[
https://issues.apache.org/jira/browse/BIGTOP-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13763465#comment-13763465
]
jay vyas commented on BIGTOP-1065:
----------------------------------
Actually, looking deeper I see the purpose of the MR2 comment: The JobTracker
is actually ephemeral, so any state issues which might arise really should be
gone. In any case, I guess the JobHistoryServer may have some such state
issues, even still. But its an important thing to note: Stress tests in MR2
will have to test different components, probably the ones in MR2 which have
more likeihood of persisting for a long time.
> Stress Tests
> -------------
>
> Key: BIGTOP-1065
> URL: https://issues.apache.org/jira/browse/BIGTOP-1065
> Project: Bigtop
> Issue Type: Test
> Reporter: jay vyas
> Priority: Minor
>
> There are issues in the past where the JobTracker has failures because of so
> called "memory-leaks" , or infinitely growing objects, unclean tmp/ folders
> from jobs, etc.
> Thus, the proposal here is a lighteweight job which can be run rapidly,
> several 1000 times, which confirms that jobtracker state does not grow out of
> bounds or infinitely with respect to number of tasks/jobs run/submitted.
> IMPLEMENTATION PROPOSAL:
> Some simple starts would be to :
> - run word count, or a the sleep job 100 or 1000 times or 10,000 times.
> - create and delete the same file over and over again several thousand times
> to see if filesystem consistency is maintained
> To start, I'd like to add all these tests in a single module , under
> test-executions/stress/. Then later we could shard it out in another way.
> UPDATE:
> As per comments below, just noting that although phrased in terms of
> "JobTracker", the spirit of this ticket is to be applicable in both mr1 and
> mr2, since in either case, the purpose is to test the impact that several
> 100/1000 mapreduce job runs has over time and confirm that tmp dirs, objects
> in memory, etc are all managed and lifecycled properly .
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
