[
https://issues.apache.org/jira/browse/BIGTOP-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jay vyas updated BIGTOP-1065:
-----------------------------
Description:
There are issues in the past where the JobTracker has failures because of so
called "memory-leaks" , or infinitely growing objects, unclean tmp/ folders
from jobs, etc.
Thus, the proposal here is a lighteweight job which can be run rapidly, several
1000 times, which confirms that jobtracker state does not grow out of bounds or
infinitely with respect to number of tasks/jobs run/submitted.
IMPLEMENTATION PROPOSAL:
Some simple starts would be to :
- run word count, or a the sleep job 100 or 1000 times or 10,000 times.
- create and delete the same file over and over again several thousand times to
see if filesystem consistency is maintained
To start, I'd like to add all these tests in a single module , under
test-executions/stress/. Then later we could shard it out in another way.
UPDATE:
As per comments below, just noting that although phrased in terms of
"JobTracker", the spirit of this ticket is to be applicable in both mr1 and
mr2, since in either case, the purpose is to test the impact that several
100/1000 mapreduce job runs has over time and confirm that tmp dirs, objects in
memory, etc are all managed and lifecycled properly .
was:
There are issues in the past where the JobTracker has failures because of so
called "memory-leaks" , or infinitely growing objects, unclean tmp/ folders
from jobs, etc.
Thus, the proposal here is a lighteweight job which can be run rapidly, several
1000 times, which confirms that jobtracker state does not grow out of bounds or
infinitely with respect to number of tasks/jobs run/submitted.
IMPLEMENTATION PROPOSAL:
Some simple starts would be to :
- run word count, or a the sleep job 100 or 1000 times or 10,000 times.
- create and delete the same file over and over again several thousand times to
see if filesystem consistency is maintained
To start, I'd like to add all these tests in a single module , under
test-executions/stress/. Then later we could shard it out in another way.
UPDATE:
As per comments below, the spirit of this ticket is to be applicable in both
mr1 and mr2, since in either case, the purpose is to test the impact that
several 100/1000 mapreduce job runs has over time and confirm that tmp dirs,
objects in memory, etc are all managed and lifecycled properly .
> Stress Tests
> -------------
>
> Key: BIGTOP-1065
> URL: https://issues.apache.org/jira/browse/BIGTOP-1065
> Project: Bigtop
> Issue Type: Test
> Reporter: jay vyas
> Priority: Minor
>
> There are issues in the past where the JobTracker has failures because of so
> called "memory-leaks" , or infinitely growing objects, unclean tmp/ folders
> from jobs, etc.
> Thus, the proposal here is a lighteweight job which can be run rapidly,
> several 1000 times, which confirms that jobtracker state does not grow out of
> bounds or infinitely with respect to number of tasks/jobs run/submitted.
> IMPLEMENTATION PROPOSAL:
> Some simple starts would be to :
> - run word count, or a the sleep job 100 or 1000 times or 10,000 times.
> - create and delete the same file over and over again several thousand times
> to see if filesystem consistency is maintained
> To start, I'd like to add all these tests in a single module , under
> test-executions/stress/. Then later we could shard it out in another way.
> UPDATE:
> As per comments below, just noting that although phrased in terms of
> "JobTracker", the spirit of this ticket is to be applicable in both mr1 and
> mr2, since in either case, the purpose is to test the impact that several
> 100/1000 mapreduce job runs has over time and confirm that tmp dirs, objects
> in memory, etc are all managed and lifecycled properly .
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
