Task Initialization should be delayed till when a job can be run
----------------------------------------------------------------
Key: MAPREDUCE-1783
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1783
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: contrib/fair-share
Affects Versions: 0.20.1
Reporter: Ramkumar Vadali
The FairScheduler task scheduler uses PoolManager to impose limits on the
number of jobs that can be running at a given time. However, jobs that are
submitted are initiaiized immediately by EagerTaskInitializationListener by
calling JobInProgress.initTasks. This causes the job split file to be read into
memory. The split information is not needed until the number of running jobs is
less than the maximum specified. If the amount of split information is large,
this leads to unnecessary memory pressure on the Job Tracker.
To ease memory pressure, FairScheduler can use another implementation of
JobInProgressListener that is aware of PoolManager limits and can delay task
initialization until the number of running jobs is below the maximum.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
