Quicker preemption causes excessive preemption in FairScheduler
---------------------------------------------------------------
Key: MAPREDUCE-2201
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2201
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: contrib/fair-share
Reporter: Joydeep Sen Sarma
One problem we are seeing is where FairScheduler repeatedly preempts for the
same job. This is presumably because our preemption interval is set to a low
number (1 minute). FS queues up N tasks to be killed - but in 1 min it is not
able to kill and schedule new tasks on all these slots. As a result, after 1
min - it again preempts a whole bunch of tasks.
We could (and probably will) workaround this by increasing the preemption
interval. However - this gives us a hard tradeoff between accurate preemption
and timely preemption. Not good. Ideally we want to make the first set of
preemptions quickly (to provide responsive behavior to new jobs for example) -
but wait (to make sure that the kill actions have actually been processed)
thereafter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
