[
https://issues.apache.org/jira/browse/MAPREDUCE-2201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935900#action_12935900
]
Joydeep Sen Sarma commented on MAPREDUCE-2201:
----------------------------------------------
there's something more to it for sure. it seems that after preempting for job X
- the freed up slots were not used for job X - they were used for other jobs
(including those just preempted). i will file a separate jira with details.
that seems like bigger issue.
yeah - two heartbeats are required to kill and reschedule. screwed up.
> Quicker preemption causes excessive preemption in FairScheduler
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2201
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2201
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: Joydeep Sen Sarma
>
> One problem we are seeing is where FairScheduler repeatedly preempts for the
> same job. This is presumably because our preemption interval is set to a low
> number (1 minute). FS queues up N tasks to be killed - but in 1 min it is not
> able to kill and schedule new tasks on all these slots. As a result, after 1
> min - it again preempts a whole bunch of tasks.
> We could (and probably will) workaround this by increasing the preemption
> interval. However - this gives us a hard tradeoff between accurate preemption
> and timely preemption. Not good. Ideally we want to make the first set of
> preemptions quickly (to provide responsive behavior to new jobs for example)
> - but wait (to make sure that the kill actions have actually been processed)
> thereafter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
