[
https://issues.apache.org/jira/browse/HADOOP-5271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12674314#action_12674314
]
Tim Williamson commented on HADOOP-5271:
----------------------------------------
Sweet: "mapred.reduce.slowstart.completed.maps" looks like essentially the same
idea, and using number of completed map tasks (as opposed to progress) is a
better approach. Not sure if there is any utility in being able to set this
per-job or not, but from my use-case perspective, this JIRA could be closed as
a duplicate.
> JobConf option for minimum progress threshold before reducers are assigned
> --------------------------------------------------------------------------
>
> Key: HADOOP-5271
> URL: https://issues.apache.org/jira/browse/HADOOP-5271
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Tim Williamson
> Attachments: HADOOP-5271.patch
>
>
> A specific sub-case of the general priority inversion problem noted in
> HADOOP-4557 is when many lower priority jobs are submitted and are waiting
> for mappers to free up. Even though they haven't actually done any work,
> they will be assigned any free reducers. If a higher priority job is
> submitted, priority inversion results not just due to the lower priority
> tasks that are in the midst of completing, but also due to the ones that
> haven't yet started but have claimed all the free reducers. A simple
> workaround is to require a job to complete some useful work before assigning
> it a reducer. This can be done in a tunable and backwards compatible manner
> by adding a "minimum map progress percentage before assigning a reducer"
> option to the JobConf. Setting this to 0 would eliminate the common case
> above, and setting it to 100 would technically eliminate the inversion of
> HADOOP-4557, though likely at an unacceptably high cost.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
