[
https://issues.apache.org/jira/browse/HADOOP-5271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12674152#action_12674152
]
Hemanth Yamijala commented on HADOOP-5271:
------------------------------------------
Tim, in HADOOP-3136, "mapred.reduce.slowstart.completed.maps" was introduced.
The documentation reads thus:
bq. Fraction of the number of maps in the job which should be complete before
reduces are scheduled for the job.
Is this similar to what you are proposing ? This option is not in JobConf
though. So, maybe the intention is to *not* let it be overridden per job.
Arun, any thoughts ?
> JobConf option for minimum progress threshold before reducers are assigned
> --------------------------------------------------------------------------
>
> Key: HADOOP-5271
> URL: https://issues.apache.org/jira/browse/HADOOP-5271
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Tim Williamson
> Attachments: HADOOP-5271.patch
>
>
> A specific sub-case of the general priority inversion problem noted in
> HADOOP-4557 is when many lower priority jobs are submitted and are waiting
> for mappers to free up. Even though they haven't actually done any work,
> they will be assigned any free reducers. If a higher priority job is
> submitted, priority inversion results not just due to the lower priority
> tasks that are in the midst of completing, but also due to the ones that
> haven't yet started but have claimed all the free reducers. A simple
> workaround is to require a job to complete some useful work before assigning
> it a reducer. This can be done in a tunable and backwards compatible manner
> by adding a "minimum map progress percentage before assigning a reducer"
> option to the JobConf. Setting this to 0 would eliminate the common case
> above, and setting it to 100 would technically eliminate the inversion of
> HADOOP-4557, though likely at an unacceptably high cost.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
