JobConf option for minimum progress threshold before reducers are assigned
--------------------------------------------------------------------------
Key: HADOOP-5271
URL: https://issues.apache.org/jira/browse/HADOOP-5271
Project: Hadoop Core
Issue Type: Improvement
Reporter: Tim Williamson
A specific sub-case of the general priority inversion problem noted in
HADOOP-4557 is when many lower priority jobs are submitted and are waiting for
mappers to free up. Even though they haven't actually done any work, they will
be assigned any free reducers. If a higher priority job is submitted, priority
inversion results not just due to the lower priority tasks that are in the
midst of completing, but also due to the ones that haven't yet started but have
claimed all the free reducers. A simple workaround is to require a job to
complete some useful work before assigning it a reducer. This can be done in a
tunable and backwards compatible manner by adding a "minimum map progress
percentage before assigning a reducer" option to the JobConf. Setting this to
0 would eliminate the common case above, and setting it to 100 would
technically eliminate the inversion of HADOOP-4557, though likely at an
unacceptably high cost.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
