[
https://issues.apache.org/jira/browse/HADOOP-5985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716808#action_12716808
]
Aaron Kimball commented on HADOOP-5985:
---------------------------------------
Here's a first stab at an algorithm to fix this:
* Let's define the "home stretch" of shuffling, from the perspective of a
reducer, as the last X% of the map shards it needs to fetch. Maybe that's 3% or
4%.
* If during the home stretch, a map task enters the penalty box and stays there
for some number of seconds (let's say 30), the reducer can send a message back
to the JobTracker voting that the task be marked as "failed for being too slow."
* The JobTracker requires a quorum of still-shuffling reducers to vote this
way. The quorum might be, say, 10%. This way if only two reducers are still
waiting, either one of them is a quorum by itself. But if 40 reducers are
shuffling, it will take the consensus of a few reducers that a mapper is indeed
too slow.
* If enough votes are given, the JobTracker blacklists a TaskTracker from the
job, marks all its map task attempts as failed, and restarts all the map tasks
from that node, on other nodes.
> A single slow (but not dead) map TaskTracker impedes MapReduce progress
> -----------------------------------------------------------------------
>
> Key: HADOOP-5985
> URL: https://issues.apache.org/jira/browse/HADOOP-5985
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.18.3
> Reporter: Aaron Kimball
>
> We see cases where there may be a large number of mapper nodes running many
> tasks (e.g., a thousand). The reducers will pull 980 of the map task
> intermediate files down, but will be unable to retrieve the final
> intermediate shards from the last node. The TaskTracker on that node returns
> data to reducers either slowly or not at all, but its heartbeat messages make
> it back to the JobTracker -- so the JobTracker doesn't mark the tasks as
> failed. Manually stopping the offending TaskTracker works to migrate the
> tasks to other nodes, where the shuffling process finishes very quickly. Left
> on its own, it can take hours to unjam itself otherwise.
> We need a mechanism for reducers to provide feedback to the JobTracker that
> one of the mapper nodes should be regarded as lost.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
