[
https://issues.apache.org/jira/browse/HADOOP-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12601832#action_12601832
]
Jothi Padmanabhan commented on HADOOP-3478:
-------------------------------------------
bq. Given that it is quite hard to sort by MapId for a given location
Actually, we do not need to sort the MapIds for a given location. We just need
to ensure that the order of map fetches is enforced -- All reducers fetch the
maps in the same order as any other. We could do this without sorting.
bq.If there are a lot of unfinished mappers, then we should not do aggressive
mapper re-executions.
Yes. This was proposed as a solution in HADOOP-3327. The JobTracker should use
the number of unfinished mappers as one of the criteria for deciding whether
maps should be re-executed aggressively or not.
bq. If reducers have a lot of un-fetched map outputs, they can wait for a
longer period time before re-fetch the map outputs that failed to fetcher
previously.
The problem with this approach is that, if the map is faulty, it takes a long
time to detect and then re-execute it. If the detection is done earlier, the
map will likely finish re-execution by the time the reducer fetches other map
outputs and the over all time is minimized.
> The algorithm to decide map re-execution on fetch failures can be improved
> --------------------------------------------------------------------------
>
> Key: HADOOP-3478
> URL: https://issues.apache.org/jira/browse/HADOOP-3478
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Jothi Padmanabhan
>
> The algorithm to decide map re-execution on fetch failures can be improved.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
