[
https://issues.apache.org/jira/browse/HADOOP-5572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12700826#action_12700826
]
Owen O'Malley commented on HADOOP-5572:
---------------------------------------
The simple change would help speculative execution, since it is a closer
approximation of the actual progress. Currently, speculative on maps often
doesn't happen when it should because the map is stuck at 100%.
I don't think we should be tracking the progress as each record is consumed in
the merge sort. That will be too much overhead, unless you are very careful. If
you want to weight the progress by how many bytes are in each segment, that
would provide a better match. Do make sure that you consider how combiners will
affect the progress meters.
> The map progress value should have a separate phase for doing the final sort.
> -----------------------------------------------------------------------------
>
> Key: HADOOP-5572
> URL: https://issues.apache.org/jira/browse/HADOOP-5572
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Ravi Gummadi
>
> Currently, the final spill and sort doesn't record any progress while it
> runs, leading to the perception that the map is done, but "stuck".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
