[
https://issues.apache.org/jira/browse/MAPREDUCE-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe reassigned MAPREDUCE-3927:
-------------------------------------
Assignee: Bhallamudi Venkata Siva Kamesh
Overall looks good. Couple of comments:
In Job.java, this code:
{code}
if (reportedAfterCompletion
&& status.getState() == JobStatus.State.SUCCEEDED) {
mapProgress = mapProgress != 1.0f ? 1.0f : mapProgress;
reduceProgress = reduceProgress != 1.0f ? 1.0f : reduceProgress;
}
{code}
would be better written as:
{code}
if (reportedAfterCompletion
&& status.getState() == JobStatus.State.SUCCEEDED) {
mapProgress = 1.0f;
reduceProgress = 1.0f;
}
{code}
Also I noticed that when one or more map task fails (after max attempts have
failed), the map completion percentage of the job isn't accurate until the job
completes. (e.g.: reducer progress shows we're well past the shuffle stage
even though map progress is <100%). If that were done properly, I'm not sure
we'd need the SUCCEEDED == progress 1.0 fixup in Job.java.
> Shuffle hang when set map.failures.percent
> ------------------------------------------
>
> Key: MAPREDUCE-3927
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3927
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0-alpha, 2.0.1-alpha, trunk
> Reporter: MengWang
> Assignee: Bhallamudi Venkata Siva Kamesh
> Priority: Critical
> Labels: patch
> Attachments: MAPREDUCE-3927-1.patch, MAPREDUCE-3927.patch,
> MAPREDUCE-3927.patch
>
>
> When set mapred.max.map.failures.percent and there does have some failed
> maps, then shuffle will hang
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
