[
https://issues.apache.org/jira/browse/MAPREDUCE-1967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892342#action_12892342
]
Doug Cutting commented on MAPREDUCE-1967:
-----------------------------------------
Perhaps this could be generalized so that there's a set of exceptions that are
considered job-killing? Quota exceptions might be in the set by default, but
others might be added.
> When a reducer fails on DFS quota, the job should fail immediately
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1967
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1967
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Dick King
>
> Suppose an M/R job has so much output that the user is certain to exceed hir
> quota. Then some of the reducers will succeed but the job will get into a
> state where the remaining reducers squabble over the remaining space. The
> remaining reducers will nibble at the remaining space, and finally one
> reducer will fail on quota. Its output file will be erased, and the other
> reducers will collectively consume that space until one of _them_ fails on
> quota. Since the incomplete reducer that fails on quota is "chosen"
> randomly, the tasks will accumulate their failures at similar rates, and the
> system will have made a substantial futile investment.
> I would like to say that if a single reducer fails on DFS quota, the job
> should be failed. There may be a corner case that induces us to think that
> we shouldn't be quite this stringent, but at least we shouldn't have to await
> four failures by one task before shutting the job down.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
