Jason Lowe created MAPREDUCE-4831:
-------------------------------------
Summary: Task commit can occur more than once due to AM retries
Key: MAPREDUCE-4831
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4831
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am
Affects Versions: 2.0.0-alpha, 0.23.0
Reporter: Jason Lowe
Priority: Critical
If a task attempt begins committing but the AM crashes before the task attempt
completes then we could end up having the task commit again when the AM is
relaunched. The subsequent AM attempt will not see the task having completed,
so it will re-run the task and it will commit again. The output committer is
user code, and the task commit may not be something repeatable. Therefore we
should treat an AM crash during a task attempt commit the same as we do for a
commit failure by the task attempt, i.e.: the task should fail since we do not
know how to recover from a commit failure.
This is similar to MAPREDUCE-4813, as this involves commit at the task level
and that involves commit at the job-level.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
