[
https://issues.apache.org/jira/browse/MAPREDUCE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305180#comment-14305180
]
Jason Lowe commented on MAPREDUCE-5718:
---------------------------------------
The issue is that the OutputCommitter is user-specified code. It may not be
doing filesystem operations at all during the commit (e.g.: committing to a
database, a REST API, etc.), and that procedure may not be restartable without
the chance of corrupting or losing data. Yes, FileOutputCommitter's commit
procedure is something that can be restarted, but jobs are not required to use
FileOutputCommitter nor dump their output in a file at all. That's why
MAPREDUCE-5485 was filed, because to do this safely the framework needs an
indication from the output committer whether or not commit is a restartable
procedure.
> MR job will fail after commit fail
> ----------------------------------
>
> Key: MAPREDUCE-5718
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5718
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am
> Affects Versions: 2.3.0, 2.6.0
> Reporter: Karthik Kambatla
> Assignee: Yang Hao
> Fix For: 2.6.0
>
> Attachments: MAPREDUCE-5718.v2.patch, mr-5718-0.patch
>
>
> when any of this happens:
> * While testing RM HA, if the RM fails over while an MR AM is in the middle
> of a commit,
> * When testing preempting, if the MR AM fails over during the middle of a
> commit
> the subsequent AM gets spawned but dies with a diagnostic message - "We
> crashed durring a commit".
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
