[ https://issues.apache.org/jira/browse/SPARK-19790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15897863#comment-15897863 ]
Steve Loughran commented on SPARK-19790: ---------------------------------------- The only time a task output committer should be making observable state changes is during the actual commit operation. If it is doing things before that commit operation, that's a bug in that it doesn't meet the goal "committer". The Hadoop output committer has two stages here: the FileOutputFormat work and then rename of files; together they are not a transaction, but on a real filesystem: fast The now deleted DirectOutputCommitter was doing things as it went along —but that's why it got pulled. That leaves: the Hadoop Output Committer committing work on object stores which implement rename() as a copy, hence slow and with a large enough failure window. HADOOP-13786 is going to make that window very small indeed, at least for job completion. One thing to look at here is the {{org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter}} protocol, where the committer can be asked whether or not it supports recovery, as well as {{isCommitJobRepeatable}} to probe for a job commit being repeatable even if it fails partway through. The committer gets to implement its policy there. > OutputCommitCoordinator should not allow another task to commit after an > ExecutorFailure > ---------------------------------------------------------------------------------------- > > Key: SPARK-19790 > URL: https://issues.apache.org/jira/browse/SPARK-19790 > Project: Spark > Issue Type: Bug > Components: Scheduler > Affects Versions: 2.1.0 > Reporter: Imran Rashid > > The OutputCommitCoordinator resets the allowed committer when the task fails. > > https://github.com/apache/spark/blob/8aa560b75e6b083b2a890c52301414285ba35c3d/core/src/main/scala/org/apache/spark/scheduler/OutputCommitCoordinator.scala#L143 > However, if a task fails because of an ExecutorFailure, we actually have no > idea what the status is of the task. The task may actually still be running, > and perhaps successfully commit its output. By allowing another task to > commit its output, there is a chance that multiple tasks commit, which can > result in corrupt output. This would be particularly problematic when > commit() is an expensive operation, eg. moving files on S3. > For other task failures, we can allow other tasks to commit. But with an > ExecutorFailure, its not clear what the right thing to do is. The only safe > thing to do may be to fail the job. > This is related to SPARK-19631, and was discovered during discussion on that > PR https://github.com/apache/spark/pull/16959#discussion_r103549134 -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org