GitHub user pwoody opened a pull request:
https://github.com/apache/spark/pull/16959
[SPARK-19631][CORE] OutputCommitCoordinator should not allow commits for
already failed tasks
## What changes were proposed in this pull request?
Previously it was possible for there to be a race between a task failure
and committing the output of a task. This ensures that any previously failed
task attempts cannot enter the commit protocol.
## How was this patch tested?
Added a unit test
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pwoody/spark pw/recordFailuresForCommitter
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16959.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16959
----
commit ce17e02fbebbac6a3e4c92e5a9ec8b2a59879f20
Author: Patrick Woody <[email protected]>
Date: 2017-02-16T15:03:35Z
Record failed attempts in the OutputCommitCoordinator
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
