GitHub user jasonmoore2k opened a pull request:
https://github.com/apache/spark/pull/12751
[SPARK-14915] [CORE] Don't re-queue a task if another attempt has already
succeeded
## What changes were proposed in this pull request?
Don't re-queue a task if another attempt has already succeeded. This
currently happens when a speculative task is denied from committing the result
due to another copy of the task already having succeeded.
## How was this patch tested?
I'm running a job which has a fair bit of skew in the processing time
across the tasks for speculation to trigger in the last quarter (default
settings), causing many commit denied exceptions to be thrown. Previously,
these tasks were then being retried over and over again until the stage
possibly completes (despite using compute resources on these superfluous
tasks). With this change (applied to the 1.6 branch), they no longer retry and
the stage completes successfully without these extra task attempts.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jasonmoore2k/spark SPARK-14915
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12751.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12751
----
commit a3e69c0234cb97c73ec57e5fbe30fde89a68899b
Author: Jason Moore <[email protected]>
Date: 2016-04-28T02:14:15Z
[SPARK-14915] [CORE] Don't re-queue a task if another attempt has already
succeeded
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
