GitHub user janewangfb opened a pull request:
https://github.com/apache/spark/pull/18492
[SPARK-19326] Speculated task attempts do not get launched in few scenarios
## What changes were proposed in this pull request?
Add a new listener event when a speculative task is created and notify it
to ExecutorAllocationManager for requesting more executor.
## How was this patch tested?
- Added Unittests.
- For the test snippet in the jira:
val n = 100
val someRDD = sc.parallelize(1 to n, n)
someRDD.mapPartitionsWithIndex( (index: Int, it: Iterator[Int]) => {
if (index == 1) {
Thread.sleep(Long.MaxValue) // fake long running task(s)
}
it.toList.map(x => index + ", " + x).iterator
}).collect
With this code change, spark indicates 101 jobs are running (99 succeeded,
2 running and 1 is speculative job)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/janewangfb/spark speculated_task_not_launched
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18492.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18492
----
commit 9d4886e621d17b8e5871e8beebbcf133f009b14b
Author: Jane Wang <[email protected]>
Date: 2017-06-29T18:50:33Z
add speculative job for executor calculation
commit d4dd56c7ef83e3061641e40ae00ef7389d7cee95
Author: Jane Wang <[email protected]>
Date: 2017-06-29T21:49:56Z
Merge branch 'master' into speculated_task_not_launched
commit 210ba70fdad5b1f4355deed1b84feeec6a265281
Author: Jane Wang <[email protected]>
Date: 2017-06-30T00:06:13Z
Add Unittests
commit cb2840496723326796cd2b36e6aa515e836bd3c1
Author: Jane Wang <[email protected]>
Date: 2017-06-30T00:41:00Z
remove debug lines
commit 41170df18bd70f58ff9c4562bc16eddbef5ad191
Author: Jane Wang <[email protected]>
Date: 2017-06-30T00:52:37Z
remove debug
commit 9784b4e6e30066fa03082ba12262fd8b0e2a5694
Author: Jane Wang <[email protected]>
Date: 2017-06-30T00:58:03Z
remove debug lines
commit 5331ab93d80515ce46bbe766036ef9dd8fef6e64
Author: Jane Wang <[email protected]>
Date: 2017-06-30T17:28:23Z
Merge branch 'master' into speculated_task_not_launched
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
