GitHub user tgravescs opened a pull request:
https://github.com/apache/spark/pull/10951
[SPARK-11701] dynamic allocation and speculation active task accounting
wrong
I am using dynamic container allocation and speculation and am seeing
issues with the active task accounting. The Executor UI still shows active
tasks on the an executor but the job/stage is all completed. I think its also
affecting the dynamic allocation being able to release containers because it
thinks there are still tasks.
There are multiple issues with this:
- If the task end for tasks (in this case probably because of speculation)
comes in after the stage is finished, then the
DAGScheduler.handleTaskCompletion will skip the task completion event
- race in the task committer where it was ignoring tasks is already
committed in some cases.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tgravescs/spark SPARK-11701
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10951.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10951
----
commit 416708e4de04ad6dcc4ce767b42865481ad68dff
Author: Tom Graves <[email protected]>
Date: 2016-01-26T15:31:20Z
[SPARK-11701] YARN - dynamic allocation and speculation active task
accounting wrong
commit 2ab1c902c5e792ad788a673959772ce8f4194124
Author: Thomas Graves <[email protected]>
Date: 2016-01-27T19:23:47Z
Add test for DAGScheduler changes
commit 574000d89379acf385d0630a6586f0a71fef67fb
Author: Thomas Graves <[email protected]>
Date: 2016-01-27T20:06:20Z
Merge remote-tracking branch 'upstream/master' into SPARK-11701
Conflicts:
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
