GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/11205
[SPARK-11334][Core] Handle maximum task failure situation in dynamic
allocation
Currently there're two problems in dynamic allocation when maximum task
failure is met:
1. Number of running tasks will possibly be negative, which will affect the
calculation of needed executors.
2. Executors may never be idle. Currently we use the executor to tasks
mapping relation to identify the status of executors, in maximum task failure
situation, some `TaskEnd` events may never be delivered, which makes the
related executor always be busy.
This patch tries to fix these two issues, please review, thanks a lot.
CC @andrewor14 and @tgravescs .
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jerryshao/apache-spark SPARK-11334
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11205.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11205
----
commit 966eb891ba8da1936412b9894335ee7ce41b4c4b
Author: jerryshao <[email protected]>
Date: 2016-02-15T06:32:01Z
Fix maximum task failure issue in dynamic allocation
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
