GitHub user KaiXinXiaoLei opened a pull request:
https://github.com/apache/spark/pull/8945
[SPARK-10515] When killing executor, the pending replacement executors
should not be lost
If the heartbeat receiver kills executors (and new ones are not registered
to replace them), the idle timeout for the old executors will be lost (and then
change a total number of executors requested by Driver), So new ones will be
not to asked to replace them.
For example, executorsPendingToRemove=Set(1), and executor 2 is idle
timeout before a new executor is asked to replace executor 1. Then driver kill
executor 2, and sending RequestExecutors to AM. But
executorsPendingToRemove=Set(1,2), So AM doesn't allocate a executor to replace
1.
see: https://github.com/apache/spark/pull/8668
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/KaiXinXiaoLei/spark pendingexecutor
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/8945.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #8945
----
commit e559482ae4404320a1a242d0aa082f00611e02bb
Author: huleilei <[email protected]>
Date: 2015-09-30T05:16:27Z
add unit test
commit 1bdde8e88ed36194e020e773db415e65181357a9
Author: huleilei <[email protected]>
Date: 2015-09-30T06:35:51Z
add numPendingExecutors
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
