GitHub user xuchenCN opened a pull request:
https://github.com/apache/spark/pull/7128
[SPARK-8374] [YARN] Job frequently hangs after YARN preemption
Issue description
[SPARK-8374](https://issues.apache.org/jira/browse/SPARK-8374)
Application starve because YARN scheduler preemption.
This PR fix two issues
1. Application starve because CoarseGrainedSchedulerBackend.removeExecutor
is happened before HeartbeatReceiver.killExecutorThread
2.SpakrContext.killExecutor less executors than before BUT
HeartbeatReceiver just want to expire the executor and expect new one
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xuchenCN/spark SPARK-8374.1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/7128.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #7128
----
commit 5c532948b125b6fd9a1e1ca48d1a7ebeb2b5f648
Author: xuchenCN <[email protected]>
Date: 2015-06-30T13:52:59Z
[SPARK-8374] [YARN] Job frequently hangs after YARN preemption
commit 316ac2e1e406e5e6401c29117f1ccbe302e64954
Author: xuchenCN <[email protected]>
Date: 2015-06-30T13:53:55Z
[SPARK-8374] [YARN] Job frequently hangs after YARN preemption
commit 5357e1363b6e3c192fc9ac64e8e9aca888618d9a
Author: xuchenCN <[email protected]>
Date: 2015-06-30T13:54:21Z
[SPARK-8374] [YARN] Job frequently hangs after YARN preemption
commit 0238754389ec1830510f4dabc1b033b6364aefb5
Author: xuchenCN <[email protected]>
Date: 2015-06-30T13:54:36Z
[SPARK-8374] [YARN] Job frequently hangs after YARN preemption
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
