GitHub user li-zhihui opened a pull request:
https://github.com/apache/spark/pull/1525
Fix race condition at SchedulerBackend.isReady in standalone mode
In SPARK-1946(PR #900), configuration
<code>spark.scheduler.minRegisteredExecutorsRatio</code> was introduced.
However, in standalone mode, there is a race condition where isReady() can
return true because totalExpectedExecutors has not been correctly set.
Because expected executors is uncertain in standalone mode, the PR try to
use CPU cores(<code>--total-executor-cores</code>) as expected resources to
judge whether SchedulerBackend is ready.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/li-zhihui/spark fixre4s
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1525.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1525
----
commit 8b54316c77d086ea3454419ebba92003707bbd76
Author: li-zhihui <[email protected]>
Date: 2014-07-22T08:15:40Z
Fix race condition at SchedulerBackend.isReady in standalone mode
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
