GitHub user squito opened a pull request:
https://github.com/apache/spark/pull/13826
[SPARK-16106][CORE] TaskSchedulerImpl should properly track executors added
to existing hosts
## What changes were proposed in this pull request?
TaskSchedulerImpl used to only set `newExecAvailable` when a new *host* was
added, not when a new executor was added to an existing host. It also didn't
update some internal state tracking live executors until a task was scheduled
on the executor. This patch changes it to properly update as soon as it knows
about a new executor.
## How was this patch tested?
added a unit test, ran everything via jenkins.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/squito/spark SPARK-16106_executorByHosts
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13826.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13826
----
commit 39d18eddee6f2f38663fa53770ab172cb68a6a5a
Author: Imran Rashid <[email protected]>
Date: 2016-06-21T22:19:04Z
refactor and fix tests
commit 195408be6749b18fc28bd8fdb1512c6da760b90b
Author: Imran Rashid <[email protected]>
Date: 2016-06-21T22:19:28Z
make sure executorsByHost and localityLevels updated on new executors for
same host
commit 40b8ee7db3a8bd54110a4b834ec14e5b0b8f150d
Author: Imran Rashid <[email protected]>
Date: 2016-06-21T22:35:03Z
slightly expand test
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
