GitHub user andrewor14 opened a pull request:
https://github.com/apache/spark/pull/7532
[WIP][SPARK-4751] Dynamic allocation in standalone mode
Dynamic allocation is a feature that allows a Spark application to scale
the number of executors up and down dynamically based on the workload. Support
was first introduced in YARN since 1.2, and then extended to Mesos
coarse-grained mode recently. Today, it is finally supported in standalone mode
as well.
I tested this locally and it works as expected. This is WIP because unit
tests are coming.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/andrewor14/spark standalone-da
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/7532.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #7532
----
commit 1c2d0010afe1df9ca79e846e6275d98cce006713
Author: Andrew Or <[email protected]>
Date: 2015-07-20T07:57:49Z
First working implementation
This adds two messages between the AppClient on the driver and
the standalone Master: request and kill. The scheduling on the
Master side handles both applications that explicitly set
`spark.executor.cores` and those that did not.
TODO: clean up shuffle files on application exit and unit tests.
commit 5807eb2397de22250dbcccf92928843fde81536d
Author: Andrew Or <[email protected]>
Date: 2015-07-20T08:04:52Z
Clean up shuffle files after application exits
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
