GitHub user rdblue opened a pull request:
https://github.com/apache/spark/pull/13338
SPARK-13723: Change behavior of --num-executors with dynamic allocation.
## What changes were proposed in this pull request?
This changes the behavior of --num-executors and spark.executor.instances
when using dynamic allocation. Instead of turning dynamic allocation off, it
uses the value for the initial number of executors.
This changes was discussed on
[SPARK-13723|https://issues.apache.org/jira/browse/SPARK-13723]. I highly
recommend using it while we can change the behavior for 2.0.0. In practice, the
1.x behavior causes unexpected behavior for users (it is not clear that it
disables dynamic allocation) and wastes cluster resources because users rarely
notice the log message.
## How was this patch tested?
This patch updates tests and adds a test for
Utils.getDynamicAllocationInitialExecutors.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rdblue/spark
SPARK-13723-num-executors-with-dynamic-allocation
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13338.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13338
----
commit 5f6954943ed9278b3ebe6b871f20b14f66efd2f6
Author: Ryan Blue <[email protected]>
Date: 2016-03-04T21:44:13Z
SPARK-13723: Change behavior of --num-executors with dynamic allocation.
Previously, using --num-executors or spark.executor.instances with
dynamic allocation enabled would disable dynamic allocation. This
updates the behavior so these settings change the initial number of
executors and do not disable dynamic allocation.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
