GitHub user andrewor14 opened a pull request:
https://github.com/apache/spark/pull/3002
[SPARK-4138][SPARK-4139] Improve dynamic allocation settings
This should be merged after #2746 (SPARK-3795).
**SPARK-4138**. If the user sets both the number of executors and
`spark.dynamicAllocation.enabled`, we should throw an exception.
**SPARK-4139**. If the user sets `spark.dynamicAllocation.enabled`, we
should use the max number of executors as the starting number of executors
because the first job is likely to run immediately after application startup.
If the latter is not set, throw an exception.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/andrewor14/spark yarn-set-executors
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3002.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3002
----
commit 60af3f664c492831eb77004238e6a0890b901697
Author: Andrew Or <[email protected]>
Date: 2014-10-29T20:29:44Z
Guard against incompatible settings of number of executors
Also, start the number of executors at the max if dynamic
allocation is enabled.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
