[
https://issues.apache.org/jira/browse/SPARK-15917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15341360#comment-15341360
]
Jonathan Taws edited comment on SPARK-15917 at 6/21/16 8:24 AM:
----------------------------------------------------------------
If I run the following command : {{spark-shell --master <master-url> --conf
spark.cores.max=2 --conf spark.executor.instances=2 --executor-cores 2
--executor-memory 4g}}, I get only one executor, as there are only 2 cores
available (same with {{--num-executors}} instead of
{{spark.executor.instances}}).
{{spark.cores.max}} does its job right here, and it just seems like the
executor instances property is just being discarded here.
If anything, adding a support for a {{--num-executors}}-like property in
standalone would allow to be consistent across deployment methods and, in my
experience, make the beginner experience slightly easier.
was (Author: jonathantaws):
If I run the following command : {{spark-shell \--master <master-url> \--conf
spark.cores.max=2 \--conf spark.executor.instances=2 \--executor-cores 2
\--executor-memory 4g}}, I get only one executor, as there are only 2 cores
available (same with {{--num-executors}} instead of
{{spark.executor.instances}}).
{{spark.cores.max}} does its job right here, and it just seems like the
executor instances property is just being discarded here.
If anything, adding a support for a {{--num-executors}}-like property in
standalone would allow to be consistent across deployment methods and, in my
experience, make the beginner experience slightly easier.
> Define the number of executors in standalone mode with an easy-to-use property
> ------------------------------------------------------------------------------
>
> Key: SPARK-15917
> URL: https://issues.apache.org/jira/browse/SPARK-15917
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, Spark Shell, Spark Submit
> Affects Versions: 1.6.1
> Reporter: Jonathan Taws
> Priority: Minor
>
> After stumbling across a few StackOverflow posts around the issue of using a
> fixed number of executors in standalone mode (non-YARN), I was wondering if
> we could not add an easier way to set this parameter than having to resort to
> some calculations based on the number of cores and the memory you have
> available on your worker.
> For example, let's say I have 8 cores and 30GB of memory available :
> - If no option is passed, one executor will be spawned with 8 cores and 1GB
> of memory allocated.
> - However, if I want to have only *2* executors, and to use 2 cores and 10GB
> of memory per executor, I will end up with *3* executors (as the available
> memory will limit the number of executors) instead of the 2 I was hoping for.
> Sure, I can set {{spark.cores.max}} as a workaround to get exactly what I
> want, but would it not be easier to add a {{--num-executors}}-like option to
> standalone mode to be able to really fine-tune the configuration ? This
> option is already available in YARN mode.
> From my understanding, I don't see any other option lying around that can
> help achieve this.
> This seems to be slightly disturbing for newcomers, and standalone mode is
> probably the first thing anyone will use to just try out Spark or test some
> configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
