[
https://issues.apache.org/jira/browse/SPARK-15801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15318392#comment-15318392
]
Sean Owen commented on SPARK-15801:
-----------------------------------
I get the result you get _without_ {{--num-executors}}. I've kind of forgotten
how standalone mode is supposed to work, so hopefully that is still expected
behavior. But {{--num-executors}} doesn't seem to do anything. I get 4
regardless of the value I set. CC [~vanzin] to see if that's supposed to
generate a warning or whatever.
> spark-submit --num-executors switch also works without YARN
> -----------------------------------------------------------
>
> Key: SPARK-15801
> URL: https://issues.apache.org/jira/browse/SPARK-15801
> Project: Spark
> Issue Type: Documentation
> Components: Spark Submit
> Affects Versions: 1.6.1
> Reporter: Jonathan Taws
> Priority: Minor
>
> Based on this [issue|https://issues.apache.org/jira/browse/SPARK-15781]
> regarding the SPARK_WORKER_INSTANCES property, I also found that the
> {{--num-executors}} switch documented in the spark-submit help is partially
> incorrect.
> Here's one part of the output (produced by {{spark-submit --help}}):
> {code}
> YARN-only:
> --driver-cores NUM Number of cores used by the driver, only in
> cluster mode
> (Default: 1).
> --queue QUEUE_NAME The YARN queue to submit to (Default:
> "default").
> --num-executors NUM Number of executors to launch (Default: 2).
> {code}
> Correct me if I am wrong, but the num-executors switch also works in Spark
> standalone mode *without YARN*.
> I tried by only launching a master and a worker with 4 executors specified,
> and they were all successfully spawned. The master switch pointed to the
> master's url, and not to the yarn value.
> Here's the exact command : {{spark-submit --master spark://[local
> machine]:7077 --num-executors 4 --executor-cores 2}}
> By default it is *1* executor per worker in Spark standalone mode without
> YARN, but this option enables to specify the number of executors (per worker
> ?) if, and only if, the {{--executor-cores}} switch is also set. I do believe
> it defaults to 2 in YARN mode.
> I would propose to move the option from the *YARN-only* section to the *Spark
> standalone and YARN only* section.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
