[
https://issues.apache.org/jira/browse/SPARK-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-2946.
------------------------------
Resolution: Duplicate
Looks like a subset of the larger request to be able to ask for all of
everything in a YARN cluster.
> Allow specifying * for --num-executors in YARN
> ----------------------------------------------
>
> Key: SPARK-2946
> URL: https://issues.apache.org/jira/browse/SPARK-2946
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 1.0.0
> Environment: Ubuntu precise, on YARN (CDH 5.1.0)
> Reporter: Shay Rojansky
> Priority: Minor
>
> It would be useful to allow specifying --num-executors * when submitting jobs
> to YARN, and to have Spark automatically determine how many total cores are
> available in the cluster by querying YARN.
> Our scenario is multiple users running research batch jobs. We never want to
> have a situation where cluster resources aren't being used, so ideally users
> would specify * and let YARN scheduling and preemption ensure fairness.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
