Shay Rojansky created SPARK-2946:
------------------------------------
Summary: Allow specifying * for --num-executors in YARN
Key: SPARK-2946
URL: https://issues.apache.org/jira/browse/SPARK-2946
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 1.0.0
Environment: Ubuntu precise, on YARN (CDH 5.1.0)
Reporter: Shay Rojansky
Priority: Minor
It would be useful to allow specifying --num-executors * when submitting jobs
to YARN, and to have Spark automatically determine how many total cores are
available in the cluster by querying YARN.
Our scenario is multiple users running research batch jobs. We never want to
have a situation where cluster resources aren't being used, so ideally users
would specify * and let YARN scheduling and preemption ensure fairness.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
