Shay Rojansky created SPARK-2946: ------------------------------------ Summary: Allow specifying * for --num-executors in YARN Key: SPARK-2946 URL: https://issues.apache.org/jira/browse/SPARK-2946 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 1.0.0 Environment: Ubuntu precise, on YARN (CDH 5.1.0) Reporter: Shay Rojansky Priority: Minor
It would be useful to allow specifying --num-executors * when submitting jobs to YARN, and to have Spark automatically determine how many total cores are available in the cluster by querying YARN. Our scenario is multiple users running research batch jobs. We never want to have a situation where cluster resources aren't being used, so ideally users would specify * and let YARN scheduling and preemption ensure fairness. -- This message was sent by Atlassian JIRA (v6.2#6252) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org