Repository: spark Updated Branches: refs/heads/master 0349b5b43 -> 67580f1f5
[SPARK-10332] [CORE] Fix yarn spark executor validation >From Jira: Running spark-submit with yarn with number-executors equal to 0 when not using dynamic allocation should error out. In spark 1.5.0 it continues and ends up hanging. yarn.ClientArguments still has the check so something else must have changed. spark-submit --master yarn --deploy-mode cluster --class org.apache.spark.examples.SparkPi --num-executors 0 .... spark 1.4.1 errors with: java.lang.IllegalArgumentException: Number of executors was 0, but must be at least 1 (or 0 if dynamic executor allocation is enabled). Author: Holden Karau <[email protected]> Closes #8580 from holdenk/SPARK-10332-spark-submit-to-yarn-executors-0-message. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/67580f1f Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/67580f1f Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/67580f1f Branch: refs/heads/master Commit: 67580f1f574d272af3712fd91458f3c87368c2e4 Parents: 0349b5b Author: Holden Karau <[email protected]> Authored: Thu Sep 3 09:30:54 2015 +0100 Committer: Sean Owen <[email protected]> Committed: Thu Sep 3 09:30:54 2015 +0100 ---------------------------------------------------------------------- .../main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala | 3 +++ 1 file changed, 3 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/67580f1f/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala ---------------------------------------------------------------------- diff --git a/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala b/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala index 4f42ffe..54f62e6 100644 --- a/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala +++ b/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala @@ -96,6 +96,9 @@ private[spark] class ClientArguments(args: Array[String], sparkConf: SparkConf) } numExecutors = initialNumExecutors + } else { + val numExecutorsConf = "spark.executor.instances" + numExecutors = sparkConf.getInt(numExecutorsConf, numExecutors) } principal = Option(principal) .orElse(sparkConf.getOption("spark.yarn.principal")) --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
