Hey all, I've recently run into an issue where spark dynamicAllocation has asked for -1 executors from YARN. Unfortunately, this raises an exception that kills the executor-allocation thread and the application can't request more resources.
Has anyone seen this before? It is spurious and the application usually works, but when this gets hit it becomes unusable when getting stuck at minimum YARN resources. Stacktrace below. Thanks! -Pat 470 ERROR [2015-06-12 16:44:39,724] org.apache.spark.util.Utils: Uncaught exception in thread spark-dynamic-executor-allocation-0 471 ! java.lang.IllegalArgumentException: Attempted to request a negative number of executor(s) -1 from the cluster manager. Please specify a positive number! 472 ! at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.requestTotalExecutors(CoarseGrainedSchedulerBackend.scala:338) ~[spark-core_2.10-1.3.1.jar:1. 473 ! at org.apache.spark.SparkContext.requestTotalExecutors(SparkContext.scala:1137) ~[spark-core_2.10-1.3.1.jar:1.3.1] 474 ! at org.apache.spark.ExecutorAllocationManager.addExecutors(ExecutorAllocationManager.scala:294) ~[spark-core_2.10-1.3.1.jar:1.3.1] 475 ! at org.apache.spark.ExecutorAllocationManager.addOrCancelExecutorRequests(ExecutorAllocationManager.scala:263) ~[spark-core_2.10-1.3.1.jar:1.3.1] 476 ! at org.apache.spark.ExecutorAllocationManager.org$apache$spark$ExecutorAllocationManager$$schedule(ExecutorAllocationManager.scala:230) ~[spark-core_2.10-1.3.1.j 477 ! at org.apache.spark.ExecutorAllocationManager$$anon$1$$anonfun$run$1.apply$mcV$sp(ExecutorAllocationManager.scala:189) ~[spark-core_2.10-1.3.1.jar:1.3.1] 478 ! at org.apache.spark.ExecutorAllocationManager$$anon$1$$anonfun$run$1.apply(ExecutorAllocationManager.scala:189) ~[spark-core_2.10-1.3.1.jar:1.3.1] 479 ! at org.apache.spark.ExecutorAllocationManager$$anon$1$$anonfun$run$1.apply(ExecutorAllocationManager.scala:189) ~[spark-core_2.10-1.3.1.jar:1.3.1] 480 ! at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1618) ~[spark-core_2.10-1.3.1.jar:1.3.1] 481 ! at org.apache.spark.ExecutorAllocationManager$$anon$1.run(ExecutorAllocationManager.scala:189) [spark-core_2.10-1.3.1.jar:1.3.1] 482 ! at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_71] 483 ! at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_71] 484 ! at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_71] 485 ! at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_71] 486 ! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_71] 487 ! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_71]
