Hi, >> Attempted to request a negative number of executor(s) -663 from the cluster manager. Please specify a positive number!
This is a bug in dynamic allocation. Here is the jira- https://issues.apache.org/jira/browse/SPARK-6954 Thanks! Cheolsoo On Thu, Apr 23, 2015 at 7:57 AM, Michael Stone <mst...@mathom.us> wrote: > If I enable dynamicAllocation and then use spark-shell or pyspark, things > start out working as expected: running simple commands causes new executors > to start and complete tasks. If the shell is left idle for a while, > executors start getting killed off: > > 15/04/23 10:52:43 INFO cluster.YarnClientSchedulerBackend: Requesting to > kill executor(s) 368 > 15/04/23 10:52:43 INFO spark.ExecutorAllocationManager: Removing executor > 368 because it has been idle for 600 seconds (new desired total will be 665) > > That makes sense. But the action also results in error messages: > > 15/04/23 10:52:47 ERROR cluster.YarnScheduler: Lost executor 368 on > hostname: remote Akka client disassociated > 15/04/23 10:52:47 INFO scheduler.DAGScheduler: Executor lost: 368 (epoch 0) > 15/04/23 10:52:47 INFO spark.ExecutorAllocationManager: Existing executor > 368 has been removed (new total is 665) > 15/04/23 10:52:47 INFO storage.BlockManagerMasterActor: Trying to remove > executor 368 from BlockManagerMaster. > 15/04/23 10:52:47 INFO storage.BlockManagerMasterActor: Removing block > manager BlockManagerId(368, hostname, 35877) > 15/04/23 10:52:47 INFO storage.BlockManagerMaster: Removed 368 > successfully in removeExecutor > > After that, trying to run a simple command results in: > > 15/04/23 10:13:30 ERROR util.Utils: Uncaught exception in thread > spark-dynamic-executor-allocation-0 > java.lang.IllegalArgumentException: Attempted to request a negative number > of executor(s) -663 from the cluster manager. Please specify a positive > number! > > And then only the single remaining executor attempts to complete the new > tasks. Am I missing some kind of simple configuration item, are other > people seeing the same behavior as a bug, or is this actually expected? > > Mike Stone > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
