Hi SaiSai, I have changed "if (numExecutorsTarget >= maxNumExecutors)" to "if (numExecutorsTarget > maxNumExecutors)" of the first line in the ExecutorAllocationManager#addExecutors() and it rans well. In my opinion,when I was set minExecutors equals maxExecutors,when the first time to add Executors,numExecutorsTarget equals maxNumExecutors and it repeat printe "DEBUG ExecutorAllocationManager: Not adding executors because our current target total is already 50 (limit 50)". Thanks Weber
2015-11-23 21:00 GMT+08:00 Saisai Shao <sai.sai.s...@gmail.com>: > Hi Tingwen, > > Would you minding sharing your changes in > ExecutorAllocationManager#addExecutors(). > > From my understanding and test, dynamic allocation can be worked when you > set the min to max number of executors to the same number. > > Please check your Spark and Yarn log to make sure the executors are > correctly started, the warning log means currently resource is not enough > to submit tasks. > > Thanks > Saisai > > > On Mon, Nov 23, 2015 at 8:41 PM, 谢廷稳 <xieting...@gmail.com> wrote: > >> Hi all, >> I ran a SparkPi on YARN with Dynamic Allocation enabled and set >> spark.dynamicAllocation.maxExecutors >> equals >> spark.dynamicAllocation.minExecutors,then I submit an application using: >> ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master >> yarn-cluster --driver-memory 4g --executor-memory 8g >> lib/spark-examples*.jar 200 >> >> then, this application was submitted successfully, but the AppMaster >> always saying “15/11/23 20:13:08 WARN cluster.YarnClusterScheduler: >> Initial job has not accepted any resources; check your cluster UI to ensure >> that workers are registered and have sufficient resources” >> and when I open DEBUG,I found “15/11/23 20:24:00 DEBUG >> ExecutorAllocationManager: Not adding executors because our current target >> total is already 50 (limit 50)” in the console. >> >> I have fixed it by modifying code in >> ExecutorAllocationManager.addExecutors,Does this a bug or it was designed >> that we can’t set maxExecutors equals minExecutors? >> >> Thanks, >> Weber >> > >
