Hi Saisai, I'm sorry for did not describe it clearly,YARN debug log said I have 50 executors,but ResourceManager showed that I only have 1 container for the AppMaster.
I have checked YARN RM logs,after AppMaster changed state from ACCEPTED to RUNNING,it did not have log about this job any more.So,the problem is I did not have any executor but ExecutorAllocationManager think I have.Would you minding having a test in your cluster environment? Thanks, Weber 2015-11-24 11:00 GMT+08:00 Saisai Shao <[email protected]>: > I think this behavior is expected, since you already have 50 executors > launched, so no need to acquire additional executors. You change is not > solid, it is just hiding the log. > > Again I think you should check the logs of Yarn and Spark to see if > executors are started correctly. Why resource is still not enough where you > already have 50 executors. > > On Tue, Nov 24, 2015 at 10:48 AM, 谢廷稳 <[email protected]> wrote: > >> Hi SaiSai, >> I have changed "if (numExecutorsTarget >= maxNumExecutors)" to "if >> (numExecutorsTarget > maxNumExecutors)" of the first line in the >> ExecutorAllocationManager#addExecutors() and it rans well. >> In my opinion,when I was set minExecutors equals maxExecutors,when the >> first time to add Executors,numExecutorsTarget equals maxNumExecutors and >> it repeat printe "DEBUG ExecutorAllocationManager: Not adding executors >> because our current target total is already 50 (limit 50)". >> Thanks >> Weber >> >> 2015-11-23 21:00 GMT+08:00 Saisai Shao <[email protected]>: >> >>> Hi Tingwen, >>> >>> Would you minding sharing your changes in >>> ExecutorAllocationManager#addExecutors(). >>> >>> From my understanding and test, dynamic allocation can be worked when >>> you set the min to max number of executors to the same number. >>> >>> Please check your Spark and Yarn log to make sure the executors are >>> correctly started, the warning log means currently resource is not enough >>> to submit tasks. >>> >>> Thanks >>> Saisai >>> >>> >>> On Mon, Nov 23, 2015 at 8:41 PM, 谢廷稳 <[email protected]> wrote: >>> >>>> Hi all, >>>> I ran a SparkPi on YARN with Dynamic Allocation enabled and set >>>> spark.dynamicAllocation.maxExecutors >>>> equals >>>> spark.dynamicAllocation.minExecutors,then I submit an application using: >>>> ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master >>>> yarn-cluster --driver-memory 4g --executor-memory 8g >>>> lib/spark-examples*.jar 200 >>>> >>>> then, this application was submitted successfully, but the AppMaster >>>> always saying “15/11/23 20:13:08 WARN cluster.YarnClusterScheduler: >>>> Initial job has not accepted any resources; check your cluster UI to ensure >>>> that workers are registered and have sufficient resources” >>>> and when I open DEBUG,I found “15/11/23 20:24:00 DEBUG >>>> ExecutorAllocationManager: Not adding executors because our current target >>>> total is already 50 (limit 50)” in the console. >>>> >>>> I have fixed it by modifying code in >>>> ExecutorAllocationManager.addExecutors,Does this a bug or it was designed >>>> that we can’t set maxExecutors equals minExecutors? >>>> >>>> Thanks, >>>> Weber >>>> >>> >>> >> >
