OK,the YARN cluster was used by myself,it have 6 node witch can run over 100 executor, and the YARN RM logs showed that the Spark application did not requested resource from it.
Is this a bug? Should I create a JIRA for this problem? 2015-11-24 12:00 GMT+08:00 Saisai Shao <[email protected]>: > OK, so this looks like your Yarn cluster does not allocate containers > which you supposed should be 50. Does the yarn cluster have enough resource > after allocating AM container, if not, that is the problem. > > The problem not lies in dynamic allocation from my guess of your > description. I said I'm OK with min and max executors to the same number. > > On Tue, Nov 24, 2015 at 11:54 AM, 谢廷稳 <[email protected]> wrote: > >> Hi Saisai, >> I'm sorry for did not describe it clearly,YARN debug log said I have 50 >> executors,but ResourceManager showed that I only have 1 container for the >> AppMaster. >> >> I have checked YARN RM logs,after AppMaster changed state from ACCEPTED >> to RUNNING,it did not have log about this job any more.So,the problem is I >> did not have any executor but ExecutorAllocationManager think I have.Would >> you minding having a test in your cluster environment? >> Thanks, >> Weber >> >> 2015-11-24 11:00 GMT+08:00 Saisai Shao <[email protected]>: >> >>> I think this behavior is expected, since you already have 50 executors >>> launched, so no need to acquire additional executors. You change is not >>> solid, it is just hiding the log. >>> >>> Again I think you should check the logs of Yarn and Spark to see if >>> executors are started correctly. Why resource is still not enough where you >>> already have 50 executors. >>> >>> On Tue, Nov 24, 2015 at 10:48 AM, 谢廷稳 <[email protected]> wrote: >>> >>>> Hi SaiSai, >>>> I have changed "if (numExecutorsTarget >= maxNumExecutors)" to "if >>>> (numExecutorsTarget > maxNumExecutors)" of the first line in the >>>> ExecutorAllocationManager#addExecutors() and it rans well. >>>> In my opinion,when I was set minExecutors equals maxExecutors,when the >>>> first time to add Executors,numExecutorsTarget equals maxNumExecutors and >>>> it repeat printe "DEBUG ExecutorAllocationManager: Not adding >>>> executors because our current target total is already 50 (limit 50)". >>>> Thanks >>>> Weber >>>> >>>> 2015-11-23 21:00 GMT+08:00 Saisai Shao <[email protected]>: >>>> >>>>> Hi Tingwen, >>>>> >>>>> Would you minding sharing your changes in >>>>> ExecutorAllocationManager#addExecutors(). >>>>> >>>>> From my understanding and test, dynamic allocation can be worked when >>>>> you set the min to max number of executors to the same number. >>>>> >>>>> Please check your Spark and Yarn log to make sure the executors are >>>>> correctly started, the warning log means currently resource is not enough >>>>> to submit tasks. >>>>> >>>>> Thanks >>>>> Saisai >>>>> >>>>> >>>>> On Mon, Nov 23, 2015 at 8:41 PM, 谢廷稳 <[email protected]> wrote: >>>>> >>>>>> Hi all, >>>>>> I ran a SparkPi on YARN with Dynamic Allocation enabled and set >>>>>> spark.dynamicAllocation.maxExecutors >>>>>> equals >>>>>> spark.dynamicAllocation.minExecutors,then I submit an application >>>>>> using: >>>>>> ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master >>>>>> yarn-cluster --driver-memory 4g --executor-memory 8g >>>>>> lib/spark-examples*.jar 200 >>>>>> >>>>>> then, this application was submitted successfully, but the AppMaster >>>>>> always saying “15/11/23 20:13:08 WARN cluster.YarnClusterScheduler: >>>>>> Initial job has not accepted any resources; check your cluster UI to >>>>>> ensure >>>>>> that workers are registered and have sufficient resources” >>>>>> and when I open DEBUG,I found “15/11/23 20:24:00 DEBUG >>>>>> ExecutorAllocationManager: Not adding executors because our current >>>>>> target >>>>>> total is already 50 (limit 50)” in the console. >>>>>> >>>>>> I have fixed it by modifying code in >>>>>> ExecutorAllocationManager.addExecutors,Does this a bug or it was designed >>>>>> that we can’t set maxExecutors equals minExecutors? >>>>>> >>>>>> Thanks, >>>>>> Weber >>>>>> >>>>> >>>>> >>>> >>> >> >
