Following the logs from the resource manager: 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent is insufficient to start a single application in queue, it is likely set too low. skipping enforcement to allow at least one application to start
2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent is insufficient to start a single application in queue for user, it is likely set too low. skipping enforcement to allow at least one application to start I’d say it has nothing to do with spark. Your master is just asking more resources than the default Yarn queue is allowed to provide. You might take a look at https://hadoop.apache.org/docs/r2.7.3/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html and search for maximum-am-resource-percent. Regards, Yohann Jardin Le 7/8/2018 à 4:40 PM, kant kodali a écrit : Hi, It's on local mac book pro machine that has 16GB RAM 512GB disk and 8 vCpu! I am not running any code since I can't even spawn spark-shell with yarn as master as described in my previous email. I just want to run simple word count using yarn as master. Thanks! Below is the resource manager log once again if that helps 2018-07-08 07:23:23,343 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Application added - appId: application_1531059242261_0001 user: xxx leaf-queue of parent: root #applications: 1 2018-07-08 07:23:23,344 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Accepted application application_1531059242261_0001 from user: xxx, in queue: default 2018-07-08 07:23:23,350 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1531059242261_0001 State change from SUBMITTED to ACCEPTED on event=APP_ACCEPTED 2018-07-08 07:23:23,370 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Registering app attempt : appattempt_1531059242261_0001_000001 2018-07-08 07:23:23,370 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1531059242261_0001_000001 State change from NEW to SUBMITTED 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent is insufficient to start a single application in queue, it is likely set too low. skipping enforcement to allow at least one application to start 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent is insufficient to start a single application in queue for user, it is likely set too low. skipping enforcement to allow at least one application to start 2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application application_1531059242261_0001 from user: xxx activated in queue: default 2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application added - appId: application_1531059242261_0001 user: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue$User@476750cd, leaf-queue: default #user-pending-applications: 0 #user-active-applications: 1 #queue-pending-applications: 0 #queue-active-applications: 1 2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Added Application Attempt appattempt_1531059242261_0001_000001 to scheduler from user xxx in queue default 2018-07-08 07:23:23,386 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1531059242261_0001_000001 State change from SUBMITTED to SCHEDULED
