Re: spark-shell gets stuck in ACCEPTED state forever when ran in YARN client mode.
@yohann Looks like something is wrong with my environment which I am yet to figure out but the theory so far makes sense and I had also tried it in another environments with very minimal configuration like my environment and it works fine so clearly something is wrong with my env I don't know why the node automatically is going to unhealthy state and INFO logs don't tell me why. On Sun, Jul 8, 2018 at 7:36 PM, kant kodali wrote: > @yohann Thanks for shining some light! It is making more sense now. > > I think you are correct when you stated: "Your application master is just > asking for more resources than the default Yarn queue is allowed to provide > ". > > Attached are the screenshots of the UI pages you mentioned. The thing that > catches my eye is the default queue resources under scheduler section. It > has the following > > The default queue as listed in the screenshot has the following > > > Max Application Master Resources: > Used Application Master Resources: > > is this why my spark-shell gets stuck in ACCEPTED stated forever? I am > pretty much using the default config so is there a config I should add to > set the Max Application Master Resources? > > Thanks! > > > > > > On Sun, Jul 8, 2018 at 10:27 AM, yohann jardin > wrote: > >> When you run on Yarn, you don’t even need to start a spark cluster (spark >> master and slaves). Yarn receives a job and then allocate resources for the >> application master and then its workers. >> >> Check the resources available in the node section of the resource manager >> UI (and is your node actually detected as alive?), as well as the scheduler >> section to check the default queue resources. >> If you seem to lack resources for your driver, you can try to reduce the >> driver memory by specifying “--driver-memory 512” for example, but I’d >> expect the default of 1g to be low enough based on what you showed us. >> >> *Yohann Jardin* >> Le 7/8/2018 à 6:11 PM, kant kodali a écrit : >> >> @yohann sorry I am assuming you meant application master if so I believe >> spark is the one that provides application master. Is there anyway to look >> for how much resources are being requested and how much yarn is allowed to >> provide? I would assume this is a common case if so I am not sure why these >> numbers are not part of resource manager logs? >> >> On Sun, Jul 8, 2018 at 8:09 AM, kant kodali wrote: >> >>> yarn.scheduler.capacity.maximum-am-resource-percent by default is set >>> to 0.1 and I tried changing it to 1.0 and still no luck. same problem >>> persists. The master here is yarn and I just trying to spawn spark-shell >>> --master yarn --deploy-mode client and run a simple world count so I am not >>> sure why it would request for more resources? >>> >>> On Sun, Jul 8, 2018 at 8:02 AM, yohann jardin >>> wrote: >>> Following the logs from the resource manager: 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server. resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent is insufficient to start a single application in queue, it is likely set too low. skipping enforcement to allow at least one application to start 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server. resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent is insufficient to start a single application in queue for user, it is likely set too low. skipping enforcement to allow at least one application to start I’d say it has nothing to do with spark. Your master is just asking more resources than the default Yarn queue is allowed to provide. You might take a look at https://hadoop.apache.org/docs /r2.7.3/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html and search for maximum-am-resource-percent. Regards, *Yohann Jardin* Le 7/8/2018 à 4:40 PM, kant kodali a écrit : Hi, It's on local mac book pro machine that has 16GB RAM 512GB disk and 8 vCpu! I am not running any code since I can't even spawn spark-shell with yarn as master as described in my previous email. I just want to run simple word count using yarn as master. Thanks! Below is the resource manager log once again if that helps 2018-07-08 07:23:23,343 INFO org.apache.hadoop.yarn.server. resourcemanager.scheduler.capacity.ParentQueue: Application added - appId: application_1531059242261_0001 user: xxx leaf-queue of parent: root #applications: 1 2018-07-08 07:23:23,344 INFO org.apache.hadoop.yarn.server. resourcemanager.scheduler.capacity.CapacityScheduler: Accepted application application_1531059242261_0001 from user: xxx, in queue: default 2018-07-08 07:23:23,350 INFO org.apache.hadoop.yarn.server. resourcemanager.rmapp.RMAppImpl: application_1531059242261_0001 State change from SUBMITTED to ACCEPTED on event=APP_ACCEPTED
Re: spark-shell gets stuck in ACCEPTED state forever when ran in YARN client mode.
When you run on Yarn, you don’t even need to start a spark cluster (spark master and slaves). Yarn receives a job and then allocate resources for the application master and then its workers. Check the resources available in the node section of the resource manager UI (and is your node actually detected as alive?), as well as the scheduler section to check the default queue resources. If you seem to lack resources for your driver, you can try to reduce the driver memory by specifying “--driver-memory 512” for example, but I’d expect the default of 1g to be low enough based on what you showed us. Yohann Jardin Le 7/8/2018 à 6:11 PM, kant kodali a écrit : @yohann sorry I am assuming you meant application master if so I believe spark is the one that provides application master. Is there anyway to look for how much resources are being requested and how much yarn is allowed to provide? I would assume this is a common case if so I am not sure why these numbers are not part of resource manager logs? On Sun, Jul 8, 2018 at 8:09 AM, kant kodali mailto:kanth...@gmail.com>> wrote: yarn.scheduler.capacity.maximum-am-resource-percent by default is set to 0.1 and I tried changing it to 1.0 and still no luck. same problem persists. The master here is yarn and I just trying to spawn spark-shell --master yarn --deploy-mode client and run a simple world count so I am not sure why it would request for more resources? On Sun, Jul 8, 2018 at 8:02 AM, yohann jardin mailto:yohannjar...@hotmail.com>> wrote: Following the logs from the resource manager: 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent is insufficient to start a single application in queue, it is likely set too low. skipping enforcement to allow at least one application to start 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent is insufficient to start a single application in queue for user, it is likely set too low. skipping enforcement to allow at least one application to start I’d say it has nothing to do with spark. Your master is just asking more resources than the default Yarn queue is allowed to provide. You might take a look at https://hadoop.apache.org/docs/r2.7.3/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html and search for maximum-am-resource-percent. Regards, Yohann Jardin Le 7/8/2018 à 4:40 PM, kant kodali a écrit : Hi, It's on local mac book pro machine that has 16GB RAM 512GB disk and 8 vCpu! I am not running any code since I can't even spawn spark-shell with yarn as master as described in my previous email. I just want to run simple word count using yarn as master. Thanks! Below is the resource manager log once again if that helps 2018-07-08 07:23:23,343 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Application added - appId: application_1531059242261_0001 user: xxx leaf-queue of parent: root #applications: 1 2018-07-08 07:23:23,344 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Accepted application application_1531059242261_0001 from user: xxx, in queue: default 2018-07-08 07:23:23,350 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1531059242261_0001 State change from SUBMITTED to ACCEPTED on event=APP_ACCEPTED 2018-07-08 07:23:23,370 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Registering app attempt : appattempt_1531059242261_0001_01 2018-07-08 07:23:23,370 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1531059242261_0001_01 State change from NEW to SUBMITTED 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent is insufficient to start a single application in queue, it is likely set too low. skipping enforcement to allow at least one application to start 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent is insufficient to start a single application in queue for user, it is likely set too low. skipping enforcement to allow at least one application to start 2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application application_1531059242261_0001 from user: xxx activated in queue: default 2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application added - appId: application_1531059242261_0001 user: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue$User@476750cd, leaf-queue: default #user-pending-applications: 0 #user-active-applications: 1 #queue-pending-applications: 0 #queue-active-applications: 1
Re: spark-shell gets stuck in ACCEPTED state forever when ran in YARN client mode.
@yohann sorry I am assuming you meant application master if so I believe spark is the one that provides application master. Is there anyway to look for how much resources are being requested and how much yarn is allowed to provide? I would assume this is a common case if so I am not sure why these numbers are not part of resource manager logs? On Sun, Jul 8, 2018 at 8:09 AM, kant kodali wrote: > yarn.scheduler.capacity.maximum-am-resource-percent by default is set to > 0.1 and I tried changing it to 1.0 and still no luck. same problem > persists. The master here is yarn and I just trying to spawn spark-shell > --master yarn --deploy-mode client and run a simple world count so I am not > sure why it would request for more resources? > > On Sun, Jul 8, 2018 at 8:02 AM, yohann jardin > wrote: > >> Following the logs from the resource manager: >> >> 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server. >> resourcemanager.scheduler.capacity.LeafQueue: >> maximum-am-resource-percent is insufficient to start a single >> application in queue, it is likely set too low. skipping enforcement to >> allow at least one application to start >> >> 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server. >> resourcemanager.scheduler.capacity.LeafQueue: >> maximum-am-resource-percent is insufficient to start a single >> application in queue for user, it is likely set too low. skipping >> enforcement to allow at least one application to start >> >> I’d say it has nothing to do with spark. Your master is just asking more >> resources than the default Yarn queue is allowed to provide. >> You might take a look at https://hadoop.apache.org/docs >> /r2.7.3/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html and search >> for maximum-am-resource-percent. >> >> Regards, >> >> *Yohann Jardin* >> Le 7/8/2018 à 4:40 PM, kant kodali a écrit : >> >> Hi, >> >> It's on local mac book pro machine that has 16GB RAM 512GB disk and 8 >> vCpu! I am not running any code since I can't even spawn spark-shell with >> yarn as master as described in my previous email. I just want to run simple >> word count using yarn as master. >> >> Thanks! >> >> Below is the resource manager log once again if that helps >> >> >> 2018-07-08 07:23:23,343 INFO org.apache.hadoop.yarn.server. >> resourcemanager.scheduler.capacity.ParentQueue: Application added - >> appId: application_1531059242261_0001 user: xxx leaf-queue of parent: root >> #applications: >> 1 >> >> 2018-07-08 07:23:23,344 INFO org.apache.hadoop.yarn.server. >> resourcemanager.scheduler.capacity.CapacityScheduler: Accepted >> application application_1531059242261_0001 from user: xxx, in queue: >> default >> >> 2018-07-08 07:23:23,350 INFO org.apache.hadoop.yarn.server. >> resourcemanager.rmapp.RMAppImpl: application_1531059242261_0001 State >> change from SUBMITTED to ACCEPTED on event=APP_ACCEPTED >> >> 2018-07-08 07:23:23,370 INFO org.apache.hadoop.yarn.server. >> resourcemanager.ApplicationMasterService: Registering app attempt : >> appattempt_1531059242261_0001_01 >> >> 2018-07-08 07:23:23,370 INFO org.apache.hadoop.yarn.server. >> resourcemanager.rmapp.attempt.RMAppAttemptImpl: >> appattempt_1531059242261_0001_01 State change from NEW to SUBMITTED >> >> 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server. >> resourcemanager.scheduler.capacity.LeafQueue: >> maximum-am-resource-percent is insufficient to start a single >> application in queue, it is likely set too low. skipping enforcement to >> allow at least one application to start >> >> 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server. >> resourcemanager.scheduler.capacity.LeafQueue: >> maximum-am-resource-percent is insufficient to start a single >> application in queue for user, it is likely set too low. skipping >> enforcement to allow at least one application to start >> >> 2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server. >> resourcemanager.scheduler.capacity.LeafQueue: Application >> application_1531059242261_0001 from user: xxx activated in queue: default >> >> 2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server. >> resourcemanager.scheduler.capacity.LeafQueue: Application added - appId: >> application_1531059242261_0001 user: org.apache.hadoop.yarn.server. >> resourcemanager.scheduler.capacity.LeafQueue$User@476750cd, leaf-queue: >> default #user-pending-applications: 0 #user-active-applications: 1 >> #queue-pending-applications: 0 #queue-active-applications: 1 >> >> 2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server. >> resourcemanager.scheduler.capacity.CapacityScheduler: Added Application >> Attempt appattempt_1531059242261_0001_01 to scheduler from user xxx >> in queue default >> >> 2018-07-08 07:23:23,386 INFO org.apache.hadoop.yarn.server. >> resourcemanager.rmapp.attempt.RMAppAttemptImpl: >> appattempt_1531059242261_0001_01 State change from SUBMITTED to >> SCHEDULED >> >> >> >> >
Re: spark-shell gets stuck in ACCEPTED state forever when ran in YARN client mode.
yarn.scheduler.capacity.maximum-am-resource-percent by default is set to 0.1 and I tried changing it to 1.0 and still no luck. same problem persists. The master here is yarn and I just trying to spawn spark-shell --master yarn --deploy-mode client and run a simple world count so I am not sure why it would request for more resources? On Sun, Jul 8, 2018 at 8:02 AM, yohann jardin wrote: > Following the logs from the resource manager: > > 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server. > resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent > is insufficient to start a single application in queue, it is likely set > too low. skipping enforcement to allow at least one application to start > > 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server. > resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent > is insufficient to start a single application in queue for user, it is > likely set too low. skipping enforcement to allow at least one > application to start > > I’d say it has nothing to do with spark. Your master is just asking more > resources than the default Yarn queue is allowed to provide. > You might take a look at https://hadoop.apache.org/ > docs/r2.7.3/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html and > search for maximum-am-resource-percent. > > Regards, > > *Yohann Jardin* > Le 7/8/2018 à 4:40 PM, kant kodali a écrit : > > Hi, > > It's on local mac book pro machine that has 16GB RAM 512GB disk and 8 > vCpu! I am not running any code since I can't even spawn spark-shell with > yarn as master as described in my previous email. I just want to run simple > word count using yarn as master. > > Thanks! > > Below is the resource manager log once again if that helps > > > 2018-07-08 07:23:23,343 INFO org.apache.hadoop.yarn.server. > resourcemanager.scheduler.capacity.ParentQueue: Application added - > appId: application_1531059242261_0001 user: xxx leaf-queue of parent: root > #applications: > 1 > > 2018-07-08 07:23:23,344 INFO org.apache.hadoop.yarn.server. > resourcemanager.scheduler.capacity.CapacityScheduler: Accepted > application application_1531059242261_0001 from user: xxx, in queue: > default > > 2018-07-08 07:23:23,350 INFO org.apache.hadoop.yarn.server. > resourcemanager.rmapp.RMAppImpl: application_1531059242261_0001 State > change from SUBMITTED to ACCEPTED on event=APP_ACCEPTED > > 2018-07-08 07:23:23,370 INFO org.apache.hadoop.yarn.server. > resourcemanager.ApplicationMasterService: Registering app attempt : > appattempt_1531059242261_0001_01 > > 2018-07-08 07:23:23,370 INFO org.apache.hadoop.yarn.server. > resourcemanager.rmapp.attempt.RMAppAttemptImpl: > appattempt_1531059242261_0001_01 State change from NEW to SUBMITTED > > 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server. > resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent > is insufficient to start a single application in queue, it is likely set > too low. skipping enforcement to allow at least one application to start > > 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server. > resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent > is insufficient to start a single application in queue for user, it is > likely set too low. skipping enforcement to allow at least one > application to start > > 2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server. > resourcemanager.scheduler.capacity.LeafQueue: Application > application_1531059242261_0001 from user: xxx activated in queue: default > > 2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server. > resourcemanager.scheduler.capacity.LeafQueue: Application added - appId: > application_1531059242261_0001 user: org.apache.hadoop.yarn.server. > resourcemanager.scheduler.capacity.LeafQueue$User@476750cd, leaf-queue: > default #user-pending-applications: 0 #user-active-applications: 1 > #queue-pending-applications: 0 #queue-active-applications: 1 > > 2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server. > resourcemanager.scheduler.capacity.CapacityScheduler: Added Application > Attempt appattempt_1531059242261_0001_01 to scheduler from user xxx in > queue default > > 2018-07-08 07:23:23,386 INFO org.apache.hadoop.yarn.server. > resourcemanager.rmapp.attempt.RMAppAttemptImpl: > appattempt_1531059242261_0001_01 State change from SUBMITTED to > SCHEDULED > > > >
Re: spark-shell gets stuck in ACCEPTED state forever when ran in YARN client mode.
Following the logs from the resource manager: 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent is insufficient to start a single application in queue, it is likely set too low. skipping enforcement to allow at least one application to start 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent is insufficient to start a single application in queue for user, it is likely set too low. skipping enforcement to allow at least one application to start I’d say it has nothing to do with spark. Your master is just asking more resources than the default Yarn queue is allowed to provide. You might take a look at https://hadoop.apache.org/docs/r2.7.3/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html and search for maximum-am-resource-percent. Regards, Yohann Jardin Le 7/8/2018 à 4:40 PM, kant kodali a écrit : Hi, It's on local mac book pro machine that has 16GB RAM 512GB disk and 8 vCpu! I am not running any code since I can't even spawn spark-shell with yarn as master as described in my previous email. I just want to run simple word count using yarn as master. Thanks! Below is the resource manager log once again if that helps 2018-07-08 07:23:23,343 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Application added - appId: application_1531059242261_0001 user: xxx leaf-queue of parent: root #applications: 1 2018-07-08 07:23:23,344 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Accepted application application_1531059242261_0001 from user: xxx, in queue: default 2018-07-08 07:23:23,350 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1531059242261_0001 State change from SUBMITTED to ACCEPTED on event=APP_ACCEPTED 2018-07-08 07:23:23,370 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Registering app attempt : appattempt_1531059242261_0001_01 2018-07-08 07:23:23,370 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1531059242261_0001_01 State change from NEW to SUBMITTED 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent is insufficient to start a single application in queue, it is likely set too low. skipping enforcement to allow at least one application to start 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent is insufficient to start a single application in queue for user, it is likely set too low. skipping enforcement to allow at least one application to start 2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application application_1531059242261_0001 from user: xxx activated in queue: default 2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application added - appId: application_1531059242261_0001 user: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue$User@476750cd, leaf-queue: default #user-pending-applications: 0 #user-active-applications: 1 #queue-pending-applications: 0 #queue-active-applications: 1 2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Added Application Attempt appattempt_1531059242261_0001_01 to scheduler from user xxx in queue default 2018-07-08 07:23:23,386 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1531059242261_0001_01 State change from SUBMITTED to SCHEDULED
Re: spark-shell gets stuck in ACCEPTED state forever when ran in YARN client mode.
Are you able to run a simple Map Reduce job on yarn without any issues?
If you have any issues: I had this problem on Mac. Use CSRUTIL in Mac, to
disable it. Then add a softlink
sudo ln –s /usr/bin/java/bin/java
The new versions of Mac from EL Captain does not allow softlinks in
/bin/java.
I got everything working by above.
Best,
Ravion
On Sun, Jul 8, 2018 at 10:20 AM Marco Mistroni wrote:
> You running on emr? You checked the emr logs?
> Was in similar situation where job was stuck in accepted and then it
> died..turned out to be an issue w. My code when running g with huge
> data.perhaps try to reduce gradually the load til it works and then start
> from there?
> Not a huge help but I followed same when. My job was stuck on accepted
> Hth
>
> On Sun, Jul 8, 2018, 2:59 PM kant kodali wrote:
>
>> Hi All,
>>
>> I am trying to run a simple word count using YARN as a cluster manager.
>> I am currently using Spark 2.3.1 and Apache hadoop 2.7.3. When I spawn
>> spark-shell like below it gets stuck in ACCEPTED stated forever.
>>
>> ./bin/spark-shell --master yarn --deploy-mode client
>>
>>
>> I set my log4j.properties in SPARK_HOME/conf to TRACE
>>
>> queue: "default" name: "Spark shell" host: "N/A" rpc_port: -1
>> yarn_application_state: ACCEPTED trackingUrl: "
>> http://Kants-MacBook-Pro-2.local:8088/proxy/application_1531056583425_0001/;
>> diagnostics: "" startTime: 1531056632496 finishTime: 0
>> final_application_status: APP_UNDEFINED app_resource_Usage {
>> num_used_containers: 0 num_reserved_containers: 0 used_resources { memory:
>> 0 virtual_cores: 0 } reserved_resources { memory: 0 virtual_cores: 0 }
>> needed_resources { memory: 0 virtual_cores: 0 } memory_seconds: 0
>> vcore_seconds: 0 } originalTrackingUrl: "N/A" currentApplicationAttemptId {
>> application_id { id: 1 cluster_timestamp: 1531056583425 } attemptId: 1 }
>> progress: 0.0 applicationType: "SPARK" }}
>>
>> 18/07/08 06:32:22 INFO Client: Application report for
>> application_1531056583425_0001 (state: ACCEPTED)
>>
>> 18/07/08 06:32:22 DEBUG Client:
>>
>> client token: N/A
>>
>> diagnostics: N/A
>>
>> ApplicationMaster host: N/A
>>
>> ApplicationMaster RPC port: -1
>>
>> queue: default
>>
>> start time: 1531056632496
>>
>> final status: UNDEFINED
>>
>> tracking URL:
>> http://xxx-MacBook-Pro-2.local:8088/proxy/application_1531056583425_0001/
>>
>> user: xxx
>>
>>
>>
>> 18/07/08 06:32:20 DEBUG Client:
>>
>> client token: N/A
>>
>> diagnostics: N/A
>>
>> ApplicationMaster host: N/A
>>
>> ApplicationMaster RPC port: -1
>>
>> queue: default
>>
>> start time: 1531056632496
>>
>> final status: UNDEFINED
>>
>> tracking URL:
>> http://Kants-MacBook-Pro-2.local:8088/proxy/application_1531056583425_0001/
>>
>> user: kantkodali
>>
>>
>> 18/07/08 06:32:21 TRACE ProtobufRpcEngine: 1: Call -> /0.0.0.0:8032:
>> getApplicationReport {application_id { id: 1 cluster_timestamp:
>> 1531056583425 }}
>>
>> 18/07/08 06:32:21 DEBUG Client: IPC Client (1608805714) connection to /
>> 0.0.0.0:8032 from kantkodali sending #136
>>
>> 18/07/08 06:32:21 DEBUG Client: IPC Client (1608805714) connection to /
>> 0.0.0.0:8032 from kantkodali got value #136
>>
>> 18/07/08 06:32:21 DEBUG ProtobufRpcEngine: Call: getApplicationReport
>> took 1ms
>>
>> 18/07/08 06:32:21 TRACE ProtobufRpcEngine: 1: Response <- /0.0.0.0:8032:
>> getApplicationReport {application_report { applicationId { id: 1
>> cluster_timestamp: 1531056583425 } user: "xxx" queue: "default" name:
>> "Spark shell" host: "N/A" rpc_port: -1 yarn_application_state: ACCEPTED
>> trackingUrl: "
>> http://xxx-MacBook-Pro-2.local:8088/proxy/application_1531056583425_0001/;
>> diagnostics: "" startTime: 1531056632496 finishTime: 0
>> final_application_status: APP_UNDEFINED app_resource_Usage {
>> num_used_containers: 0 num_reserved_containers: 0 used_resources { memory:
>> 0 virtual_cores: 0 } reserved_resources { memory: 0 virtual_cores: 0 }
>> needed_resources { memory: 0 virtual_cores: 0 } memory_seconds: 0
>> vcore_seconds: 0 } originalTrackingUrl: "N/A" currentApplicationAttemptId {
>> application_id { id: 1 cluster_timestamp: 1531056583425 } attemptId: 1 }
>> progress: 0.0 applicationType: "SPARK" }}
>>
>> 18/07/08 06:32:21 INFO Client: Application report for
>> application_1531056583425_0001 (state: ACCEPTED)
>>
>>
>> I have read this link
>>
>> and
>> here are the conf files that are different from default settings
>>
>>
>> *yarn-site.xml*
>>
>>
>>
>>
>>
>>
>>
>> yarn.nodemanager.aux-services
>>
>> mapreduce_shuffle
>>
>>
>>
>>
>>
>>
>> yarn.nodemanager.resource.memory-mb
>>
>> 16384
>>
>>
>>
>>
>>
>>
>>yarn.scheduler.minimum-allocation-mb
>>
>>256
>>
>>
>>
>>
>>
>>
>>yarn.scheduler.maximum-allocation-mb
>>
>>8192
>>
>>
>>
>>
>>
>>
>>yarn.nodemanager.resource.cpu-vcores
>>
>>8
>>
>>
Re: spark-shell gets stuck in ACCEPTED state forever when ran in YARN client mode.
Hi, It's on local mac book pro machine that has 16GB RAM 512GB disk and 8 vCpu! I am not running any code since I can't even spawn spark-shell with yarn as master as described in my previous email. I just want to run simple word count using yarn as master. Thanks! Below is the resource manager log once again if that helps 2018-07-08 07:23:23,343 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Application added - appId: application_1531059242261_0001 user: xxx leaf-queue of parent: root #applications: 1 2018-07-08 07:23:23,344 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Accepted application application_1531059242261_0001 from user: xxx, in queue: default 2018-07-08 07:23:23,350 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1531059242261_0001 State change from SUBMITTED to ACCEPTED on event=APP_ACCEPTED 2018-07-08 07:23:23,370 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Registering app attempt : appattempt_1531059242261_0001_01 2018-07-08 07:23:23,370 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1531059242261_0001_01 State change from NEW to SUBMITTED 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent is insufficient to start a single application in queue, it is likely set too low. skipping enforcement to allow at least one application to start 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent is insufficient to start a single application in queue for user, it is likely set too low. skipping enforcement to allow at least one application to start 2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application application_1531059242261_0001 from user: xxx activated in queue: default 2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application added - appId: application_1531059242261_0001 user: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue$User@ 476750cd, leaf-queue: default #user-pending-applications: 0 #user-active-applications: 1 #queue-pending-applications: 0 #queue-active-applications: 1 2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Added Application Attempt appattempt_1531059242261_0001_01 to scheduler from user xxx in queue default 2018-07-08 07:23:23,386 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1531059242261_0001_01 State change from SUBMITTED to SCHEDULED
Re: spark-shell gets stuck in ACCEPTED state forever when ran in YARN client mode.
You running on emr? You checked the emr logs?
Was in similar situation where job was stuck in accepted and then it
died..turned out to be an issue w. My code when running g with huge
data.perhaps try to reduce gradually the load til it works and then start
from there?
Not a huge help but I followed same when. My job was stuck on accepted
Hth
On Sun, Jul 8, 2018, 2:59 PM kant kodali wrote:
> Hi All,
>
> I am trying to run a simple word count using YARN as a cluster manager. I
> am currently using Spark 2.3.1 and Apache hadoop 2.7.3. When I spawn
> spark-shell like below it gets stuck in ACCEPTED stated forever.
>
> ./bin/spark-shell --master yarn --deploy-mode client
>
>
> I set my log4j.properties in SPARK_HOME/conf to TRACE
>
> queue: "default" name: "Spark shell" host: "N/A" rpc_port: -1
> yarn_application_state: ACCEPTED trackingUrl: "
> http://Kants-MacBook-Pro-2.local:8088/proxy/application_1531056583425_0001/;
> diagnostics: "" startTime: 1531056632496 finishTime: 0
> final_application_status: APP_UNDEFINED app_resource_Usage {
> num_used_containers: 0 num_reserved_containers: 0 used_resources { memory:
> 0 virtual_cores: 0 } reserved_resources { memory: 0 virtual_cores: 0 }
> needed_resources { memory: 0 virtual_cores: 0 } memory_seconds: 0
> vcore_seconds: 0 } originalTrackingUrl: "N/A" currentApplicationAttemptId {
> application_id { id: 1 cluster_timestamp: 1531056583425 } attemptId: 1 }
> progress: 0.0 applicationType: "SPARK" }}
>
> 18/07/08 06:32:22 INFO Client: Application report for
> application_1531056583425_0001 (state: ACCEPTED)
>
> 18/07/08 06:32:22 DEBUG Client:
>
> client token: N/A
>
> diagnostics: N/A
>
> ApplicationMaster host: N/A
>
> ApplicationMaster RPC port: -1
>
> queue: default
>
> start time: 1531056632496
>
> final status: UNDEFINED
>
> tracking URL:
> http://xxx-MacBook-Pro-2.local:8088/proxy/application_1531056583425_0001/
>
> user: xxx
>
>
>
> 18/07/08 06:32:20 DEBUG Client:
>
> client token: N/A
>
> diagnostics: N/A
>
> ApplicationMaster host: N/A
>
> ApplicationMaster RPC port: -1
>
> queue: default
>
> start time: 1531056632496
>
> final status: UNDEFINED
>
> tracking URL:
> http://Kants-MacBook-Pro-2.local:8088/proxy/application_1531056583425_0001/
>
> user: kantkodali
>
>
> 18/07/08 06:32:21 TRACE ProtobufRpcEngine: 1: Call -> /0.0.0.0:8032:
> getApplicationReport {application_id { id: 1 cluster_timestamp:
> 1531056583425 }}
>
> 18/07/08 06:32:21 DEBUG Client: IPC Client (1608805714) connection to /
> 0.0.0.0:8032 from kantkodali sending #136
>
> 18/07/08 06:32:21 DEBUG Client: IPC Client (1608805714) connection to /
> 0.0.0.0:8032 from kantkodali got value #136
>
> 18/07/08 06:32:21 DEBUG ProtobufRpcEngine: Call: getApplicationReport took
> 1ms
>
> 18/07/08 06:32:21 TRACE ProtobufRpcEngine: 1: Response <- /0.0.0.0:8032:
> getApplicationReport {application_report { applicationId { id: 1
> cluster_timestamp: 1531056583425 } user: "xxx" queue: "default" name:
> "Spark shell" host: "N/A" rpc_port: -1 yarn_application_state: ACCEPTED
> trackingUrl: "
> http://xxx-MacBook-Pro-2.local:8088/proxy/application_1531056583425_0001/;
> diagnostics: "" startTime: 1531056632496 finishTime: 0
> final_application_status: APP_UNDEFINED app_resource_Usage {
> num_used_containers: 0 num_reserved_containers: 0 used_resources { memory:
> 0 virtual_cores: 0 } reserved_resources { memory: 0 virtual_cores: 0 }
> needed_resources { memory: 0 virtual_cores: 0 } memory_seconds: 0
> vcore_seconds: 0 } originalTrackingUrl: "N/A" currentApplicationAttemptId {
> application_id { id: 1 cluster_timestamp: 1531056583425 } attemptId: 1 }
> progress: 0.0 applicationType: "SPARK" }}
>
> 18/07/08 06:32:21 INFO Client: Application report for
> application_1531056583425_0001 (state: ACCEPTED)
>
>
> I have read this link
>
> and
> here are the conf files that are different from default settings
>
>
> *yarn-site.xml*
>
>
>
>
>
>
>
> yarn.nodemanager.aux-services
>
> mapreduce_shuffle
>
>
>
>
>
>
> yarn.nodemanager.resource.memory-mb
>
> 16384
>
>
>
>
>
>
>yarn.scheduler.minimum-allocation-mb
>
>256
>
>
>
>
>
>
>yarn.scheduler.maximum-allocation-mb
>
>8192
>
>
>
>
>
>
>yarn.nodemanager.resource.cpu-vcores
>
>8
>
>
>
>
>
>
> *core-site.xml*
>
>
>
>
>
>
> fs.defaultFS
>
> hdfs://localhost:9000
>
>
>
>
>
> *hdfs-site.xml*
>
>
>
>
>
>
> dfs.replication
>
> 1
>
>
>
>
>
>
> you can imagine every other config remains untouched(so everything else
> has default settings) Finally, I have also tried to see if there any clues
> in resource manager logs but they dont seem to be helpful in terms of
> fixing the issue however I am newbie to yarn so please let me know if I
> missed out on something.
>
>
>
> 2018-07-08 06:54:57,345 INFO
>
spark-shell gets stuck in ACCEPTED state forever when ran in YARN client mode.
Hi All,
I am trying to run a simple word count using YARN as a cluster manager. I
am currently using Spark 2.3.1 and Apache hadoop 2.7.3. When I spawn
spark-shell like below it gets stuck in ACCEPTED stated forever.
./bin/spark-shell --master yarn --deploy-mode client
I set my log4j.properties in SPARK_HOME/conf to TRACE
queue: "default" name: "Spark shell" host: "N/A" rpc_port: -1
yarn_application_state: ACCEPTED trackingUrl: "
http://Kants-MacBook-Pro-2.local:8088/proxy/application_1531056583425_0001/;
diagnostics: "" startTime: 1531056632496 finishTime: 0
final_application_status: APP_UNDEFINED app_resource_Usage {
num_used_containers: 0 num_reserved_containers: 0 used_resources { memory:
0 virtual_cores: 0 } reserved_resources { memory: 0 virtual_cores: 0 }
needed_resources { memory: 0 virtual_cores: 0 } memory_seconds: 0
vcore_seconds: 0 } originalTrackingUrl: "N/A" currentApplicationAttemptId {
application_id { id: 1 cluster_timestamp: 1531056583425 } attemptId: 1 }
progress: 0.0 applicationType: "SPARK" }}
18/07/08 06:32:22 INFO Client: Application report for
application_1531056583425_0001 (state: ACCEPTED)
18/07/08 06:32:22 DEBUG Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1531056632496
final status: UNDEFINED
tracking URL:
http://xxx-MacBook-Pro-2.local:8088/proxy/application_1531056583425_0001/
user: xxx
18/07/08 06:32:20 DEBUG Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1531056632496
final status: UNDEFINED
tracking URL:
http://Kants-MacBook-Pro-2.local:8088/proxy/application_1531056583425_0001/
user: kantkodali
18/07/08 06:32:21 TRACE ProtobufRpcEngine: 1: Call -> /0.0.0.0:8032:
getApplicationReport {application_id { id: 1 cluster_timestamp:
1531056583425 }}
18/07/08 06:32:21 DEBUG Client: IPC Client (1608805714) connection to /
0.0.0.0:8032 from kantkodali sending #136
18/07/08 06:32:21 DEBUG Client: IPC Client (1608805714) connection to /
0.0.0.0:8032 from kantkodali got value #136
18/07/08 06:32:21 DEBUG ProtobufRpcEngine: Call: getApplicationReport took
1ms
18/07/08 06:32:21 TRACE ProtobufRpcEngine: 1: Response <- /0.0.0.0:8032:
getApplicationReport {application_report { applicationId { id: 1
cluster_timestamp: 1531056583425 } user: "xxx" queue: "default" name:
"Spark shell" host: "N/A" rpc_port: -1 yarn_application_state: ACCEPTED
trackingUrl: "
http://xxx-MacBook-Pro-2.local:8088/proxy/application_1531056583425_0001/;
diagnostics: "" startTime: 1531056632496 finishTime: 0
final_application_status: APP_UNDEFINED app_resource_Usage {
num_used_containers: 0 num_reserved_containers: 0 used_resources { memory:
0 virtual_cores: 0 } reserved_resources { memory: 0 virtual_cores: 0 }
needed_resources { memory: 0 virtual_cores: 0 } memory_seconds: 0
vcore_seconds: 0 } originalTrackingUrl: "N/A" currentApplicationAttemptId {
application_id { id: 1 cluster_timestamp: 1531056583425 } attemptId: 1 }
progress: 0.0 applicationType: "SPARK" }}
18/07/08 06:32:21 INFO Client: Application report for
application_1531056583425_0001 (state: ACCEPTED)
I have read this link
and
here are the conf files that are different from default settings
*yarn-site.xml*
yarn.nodemanager.aux-services
mapreduce_shuffle
yarn.nodemanager.resource.memory-mb
16384
yarn.scheduler.minimum-allocation-mb
256
yarn.scheduler.maximum-allocation-mb
8192
yarn.nodemanager.resource.cpu-vcores
8
*core-site.xml*
fs.defaultFS
hdfs://localhost:9000
*hdfs-site.xml*
dfs.replication
1
you can imagine every other config remains untouched(so everything else has
default settings) Finally, I have also tried to see if there any clues in
resource manager logs but they dont seem to be helpful in terms of fixing
the issue however I am newbie to yarn so please let me know if I missed out
on something.
2018-07-08 06:54:57,345 INFO
org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Allocated
new applicationId: 1
2018-07-08 06:55:09,413 WARN
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: The specific
max attempts: 0 for application: 1 is invalid, because it is out of the
range [1, 2]. Use the global max attempts instead.
2018-07-08 06:55:09,414 INFO
org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Application
with id 1 submitted by user xxx
2018-07-08 06:55:09,415 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Storing
application with id application_1531058076308_0001
2018-07-08 06:55:09,416 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=kantkodali
