[ 
https://issues.apache.org/jira/browse/YARN-2918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14644869#comment-14644869
 ] 

Dheeren Beborrtha commented on YARN-2918:
-----------------------------------------

Thanks Wangda Tan. The problem is that I had a cluster up and running with HDP 
2.2.0. Followed HortonWorks instructions to add the labels and configured the 
CS queues and that was easy.Few days went by. The cluster went down one day and 
on restart RM wouldn't come back up. The only way we could bring it back was to 
use an older version of the CS xml to bring it back to life. Unfortunately this 
was not documented any where. 
By the way, where are the node labels persisted? I had to re-add the labels too!

> Don't fail RM if queue's configured labels are not existed in 
> cluster-node-labels
> ---------------------------------------------------------------------------------
>
>                 Key: YARN-2918
>                 URL: https://issues.apache.org/jira/browse/YARN-2918
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Rohith Sharma K S
>            Assignee: Wangda Tan
>              Labels: 2.6.1-candidate
>             Fix For: 2.8.0, 2.7.1
>
>         Attachments: YARN-2918.1.patch, YARN-2918.2.patch, YARN-2918.3.patch
>
>
> Currently, if admin setup labels on queues 
> {{<queue-path>.accessible-node-labels = ...}}. And the label is not added to 
> RM, queue's initialization will fail and RM will fail too:
> {noformat}
> 2014-12-03 20:11:50,126 FATAL 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting 
> ResourceManager
> ...
> Caused by: java.io.IOException: NodeLabelManager doesn't include label = x, 
> please check.
>       at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.checkIfLabelInClusterNodeLabels(SchedulerUtils.java:287)
>       at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.<init>(AbstractCSQueue.java:109)
>       at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.<init>(LeafQueue.java:120)
>       at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:567)
>       at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:587)
>       at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:462)
>       at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:294)
>       at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:324)
>       at 
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> {noformat}
> This is not a good user experience, we should stop fail RM so that admin can 
> configure queue/labels in following steps:
> - Configure queue (with label)
> - Start RM
> - Add labels to RM
> - Submit applications
> Now admin has to:
> - Configure queue (without label)
> - Start RM
> - Add labels to RM
> - Refresh queue's config (with label)
> - Submit applications



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to