[ https://issues.apache.org/jira/browse/YARN-10108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17025132#comment-17025132 ]
Peter Bacsko edited comment on YARN-10108 at 1/28/20 2:11 PM: -------------------------------------------------------------- Ok, here is the problem in {{UserGroupMappingPlacementRule}}: {noformat} private static QueueMapping validateAndGetAutoCreatedQueueMapping( CapacitySchedulerQueueManager queueManager, QueueMapping mapping, QueuePath queuePath) throws IOException { if (queuePath.hasParentQueue() && (queuePath.getParentQueue().equals(PRIMARY_GROUP_MAPPING) || queuePath.getParentQueue().equals(SECONDARY_GROUP_MAPPING))) { [...] } else if (queuePath.hasParentQueue()) { //if parent queue is specified, // then it should exist and be an instance of ManagedParentQueue QueuePlacementRuleUtils.validateQueueMappingUnderParentQueue( queueManager.getQueue(queuePath.getParentQueue()), <--- queuePath.getParentQueue() is "root.users" queuePath.getParentQueue(), queuePath.getLeafQueue()); return QueueMappingBuilder.create() .type(mapping.getType()) .source(mapping.getSource()) .queue(queuePath.getLeafQueue()) .parentQueue(queuePath.getParentQueue()) .build(); } {noformat} The problem is that {{queueManager.getQueue()}} expects a leaf queue name, not a full path. Since [~shuzirra] is working on YARN-9879 which enhances {{getQueue()}}, let's wait until that JIRA is committed. was (Author: pbacsko): Ok, here is the problem in {{UserGroupMappingPlacementRule}}: {noformat} [...] private static QueueMapping validateAndGetAutoCreatedQueueMapping( CapacitySchedulerQueueManager queueManager, QueueMapping mapping, QueuePath queuePath) throws IOException { if (queuePath.hasParentQueue() && (queuePath.getParentQueue().equals(PRIMARY_GROUP_MAPPING) || queuePath.getParentQueue().equals(SECONDARY_GROUP_MAPPING))) { [...] } else if (queuePath.hasParentQueue()) { //if parent queue is specified, // then it should exist and be an instance of ManagedParentQueue QueuePlacementRuleUtils.validateQueueMappingUnderParentQueue( queueManager.getQueue(queuePath.getParentQueue()), <--- queuePath.getParentQueue() is "root.users" queuePath.getParentQueue(), queuePath.getLeafQueue()); return QueueMappingBuilder.create() .type(mapping.getType()) .source(mapping.getSource()) .queue(queuePath.getLeafQueue()) .parentQueue(queuePath.getParentQueue()) .build(); } [...] {noformat} The problem is that {{queueManager.getQueue()}} expects a leaf queue name, not a full path. Since [~shuzirra] is working on YARN-9879 which enhances {{getQueue()}}, let's wait until that JIRA is committed. > FS-CS converter: nestedUserQueue with default rule maps to invalid queue > mapping > -------------------------------------------------------------------------------- > > Key: YARN-10108 > URL: https://issues.apache.org/jira/browse/YARN-10108 > Project: Hadoop YARN > Issue Type: Sub-task > Affects Versions: 3.3.0 > Reporter: Prabhu Joseph > Assignee: Peter Bacsko > Priority: Major > > FS Queue Placement Policy > {code:java} > <queuePlacementPolicy> > <rule name="nestedUserQueue"> > <rule name="default" queue="users"/> > </rule> > </queuePlacementPolicy> > {code} > gets mapped to an invalid CS queue mapping "u:%user:root.users.%user" > RM fails to start with above queue mapping in CS > {code:java} > 2020-01-28 00:19:12,889 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting > ResourceManager > org.apache.hadoop.service.ServiceStateException: java.io.IOException: mapping > contains invalid or non-leaf queue [%user] and invalid parent queue > [root.users] > at > org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:173) > at > org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:829) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1247) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:324) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1534) > Caused by: java.io.IOException: mapping contains invalid or non-leaf queue > [%user] and invalid parent queue [root.users] > at > org.apache.hadoop.yarn.server.resourcemanager.placement.QueuePlacementRuleUtils.validateQueueMappingUnderParentQueue(QueuePlacementRuleUtils.java:48) > at > org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule.validateAndGetAutoCreatedQueueMapping(UserGroupMappingPlacementRule.java:363) > at > org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule.initialize(UserGroupMappingPlacementRule.java:300) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.getUserGroupMappingPlacementRule(CapacityScheduler.java:671) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.updatePlacementRules(CapacityScheduler.java:712) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:753) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > ... 7 more > {code} > QueuePlacementConverter#handleNestedRule has to be fixed. > {code:java} > else if (pr instanceof DefaultPlacementRule) { > DefaultPlacementRule defaultRule = (DefaultPlacementRule) pr; > mapping.append("u:" + USER + ":") > .append(defaultRule.defaultQueueName) > .append("." + USER); > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org