[jira] [Comment Edited] (YARN-10127) FSQueueConverter should not set App Ordering Policy to Parent Queue
[ https://issues.apache.org/jira/browse/YARN-10127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17034447#comment-17034447 ] Prabhu Joseph edited comment on YARN-10127 at 2/11/20 1:44 PM: --- [~snemeth] CS iterates ParentQueue (root queue as well) with Queue Ordering Policy. (supported values: utilization, priority-utilization) and iterates LeafQueue with App Ordering Policy (supported values: fair, fifo). But the config name is same for both Parent and Leaf queue - yarn.scheduler.capacity..ordering-policy causing the confusion. Setting fair to a ParentQueue is wrong as it accepts only utilization, priority-utilization. was (Author: prabhu joseph): [~snemeth] CS iterates ParentQueue (root queue as well) with Queue Ordering Policy. (supported values: utilization, priority-utilization) and iterates LeafQueue with App Ordering Policy (supported values: fair, fifo). But the config name is same for both Parent and Leaf queue - yarn.scheduler.capacity..ordering-policy causing the confusion. > FSQueueConverter should not set App Ordering Policy to Parent Queue > --- > > Key: YARN-10127 > URL: https://issues.apache.org/jira/browse/YARN-10127 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Peter Bacsko >Priority: Major > > FSQueueConverter should not set App Ordering Policy (fair, fifo) to Parent > Queue. RM will fail to start if Parent Queue is set with App Ordering Policy. > {code} > Error starting ResourceManager > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct > queue ordering policy=fair queue=root > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:1584) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:145) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractManagedParentQueue.(AbstractManagedParentQueue.java:51) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue.(ManagedParentQueue.java:56) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:272) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:158) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:751) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:829) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1247) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:324) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1534) > > {code} > Input fair-scheduler.xml: > {code} > [yarn@mradha-s1-1 /]$ cat /tmp/fair-scheduler.xml > > > > fair > > fair > > > fair > > > > > > > > > > > {code} > Command Used: > {code} > yarn fs2cs -t -f /tmp/fair-scheduler.xml -y > /var/run/cloudera-scm-agent/process/11-yarn-RESOURCEMANAGER/yarn-site.xml -o > /tmp/CS > {code} > Output capacity-scheduler.xml > {code} > > yarn.scheduler.capacity.root.auto-create-child-queue.enabledtruefalseprogrammatically > yarn.scheduler.capacity.root.users.capacity50.000falseprogrammatically > yarn.scheduler.capacity.root.queuesdefault,usersfalseprogrammatically > yarn.scheduler.capacity.queue-mappings-override.enablefalsefalseprogramma
[jira] [Comment Edited] (YARN-10127) FSQueueConverter should not set App Ordering Policy to Parent Queue
[ https://issues.apache.org/jira/browse/YARN-10127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17034388#comment-17034388 ] Szilard Nemeth edited comment on YARN-10127 at 2/11/20 12:09 PM: - Hi [~prabhujoseph], What's the rationale behind {quote} Root Queue is set with App Ordering Policy fair which is wrong {quote} ? Why CS doesn't accept this config? was (Author: snemeth): Hi [~prabhujoseph], What's the rationale behind {quote} Root Queue is set with App Ordering Policy fair which is wrong {quote} ? Why CS doesn't accept this config? > FSQueueConverter should not set App Ordering Policy to Parent Queue > --- > > Key: YARN-10127 > URL: https://issues.apache.org/jira/browse/YARN-10127 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Priority: Major > > FSQueueConverter should not set App Ordering Policy (fair, fifo) to Parent > Queue. RM will fail to start if Parent Queue is set with App Ordering Policy. > {code} > Error starting ResourceManager > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct > queue ordering policy=fair queue=root > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:1584) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:145) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractManagedParentQueue.(AbstractManagedParentQueue.java:51) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue.(ManagedParentQueue.java:56) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:272) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:158) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:751) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:829) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1247) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:324) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1534) > > {code} > Input fair-scheduler.xml: > {code} > [yarn@mradha-s1-1 /]$ cat /tmp/fair-scheduler.xml > > > > fair > > fair > > > fair > > > > > > > > > > > {code} > Command Used: > {code} > yarn fs2cs -t -f /tmp/fair-scheduler.xml -y > /var/run/cloudera-scm-agent/process/11-yarn-RESOURCEMANAGER/yarn-site.xml -o > /tmp/CS > {code} > Output capacity-scheduler.xml > {code} > > yarn.scheduler.capacity.root.auto-create-child-queue.enabledtruefalseprogrammatically > yarn.scheduler.capacity.root.users.capacity50.000falseprogrammatically > yarn.scheduler.capacity.root.queuesdefault,usersfalseprogrammatically > yarn.scheduler.capacity.queue-mappings-override.enablefalsefalseprogrammatically > yarn.scheduler.capacity.root.default.capacity50.000falseprogrammatically > yarn.scheduler.capacity.root.default.auto-create-child-queue.enabledtruefalseprogrammatically > yarn.scheduler.capacity.maximum-am-resource-percent0.5falseprogrammatically > yarn.scheduler.capacity.root.users.auto-create-child-queue.enabledtruefalseprogrammatically > yarn.scheduler.capacity.root.default.ordering-policyfairfalseprogrammatically > yarn.scheduler.capacity.queue-mappingsu:%user:%user;u:%use