[jira] [Commented] (YARN-10127) FSQueueConverter should not set App Ordering Policy to Parent Queue
[ https://issues.apache.org/jira/browse/YARN-10127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034597#comment-17034597 ] Hudson commented on YARN-10127: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17941 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17941/]) YARN-10127. Remove setting App Ordering Policy to ParentQueue in (pjoseph: rev e6377972113db844d7e7b99867fad19d3cbc58bd) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSQueueConverter.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSQueueConverter.java > FSQueueConverter should not set App Ordering Policy to Parent Queue > --- > > Key: YARN-10127 > URL: https://issues.apache.org/jira/browse/YARN-10127 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Peter Bacsko >Priority: Major > Labels: Reviewed > Fix For: 3.3.0 > > Attachments: YARN-10127-001.patch > > > FSQueueConverter should not set App Ordering Policy (fair, fifo) to Parent > Queue. RM will fail to start if Parent Queue is set with App Ordering Policy. > {code} > Error starting ResourceManager > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct > queue ordering policy=fair queue=root > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:1584) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:145) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractManagedParentQueue.(AbstractManagedParentQueue.java:51) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue.(ManagedParentQueue.java:56) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:272) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:158) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:751) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:829) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1247) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:324) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1534) > > {code} > Input fair-scheduler.xml: > {code} > [yarn@mradha-s1-1 /]$ cat /tmp/fair-scheduler.xml > > > > fair > > fair > > > fair > > > > > > > > > > > {code} > Command Used: > {code} > yarn fs2cs -t -f /tmp/fair-scheduler.xml -y > /var/run/cloudera-scm-agent/process/11-yarn-RESOURCEMANAGER/yarn-site.xml -o > /tmp/CS > {code} > Output capacity-scheduler.xml > {code} > > yarn.scheduler.capacity.root.auto-create-child-queue.enabledtruefalseprogrammatically > yarn.scheduler.capacity.root.users.capacity50.000falseprogrammatically > yarn.scheduler.capacity.root.queuesdefault,usersfalseprogrammatically > yarn.scheduler.capacity.queue-mappings-override.enablefalsefalseprogrammatically > yarn.scheduler.capacity.root.default.capacity50.000falseprogrammatically >
[jira] [Commented] (YARN-10127) FSQueueConverter should not set App Ordering Policy to Parent Queue
[ https://issues.apache.org/jira/browse/YARN-10127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034589#comment-17034589 ] Prabhu Joseph commented on YARN-10127: -- Have committed the patch to trunk. Will resolve the Jira. > FSQueueConverter should not set App Ordering Policy to Parent Queue > --- > > Key: YARN-10127 > URL: https://issues.apache.org/jira/browse/YARN-10127 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-10127-001.patch > > > FSQueueConverter should not set App Ordering Policy (fair, fifo) to Parent > Queue. RM will fail to start if Parent Queue is set with App Ordering Policy. > {code} > Error starting ResourceManager > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct > queue ordering policy=fair queue=root > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:1584) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:145) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractManagedParentQueue.(AbstractManagedParentQueue.java:51) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue.(ManagedParentQueue.java:56) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:272) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:158) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:751) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:829) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1247) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:324) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1534) > > {code} > Input fair-scheduler.xml: > {code} > [yarn@mradha-s1-1 /]$ cat /tmp/fair-scheduler.xml > > > > fair > > fair > > > fair > > > > > > > > > > > {code} > Command Used: > {code} > yarn fs2cs -t -f /tmp/fair-scheduler.xml -y > /var/run/cloudera-scm-agent/process/11-yarn-RESOURCEMANAGER/yarn-site.xml -o > /tmp/CS > {code} > Output capacity-scheduler.xml > {code} > > yarn.scheduler.capacity.root.auto-create-child-queue.enabledtruefalseprogrammatically > yarn.scheduler.capacity.root.users.capacity50.000falseprogrammatically > yarn.scheduler.capacity.root.queuesdefault,usersfalseprogrammatically > yarn.scheduler.capacity.queue-mappings-override.enablefalsefalseprogrammatically > yarn.scheduler.capacity.root.default.capacity50.000falseprogrammatically > yarn.scheduler.capacity.root.default.auto-create-child-queue.enabledtruefalseprogrammatically > yarn.scheduler.capacity.maximum-am-resource-percent0.5falseprogrammatically > yarn.scheduler.capacity.root.users.auto-create-child-queue.enabledtruefalseprogrammatically > yarn.scheduler.capacity.root.default.ordering-policyfairfalseprogrammatically > yarn.scheduler.capacity.queue-mappingsu:%user:%user;u:%user:root.users.%user;u:%user:root.defaultfalseprogrammatically > yarn.scheduler.capacity.root.users.ordering-policyfairfalseprogrammatically > yarn.scheduler.capacity.root.ordering-policyfairfalseprogrammatically > > {code} > Root Queue is set with App Ordering Policy fair which is wrong >
[jira] [Commented] (YARN-10127) FSQueueConverter should not set App Ordering Policy to Parent Queue
[ https://issues.apache.org/jira/browse/YARN-10127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034584#comment-17034584 ] Hadoop QA commented on YARN-10127: -- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 38s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 24s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 35s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 47s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 15m 28s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 37s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 9s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 87m 56s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 30s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}147m 57s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 | | JIRA Issue | YARN-10127 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12993136/YARN-10127-001.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 8a68761a3ae8 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / cc8ae59 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_242 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/25514/testReport/ | | Max. process+thread count | 820 (vs. ulimit of 5500) | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/25514/console | | Powered by | Apache Yetus 0.8.0 http://yetus.apache.org | This message was automatically generated. > FSQueueConverter should not set App Ordering
[jira] [Commented] (YARN-10127) FSQueueConverter should not set App Ordering Policy to Parent Queue
[ https://issues.apache.org/jira/browse/YARN-10127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034529#comment-17034529 ] Szilard Nemeth commented on YARN-10127: --- Thanks [~prabhujoseph] for your explanation. Thanks [~pbacsko], patch looks good to me as well. > FSQueueConverter should not set App Ordering Policy to Parent Queue > --- > > Key: YARN-10127 > URL: https://issues.apache.org/jira/browse/YARN-10127 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-10127-001.patch > > > FSQueueConverter should not set App Ordering Policy (fair, fifo) to Parent > Queue. RM will fail to start if Parent Queue is set with App Ordering Policy. > {code} > Error starting ResourceManager > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct > queue ordering policy=fair queue=root > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:1584) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:145) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractManagedParentQueue.(AbstractManagedParentQueue.java:51) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue.(ManagedParentQueue.java:56) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:272) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:158) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:751) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:829) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1247) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:324) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1534) > > {code} > Input fair-scheduler.xml: > {code} > [yarn@mradha-s1-1 /]$ cat /tmp/fair-scheduler.xml > > > > fair > > fair > > > fair > > > > > > > > > > > {code} > Command Used: > {code} > yarn fs2cs -t -f /tmp/fair-scheduler.xml -y > /var/run/cloudera-scm-agent/process/11-yarn-RESOURCEMANAGER/yarn-site.xml -o > /tmp/CS > {code} > Output capacity-scheduler.xml > {code} > > yarn.scheduler.capacity.root.auto-create-child-queue.enabledtruefalseprogrammatically > yarn.scheduler.capacity.root.users.capacity50.000falseprogrammatically > yarn.scheduler.capacity.root.queuesdefault,usersfalseprogrammatically > yarn.scheduler.capacity.queue-mappings-override.enablefalsefalseprogrammatically > yarn.scheduler.capacity.root.default.capacity50.000falseprogrammatically > yarn.scheduler.capacity.root.default.auto-create-child-queue.enabledtruefalseprogrammatically > yarn.scheduler.capacity.maximum-am-resource-percent0.5falseprogrammatically > yarn.scheduler.capacity.root.users.auto-create-child-queue.enabledtruefalseprogrammatically > yarn.scheduler.capacity.root.default.ordering-policyfairfalseprogrammatically > yarn.scheduler.capacity.queue-mappingsu:%user:%user;u:%user:root.users.%user;u:%user:root.defaultfalseprogrammatically > yarn.scheduler.capacity.root.users.ordering-policyfairfalseprogrammatically > yarn.scheduler.capacity.root.ordering-policyfairfalseprogrammatically > > {code} > Root Queue is set with App
[jira] [Commented] (YARN-10127) FSQueueConverter should not set App Ordering Policy to Parent Queue
[ https://issues.apache.org/jira/browse/YARN-10127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034473#comment-17034473 ] Prabhu Joseph commented on YARN-10127: -- Thanks [~pbacsko] for the patch. +1, waiting for jenkins. > FSQueueConverter should not set App Ordering Policy to Parent Queue > --- > > Key: YARN-10127 > URL: https://issues.apache.org/jira/browse/YARN-10127 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-10127-001.patch > > > FSQueueConverter should not set App Ordering Policy (fair, fifo) to Parent > Queue. RM will fail to start if Parent Queue is set with App Ordering Policy. > {code} > Error starting ResourceManager > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct > queue ordering policy=fair queue=root > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:1584) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:145) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractManagedParentQueue.(AbstractManagedParentQueue.java:51) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue.(ManagedParentQueue.java:56) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:272) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:158) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:751) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:829) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1247) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:324) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1534) > > {code} > Input fair-scheduler.xml: > {code} > [yarn@mradha-s1-1 /]$ cat /tmp/fair-scheduler.xml > > > > fair > > fair > > > fair > > > > > > > > > > > {code} > Command Used: > {code} > yarn fs2cs -t -f /tmp/fair-scheduler.xml -y > /var/run/cloudera-scm-agent/process/11-yarn-RESOURCEMANAGER/yarn-site.xml -o > /tmp/CS > {code} > Output capacity-scheduler.xml > {code} > > yarn.scheduler.capacity.root.auto-create-child-queue.enabledtruefalseprogrammatically > yarn.scheduler.capacity.root.users.capacity50.000falseprogrammatically > yarn.scheduler.capacity.root.queuesdefault,usersfalseprogrammatically > yarn.scheduler.capacity.queue-mappings-override.enablefalsefalseprogrammatically > yarn.scheduler.capacity.root.default.capacity50.000falseprogrammatically > yarn.scheduler.capacity.root.default.auto-create-child-queue.enabledtruefalseprogrammatically > yarn.scheduler.capacity.maximum-am-resource-percent0.5falseprogrammatically > yarn.scheduler.capacity.root.users.auto-create-child-queue.enabledtruefalseprogrammatically > yarn.scheduler.capacity.root.default.ordering-policyfairfalseprogrammatically > yarn.scheduler.capacity.queue-mappingsu:%user:%user;u:%user:root.users.%user;u:%user:root.defaultfalseprogrammatically > yarn.scheduler.capacity.root.users.ordering-policyfairfalseprogrammatically > yarn.scheduler.capacity.root.ordering-policyfairfalseprogrammatically > > {code} > Root Queue is set with App Ordering Policy fair which is wrong >
[jira] [Commented] (YARN-10127) FSQueueConverter should not set App Ordering Policy to Parent Queue
[ https://issues.apache.org/jira/browse/YARN-10127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034467#comment-17034467 ] Peter Bacsko commented on YARN-10127: - Thanks for the explanation [~prabhujoseph] - looks like the solution is to only emit this property for leaf queues. Otherwise skip it. Uploaded patch v1. > FSQueueConverter should not set App Ordering Policy to Parent Queue > --- > > Key: YARN-10127 > URL: https://issues.apache.org/jira/browse/YARN-10127 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-10127-001.patch > > > FSQueueConverter should not set App Ordering Policy (fair, fifo) to Parent > Queue. RM will fail to start if Parent Queue is set with App Ordering Policy. > {code} > Error starting ResourceManager > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct > queue ordering policy=fair queue=root > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:1584) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:145) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractManagedParentQueue.(AbstractManagedParentQueue.java:51) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue.(ManagedParentQueue.java:56) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:272) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:158) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:751) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:829) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1247) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:324) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1534) > > {code} > Input fair-scheduler.xml: > {code} > [yarn@mradha-s1-1 /]$ cat /tmp/fair-scheduler.xml > > > > fair > > fair > > > fair > > > > > > > > > > > {code} > Command Used: > {code} > yarn fs2cs -t -f /tmp/fair-scheduler.xml -y > /var/run/cloudera-scm-agent/process/11-yarn-RESOURCEMANAGER/yarn-site.xml -o > /tmp/CS > {code} > Output capacity-scheduler.xml > {code} > > yarn.scheduler.capacity.root.auto-create-child-queue.enabledtruefalseprogrammatically > yarn.scheduler.capacity.root.users.capacity50.000falseprogrammatically > yarn.scheduler.capacity.root.queuesdefault,usersfalseprogrammatically > yarn.scheduler.capacity.queue-mappings-override.enablefalsefalseprogrammatically > yarn.scheduler.capacity.root.default.capacity50.000falseprogrammatically > yarn.scheduler.capacity.root.default.auto-create-child-queue.enabledtruefalseprogrammatically > yarn.scheduler.capacity.maximum-am-resource-percent0.5falseprogrammatically > yarn.scheduler.capacity.root.users.auto-create-child-queue.enabledtruefalseprogrammatically > yarn.scheduler.capacity.root.default.ordering-policyfairfalseprogrammatically > yarn.scheduler.capacity.queue-mappingsu:%user:%user;u:%user:root.users.%user;u:%user:root.defaultfalseprogrammatically > yarn.scheduler.capacity.root.users.ordering-policyfairfalseprogrammatically >
[jira] [Commented] (YARN-10127) FSQueueConverter should not set App Ordering Policy to Parent Queue
[ https://issues.apache.org/jira/browse/YARN-10127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034447#comment-17034447 ] Prabhu Joseph commented on YARN-10127: -- [~snemeth] CS iterates ParentQueue (root queue as well) with Queue Ordering Policy. (supported values: utilization, priority-utilization) and iterates LeafQueue with App Ordering Policy (supported values: fair, fifo). But the config name is same for both Parent and Leaf queue - yarn.scheduler.capacity..ordering-policy causing the confusion. > FSQueueConverter should not set App Ordering Policy to Parent Queue > --- > > Key: YARN-10127 > URL: https://issues.apache.org/jira/browse/YARN-10127 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Peter Bacsko >Priority: Major > > FSQueueConverter should not set App Ordering Policy (fair, fifo) to Parent > Queue. RM will fail to start if Parent Queue is set with App Ordering Policy. > {code} > Error starting ResourceManager > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct > queue ordering policy=fair queue=root > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:1584) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:145) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractManagedParentQueue.(AbstractManagedParentQueue.java:51) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue.(ManagedParentQueue.java:56) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:272) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:158) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:751) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:829) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1247) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:324) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1534) > > {code} > Input fair-scheduler.xml: > {code} > [yarn@mradha-s1-1 /]$ cat /tmp/fair-scheduler.xml > > > > fair > > fair > > > fair > > > > > > > > > > > {code} > Command Used: > {code} > yarn fs2cs -t -f /tmp/fair-scheduler.xml -y > /var/run/cloudera-scm-agent/process/11-yarn-RESOURCEMANAGER/yarn-site.xml -o > /tmp/CS > {code} > Output capacity-scheduler.xml > {code} > > yarn.scheduler.capacity.root.auto-create-child-queue.enabledtruefalseprogrammatically > yarn.scheduler.capacity.root.users.capacity50.000falseprogrammatically > yarn.scheduler.capacity.root.queuesdefault,usersfalseprogrammatically > yarn.scheduler.capacity.queue-mappings-override.enablefalsefalseprogrammatically > yarn.scheduler.capacity.root.default.capacity50.000falseprogrammatically > yarn.scheduler.capacity.root.default.auto-create-child-queue.enabledtruefalseprogrammatically > yarn.scheduler.capacity.maximum-am-resource-percent0.5falseprogrammatically > yarn.scheduler.capacity.root.users.auto-create-child-queue.enabledtruefalseprogrammatically > yarn.scheduler.capacity.root.default.ordering-policyfairfalseprogrammatically >
[jira] [Commented] (YARN-10127) FSQueueConverter should not set App Ordering Policy to Parent Queue
[ https://issues.apache.org/jira/browse/YARN-10127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034388#comment-17034388 ] Szilard Nemeth commented on YARN-10127: --- Hi [~prabhujoseph], What's the rationale behind {quote} Root Queue is set with App Ordering Policy fair which is wrong {quote} ? Why CS doesn't accept this config? > FSQueueConverter should not set App Ordering Policy to Parent Queue > --- > > Key: YARN-10127 > URL: https://issues.apache.org/jira/browse/YARN-10127 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Priority: Major > > FSQueueConverter should not set App Ordering Policy (fair, fifo) to Parent > Queue. RM will fail to start if Parent Queue is set with App Ordering Policy. > {code} > Error starting ResourceManager > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct > queue ordering policy=fair queue=root > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:1584) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:145) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractManagedParentQueue.(AbstractManagedParentQueue.java:51) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue.(ManagedParentQueue.java:56) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:272) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:158) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:751) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:829) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1247) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:324) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1534) > > {code} > Input fair-scheduler.xml: > {code} > [yarn@mradha-s1-1 /]$ cat /tmp/fair-scheduler.xml > > > > fair > > fair > > > fair > > > > > > > > > > > {code} > Command Used: > {code} > yarn fs2cs -t -f /tmp/fair-scheduler.xml -y > /var/run/cloudera-scm-agent/process/11-yarn-RESOURCEMANAGER/yarn-site.xml -o > /tmp/CS > {code} > Output capacity-scheduler.xml > {code} > > yarn.scheduler.capacity.root.auto-create-child-queue.enabledtruefalseprogrammatically > yarn.scheduler.capacity.root.users.capacity50.000falseprogrammatically > yarn.scheduler.capacity.root.queuesdefault,usersfalseprogrammatically > yarn.scheduler.capacity.queue-mappings-override.enablefalsefalseprogrammatically > yarn.scheduler.capacity.root.default.capacity50.000falseprogrammatically > yarn.scheduler.capacity.root.default.auto-create-child-queue.enabledtruefalseprogrammatically > yarn.scheduler.capacity.maximum-am-resource-percent0.5falseprogrammatically > yarn.scheduler.capacity.root.users.auto-create-child-queue.enabledtruefalseprogrammatically > yarn.scheduler.capacity.root.default.ordering-policyfairfalseprogrammatically > yarn.scheduler.capacity.queue-mappingsu:%user:%user;u:%user:root.users.%user;u:%user:root.defaultfalseprogrammatically > yarn.scheduler.capacity.root.users.ordering-policyfairfalseprogrammatically > yarn.scheduler.capacity.root.ordering-policyfairfalseprogrammatically > > {code} > Root Queue is set with App Ordering