[jira] [Commented] (YARN-8268) Fair scheduler: reservable queue is configured both as parent and leaf queue
[ https://issues.apache.org/jira/browse/YARN-8268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473906#comment-16473906 ] Gergo Repas commented on YARN-8268: --- [~haibochen] - thanks for the review and committing the change! [~wilfreds] - thanks for the review! > Fair scheduler: reservable queue is configured both as parent and leaf queue > > > Key: YARN-8268 > URL: https://issues.apache.org/jira/browse/YARN-8268 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 3.1.0 >Reporter: Gergo Repas >Assignee: Gergo Repas >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8268.000.patch, YARN-8268.001.patch > > > The following allocation file > {code:java} > > > > someuser > > someuser > > > > > someuser > > > drf > > {code} > is being parsed as: {{PARENT=[root, root.dedicated], LEAF=[root.default, > root.dedicated]}} (AllocationConfiguration.configuredQueues). > The root.dedicated should only appear as a PARENT queue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8268) Fair scheduler: reservable queue is configured both as parent and leaf queue
[ https://issues.apache.org/jira/browse/YARN-8268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472502#comment-16472502 ] Hudson commented on YARN-8268: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14175 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14175/]) YARN-8268. Fair scheduler: reservable queue is configured both as parent (haibochen: rev 1f10a360219c91ac13d31bdb5c8d302b1b45afc3) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestAllocationFileLoaderService.java > Fair scheduler: reservable queue is configured both as parent and leaf queue > > > Key: YARN-8268 > URL: https://issues.apache.org/jira/browse/YARN-8268 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 3.1.0 >Reporter: Gergo Repas >Assignee: Gergo Repas >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8268.000.patch, YARN-8268.001.patch > > > The following allocation file > {code:java} > > > > someuser > > someuser > > > > > someuser > > > drf > > {code} > is being parsed as: {{PARENT=[root, root.dedicated], LEAF=[root.default, > root.dedicated]}} (AllocationConfiguration.configuredQueues). > The root.dedicated should only appear as a PARENT queue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8268) Fair scheduler: reservable queue is configured both as parent and leaf queue
[ https://issues.apache.org/jira/browse/YARN-8268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472452#comment-16472452 ] Haibo Chen commented on YARN-8268: -- Thanks [~grepas] for the contribution, [~wilfreds] for the additional review. I have checked in the patch to trunk > Fair scheduler: reservable queue is configured both as parent and leaf queue > > > Key: YARN-8268 > URL: https://issues.apache.org/jira/browse/YARN-8268 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 3.1.0 >Reporter: Gergo Repas >Assignee: Gergo Repas >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8268.000.patch, YARN-8268.001.patch > > > The following allocation file > {code:java} > > > > someuser > > someuser > > > > > someuser > > > drf > > {code} > is being parsed as: {{PARENT=[root, root.dedicated], LEAF=[root.default, > root.dedicated]}} (AllocationConfiguration.configuredQueues). > The root.dedicated should only appear as a PARENT queue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8268) Fair scheduler: reservable queue is configured both as parent and leaf queue
[ https://issues.apache.org/jira/browse/YARN-8268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472423#comment-16472423 ] Haibo Chen commented on YARN-8268: -- +1. Checking this in shortly. > Fair scheduler: reservable queue is configured both as parent and leaf queue > > > Key: YARN-8268 > URL: https://issues.apache.org/jira/browse/YARN-8268 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 3.1.0 >Reporter: Gergo Repas >Assignee: Gergo Repas >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8268.000.patch, YARN-8268.001.patch > > > The following allocation file > {code:java} > > > > someuser > > someuser > > > > > someuser > > > drf > > {code} > is being parsed as: {{PARENT=[root, root.dedicated], LEAF=[root.default, > root.dedicated]}} (AllocationConfiguration.configuredQueues). > The root.dedicated should only appear as a PARENT queue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8268) Fair scheduler: reservable queue is configured both as parent and leaf queue
[ https://issues.apache.org/jira/browse/YARN-8268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469833#comment-16469833 ] Wilfred Spiegelenburg commented on YARN-8268: - [~grepas] changes looking good no comments +1 (non binding) > Fair scheduler: reservable queue is configured both as parent and leaf queue > > > Key: YARN-8268 > URL: https://issues.apache.org/jira/browse/YARN-8268 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 3.1.0 >Reporter: Gergo Repas >Assignee: Gergo Repas >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8268.000.patch, YARN-8268.001.patch > > > The following allocation file > {code} > > > > someuser > > someuser > > > > > someuser > > > drf > > {code} > is being parsed as: {{PARENT=[root, root.dedicated], LEAF=[root.default, > root.dedicated]}} (AllocationConfiguration.configuredQueues). > The root.dedicated should only appear as a LEAF queue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8268) Fair scheduler: reservable queue is configured both as parent and leaf queue
[ https://issues.apache.org/jira/browse/YARN-8268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469108#comment-16469108 ] genericqa commented on YARN-8268: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 37s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 41s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 23s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 29s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 62m 39s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 24s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}121m 58s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd | | JIRA Issue | YARN-8268 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12922648/YARN-8268.001.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 05a3c05f9d99 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 343b51d | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_162 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/20662/testReport/ | | Max. process+thread count | 819 (vs. ulimit of 1) | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/20662/console | | Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Fair scheduler: reservable queue is configured
[jira] [Commented] (YARN-8268) Fair scheduler: reservable queue is configured both as parent and leaf queue
[ https://issues.apache.org/jira/browse/YARN-8268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468931#comment-16468931 ] Gergo Repas commented on YARN-8268: --- By looking at the test failures, the reservation queues should have been dropped from the LEAF queue set, I'm uploading a mew patch. > Fair scheduler: reservable queue is configured both as parent and leaf queue > > > Key: YARN-8268 > URL: https://issues.apache.org/jira/browse/YARN-8268 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 3.1.0 >Reporter: Gergo Repas >Assignee: Gergo Repas >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8268.000.patch, YARN-8268.001.patch > > > The following allocation file > {code} > > > > someuser > > someuser > > > > > someuser > > > drf > > {code} > is being parsed as: {{PARENT=[root, root.dedicated], LEAF=[root.default, > root.dedicated]}} (AllocationConfiguration.configuredQueues). > The root.dedicated should only appear as a LEAF queue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8268) Fair scheduler: reservable queue is configured both as parent and leaf queue
[ https://issues.apache.org/jira/browse/YARN-8268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468886#comment-16468886 ] Gergo Repas commented on YARN-8268: --- I'm checking the unit test failures. > Fair scheduler: reservable queue is configured both as parent and leaf queue > > > Key: YARN-8268 > URL: https://issues.apache.org/jira/browse/YARN-8268 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 3.1.0 >Reporter: Gergo Repas >Assignee: Gergo Repas >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8268.000.patch > > > The following allocation file > {code} > > > > someuser > > someuser > > > > > someuser > > > drf > > {code} > is being parsed as: {{PARENT=[root, root.dedicated], LEAF=[root.default, > root.dedicated]}} (AllocationConfiguration.configuredQueues). > The root.dedicated should only appear as a LEAF queue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8268) Fair scheduler: reservable queue is configured both as parent and leaf queue
[ https://issues.apache.org/jira/browse/YARN-8268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468821#comment-16468821 ] genericqa commented on YARN-8268: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 27s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 28s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 31s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 42s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 40s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 4s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 28s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 71m 43s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 19s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}124m 28s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.resourcemanager.reservation.TestFairSchedulerPlanFollower | | | hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd | | JIRA Issue | YARN-8268 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12922628/YARN-8268.000.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux e0659e1275b0 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / cd68c7c | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_162 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-YARN-Build/20658/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/20658/testReport/ | | Max. process+thread count | 871 (vs. ulimit of 1) | | modules | C:
[jira] [Commented] (YARN-8268) Fair scheduler: reservable queue is configured both as parent and leaf queue
[ https://issues.apache.org/jira/browse/YARN-8268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468701#comment-16468701 ] Gergo Repas commented on YARN-8268: --- When only applying the test changes (in TestAllocationFileLoaderService) to current trunk and running this test, the following assertion fails: {code:java} java.lang.AssertionError: reservable queue should not be a parent queue at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertFalse(Assert.java:64) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService.testReservableQueue(TestAllocationFileLoaderService.java:811) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ... {code} This demonstrates the problem with the current allocation file processing. The rest of the changes in the attached patch addresses this issue. > Fair scheduler: reservable queue is configured both as parent and leaf queue > > > Key: YARN-8268 > URL: https://issues.apache.org/jira/browse/YARN-8268 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 3.1.0 >Reporter: Gergo Repas >Assignee: Gergo Repas >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8268.000.patch > > > The following allocation file > {code} > > > > someuser > > someuser > > > > > someuser > > > drf > > {code} > is being parsed as: {{PARENT=[root, root.dedicated], LEAF=[root.default, > root.dedicated]}} (AllocationConfiguration.configuredQueues). > The root.dedicated should only appear as a LEAF queue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org