[jira] [Commented] (YARN-9580) Fulfilled reservation information in assignment is lost when transferring in ParentQueue#assignContainers
[ https://issues.apache.org/jira/browse/YARN-9580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855366#comment-16855366 ] Hadoop QA commented on YARN-9580: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} branch-3.2 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 57s{color} | {color:green} branch-3.2 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s{color} | {color:green} branch-3.2 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 30s{color} | {color:green} branch-3.2 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 47s{color} | {color:green} branch-3.2 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 50s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 13s{color} | {color:green} branch-3.2 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s{color} | {color:green} branch-3.2 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 4s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 68m 51s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 22s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}123m 1s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:63396be | | JIRA Issue | YARN-9580 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12970765/YARN-9580.branch-3.2.002.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 797c6d086e44 4.4.0-143-generic #169~14.04.2-Ubuntu SMP Wed Feb 13 15:00:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | branch-3.2 / 2f01204 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_212 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/24214/testReport/ | | Max. process+thread count | 914 (vs. ulimit of 1) | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/24214/console | | Powered by | Apache Yetus 0.8.0 http://yetus.apache.org | This message was aut
[jira] [Commented] (YARN-9580) Fulfilled reservation information in assignment is lost when transferring in ParentQueue#assignContainers
[ https://issues.apache.org/jira/browse/YARN-9580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855289#comment-16855289 ] Tao Yang commented on YARN-9580: Sorry about forgetting to check imports in UT class. Attached v2 patch for branch-3.2 to correct it. > Fulfilled reservation information in assignment is lost when transferring in > ParentQueue#assignContainers > - > > Key: YARN-9580 > URL: https://issues.apache.org/jira/browse/YARN-9580 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Reporter: Tao Yang >Assignee: Tao Yang >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9580.001.patch, YARN-9580.branch-3.2.001.patch, > YARN-9580.branch-3.2.002.patch > > > When transferring assignment from child queue to parent queue, fulfilled > reservation information including fulfilledReservation and > fulfilledReservedContainer in assignment is lost. > When multi-nodes enabled, this lost can raise a problem that allocation > proposal is generated but can't be accepted because there is a check for > fulfilled reservation information in > FiCaSchedulerApp#commonCheckContainerAllocation, this endless loop will > always be there and the resource of the node can't be used anymore. > In HB-driven scheduling mode, fulfilled reservation can be allocated via > another calling stack: CapacityScheduler#allocateContainersToNode --> > CapacityScheduler#allocateContainerOnSingleNode --> > CapacityScheduler#allocateFromReservedContainer, in this way assignment can > be generated by leaf queue and directly submitted, I think that's why we > hardly find this problem before. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9580) Fulfilled reservation information in assignment is lost when transferring in ParentQueue#assignContainers
[ https://issues.apache.org/jira/browse/YARN-9580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855277#comment-16855277 ] Weiwei Yang commented on YARN-9580: --- Hi [~Tao Yang] There seems to have issues in the patch for branch-3.2, could you please take a look? > Fulfilled reservation information in assignment is lost when transferring in > ParentQueue#assignContainers > - > > Key: YARN-9580 > URL: https://issues.apache.org/jira/browse/YARN-9580 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Reporter: Tao Yang >Assignee: Tao Yang >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9580.001.patch, YARN-9580.branch-3.2.001.patch > > > When transferring assignment from child queue to parent queue, fulfilled > reservation information including fulfilledReservation and > fulfilledReservedContainer in assignment is lost. > When multi-nodes enabled, this lost can raise a problem that allocation > proposal is generated but can't be accepted because there is a check for > fulfilled reservation information in > FiCaSchedulerApp#commonCheckContainerAllocation, this endless loop will > always be there and the resource of the node can't be used anymore. > In HB-driven scheduling mode, fulfilled reservation can be allocated via > another calling stack: CapacityScheduler#allocateContainersToNode --> > CapacityScheduler#allocateContainerOnSingleNode --> > CapacityScheduler#allocateFromReservedContainer, in this way assignment can > be generated by leaf queue and directly submitted, I think that's why we > hardly find this problem before. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9580) Fulfilled reservation information in assignment is lost when transferring in ParentQueue#assignContainers
[ https://issues.apache.org/jira/browse/YARN-9580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855274#comment-16855274 ] Hadoop QA commented on YARN-9580: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} branch-3.2 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m 24s{color} | {color:green} branch-3.2 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s{color} | {color:green} branch-3.2 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 35s{color} | {color:green} branch-3.2 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s{color} | {color:green} branch-3.2 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 27s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 35s{color} | {color:green} branch-3.2 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s{color} | {color:green} branch-3.2 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 49s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 46s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 46s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 31s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 50s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 4m 30s{color} | {color:red} patch has errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 37s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 55s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 33s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 57m 31s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:63396be | | JIRA Issue | YARN-9580 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12970763/YARN-9580.branch-3.2.001.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 1f1e5c31ef5e 4.4.0-143-generic #169~14.04.2-Ubuntu SMP Wed Feb 13 15:00:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | branch-3.2 / 2f01204 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_212 | | findbugs | v3.1.0-RC1 | | mvninstall | https://builds.apache.org/job/PreCommit-YARN-Build/24213/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | compile | https://builds.apache.org/job/PreCommit-YARN-Build/24213/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager
[jira] [Commented] (YARN-9580) Fulfilled reservation information in assignment is lost when transferring in ParentQueue#assignContainers
[ https://issues.apache.org/jira/browse/YARN-9580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855221#comment-16855221 ] Tao Yang commented on YARN-9580: Sure, thanks [~cheersyang] for the review and commit. I have found another problems about reservation and commented above, can you take a look and I think we should have a discuss about reservation and make it able to work when multi-node enabled. > Fulfilled reservation information in assignment is lost when transferring in > ParentQueue#assignContainers > - > > Key: YARN-9580 > URL: https://issues.apache.org/jira/browse/YARN-9580 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Reporter: Tao Yang >Assignee: Tao Yang >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9580.001.patch > > > When transferring assignment from child queue to parent queue, fulfilled > reservation information including fulfilledReservation and > fulfilledReservedContainer in assignment is lost. > When multi-nodes enabled, this lost can raise a problem that allocation > proposal is generated but can't be accepted because there is a check for > fulfilled reservation information in > FiCaSchedulerApp#commonCheckContainerAllocation, this endless loop will > always be there and the resource of the node can't be used anymore. > In HB-driven scheduling mode, fulfilled reservation can be allocated via > another calling stack: CapacityScheduler#allocateContainersToNode --> > CapacityScheduler#allocateContainerOnSingleNode --> > CapacityScheduler#allocateFromReservedContainer, in this way assignment can > be generated by leaf queue and directly submitted, I think that's why we > hardly find this problem before. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9580) Fulfilled reservation information in assignment is lost when transferring in ParentQueue#assignContainers
[ https://issues.apache.org/jira/browse/YARN-9580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854721#comment-16854721 ] Hudson commented on YARN-9580: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16654 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16654/]) YARN-9580. Fulfilled reservation information in assignment is lost when (wwei: rev bd2590d71ba1f3db1c686f7afeaf51382f8d8a2f) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerMultiNodes.java > Fulfilled reservation information in assignment is lost when transferring in > ParentQueue#assignContainers > - > > Key: YARN-9580 > URL: https://issues.apache.org/jira/browse/YARN-9580 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Reporter: Tao Yang >Assignee: Tao Yang >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9580.001.patch > > > When transferring assignment from child queue to parent queue, fulfilled > reservation information including fulfilledReservation and > fulfilledReservedContainer in assignment is lost. > When multi-nodes enabled, this lost can raise a problem that allocation > proposal is generated but can't be accepted because there is a check for > fulfilled reservation information in > FiCaSchedulerApp#commonCheckContainerAllocation, this endless loop will > always be there and the resource of the node can't be used anymore. > In HB-driven scheduling mode, fulfilled reservation can be allocated via > another calling stack: CapacityScheduler#allocateContainersToNode --> > CapacityScheduler#allocateContainerOnSingleNode --> > CapacityScheduler#allocateFromReservedContainer, in this way assignment can > be generated by leaf queue and directly submitted, I think that's why we > hardly find this problem before. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9580) Fulfilled reservation information in assignment is lost when transferring in ParentQueue#assignContainers
[ https://issues.apache.org/jira/browse/YARN-9580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854701#comment-16854701 ] Weiwei Yang commented on YARN-9580: --- Committed to trunk. [~Tao Yang], can you please provide a patch for branch-3.2 too? > Fulfilled reservation information in assignment is lost when transferring in > ParentQueue#assignContainers > - > > Key: YARN-9580 > URL: https://issues.apache.org/jira/browse/YARN-9580 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Reporter: Tao Yang >Assignee: Tao Yang >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9580.001.patch > > > When transferring assignment from child queue to parent queue, fulfilled > reservation information including fulfilledReservation and > fulfilledReservedContainer in assignment is lost. > When multi-nodes enabled, this lost can raise a problem that allocation > proposal is generated but can't be accepted because there is a check for > fulfilled reservation information in > FiCaSchedulerApp#commonCheckContainerAllocation, this endless loop will > always be there and the resource of the node can't be used anymore. > In HB-driven scheduling mode, fulfilled reservation can be allocated via > another calling stack: CapacityScheduler#allocateContainersToNode --> > CapacityScheduler#allocateContainerOnSingleNode --> > CapacityScheduler#allocateFromReservedContainer, in this way assignment can > be generated by leaf queue and directly submitted, I think that's why we > hardly find this problem before. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9580) Fulfilled reservation information in assignment is lost when transferring in ParentQueue#assignContainers
[ https://issues.apache.org/jira/browse/YARN-9580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854697#comment-16854697 ] Weiwei Yang commented on YARN-9580: --- [~Tao Yang], thanks for the patch, it makes sense to me. +1. > Fulfilled reservation information in assignment is lost when transferring in > ParentQueue#assignContainers > - > > Key: YARN-9580 > URL: https://issues.apache.org/jira/browse/YARN-9580 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Reporter: Tao Yang >Assignee: Tao Yang >Priority: Major > Attachments: YARN-9580.001.patch > > > When transferring assignment from child queue to parent queue, fulfilled > reservation information including fulfilledReservation and > fulfilledReservedContainer in assignment is lost. > When multi-nodes enabled, this lost can raise a problem that allocation > proposal is generated but can't be accepted because there is a check for > fulfilled reservation information in > FiCaSchedulerApp#commonCheckContainerAllocation, this endless loop will > always be there and the resource of the node can't be used anymore. > In HB-driven scheduling mode, fulfilled reservation can be allocated via > another calling stack: CapacityScheduler#allocateContainersToNode --> > CapacityScheduler#allocateContainerOnSingleNode --> > CapacityScheduler#allocateFromReservedContainer, in this way assignment can > be generated by leaf queue and directly submitted, I think that's why we > hardly find this problem before. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9580) Fulfilled reservation information in assignment is lost when transferring in ParentQueue#assignContainers
[ https://issues.apache.org/jira/browse/YARN-9580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16851623#comment-16851623 ] Tao Yang commented on YARN-9580: Found another problem about reservation when multi-node enabled, scheduler iterates all nodes and try to allocate for reserved container in LeafQueue#allocateFromReservedContainer. Here there are two problems: # The node of reserved container should be taken as candidates instead of all nodes when calling FiCaSchedulerApp#assignContainers, otherwise later scheduler may generate a reservation-fulfilled proposal on another node, which will always be rejected when committing. # Assignment returned by FiCaSchedulerApp#assignContainers could never be null even if it's just skipped, it will break the normal scheduling process for this leaf queue because of the if clause in LeafQueue#assignContainers: "if (null != assignment) \{ return assignment;}" > Fulfilled reservation information in assignment is lost when transferring in > ParentQueue#assignContainers > - > > Key: YARN-9580 > URL: https://issues.apache.org/jira/browse/YARN-9580 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Reporter: Tao Yang >Assignee: Tao Yang >Priority: Major > Attachments: YARN-9580.001.patch > > > When transferring assignment from child queue to parent queue, fulfilled > reservation information including fulfilledReservation and > fulfilledReservedContainer in assignment is lost. > When multi-nodes enabled, this lost can raise a problem that allocation > proposal is generated but can't be accepted because there is a check for > fulfilled reservation information in > FiCaSchedulerApp#commonCheckContainerAllocation, this endless loop will > always be there and the resource of the node can't be used anymore. > In HB-driven scheduling mode, fulfilled reservation can be allocated via > another calling stack: CapacityScheduler#allocateContainersToNode --> > CapacityScheduler#allocateContainerOnSingleNode --> > CapacityScheduler#allocateFromReservedContainer, in this way assignment can > be generated by leaf queue and directly submitted, I think that's why we > hardly find this problem before. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9580) Fulfilled reservation information in assignment is lost when transferring in ParentQueue#assignContainers
[ https://issues.apache.org/jira/browse/YARN-9580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16848603#comment-16848603 ] Tao Yang commented on YARN-9580: UT Failures about fair scheduler and state store seem unrelated to this patch, can't reproduce them on my local environment. [~cheersyang], could you please help to review this patch? > Fulfilled reservation information in assignment is lost when transferring in > ParentQueue#assignContainers > - > > Key: YARN-9580 > URL: https://issues.apache.org/jira/browse/YARN-9580 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Reporter: Tao Yang >Assignee: Tao Yang >Priority: Major > Attachments: YARN-9580.001.patch > > > When transferring assignment from child queue to parent queue, fulfilled > reservation information including fulfilledReservation and > fulfilledReservedContainer in assignment is lost. > When multi-nodes enabled, this lost can raise a problem that allocation > proposal is generated but can't be accepted because there is a check for > fulfilled reservation information in > FiCaSchedulerApp#commonCheckContainerAllocation, this endless loop will > always be there and the resource of the node can't be used anymore. > In HB-driven scheduling mode, fulfilled reservation can be allocated via > another calling stack: CapacityScheduler#allocateContainersToNode --> > CapacityScheduler#allocateContainerOnSingleNode --> > CapacityScheduler#allocateFromReservedContainer, in this way assignment can > be generated by leaf queue and directly submitted, I think that's why we > hardly find this problem before. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9580) Fulfilled reservation information in assignment is lost when transferring in ParentQueue#assignContainers
[ https://issues.apache.org/jira/browse/YARN-9580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16847468#comment-16847468 ] Hadoop QA commented on YARN-9580: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 29s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 34s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 46s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 22s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 23s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 11s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 9s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 25s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}128m 46s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.resourcemanager.TestLeaderElectorService | | | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e | | JIRA Issue | YARN-9580 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12969609/YARN-9580.001.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 4005624868e2 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 460ba7f | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_212 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-YARN-Build/24145/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/24145/testReport/ | | Max. process+thread count | 919 (vs. ulimit of 1) | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoo