[jira] [Commented] (YARN-10589) Improve logic of multi-node allocation
[ https://issues.apache.org/jira/browse/YARN-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286909#comment-17286909 ] Qi Zhu commented on YARN-10589: --- [~tanu.ajmera] There still some checkstyle should handle. > Improve logic of multi-node allocation > -- > > Key: YARN-10589 > URL: https://issues.apache.org/jira/browse/YARN-10589 > Project: Hadoop YARN > Issue Type: Task >Affects Versions: 3.3.0 >Reporter: Tanu Ajmera >Assignee: Tanu Ajmera >Priority: Major > Attachments: YARN-10589-001.patch, YARN-10589-002.patch, > YARN-10589-003.patch, YARN-10589-004.patch, YARN-10589-005.patch > > > {code:java} > for (String partititon : partitions) { > if (current++ > start) { > break; > } > CandidateNodeSet candidates = > cs.getCandidateNodeSet(partititon); > if (candidates == null) { > continue; > } > cs.allocateContainersToNode(candidates, false); > }{code} > In above logic, if we have thousands of node in one partition, we will still > repeatedly access all nodes of the partition thousands of times. There is no > break point where if the partition is not same for the first node, it should > stop checking other nodes in that partition. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10589) Improve logic of multi-node allocation
[ https://issues.apache.org/jira/browse/YARN-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17280866#comment-17280866 ] Hadoop QA commented on YARN-10589: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 12s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 20s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 45s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 17m 0s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m 50s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 49s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 48s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 46s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 39s{color} | {color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/601/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 2 new + 37 unchanged - 0 fixed = 39 total (was 37) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 48s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 1s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 58s{color} | {color:green}{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | |
[jira] [Commented] (YARN-10589) Improve logic of multi-node allocation
[ https://issues.apache.org/jira/browse/YARN-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17280774#comment-17280774 ] Tanu Ajmera commented on YARN-10589: [~zhuqi] Thanks for the review. I have fixed the checkstyle and uploaded a new patch > Improve logic of multi-node allocation > -- > > Key: YARN-10589 > URL: https://issues.apache.org/jira/browse/YARN-10589 > Project: Hadoop YARN > Issue Type: Task >Affects Versions: 3.3.0 >Reporter: Tanu Ajmera >Assignee: Tanu Ajmera >Priority: Major > Attachments: YARN-10589-001.patch, YARN-10589-002.patch, > YARN-10589-003.patch, YARN-10589-004.patch, YARN-10589-005.patch > > > {code:java} > for (String partititon : partitions) { > if (current++ > start) { > break; > } > CandidateNodeSet candidates = > cs.getCandidateNodeSet(partititon); > if (candidates == null) { > continue; > } > cs.allocateContainersToNode(candidates, false); > }{code} > In above logic, if we have thousands of node in one partition, we will still > repeatedly access all nodes of the partition thousands of times. There is no > break point where if the partition is not same for the first node, it should > stop checking other nodes in that partition. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10589) Improve logic of multi-node allocation
[ https://issues.apache.org/jira/browse/YARN-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17279389#comment-17279389 ] Qi Zhu commented on YARN-10589: --- Thanks for [~tanu.ajmera] new patch. The new code LGTM. Should also fix the checkstyle. The test error is not related to this. > Improve logic of multi-node allocation > -- > > Key: YARN-10589 > URL: https://issues.apache.org/jira/browse/YARN-10589 > Project: Hadoop YARN > Issue Type: Task >Affects Versions: 3.3.0 >Reporter: Tanu Ajmera >Assignee: Tanu Ajmera >Priority: Major > Attachments: YARN-10589-001.patch, YARN-10589-002.patch, > YARN-10589-003.patch, YARN-10589-004.patch > > > {code:java} > for (String partititon : partitions) { > if (current++ > start) { > break; > } > CandidateNodeSet candidates = > cs.getCandidateNodeSet(partititon); > if (candidates == null) { > continue; > } > cs.allocateContainersToNode(candidates, false); > }{code} > In above logic, if we have thousands of node in one partition, we will still > repeatedly access all nodes of the partition thousands of times. There is no > break point where if the partition is not same for the first node, it should > stop checking other nodes in that partition. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10589) Improve logic of multi-node allocation
[ https://issues.apache.org/jira/browse/YARN-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17278770#comment-17278770 ] Hadoop QA commented on YARN-10589: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 54s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 38m 16s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 12s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 54s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 5s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 17m 44s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m 28s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 24s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 58s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 10s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 10s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 59s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 48s{color} | {color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/586/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 11 new + 37 unchanged - 0 fixed = 48 total (was 37) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 59s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 47s{color} | {color:green}{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | |
[jira] [Commented] (YARN-10589) Improve logic of multi-node allocation
[ https://issues.apache.org/jira/browse/YARN-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17278747#comment-17278747 ] Tanu Ajmera commented on YARN-10589: [~zhuqi] [~ztang] Thanks for the review. I have split the code for partition and improved the patch. Please review the latest patch > Improve logic of multi-node allocation > -- > > Key: YARN-10589 > URL: https://issues.apache.org/jira/browse/YARN-10589 > Project: Hadoop YARN > Issue Type: Task >Affects Versions: 3.3.0 >Reporter: Tanu Ajmera >Assignee: Tanu Ajmera >Priority: Major > Attachments: YARN-10589-001.patch, YARN-10589-002.patch, > YARN-10589-003.patch, YARN-10589-004.patch > > > {code:java} > for (String partititon : partitions) { > if (current++ > start) { > break; > } > CandidateNodeSet candidates = > cs.getCandidateNodeSet(partititon); > if (candidates == null) { > continue; > } > cs.allocateContainersToNode(candidates, false); > }{code} > In above logic, if we have thousands of node in one partition, we will still > repeatedly access all nodes of the partition thousands of times. There is no > break point where if the partition is not same for the first node, it should > stop checking other nodes in that partition. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10589) Improve logic of multi-node allocation
[ https://issues.apache.org/jira/browse/YARN-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17277085#comment-17277085 ] Hadoop QA commented on YARN-10589: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 20s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 51s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 45s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 58s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 17m 0s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m 49s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 47s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/571/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in trunk has 1 extant findbugs warnings. {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 49s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 54s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 38s{color} | {color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/571/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 16 unchanged - 0 fixed = 17 total (was 16) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 48s{color} | {color:green}{color} | {color:green} the
[jira] [Commented] (YARN-10589) Improve logic of multi-node allocation
[ https://issues.apache.org/jira/browse/YARN-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17277031#comment-17277031 ] Qi Zhu commented on YARN-10589: --- [~tanu.ajmera] I agree with [~ztang] that we need to split the code to partition related logic, then to return: {code:java} public boolean precheckNode(SchedulerRequestKey schedulerKey, SchedulerNode schedulerNode, SchedulingMode schedulingMode, Optional dcOpt) { this.readLock.lock(); try { AppPlacementAllocator ap = schedulerKeyToAppPlacementAllocator.get(schedulerKey); return (ap != null) && (ap.getPlacementAttempt() < retryAttempts) && ap.precheckNode(schedulerNode, schedulingMode, dcOpt); } finally { this.readLock.unlock(); } } {code} > Improve logic of multi-node allocation > -- > > Key: YARN-10589 > URL: https://issues.apache.org/jira/browse/YARN-10589 > Project: Hadoop YARN > Issue Type: Task >Affects Versions: 3.3.0 >Reporter: Tanu Ajmera >Assignee: Tanu Ajmera >Priority: Major > Attachments: YARN-10589-001.patch, YARN-10589-002.patch, > YARN-10589-003.patch > > > {code:java} > for (String partititon : partitions) { > if (current++ > start) { > break; > } > CandidateNodeSet candidates = > cs.getCandidateNodeSet(partititon); > if (candidates == null) { > continue; > } > cs.allocateContainersToNode(candidates, false); > }{code} > In above logic, if we have thousands of node in one partition, we will still > repeatedly access all nodes of the partition thousands of times. There is no > break point where if the partition is not same for the first node, it should > stop checking other nodes in that partition. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10589) Improve logic of multi-node allocation
[ https://issues.apache.org/jira/browse/YARN-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276992#comment-17276992 ] Qi Zhu commented on YARN-10589: --- [~ztang] [~tanu.ajmera] I attached a new patch 003 based 002 , i think return PARTITION_SKIPPED will return and get next PRIORITY_SKIPPED in : {code:java} while (iter.hasNext()) { FiCaSchedulerNode node = iter.next(); if (reservedContainer == null) { result = preCheckForNodeCandidateSet(clusterResource, node, schedulingMode, resourceLimits, schedulerKey); if (null != result) { if (result == ContainerAllocation.PARTITION_SKIPPED) { return result; } else { continue; } } } {code} Thanks. > Improve logic of multi-node allocation > -- > > Key: YARN-10589 > URL: https://issues.apache.org/jira/browse/YARN-10589 > Project: Hadoop YARN > Issue Type: Task >Affects Versions: 3.3.0 >Reporter: Tanu Ajmera >Assignee: Tanu Ajmera >Priority: Major > Attachments: YARN-10589-001.patch, YARN-10589-002.patch, > YARN-10589-003.patch > > > {code:java} > for (String partititon : partitions) { > if (current++ > start) { > break; > } > CandidateNodeSet candidates = > cs.getCandidateNodeSet(partititon); > if (candidates == null) { > continue; > } > cs.allocateContainersToNode(candidates, false); > }{code} > In above logic, if we have thousands of node in one partition, we will still > repeatedly access all nodes of the partition thousands of times. There is no > break point where if the partition is not same for the first node, it should > stop checking other nodes in that partition. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10589) Improve logic of multi-node allocation
[ https://issues.apache.org/jira/browse/YARN-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276973#comment-17276973 ] Zhankun Tang commented on YARN-10589: - [~zhuqi], Thanks a lot for the review! [~tanu.ajmera], I'm not very clear what we are doing now. When we change PRIORITY_SKIPPED to PARTITION_SKIPPED, what's the difference if we use PRIORITY_SKIPPED to skip the node iteration? > Improve logic of multi-node allocation > -- > > Key: YARN-10589 > URL: https://issues.apache.org/jira/browse/YARN-10589 > Project: Hadoop YARN > Issue Type: Task >Affects Versions: 3.3.0 >Reporter: Tanu Ajmera >Assignee: Tanu Ajmera >Priority: Major > Attachments: YARN-10589-001.patch, YARN-10589-002.patch > > > {code:java} > for (String partititon : partitions) { > if (current++ > start) { > break; > } > CandidateNodeSet candidates = > cs.getCandidateNodeSet(partititon); > if (candidates == null) { > continue; > } > cs.allocateContainersToNode(candidates, false); > }{code} > In above logic, if we have thousands of node in one partition, we will still > repeatedly access all nodes of the partition thousands of times. There is no > break point where if the partition is not same for the first node, it should > stop checking other nodes in that partition. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10589) Improve logic of multi-node allocation
[ https://issues.apache.org/jira/browse/YARN-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276427#comment-17276427 ] Hadoop QA commented on YARN-10589: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 50s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m 53s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 0s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 4s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 19m 1s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/569/artifact/out/branch-shadedclient.txt{color} | {color:red} branch has errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 28s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/569/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04.txt{color} | {color:red} hadoop-yarn-server-resourcemanager in trunk failed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 29s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/569/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01.txt{color} | {color:red} hadoop-yarn-server-resourcemanager in trunk failed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01. {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 20m 29s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 28s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/569/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color} | {color:red} hadoop-yarn-server-resourcemanager in trunk failed. {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 23s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/569/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 23s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/569/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04.txt{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 23s{color} |
[jira] [Commented] (YARN-10589) Improve logic of multi-node allocation
[ https://issues.apache.org/jira/browse/YARN-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276369#comment-17276369 ] zhuqi commented on YARN-10589: -- [~ztang] [~tanu.ajmera] I have reviewed the patch, i don't think the logic is needed, and it break the PRIORITY_SKIPPED logic. In Schedule in priority order: {code:java} // Schedule in priority order for (SchedulerRequestKey schedulerKey : application.getSchedulerKeys()) { ContainerAllocation result = allocate(clusterResource, candidates, schedulingMode, resourceLimits, schedulerKey, null); AllocationState allocationState = result.getAllocationState(); if (allocationState == AllocationState.PRIORITY_SKIPPED) { continue; } return getCSAssignmentFromAllocateResult(clusterResource, result, null, node); } if (reservedContainer == null) { result = preCheckForNodeCandidateSet(clusterResource, node, schedulingMode, resourceLimits, schedulerKey); if (null != result) { continue; } } {code} I think the original logic is right, in preCheckForNodeCandidateSet: {code:java} // Is the nodePartition of pending request matches the node's partition // If not match, jump to next priority. Optional dcOpt = activitiesManager == null ? Optional.empty() : activitiesManager.getOptionalDiagnosticsCollector(); if (!appInfo.precheckNode(schedulerKey, node, schedulingMode, dcOpt)) { ActivitiesLogger.APP.recordSkippedAppActivityWithoutAllocation( activitiesManager, node, application, schedulerKey, ActivityDiagnosticConstant. NODE_DO_NOT_MATCH_PARTITION_OR_PLACEMENT_CONSTRAINTS + ActivitiesManager.getDiagnostics(dcOpt), ActivityLevel.NODE); return ContainerAllocation.PRIORITY_SKIPPED; } {code} We have skipped the priority, and the next priority may need this node partition. We just skipped the priority of application. Any other thoughts? > Improve logic of multi-node allocation > -- > > Key: YARN-10589 > URL: https://issues.apache.org/jira/browse/YARN-10589 > Project: Hadoop YARN > Issue Type: Task >Affects Versions: 3.3.0 >Reporter: Tanu Ajmera >Assignee: Tanu Ajmera >Priority: Major > Attachments: YARN-10589-001.patch > > > {code:java} > for (String partititon : partitions) { > if (current++ > start) { > break; > } > CandidateNodeSet candidates = > cs.getCandidateNodeSet(partititon); > if (candidates == null) { > continue; > } > cs.allocateContainersToNode(candidates, false); > }{code} > In above logic, if we have thousands of node in one partition, we will still > repeatedly access all nodes of the partition thousands of times. There is no > break point where if the partition is not same for the first node, it should > stop checking other nodes in that partition. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10589) Improve logic of multi-node allocation
[ https://issues.apache.org/jira/browse/YARN-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276170#comment-17276170 ] Zhankun Tang commented on YARN-10589: - [~zhuqi], could you please review Tanu's patch too? > Improve logic of multi-node allocation > -- > > Key: YARN-10589 > URL: https://issues.apache.org/jira/browse/YARN-10589 > Project: Hadoop YARN > Issue Type: Task >Affects Versions: 3.3.0 >Reporter: Tanu Ajmera >Assignee: Tanu Ajmera >Priority: Major > Attachments: YARN-10589-001.patch > > > {code:java} > for (String partititon : partitions) { > if (current++ > start) { > break; > } > CandidateNodeSet candidates = > cs.getCandidateNodeSet(partititon); > if (candidates == null) { > continue; > } > cs.allocateContainersToNode(candidates, false); > }{code} > In above logic, if we have thousands of node in one partition, we will still > repeatedly access all nodes of the partition thousands of times. There is no > break point where if the partition is not same for the first node, it should > stop checking other nodes in that partition. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10589) Improve logic of multi-node allocation
[ https://issues.apache.org/jira/browse/YARN-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17273523#comment-17273523 ] Hadoop QA commented on YARN-10589: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 15s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 49s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 46s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 58s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 17m 40s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m 56s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 54s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/559/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in trunk has 1 extant findbugs warnings. {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 52s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 58s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 41s{color} | {color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/559/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 9 new + 28 unchanged - 0 fixed = 37 total (was 28) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 49s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} |
[jira] [Commented] (YARN-10589) Improve logic of multi-node allocation
[ https://issues.apache.org/jira/browse/YARN-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17273499#comment-17273499 ] Tanu Ajmera commented on YARN-10589: [~zhuqi] In the code block I attached, it will create a set of all those nodes of one partition and send for allocation. If the partition doesn't match in preCheckNodeCandidateSet, it should stop checking for all other nodes in that set. I'm just creating a break point so that it stops after one node. I have attached patch, please review and give comments > Improve logic of multi-node allocation > -- > > Key: YARN-10589 > URL: https://issues.apache.org/jira/browse/YARN-10589 > Project: Hadoop YARN > Issue Type: Task >Affects Versions: 3.3.0 >Reporter: Tanu Ajmera >Assignee: Tanu Ajmera >Priority: Major > Attachments: YARN-10589-001.patch > > > {code:java} > for (String partititon : partitions) { > if (current++ > start) { > break; > } > CandidateNodeSet candidates = > cs.getCandidateNodeSet(partititon); > if (candidates == null) { > continue; > } > cs.allocateContainersToNode(candidates, false); > }{code} > In above logic, if we have thousands of node in one partition, we will still > repeatedly access all nodes of the partition thousands of times. There is no > break point where if the partition is not same for the first node, it should > stop checking other nodes in that partition. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10589) Improve logic of multi-node allocation
[ https://issues.apache.org/jira/browse/YARN-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17270635#comment-17270635 ] zhuqi commented on YARN-10589: -- [~tanu.ajmera] The logic has been improved from multi (node->partition) to one (partition). > Improve logic of multi-node allocation > -- > > Key: YARN-10589 > URL: https://issues.apache.org/jira/browse/YARN-10589 > Project: Hadoop YARN > Issue Type: Task >Affects Versions: 3.3.0 >Reporter: Tanu Ajmera >Assignee: Tanu Ajmera >Priority: Major > Fix For: 3.4.0 > > > {code:java} > for (String partititon : partitions) { > if (current++ > start) { > break; > } > CandidateNodeSet candidates = > cs.getCandidateNodeSet(partititon); > if (candidates == null) { > continue; > } > cs.allocateContainersToNode(candidates, false); > }{code} > In above logic, if we have thousands of node in one partition, we will still > repeatedly access all nodes of the partition thousands of times. There is no > break point where if the partition is not same for the first node, it should > stop checking other nodes in that partition. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org