[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331316#comment-16331316 ] Arun Suresh commented on YARN-6599: --- Thanks [~leftnoteasy] for the patch, and reviews [~sunilg], [~kkaranasos] and [~cheersyang]. Committing this shortly > Support rich placement constraints in scheduler > --- > > Key: YARN-6599 > URL: https://issues.apache.org/jira/browse/YARN-6599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > Attachments: YARN-6599-YARN-6592.003.patch, > YARN-6599-YARN-6592.004.patch, YARN-6599-YARN-6592.005.patch, > YARN-6599-YARN-6592.006.patch, YARN-6599-YARN-6592.007.patch, > YARN-6599-YARN-6592.008.patch, YARN-6599-YARN-6592.009.patch, > YARN-6599-YARN-6592.010.patch, YARN-6599-YARN-6592.011.patch, > YARN-6599-YARN-6592.012.patch, YARN-6599-YARN-6592.013.patch, > YARN-6599-YARN-6592.014.patch, YARN-6599-YARN-6592.015.patch, > YARN-6599-YARN-6592.wip.002.patch, YARN-6599.poc.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331214#comment-16331214 ] Wangda Tan commented on YARN-6599: -- The failed test cases is not related, which is tracked by: https://issues.apache.org/jira/browse/YARN-6272 > Support rich placement constraints in scheduler > --- > > Key: YARN-6599 > URL: https://issues.apache.org/jira/browse/YARN-6599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > Attachments: YARN-6599-YARN-6592.003.patch, > YARN-6599-YARN-6592.004.patch, YARN-6599-YARN-6592.005.patch, > YARN-6599-YARN-6592.006.patch, YARN-6599-YARN-6592.007.patch, > YARN-6599-YARN-6592.008.patch, YARN-6599-YARN-6592.009.patch, > YARN-6599-YARN-6592.010.patch, YARN-6599-YARN-6592.011.patch, > YARN-6599-YARN-6592.012.patch, YARN-6599-YARN-6592.013.patch, > YARN-6599-YARN-6592.014.patch, YARN-6599-YARN-6592.015.patch, > YARN-6599-YARN-6592.wip.002.patch, YARN-6599.poc.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331162#comment-16331162 ] genericqa commented on YARN-6599: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 23 new or modified test files. {color} | || || || || {color:brown} YARN-6592 Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 20s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 15s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 45s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 0s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m 12s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 17m 0s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 1s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in YARN-6592 has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 44s{color} | {color:green} YARN-6592 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 34s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 59s{color} | {color:orange} root: The patch generated 126 new + 1620 unchanged - 19 fixed = 1746 total (was 1639) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m 22s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 8m 25s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 49s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}104m 41s{color} | {color:red} hadoop-yarn in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 38s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 13s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 14s{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 66m 42s{color} |
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329892#comment-16329892 ] genericqa commented on YARN-6599: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 23s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 23 new or modified test files. {color} | || || || || {color:brown} YARN-6592 Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 23s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 30s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 1s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m 31s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 17m 41s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 3s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in YARN-6592 has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 2s{color} | {color:green} YARN-6592 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 6s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 19s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 21s{color} | {color:orange} root: The patch generated 129 new + 1618 unchanged - 19 fixed = 1747 total (was 1637) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m 49s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 9m 31s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 49s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}102m 4s{color} | {color:red} hadoop-yarn in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 47s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 19s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 19s{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 65m 0s{color} |
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329473#comment-16329473 ] Wangda Tan commented on YARN-6599: -- Attached ver.13 patch which rebased to latest branch after YARN-6619 > Support rich placement constraints in scheduler > --- > > Key: YARN-6599 > URL: https://issues.apache.org/jira/browse/YARN-6599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > Attachments: YARN-6599-YARN-6592.003.patch, > YARN-6599-YARN-6592.004.patch, YARN-6599-YARN-6592.005.patch, > YARN-6599-YARN-6592.006.patch, YARN-6599-YARN-6592.007.patch, > YARN-6599-YARN-6592.008.patch, YARN-6599-YARN-6592.009.patch, > YARN-6599-YARN-6592.010.patch, YARN-6599-YARN-6592.011.patch, > YARN-6599-YARN-6592.012.patch, YARN-6599-YARN-6592.013.patch, > YARN-6599-YARN-6592.wip.002.patch, YARN-6599.poc.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329112#comment-16329112 ] Sunil G commented on YARN-6599: --- Thanks [~leftnoteasy] for clarifying my comments. Latest patch seems fine to me. +1. > Support rich placement constraints in scheduler > --- > > Key: YARN-6599 > URL: https://issues.apache.org/jira/browse/YARN-6599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > Attachments: YARN-6599-YARN-6592.003.patch, > YARN-6599-YARN-6592.004.patch, YARN-6599-YARN-6592.005.patch, > YARN-6599-YARN-6592.006.patch, YARN-6599-YARN-6592.007.patch, > YARN-6599-YARN-6592.008.patch, YARN-6599-YARN-6592.009.patch, > YARN-6599-YARN-6592.010.patch, YARN-6599-YARN-6592.011.patch, > YARN-6599-YARN-6592.012.patch, YARN-6599-YARN-6592.wip.002.patch, > YARN-6599.poc.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328659#comment-16328659 ] genericqa commented on YARN-6599: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 23 new or modified test files. {color} | || || || || {color:brown} YARN-6592 Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 28s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 23s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 28s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m 59s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 19m 41s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 8s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in YARN-6592 has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 34s{color} | {color:green} YARN-6592 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 44s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 26s{color} | {color:orange} root: The patch generated 129 new + 1621 unchanged - 19 fixed = 1750 total (was 1640) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 8s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 18s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 93m 30s{color} | {color:red} hadoop-yarn in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 44s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 12s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 7s{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 61m 15s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. {color} | |
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328556#comment-16328556 ] genericqa commented on YARN-6599: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 20m 23s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 23 new or modified test files. {color} | || || || || {color:brown} YARN-6592 Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}114m 14s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 50s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 7s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 29s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 7m 3s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 19m 46s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 8s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in YARN-6592 has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 16s{color} | {color:green} YARN-6592 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 55s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m 5s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 31s{color} | {color:orange} root: The patch generated 129 new + 1619 unchanged - 19 fixed = 1748 total (was 1638) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 7m 6s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 9s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 20s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}104m 34s{color} | {color:red} hadoop-yarn in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 43s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 11s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 10s{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 64m 4s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. {color} | |
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327987#comment-16327987 ] Wangda Tan commented on YARN-6599: -- ver.11 patch has some issues, attached ver.12 patch. > Support rich placement constraints in scheduler > --- > > Key: YARN-6599 > URL: https://issues.apache.org/jira/browse/YARN-6599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > Attachments: YARN-6599-YARN-6592.003.patch, > YARN-6599-YARN-6592.004.patch, YARN-6599-YARN-6592.005.patch, > YARN-6599-YARN-6592.006.patch, YARN-6599-YARN-6592.007.patch, > YARN-6599-YARN-6592.008.patch, YARN-6599-YARN-6592.009.patch, > YARN-6599-YARN-6592.010.patch, YARN-6599-YARN-6592.011.patch, > YARN-6599-YARN-6592.012.patch, YARN-6599-YARN-6592.wip.002.patch, > YARN-6599.poc.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327977#comment-16327977 ] genericqa commented on YARN-6599: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 23 new or modified test files. {color} | || || || || {color:brown} YARN-6592 Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 4m 37s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 10s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 43s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 2s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m 29s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 17m 49s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 4s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in YARN-6592 has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 59s{color} | {color:green} YARN-6592 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 2m 17s{color} | {color:red} hadoop-yarn in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 21s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 21s{color} | {color:red} hadoop-yarn-client in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 24s{color} | {color:red} hadoop-mapreduce-client-app in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 17s{color} | {color:red} hadoop-sls in the patch failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m 55s{color} | {color:red} root in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 3m 55s{color} | {color:red} root in the patch failed. {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 6s{color} | {color:orange} root: The patch generated 131 new + 1620 unchanged - 19 fixed = 1751 total (was 1639) {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 1m 43s{color} | {color:red} hadoop-yarn in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 25s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 24s{color} | {color:red} hadoop-yarn-client in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 27s{color} | {color:red} hadoop-mapreduce-client-app in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 18s{color} | {color:red} hadoop-sls in the patch failed. {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 1s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 2m 39s{color} | {color:red} patch has errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue}
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327670#comment-16327670 ] Wangda Tan commented on YARN-6599: -- Thanks comments from [~sunilg] / [~asuresh], For comments from [~sunilg] , # Removed all APP_ID_TAG_PREFIX, it should be replaced by allocationTagIntraApp already. # The reason is we want to make sure: a. Since this is a new code, we will not accept SchedulingRequest as it may cause damages to RM. b. PlacementProcessor is doing the same thing with a different approach, user needs to enable one of them to work. # Currently, intra-app anti-affinity is not mutual, which means if SchedulRequest_A (with allocation tag x) want to anti-affinity to allocation tags=y, the other SchedulingRequest_B (with allocation tag y) need explicitly specify anti-affinity to allocation tag=x. # Since this involves PlacementProcessor, let's move the change to a different Jira. For comments from [~asuresh] , As we discussed offline, we will add SchedulingRequest as part of the PlacementConstraintUtil method and both of processor/scheduler implementation will use the same logic. The logic looks like: {code:java} PlacementConstraint pc = schedulingRequest.getPlacementConstraint(); If (pc == null) { pc = PlacementConstraintMgr.getPlacementConstraint(schedulingRequest.getAllocationTags()); } // Do placement constraint match ...{code} Let's move it to a separate Jira. Attached ver.11 patch with changes mentioned. > Support rich placement constraints in scheduler > --- > > Key: YARN-6599 > URL: https://issues.apache.org/jira/browse/YARN-6599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > Attachments: YARN-6599-YARN-6592.003.patch, > YARN-6599-YARN-6592.004.patch, YARN-6599-YARN-6592.005.patch, > YARN-6599-YARN-6592.006.patch, YARN-6599-YARN-6592.007.patch, > YARN-6599-YARN-6592.008.patch, YARN-6599-YARN-6592.009.patch, > YARN-6599-YARN-6592.010.patch, YARN-6599-YARN-6592.011.patch, > YARN-6599-YARN-6592.wip.002.patch, YARN-6599.poc.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327517#comment-16327517 ] Arun Suresh commented on YARN-6599: --- Thanks for the update [~leftnoteasy] I just have couple of minor comments: As we discussed offline, to maintain consistency with the processor approach, atleast at an API level, can we do the following: Currently, the processor expects all placement constraints for the app to sent in the register call. Any placement constraint specified in the SchedulingRequest should be an override of the constraints specified in the mapping provided in the registerAM call. The mapping that is sent in the register call is of the form {{Map}}, where the key is a set of allocation tags. Then in the specific schedulingRequest, we refer to the PlacementConstraint using the allocation tags (The PlacementConstraintsManager holds the mapping). This means you need to set the allocation tags in the scheduling request. If the SchedulingRequest contains a PlacementRequest, then it will first replace the mapping in the PlacementConstraintsManager and then apply the constraints to the SchedulingRequest. Am +1 after that > Support rich placement constraints in scheduler > --- > > Key: YARN-6599 > URL: https://issues.apache.org/jira/browse/YARN-6599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > Attachments: YARN-6599-YARN-6592.003.patch, > YARN-6599-YARN-6592.004.patch, YARN-6599-YARN-6592.005.patch, > YARN-6599-YARN-6592.006.patch, YARN-6599-YARN-6592.007.patch, > YARN-6599-YARN-6592.008.patch, YARN-6599-YARN-6592.009.patch, > YARN-6599-YARN-6592.010.patch, YARN-6599-YARN-6592.wip.002.patch, > YARN-6599.poc.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327451#comment-16327451 ] Sunil G commented on YARN-6599: --- Thanks [~leftnoteasy]. Few comments. # {{public static final String APP_ID_TAG_PREFIX = "yarn_app_id/";}} Could this be categorized under {{yarn_application_label}} as sub category. I think if all app related could be under same tag, its better. # {{CapacitySchedulerConfiguration.SCHEDULING_REQUEST_ALLOWED}} I think this flag is not needed. Could we simply check whether *schedulingRequests* is present along with *resourceRequests* and handle error scenarios. Any advantage of having this flag other than validation? # scope of anti-affinity is only to intra-app when PlacementConstraints is Node. *intra-app* is a bit confusing term to me. For eg, we are at high level now doing an anti-affinity for app1's region server not run with app1's hmaster. Does this mean that app2's hmaster can run with app1's region server. cud u pls help to clarify this. # Levels of iterations to check whether a constraint is valid or not is a bit sequential. So could we have a state level verfication whether an incoming request with a constraint is valid or not. What i meant is tat, we can move these validations to a separate class so that its very much easy to define error states etc. > Support rich placement constraints in scheduler > --- > > Key: YARN-6599 > URL: https://issues.apache.org/jira/browse/YARN-6599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > Attachments: YARN-6599-YARN-6592.003.patch, > YARN-6599-YARN-6592.004.patch, YARN-6599-YARN-6592.005.patch, > YARN-6599-YARN-6592.006.patch, YARN-6599-YARN-6592.007.patch, > YARN-6599-YARN-6592.008.patch, YARN-6599-YARN-6592.009.patch, > YARN-6599-YARN-6592.010.patch, YARN-6599-YARN-6592.wip.002.patch, > YARN-6599.poc.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326591#comment-16326591 ] Arun Suresh commented on YARN-6599: --- Sure - will check it out today eve. > Support rich placement constraints in scheduler > --- > > Key: YARN-6599 > URL: https://issues.apache.org/jira/browse/YARN-6599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > Attachments: YARN-6599-YARN-6592.003.patch, > YARN-6599-YARN-6592.004.patch, YARN-6599-YARN-6592.005.patch, > YARN-6599-YARN-6592.006.patch, YARN-6599-YARN-6592.007.patch, > YARN-6599-YARN-6592.008.patch, YARN-6599-YARN-6592.009.patch, > YARN-6599-YARN-6592.010.patch, YARN-6599-YARN-6592.wip.002.patch, > YARN-6599.poc.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325008#comment-16325008 ] genericqa commented on YARN-6599: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 22 new or modified test files. {color} | || || || || {color:brown} YARN-6592 Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 18s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 42s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 8s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 59s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m 30s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 17m 25s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 5s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in YARN-6592 has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 26s{color} | {color:green} YARN-6592 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 2s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 8s{color} | {color:orange} root: The patch generated 122 new + 1606 unchanged - 19 fixed = 1728 total (was 1625) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 8m 35s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 22s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m 35s{color} | {color:red} hadoop-yarn-project_hadoop-yarn generated 1 new + 4204 unchanged - 0 fixed = 4205 total (was 4204) {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 21s{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}102m 43s{color} | {color:red} hadoop-yarn in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 40s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 9s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320979#comment-16320979 ] Arun Suresh commented on YARN-6599: --- Another comment: In YARN-7669, we added support for a {{RejectedSchedulingRequest}}. Maybe you use that to notify the AM when you get a {{SchedulerInvalidResoureRequestException}} > Support rich placement constraints in scheduler > --- > > Key: YARN-6599 > URL: https://issues.apache.org/jira/browse/YARN-6599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6599-YARN-6592.003.patch, > YARN-6599-YARN-6592.004.patch, YARN-6599-YARN-6592.005.patch, > YARN-6599-YARN-6592.006.patch, YARN-6599-YARN-6592.007.patch, > YARN-6599-YARN-6592.008.patch, YARN-6599-YARN-6592.wip.002.patch, > YARN-6599.poc.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320977#comment-16320977 ] Wangda Tan commented on YARN-6599: -- [~asuresh], thanks, sure will make changes today. > Support rich placement constraints in scheduler > --- > > Key: YARN-6599 > URL: https://issues.apache.org/jira/browse/YARN-6599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6599-YARN-6592.003.patch, > YARN-6599-YARN-6592.004.patch, YARN-6599-YARN-6592.005.patch, > YARN-6599-YARN-6592.006.patch, YARN-6599-YARN-6592.007.patch, > YARN-6599-YARN-6592.008.patch, YARN-6599-YARN-6592.wip.002.patch, > YARN-6599.poc.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320959#comment-16320959 ] Arun Suresh commented on YARN-6599: --- {noformat} Since the whole scheduling request is a new feature, we will include two configs: a. Enable placement processor (YARN-7612 already includes configs) b. Enable scheduling request handled by app placement allocator. Both a/b are disabled by default and cannot be enabled at the same time. {noformat} Also, as stated above, will you be adding the new config param - and the check in this patch ? > Support rich placement constraints in scheduler > --- > > Key: YARN-6599 > URL: https://issues.apache.org/jira/browse/YARN-6599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6599-YARN-6592.003.patch, > YARN-6599-YARN-6592.004.patch, YARN-6599-YARN-6592.005.patch, > YARN-6599-YARN-6592.006.patch, YARN-6599-YARN-6592.007.patch, > YARN-6599-YARN-6592.008.patch, YARN-6599-YARN-6592.wip.002.patch, > YARN-6599.poc.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320928#comment-16320928 ] Arun Suresh commented on YARN-6599: --- [~leftnoteasy], sure - given our discussion on adding the {{APPLICATION_LABEL/}} targetKey, can you remove the applicationId tag prefix and related code from the latest patch ? > Support rich placement constraints in scheduler > --- > > Key: YARN-6599 > URL: https://issues.apache.org/jira/browse/YARN-6599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6599-YARN-6592.003.patch, > YARN-6599-YARN-6592.004.patch, YARN-6599-YARN-6592.005.patch, > YARN-6599-YARN-6592.006.patch, YARN-6599-YARN-6592.007.patch, > YARN-6599-YARN-6592.008.patch, YARN-6599-YARN-6592.wip.002.patch, > YARN-6599.poc.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320821#comment-16320821 ] Wangda Tan commented on YARN-6599: -- [~asuresh]/[~kkaranasos], could you help to check the latest patch? I want to get this done by next Monday. > Support rich placement constraints in scheduler > --- > > Key: YARN-6599 > URL: https://issues.apache.org/jira/browse/YARN-6599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6599-YARN-6592.003.patch, > YARN-6599-YARN-6592.004.patch, YARN-6599-YARN-6592.005.patch, > YARN-6599-YARN-6592.006.patch, YARN-6599-YARN-6592.007.patch, > YARN-6599-YARN-6592.008.patch, YARN-6599-YARN-6592.wip.002.patch, > YARN-6599.poc.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319662#comment-16319662 ] Wangda Tan commented on YARN-6599: -- Discussed with [~asuresh] and [~kkaranasos] today, a summary of what we discussed. 1) To support check cardinality for intra-app/inter-app. We will reuse the TargetExpression.targetKey to specify application label. Application label is a new concept which could be: - {{APPLICATION_LABEL/SELF}}: to check cardinality within the app. - {{APPLICATION_LABEL/}}: to check cardinality under app with given app id. - {{APPLICATION_LABEL/}}: to check cardinality under app with given label. What we will immediately add is {{APPLICATION_LABEL/SELF}}, and move all other staffs to future when we want to do cross-application affinity/anti-affinity. 2) SELF target-type will be removed in YARN-7709. Application needs to copy source tags to target tags when wants to affinity/anti-affinity to itself. 3) Will keep node partition related logic in the patch for reviews. In the future after YARN-3409, we need to add all node-related properties to node attributes, such as {{yarn.io/host.name/}}, {{yarn.io.failure.domain/}}, {{yarn.io.node.partition/}}. Reason of doing this is, we can support all node related attribute in the same logic, no need to do if checks like: {code} if (targetKey == node_partition) { // check node.getPartition } {code} 4) Since we have two approaches: - Placement processor. (Added by YARN-7612) - AppPlacementAllocator for SchedulingRequest (added by this patch). Both approaches have merits, the first one can consider multiple-apps/multiple-scheduling-requests at the same time. And the latter one keeps same scheduler behavior and support features like node partition, allocation order under different priority/allocation-id/app-priority/queue-order etc. Since the whole scheduling request is a new feature, we will include two configs: a. Enable placement processor (YARN-7612 already includes configs) b. Enable scheduling request handled by app placement allocator. Both a/b are disabled by default and cannot be enabled at the same time. An alternative solution is to have one config which supports enum values like processor/scheduler. (we need a better name for this). Thoughts here? 5) For Scheduler allocate interface, we will stick to approach introduced by this patch. Extracting a common class for SchedulerRequest/ResourceRequest cannot simplify the problem and might introduce compatibility issues. [~asuresh]/[~kkaranasos], please add if any missed/inaccurate. > Support rich placement constraints in scheduler > --- > > Key: YARN-6599 > URL: https://issues.apache.org/jira/browse/YARN-6599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6599-YARN-6592.003.patch, > YARN-6599-YARN-6592.004.patch, YARN-6599-YARN-6592.005.patch, > YARN-6599-YARN-6592.006.patch, YARN-6599-YARN-6592.007.patch, > YARN-6599-YARN-6592.008.patch, YARN-6599-YARN-6592.wip.002.patch, > YARN-6599.poc.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16317379#comment-16317379 ] Arun Suresh commented on YARN-6599: --- [~leftnoteasy], did a slightly deeper dive. This is my understanding of the general flow - I'd break this down the 2 phases. # When the app makes the allocate call: This is where you do the updatePendingAsk etc for each SchedulingRequest. This is also where you instantiate the AppPlacementAllocator and map it to the request. Looks like the really interesting method is {{validateAndSetSchedulingRequest}}, which is where you validate the placement constraints. This method sets the valid targetAllocationTags etc. # When the node heartbeats: At this point, in the leafQueue, we pick the SchedulingRequest with highest priority and finally, we call the {{canAllocate}} method which checks if the Node can be assigned to the scheduling request based on the placementConstratint. right ? Given the above, we should agree that: This approach - while it ensures that allocation of a SchedulingRequest to a node will guarantee that Constraints are NOT violated, It does nothing in the way of trying to FIND the nodes that will meet the constraints.. agreed ? My opinion - and something we should call out / document is that: * If an app is more concerned about priority ordering of its requests, then we can recommend using this approach. * If the app the more concerned about an optimal placement, then the processor route will give better results - since it activly tries to find nodes that satisfy constraints by considering multiple schedulingRequests and multiple nodes. Thuoghts ? Also some other comments: * In the Scheduler, you are adding a new List parameter to the allocate() method. Do you think, a better approach might be to create a common superclass / interface which both the SchedulingRequest and ResourceRequest extends and change the existing parameter to {{List}} ? * If we do the above, then you wont have to duplicate methods like : application.updateResourceRequests and normalizeSchedulingRequests > Support rich placement constraints in scheduler > --- > > Key: YARN-6599 > URL: https://issues.apache.org/jira/browse/YARN-6599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6599-YARN-6592.003.patch, > YARN-6599-YARN-6592.004.patch, YARN-6599-YARN-6592.005.patch, > YARN-6599-YARN-6592.006.patch, YARN-6599-YARN-6592.007.patch, > YARN-6599-YARN-6592.008.patch, YARN-6599-YARN-6592.wip.002.patch, > YARN-6599.poc.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314934#comment-16314934 ] Konstantinos Karanasos commented on YARN-6599: -- bq. different implementations can choose to support subset of the SchedulingRequest, but we should not change semantics of the API unless discussed Fair point. Let's discuss to see what is the right balance between usability and complicating the code for the first version. bq. Filed YARN-7709 for this. bq. but let's discuss before taking action on this. I need to think a bit more about it – it might not be just syntactic sugar given we have decided to not include the source tags in the constraint itself. > Support rich placement constraints in scheduler > --- > > Key: YARN-6599 > URL: https://issues.apache.org/jira/browse/YARN-6599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6599-YARN-6592.003.patch, > YARN-6599-YARN-6592.004.patch, YARN-6599-YARN-6592.005.patch, > YARN-6599-YARN-6592.006.patch, YARN-6599-YARN-6592.007.patch, > YARN-6599-YARN-6592.008.patch, YARN-6599-YARN-6592.wip.002.patch, > YARN-6599.poc.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314907#comment-16314907 ] Wangda Tan commented on YARN-6599: -- [~kkaranasos], thanks for comments. bq. I think we should not introduce node partitions in this JIRA. Not only for the size of the patch, but because semantically it belongs to a different JIRA. I can move it to a separate JIRA, but we need to support node partition in first released version. bq. What we did in YARN-7613 is to restrict the scope of a constraint to the application that is defining it for the first version, so that we do not have to deal with prefixes This is not correct to me, different implementations can choose to support *subset* of the SchedulingRequest, but we should not change semantics of the API unless discussed. So a client should expect same result (not have to be exactly same placement, but the constraint should be strictly respected) of a SinglePlacementConstraint handled by PlacementProcessor or logic from this patch. We should revisit changes done by YARN-7613 and follow what we have decided in design doc. bq. I can be convinced to remove the SELF target Filed YARN-7709 for this. > Support rich placement constraints in scheduler > --- > > Key: YARN-6599 > URL: https://issues.apache.org/jira/browse/YARN-6599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6599-YARN-6592.003.patch, > YARN-6599-YARN-6592.004.patch, YARN-6599-YARN-6592.005.patch, > YARN-6599-YARN-6592.006.patch, YARN-6599-YARN-6592.007.patch, > YARN-6599-YARN-6592.008.patch, YARN-6599-YARN-6592.wip.002.patch, > YARN-6599.poc.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314893#comment-16314893 ] Konstantinos Karanasos commented on YARN-6599: -- [~wangda] and [~asuresh]: * I think we should not introduce node partitions in this JIRA. Not only for the size of the patch, but because semantically it belongs to a different JIRA. In general, this is a complicated feature we are adding. Let's get a first version with node and rack, and once we are convinced this is working, we can add the node partitions, after we agree on how we do this properly and in a clean way. * What we did in YARN-7613 is to restrict the scope of a constraint to the application that is defining it for the first version, so that we do not have to deal with prefixes. Again, we will eventually need something like this, but I don't think it should be introduced in this JIRA. * I can be convinced to remove the SELF target, but let's discuss before taking action on this. I need to think a bit more about it -- it might not be just syntactic sugar given have decided to not include the source tags in the constraint itself. I think we should chat offline to converge faster on this couple of things. > Support rich placement constraints in scheduler > --- > > Key: YARN-6599 > URL: https://issues.apache.org/jira/browse/YARN-6599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6599-YARN-6592.003.patch, > YARN-6599-YARN-6592.004.patch, YARN-6599-YARN-6592.005.patch, > YARN-6599-YARN-6592.006.patch, YARN-6599-YARN-6592.007.patch, > YARN-6599-YARN-6592.008.patch, YARN-6599-YARN-6592.wip.002.patch, > YARN-6599.poc.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314876#comment-16314876 ] Wangda Tan commented on YARN-6599: -- Thanks [~asuresh], bq. Wangda Tan, apologize, but I think we really should split this part. Let me explain a bit, it may not be as hard as you thought about. Following changes are related to partition: 1) Added nodePartition to PlacementConstraints 2) Renamed acceptNodePartition to precheckNode so we can blacklist nodes in AppPlaementAllocator. 3) Check node partition inside new AppPlaementAllocator implementation. 4) Added partition check in PlacementConstraintsUtil. 5) Unit test related changes. #1/#3 is very straightforward. #2 is needed anyway since we want to plug logics to check SchedulingRequest while allocating/validating container allocation, regardless of splitting logic. To me, #3 is also not hard. Please let me know your thoughts on this. It may take me several hours to split the logic, I'm not sure if it worths since we're pushing the branch merge soon. bq. It just means source tag == target expression tag. This looks not correct, first of all, it is not a new type, it's more like a syntactic sugar which should be built inside PlacementConstraints, and I'm doubt if it is really needed. We can discuss it in a separate JIRA. bq. As I mentioned in the previous comment. There is is this issue of application priority To me, this is not different from affinity/anti-affinity to different tags. For example a Tensorflow "worker" affinity to "parameter server" which is in different priority. A wrong priority setting could still cause deadlock issue. And in order to specify intra-app affinity/anti-affinity, we have to include application id inside allocation tags since by default allocation tag query should include all apps. > Support rich placement constraints in scheduler > --- > > Key: YARN-6599 > URL: https://issues.apache.org/jira/browse/YARN-6599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6599-YARN-6592.003.patch, > YARN-6599-YARN-6592.004.patch, YARN-6599-YARN-6592.005.patch, > YARN-6599-YARN-6592.006.patch, YARN-6599-YARN-6592.007.patch, > YARN-6599-YARN-6592.008.patch, YARN-6599-YARN-6592.wip.002.patch, > YARN-6599.poc.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314436#comment-16314436 ] Arun Suresh commented on YARN-6599: --- bq. I can rename canSatisfySingleConstraint to canSatisfyConstraints, but it seems that we still need a separate method. In this patch, we need to pass in PlacementConstraint directly. Are you fine with this? Makes Sense {noformat} Can we split the node-partition target expression handling aspect into a separate JIRA maybe ? I would prefer not, there're only a few logics related to partition handling. Splitting this logic cannot help reduce the size of the patch. {noformat} [~leftnoteasy], apologize, but I think we really should split this part. I see a lot of if then also code related to node partitions peppered everywhere - which makes it a bit hard to follow the code. Lets start with a target expression that can handle node/rack scope and then build on that. [~kkaranasos], thoughts ? bq. Also, I'm not sure what is the SELF as target type means It just means source tag == target expression tag. bq. I'm open to any feedbacks on this. I think this is documented in our design doc attached to YARN-6592. Check chapter "Applying constraints during scheduling". I understand - but my issue with with regard to implementation and from a general usability perspective. As I mentioned in the previous comment. There is is this issue of application priority - If app1 has a lower priority than app2 but is scheduled first and app2 has issues scheduling requests that depend on app1, allocation might stall. I think more thought should be put into this. > Support rich placement constraints in scheduler > --- > > Key: YARN-6599 > URL: https://issues.apache.org/jira/browse/YARN-6599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6599-YARN-6592.003.patch, > YARN-6599-YARN-6592.004.patch, YARN-6599-YARN-6592.005.patch, > YARN-6599-YARN-6592.006.patch, YARN-6599-YARN-6592.007.patch, > YARN-6599-YARN-6592.008.patch, YARN-6599-YARN-6592.wip.002.patch, > YARN-6599.poc.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314218#comment-16314218 ] Wangda Tan commented on YARN-6599: -- Thanks [~asuresh] for reviewing this. bq. I think you should keep the name of the main entry point method as canSatisfyConstraints. I can rename canSatisfySingleConstraint to canSatisfyConstraints, but it seems that we still need a separate method. In this patch, we need to pass in PlacementConstraint directly. Are you fine with this? bq. Shouldnt you have to do the checkMinCardinality in all cases ? No, if minCardinality = 0, we don't need to check it at all. bq. Can we split the node-partition target expression handling aspect into a separate JIRA maybe ? I would prefer not, there're only a few logics related to partition handling. Splitting this logic cannot help reduce the size of the patch. bq. To check if the given constraint is a SingleConstraint, you should use the transformer first before doing a type cast. This part I do not quite understand, could you explain a bit? And another question is I'm using following logic to copy SchedulingRequest, is it correct? {code} 396 this.schedulingRequest = new SchedulingRequestPBImpl( 397 ((SchedulingRequestPBImpl) newSchedulingRequest).getProto()); {code} bq. Can we target this JIRA for just intra-app anti-affinity. I'm open about this, could you respond to my comment below? bq. I am also not really sold on the application-id/allocation-tag format yet. I'm open to any feedbacks on this. I think this is documented in our design doc attached to YARN-6592. Check chapter "Applying constraints during scheduling". We have to include anti-affinity to a specific app in this patch to support intra-app anti-affinity. This is because by default allocation-tag specified inside PlacementConstraint should take account of tags from all apps. Are you fine with anti-affinity to its own application by using the syntax I proposed in the patch? (The only different between my implementation and design doc is my impl includes a prefix). Also, I'm not sure what is the SELF as target type means, does it mean the same app/priority+allocationId/SchedulingRequest? It is not part of design doc, I'm not sure what's the reason to include it. > Support rich placement constraints in scheduler > --- > > Key: YARN-6599 > URL: https://issues.apache.org/jira/browse/YARN-6599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6599-YARN-6592.003.patch, > YARN-6599-YARN-6592.004.patch, YARN-6599-YARN-6592.005.patch, > YARN-6599-YARN-6592.006.patch, YARN-6599-YARN-6592.007.patch, > YARN-6599-YARN-6592.008.patch, YARN-6599-YARN-6592.wip.002.patch, > YARN-6599.poc.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314152#comment-16314152 ] Arun Suresh commented on YARN-6599: --- [~leftnoteasy], thanks for working on this. Couple of comments: * I think you should keep the name of the main entry point method as {{canSatisfyConstraints}}. The single constraint is a special case and you can have a private method for it, once we decide in the above function that the constraint is a single constraint type. * Shouldnt you have to do the checkMinCardinality in all cases ? * Can we split the node-partition target expression handling aspect into a separate JIRA maybe ? * To check if the given constraint is a SingleConstraint, you should use the transformer first before doing a type cast. * Can we target this JIRA for just intra-app anti-affinity. I am also not really sold on the application-id/allocation-tag format yet. It requires that the first app be running and then the second app (since we wont have a application-Id till then) - Also if we couple this with application priorities, I can think of a situation where we can deadlock. > Support rich placement constraints in scheduler > --- > > Key: YARN-6599 > URL: https://issues.apache.org/jira/browse/YARN-6599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6599-YARN-6592.003.patch, > YARN-6599-YARN-6592.004.patch, YARN-6599-YARN-6592.005.patch, > YARN-6599-YARN-6592.006.patch, YARN-6599-YARN-6592.007.patch, > YARN-6599-YARN-6592.008.patch, YARN-6599-YARN-6592.wip.002.patch, > YARN-6599.poc.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313734#comment-16313734 ] Sunil G commented on YARN-6599: --- Thanks [~leftnoteasy] I think latest patch seems fine. If there are no objections, i could commit this over weekend. > Support rich placement constraints in scheduler > --- > > Key: YARN-6599 > URL: https://issues.apache.org/jira/browse/YARN-6599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6599-YARN-6592.003.patch, > YARN-6599-YARN-6592.004.patch, YARN-6599-YARN-6592.005.patch, > YARN-6599-YARN-6592.006.patch, YARN-6599-YARN-6592.007.patch, > YARN-6599-YARN-6592.008.patch, YARN-6599-YARN-6592.wip.002.patch, > YARN-6599.poc.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312459#comment-16312459 ] genericqa commented on YARN-6599: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 21 new or modified test files. {color} | || || || || {color:brown} YARN-6592 Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 26s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 16s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 16s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 25s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 7m 19s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 19m 58s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 7s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in YARN-6592 has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 56s{color} | {color:green} YARN-6592 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m 33s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 3m 15s{color} | {color:orange} root: The patch generated 129 new + 1501 unchanged - 15 fixed = 1630 total (was 1516) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 8m 7s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 9m 56s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 58s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}105m 19s{color} | {color:red} hadoop-yarn in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 44s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 8s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 8s{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 65m 35s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. {color} | |
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312317#comment-16312317 ] genericqa commented on YARN-6599: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 21 new or modified test files. {color} | || || || || {color:brown} YARN-6592 Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 20s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 11s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 18s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 2s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m 17s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 17m 10s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 3s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in YARN-6592 has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 3s{color} | {color:green} YARN-6592 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 54s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m 36s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 9s{color} | {color:orange} root: The patch generated 128 new + 1502 unchanged - 15 fixed = 1630 total (was 1517) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 8m 38s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 56s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}109m 4s{color} | {color:red} hadoop-yarn in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 45s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 17s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 18s{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 21s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | |
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311298#comment-16311298 ] Sunil G commented on YARN-6599: --- Thanks [~leftnoteasy] for working on this. Some comments/doubts from myside: # I think {{PlacementTags}} could be part of a util class. We are trying to fetch app id from tags hence may be it could be renamed are placement util class or similar. # {{AYS#normalizeSchedulingRequests}} could be moved to CS for now as we are supporting this only in CS currently. # {{AppSchedulingInfo#addSchedulingRequests}} could avoid second for loop. I think {{MapschedulingRequestMap}} is not needed. If we iterate on {{List schedulingRequests}}, *SchedulerRequestKey* could created for each iteration and use it internally. # Why are we throwing exception in SingleConstraintAppPlacementAllocator #updatePendingAsk when requests are not NULL. # Also i really didnt understand why we need {{private SchedulingRequest schedulingRequest = null;}} in SingleConstraintAppPlacementAllocator. This should be for all type of allocators, correct?. So do we need an abstract class to hold such common functionalities. # In {{internalUpdatePendingAsk}}, could we check {{if (!schedulingRequest.equals(newSchedulingRequest))}} earlier in method. # Is this correct? we set sizing multiple times here. {code} 151 sizing.setNumAllocations(existingNumAllocations); 152 153 // Compare two objects 154 if (!schedulingRequest.equals(newSchedulingRequest)) { 155 // Rollback #numAllocations 156 sizing.setNumAllocations(newNumAllocations); . 166 } 167 168 // Rollback #numAllocations 169 sizing.setNumAllocations(newNumAllocations); {code} # *validateAndSetSchedulingRequest* for loop has a continue statement. May not be needed. # I think we can try to have a common evaluator to process PlacementConstraint.TargetExpression based on each TargetType. For any target type, we are trying to get *nodePartition* and *targetAllocationTags*. A typed evaluator could process and evaluate it based on TargetType. # Are we supporting a case where app is changing partition for a given resource request? Or will be considered as a new requests and old one ll be cleared. # For TargetType.ALLOCATION_TAG processing in validateAndSetSchedulingRequest, I think we can directly reject if any tag is not starting with {{tag.startsWith(APP_ID_TAG_PREFIX)}} ? # Could we get appId from appInfo itself? {code} 379 ApplicationId appId = PlacementTags.extractApplicationIdFromTag( 380 targetTag); {code} # > Support rich placement constraints in scheduler > --- > > Key: YARN-6599 > URL: https://issues.apache.org/jira/browse/YARN-6599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6599-YARN-6592.003.patch, > YARN-6599-YARN-6592.004.patch, YARN-6599-YARN-6592.005.patch, > YARN-6599-YARN-6592.006.patch, YARN-6599-YARN-6592.wip.002.patch, > YARN-6599.poc.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16309343#comment-16309343 ] genericqa commented on YARN-6599: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 20 new or modified test files. {color} | || || || || {color:brown} YARN-6592 Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 40s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 29s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 28s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 7m 2s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 19m 46s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 8s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in YARN-6592 has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 23s{color} | {color:green} YARN-6592 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m 19s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 26s{color} | {color:orange} root: The patch generated 87 new + 1481 unchanged - 15 fixed = 1568 total (was 1496) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 7m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 9m 53s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 23s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}100m 39s{color} | {color:red} hadoop-yarn in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 44s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 9s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 13s{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 21s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | |
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16309079#comment-16309079 ] genericqa commented on YARN-6599: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 20 new or modified test files. {color} | || || || || {color:brown} YARN-6592 Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 18s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 46s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 29s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 27s{color} | {color:green} YARN-6592 passed {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 29s{color} | {color:red} hadoop-yarn-common in YARN-6592 failed. {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 33s{color} | {color:red} hadoop-yarn-server-resourcemanager in YARN-6592 failed. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 20m 33s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 8s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in YARN-6592 has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 25s{color} | {color:green} YARN-6592 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 38s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 35s{color} | {color:red} hadoop-yarn in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 30s{color} | {color:red} hadoop-yarn-api in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 21s{color} | {color:red} hadoop-yarn-common in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 19s{color} | {color:red} hadoop-yarn-server-common in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 22s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 22s{color} | {color:red} hadoop-yarn-client in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 23s{color} | {color:red} hadoop-mapreduce-client-app in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 16s{color} | {color:red} hadoop-sls in the patch failed. {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m 23s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 27s{color} | {color:orange} root: The patch generated 87 new + 1481 unchanged - 15 fixed = 1568 total (was 1496) {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 30s{color} | {color:red} hadoop-yarn-common in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 28s{color} | {color:red} hadoop-yarn-server-common in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 29s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 29s{color} | {color:red} hadoop-yarn-client in the patch failed. {color} | | {color:red}-1{color} |
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308875#comment-16308875 ] genericqa commented on YARN-6599: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 25s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 20 new or modified test files. {color} | || || || || {color:brown} YARN-6592 Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 5m 54s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 14s{color} | {color:red} root in YARN-6592 failed. {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 17s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 18s{color} | {color:green} YARN-6592 passed {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 22s{color} | {color:red} hadoop-yarn in YARN-6592 failed. {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 22s{color} | {color:red} hadoop-yarn-api in YARN-6592 failed. {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 22s{color} | {color:red} hadoop-yarn-common in YARN-6592 failed. {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 22s{color} | {color:red} hadoop-yarn-server-common in YARN-6592 failed. {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 23s{color} | {color:red} hadoop-yarn-server-resourcemanager in YARN-6592 failed. {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 19s{color} | {color:red} hadoop-yarn-client in YARN-6592 failed. {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 39s{color} | {color:red} hadoop-mapreduce-client-app in YARN-6592 failed. {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 23s{color} | {color:red} hadoop-sls in YARN-6592 failed. {color} | | {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 6m 28s{color} | {color:red} branch has errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 21s{color} | {color:red} hadoop-yarn-api in YARN-6592 failed. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 22s{color} | {color:red} hadoop-yarn-common in YARN-6592 failed. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 22s{color} | {color:red} hadoop-yarn-server-common in YARN-6592 failed. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 21s{color} | {color:red} hadoop-yarn-server-resourcemanager in YARN-6592 failed. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 22s{color} | {color:red} hadoop-yarn-client in YARN-6592 failed. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 26s{color} | {color:red} hadoop-yarn in YARN-6592 failed. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 21s{color} | {color:red} hadoop-yarn-api in YARN-6592 failed. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 22s{color} | {color:red} hadoop-yarn-common in YARN-6592 failed. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 21s{color} | {color:red} hadoop-yarn-server-common in YARN-6592 failed. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 21s{color} | {color:red} hadoop-yarn-server-resourcemanager in YARN-6592 failed. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 22s{color} | {color:red} hadoop-yarn-client in YARN-6592 failed. {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 29s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 24s{color} | {color:red} hadoop-yarn in the patch
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308781#comment-16308781 ] genericqa commented on YARN-6599: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 12s{color} | {color:red} YARN-6599 does not apply to YARN-6592. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | YARN-6599 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12904288/YARN-6599-YARN-6592.005.patch | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/19076/console | | Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Support rich placement constraints in scheduler > --- > > Key: YARN-6599 > URL: https://issues.apache.org/jira/browse/YARN-6599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6599-YARN-6592.003.patch, > YARN-6599-YARN-6592.004.patch, YARN-6599-YARN-6592.005.patch, > YARN-6599-YARN-6592.wip.002.patch, YARN-6599.poc.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16307846#comment-16307846 ] genericqa commented on YARN-6599: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 20 new or modified test files. {color} | || || || || {color:brown} YARN-6592 Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 42s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 32s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 51s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 26s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 34s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 17m 39s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 20s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in YARN-6592 has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 39s{color} | {color:green} YARN-6592 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m 15s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 23s{color} | {color:orange} root: The patch generated 87 new + 1481 unchanged - 15 fixed = 1568 total (was 1496) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 4s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 29s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 25s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 43s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 11s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 14s{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 49s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 35m 22s{color} | {color:red} hadoop-yarn-client in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 1s{color} | {color:red} hadoop-mapreduce-client-app in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 2s{color} | {color:green} hadoop-sls in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 29s{color} | {color:green} The patch
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16307571#comment-16307571 ] genericqa commented on YARN-6599: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 29m 2s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 18 new or modified test files. {color} | || || || || {color:brown} YARN-6592 Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 20s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 56s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 32s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 27s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 45s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 41s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 13s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in YARN-6592 has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 43s{color} | {color:green} YARN-6592 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 54s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m 22s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 24s{color} | {color:orange} root: The patch generated 81 new + 1458 unchanged - 15 fixed = 1539 total (was 1473) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 2s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 21s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 43s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 42s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 8s{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 30s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 23m 9s{color} | {color:red} hadoop-yarn-client in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 5s{color} | {color:red} hadoop-mapreduce-client-app in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 57s{color} | {color:green} hadoop-sls in the patch passed. {color} | | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 34s{color} | {color:red} The patch generated 1 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}235m 18s{color} | {color:black} {color} | \\ \\ ||
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16306657#comment-16306657 ] genericqa commented on YARN-6599: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 11 new or modified test files. {color} | || || || || {color:brown} YARN-6592 Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 18s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 34s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 25s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 23s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 46s{color} | {color:green} YARN-6592 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 46s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 15s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in YARN-6592 has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 42s{color} | {color:green} YARN-6592 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 56s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m 15s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 24s{color} | {color:orange} root: The patch generated 44 new + 1219 unchanged - 7 fixed = 1263 total (was 1226) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 1s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 22s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 42s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 42s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 9s{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m 57s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 17m 31s{color} | {color:red} hadoop-yarn-client in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 58s{color} | {color:red} hadoop-mapreduce-client-app in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 57s{color} | {color:green} hadoop-sls in the patch passed. {color} | | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 33s{color} | {color:red} The patch generated 1 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}197m 37s{color} | {color:black} {color} | \\ \\ ||
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16031469#comment-16031469 ] Carlo Curino commented on YARN-6599: (Pardon my own delay... moving/travelling) I hear you... There is a tension between an ever-growing overly-complicated scheduler code, and replication of functionalities. This, in fact, aligns with my ongoing rant about the fact that we do not have clear semantics of how all the invariants/constraints interact... if we did, it would be much easier to carve out / layer this solution. My main fear is to introduce further (complex) logic within the scheduler guts, which are already very hard to maintain. At the same time, I hear you that respecting limits/priorities/etc.. is non-trivial while sitting outside the scheduler. My preference would be for us to move towards a two-components solutions, where there is a planner/preprocessor that makes complex decisions using a rich language of constraints (and I mean a formal language) and a general purpose solver/compiler/optimizer type design. This would include all constraints/preferences that capture limits/priorities/etc. The second component would be a lean and mean inner-loop scheduler that only maintain states, and quickly instantiate those high level decisions into actual container placements. I think POCs and investigations can definitely help... my only concern is to have you (or anyone else) do lots of work that we eventually decide can't be used (hence my alarmed comments). If you are ok with it, than I am all in for POCs. > Support rich placement constraints in scheduler > --- > > Key: YARN-6599 > URL: https://issues.apache.org/jira/browse/YARN-6599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021536#comment-16021536 ] Wangda Tan commented on YARN-6599: -- [~curino], Apologize for the late responses, somehow I missed your last comment. IIRC, the only thing we decided is to try best to make placement logics can be shared between schedulers. This include and not limited to, finding a node to place a container, score nodes according to constraint, etc. To me placing a container should be common logic of schedulers. However, this doesn't mean nothing will be added to scheduler. As I mentioned to [~kkaranasos]/[~asuresh] offline, the reason we cannot add this logics completely outside of scheduler is, how and when to do placement for requests depends on internal state, such as limit/ordering, etc. of queue/app/user's. These logics are related to different scheduler implementations, and I don't think logics out side of scheduler can access it without duplicating a large part of scheduler logics. Just like preemption logics of CS, it is completely outside of scheduler, however we have to duplicate a lot logics to make its behavior is synced to how scheduler behaves, which is also expensive to maintain, that's my last two years experiences. To me, the solution should be something in-between: not all in scheduler, and not all in outside of scheduler. I don't think we're ready for an answer now, I suggest to do some investigation/POC before make a decision. Thoughts? > Support rich placement constraints in scheduler > --- > > Key: YARN-6599 > URL: https://issues.apache.org/jira/browse/YARN-6599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6599) Support rich placement constraints in scheduler
[ https://issues.apache.org/jira/browse/YARN-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16016765#comment-16016765 ] Carlo Curino commented on YARN-6599: [~leftnoteasy] I am not sure I understand this JIRA. I thought we discussed this offline with [~vinodkv], [~chris.douglas], [~kkaranasos], [~subru] and [~asuresh], and agreed that we were *not* going to add extra stuff inside the scheduler to support constraints, as the scheduler(s) code is becoming untenable to maintain and scale. I think it would be best to keep the constraints management code in the scheduler pre-processor / {{PlacementConstraintManager}} so that the feature is orthogonal to the scheduler(s) and does not add complexity there. Can you explain what has changed since that discussion? I noticed that in the document you also stated there is no agreement on this, while I thought was (after a lengthy discussion) a settled issue. > Support rich placement constraints in scheduler > --- > > Key: YARN-6599 > URL: https://issues.apache.org/jira/browse/YARN-6599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org