[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[ https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15002783#comment-15002783 ] Hudson commented on YARN-1509: -- FAILURE: Integrated in Hadoop-trunk-Commit #8800 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8800/]) YARN-1509. Make AMRMClient support send increase container request and (wangda: rev 7ff280fca9af45b98cee2336e78803da46b0f8a5) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClientOnRMRestart.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/AMRMClientAsync.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/impl/AMRMClientAsyncImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/async/impl/TestAMRMClientAsync.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * hadoop-yarn-project/CHANGES.txt > Make AMRMClient support send increase container request and get > increased/decreased containers > -- > > Key: YARN-1509 > URL: https://issues.apache.org/jira/browse/YARN-1509 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Wangda Tan (No longer used) >Assignee: MENG DING > Fix For: 2.8.0 > > Attachments: YARN-1509.1.patch, YARN-1509.10.patch, > YARN-1509.2.patch, YARN-1509.3.patch, YARN-1509.4.patch, YARN-1509.5.patch, > YARN-1509.6.patch, YARN-1509.7.patch, YARN-1509.8.patch, YARN-1509.9.patch > > > As described in YARN-1197, we need add API in AMRMClient to support > 1) Add increase request > 2) Can get successfully increased/decreased containers from RM -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[ https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15003005#comment-15003005 ] Hudson commented on YARN-1509: -- SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #675 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/675/]) YARN-1509. Make AMRMClient support send increase container request and (wangda: rev 7ff280fca9af45b98cee2336e78803da46b0f8a5) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/impl/AMRMClientAsyncImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/AMRMClientAsync.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/async/impl/TestAMRMClientAsync.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClientOnRMRestart.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java > Make AMRMClient support send increase container request and get > increased/decreased containers > -- > > Key: YARN-1509 > URL: https://issues.apache.org/jira/browse/YARN-1509 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Wangda Tan (No longer used) >Assignee: MENG DING > Fix For: 2.8.0 > > Attachments: YARN-1509.1.patch, YARN-1509.10.patch, > YARN-1509.2.patch, YARN-1509.3.patch, YARN-1509.4.patch, YARN-1509.5.patch, > YARN-1509.6.patch, YARN-1509.7.patch, YARN-1509.8.patch, YARN-1509.9.patch > > > As described in YARN-1197, we need add API in AMRMClient to support > 1) Add increase request > 2) Can get successfully increased/decreased containers from RM -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[ https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15002982#comment-15002982 ] Hudson commented on YARN-1509: -- FAILURE: Integrated in Hadoop-Yarn-trunk #1399 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1399/]) YARN-1509. Make AMRMClient support send increase container request and (wangda: rev 7ff280fca9af45b98cee2336e78803da46b0f8a5) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClientOnRMRestart.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/impl/AMRMClientAsyncImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/AMRMClientAsync.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/async/impl/TestAMRMClientAsync.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java > Make AMRMClient support send increase container request and get > increased/decreased containers > -- > > Key: YARN-1509 > URL: https://issues.apache.org/jira/browse/YARN-1509 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Wangda Tan (No longer used) >Assignee: MENG DING > Fix For: 2.8.0 > > Attachments: YARN-1509.1.patch, YARN-1509.10.patch, > YARN-1509.2.patch, YARN-1509.3.patch, YARN-1509.4.patch, YARN-1509.5.patch, > YARN-1509.6.patch, YARN-1509.7.patch, YARN-1509.8.patch, YARN-1509.9.patch > > > As described in YARN-1197, we need add API in AMRMClient to support > 1) Add increase request > 2) Can get successfully increased/decreased containers from RM -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[ https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15003103#comment-15003103 ] Hudson commented on YARN-1509: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #663 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/663/]) YARN-1509. Make AMRMClient support send increase container request and (wangda: rev 7ff280fca9af45b98cee2336e78803da46b0f8a5) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClientOnRMRestart.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/impl/AMRMClientAsyncImpl.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/AMRMClientAsync.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/async/impl/TestAMRMClientAsync.java > Make AMRMClient support send increase container request and get > increased/decreased containers > -- > > Key: YARN-1509 > URL: https://issues.apache.org/jira/browse/YARN-1509 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Wangda Tan (No longer used) >Assignee: MENG DING > Fix For: 2.8.0 > > Attachments: YARN-1509.1.patch, YARN-1509.10.patch, > YARN-1509.2.patch, YARN-1509.3.patch, YARN-1509.4.patch, YARN-1509.5.patch, > YARN-1509.6.patch, YARN-1509.7.patch, YARN-1509.8.patch, YARN-1509.9.patch > > > As described in YARN-1197, we need add API in AMRMClient to support > 1) Add increase request > 2) Can get successfully increased/decreased containers from RM -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[ https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15003406#comment-15003406 ] Hudson commented on YARN-1509: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2604 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2604/]) YARN-1509. Make AMRMClient support send increase container request and (wangda: rev 7ff280fca9af45b98cee2336e78803da46b0f8a5) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClientOnRMRestart.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/AMRMClientAsync.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/impl/AMRMClientAsyncImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/async/impl/TestAMRMClientAsync.java > Make AMRMClient support send increase container request and get > increased/decreased containers > -- > > Key: YARN-1509 > URL: https://issues.apache.org/jira/browse/YARN-1509 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Wangda Tan (No longer used) >Assignee: MENG DING > Fix For: 2.8.0 > > Attachments: YARN-1509.1.patch, YARN-1509.10.patch, > YARN-1509.2.patch, YARN-1509.3.patch, YARN-1509.4.patch, YARN-1509.5.patch, > YARN-1509.6.patch, YARN-1509.7.patch, YARN-1509.8.patch, YARN-1509.9.patch > > > As described in YARN-1197, we need add API in AMRMClient to support > 1) Add increase request > 2) Can get successfully increased/decreased containers from RM -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[ https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15003437#comment-15003437 ] Hudson commented on YARN-1509: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #602 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/602/]) YARN-1509. Make AMRMClient support send increase container request and (wangda: rev 7ff280fca9af45b98cee2336e78803da46b0f8a5) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClientOnRMRestart.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/async/impl/TestAMRMClientAsync.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/AMRMClientAsync.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/impl/AMRMClientAsyncImpl.java > Make AMRMClient support send increase container request and get > increased/decreased containers > -- > > Key: YARN-1509 > URL: https://issues.apache.org/jira/browse/YARN-1509 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Wangda Tan (No longer used) >Assignee: MENG DING > Fix For: 2.8.0 > > Attachments: YARN-1509.1.patch, YARN-1509.10.patch, > YARN-1509.2.patch, YARN-1509.3.patch, YARN-1509.4.patch, YARN-1509.5.patch, > YARN-1509.6.patch, YARN-1509.7.patch, YARN-1509.8.patch, YARN-1509.9.patch > > > As described in YARN-1197, we need add API in AMRMClient to support > 1) Add increase request > 2) Can get successfully increased/decreased containers from RM -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[ https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15003313#comment-15003313 ] Hudson commented on YARN-1509: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2539 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2539/]) YARN-1509. Make AMRMClient support send increase container request and (wangda: rev 7ff280fca9af45b98cee2336e78803da46b0f8a5) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClientOnRMRestart.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/AMRMClientAsync.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/async/impl/TestAMRMClientAsync.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/impl/AMRMClientAsyncImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * hadoop-yarn-project/CHANGES.txt > Make AMRMClient support send increase container request and get > increased/decreased containers > -- > > Key: YARN-1509 > URL: https://issues.apache.org/jira/browse/YARN-1509 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Wangda Tan (No longer used) >Assignee: MENG DING > Fix For: 2.8.0 > > Attachments: YARN-1509.1.patch, YARN-1509.10.patch, > YARN-1509.2.patch, YARN-1509.3.patch, YARN-1509.4.patch, YARN-1509.5.patch, > YARN-1509.6.patch, YARN-1509.7.patch, YARN-1509.8.patch, YARN-1509.9.patch > > > As described in YARN-1197, we need add API in AMRMClient to support > 1) Add increase request > 2) Can get successfully increased/decreased containers from RM -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[ https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15001134#comment-15001134 ] Jian He commented on YARN-1509: --- lgtm, thanks Meng ! > Make AMRMClient support send increase container request and get > increased/decreased containers > -- > > Key: YARN-1509 > URL: https://issues.apache.org/jira/browse/YARN-1509 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Wangda Tan (No longer used) >Assignee: MENG DING > Attachments: YARN-1509.1.patch, YARN-1509.10.patch, > YARN-1509.2.patch, YARN-1509.3.patch, YARN-1509.4.patch, YARN-1509.5.patch, > YARN-1509.6.patch, YARN-1509.7.patch, YARN-1509.8.patch, YARN-1509.9.patch > > > As described in YARN-1197, we need add API in AMRMClient to support > 1) Add increase request > 2) Can get successfully increased/decreased containers from RM -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[ https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15001141#comment-15001141 ] Wangda Tan commented on YARN-1509: -- +1 to latest patch, will commit it tomorrow if there's no opposite opinions. > Make AMRMClient support send increase container request and get > increased/decreased containers > -- > > Key: YARN-1509 > URL: https://issues.apache.org/jira/browse/YARN-1509 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Wangda Tan (No longer used) >Assignee: MENG DING > Attachments: YARN-1509.1.patch, YARN-1509.10.patch, > YARN-1509.2.patch, YARN-1509.3.patch, YARN-1509.4.patch, YARN-1509.5.patch, > YARN-1509.6.patch, YARN-1509.7.patch, YARN-1509.8.patch, YARN-1509.9.patch > > > As described in YARN-1197, we need add API in AMRMClient to support > 1) Add increase request > 2) Can get successfully increased/decreased containers from RM -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[
https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999410#comment-14999410
]
Hadoop QA commented on YARN-1509:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 7s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 13s
{color} | {color:red} hadoop-yarn-applications-distributedshell in the patch
failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 59s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 31s
{color} | {color:red} Patch generated 3 new checkstyle issues in
hadoop-yarn-project/hadoop-yarn (total was 124, now 120). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 3s
{color} | {color:green} hadoop-yarn-applications-distributedshell in the patch
passed with JDK v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 49m 35s {color}
| {color:red} hadoop-yarn-client in the patch failed with JDK v1.8.0_60.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 2s
{color} | {color:green} hadoop-yarn-applications-distributedshell in the patch
passed with JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 49m 37s {color}
| {color:red} hadoop-yarn-client in the patch failed with JDK v1.7.0_79.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 129m 35s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_60 Failed junit tests | hadoop.yarn.client.TestGetGroups |
| JDK v1.8.0_60 Timed out junit tests |
org.apache.hadoop.yarn.client.api.impl.TestYarnClient |
| | org.apache.hadoop.yarn.client.api.impl.TestAMRMClient |
| | org.apache.hadoop.yarn.client.api.impl.TestNMClient |
| JDK v1.7.0_79 Failed junit tests | hadoop.yarn.client.TestGetGroups |
| JDK v1.7.0_79 Timed out junit tests |
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[
https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987819#comment-14987819
]
Hadoop QA commented on YARN-1509:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 7s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
53s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 13s
{color} | {color:red} hadoop-yarn-applications-distributedshell in the patch
failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 28s
{color} | {color:red} Patch generated 3 new checkstyle issues in
hadoop-yarn-project/hadoop-yarn (total was 124, now 120). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 55s {color}
| {color:red} hadoop-yarn-applications-distributedshell in the patch failed
with JDK v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 49m 33s {color}
| {color:red} hadoop-yarn-client in the patch failed with JDK v1.8.0_60.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 57s {color}
| {color:red} hadoop-yarn-applications-distributedshell in the patch failed
with JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 49m 36s {color}
| {color:red} hadoop-yarn-client in the patch failed with JDK v1.7.0_79.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 149m 45s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_60 Failed junit tests |
hadoop.yarn.applications.distributedshell.TestDistributedShellWithNodeLabels |
| | hadoop.yarn.client.TestGetGroups |
| JDK v1.8.0_60 Timed out junit tests |
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell |
| | org.apache.hadoop.yarn.client.api.impl.TestAMRMClient |
| | org.apache.hadoop.yarn.client.api.impl.TestYarnClient |
| |
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[
https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987878#comment-14987878
]
Hadoop QA commented on YARN-1509:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 7s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 8s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 15s
{color} | {color:red} hadoop-yarn-applications-distributedshell in the patch
failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 28s
{color} | {color:red} Patch generated 3 new checkstyle issues in
hadoop-yarn-project/hadoop-yarn (total was 125, now 121). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 54s {color}
| {color:red} hadoop-yarn-applications-distributedshell in the patch failed
with JDK v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 49m 25s {color}
| {color:red} hadoop-yarn-client in the patch failed with JDK v1.8.0_60.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 59s {color}
| {color:red} hadoop-yarn-applications-distributedshell in the patch failed
with JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 49m 34s {color}
| {color:red} hadoop-yarn-client in the patch failed with JDK v1.7.0_79.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 148m 27s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_60 Failed junit tests |
hadoop.yarn.applications.distributedshell.TestDistributedShellWithNodeLabels |
| | hadoop.yarn.client.TestGetGroups |
| JDK v1.8.0_60 Timed out junit tests |
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell |
| | org.apache.hadoop.yarn.client.api.impl.TestYarnClient |
| | org.apache.hadoop.yarn.client.api.impl.TestAMRMClient |
| |
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[
https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987868#comment-14987868
]
MENG DING commented on YARN-1509:
-
The test failure should be related to YARN-4326.
In addition, the mvn install test script is flawed. It goes directly into
{{hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell}}
directory and does a {{mvn install}} which still uses the old local maven repo
for yarn client. This causes the build to fail. The mvn install test should be
done in the root hadoop directory.
{code}
Tue Nov 3 16:18:38 UTC 2015
cd
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
mvn -Dmaven.repo.local=/home/jenkins/yetus-m2/hadoop-trunk-0 -DskipTests -fae
clean install -DskipTests=true -Dmaven.javadoc.skip=true
[INFO] Scanning for projects...
...
...
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @
hadoop-yarn-applications-distributedshell ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 4 source files to
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/target/classes
[INFO] -
[ERROR] COMPILATION ERROR :
[INFO] -
[ERROR]
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java:[735,50]
cannot find symbol
symbol: class AbstractCallbackHandler
location: class org.apache.hadoop.yarn.client.api.async.AMRMClientAsync
[ERROR]
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java:[559,20]
cannot find symbol
symbol: class AbstractCallbackHandler
location: class org.apache.hadoop.yarn.client.api.async.AMRMClientAsync
[ERROR]
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java:[737,5]
method does not override or implement a method from a supertype
[ERROR]
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java:[805,5]
method does not override or implement a method from a supertype
[ERROR]
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java:[838,5]
method does not override or implement a method from a supertype
[ERROR]
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java:[841,5]
method does not override or implement a method from a supertype
[ERROR]
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java:[846,5]
method does not override or implement a method from a supertype
[ERROR]
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java:[849,5]
method does not override or implement a method from a supertype
[ERROR]
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java:[857,5]
method does not override or implement a method from a supertype
[INFO] 9 errors
[INFO] -
[INFO]
[INFO] BUILD FAILURE
{code}
> Make AMRMClient support send increase container request and get
> increased/decreased containers
> --
>
> Key: YARN-1509
> URL: https://issues.apache.org/jira/browse/YARN-1509
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Wangda Tan (No longer used)
>Assignee: MENG DING
> Attachments: YARN-1509.1.patch, YARN-1509.10.patch,
> YARN-1509.2.patch,
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[ https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978430#comment-14978430 ] MENG DING commented on YARN-1509: - The failed tests are not related. > Make AMRMClient support send increase container request and get > increased/decreased containers > -- > > Key: YARN-1509 > URL: https://issues.apache.org/jira/browse/YARN-1509 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Wangda Tan (No longer used) >Assignee: MENG DING > Attachments: YARN-1509.1.patch, YARN-1509.2.patch, YARN-1509.3.patch, > YARN-1509.4.patch, YARN-1509.5.patch, YARN-1509.6.patch, YARN-1509.7.patch, > YARN-1509.8.patch > > > As described in YARN-1197, we need add API in AMRMClient to support > 1) Add increase request > 2) Can get successfully increased/decreased containers from RM -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[
https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976887#comment-14976887
]
Hadoop QA commented on YARN-1509:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 23m 38s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 11m 14s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 15m 22s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 36s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 7s | The applied patch generated 3
new checkstyle issues (total was 79, now 75). |
| {color:green}+1{color} | whitespace | 0m 14s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 2m 16s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 51s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 22s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | yarn tests | 6m 46s | Tests failed in
hadoop-yarn-applications-distributedshell. |
| {color:red}-1{color} | yarn tests | 8m 3s | Tests failed in
hadoop-yarn-client. |
| | | 72m 36s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.yarn.applications.distributedshell.TestDistributedShell |
| |
hadoop.yarn.applications.distributedshell.TestDistributedShellWithNodeLabels |
| | hadoop.yarn.client.api.impl.TestYarnClient |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12769009/YARN-1509.8.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / ed9806e |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/9590/artifact/patchprocess/diffcheckstylehadoop-yarn-client.txt
|
| hadoop-yarn-applications-distributedshell test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9590/artifact/patchprocess/testrun_hadoop-yarn-applications-distributedshell.txt
|
| hadoop-yarn-client test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9590/artifact/patchprocess/testrun_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9590/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9590/console |
This message was automatically generated.
> Make AMRMClient support send increase container request and get
> increased/decreased containers
> --
>
> Key: YARN-1509
> URL: https://issues.apache.org/jira/browse/YARN-1509
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Wangda Tan (No longer used)
>Assignee: MENG DING
> Attachments: YARN-1509.1.patch, YARN-1509.2.patch, YARN-1509.3.patch,
> YARN-1509.4.patch, YARN-1509.5.patch, YARN-1509.6.patch, YARN-1509.7.patch,
> YARN-1509.8.patch
>
>
> As described in YARN-1197, we need add API in AMRMClient to support
> 1) Add increase request
> 2) Can get successfully increased/decreased containers from RM
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[
https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975432#comment-14975432
]
Wangda Tan commented on YARN-1509:
--
[~mding], thanks for update, generally looks good, only one comment is, is it
possible to avoid adding dependency of commons-lang3? I'm afraid this new
dependency add some risks of jar dependency conflicts, etc.
I found the only place using commons-lang3 is Pair, could you take a look at if
{{AbstractMap.SimpleEntry}} meets your requirements?
> Make AMRMClient support send increase container request and get
> increased/decreased containers
> --
>
> Key: YARN-1509
> URL: https://issues.apache.org/jira/browse/YARN-1509
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Wangda Tan (No longer used)
>Assignee: MENG DING
> Attachments: YARN-1509.1.patch, YARN-1509.2.patch, YARN-1509.3.patch,
> YARN-1509.4.patch, YARN-1509.5.patch, YARN-1509.6.patch, YARN-1509.7.patch
>
>
> As described in YARN-1197, we need add API in AMRMClient to support
> 1) Add increase request
> 2) Can get successfully increased/decreased containers from RM
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[
https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975470#comment-14975470
]
Hadoop QA commented on YARN-1509:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 34s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 8m 4s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 37s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 25s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 47s | The applied patch generated 3
new checkstyle issues (total was 79, now 75). |
| {color:green}+1{color} | whitespace | 0m 12s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 31s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 37s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 40s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 6m 56s | Tests passed in
hadoop-yarn-applications-distributedshell. |
| {color:green}+1{color} | yarn tests | 7m 38s | Tests passed in
hadoop-yarn-client. |
| | | 56m 6s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12768817/YARN-1509.7.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 56e4f62 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/9583/artifact/patchprocess/diffcheckstylehadoop-yarn-client.txt
|
| hadoop-yarn-applications-distributedshell test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9583/artifact/patchprocess/testrun_hadoop-yarn-applications-distributedshell.txt
|
| hadoop-yarn-client test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9583/artifact/patchprocess/testrun_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9583/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9583/console |
This message was automatically generated.
> Make AMRMClient support send increase container request and get
> increased/decreased containers
> --
>
> Key: YARN-1509
> URL: https://issues.apache.org/jira/browse/YARN-1509
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Wangda Tan (No longer used)
>Assignee: MENG DING
> Attachments: YARN-1509.1.patch, YARN-1509.2.patch, YARN-1509.3.patch,
> YARN-1509.4.patch, YARN-1509.5.patch, YARN-1509.6.patch, YARN-1509.7.patch
>
>
> As described in YARN-1197, we need add API in AMRMClient to support
> 1) Add increase request
> 2) Can get successfully increased/decreased containers from RM
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[ https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14961035#comment-14961035 ] Bikas Saha commented on YARN-1509: -- Does issuing an increase followed by the decrease actually remove the pending change request on the RM or will it cause the RM to try to change a container resource to the same size as the existing resource and then go down the code path of increase (new token) or decrease (update NM). This would be a corner case that would be good to double check. If that works and the RM actually removes the pending container change request then we could use this mechanism to implement a cancel method wrapper in the AMRMClient. Otherwise, if fixes are needed on the RM side, we could so it separately when we fix the RM. > Make AMRMClient support send increase container request and get > increased/decreased containers > -- > > Key: YARN-1509 > URL: https://issues.apache.org/jira/browse/YARN-1509 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Wangda Tan (No longer used) >Assignee: MENG DING > Attachments: YARN-1509.1.patch, YARN-1509.2.patch, YARN-1509.3.patch, > YARN-1509.4.patch, YARN-1509.5.patch > > > As described in YARN-1197, we need add API in AMRMClient to support > 1) Add increase request > 2) Can get successfully increased/decreased containers from RM -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[ https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14961154#comment-14961154 ] Bikas Saha commented on YARN-1509: -- Sounds good > Make AMRMClient support send increase container request and get > increased/decreased containers > -- > > Key: YARN-1509 > URL: https://issues.apache.org/jira/browse/YARN-1509 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Wangda Tan (No longer used) >Assignee: MENG DING > Attachments: YARN-1509.1.patch, YARN-1509.2.patch, YARN-1509.3.patch, > YARN-1509.4.patch, YARN-1509.5.patch > > > As described in YARN-1197, we need add API in AMRMClient to support > 1) Add increase request > 2) Can get successfully increased/decreased containers from RM -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[
https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14961090#comment-14961090
]
Wangda Tan commented on YARN-1509:
--
Sync up with [~mding]. I think we're all on the same page now. Some thoughts
from my side:
- We can separate cancel container resource request change to a separated JIRA.
Now using decrease container to same size to cancel pending increase request is
like a workaround, we'd better add a protocol to AMRMProtocol so AMRMClient can
do things correct.
- For the onContainerResourceChanged API
{code}
public abstract void onContainersResourceChanged(List containers);
{code}
This looks good to me. Since we have a unified API to request change container,
return the updated container to AM should be enough, AM knows if it's an
increased or decreased container.
Thoughts? [~bikassaha].
> Make AMRMClient support send increase container request and get
> increased/decreased containers
> --
>
> Key: YARN-1509
> URL: https://issues.apache.org/jira/browse/YARN-1509
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Wangda Tan (No longer used)
>Assignee: MENG DING
> Attachments: YARN-1509.1.patch, YARN-1509.2.patch, YARN-1509.3.patch,
> YARN-1509.4.patch, YARN-1509.5.patch
>
>
> As described in YARN-1197, we need add API in AMRMClient to support
> 1) Add increase request
> 2) Can get successfully increased/decreased containers from RM
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[
https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14960927#comment-14960927
]
MENG DING commented on YARN-1509:
-
Hi, [~bikassaha]
Thanks for the comments.
I probably didn't make myself clear. We are on the SAME page that for the sake
of point 2 alone, it already makes sense to combine increase/decrease API into
one change API: {code}public abstract void
requestContainerResourceChange(Container container, Resource capability);{code}
What I was trying to say in the previous post is that to support mix of
increase and decrease in one change request (point 1) doesn't seem to be very
feasible (even at a later date). But I don't think we need to worry about that
for now.
Since we are combining increase/decrease API, we definitely should combine the
callback methods into one as well: onContainerResourceChanged(). At this point,
I am inclined to simply do the following which doesn't incur much code changes.
I will discuss further with [~leftnoteasy] on this. {code}public abstract void
onContainersResourceChanged(List containers);{code}
bq. Or are you saying that invalid container resource change requests are
immediately rejected by the RM synchronously in the allocate RPC?
Yes, the ApplicationMasterService will perform a series of sanity checks (e.g.,
requested resource <= maximum allocation, etc), and reject invalid requests
immediately. This is the same for other requests too.
bq. Having a simple cancel request regardless of increase or decrease is
preferable since then we are not leaking the current state of the
implementation to the user. It is future safe
Make sense to me. We can probably have something like
{{cancelContainerResourceChange(Container container)}}, which applies to the
container that has an outstanding pending increase sitting in the pendingChange
map. There is no explicit protocol to support cancellation of resource change
yet. For now we can achieve that by issuing a backend decrease request with the
target resource set to the same as the current resource, which effectively
cancels any outstanding increase request.
> Make AMRMClient support send increase container request and get
> increased/decreased containers
> --
>
> Key: YARN-1509
> URL: https://issues.apache.org/jira/browse/YARN-1509
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Wangda Tan (No longer used)
>Assignee: MENG DING
> Attachments: YARN-1509.1.patch, YARN-1509.2.patch, YARN-1509.3.patch,
> YARN-1509.4.patch, YARN-1509.5.patch
>
>
> As described in YARN-1197, we need add API in AMRMClient to support
> 1) Add increase request
> 2) Can get successfully increased/decreased containers from RM
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[
https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14961578#comment-14961578
]
Hadoop QA commented on YARN-1509:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 18s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 8m 2s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 44s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 54s | The applied patch generated 3
new checkstyle issues (total was 79, now 76). |
| {color:green}+1{color} | whitespace | 0m 12s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 38s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 35s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 40s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 7m 0s | Tests passed in
hadoop-yarn-applications-distributedshell. |
| {color:green}+1{color} | yarn tests | 7m 29s | Tests passed in
hadoop-yarn-client. |
| | | 55m 59s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12767154/YARN-1509.6.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f9da5cd |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/9467/artifact/patchprocess/diffcheckstylehadoop-yarn-client.txt
|
| hadoop-yarn-applications-distributedshell test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9467/artifact/patchprocess/testrun_hadoop-yarn-applications-distributedshell.txt
|
| hadoop-yarn-client test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9467/artifact/patchprocess/testrun_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9467/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9467/console |
This message was automatically generated.
> Make AMRMClient support send increase container request and get
> increased/decreased containers
> --
>
> Key: YARN-1509
> URL: https://issues.apache.org/jira/browse/YARN-1509
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Wangda Tan (No longer used)
>Assignee: MENG DING
> Attachments: YARN-1509.1.patch, YARN-1509.2.patch, YARN-1509.3.patch,
> YARN-1509.4.patch, YARN-1509.5.patch, YARN-1509.6.patch
>
>
> As described in YARN-1197, we need add API in AMRMClient to support
> 1) Add increase request
> 2) Can get successfully increased/decreased containers from RM
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[
https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959077#comment-14959077
]
MENG DING commented on YARN-1509:
-
Hi, [~bikassaha]
Apologize for the late response, I was out traveling and just came back.
bq. A change container request (maybe not supported now) can be increase cpu +
decrease memory. Hence a built in concept of increase and decrease in the API
is something I am wary off
>From the design stage of this project, I believe the semantics of "changing
>container resource" was meant to be either "increase" or "decrease", this was
>re-enforced with the design choices that successful increase and decrease of
>resource go through different paths. I have some concerns of extending the
>semantics to something like "increase cpu + decrease memory" inside one change
>request:
* Decrease resource happens immediately, while increase resource involves
handing out a token together with a user action to increase on NM. If we extend
the semantics, we need to educate the user that once a change request is
approved, it means that the decrease part of the request is effective
immediately, while the increase part of the request is still pending on user
action. Could it be too confusing?
* To make matters worse, if the increase token expires and the RM rolls back
the allocation of the increase part of the request, we end up with a partially
fulfilled request, as we are not able to rollback the decrease part of the
request.
IMHO, it is much cleaner to clearly separate increase and decrease requests at
the user API level. If a user wants to increase cpu and decrease memory, he
should send out two separate requests. Thoughts?
bq. So how about {code}public abstract void
onContainersResourceChanged(Map oldToNewContainers);
{code} OR {code}public abstract void
onContainersResourceChanged(List
updatedContainerInfo);{code}
I thought about providing the old containers in the callback method. Right now
{{AMRMClientImpl}} remembers old containers in the {{pendingChange}} map, but
the problem is, in the {{AMRMClientImpl.allocate}} call, once an
increase/decrease approval is received, the old containers are immediately
removed from the pending map. So by the time the {{AMRMClientAsyncImpl}}
callback handler thread starts to process the response, the old containers
won't be there any more:
{code}
+if (!pendingIncrease.isEmpty() &&
!allocateResponse.getIncreasedContainers().isEmpty()) {
+
removePendingChangeRequests(allocateResponse.getDecreasedContainers(), true);
+}
+if (!pendingDecrease.isEmpty() &&
!allocateResponse.getDecreasedContainers().isEmpty()) {
+
removePendingChangeRequests(allocateResponse.getDecreasedContainers(), false);
+}
{code}
My thought is, since we already ask the user to provide the old container when
he sends out the change request, he should have the old container already, so
we don't necessarily have to provide the old container info in the callback
method. Thoughts?
bq. Would there be a case (maybe not currently) when a change container request
can fail on the RM? Should the callback allow notifying about a failure to
change the container?
The {{AbstractCallbackHandler.onError}} will be called when the change
container request throws exception on the RM side.
bq. What is the RM notifies AMRMClient about a container completed. That
container happens to have a pending change request? What should happen in this
case? Should the AMRM client clear that pending request? Should it also notify
the user that pending container change request has failed or just rely on
onContainerCompleted() to let the AM get that information.
I think in this case AMRMClient should clear all pending requests that belong
to this container. I will add that logic in. Thanks!
bq. I would be wary of overloading cancel with a second container change
request. To be clear, here we are discussing user facing semantics and API.
Having clear semantics is important vs implicit or overloaded behavior.
I am not against providing a separate cancel API. But I think the API needs to
be clear that the cancel is only for increase request, NOT decrease request
(just like we don't have something like cancel release container). For example,
we can have something like the following. Thoughts?
{code}
public abstract void cancelContainerResourceIncrease(Container container)
{code}
bq. Is there an existing test for that code path that could be augmented to
make sure that the new changes are tested?
I didn't find existing tests that test the pending list on RM restart, I will
try to add a test case for that. Thanks.
> Make AMRMClient support send increase container request and get
> increased/decreased containers
> --
>
>
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[ https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959502#comment-14959502 ] MENG DING commented on YARN-1509: - bq. I didn't find existing tests that test the pending list on RM restart, I will try to add a test case for that Correction, I found an existing test (i.e. testAMRMClientResendsRequestsOnRMRestart) that test the pending list on RM restart. Will augment that test to test pendingChange map. > Make AMRMClient support send increase container request and get > increased/decreased containers > -- > > Key: YARN-1509 > URL: https://issues.apache.org/jira/browse/YARN-1509 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Wangda Tan (No longer used) >Assignee: MENG DING > Attachments: YARN-1509.1.patch, YARN-1509.2.patch, YARN-1509.3.patch, > YARN-1509.4.patch, YARN-1509.5.patch > > > As described in YARN-1197, we need add API in AMRMClient to support > 1) Add increase request > 2) Can get successfully increased/decreased containers from RM -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[ https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959877#comment-14959877 ] Bikas Saha commented on YARN-1509: -- bq. Increase/Decrease/Change Not sure why the implementation went down that route of creating a separation of increase vs decrease throughout the flow. In any case, that is the back-end implementation. That can change and evolve without affecting user code. This is the user facing API. So once code is written against this API making backwards incompatible changes or providing new functionality in the future needs to be considered. Also just API simplicity needs to be considered. 1) Having a changeRequest API allows us to support other kinds of changes (mix of increase or decrease) at a later point. For now, the API could check for increase/decrease (which it has to do anyways for sanity checking) and reject unsupported scenarios. 2) Even for just increase or decrease, given the user can provide the old Container + new Request Size, it should be easy for the library to figure out if an increase or decrease is needed. Why burden the user by having them call 2 different API's that are essentially making the same request. Having to handle container token for increase but do nothing for decrease is something that already has to be explained to the user. Same for the fact that decrease is quick but not increase. So those aspects of user education are probably orthogonal. We have iterated on this API question a few times now. In the above 2 points I have tried to summarize the reasons for requesting to change instead of increase/decrease. Even if we dont support both increase and decrease (point 1) I think the simplicity of having a single API (point 2) would be an advantage of having change vs increase/decrease. This also simplifies to having a single callback onContainerResourceChanged() instead of 2 callbacks. At this point, I will request you and [~leftnoteasy] to consider both future extensions and API simplicity to make a final call on having 2 APIs and 2 callbacks vs 1. bq. My thought is, since we already ask the user to provide the old container when he sends out the change request, he should have the old container already, so we don't necessarily have to provide the old container info in the callback method I am fine either ways. I discussed the same thing offline with [~leftnoteasy] and he thought that having the old container info would make life easier for the user. A user who is using this API is very likely going to have state about the container for which a change has been requested and can match them up using the containerId. bq. The AbstractCallbackHandler.onError will be called when the change container request throws exception on the RM side. The change request is sent inside the allocate heartbeat request, right? So i am not sure how we get an exception back for the specific case of a failed container change request. Or are you saying that invalid container resource change requests are immediately rejected by the RM synchronously in the allocate RPC? bq. I am not against providing a separate cancel API. But I think the API needs to be clear that the cancel is only for increase request, NOT decrease request (just like we don't have something like cancel release container). Having a simple cancel request regardless of increase or decrease is preferable since then we are not leaking the current state of the implementation to the user. It is future safe E.g. later if we find an issue with decreasing and fixing that makes it non-instantaneous, then we dont want to have to change the API to support that. But today, given that it is instantaneous, we can simply ignore the cancellation of a decrease in the cancel method of AMRMClient. I think the RM does not support a cancel container resource change request. Does it? If it does not, then perhaps this API can be discussed/added in a separate jira after there is back end support for it. > Make AMRMClient support send increase container request and get > increased/decreased containers > -- > > Key: YARN-1509 > URL: https://issues.apache.org/jira/browse/YARN-1509 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Wangda Tan (No longer used) >Assignee: MENG DING > Attachments: YARN-1509.1.patch, YARN-1509.2.patch, YARN-1509.3.patch, > YARN-1509.4.patch, YARN-1509.5.patch > > > As described in YARN-1197, we need add API in AMRMClient to support > 1) Add increase request > 2) Can get successfully increased/decreased containers from RM -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[
https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950835#comment-14950835
]
Bikas Saha commented on YARN-1509:
--
A change container request (maybe not supported now) can be increase cpu +
decrease memory. Hence a built in concept of increase and decrease in the API
is something I am wary off.
So how about
{code} public abstract void onContainersResourceChanged(
Map oldToNewContainers);
OR
public abstract void onContainersResourceChanged(
List updatedContainerInfo);{code}
Would there be a case (maybe not currently) when a change container request can
fail on the RM? Should the callback allow notifying about a failure to change
the container?
What is the RM notifies AMRMClient about a container completed. That container
happens to have a pending change request? What should happen in this case?
Should the AMRM client clear that pending request? Should it also notify the
user that pending container change request has failed or just rely on
onContainerCompleted() to let the AM get that information.
I would be wary of overloading cancel with a second container change request.
To be clear, here we are discussing user facing semantics and API. Having clear
semantics is important vs implicit or overloaded behavior. E.g. are there cases
where an increase followed by a decrease request is a valid scenario and how
would that be different compare to an increase followed by a cancel. Should the
RM do different things for increase followed by cancel vs increase followed by
decrease?
AM restart does not need any handling since the AM is going to start from a
clean slate. Sorry, my bad.
I missed the handling of the RM restart case. Is there an existing test for
that code path that could be augmented to make sure that the new changes are
tested?
> Make AMRMClient support send increase container request and get
> increased/decreased containers
> --
>
> Key: YARN-1509
> URL: https://issues.apache.org/jira/browse/YARN-1509
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Wangda Tan (No longer used)
>Assignee: MENG DING
> Attachments: YARN-1509.1.patch, YARN-1509.2.patch, YARN-1509.3.patch,
> YARN-1509.4.patch, YARN-1509.5.patch
>
>
> As described in YARN-1197, we need add API in AMRMClient to support
> 1) Add increase request
> 2) Can get successfully increased/decreased containers from RM
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[ https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14948709#comment-14948709 ] MENG DING commented on YARN-1509: - Hi, [~bikassaha] Thanks a lot for the valuable comments! bq. Why are there separate methods for increase and decrease instead of a single method to change the container resource size? By comparing the existing resource allocation to a container and the new requested resource allocation, it should be clear whether an increase or decrease is being requested. As discussed in the design stage, and also described in the design doc, the reason to separate the increase/decrease requests in the APIs and AMRM protocol is to make sure that users will make a conscious decision when they are making these requests. It is also much easier to catch any potential mistakes that the user could make. For example, if a user intends to increase resource of a container, but for whatever reason mistakenly specifies a target resource that is smaller than the current resource, RM can catch that and throw exception. bq. Also, for completeness, is there a need for a cancelContainerResourceChange()? After a container resource change request has been submitted, what are my options as a user other than to wait for the request to be satisfied by the RM? For container resource decrease request, there is practically no chance (and probably no need) to cancel the request, as it happens immediately when scheduler process the request (this is similar to the release container request). For container resource increase, the user can cancel any pending increase request still sitting in RM by sending a decrease request of the same size of the current container size. I will improve the Javadoc description to make it clear on this. bq. If I release the container, then does it mean all pending change requests for that container should be removed? From a quick look at the patch, it does not look like that is being covered, unless I am missing something. You are right that releasing a container should cancel all pending change requests for that container. This is missing in the current implementation, I will add that. bq. What will happen if the AM restarts after submitting a change request. Does the AM-RM re-register protocol need an update to handle the case of re-synchronizing on the change requests? Whats happens if the RM restarts? If these are explained in a document, then please point me to the document. The patch did not seem to have anything around this area. So I thought I would ask The current implementation handles RM restarts by maintaining a pendingIncrease and pendingDecrease map, just like the pendingRelease list. This is covered in the design doc. For AM restarts, I am not sure what we need to do here. Does AM-RM re-register protocol currently handle the re-synchronize of outstanding new container requests after AM is restarted? Will you be able to elaborate a little bit on this? bq. Also, why have the callback interface methods been made non-public? Would that be an incompatible change? All interface methods are implicitly public and abstract. The existing public modifier on these methods are redundant, so I removed them. > Make AMRMClient support send increase container request and get > increased/decreased containers > -- > > Key: YARN-1509 > URL: https://issues.apache.org/jira/browse/YARN-1509 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Wangda Tan (No longer used) >Assignee: MENG DING > Attachments: YARN-1509.1.patch, YARN-1509.2.patch, YARN-1509.3.patch, > YARN-1509.4.patch, YARN-1509.5.patch > > > As described in YARN-1197, we need add API in AMRMClient to support > 1) Add increase request > 2) Can get successfully increased/decreased containers from RM -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[
https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14949639#comment-14949639
]
MENG DING commented on YARN-1509:
-
Had an offline discussion with [~leftnoteasy] and [~bikassaha]. Overall we
agreed that we can combine the separate increase/decrease requests into one API
in the client:
* Combine {{requestContainerResourceIncrease}} and
{{requestContainerResourceDecrease}} into one API. For example:
{code}
/**
* Request container resource change before calling allocate.
* Any previous pending resource change request of the same container will be
* cancelled.
*
* @param container The container returned from the last successful resource
* allocation or resource change
* @param capability The target resource capability of the container
*/
public abstract void requestContainerResourceChange(
Container container, Resource capability);
{code}
User must pass in a container object (instead of just a container ID), and the
target resource capability. Because the container object contains the existing
container Resource, the AMRMClient can use that information to compare against
the target resource to figure out if this is an increase or decrease request.
* There is *NO* need to change the AMRM protocol.
* For the CallbackHandler methods, we can also combine
{{onContainersResourceDecreased}} and {{onContainersResourceIncreased}} into
one API:
{code}
public abstract void onContainersResourceChanged(
List containers);
{code}
The user can compare the passed-in containers with the containers they have
remembered to determine if this is an increase or decrease request. Or maybe we
can make it even simpler by doing something like the following? Thoughts?
{code}
public abstract void onContainersResourceChanged(
List increasedContainers, List
decreasedContainers);
{code}
* We can *deprecate* the existing CallbackHandler interface and use the
AbstractCallbackHandler instead.
[~bikassaha], [~leftnoteasy], any comments?
> Make AMRMClient support send increase container request and get
> increased/decreased containers
> --
>
> Key: YARN-1509
> URL: https://issues.apache.org/jira/browse/YARN-1509
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Wangda Tan (No longer used)
>Assignee: MENG DING
> Attachments: YARN-1509.1.patch, YARN-1509.2.patch, YARN-1509.3.patch,
> YARN-1509.4.patch, YARN-1509.5.patch
>
>
> As described in YARN-1197, we need add API in AMRMClient to support
> 1) Add increase request
> 2) Can get successfully increased/decreased containers from RM
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[
https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14947025#comment-14947025
]
Hadoop QA commented on YARN-1509:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 52s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 8m 3s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 28s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 19s | The applied patch generated
1 release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 28s | The applied patch generated 5
new checkstyle issues (total was 79, now 78). |
| {color:green}+1{color} | whitespace | 0m 8s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 52s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | yarn tests | 7m 23s | Tests failed in
hadoop-yarn-client. |
| | | 46m 45s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.yarn.client.api.impl.TestYarnClient |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12765393/YARN-1509.5.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 61b3547 |
| Release Audit |
https://builds.apache.org/job/PreCommit-YARN-Build/9371/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/9371/artifact/patchprocess/diffcheckstylehadoop-yarn-client.txt
|
| hadoop-yarn-client test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9371/artifact/patchprocess/testrun_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9371/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9371/console |
This message was automatically generated.
> Make AMRMClient support send increase container request and get
> increased/decreased containers
> --
>
> Key: YARN-1509
> URL: https://issues.apache.org/jira/browse/YARN-1509
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Wangda Tan (No longer used)
>Assignee: MENG DING
> Attachments: YARN-1509.1.patch, YARN-1509.2.patch, YARN-1509.3.patch,
> YARN-1509.4.patch, YARN-1509.5.patch
>
>
> As described in YARN-1197, we need add API in AMRMClient to support
> 1) Add increase request
> 2) Can get successfully increased/decreased containers from RM
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[ https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14947056#comment-14947056 ] MENG DING commented on YARN-1509: - Test failure is not related to this patch. Checkstyle warning is the same as before. > Make AMRMClient support send increase container request and get > increased/decreased containers > -- > > Key: YARN-1509 > URL: https://issues.apache.org/jira/browse/YARN-1509 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Wangda Tan (No longer used) >Assignee: MENG DING > Attachments: YARN-1509.1.patch, YARN-1509.2.patch, YARN-1509.3.patch, > YARN-1509.4.patch, YARN-1509.5.patch > > > As described in YARN-1197, we need add API in AMRMClient to support > 1) Add increase request > 2) Can get successfully increased/decreased containers from RM -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[ https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14947315#comment-14947315 ] Bikas Saha commented on YARN-1509: -- Sorry for coming in late on this. I have some questions on the API. Why are there separate methods for increase and decrease instead of a single method to change the container resource size? By comparing the existing resource allocation to a container and the new requested resource allocation, it should be clear whether an increase or decrease is being requested. Also, for completeness, is there a need for a cancelContainerResourceChange()? After a container resource change request has been submitted, what are my options as a user other than to wait for the request to be satisfied by the RM? If I release the container, then does it mean all pending change requests for that container should be removed? From a quick look at the patch, it does not look like that is being covered, unless I am missing something. What will happen if the AM restarts after submitting a change request. Does the AM-RM re-register protocol need an update to handle the case of re-synchronizing on the change requests? Whats happens if the RM restarts? If these are explained in a document, then please point me to the document. The patch did not seem to have anything around this area. So I thought I would ask. Also, why have the callback interface methods been made non-public? Would that be an incompatible change? > Make AMRMClient support send increase container request and get > increased/decreased containers > -- > > Key: YARN-1509 > URL: https://issues.apache.org/jira/browse/YARN-1509 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Wangda Tan (No longer used) >Assignee: MENG DING > Attachments: YARN-1509.1.patch, YARN-1509.2.patch, YARN-1509.3.patch, > YARN-1509.4.patch, YARN-1509.5.patch > > > As described in YARN-1197, we need add API in AMRMClient to support > 1) Add increase request > 2) Can get successfully increased/decreased containers from RM -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[ https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946020#comment-14946020 ] Wangda Tan commented on YARN-1509: -- Thanks [~mding], I think patch looks good, and your response makes sense to me. One nit is: could you wrap Log.debug with Log.isDebugEnabled? Will commit once Jenkins get back. > Make AMRMClient support send increase container request and get > increased/decreased containers > -- > > Key: YARN-1509 > URL: https://issues.apache.org/jira/browse/YARN-1509 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Wangda Tan (No longer used) >Assignee: MENG DING > Attachments: YARN-1509.1.patch, YARN-1509.2.patch, YARN-1509.3.patch, > YARN-1509.4.patch > > > As described in YARN-1197, we need add API in AMRMClient to support > 1) Add increase request > 2) Can get successfully increased/decreased containers from RM -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[
https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14943567#comment-14943567
]
Hadoop QA commented on YARN-1509:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 17s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 59s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 18s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 15s | The applied patch generated
1 release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 30s | The applied patch generated 5
new checkstyle issues (total was 79, now 78). |
| {color:green}+1{color} | whitespace | 0m 8s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 30s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 36s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 53s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 7m 31s | Tests passed in
hadoop-yarn-client. |
| | | 46m 1s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12765011/YARN-1509.4.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / b925cf1 |
| Release Audit |
https://builds.apache.org/job/PreCommit-YARN-Build/9346/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/9346/artifact/patchprocess/diffcheckstylehadoop-yarn-client.txt
|
| hadoop-yarn-client test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9346/artifact/patchprocess/testrun_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9346/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9346/console |
This message was automatically generated.
> Make AMRMClient support send increase container request and get
> increased/decreased containers
> --
>
> Key: YARN-1509
> URL: https://issues.apache.org/jira/browse/YARN-1509
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Wangda Tan (No longer used)
>Assignee: MENG DING
> Attachments: YARN-1509.1.patch, YARN-1509.2.patch, YARN-1509.3.patch,
> YARN-1509.4.patch
>
>
> As described in YARN-1197, we need add API in AMRMClient to support
> 1) Add increase request
> 2) Can get successfully increased/decreased containers from RM
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[ https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14943587#comment-14943587 ] MENG DING commented on YARN-1509: - * release audit is not related * will apply for exception for checkstyle: ** relaxed visibility is for testing purposes. ** function length exceeding limit is caused by long comments. > Make AMRMClient support send increase container request and get > increased/decreased containers > -- > > Key: YARN-1509 > URL: https://issues.apache.org/jira/browse/YARN-1509 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Wangda Tan (No longer used) >Assignee: MENG DING > Attachments: YARN-1509.1.patch, YARN-1509.2.patch, YARN-1509.3.patch, > YARN-1509.4.patch > > > As described in YARN-1197, we need add API in AMRMClient to support > 1) Add increase request > 2) Can get successfully increased/decreased containers from RM -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[
https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14941992#comment-14941992
]
Hadoop QA commented on YARN-1509:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 52s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 8m 5s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 29s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 17s | The applied patch generated
1 release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 36s | The applied patch generated 5
new checkstyle issues (total was 79, now 78). |
| {color:red}-1{color} | whitespace | 0m 5s | The patch has 4 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 29s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 56s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 7m 26s | Tests passed in
hadoop-yarn-client. |
| | | 46m 52s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12764874/YARN-1509.3.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / fdf02d1 |
| Release Audit |
https://builds.apache.org/job/PreCommit-YARN-Build/9338/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/9338/artifact/patchprocess/diffcheckstylehadoop-yarn-client.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/9338/artifact/patchprocess/whitespace.txt
|
| hadoop-yarn-client test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9338/artifact/patchprocess/testrun_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9338/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9338/console |
This message was automatically generated.
> Make AMRMClient support send increase container request and get
> increased/decreased containers
> --
>
> Key: YARN-1509
> URL: https://issues.apache.org/jira/browse/YARN-1509
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Wangda Tan (No longer used)
>Assignee: MENG DING
> Attachments: YARN-1509.1.patch, YARN-1509.2.patch, YARN-1509.3.patch
>
>
> As described in YARN-1197, we need add API in AMRMClient to support
> 1) Add increase request
> 2) Can get successfully increased/decreased containers from RM
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[
https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14935681#comment-14935681
]
MENG DING commented on YARN-1509:
-
Thanks for the review [~leftnoteasy]!
bq. I think we can simply add decreaseList to decrease and increaseList to
increase.
In most cases, the current logic effectively adds decreaseList to decrease map,
and increaseList to increase map. But since the allocate call
{{allocateResponse = allocate(progressIndicator)}} is not synchronized, during
the allocation, new increase/decrease requests may have been added to the
increase/decrease table, which IMO should not be overwritten by the old
requests cached in increaseList and decreaseList. This is similar to the new
container requests logic when allocation fails. Let me know if you think
otherwise.
bq, if request matches, we can print some logs to show this
Will do.
> Make AMRMClient support send increase container request and get
> increased/decreased containers
> --
>
> Key: YARN-1509
> URL: https://issues.apache.org/jira/browse/YARN-1509
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Wangda Tan (No longer used)
>Assignee: MENG DING
> Attachments: YARN-1509.1.patch, YARN-1509.2.patch
>
>
> As described in YARN-1197, we need add API in AMRMClient to support
> 1) Add increase request
> 2) Can get successfully increased/decreased containers from RM
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[
https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14935618#comment-14935618
]
Wangda Tan commented on YARN-1509:
--
Thanks [~mding],
I think patch generally looks good, one query:
{code}
// increase/decrease requests could have been added during the
// allocate call. Those are the newest requests which take precedence
// over requests cached in the increaseList and decreaseList.
//
// Only insert entries from the cached increaseList and decreaseList
// that do not exist in either of current decrease and increase maps:
// 1. If the cached increaseList contains the same container as that
//in the new increase map, then there is nothing to do as the
//the request in the new increase map has the latest value.
// 2. If the cached increaseList contains the same container as that
//in the new decrease map, then there is nothing to do either as
//the request in the new decrease map is newer and should cancel
//the old increase request.
// 3. The above also apply to the decreaseList.
for (ContainerResourceChangeRequest oldIncrease : increaseList) {
ContainerId oldContainerId = oldIncrease.getContainerId();
if (increase.get(oldContainerId) == null
&& decrease.get(oldContainerId) == null) {
increase.put(oldContainerId, oldIncrease.getCapability());
}
}
for (ContainerResourceChangeRequest oldDecrease : decreaseList) {
ContainerId oldContainerId = oldDecrease.getContainerId();
if (decrease.get(oldContainerId) == null
&& increase.get(oldContainerId) == null) {
decrease.put(oldContainerId, oldDecrease.getCapability());
}
}
{code}
I think we can simply add decreaseList to decrease and increaseList to
increase. If AllocateResponse == null, we assume allocation fails, and
scheduler's increase/decrease table isn't updated. In this case, I think we
should simply revert changes to increase/decrease table. Thoughts?
And I think we can add some debug/info message as well. For example, at
{{removePendingChangeRequests}}, if request matches, we can print some logs to
show this.
> Make AMRMClient support send increase container request and get
> increased/decreased containers
> --
>
> Key: YARN-1509
> URL: https://issues.apache.org/jira/browse/YARN-1509
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Wangda Tan (No longer used)
>Assignee: MENG DING
> Attachments: YARN-1509.1.patch, YARN-1509.2.patch
>
>
> As described in YARN-1197, we need add API in AMRMClient to support
> 1) Add increase request
> 2) Can get successfully increased/decreased containers from RM
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1509) Make AMRMClient support send increase container request and get increased/decreased containers
[
https://issues.apache.org/jira/browse/YARN-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14934196#comment-14934196
]
Hadoop QA commented on YARN-1509:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 27s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 55s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 16s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 8s | The applied patch generated 5
new checkstyle issues (total was 79, now 84). |
| {color:green}+1{color} | whitespace | 0m 10s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 28s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 3s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | yarn tests | 1m 17s | Tests failed in
hadoop-yarn-applications-distributedshell. |
| {color:green}+1{color} | yarn tests | 7m 28s | Tests passed in
hadoop-yarn-client. |
| {color:red}-1{color} | yarn tests | 56m 16s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 108m 56s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.yarn.applications.distributedshell.TestDistributedShellWithNodeLabels |
| | hadoop.yarn.applications.distributedshell.TestDistributedShell |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService
|
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12764082/YARN-1509.2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 50741cb |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/9290/artifact/patchprocess/diffcheckstylehadoop-yarn-client.txt
|
| hadoop-yarn-applications-distributedshell test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9290/artifact/patchprocess/testrun_hadoop-yarn-applications-distributedshell.txt
|
| hadoop-yarn-client test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9290/artifact/patchprocess/testrun_hadoop-yarn-client.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9290/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9290/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9290/console |
This message was automatically generated.
> Make AMRMClient support send increase container request and get
> increased/decreased containers
> --
>
> Key: YARN-1509
> URL: https://issues.apache.org/jira/browse/YARN-1509
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Wangda Tan (No longer used)
>Assignee: MENG DING
> Attachments: YARN-1509.1.patch, YARN-1509.2.patch
>
>
> As described in YARN-1197, we need add API in AMRMClient to support
> 1) Add increase request
> 2) Can get successfully increased/decreased containers from RM
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
