[jira] [Commented] (YARN-2354) DistributedShell may allocate more containers than client specified after it restarts
[ https://issues.apache.org/jira/browse/YARN-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14079339#comment-14079339 ] Hudson commented on YARN-2354: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1847 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1847/]) YARN-2354. DistributedShell may allocate more containers than client specified after AM restarts. Contributed by Li Lu (jianhe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1614538) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDSFailedAppMaster.java > DistributedShell may allocate more containers than client specified after it > restarts > - > > Key: YARN-2354 > URL: https://issues.apache.org/jira/browse/YARN-2354 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Li Lu > Fix For: 2.6.0 > > Attachments: YARN-2354-072514.patch, YARN-2354-072814.patch, > YARN-2354-072914.patch > > > To reproduce, run distributed shell with -num_containers option, > In ApplicationMaster.java, the following code has some issue. > {code} > int numTotalContainersToRequest = > numTotalContainers - previousAMRunningContainers.size(); > for (int i = 0; i < numTotalContainersToRequest; ++i) { > ContainerRequest containerAsk = setupContainerAskForRM(); > amRMClient.addContainerRequest(containerAsk); > } > numRequestedContainers.set(numTotalContainersToRequest); > {code} > numRequestedContainers doesn't account for previous AM's requested > containers. so numRequestedContainers should be set to numTotalContainers -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2354) DistributedShell may allocate more containers than client specified after it restarts
[ https://issues.apache.org/jira/browse/YARN-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14079290#comment-14079290 ] Hudson commented on YARN-2354: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1821 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1821/]) YARN-2354. DistributedShell may allocate more containers than client specified after AM restarts. Contributed by Li Lu (jianhe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1614538) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDSFailedAppMaster.java > DistributedShell may allocate more containers than client specified after it > restarts > - > > Key: YARN-2354 > URL: https://issues.apache.org/jira/browse/YARN-2354 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Li Lu > Fix For: 2.6.0 > > Attachments: YARN-2354-072514.patch, YARN-2354-072814.patch, > YARN-2354-072914.patch > > > To reproduce, run distributed shell with -num_containers option, > In ApplicationMaster.java, the following code has some issue. > {code} > int numTotalContainersToRequest = > numTotalContainers - previousAMRunningContainers.size(); > for (int i = 0; i < numTotalContainersToRequest; ++i) { > ContainerRequest containerAsk = setupContainerAskForRM(); > amRMClient.addContainerRequest(containerAsk); > } > numRequestedContainers.set(numTotalContainersToRequest); > {code} > numRequestedContainers doesn't account for previous AM's requested > containers. so numRequestedContainers should be set to numTotalContainers -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2354) DistributedShell may allocate more containers than client specified after it restarts
[ https://issues.apache.org/jira/browse/YARN-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14079195#comment-14079195 ] Hudson commented on YARN-2354: -- FAILURE: Integrated in Hadoop-Yarn-trunk #628 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/628/]) YARN-2354. DistributedShell may allocate more containers than client specified after AM restarts. Contributed by Li Lu (jianhe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1614538) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDSFailedAppMaster.java > DistributedShell may allocate more containers than client specified after it > restarts > - > > Key: YARN-2354 > URL: https://issues.apache.org/jira/browse/YARN-2354 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Li Lu > Fix For: 2.6.0 > > Attachments: YARN-2354-072514.patch, YARN-2354-072814.patch, > YARN-2354-072914.patch > > > To reproduce, run distributed shell with -num_containers option, > In ApplicationMaster.java, the following code has some issue. > {code} > int numTotalContainersToRequest = > numTotalContainers - previousAMRunningContainers.size(); > for (int i = 0; i < numTotalContainersToRequest; ++i) { > ContainerRequest containerAsk = setupContainerAskForRM(); > amRMClient.addContainerRequest(containerAsk); > } > numRequestedContainers.set(numTotalContainersToRequest); > {code} > numRequestedContainers doesn't account for previous AM's requested > containers. so numRequestedContainers should be set to numTotalContainers -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2354) DistributedShell may allocate more containers than client specified after it restarts
[ https://issues.apache.org/jira/browse/YARN-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14078859#comment-14078859 ] Hudson commented on YARN-2354: -- FAILURE: Integrated in Hadoop-trunk-Commit #5983 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5983/]) YARN-2354. DistributedShell may allocate more containers than client specified after AM restarts. Contributed by Li Lu (jianhe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1614538) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDSFailedAppMaster.java > DistributedShell may allocate more containers than client specified after it > restarts > - > > Key: YARN-2354 > URL: https://issues.apache.org/jira/browse/YARN-2354 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Li Lu > Fix For: 2.6.0 > > Attachments: YARN-2354-072514.patch, YARN-2354-072814.patch, > YARN-2354-072914.patch > > > To reproduce, run distributed shell with -num_containers option, > In ApplicationMaster.java, the following code has some issue. > {code} > int numTotalContainersToRequest = > numTotalContainers - previousAMRunningContainers.size(); > for (int i = 0; i < numTotalContainersToRequest; ++i) { > ContainerRequest containerAsk = setupContainerAskForRM(); > amRMClient.addContainerRequest(containerAsk); > } > numRequestedContainers.set(numTotalContainersToRequest); > {code} > numRequestedContainers doesn't account for previous AM's requested > containers. so numRequestedContainers should be set to numTotalContainers -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2354) DistributedShell may allocate more containers than client specified after it restarts
[ https://issues.apache.org/jira/browse/YARN-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14078851#comment-14078851 ] Li Lu commented on YARN-2354: - Thanks [~jianhe]. In the second time I think the only problem left is the persistent one, but unrelated to this patch. > DistributedShell may allocate more containers than client specified after it > restarts > - > > Key: YARN-2354 > URL: https://issues.apache.org/jira/browse/YARN-2354 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Li Lu > Attachments: YARN-2354-072514.patch, YARN-2354-072814.patch, > YARN-2354-072914.patch > > > To reproduce, run distributed shell with -num_containers option, > In ApplicationMaster.java, the following code has some issue. > {code} > int numTotalContainersToRequest = > numTotalContainers - previousAMRunningContainers.size(); > for (int i = 0; i < numTotalContainersToRequest; ++i) { > ContainerRequest containerAsk = setupContainerAskForRM(); > amRMClient.addContainerRequest(containerAsk); > } > numRequestedContainers.set(numTotalContainersToRequest); > {code} > numRequestedContainers doesn't account for previous AM's requested > containers. so numRequestedContainers should be set to numTotalContainers -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2354) DistributedShell may allocate more containers than client specified after it restarts
[ https://issues.apache.org/jira/browse/YARN-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14078849#comment-14078849 ] Jian He commented on YARN-2354: --- looks good, checking this in. > DistributedShell may allocate more containers than client specified after it > restarts > - > > Key: YARN-2354 > URL: https://issues.apache.org/jira/browse/YARN-2354 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Li Lu > Attachments: YARN-2354-072514.patch, YARN-2354-072814.patch, > YARN-2354-072914.patch > > > To reproduce, run distributed shell with -num_containers option, > In ApplicationMaster.java, the following code has some issue. > {code} > int numTotalContainersToRequest = > numTotalContainers - previousAMRunningContainers.size(); > for (int i = 0; i < numTotalContainersToRequest; ++i) { > ContainerRequest containerAsk = setupContainerAskForRM(); > amRMClient.addContainerRequest(containerAsk); > } > numRequestedContainers.set(numTotalContainersToRequest); > {code} > numRequestedContainers doesn't account for previous AM's requested > containers. so numRequestedContainers should be set to numTotalContainers -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2354) DistributedShell may allocate more containers than client specified after it restarts
[ https://issues.apache.org/jira/browse/YARN-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14078740#comment-14078740 ] Hadoop QA commented on YARN-2354: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12658544/YARN-2354-072914.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell: org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4476//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4476//console This message is automatically generated. > DistributedShell may allocate more containers than client specified after it > restarts > - > > Key: YARN-2354 > URL: https://issues.apache.org/jira/browse/YARN-2354 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Li Lu > Attachments: YARN-2354-072514.patch, YARN-2354-072814.patch, > YARN-2354-072914.patch > > > To reproduce, run distributed shell with -num_containers option, > In ApplicationMaster.java, the following code has some issue. > {code} > int numTotalContainersToRequest = > numTotalContainers - previousAMRunningContainers.size(); > for (int i = 0; i < numTotalContainersToRequest; ++i) { > ContainerRequest containerAsk = setupContainerAskForRM(); > amRMClient.addContainerRequest(containerAsk); > } > numRequestedContainers.set(numTotalContainersToRequest); > {code} > numRequestedContainers doesn't account for previous AM's requested > containers. so numRequestedContainers should be set to numTotalContainers -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2354) DistributedShell may allocate more containers than client specified after it restarts
[ https://issues.apache.org/jira/browse/YARN-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14078648#comment-14078648 ] Jian He commented on YARN-2354: --- Hi Li, the test failures are different from previous one, can you make sure these failures are not related to this patch ? th! > DistributedShell may allocate more containers than client specified after it > restarts > - > > Key: YARN-2354 > URL: https://issues.apache.org/jira/browse/YARN-2354 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Li Lu > Attachments: YARN-2354-072514.patch, YARN-2354-072814.patch > > > To reproduce, run distributed shell with -num_containers option, > In ApplicationMaster.java, the following code has some issue. > {code} > int numTotalContainersToRequest = > numTotalContainers - previousAMRunningContainers.size(); > for (int i = 0; i < numTotalContainersToRequest; ++i) { > ContainerRequest containerAsk = setupContainerAskForRM(); > amRMClient.addContainerRequest(containerAsk); > } > numRequestedContainers.set(numTotalContainersToRequest); > {code} > numRequestedContainers doesn't account for previous AM's requested > containers. so numRequestedContainers should be set to numTotalContainers -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2354) DistributedShell may allocate more containers than client specified after it restarts
[ https://issues.apache.org/jira/browse/YARN-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14077279#comment-14077279 ] Hadoop QA commented on YARN-2354: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12658299/YARN-2354-072814.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell: org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4464//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4464//console This message is automatically generated. > DistributedShell may allocate more containers than client specified after it > restarts > - > > Key: YARN-2354 > URL: https://issues.apache.org/jira/browse/YARN-2354 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Li Lu > Attachments: YARN-2354-072514.patch, YARN-2354-072814.patch > > > To reproduce, run distributed shell with -num_containers option, > In ApplicationMaster.java, the following code has some issue. > {code} > int numTotalContainersToRequest = > numTotalContainers - previousAMRunningContainers.size(); > for (int i = 0; i < numTotalContainersToRequest; ++i) { > ContainerRequest containerAsk = setupContainerAskForRM(); > amRMClient.addContainerRequest(containerAsk); > } > numRequestedContainers.set(numTotalContainersToRequest); > {code} > numRequestedContainers doesn't account for previous AM's requested > containers. so numRequestedContainers should be set to numTotalContainers -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2354) DistributedShell may allocate more containers than client specified after it restarts
[ https://issues.apache.org/jira/browse/YARN-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14076439#comment-14076439 ] Li Lu commented on YARN-2354: - Same error message as YARN-2295, and could not reproduce locally. Seems like this is connected with the network settings of the server, causing the following lines to fail {code} if (appReport.getHost().startsWith(hostName) && appReport.getRpcPort() == -1) { verified = true; } {code} If such check failed, verified will never be set to true, hence the test will fail. This failure appears to be unrelated to the problem fixed by this patch. > DistributedShell may allocate more containers than client specified after it > restarts > - > > Key: YARN-2354 > URL: https://issues.apache.org/jira/browse/YARN-2354 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Li Lu > Attachments: YARN-2354-072514.patch > > > To reproduce, run distributed shell with -num_containers option, > In ApplicationMaster.java, the following code has some issue. > {code} > int numTotalContainersToRequest = > numTotalContainers - previousAMRunningContainers.size(); > for (int i = 0; i < numTotalContainersToRequest; ++i) { > ContainerRequest containerAsk = setupContainerAskForRM(); > amRMClient.addContainerRequest(containerAsk); > } > numRequestedContainers.set(numTotalContainersToRequest); > {code} > numRequestedContainers doesn't account for previous AM's requested > containers. so numRequestedContainers should be set to numTotalContainers -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2354) DistributedShell may allocate more containers than client specified after it restarts
[ https://issues.apache.org/jira/browse/YARN-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14075224#comment-14075224 ] Hadoop QA commented on YARN-2354: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12657943/YARN-2354-072514.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell: org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build///testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build///console This message is automatically generated. > DistributedShell may allocate more containers than client specified after it > restarts > - > > Key: YARN-2354 > URL: https://issues.apache.org/jira/browse/YARN-2354 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Li Lu > Attachments: YARN-2354-072514.patch > > > To reproduce, run distributed shell with -num_containers option, > In ApplicationMaster.java, the following code has some issue. > {code} > int numTotalContainersToRequest = > numTotalContainers - previousAMRunningContainers.size(); > for (int i = 0; i < numTotalContainersToRequest; ++i) { > ContainerRequest containerAsk = setupContainerAskForRM(); > amRMClient.addContainerRequest(containerAsk); > } > numRequestedContainers.set(numTotalContainersToRequest); > {code} > numRequestedContainers doesn't account for previous AM's requested > containers. so numRequestedContainers should be set to numTotalContainers -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2354) DistributedShell may allocate more containers than client specified after it restarts
[ https://issues.apache.org/jira/browse/YARN-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14075165#comment-14075165 ] Jian He commented on YARN-2354: --- looks good. thanks for working on the patch Li ! > DistributedShell may allocate more containers than client specified after it > restarts > - > > Key: YARN-2354 > URL: https://issues.apache.org/jira/browse/YARN-2354 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Li Lu > Attachments: YARN-2354-072514.patch > > > To reproduce, run distributed shell with -num_containers option, > In ApplicationMaster.java, the following code has some issue. > {code} > int numTotalContainersToRequest = > numTotalContainers - previousAMRunningContainers.size(); > for (int i = 0; i < numTotalContainersToRequest; ++i) { > ContainerRequest containerAsk = setupContainerAskForRM(); > amRMClient.addContainerRequest(containerAsk); > } > numRequestedContainers.set(numTotalContainersToRequest); > {code} > numRequestedContainers doesn't account for previous AM's requested > containers. so numRequestedContainers should be set to numTotalContainers -- This message was sent by Atlassian JIRA (v6.2#6252)