[jira] [Commented] (YARN-660) Improve AMRMClient with matching requests
[ https://issues.apache.org/jira/browse/YARN-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13672112#comment-13672112 ] Hudson commented on YARN-660: - Integrated in Hadoop-Mapreduce-trunk #1443 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1443/]) YARN-660. Improve AMRMClient with matching requests (bikas) (Revision 1488485) Result = SUCCESS bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1488485 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestNMClient.java > Improve AMRMClient with matching requests > - > > Key: YARN-660 > URL: https://issues.apache.org/jira/browse/YARN-660 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Bikas Saha > Fix For: 2.1.0-beta > > Attachments: YARN-660.1.patch, YARN-660.2.patch, YARN-660.3.patch, > YARN-660.4.patch, YARN-660.5.patch, YARN-660.6.patch, YARN-660.7.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-660) Improve AMRMClient with matching requests
[ https://issues.apache.org/jira/browse/YARN-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13672093#comment-13672093 ] Hudson commented on YARN-660: - Integrated in Hadoop-Hdfs-trunk #1417 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1417/]) YARN-660. Improve AMRMClient with matching requests (bikas) (Revision 1488485) Result = FAILURE bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1488485 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestNMClient.java > Improve AMRMClient with matching requests > - > > Key: YARN-660 > URL: https://issues.apache.org/jira/browse/YARN-660 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Bikas Saha > Fix For: 2.1.0-beta > > Attachments: YARN-660.1.patch, YARN-660.2.patch, YARN-660.3.patch, > YARN-660.4.patch, YARN-660.5.patch, YARN-660.6.patch, YARN-660.7.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-660) Improve AMRMClient with matching requests
[ https://issues.apache.org/jira/browse/YARN-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13672048#comment-13672048 ] Hudson commented on YARN-660: - Integrated in Hadoop-Yarn-trunk #227 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/227/]) YARN-660. Improve AMRMClient with matching requests (bikas) (Revision 1488485) Result = SUCCESS bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1488485 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestNMClient.java > Improve AMRMClient with matching requests > - > > Key: YARN-660 > URL: https://issues.apache.org/jira/browse/YARN-660 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Bikas Saha > Fix For: 2.1.0-beta > > Attachments: YARN-660.1.patch, YARN-660.2.patch, YARN-660.3.patch, > YARN-660.4.patch, YARN-660.5.patch, YARN-660.6.patch, YARN-660.7.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-660) Improve AMRMClient with matching requests
[ https://issues.apache.org/jira/browse/YARN-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13672021#comment-13672021 ] Hudson commented on YARN-660: - Integrated in Hadoop-trunk-Commit #3827 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3827/]) YARN-660. Improve AMRMClient with matching requests (bikas) (Revision 1488485) Result = SUCCESS bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1488485 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestNMClient.java > Improve AMRMClient with matching requests > - > > Key: YARN-660 > URL: https://issues.apache.org/jira/browse/YARN-660 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Bikas Saha > Fix For: 2.1.0-beta > > Attachments: YARN-660.1.patch, YARN-660.2.patch, YARN-660.3.patch, > YARN-660.4.patch, YARN-660.5.patch, YARN-660.6.patch, YARN-660.7.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-660) Improve AMRMClient with matching requests
[
https://issues.apache.org/jira/browse/YARN-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13672016#comment-13672016
]
Hadoop QA commented on YARN-660:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12585709/YARN-660.7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1062//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1062//console
This message is automatically generated.
> Improve AMRMClient with matching requests
> -
>
> Key: YARN-660
> URL: https://issues.apache.org/jira/browse/YARN-660
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 2.1.0-beta
>Reporter: Bikas Saha
>Assignee: Bikas Saha
> Fix For: 2.1.0-beta
>
> Attachments: YARN-660.1.patch, YARN-660.2.patch, YARN-660.3.patch,
> YARN-660.4.patch, YARN-660.5.patch, YARN-660.6.patch, YARN-660.7.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-660) Improve AMRMClient with matching requests
[ https://issues.apache.org/jira/browse/YARN-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668714#comment-13668714 ] Hitesh Shah commented on YARN-660: -- +1. Mostly looks good. Minor nit: Should we be logging the InterruptedException at WARN level when shutting down? Potentially gives the user a wrong impression that there was a problem in shutdown? > Improve AMRMClient with matching requests > - > > Key: YARN-660 > URL: https://issues.apache.org/jira/browse/YARN-660 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.0.5-beta >Reporter: Bikas Saha >Assignee: Bikas Saha > Fix For: 2.0.5-beta > > Attachments: YARN-660.1.patch, YARN-660.2.patch, YARN-660.3.patch, > YARN-660.4.patch, YARN-660.5.patch, YARN-660.6.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-660) Improve AMRMClient with matching requests
[
https://issues.apache.org/jira/browse/YARN-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668703#comment-13668703
]
Hadoop QA commented on YARN-660:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12585086/YARN-660.6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1012//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1012//console
This message is automatically generated.
> Improve AMRMClient with matching requests
> -
>
> Key: YARN-660
> URL: https://issues.apache.org/jira/browse/YARN-660
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 2.0.5-beta
>Reporter: Bikas Saha
>Assignee: Bikas Saha
> Fix For: 2.0.5-beta
>
> Attachments: YARN-660.1.patch, YARN-660.2.patch, YARN-660.3.patch,
> YARN-660.4.patch, YARN-660.5.patch, YARN-660.6.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-660) Improve AMRMClient with matching requests
[
https://issues.apache.org/jira/browse/YARN-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13665931#comment-13665931
]
Hadoop QA commented on YARN-660:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584628/YARN-660.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1001//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1001//console
This message is automatically generated.
> Improve AMRMClient with matching requests
> -
>
> Key: YARN-660
> URL: https://issues.apache.org/jira/browse/YARN-660
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 2.0.5-beta
>Reporter: Bikas Saha
>Assignee: Bikas Saha
> Fix For: 2.0.5-beta
>
> Attachments: YARN-660.1.patch, YARN-660.2.patch, YARN-660.3.patch,
> YARN-660.4.patch, YARN-660.5.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-660) Improve AMRMClient with matching requests
[ https://issues.apache.org/jira/browse/YARN-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13665918#comment-13665918 ] Bikas Saha commented on YARN-660: - New patch removes cookies from the code. Title updated to reflect that. Additionally, it fixes synchronization issue in AMRMClientAsync. TestAMRMClientAsync verifies fix. The other main change is to AMRMClient.getMatchingContainers(). The old impl was incorrectly assuming that the RM will return containers with exact resource matching. However, the RM can return containers that are larger (because of normalization etc). Container may be larger by memory or cpu (e.g. container (1000M, 3cpu) can fit request (1000M, 1cpu) or request (500M, 3cpu). Only the user can decide which one matches better. Thus requests for different resource capabilities must be returned to the user. Tests improved to show lifecycle of using storedcontainerrequests per previous comment. Tests added to verify new matching functionality. > Improve AMRMClient with matching requests > - > > Key: YARN-660 > URL: https://issues.apache.org/jira/browse/YARN-660 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.0.5-beta >Reporter: Bikas Saha >Assignee: Bikas Saha > Fix For: 2.0.5-beta > > Attachments: YARN-660.1.patch, YARN-660.2.patch, YARN-660.3.patch, > YARN-660.4.patch, YARN-660.5.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
