[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-17 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13685445#comment-13685445
 ] 

Hudson commented on YARN-752:
-

Integrated in Hadoop-Yarn-trunk #243 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/243/])
YARN-752. In AMRMClient, automatically add corresponding rack requests for 
requested nodes. (sandyr via tucu) (Revision 1493599)

 Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1493599
Files : 
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClientContainerRequest.java


 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Fix For: 2.1.0-beta

 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, 
 YARN-752.3.patch, YARN-752.4.patch, YARN-752-5.patch, YARN-752-6.patch, 
 YARN-752-7.patch, YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-17 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13685531#comment-13685531
 ] 

Hudson commented on YARN-752:
-

Integrated in Hadoop-Hdfs-trunk #1433 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1433/])
YARN-752. In AMRMClient, automatically add corresponding rack requests for 
requested nodes. (sandyr via tucu) (Revision 1493599)

 Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1493599
Files : 
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClientContainerRequest.java


 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Fix For: 2.1.0-beta

 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, 
 YARN-752.3.patch, YARN-752.4.patch, YARN-752-5.patch, YARN-752-6.patch, 
 YARN-752-7.patch, YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-17 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13685585#comment-13685585
 ] 

Hudson commented on YARN-752:
-

Integrated in Hadoop-Mapreduce-trunk #1460 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1460/])
YARN-752. In AMRMClient, automatically add corresponding rack requests for 
requested nodes. (sandyr via tucu) (Revision 1493599)

 Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1493599
Files : 
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClientContainerRequest.java


 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Fix For: 2.1.0-beta

 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, 
 YARN-752.3.patch, YARN-752.4.patch, YARN-752-5.patch, YARN-752-6.patch, 
 YARN-752-7.patch, YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-16 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13684824#comment-13684824
 ] 

Hudson commented on YARN-752:
-

Integrated in Hadoop-trunk-Commit #3942 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/3942/])
YARN-752. In AMRMClient, automatically add corresponding rack requests for 
requested nodes. (sandyr via tucu) (Revision 1493599)

 Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1493599
Files : 
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClientContainerRequest.java


 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Fix For: 2.1.0-beta

 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, 
 YARN-752.3.patch, YARN-752.4.patch, YARN-752-5.patch, YARN-752-6.patch, 
 YARN-752-7.patch, YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-14 Thread Alejandro Abdelnur (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13683566#comment-13683566
 ] 

Alejandro Abdelnur commented on YARN-752:
-

patch is not applying anymore, some changes that went in AMRMClientImpl.java 
are breaking it. Mind rebasing?

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, 
 YARN-752.3.patch, YARN-752.4.patch, YARN-752-5.patch, YARN-752-6.patch, 
 YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-14 Thread Vinod Kumar Vavilapalli (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13683695#comment-13683695
 ] 

Vinod Kumar Vavilapalli commented on YARN-752:
--

bq. Some additional explanation will be needed, but in general, I don't think 
all schedulers will be required to handle the relaxLocality flag.
Not blocking this patch, but that doesn't seem right. All schedulers MUST 
handle it. AMs shouldn't be burdened with coding for a specific scheduler.

Anyways, Sandy, can you please update it? I'd like to get this in for 2.1.0 
that'll happen sometime soon.

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, 
 YARN-752.3.patch, YARN-752.4.patch, YARN-752-5.patch, YARN-752-6.patch, 
 YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-14 Thread Sandy Ryza (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13683703#comment-13683703
 ] 

Sandy Ryza commented on YARN-752:
-

Attached a rebased patch.

bq. Not blocking this patch, but that doesn't seem right. All schedulers MUST 
handle it. AMs shouldn't be burdened with coding for a specific scheduler.

OK, making this required sounds reasonable to me.  We can continue the 
discussion on YARN-521?

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, 
 YARN-752.3.patch, YARN-752.4.patch, YARN-752-5.patch, YARN-752-6.patch, 
 YARN-752-7.patch, YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-14 Thread Bikas Saha (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13683817#comment-13683817
 ] 

Bikas Saha commented on YARN-752:
-

Probably just refer to the javadoc for AllocateRequest/ResourceRequest where 
the semantics are defined.

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, 
 YARN-752.3.patch, YARN-752.4.patch, YARN-752-5.patch, YARN-752-6.patch, 
 YARN-752-7.patch, YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-12 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13681581#comment-13681581
 ] 

Hadoop QA commented on YARN-752:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12587497/YARN-752-6.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/1204//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1204//console

This message is automatically generated.

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, 
 YARN-752.3.patch, YARN-752.4.patch, YARN-752-5.patch, YARN-752-6.patch, 
 YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-11 Thread Bikas Saha (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13680953#comment-13680953
 ] 

Bikas Saha commented on YARN-752:
-

Would be great if you could also rename ContainerRequest.getHosts() to 
getNodes() and make it consistent with YARN's use of node for machine.
Since I made a non-trivial change to the patch, some other committer probably 
needs to +1 and commit the patch.

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, 
 YARN-752.3.patch, YARN-752.4.patch, YARN-752-5.patch, YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-10 Thread Sandy Ryza (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679899#comment-13679899
 ] 

Sandy Ryza commented on YARN-752:
-

[~bikassaha], your updated patch looks good to me.  Noticed a minor spacing 
issue that originated in my code, uploading a patch to fix it.

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, 
 YARN-752.3.patch, YARN-752.4.patch, YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-10 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679915#comment-13679915
 ] 

Hadoop QA commented on YARN-752:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12587130/YARN-752-5.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/1181//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1181//console

This message is automatically generated.

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, 
 YARN-752.3.patch, YARN-752.4.patch, YARN-752-5.patch, YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-09 Thread Bikas Saha (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679235#comment-13679235
 ] 

Bikas Saha commented on YARN-752:
-

Its not 100% clear what the API should be on the AMRMClient for specific 
requests. A set of checks need to be made so that unsatisfiable requests cannot 
be made or illegal mix-n-match cannot happen. The server side impl is simple 
and does no checks for the sake of overhead reduction.
Regardless, given the protocol API that requires relaxLocality flag to be 
specified on racks for the specific nodes, the resolution of nodes to racks is 
required, even in that case. This jira is simply for the addition of rack 
resolution functionality to the AMRMClient. Its current use is to make sure all 
nodes are resolved to their racks or else those requests will be unsatisfiable 
on the server side, resulting in correctness issues.

The patch by itself has a bug. After a container request has been removed, it 
should not be returned via getMatchingRequests() for the inferred rack. i.e. 
the patch was doing addResourceRequest() for the inferred rack but not doing 
the corresponding removeResourceRequest(). The simple fix is to resolve/dedupe 
not only during addContainerRequest() but also during removeContainerRequest(). 
Attaching patch with simple fix and minor changes to the javadoc.

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, 
 YARN-752.3.patch, YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-09 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679241#comment-13679241
 ] 

Hadoop QA commented on YARN-752:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12587002/YARN-752.3.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client:

  org.apache.hadoop.yarn.client.TestAMRMClient

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/1174//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1174//console

This message is automatically generated.

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, 
 YARN-752.3.patch, YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-09 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679254#comment-13679254
 ] 

Hadoop QA commented on YARN-752:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12587008/YARN-752.4.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/1175//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1175//console

This message is automatically generated.

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, 
 YARN-752.3.patch, YARN-752.4.patch, YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-07 Thread Siddharth Seth (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678228#comment-13678228
 ] 

Siddharth Seth commented on YARN-752:
-

Can we make this rack resolution optional
1) If the user has provided racks - just trust that information instead of 
resolving and then checking whether the user has provided the correct 
information.
2) Does it make sense to have an option to disable this automatic resolution. I 
haven't been following the whitelisting / blacklisting closely - but isn't it 
possible for users to ask for specific hosts, without specifying racks 
(Ignoring scheduler implementation here)

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-07 Thread Sandy Ryza (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678270#comment-13678270
 ] 

Sandy Ryza commented on YARN-752:
-

bq. If the user has provided racks - just trust that information instead of 
resolving and then checking whether the user has provided the correct 
information.
If we do this, I think it would be good at least to log some sort of warnings 
when the given and expected racks don't match up. The consequences of a user 
getting it wrong can be pretty subtle.

bq.  but isn't it possible for users to ask for specific hosts, without 
specifying racks (Ignoring scheduler implementation here)
Currently, it does not make sense for users to ask for specific hosts without 
specifying racks in any scheduler implementation.  Containers may still be 
scheduled on those nodes, but they will be treated like any other non-local 
nodes for the purposes of delay-scheduling and locality-specific requests. I 
believe that this is not just an implementation issue, but a result of the 
protocol.  A scheduler having zero containers requested on rack1, but non-zero 
containers requested on nodes on rack1, can occur when a single container was 
requested for any of multiple nodes on rack1.



 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-07 Thread Bikas Saha (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678273#comment-13678273
 ] 

Bikas Saha commented on YARN-752:
-

YARN's default and most efficient scheduling behavior comes when locality can 
relaxed from node to rack etc. The AMRMClient should simply do the right thing 
in this context and add racks for the nodes provided by the user, if the user 
has already not done so. If this is not done then its the burden of the user to 
do it and if the user forgets to do it then YARN will probably not give it any 
containers and the job will be stuck. Hence, by default the AMRMClient should 
always fill in missing racks. If the user has already done the right thing then 
this functionality will be a no-op. This cannot be optional. The only time this 
should not be done is when the user has asked for specific nodes/racks. Even 
there the AMRMClient will have to add missing racks but with a special flag or 
else the schedulers will not assign containers to the app. Support for that 
feature is yet to be added in AMRMClient and is tracked by another jira. 
If the user has specified a mix of nodes and racks, even then we need to ensure 
that racks are added for nodes whose racks are missing.

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-07 Thread Bikas Saha (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678275#comment-13678275
 ] 

Bikas Saha commented on YARN-752:
-

The patch looks good to me as is. Need to do a final review.

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-07 Thread Alejandro Abdelnur (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678401#comment-13678401
 ] 

Alejandro Abdelnur commented on YARN-752:
-

agree with [~bikassaha]'s, only question left, what if the user provides wrong 
rack for node? As Sandy mentioned, that could lead to odd/subtle behavior.

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-07 Thread Bikas Saha (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678405#comment-13678405
 ] 

Bikas Saha commented on YARN-752:
-

User cannot provide a wrong rack for a node. User can provide a wrong rack by 
itself. If a rack is missing then it will be corrected by AMRMClient. If there 
is another different rack, its either a valid rack or a rack that does not 
exist. We should probably resolve nodes and racks to make sure they are all 
valid locations. Different jira perhaps. This kind of depends on YARN-435.

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-07 Thread Alejandro Abdelnur (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678418#comment-13678418
 ] 

Alejandro Abdelnur commented on YARN-752:
-

Ah!, ok, so before this patch if I would do a request of only NODES allocations 
would never happen. I think we should clearly document in the API that 
specifying RACKS is only needed if you want an allocation in any node of that 
RACK, if you want the allocation in a NODE you don't need to specify the rack 
the node is.

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-07 Thread Siddharth Seth (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678445#comment-13678445
 ] 

Siddharth Seth commented on YARN-752:
-

Alright - I guess the node only request will be handled in the other jira to 
track it. And from Bikas' explanation - that'll require rack resolution in any 
case. Seems a little strange, but required based on scheduler implementation.

If we're always doing the rack lookup, we should change the MR AM avoid rack 
lookups if possible - may be difficult considering it tracks RACK_LOCAL 
allocations. Separate jira though.

The rack lookup in the patch itself, shouldn't it be adding a rack only once.
e.g. ResourceRequest: h1, h2, h3 - numContainers=1
If all of them resolve to the same rack - num containers on the rack would go 
to 3 after this ? Is that correct behaviour.

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-07 Thread Sandy Ryza (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678468#comment-13678468
 ] 

Sandy Ryza commented on YARN-752:
-

bq. The rack lookup in the patch itself, shouldn't it be adding a rack only 
once.
You're right.  I'll fix this.

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-07 Thread Bikas Saha (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678470#comment-13678470
 ] 

Bikas Saha commented on YARN-752:
-

The user needs to only specify a node (and may end up getting a container on 
that node, on that node's rack or somewhere else. There is no need to specify 
the rack of a node explicitly but users may do so. If a user wants any node in 
a set of racks then they should simply directly specify those racks.

This code is not quite correct. If nodes n1, n2, n3 (all in rack1) are present 
then this will end up calling addResourceRequest 3 times for rack1. Should 
probably create a local set of racks, initialized by CR.racks, add nodes' racks 
to the set. Then call addResourceRequest for the rack set.
{code}
+String rack = RackResolver.resolve(host).getNetworkLocation();
+if (rack == null) {
+  LOG.warn(Failed to resolve rack for host  + host + .);
+} else if (req.racks == null || !req.racks.contains(rack)) {
+  addResourceRequest(req.priority, rack, req.capability,
+  req.containerCount, req);
+}
{code}

The test needs to improve to catch the above case. Also, should check container 
count for the asks to ensure that counts are correct (with or without user 
specifying the racks). The test is also calling api's without calling start(). 
It works now, but will probably break later once we enforce the life cycle 
inside the client. Things like rack resolution etc probably depend on topoplogy 
information coming from the RM upon registration.

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-07 Thread Bikas Saha (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678472#comment-13678472
 ] 

Bikas Saha commented on YARN-752:
-

Would be great if the next patch added documentation requested by [~tucu00]

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-07 Thread Sandy Ryza (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678535#comment-13678535
 ] 

Sandy Ryza commented on YARN-752:
-

Uploading a patch that modifies the test the catch the issue identified by 
Siddharth, fixes it, and adds the documentation requested by Tucu

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, 
 YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-07 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678557#comment-13678557
 ] 

Hadoop QA commented on YARN-752:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12586821/YARN-752-2.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/1164//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1164//console

This message is automatically generated.

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, 
 YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-07 Thread Vinod Kumar Vavilapalli (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678561#comment-13678561
 ] 

Vinod Kumar Vavilapalli commented on YARN-752:
--

Haven't been paying attention. But quickly looked through.
 - I think if the user gives a wrong rack, we should throw an error.
 - IIUC, this code will change after YARN-521 or is YARN-521 just a new API 
instead of changes to the current API with optional flags?
 - In any case, the algo should eventually look like
{code}
For each explicitly added rack, if it doesn't exist, throw error;
if (asking for specific nodes) {
  don't add racks
} else {
  add racks
}
{code}
 - Either ways I think YARN-521 is a priority, depend on how the API is 
designed, it can result in API signature changes.

If you agree that the logic is based on enabling/disabling strict allocations, 
the following in the patch won't be correct any longer, right?
{code}
Scheduler
+   * documentation should be consulted for the specifics of how the parameters
+   * are honored.
{code}

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, 
 YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-07 Thread Alejandro Abdelnur (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678578#comment-13678578
 ] 

Alejandro Abdelnur commented on YARN-752:
-

bq. I think if the user gives a wrong rack, we should throw an error.

Per previous discussion this does not happen (except if you are referring to an 
invalid rack altogether). If a ContainerRequest has host1 and rack3, it means 
that the containers are desired either in the host1 or rack3. it does not mean 
that host1 is in rack3.


 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, 
 YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-07 Thread Sandy Ryza (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678613#comment-13678613
 ] 

Sandy Ryza commented on YARN-752:
-

In my vision of YARN-521, we will add an additional boolean parameter to the 
ContainerRequest constructor, relaxLocality.  If a ContainerRequest's 
relaxLocality is false, and both nodes and racks are passed in to a 
ContainerRequest, we have a few options for how to handle it:
* Set relaxLocality to false only for all non-explicit ResourceRequests.  This 
would mean that if I submit a ContainerRequest with both nodes and the racks 
that those nodes are on, containers could be given on any nodes on those racks. 
* If there are any node requests, set relaxLocality to false for all rack 
requests that those nodes are on, even if the racks are explicitly given.  This 
would mean that if I submit a request with a node, the rack it is on, and 
another rack, containers could be given on that node or on any node on the 
other rack.
* If there are any node requests, set relaxLocality to false for all rack 
requests.  This would mean that if I submit a request with a node, the rack it 
is on, and another rack, containers could be given only on the node.

bq. If you agree that the logic is based on enabling/disabling strict 
allocations, the following in the patch won't be correct any longer, right?
Some additional explanation will be needed, but in general, I don't think all 
schedulers will be required to handle the relaxLocality flag.

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, 
 YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-06 Thread Bikas Saha (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13677287#comment-13677287
 ] 

Bikas Saha commented on YARN-752:
-

First of all I think I made a mistake in putting ImmutableList as part of the 
signature of the getHosts() etc. It ties down the implementation vs having 
returned a generic list. We should probably fix that. I can do that in a 
different jira. We should be storing a simple memory efficient list and 
returning an unmodifiablelist instead of ImmutableList.

Changing the storage type to set is not desired. The set is only needed 1 time 
for ease of lookup(and probably isnt much different in perf vs linear search 
for single digit sizes). Lets not change the actual storage to a set because 
its just more memory (for millions of container request) and it also loses the 
ordering given by the user. At some point we may want to match requests in the 
preferred order given by the user.

What if the RackResolver is improperly initialized because of a bad config? And 
does not return valid/non-null values? Can you please check?
{code}
+String rack = RackResolver.resolve(host).getNetworkLocation();
+if (!req.racks.contains(rack)) {
{code}

The test could probably go into the existing tests for AMRMClient so that it 
stays in one place. Also, would be great to make sure that 
getMatchingRequests() work properly in this case - user should get the 
container request for a rack for which they had originally specified only 
nodes. They should but a test would verify the expected behavior.

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-06 Thread Sandy Ryza (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13677354#comment-13677354
 ] 

Sandy Ryza commented on YARN-752:
-

bq. Changing the storage type to set is not desired.
Makes sense to me.  I'll revert that part.

bq. What if the RackResolver is improperly initialized
It looks like TableMapping returns NetworkTopology.DEFAULT_RACK whenever it 
can't find a node, but ScriptBasedMapping returns null when it encounters 
problems.  I'll modify the patch to log something and not try to file a 
ResourceRequest with a null rack.

bq. The test could probably go into the existing tests for AMRMClient
As unit tests, it seems to me that the closer we can hone in on the 
functionality being tested the better. Wrapping the test in a MiniYARNCluster 
and other structures used for the AMRMClient lifecycle testing in 
TestAMRMClient will make debugging this specific functionality more difficult 
in the future.

bq. Also, would be great to make sure that getMatchingRequests() work properly 
in this case
I'll try to write a test that verifies this

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-06 Thread Bikas Saha (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13677373#comment-13677373
 ] 

Bikas Saha commented on YARN-752:
-

bq. Wrapping the test in a MiniYARNCluster and other structures used for the 
AMRMClient lifecycle testing in TestAMRMClient will make debugging this 
specific functionality more difficult in the future.
Thats fine. Can that not be done within the existing file TestAMRMClient.java 
instead of creating a new one? Just helps to keep all related tests in one 
place. Not a big deal if thats not simple.

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-06 Thread Sandy Ryza (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13677414#comment-13677414
 ] 

Sandy Ryza commented on YARN-752:
-

Uploading a patch that addresses Bikas's comments.

bq. Can that not be done within the existing file TestAMRMClient.java instead 
of creating a new one?
The only ways I can think of to do it seem a little ugly to me, but I'm open to 
suggestions.


 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-06 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13677425#comment-13677425
 ] 

Hadoop QA commented on YARN-752:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12586559/YARN-752-1.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client:

  org.apache.hadoop.yarn.client.TestNMClientAsync

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/1145//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1145//console

This message is automatically generated.

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752-1.patch, YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-06 Thread Sandy Ryza (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13677479#comment-13677479
 ] 

Sandy Ryza commented on YARN-752:
-

I'm unable to reproduce the failure locally, and don't see how the changes 
could have caused it.  I'm going to try re-uploading.

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-06 Thread Bikas Saha (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13677482#comment-13677482
 ] 

Bikas Saha commented on YARN-752:
-

Its a known issue. YARN-761

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-06 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13677492#comment-13677492
 ] 

Hadoop QA commented on YARN-752:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12586579/YARN-752-1.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/1146//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1146//console

This message is automatically generated.

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-03 Thread Sandy Ryza (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673526#comment-13673526
 ] 

Sandy Ryza commented on YARN-752:
-

Cool, I think that option is the best too.  Updated the title and description.

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza

 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-03 Thread Sandy Ryza (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673577#comment-13673577
 ] 

Sandy Ryza commented on YARN-752:
-

[~bikassaha], do you have an opinion on whether it makes more sense to add the 
missing racks when a ContainerRequest is constructed or when it's submitted to 
the AMRMClient?

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza

 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-03 Thread Sandy Ryza (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673859#comment-13673859
 ] 

Sandy Ryza commented on YARN-752:
-

Attached a patch that resolves racks in the AMRMClient

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes

2013-06-03 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673879#comment-13673879
 ] 

Hadoop QA commented on YARN-752:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12586005/YARN-752.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/1094//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1094//console

This message is automatically generated.

 In AMRMClient, automatically add corresponding rack requests for requested 
 nodes
 

 Key: YARN-752
 URL: https://issues.apache.org/jira/browse/YARN-752
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-752.patch


 A ContainerRequest that includes node-level requests must also include 
 matching rack-level requests for the racks that those nodes are on.  When a 
 node is present without its rack, it makes sense for the client to 
 automatically add the node's rack.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira