[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[ https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13685445#comment-13685445 ] Hudson commented on YARN-752: - Integrated in Hadoop-Yarn-trunk #243 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/243/]) YARN-752. In AMRMClient, automatically add corresponding rack requests for requested nodes. (sandyr via tucu) (Revision 1493599) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1493599 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClientContainerRequest.java In AMRMClient, automatically add corresponding rack requests for requested nodes Key: YARN-752 URL: https://issues.apache.org/jira/browse/YARN-752 Project: Hadoop YARN Issue Type: Improvement Components: api, applications Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.1.0-beta Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, YARN-752.3.patch, YARN-752.4.patch, YARN-752-5.patch, YARN-752-6.patch, YARN-752-7.patch, YARN-752.patch A ContainerRequest that includes node-level requests must also include matching rack-level requests for the racks that those nodes are on. When a node is present without its rack, it makes sense for the client to automatically add the node's rack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[ https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13685531#comment-13685531 ] Hudson commented on YARN-752: - Integrated in Hadoop-Hdfs-trunk #1433 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1433/]) YARN-752. In AMRMClient, automatically add corresponding rack requests for requested nodes. (sandyr via tucu) (Revision 1493599) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1493599 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClientContainerRequest.java In AMRMClient, automatically add corresponding rack requests for requested nodes Key: YARN-752 URL: https://issues.apache.org/jira/browse/YARN-752 Project: Hadoop YARN Issue Type: Improvement Components: api, applications Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.1.0-beta Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, YARN-752.3.patch, YARN-752.4.patch, YARN-752-5.patch, YARN-752-6.patch, YARN-752-7.patch, YARN-752.patch A ContainerRequest that includes node-level requests must also include matching rack-level requests for the racks that those nodes are on. When a node is present without its rack, it makes sense for the client to automatically add the node's rack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[ https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13685585#comment-13685585 ] Hudson commented on YARN-752: - Integrated in Hadoop-Mapreduce-trunk #1460 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1460/]) YARN-752. In AMRMClient, automatically add corresponding rack requests for requested nodes. (sandyr via tucu) (Revision 1493599) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1493599 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClientContainerRequest.java In AMRMClient, automatically add corresponding rack requests for requested nodes Key: YARN-752 URL: https://issues.apache.org/jira/browse/YARN-752 Project: Hadoop YARN Issue Type: Improvement Components: api, applications Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.1.0-beta Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, YARN-752.3.patch, YARN-752.4.patch, YARN-752-5.patch, YARN-752-6.patch, YARN-752-7.patch, YARN-752.patch A ContainerRequest that includes node-level requests must also include matching rack-level requests for the racks that those nodes are on. When a node is present without its rack, it makes sense for the client to automatically add the node's rack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[ https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13684824#comment-13684824 ] Hudson commented on YARN-752: - Integrated in Hadoop-trunk-Commit #3942 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3942/]) YARN-752. In AMRMClient, automatically add corresponding rack requests for requested nodes. (sandyr via tucu) (Revision 1493599) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1493599 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClientContainerRequest.java In AMRMClient, automatically add corresponding rack requests for requested nodes Key: YARN-752 URL: https://issues.apache.org/jira/browse/YARN-752 Project: Hadoop YARN Issue Type: Improvement Components: api, applications Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.1.0-beta Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, YARN-752.3.patch, YARN-752.4.patch, YARN-752-5.patch, YARN-752-6.patch, YARN-752-7.patch, YARN-752.patch A ContainerRequest that includes node-level requests must also include matching rack-level requests for the racks that those nodes are on. When a node is present without its rack, it makes sense for the client to automatically add the node's rack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[ https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13683566#comment-13683566 ] Alejandro Abdelnur commented on YARN-752: - patch is not applying anymore, some changes that went in AMRMClientImpl.java are breaking it. Mind rebasing? In AMRMClient, automatically add corresponding rack requests for requested nodes Key: YARN-752 URL: https://issues.apache.org/jira/browse/YARN-752 Project: Hadoop YARN Issue Type: Improvement Components: api, applications Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, YARN-752.3.patch, YARN-752.4.patch, YARN-752-5.patch, YARN-752-6.patch, YARN-752.patch A ContainerRequest that includes node-level requests must also include matching rack-level requests for the racks that those nodes are on. When a node is present without its rack, it makes sense for the client to automatically add the node's rack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[ https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13683695#comment-13683695 ] Vinod Kumar Vavilapalli commented on YARN-752: -- bq. Some additional explanation will be needed, but in general, I don't think all schedulers will be required to handle the relaxLocality flag. Not blocking this patch, but that doesn't seem right. All schedulers MUST handle it. AMs shouldn't be burdened with coding for a specific scheduler. Anyways, Sandy, can you please update it? I'd like to get this in for 2.1.0 that'll happen sometime soon. In AMRMClient, automatically add corresponding rack requests for requested nodes Key: YARN-752 URL: https://issues.apache.org/jira/browse/YARN-752 Project: Hadoop YARN Issue Type: Improvement Components: api, applications Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, YARN-752.3.patch, YARN-752.4.patch, YARN-752-5.patch, YARN-752-6.patch, YARN-752.patch A ContainerRequest that includes node-level requests must also include matching rack-level requests for the racks that those nodes are on. When a node is present without its rack, it makes sense for the client to automatically add the node's rack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[ https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13683703#comment-13683703 ] Sandy Ryza commented on YARN-752: - Attached a rebased patch. bq. Not blocking this patch, but that doesn't seem right. All schedulers MUST handle it. AMs shouldn't be burdened with coding for a specific scheduler. OK, making this required sounds reasonable to me. We can continue the discussion on YARN-521? In AMRMClient, automatically add corresponding rack requests for requested nodes Key: YARN-752 URL: https://issues.apache.org/jira/browse/YARN-752 Project: Hadoop YARN Issue Type: Improvement Components: api, applications Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, YARN-752.3.patch, YARN-752.4.patch, YARN-752-5.patch, YARN-752-6.patch, YARN-752-7.patch, YARN-752.patch A ContainerRequest that includes node-level requests must also include matching rack-level requests for the racks that those nodes are on. When a node is present without its rack, it makes sense for the client to automatically add the node's rack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[ https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13683817#comment-13683817 ] Bikas Saha commented on YARN-752: - Probably just refer to the javadoc for AllocateRequest/ResourceRequest where the semantics are defined. In AMRMClient, automatically add corresponding rack requests for requested nodes Key: YARN-752 URL: https://issues.apache.org/jira/browse/YARN-752 Project: Hadoop YARN Issue Type: Improvement Components: api, applications Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, YARN-752.3.patch, YARN-752.4.patch, YARN-752-5.patch, YARN-752-6.patch, YARN-752-7.patch, YARN-752.patch A ContainerRequest that includes node-level requests must also include matching rack-level requests for the racks that those nodes are on. When a node is present without its rack, it makes sense for the client to automatically add the node's rack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13681581#comment-13681581
]
Hadoop QA commented on YARN-752:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12587497/YARN-752-6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1204//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1204//console
This message is automatically generated.
In AMRMClient, automatically add corresponding rack requests for requested
nodes
Key: YARN-752
URL: https://issues.apache.org/jira/browse/YARN-752
Project: Hadoop YARN
Issue Type: Improvement
Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch,
YARN-752.3.patch, YARN-752.4.patch, YARN-752-5.patch, YARN-752-6.patch,
YARN-752.patch
A ContainerRequest that includes node-level requests must also include
matching rack-level requests for the racks that those nodes are on. When a
node is present without its rack, it makes sense for the client to
automatically add the node's rack.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[ https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679899#comment-13679899 ] Sandy Ryza commented on YARN-752: - [~bikassaha], your updated patch looks good to me. Noticed a minor spacing issue that originated in my code, uploading a patch to fix it. In AMRMClient, automatically add corresponding rack requests for requested nodes Key: YARN-752 URL: https://issues.apache.org/jira/browse/YARN-752 Project: Hadoop YARN Issue Type: Improvement Components: api, applications Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, YARN-752.3.patch, YARN-752.4.patch, YARN-752.patch A ContainerRequest that includes node-level requests must also include matching rack-level requests for the racks that those nodes are on. When a node is present without its rack, it makes sense for the client to automatically add the node's rack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679915#comment-13679915
]
Hadoop QA commented on YARN-752:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12587130/YARN-752-5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1181//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1181//console
This message is automatically generated.
In AMRMClient, automatically add corresponding rack requests for requested
nodes
Key: YARN-752
URL: https://issues.apache.org/jira/browse/YARN-752
Project: Hadoop YARN
Issue Type: Improvement
Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch,
YARN-752.3.patch, YARN-752.4.patch, YARN-752-5.patch, YARN-752.patch
A ContainerRequest that includes node-level requests must also include
matching rack-level requests for the racks that those nodes are on. When a
node is present without its rack, it makes sense for the client to
automatically add the node's rack.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[ https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679235#comment-13679235 ] Bikas Saha commented on YARN-752: - Its not 100% clear what the API should be on the AMRMClient for specific requests. A set of checks need to be made so that unsatisfiable requests cannot be made or illegal mix-n-match cannot happen. The server side impl is simple and does no checks for the sake of overhead reduction. Regardless, given the protocol API that requires relaxLocality flag to be specified on racks for the specific nodes, the resolution of nodes to racks is required, even in that case. This jira is simply for the addition of rack resolution functionality to the AMRMClient. Its current use is to make sure all nodes are resolved to their racks or else those requests will be unsatisfiable on the server side, resulting in correctness issues. The patch by itself has a bug. After a container request has been removed, it should not be returned via getMatchingRequests() for the inferred rack. i.e. the patch was doing addResourceRequest() for the inferred rack but not doing the corresponding removeResourceRequest(). The simple fix is to resolve/dedupe not only during addContainerRequest() but also during removeContainerRequest(). Attaching patch with simple fix and minor changes to the javadoc. In AMRMClient, automatically add corresponding rack requests for requested nodes Key: YARN-752 URL: https://issues.apache.org/jira/browse/YARN-752 Project: Hadoop YARN Issue Type: Improvement Components: api, applications Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, YARN-752.3.patch, YARN-752.patch A ContainerRequest that includes node-level requests must also include matching rack-level requests for the racks that those nodes are on. When a node is present without its rack, it makes sense for the client to automatically add the node's rack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679241#comment-13679241
]
Hadoop QA commented on YARN-752:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12587002/YARN-752.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client:
org.apache.hadoop.yarn.client.TestAMRMClient
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1174//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1174//console
This message is automatically generated.
In AMRMClient, automatically add corresponding rack requests for requested
nodes
Key: YARN-752
URL: https://issues.apache.org/jira/browse/YARN-752
Project: Hadoop YARN
Issue Type: Improvement
Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch,
YARN-752.3.patch, YARN-752.patch
A ContainerRequest that includes node-level requests must also include
matching rack-level requests for the racks that those nodes are on. When a
node is present without its rack, it makes sense for the client to
automatically add the node's rack.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679254#comment-13679254
]
Hadoop QA commented on YARN-752:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12587008/YARN-752.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1175//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1175//console
This message is automatically generated.
In AMRMClient, automatically add corresponding rack requests for requested
nodes
Key: YARN-752
URL: https://issues.apache.org/jira/browse/YARN-752
Project: Hadoop YARN
Issue Type: Improvement
Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch,
YARN-752.3.patch, YARN-752.4.patch, YARN-752.patch
A ContainerRequest that includes node-level requests must also include
matching rack-level requests for the racks that those nodes are on. When a
node is present without its rack, it makes sense for the client to
automatically add the node's rack.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[ https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678228#comment-13678228 ] Siddharth Seth commented on YARN-752: - Can we make this rack resolution optional 1) If the user has provided racks - just trust that information instead of resolving and then checking whether the user has provided the correct information. 2) Does it make sense to have an option to disable this automatic resolution. I haven't been following the whitelisting / blacklisting closely - but isn't it possible for users to ask for specific hosts, without specifying racks (Ignoring scheduler implementation here) In AMRMClient, automatically add corresponding rack requests for requested nodes Key: YARN-752 URL: https://issues.apache.org/jira/browse/YARN-752 Project: Hadoop YARN Issue Type: Improvement Components: api, applications Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752.patch A ContainerRequest that includes node-level requests must also include matching rack-level requests for the racks that those nodes are on. When a node is present without its rack, it makes sense for the client to automatically add the node's rack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[ https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678270#comment-13678270 ] Sandy Ryza commented on YARN-752: - bq. If the user has provided racks - just trust that information instead of resolving and then checking whether the user has provided the correct information. If we do this, I think it would be good at least to log some sort of warnings when the given and expected racks don't match up. The consequences of a user getting it wrong can be pretty subtle. bq. but isn't it possible for users to ask for specific hosts, without specifying racks (Ignoring scheduler implementation here) Currently, it does not make sense for users to ask for specific hosts without specifying racks in any scheduler implementation. Containers may still be scheduled on those nodes, but they will be treated like any other non-local nodes for the purposes of delay-scheduling and locality-specific requests. I believe that this is not just an implementation issue, but a result of the protocol. A scheduler having zero containers requested on rack1, but non-zero containers requested on nodes on rack1, can occur when a single container was requested for any of multiple nodes on rack1. In AMRMClient, automatically add corresponding rack requests for requested nodes Key: YARN-752 URL: https://issues.apache.org/jira/browse/YARN-752 Project: Hadoop YARN Issue Type: Improvement Components: api, applications Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752.patch A ContainerRequest that includes node-level requests must also include matching rack-level requests for the racks that those nodes are on. When a node is present without its rack, it makes sense for the client to automatically add the node's rack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[ https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678273#comment-13678273 ] Bikas Saha commented on YARN-752: - YARN's default and most efficient scheduling behavior comes when locality can relaxed from node to rack etc. The AMRMClient should simply do the right thing in this context and add racks for the nodes provided by the user, if the user has already not done so. If this is not done then its the burden of the user to do it and if the user forgets to do it then YARN will probably not give it any containers and the job will be stuck. Hence, by default the AMRMClient should always fill in missing racks. If the user has already done the right thing then this functionality will be a no-op. This cannot be optional. The only time this should not be done is when the user has asked for specific nodes/racks. Even there the AMRMClient will have to add missing racks but with a special flag or else the schedulers will not assign containers to the app. Support for that feature is yet to be added in AMRMClient and is tracked by another jira. If the user has specified a mix of nodes and racks, even then we need to ensure that racks are added for nodes whose racks are missing. In AMRMClient, automatically add corresponding rack requests for requested nodes Key: YARN-752 URL: https://issues.apache.org/jira/browse/YARN-752 Project: Hadoop YARN Issue Type: Improvement Components: api, applications Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752.patch A ContainerRequest that includes node-level requests must also include matching rack-level requests for the racks that those nodes are on. When a node is present without its rack, it makes sense for the client to automatically add the node's rack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[ https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678275#comment-13678275 ] Bikas Saha commented on YARN-752: - The patch looks good to me as is. Need to do a final review. In AMRMClient, automatically add corresponding rack requests for requested nodes Key: YARN-752 URL: https://issues.apache.org/jira/browse/YARN-752 Project: Hadoop YARN Issue Type: Improvement Components: api, applications Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752.patch A ContainerRequest that includes node-level requests must also include matching rack-level requests for the racks that those nodes are on. When a node is present without its rack, it makes sense for the client to automatically add the node's rack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[ https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678401#comment-13678401 ] Alejandro Abdelnur commented on YARN-752: - agree with [~bikassaha]'s, only question left, what if the user provides wrong rack for node? As Sandy mentioned, that could lead to odd/subtle behavior. In AMRMClient, automatically add corresponding rack requests for requested nodes Key: YARN-752 URL: https://issues.apache.org/jira/browse/YARN-752 Project: Hadoop YARN Issue Type: Improvement Components: api, applications Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752.patch A ContainerRequest that includes node-level requests must also include matching rack-level requests for the racks that those nodes are on. When a node is present without its rack, it makes sense for the client to automatically add the node's rack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[ https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678405#comment-13678405 ] Bikas Saha commented on YARN-752: - User cannot provide a wrong rack for a node. User can provide a wrong rack by itself. If a rack is missing then it will be corrected by AMRMClient. If there is another different rack, its either a valid rack or a rack that does not exist. We should probably resolve nodes and racks to make sure they are all valid locations. Different jira perhaps. This kind of depends on YARN-435. In AMRMClient, automatically add corresponding rack requests for requested nodes Key: YARN-752 URL: https://issues.apache.org/jira/browse/YARN-752 Project: Hadoop YARN Issue Type: Improvement Components: api, applications Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752.patch A ContainerRequest that includes node-level requests must also include matching rack-level requests for the racks that those nodes are on. When a node is present without its rack, it makes sense for the client to automatically add the node's rack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[ https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678418#comment-13678418 ] Alejandro Abdelnur commented on YARN-752: - Ah!, ok, so before this patch if I would do a request of only NODES allocations would never happen. I think we should clearly document in the API that specifying RACKS is only needed if you want an allocation in any node of that RACK, if you want the allocation in a NODE you don't need to specify the rack the node is. In AMRMClient, automatically add corresponding rack requests for requested nodes Key: YARN-752 URL: https://issues.apache.org/jira/browse/YARN-752 Project: Hadoop YARN Issue Type: Improvement Components: api, applications Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752.patch A ContainerRequest that includes node-level requests must also include matching rack-level requests for the racks that those nodes are on. When a node is present without its rack, it makes sense for the client to automatically add the node's rack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[ https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678445#comment-13678445 ] Siddharth Seth commented on YARN-752: - Alright - I guess the node only request will be handled in the other jira to track it. And from Bikas' explanation - that'll require rack resolution in any case. Seems a little strange, but required based on scheduler implementation. If we're always doing the rack lookup, we should change the MR AM avoid rack lookups if possible - may be difficult considering it tracks RACK_LOCAL allocations. Separate jira though. The rack lookup in the patch itself, shouldn't it be adding a rack only once. e.g. ResourceRequest: h1, h2, h3 - numContainers=1 If all of them resolve to the same rack - num containers on the rack would go to 3 after this ? Is that correct behaviour. In AMRMClient, automatically add corresponding rack requests for requested nodes Key: YARN-752 URL: https://issues.apache.org/jira/browse/YARN-752 Project: Hadoop YARN Issue Type: Improvement Components: api, applications Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752.patch A ContainerRequest that includes node-level requests must also include matching rack-level requests for the racks that those nodes are on. When a node is present without its rack, it makes sense for the client to automatically add the node's rack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[ https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678468#comment-13678468 ] Sandy Ryza commented on YARN-752: - bq. The rack lookup in the patch itself, shouldn't it be adding a rack only once. You're right. I'll fix this. In AMRMClient, automatically add corresponding rack requests for requested nodes Key: YARN-752 URL: https://issues.apache.org/jira/browse/YARN-752 Project: Hadoop YARN Issue Type: Improvement Components: api, applications Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752.patch A ContainerRequest that includes node-level requests must also include matching rack-level requests for the racks that those nodes are on. When a node is present without its rack, it makes sense for the client to automatically add the node's rack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678470#comment-13678470
]
Bikas Saha commented on YARN-752:
-
The user needs to only specify a node (and may end up getting a container on
that node, on that node's rack or somewhere else. There is no need to specify
the rack of a node explicitly but users may do so. If a user wants any node in
a set of racks then they should simply directly specify those racks.
This code is not quite correct. If nodes n1, n2, n3 (all in rack1) are present
then this will end up calling addResourceRequest 3 times for rack1. Should
probably create a local set of racks, initialized by CR.racks, add nodes' racks
to the set. Then call addResourceRequest for the rack set.
{code}
+String rack = RackResolver.resolve(host).getNetworkLocation();
+if (rack == null) {
+ LOG.warn(Failed to resolve rack for host + host + .);
+} else if (req.racks == null || !req.racks.contains(rack)) {
+ addResourceRequest(req.priority, rack, req.capability,
+ req.containerCount, req);
+}
{code}
The test needs to improve to catch the above case. Also, should check container
count for the asks to ensure that counts are correct (with or without user
specifying the racks). The test is also calling api's without calling start().
It works now, but will probably break later once we enforce the life cycle
inside the client. Things like rack resolution etc probably depend on topoplogy
information coming from the RM upon registration.
In AMRMClient, automatically add corresponding rack requests for requested
nodes
Key: YARN-752
URL: https://issues.apache.org/jira/browse/YARN-752
Project: Hadoop YARN
Issue Type: Improvement
Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752.patch
A ContainerRequest that includes node-level requests must also include
matching rack-level requests for the racks that those nodes are on. When a
node is present without its rack, it makes sense for the client to
automatically add the node's rack.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[ https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678472#comment-13678472 ] Bikas Saha commented on YARN-752: - Would be great if the next patch added documentation requested by [~tucu00] In AMRMClient, automatically add corresponding rack requests for requested nodes Key: YARN-752 URL: https://issues.apache.org/jira/browse/YARN-752 Project: Hadoop YARN Issue Type: Improvement Components: api, applications Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752.patch A ContainerRequest that includes node-level requests must also include matching rack-level requests for the racks that those nodes are on. When a node is present without its rack, it makes sense for the client to automatically add the node's rack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[ https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678535#comment-13678535 ] Sandy Ryza commented on YARN-752: - Uploading a patch that modifies the test the catch the issue identified by Siddharth, fixes it, and adds the documentation requested by Tucu In AMRMClient, automatically add corresponding rack requests for requested nodes Key: YARN-752 URL: https://issues.apache.org/jira/browse/YARN-752 Project: Hadoop YARN Issue Type: Improvement Components: api, applications Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, YARN-752.patch A ContainerRequest that includes node-level requests must also include matching rack-level requests for the racks that those nodes are on. When a node is present without its rack, it makes sense for the client to automatically add the node's rack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678557#comment-13678557
]
Hadoop QA commented on YARN-752:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12586821/YARN-752-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1164//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1164//console
This message is automatically generated.
In AMRMClient, automatically add corresponding rack requests for requested
nodes
Key: YARN-752
URL: https://issues.apache.org/jira/browse/YARN-752
Project: Hadoop YARN
Issue Type: Improvement
Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch,
YARN-752.patch
A ContainerRequest that includes node-level requests must also include
matching rack-level requests for the racks that those nodes are on. When a
node is present without its rack, it makes sense for the client to
automatically add the node's rack.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678561#comment-13678561
]
Vinod Kumar Vavilapalli commented on YARN-752:
--
Haven't been paying attention. But quickly looked through.
- I think if the user gives a wrong rack, we should throw an error.
- IIUC, this code will change after YARN-521 or is YARN-521 just a new API
instead of changes to the current API with optional flags?
- In any case, the algo should eventually look like
{code}
For each explicitly added rack, if it doesn't exist, throw error;
if (asking for specific nodes) {
don't add racks
} else {
add racks
}
{code}
- Either ways I think YARN-521 is a priority, depend on how the API is
designed, it can result in API signature changes.
If you agree that the logic is based on enabling/disabling strict allocations,
the following in the patch won't be correct any longer, right?
{code}
Scheduler
+ * documentation should be consulted for the specifics of how the parameters
+ * are honored.
{code}
In AMRMClient, automatically add corresponding rack requests for requested
nodes
Key: YARN-752
URL: https://issues.apache.org/jira/browse/YARN-752
Project: Hadoop YARN
Issue Type: Improvement
Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch,
YARN-752.patch
A ContainerRequest that includes node-level requests must also include
matching rack-level requests for the racks that those nodes are on. When a
node is present without its rack, it makes sense for the client to
automatically add the node's rack.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[ https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678578#comment-13678578 ] Alejandro Abdelnur commented on YARN-752: - bq. I think if the user gives a wrong rack, we should throw an error. Per previous discussion this does not happen (except if you are referring to an invalid rack altogether). If a ContainerRequest has host1 and rack3, it means that the containers are desired either in the host1 or rack3. it does not mean that host1 is in rack3. In AMRMClient, automatically add corresponding rack requests for requested nodes Key: YARN-752 URL: https://issues.apache.org/jira/browse/YARN-752 Project: Hadoop YARN Issue Type: Improvement Components: api, applications Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, YARN-752.patch A ContainerRequest that includes node-level requests must also include matching rack-level requests for the racks that those nodes are on. When a node is present without its rack, it makes sense for the client to automatically add the node's rack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[ https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678613#comment-13678613 ] Sandy Ryza commented on YARN-752: - In my vision of YARN-521, we will add an additional boolean parameter to the ContainerRequest constructor, relaxLocality. If a ContainerRequest's relaxLocality is false, and both nodes and racks are passed in to a ContainerRequest, we have a few options for how to handle it: * Set relaxLocality to false only for all non-explicit ResourceRequests. This would mean that if I submit a ContainerRequest with both nodes and the racks that those nodes are on, containers could be given on any nodes on those racks. * If there are any node requests, set relaxLocality to false for all rack requests that those nodes are on, even if the racks are explicitly given. This would mean that if I submit a request with a node, the rack it is on, and another rack, containers could be given on that node or on any node on the other rack. * If there are any node requests, set relaxLocality to false for all rack requests. This would mean that if I submit a request with a node, the rack it is on, and another rack, containers could be given only on the node. bq. If you agree that the logic is based on enabling/disabling strict allocations, the following in the patch won't be correct any longer, right? Some additional explanation will be needed, but in general, I don't think all schedulers will be required to handle the relaxLocality flag. In AMRMClient, automatically add corresponding rack requests for requested nodes Key: YARN-752 URL: https://issues.apache.org/jira/browse/YARN-752 Project: Hadoop YARN Issue Type: Improvement Components: api, applications Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752-2.patch, YARN-752.patch A ContainerRequest that includes node-level requests must also include matching rack-level requests for the racks that those nodes are on. When a node is present without its rack, it makes sense for the client to automatically add the node's rack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13677287#comment-13677287
]
Bikas Saha commented on YARN-752:
-
First of all I think I made a mistake in putting ImmutableList as part of the
signature of the getHosts() etc. It ties down the implementation vs having
returned a generic list. We should probably fix that. I can do that in a
different jira. We should be storing a simple memory efficient list and
returning an unmodifiablelist instead of ImmutableList.
Changing the storage type to set is not desired. The set is only needed 1 time
for ease of lookup(and probably isnt much different in perf vs linear search
for single digit sizes). Lets not change the actual storage to a set because
its just more memory (for millions of container request) and it also loses the
ordering given by the user. At some point we may want to match requests in the
preferred order given by the user.
What if the RackResolver is improperly initialized because of a bad config? And
does not return valid/non-null values? Can you please check?
{code}
+String rack = RackResolver.resolve(host).getNetworkLocation();
+if (!req.racks.contains(rack)) {
{code}
The test could probably go into the existing tests for AMRMClient so that it
stays in one place. Also, would be great to make sure that
getMatchingRequests() work properly in this case - user should get the
container request for a rack for which they had originally specified only
nodes. They should but a test would verify the expected behavior.
In AMRMClient, automatically add corresponding rack requests for requested
nodes
Key: YARN-752
URL: https://issues.apache.org/jira/browse/YARN-752
Project: Hadoop YARN
Issue Type: Improvement
Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-752.patch
A ContainerRequest that includes node-level requests must also include
matching rack-level requests for the racks that those nodes are on. When a
node is present without its rack, it makes sense for the client to
automatically add the node's rack.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[ https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13677373#comment-13677373 ] Bikas Saha commented on YARN-752: - bq. Wrapping the test in a MiniYARNCluster and other structures used for the AMRMClient lifecycle testing in TestAMRMClient will make debugging this specific functionality more difficult in the future. Thats fine. Can that not be done within the existing file TestAMRMClient.java instead of creating a new one? Just helps to keep all related tests in one place. Not a big deal if thats not simple. In AMRMClient, automatically add corresponding rack requests for requested nodes Key: YARN-752 URL: https://issues.apache.org/jira/browse/YARN-752 Project: Hadoop YARN Issue Type: Improvement Components: api, applications Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-752.patch A ContainerRequest that includes node-level requests must also include matching rack-level requests for the racks that those nodes are on. When a node is present without its rack, it makes sense for the client to automatically add the node's rack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[ https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13677414#comment-13677414 ] Sandy Ryza commented on YARN-752: - Uploading a patch that addresses Bikas's comments. bq. Can that not be done within the existing file TestAMRMClient.java instead of creating a new one? The only ways I can think of to do it seem a little ugly to me, but I'm open to suggestions. In AMRMClient, automatically add corresponding rack requests for requested nodes Key: YARN-752 URL: https://issues.apache.org/jira/browse/YARN-752 Project: Hadoop YARN Issue Type: Improvement Components: api, applications Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-752.patch A ContainerRequest that includes node-level requests must also include matching rack-level requests for the racks that those nodes are on. When a node is present without its rack, it makes sense for the client to automatically add the node's rack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13677425#comment-13677425
]
Hadoop QA commented on YARN-752:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12586559/YARN-752-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client:
org.apache.hadoop.yarn.client.TestNMClientAsync
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1145//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1145//console
This message is automatically generated.
In AMRMClient, automatically add corresponding rack requests for requested
nodes
Key: YARN-752
URL: https://issues.apache.org/jira/browse/YARN-752
Project: Hadoop YARN
Issue Type: Improvement
Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-752-1.patch, YARN-752.patch
A ContainerRequest that includes node-level requests must also include
matching rack-level requests for the racks that those nodes are on. When a
node is present without its rack, it makes sense for the client to
automatically add the node's rack.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[ https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13677479#comment-13677479 ] Sandy Ryza commented on YARN-752: - I'm unable to reproduce the failure locally, and don't see how the changes could have caused it. I'm going to try re-uploading. In AMRMClient, automatically add corresponding rack requests for requested nodes Key: YARN-752 URL: https://issues.apache.org/jira/browse/YARN-752 Project: Hadoop YARN Issue Type: Improvement Components: api, applications Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752.patch A ContainerRequest that includes node-level requests must also include matching rack-level requests for the racks that those nodes are on. When a node is present without its rack, it makes sense for the client to automatically add the node's rack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13677492#comment-13677492
]
Hadoop QA commented on YARN-752:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12586579/YARN-752-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1146//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1146//console
This message is automatically generated.
In AMRMClient, automatically add corresponding rack requests for requested
nodes
Key: YARN-752
URL: https://issues.apache.org/jira/browse/YARN-752
Project: Hadoop YARN
Issue Type: Improvement
Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-752-1.patch, YARN-752-1.patch, YARN-752.patch
A ContainerRequest that includes node-level requests must also include
matching rack-level requests for the racks that those nodes are on. When a
node is present without its rack, it makes sense for the client to
automatically add the node's rack.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[ https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673526#comment-13673526 ] Sandy Ryza commented on YARN-752: - Cool, I think that option is the best too. Updated the title and description. In AMRMClient, automatically add corresponding rack requests for requested nodes Key: YARN-752 URL: https://issues.apache.org/jira/browse/YARN-752 Project: Hadoop YARN Issue Type: Improvement Components: api, applications Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza A ContainerRequest that includes node-level requests must also include matching rack-level requests for the racks that those nodes are on. When a node is present without its rack, it makes sense for the client to automatically add the node's rack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[ https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673577#comment-13673577 ] Sandy Ryza commented on YARN-752: - [~bikassaha], do you have an opinion on whether it makes more sense to add the missing racks when a ContainerRequest is constructed or when it's submitted to the AMRMClient? In AMRMClient, automatically add corresponding rack requests for requested nodes Key: YARN-752 URL: https://issues.apache.org/jira/browse/YARN-752 Project: Hadoop YARN Issue Type: Improvement Components: api, applications Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza A ContainerRequest that includes node-level requests must also include matching rack-level requests for the racks that those nodes are on. When a node is present without its rack, it makes sense for the client to automatically add the node's rack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[ https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673859#comment-13673859 ] Sandy Ryza commented on YARN-752: - Attached a patch that resolves racks in the AMRMClient In AMRMClient, automatically add corresponding rack requests for requested nodes Key: YARN-752 URL: https://issues.apache.org/jira/browse/YARN-752 Project: Hadoop YARN Issue Type: Improvement Components: api, applications Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-752.patch A ContainerRequest that includes node-level requests must also include matching rack-level requests for the racks that those nodes are on. When a node is present without its rack, it makes sense for the client to automatically add the node's rack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-752) In AMRMClient, automatically add corresponding rack requests for requested nodes
[
https://issues.apache.org/jira/browse/YARN-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673879#comment-13673879
]
Hadoop QA commented on YARN-752:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12586005/YARN-752.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1094//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1094//console
This message is automatically generated.
In AMRMClient, automatically add corresponding rack requests for requested
nodes
Key: YARN-752
URL: https://issues.apache.org/jira/browse/YARN-752
Project: Hadoop YARN
Issue Type: Improvement
Components: api, applications
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-752.patch
A ContainerRequest that includes node-level requests must also include
matching rack-level requests for the racks that those nodes are on. When a
node is present without its rack, it makes sense for the client to
automatically add the node's rack.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
