[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13672107#comment-13672107 ] Hudson commented on YARN-392: - Integrated in Hadoop-Mapreduce-trunk #1443 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1443/]) YARN-392. Make it possible to specify hard locality constraints in resource requests. (sandyr via tucu) (Revision 1488326) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1488326 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceRequest.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourceRequestPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AppSchedulable.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java > Make it possible to specify hard locality constraints in resource requests > -- > > Key: YARN-392 > URL: https://issues.apache.org/jira/browse/YARN-392 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Sandy Ryza > Fix For: 2.1.0-beta > > Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch, > YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392-5.patch, > YARN-392-6.patch, YARN-392-7.patch, YARN-392-8.patch, YARN-392.patch > > > Currently its not possible to specify scheduling requests for specific nodes > and nowhere else. The RM automatically relaxes locality to rack and * and > assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13672088#comment-13672088 ] Hudson commented on YARN-392: - Integrated in Hadoop-Hdfs-trunk #1417 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1417/]) YARN-392. Make it possible to specify hard locality constraints in resource requests. (sandyr via tucu) (Revision 1488326) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1488326 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceRequest.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourceRequestPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AppSchedulable.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java > Make it possible to specify hard locality constraints in resource requests > -- > > Key: YARN-392 > URL: https://issues.apache.org/jira/browse/YARN-392 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Sandy Ryza > Fix For: 2.1.0-beta > > Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch, > YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392-5.patch, > YARN-392-6.patch, YARN-392-7.patch, YARN-392-8.patch, YARN-392.patch > > > Currently its not possible to specify scheduling requests for specific nodes > and nowhere else. The RM automatically relaxes locality to rack and * and > assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13672043#comment-13672043 ] Hudson commented on YARN-392: - Integrated in Hadoop-Yarn-trunk #227 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/227/]) YARN-392. Make it possible to specify hard locality constraints in resource requests. (sandyr via tucu) (Revision 1488326) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1488326 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceRequest.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourceRequestPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AppSchedulable.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java > Make it possible to specify hard locality constraints in resource requests > -- > > Key: YARN-392 > URL: https://issues.apache.org/jira/browse/YARN-392 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Sandy Ryza > Fix For: 2.1.0-beta > > Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch, > YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392-5.patch, > YARN-392-6.patch, YARN-392-7.patch, YARN-392-8.patch, YARN-392.patch > > > Currently its not possible to specify scheduling requests for specific nodes > and nowhere else. The RM automatically relaxes locality to rack and * and > assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13671680#comment-13671680 ] Hudson commented on YARN-392: - Integrated in Hadoop-trunk-Commit #3821 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3821/]) YARN-392. Make it possible to specify hard locality constraints in resource requests. (sandyr via tucu) (Revision 1488326) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1488326 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceRequest.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourceRequestPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AppSchedulable.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java > Make it possible to specify hard locality constraints in resource requests > -- > > Key: YARN-392 > URL: https://issues.apache.org/jira/browse/YARN-392 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Sandy Ryza > Fix For: 2.0.5-beta > > Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch, > YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392-5.patch, > YARN-392-6.patch, YARN-392-7.patch, YARN-392-8.patch, YARN-392.patch > > > Currently its not possible to specify scheduling requests for specific nodes > and nowhere else. The RM automatically relaxes locality to rack and * and > assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13671065#comment-13671065 ] Arun C Murthy commented on YARN-392: +1 for the api - I haven't reviewed FS changes though, thanks Sandy. > Make it possible to specify hard locality constraints in resource requests > -- > > Key: YARN-392 > URL: https://issues.apache.org/jira/browse/YARN-392 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Sandy Ryza > Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch, > YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392-5.patch, > YARN-392-6.patch, YARN-392-7.patch, YARN-392-8.patch, YARN-392.patch > > > Currently its not possible to specify scheduling requests for specific nodes > and nowhere else. The RM automatically relaxes locality to rack and * and > assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670458#comment-13670458 ] Alejandro Abdelnur commented on YARN-392: - [~bikassaha], [~acmurthy], all comments have been addressed, any new one? Else I'd like to commit this to get things moving on the AM client API side. > Make it possible to specify hard locality constraints in resource requests > -- > > Key: YARN-392 > URL: https://issues.apache.org/jira/browse/YARN-392 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Sandy Ryza > Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch, > YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392-5.patch, > YARN-392-6.patch, YARN-392-7.patch, YARN-392-8.patch, YARN-392.patch > > > Currently its not possible to specify scheduling requests for specific nodes > and nowhere else. The RM automatically relaxes locality to rack and * and > assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668677#comment-13668677 ] Bikas Saha commented on YARN-392: - Sorry got caught up with other stuff. I was waiting for [~acmurthy] to chime in and make sure he is on the same page. > Make it possible to specify hard locality constraints in resource requests > -- > > Key: YARN-392 > URL: https://issues.apache.org/jira/browse/YARN-392 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Sandy Ryza > Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch, > YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392-5.patch, > YARN-392-6.patch, YARN-392-7.patch, YARN-392-8.patch, YARN-392.patch > > > Currently its not possible to specify scheduling requests for specific nodes > and nowhere else. The RM automatically relaxes locality to rack and * and > assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668669#comment-13668669 ] Sandy Ryza commented on YARN-392: - [~bikassaha], thanks for taking the time to iterate on all this. Does the latest patch address all of your concerns? > Make it possible to specify hard locality constraints in resource requests > -- > > Key: YARN-392 > URL: https://issues.apache.org/jira/browse/YARN-392 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Sandy Ryza > Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch, > YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392-5.patch, > YARN-392-6.patch, YARN-392-7.patch, YARN-392-8.patch, YARN-392.patch > > > Currently its not possible to specify scheduling requests for specific nodes > and nowhere else. The RM automatically relaxes locality to rack and * and > assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[
https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13665483#comment-13665483
]
Hadoop QA commented on YARN-392:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584541/YARN-392-8.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/993//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/993//console
This message is automatically generated.
> Make it possible to specify hard locality constraints in resource requests
> --
>
> Key: YARN-392
> URL: https://issues.apache.org/jira/browse/YARN-392
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Bikas Saha
>Assignee: Sandy Ryza
> Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch,
> YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392-5.patch,
> YARN-392-6.patch, YARN-392-7.patch, YARN-392-8.patch, YARN-392.patch
>
>
> Currently its not possible to specify scheduling requests for specific nodes
> and nowhere else. The RM automatically relaxes locality to rack and * and
> assigns non-specified machines to the app.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13665465#comment-13665465 ] Sandy Ryza commented on YARN-392: - Uploaded a patch that addresses Bikas's latest comments. > Make it possible to specify hard locality constraints in resource requests > -- > > Key: YARN-392 > URL: https://issues.apache.org/jira/browse/YARN-392 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Sandy Ryza > Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch, > YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392-5.patch, > YARN-392-6.patch, YARN-392-7.patch, YARN-392-8.patch, YARN-392.patch > > > Currently its not possible to specify scheduling requests for specific nodes > and nowhere else. The RM automatically relaxes locality to rack and * and > assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13665433#comment-13665433 ] Sandy Ryza commented on YARN-392: - bq. Can anyRequest be null. Others are checked for null but this one isn't. In the contexts that the method is called, it can't be null, but I'll add in a check to be defensive. Will fix the comments as well. Based on Arun's April 4th comment, my understanding was that capacity scheduler work would be done in YARN-398. > Make it possible to specify hard locality constraints in resource requests > -- > > Key: YARN-392 > URL: https://issues.apache.org/jira/browse/YARN-392 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Sandy Ryza > Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch, > YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392-5.patch, > YARN-392-6.patch, YARN-392-7.patch, YARN-392.patch > > > Currently its not possible to specify scheduling requests for specific nodes > and nowhere else. The RM automatically relaxes locality to rack and * and > assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13664339#comment-13664339 ] Bikas Saha commented on YARN-392: - btw, what are the plans for the capacity scheduler? > Make it possible to specify hard locality constraints in resource requests > -- > > Key: YARN-392 > URL: https://issues.apache.org/jira/browse/YARN-392 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Sandy Ryza > Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch, > YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392-5.patch, > YARN-392-6.patch, YARN-392-7.patch, YARN-392.patch > > > Currently its not possible to specify scheduling requests for specific nodes > and nowhere else. The RM automatically relaxes locality to rack and * and > assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[
https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13664333#comment-13664333
]
Bikas Saha commented on YARN-392:
-
Minor nits
You would want to put ResourceRequest#ANY to be clear what "any" means in the
javadoc.
Can anyRequest be null. Others are checked for null but this one isnt.
Typo in comment. Duplicate non-zero.
{code}
+return
+// There must be outstanding requests at the given priority:
+anyRequest.getNumContainers() > 0 &&
+// If locality relaxation is turned off at *-level, there must be a
non-zero
+// non-zero request for the node's rack:
{code}
> Make it possible to specify hard locality constraints in resource requests
> --
>
> Key: YARN-392
> URL: https://issues.apache.org/jira/browse/YARN-392
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Bikas Saha
>Assignee: Sandy Ryza
> Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch,
> YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392-5.patch,
> YARN-392-6.patch, YARN-392-7.patch, YARN-392.patch
>
>
> Currently its not possible to specify scheduling requests for specific nodes
> and nowhere else. The RM automatically relaxes locality to rack and * and
> assigns non-specified machines to the app.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[
https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13663472#comment-13663472
]
Hadoop QA commented on YARN-392:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584181/YARN-392-7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/978//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/978//console
This message is automatically generated.
> Make it possible to specify hard locality constraints in resource requests
> --
>
> Key: YARN-392
> URL: https://issues.apache.org/jira/browse/YARN-392
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Bikas Saha
>Assignee: Sandy Ryza
> Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch,
> YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392-5.patch,
> YARN-392-6.patch, YARN-392-7.patch, YARN-392.patch
>
>
> Currently its not possible to specify scheduling requests for specific nodes
> and nowhere else. The RM automatically relaxes locality to rack and * and
> assigns non-specified machines to the app.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13663453#comment-13663453 ] Sandy Ryza commented on YARN-392: - Uploaded a new patch to address Bikas' comments. > Make it possible to specify hard locality constraints in resource requests > -- > > Key: YARN-392 > URL: https://issues.apache.org/jira/browse/YARN-392 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Sandy Ryza > Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch, > YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392-5.patch, > YARN-392-6.patch, YARN-392-7.patch, YARN-392.patch > > > Currently its not possible to specify scheduling requests for specific nodes > and nowhere else. The RM automatically relaxes locality to rack and * and > assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13663309#comment-13663309 ] Sandy Ryza commented on YARN-392: - bq. We mean to that level and beyond? You're right. I'll update the javadoc to better reflect the code. bq. It might help if we write the example in the javadoc by specifying how to use the flag to enable strict node locality. Agreed. bq. My grey cells are not what they used to be Can you please help me grok the logic following statement? I'll add in some comments. bq. You mean node2 right? Right. > Make it possible to specify hard locality constraints in resource requests > -- > > Key: YARN-392 > URL: https://issues.apache.org/jira/browse/YARN-392 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Sandy Ryza > Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch, > YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392-5.patch, > YARN-392-6.patch, YARN-392.patch > > > Currently its not possible to specify scheduling requests for specific nodes > and nowhere else. The RM automatically relaxes locality to rack and * and > assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[
https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13663267#comment-13663267
]
Bikas Saha commented on YARN-392:
-
bq. in a followup JIRA we can decide that setting the flag on a node-level
request warrants an exception. Or, in a followup JIRA, we can decide that
setting the flag on a node-level request is supported. Neither of these future
changes will be a backwards-incompatible change.
Makes sense to me.
We mean to that level and beyond? e.g. if the flag is set on a rack then it
implies locality wont be relaxed for that rack and * right? Thats what the code
in the patch does (continue if rack is false). OR does the flag need to be set
at rack AND * when asking for specific node?
{code}
+ * For a request at a network hierarchy level, set whether locality can be
relaxed
+ * to that level.
{code}
It might help if we write the example in the javadoc by specifying how to use
the flag to enable strict node locality. We dont have code checks about what it
legal wrt this flag but we should spell them out in the javadoc.
{code}
+ *
+ * For example, if the flag is off on a rack-level
ResourceRequest,
+ * containers at that request's priority will not be assigned to nodes on
that
+ * request's rack unless requests specifically for those nodes have also been
+ * submitted.
+ *
+ * If the flag is off on an any-level ResourceRequest,
containers at
+ * that request's priority will only be assigned on racks for which specific
+ * requests have also been submitted.
{code}
My grey cells are not what they used to be :) Can you please help me grok the
logic following statement?
{code}
+return anyRequest.getNumContainers() > 0 &&
+(nodeRequest == null || nodeRequest.getRelaxLocality()) &&
+(anyRequest.getRelaxLocality() ||
+(rackRequest != null && rackRequest.getNumContainers() > 0)) &&
+(rackRequest == null || rackRequest.getRelaxLocality() ||
+(nodeRequest != null && nodeRequest.getNumContainers() > 0)) &&
Resources.lessThanOrEqual(RESOURCE_CALCULATOR, null,
-request.getCapability(), node.getRMNode().getTotalCapability());
+anyRequest.getCapability(), node.getRMNode().getTotalCapability());
{code}
You mean node2 right?
{code}
+// then node1 should get the container
+scheduler.handle(node2UpdateEvent);
+assertEquals(1, app.getLiveContainers().size());
{code}
> Make it possible to specify hard locality constraints in resource requests
> --
>
> Key: YARN-392
> URL: https://issues.apache.org/jira/browse/YARN-392
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Bikas Saha
>Assignee: Sandy Ryza
> Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch,
> YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392-5.patch,
> YARN-392-6.patch, YARN-392.patch
>
>
> Currently its not possible to specify scheduling requests for specific nodes
> and nowhere else. The RM automatically relaxes locality to rack and * and
> assigns non-specified machines to the app.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13663126#comment-13663126 ] Alejandro Abdelnur commented on YARN-392: - LGTM, +1. [~bikassaha], [~acmurthy] or [~vinodkv], any concern with the latest patch? > Make it possible to specify hard locality constraints in resource requests > -- > > Key: YARN-392 > URL: https://issues.apache.org/jira/browse/YARN-392 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Sandy Ryza > Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch, > YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392-5.patch, > YARN-392-6.patch, YARN-392.patch > > > Currently its not possible to specify scheduling requests for specific nodes > and nowhere else. The RM automatically relaxes locality to rack and * and > assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[
https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13662513#comment-13662513
]
Hadoop QA commented on YARN-392:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12583920/YARN-392-6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/963//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/963//console
This message is automatically generated.
> Make it possible to specify hard locality constraints in resource requests
> --
>
> Key: YARN-392
> URL: https://issues.apache.org/jira/browse/YARN-392
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Bikas Saha
>Assignee: Sandy Ryza
> Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch,
> YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392-5.patch,
> YARN-392-6.patch, YARN-392.patch
>
>
> Currently its not possible to specify scheduling requests for specific nodes
> and nowhere else. The RM automatically relaxes locality to rack and * and
> assigns non-specified machines to the app.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13662493#comment-13662493 ] Sandy Ryza commented on YARN-392: - Looks like I hadn't rebased like I thought I had. Uploaded a new patch. > Make it possible to specify hard locality constraints in resource requests > -- > > Key: YARN-392 > URL: https://issues.apache.org/jira/browse/YARN-392 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Sandy Ryza > Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch, > YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392-5.patch, > YARN-392-6.patch, YARN-392.patch > > > Currently its not possible to specify scheduling requests for specific nodes > and nowhere else. The RM automatically relaxes locality to rack and * and > assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13662471#comment-13662471 ] Alejandro Abdelnur commented on YARN-392: - Glad to see we've reached consensus, thanks. > Make it possible to specify hard locality constraints in resource requests > -- > > Key: YARN-392 > URL: https://issues.apache.org/jira/browse/YARN-392 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Sandy Ryza > Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch, > YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392-5.patch, > YARN-392.patch > > > Currently its not possible to specify scheduling requests for specific nodes > and nowhere else. The RM automatically relaxes locality to rack and * and > assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13662435#comment-13662435 ] Sandy Ryza commented on YARN-392: - Weird, those tests passed locally for me. Will look into it. > Make it possible to specify hard locality constraints in resource requests > -- > > Key: YARN-392 > URL: https://issues.apache.org/jira/browse/YARN-392 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Sandy Ryza > Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch, > YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392-5.patch, > YARN-392.patch > > > Currently its not possible to specify scheduling requests for specific nodes > and nowhere else. The RM automatically relaxes locality to rack and * and > assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[
https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13662395#comment-13662395
]
Hadoop QA commented on YARN-392:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12583887/YARN-392-5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/960//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/960//console
This message is automatically generated.
> Make it possible to specify hard locality constraints in resource requests
> --
>
> Key: YARN-392
> URL: https://issues.apache.org/jira/browse/YARN-392
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Bikas Saha
>Assignee: Sandy Ryza
> Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch,
> YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392-5.patch,
> YARN-392.patch
>
>
> Currently its not possible to specify scheduling requests for specific nodes
> and nowhere else. The RM automatically relaxes locality to rack and * and
> assigns non-specified machines to the app.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13662382#comment-13662382 ] Sandy Ryza commented on YARN-392: - Uploaded a patch that takes away node-level blacklisting and changes the name of the flag accordingly to "relax locality". > Make it possible to specify hard locality constraints in resource requests > -- > > Key: YARN-392 > URL: https://issues.apache.org/jira/browse/YARN-392 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Sandy Ryza > Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch, > YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392-5.patch, > YARN-392.patch > > > Currently its not possible to specify scheduling requests for specific nodes > and nowhere else. The RM automatically relaxes locality to rack and * and > assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13662326#comment-13662326 ] Arun C Murthy commented on YARN-392: bq. The alternative is "if disableAllocation=true for a ResourceRequest at a network hierarchy level, then do not relax locality beyond that level." This is exactly what YARN-398 does too. I buy Bikas's argument that blacklist should be separate from whitelist and it shouldn't be per ResourceRequest. > Make it possible to specify hard locality constraints in resource requests > -- > > Key: YARN-392 > URL: https://issues.apache.org/jira/browse/YARN-392 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Sandy Ryza > Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch, > YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392.patch > > > Currently its not possible to specify scheduling requests for specific nodes > and nowhere else. The RM automatically relaxes locality to rack and * and > assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13662302#comment-13662302 ] Sandy Ryza commented on YARN-392: - The crisp definition I had in mind was: "if disableAllocation=true for a ResourceRequest, don't use that ResourceRequest by itself for an allocation." The alternative is "if disableAllocation=true for a ResourceRequest at a network hierarchy level, then do not relax locality beyond that level." I think both of these are crisp, and I would be ok with both of them. How about: I remove that line in the scheduler that honors the property on node-level requests. We document that the flag is not supported on node-level requests. If we wish to, in a followup JIRA we can decide that setting the flag on a node-level request warrants an exception. Or, in a followup JIRA, we can decide that setting the flag on a node-level request is supported. Neither of these future changes will be a backwards-incompatible change. > Make it possible to specify hard locality constraints in resource requests > -- > > Key: YARN-392 > URL: https://issues.apache.org/jira/browse/YARN-392 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Sandy Ryza > Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch, > YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392.patch > > > Currently its not possible to specify scheduling requests for specific nodes > and nowhere else. The RM automatically relaxes locality to rack and * and > assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13662276#comment-13662276 ] Bikas Saha commented on YARN-392: - [~tucu00] We were in agreement with the delayed locality relaxation approach where we wanted to add a time interval that specifies how long to wait before doing the relaxation. https://issues.apache.org/jira/browse/YARN-392?focusedCommentId=13639491&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13639491 At that point there was no ambiguity about the meaning of the new field. It was then suggested that implementing the time interval version of it would be hard and so we decided to do the boolean version of it. Now the boolean has an ambiguity because in a non-intended way it seems like it would support black listing too. Boolean by definition implies an inverse relation. [~sandyr] I meant blacklisting on an app level (and not cluster wide) vs blacklisting at different RRs. I am not advocating that using the non-intended aspect of the boolean is not useful. What I dont want is additional ambiguity and overload in the protocol. If we can resolve the ambiguity/overload with a single boolean then it would be great to see a crisp definition of it. If not, and if we want to use a boolean based per RR blacklisting then perhaps we can add a relaxLocality flag which affects only locality relaxation and a blacklist flag that does blacklisting per RR. These may seem duplicate but in the users mind and in the scheduler code their definitions are crisp. The new flag would be a different jira though. The way the code is written in the last patch I saw, the only change would be to not check the flag at the node level. As far as the sanity checking is concerned, it would be great to have these and other sanity checks I have already mentioned in previous comments. Sandy responded that those checks would be unnecessary overhead. > Make it possible to specify hard locality constraints in resource requests > -- > > Key: YARN-392 > URL: https://issues.apache.org/jira/browse/YARN-392 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Sandy Ryza > Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch, > YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392.patch > > > Currently its not possible to specify scheduling requests for specific nodes > and nowhere else. The RM automatically relaxes locality to rack and * and > assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13662067#comment-13662067 ] Alejandro Abdelnur commented on YARN-392: - [~ bikassaha], as [~sandyr] points out, blacklisting is a natural consequence of whitelisting and viceversa. The driver of this JIRA is to enable specific node/rack allocation. This is achieved, leveraging the existing protocol/scheduler semantics via a blacklisting of the rack and ALL. As we use the same type of request for node/rack/ALL, doing the blacklisting in the node comes as a freebie, not an intended one. In this case, the AM is the one that decides to do the node blacklisting, it has nothing to do with the health of the node (still, it is the RM's responsibility to take care of this). >From my end, I don't have an immediate use of app node blacklisting. And Sandy >indicates that modifying the scheduler not to handle node blackisting it will >require additional changes. If this is a concern, we could do that as a sanity >check in the RM when resource requests arrive (if I recall correctly, there is >a JIRA for this, I've done a quick search, but I could not find it). > Make it possible to specify hard locality constraints in resource requests > -- > > Key: YARN-392 > URL: https://issues.apache.org/jira/browse/YARN-392 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Sandy Ryza > Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch, > YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392.patch > > > Currently its not possible to specify scheduling requests for specific nodes > and nowhere else. The RM automatically relaxes locality to rack and * and > assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13661767#comment-13661767 ] Sandy Ryza commented on YARN-392: - We currently have a working patch that has gone through multiple phases of review. This patch implements the proposal made by Arun on YARN-398, which many comments led me to believe we had consensus on. The approach enables whitelisting by setting a disable-allocation flag on certain requests, so some form of "blacklisting" is a natural extension of it. The changes to the scheduler are about 10 lines. Modifying the proposal to support *only* whitelisting would require many additional changes, and do nothing to simplify the current changes. As I said, if everyone else participating agrees on these additional changes, I am happy to implement them. But my opinion is that the best way to get this into 2.0.5, both in terms of soundness of the approach and in terms of punctuality, is to go with what we have worked on so far. > Make it possible to specify hard locality constraints in resource requests > -- > > Key: YARN-392 > URL: https://issues.apache.org/jira/browse/YARN-392 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Sandy Ryza > Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch, > YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392.patch > > > Currently its not possible to specify scheduling requests for specific nodes > and nowhere else. The RM automatically relaxes locality to rack and * and > assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13661747#comment-13661747 ] Vinod Kumar Vavilapalli commented on YARN-392: -- bq. The most common use case of black listing is to specify a set of nodes on which no allocations should be made bq. I am not suggesting that this blacklisting mechanism is there to address the most common case. ... IIUC, there is no point in supporting black-listing per resource-type. I don't see a use-case for it. When you blacklist a node or a rack, you blacklist it. You don't blacklist it for 5GB,5core containers but want to use it for 1GB/1core container. Still catching up the discussion. But wanted to say that this has gone on for too long. We should try and get this into 2.0.5. Sandy/Bikas, can we just focus this for 'white-listing- per resource type through the flag that was proposed (and seems to be the consensus earlier) and use YARN-395 for blacklisting. I can close YARN-398 as duplicate. > Make it possible to specify hard locality constraints in resource requests > -- > > Key: YARN-392 > URL: https://issues.apache.org/jira/browse/YARN-392 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Sandy Ryza > Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch, > YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392.patch > > > Currently its not possible to specify scheduling requests for specific nodes > and nowhere else. The RM automatically relaxes locality to rack and * and > assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13661705#comment-13661705 ] Sandy Ryza commented on YARN-392: - As I mentioned, any mechanism that allows whitelisting also allows blacklisting by definition, as it is always possible to whitelist all the nodes but the one that one doesn't want. So I don't see it as overloading. bq. The most common use case of black listing is to specify a set of nodes on which no allocations should be made I am not suggesting that this blacklisting mechanism is there to address the most common case. In the same way that the most common use of delay scheduling is probably a cluster-wide setting, but allowing customization on specific requests in the way that you suggested earlier on this thread would still be useful, the ability to blacklist nodes for specific requests does not preclude a cluster-wide setting to address the common case. Does the following seem like a fair representation of the mechanics of the alternative? When a node-level request comes with disableAllocation=true, an InvalidAllocationException is thrown. When a rack-level request comes with disableAllocation=true, we check to make sure that there are nodes under it. If not, an InvalidAllocationException is thrown. When a node-level request is cancelled, we check the rack above it to make sure that if its disableAllocation=true, there are other non-zero node-level requests below it. If not, we throw an InvalidAllocationException. To me, this seems both more complicated and gives up functionality unnecessarily. That said, if we can get some consensus on an alternative, I am happy to implement that instead. > Make it possible to specify hard locality constraints in resource requests > -- > > Key: YARN-392 > URL: https://issues.apache.org/jira/browse/YARN-392 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Sandy Ryza > Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch, > YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392.patch > > > Currently its not possible to specify scheduling requests for specific nodes > and nowhere else. The RM automatically relaxes locality to rack and * and > assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13661513#comment-13661513 ] Bikas Saha commented on YARN-392: - I am not sure how we are making the semantics consistent by overloading something for 2 things. When the flag is set at a network hierarchy level then it means then scheduler will not relax locality beyond that level. The same flag can also be used to blacklist locations. The most common use case of black listing is to specify a set of nodes on which no allocations should be made (eg they are badly behaving nodes). How does this scheme address that case? Will we have to specify the same blacklist information for every priority that is used by an application (because resource request is per priority). Every time an app uses a new priority we will have to issue a new set of resource requests to blacklist at that priority? > Make it possible to specify hard locality constraints in resource requests > -- > > Key: YARN-392 > URL: https://issues.apache.org/jira/browse/YARN-392 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Sandy Ryza > Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch, > YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392.patch > > > Currently its not possible to specify scheduling requests for specific nodes > and nowhere else. The RM automatically relaxes locality to rack and * and > assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13661427#comment-13661427 ] Alejandro Abdelnur commented on YARN-392: - latest patch LGTM. Bikas, does Sandy's responses address your concerns? I'd like to get this in so we can move to the next step which is getting this exposed in the client API. > Make it possible to specify hard locality constraints in resource requests > -- > > Key: YARN-392 > URL: https://issues.apache.org/jira/browse/YARN-392 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Sandy Ryza > Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch, > YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392.patch > > > Currently its not possible to specify scheduling requests for specific nodes > and nowhere else. The RM automatically relaxes locality to rack and * and > assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-392) Make it possible to specify hard locality constraints in resource requests
[ https://issues.apache.org/jira/browse/YARN-392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13661403#comment-13661403 ] Sandy Ryza commented on YARN-392: - bq. it would still be possible to blacklist racks by setting the disable flag on a rack and submitting node requests for nodes under it. By which I mean: it would still be possible to blacklist racks by setting the disable flag on a rack and submitting *no" node requests for nodes under it. > Make it possible to specify hard locality constraints in resource requests > -- > > Key: YARN-392 > URL: https://issues.apache.org/jira/browse/YARN-392 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Sandy Ryza > Attachments: YARN-392-1.patch, YARN-392-2.patch, YARN-392-2.patch, > YARN-392-2.patch, YARN-392-3.patch, YARN-392-4.patch, YARN-392.patch > > > Currently its not possible to specify scheduling requests for specific nodes > and nowhere else. The RM automatically relaxes locality to rack and * and > assigns non-specified machines to the app. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
