[jira] [Commented] (YARN-2448) RM should expose the name of the ResourceCalculator being used when AMs register
[ https://issues.apache.org/jira/browse/YARN-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14118402#comment-14118402 ] Vinod Kumar Vavilapalli commented on YARN-2448: --- I believe YARN-2463 was filed in conjunction with this. Linking them so as to resolve them together. Anyways, I too agree we shouldn't pass the entire calculator - it's a server artifact - the apps shouldn't really need to figure out which resource is dominant etc. May be we need a smaller subset to be a public interface which only does the comparison math between resources? If that is not needed, and as you said, if we just want to return the resource types that the scheduler considers as part of its functioning, we may as well expose the RM configuration that says which resources are respected. RM should expose the name of the ResourceCalculator being used when AMs register Key: YARN-2448 URL: https://issues.apache.org/jira/browse/YARN-2448 Project: Hadoop YARN Issue Type: Improvement Reporter: Varun Vasudev Assignee: Varun Vasudev Attachments: apache-yarn-2448.0.patch, apache-yarn-2448.1.patch The RM should expose the name of the ResourceCalculator being used when AMs register, as part of the RegisterApplicationMasterResponse. This will allow applications to make better decisions when scheduling. MapReduce for example, only looks at memory when deciding it's scheduling, even though the RM could potentially be using the DominantResourceCalculator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2448) RM should expose the name of the ResourceCalculator being used when AMs register
[ https://issues.apache.org/jira/browse/YARN-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117506#comment-14117506 ] Varun Vasudev commented on YARN-2448: - [~sandyr], [~kasha] thanks for your extremely helpful input. I think what [~sandyr] is suggesting should be ok. Is it ok to generalize it to return a representation of the resource types that the scheduler considers as part of its functioning? So that in the future if we add support for more resource types, we don't have to change much? RM should expose the name of the ResourceCalculator being used when AMs register Key: YARN-2448 URL: https://issues.apache.org/jira/browse/YARN-2448 Project: Hadoop YARN Issue Type: Improvement Reporter: Varun Vasudev Assignee: Varun Vasudev Attachments: apache-yarn-2448.0.patch, apache-yarn-2448.1.patch The RM should expose the name of the ResourceCalculator being used when AMs register, as part of the RegisterApplicationMasterResponse. This will allow applications to make better decisions when scheduling. MapReduce for example, only looks at memory when deciding it's scheduling, even though the RM could potentially be using the DominantResourceCalculator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2448) RM should expose the name of the ResourceCalculator being used when AMs register
[ https://issues.apache.org/jira/browse/YARN-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14116828#comment-14116828 ] Sandy Ryza commented on YARN-2448: -- As Karthik mentioned, the ResourceCalculator is an abstraction used by the Capacity Scheduler that isn't a great fit for the Fair Scheduler, which always enforces CPU limits but can be configured with a different fairness policy at each queue in the hierarchy. If this is necessary, can we provide a narrower interface such as a boolean indicating whether the scheduler considers CPU in its decisions? RM should expose the name of the ResourceCalculator being used when AMs register Key: YARN-2448 URL: https://issues.apache.org/jira/browse/YARN-2448 Project: Hadoop YARN Issue Type: Improvement Reporter: Varun Vasudev Assignee: Varun Vasudev Attachments: apache-yarn-2448.0.patch, apache-yarn-2448.1.patch The RM should expose the name of the ResourceCalculator being used when AMs register, as part of the RegisterApplicationMasterResponse. This will allow applications to make better decisions when scheduling. MapReduce for example, only looks at memory when deciding it's scheduling, even though the RM could potentially be using the DominantResourceCalculator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2448) RM should expose the name of the ResourceCalculator being used when AMs register
[ https://issues.apache.org/jira/browse/YARN-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113344#comment-14113344 ] Jian He commented on YARN-2448: --- looks good overall, thinking if it makes sense to use enum type to indicate the resource calculator type instead of returning class name, like adding an enum field into DefaultResourceCalculator class, because we can't rename these classes anymore if we choose to expose class name. RM should expose the name of the ResourceCalculator being used when AMs register Key: YARN-2448 URL: https://issues.apache.org/jira/browse/YARN-2448 Project: Hadoop YARN Issue Type: Improvement Reporter: Varun Vasudev Assignee: Varun Vasudev Attachments: apache-yarn-2448.0.patch, apache-yarn-2448.1.patch The RM should expose the name of the ResourceCalculator being used when AMs register, as part of the RegisterApplicationMasterResponse. This will allow applications to make better decisions when scheduling. MapReduce for example, only looks at memory when deciding it's scheduling, even though the RM could potentially be using the DominantResourceCalculator. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2448) RM should expose the name of the ResourceCalculator being used when AMs register
[ https://issues.apache.org/jira/browse/YARN-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14108822#comment-14108822 ] Wangda Tan commented on YARN-2448: -- [~vvasudev], Thanks for working on the patch, it is LGTM, +1 Wangda RM should expose the name of the ResourceCalculator being used when AMs register Key: YARN-2448 URL: https://issues.apache.org/jira/browse/YARN-2448 Project: Hadoop YARN Issue Type: Improvement Reporter: Varun Vasudev Assignee: Varun Vasudev Attachments: apache-yarn-2448.0.patch, apache-yarn-2448.1.patch The RM should expose the name of the ResourceCalculator being used when AMs register, as part of the RegisterApplicationMasterResponse. This will allow applications to make better decisions when scheduling. MapReduce for example, only looks at memory when deciding it's scheduling, even though the RM could potentially be using the DominantResourceCalculator. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2448) RM should expose the name of the ResourceCalculator being used when AMs register
[
https://issues.apache.org/jira/browse/YARN-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14108845#comment-14108845
]
Hadoop QA commented on YARN-2448:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12664105/apache-yarn-2448.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-tools/hadoop-sls hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.sls.TestSLSRunner
org.apache.hadoop.yarn.sls.nodemanager.TestNMSimulator
org.apache.hadoop.yarn.sls.appmaster.TestAMSimulator
org.apache.hadoop.yarn.server.resourcemanager.security.TestClientToAMTokens
org.apache.hadoop.yarn.server.resourcemanager.security.TestAMRMTokens
org.apache.hadoop.yarn.server.resourcemanager.TestMoveApplication
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4717//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4717//console
This message is automatically generated.
RM should expose the name of the ResourceCalculator being used when AMs
register
Key: YARN-2448
URL: https://issues.apache.org/jira/browse/YARN-2448
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Varun Vasudev
Assignee: Varun Vasudev
Attachments: apache-yarn-2448.0.patch, apache-yarn-2448.1.patch
The RM should expose the name of the ResourceCalculator being used when AMs
register, as part of the RegisterApplicationMasterResponse.
This will allow applications to make better decisions when scheduling.
MapReduce for example, only looks at memory when deciding it's scheduling,
even though the RM could potentially be using the DominantResourceCalculator.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2448) RM should expose the name of the ResourceCalculator being used when AMs register
[ https://issues.apache.org/jira/browse/YARN-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14109577#comment-14109577 ] Karthik Kambatla commented on YARN-2448: I am not sure I understand the usecase very well. The AM's requirements shouldn't change based on what the RM does internally. Shouldn't the application ask for all the resources that YARN supports? It is upto the scheduler (queue, user, app type etc.) to decide on what resources it would consider for scheduling. If the app doesn't specify any resources at all for a type, we can assume zero for that type (e.g. in clusters not configured to use a particular type). RM should expose the name of the ResourceCalculator being used when AMs register Key: YARN-2448 URL: https://issues.apache.org/jira/browse/YARN-2448 Project: Hadoop YARN Issue Type: Improvement Reporter: Varun Vasudev Assignee: Varun Vasudev Attachments: apache-yarn-2448.0.patch, apache-yarn-2448.1.patch The RM should expose the name of the ResourceCalculator being used when AMs register, as part of the RegisterApplicationMasterResponse. This will allow applications to make better decisions when scheduling. MapReduce for example, only looks at memory when deciding it's scheduling, even though the RM could potentially be using the DominantResourceCalculator. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2448) RM should expose the name of the ResourceCalculator being used when AMs register
[ https://issues.apache.org/jira/browse/YARN-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14109589#comment-14109589 ] Varun Vasudev commented on YARN-2448: - The use case that springs to mind is adding support for cpu to map-reduce. Currently the map-reduce AM only looks at memory when it is deciding things like pre-empting reducers. If we wish to add support for cpu as a resource to map-reduce, it needs to consider vcores as well. However, if the YARN scheduler if using the DefaultResourceCalculator, which ignores cpu, and the map-reduce AM doesn't know this, it leads to inefficient asks and allocations. The aim is just to let the AM know which calculator is being used and let the AM go from there. Does that make sense? RM should expose the name of the ResourceCalculator being used when AMs register Key: YARN-2448 URL: https://issues.apache.org/jira/browse/YARN-2448 Project: Hadoop YARN Issue Type: Improvement Reporter: Varun Vasudev Assignee: Varun Vasudev Attachments: apache-yarn-2448.0.patch, apache-yarn-2448.1.patch The RM should expose the name of the ResourceCalculator being used when AMs register, as part of the RegisterApplicationMasterResponse. This will allow applications to make better decisions when scheduling. MapReduce for example, only looks at memory when deciding it's scheduling, even though the RM could potentially be using the DominantResourceCalculator. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2448) RM should expose the name of the ResourceCalculator being used when AMs register
[ https://issues.apache.org/jira/browse/YARN-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14109603#comment-14109603 ] Karthik Kambatla commented on YARN-2448: May be I am missing your point. Why not have the MR AM request both CPU and memory? If the RM/scheduler doesn't consider CPU, it will just ignore it. Related, but orthogonal point: In the case of FairScheduler, the policy depends on the queue the app is submitted to. So, some queues might consider only CPU, some only memory, and some both. So, exposing the ResourceCalculator doesn't really tell the AM anything, it has to look at the queue configuration. RM should expose the name of the ResourceCalculator being used when AMs register Key: YARN-2448 URL: https://issues.apache.org/jira/browse/YARN-2448 Project: Hadoop YARN Issue Type: Improvement Reporter: Varun Vasudev Assignee: Varun Vasudev Attachments: apache-yarn-2448.0.patch, apache-yarn-2448.1.patch The RM should expose the name of the ResourceCalculator being used when AMs register, as part of the RegisterApplicationMasterResponse. This will allow applications to make better decisions when scheduling. MapReduce for example, only looks at memory when deciding it's scheduling, even though the RM could potentially be using the DominantResourceCalculator. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2448) RM should expose the name of the ResourceCalculator being used when AMs register
[
https://issues.apache.org/jira/browse/YARN-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14110319#comment-14110319
]
Hadoop QA commented on YARN-2448:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12664105/apache-yarn-2448.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The test build failed in
hadoop-tools/hadoop-sls
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4729//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4729//console
This message is automatically generated.
RM should expose the name of the ResourceCalculator being used when AMs
register
Key: YARN-2448
URL: https://issues.apache.org/jira/browse/YARN-2448
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Varun Vasudev
Assignee: Varun Vasudev
Attachments: apache-yarn-2448.0.patch, apache-yarn-2448.1.patch
The RM should expose the name of the ResourceCalculator being used when AMs
register, as part of the RegisterApplicationMasterResponse.
This will allow applications to make better decisions when scheduling.
MapReduce for example, only looks at memory when deciding it's scheduling,
even though the RM could potentially be using the DominantResourceCalculator.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2448) RM should expose the name of the ResourceCalculator being used when AMs register
[
https://issues.apache.org/jira/browse/YARN-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14108750#comment-14108750
]
Hadoop QA commented on YARN-2448:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12664094/apache-yarn-2448.0.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4715//console
This message is automatically generated.
RM should expose the name of the ResourceCalculator being used when AMs
register
Key: YARN-2448
URL: https://issues.apache.org/jira/browse/YARN-2448
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Varun Vasudev
Assignee: Varun Vasudev
Attachments: apache-yarn-2448.0.patch
The RM should expose the name of the ResourceCalculator being used when AMs
register, as part of the RegisterApplicationMasterResponse.
This will allow applications to make better decisions when scheduling.
MapReduce for example, only looks at memory when deciding it's scheduling,
even though the RM could potentially be using the DominantResourceCalculator.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
