[jira] [Commented] (YARN-3494) Expose AM resource limit and user limit in QueueMetrics
[
https://issues.apache.org/jira/browse/YARN-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504958#comment-14504958
]
Hadoop QA commented on YARN-3494:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12726861/0002-YARN-3494.patch
against trunk revision 8ddbb8d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/7423//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7423//console
This message is automatically generated.
Expose AM resource limit and user limit in QueueMetrics
Key: YARN-3494
URL: https://issues.apache.org/jira/browse/YARN-3494
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Jian He
Assignee: Rohith
Attachments: 0001-YARN-3494.patch, 0002-YARN-3494.patch,
0002-YARN-3494.patch
Now we have the AM resource limit and user limit shown on the web UI, it
would be useful to expose them in the QueueMetrics as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3494) Expose AM resource limit and user limit in QueueMetrics
[
https://issues.apache.org/jira/browse/YARN-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504830#comment-14504830
]
Hadoop QA commented on YARN-3494:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12726858/0002-YARN-3494.patch
against trunk revision 8ddbb8d.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7422//console
This message is automatically generated.
Expose AM resource limit and user limit in QueueMetrics
Key: YARN-3494
URL: https://issues.apache.org/jira/browse/YARN-3494
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Jian He
Assignee: Rohith
Attachments: 0001-YARN-3494.patch, 0002-YARN-3494.patch
Now we have the AM resource limit and user limit shown on the web UI, it
would be useful to expose them in the QueueMetrics as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3494) Expose AM resource limit and user limit in QueueMetrics
[ https://issues.apache.org/jira/browse/YARN-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14506355#comment-14506355 ] Hudson commented on YARN-3494: -- FAILURE: Integrated in Hadoop-trunk-Commit #7632 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7632/]) YARN-3494. Expose AM resource limit and usage in CS QueueMetrics. Contributed by Rohith Sharmaks (jianhe: rev bdd90110e6904b59746812d9a093924a65e72280) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueMetrics.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java Expose AM resource limit and user limit in QueueMetrics Key: YARN-3494 URL: https://issues.apache.org/jira/browse/YARN-3494 Project: Hadoop YARN Issue Type: Improvement Reporter: Jian He Assignee: Rohith Fix For: 2.8.0 Attachments: 0001-YARN-3494.patch, 0002-YARN-3494.patch, 0002-YARN-3494.patch Now we have the AM resource limit and user limit shown on the web UI, it would be useful to expose them in the QueueMetrics as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3494) Expose AM resource limit and user limit in QueueMetrics
[
https://issues.apache.org/jira/browse/YARN-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14503751#comment-14503751
]
Jian He commented on YARN-3494:
---
Rohith, thanks for working on this !
couple comments:
- remove below ?
{code}
stats.setReservedVCores(getMetrics().getAMResourceLimitMB());
stats.setReservedVCores(getMetrics().getAMResourceLimitVCores());
stats.setReservedVCores(getMetrics().getUsedAMResourceMB());
stats.setReservedVCores(getMetrics().getUsedAMResourceVCores());
stats.setReservedVCores(getMetrics().getUserAMResourceLimitMB());
stats.setReservedVCores(getMetrics().getUserAMResourceLimitVCores());
{code}
- usedAMResource for a given user seems not updated. Existing code re-uses the
QueueMetrics for userMetrics. Following that, we may not need separate
userAMResourceLimitMB/userAMResourceLimitVCores, we can use the
AMResourceLimitMB and AMResourceLimitVCores
Expose AM resource limit and user limit in QueueMetrics
Key: YARN-3494
URL: https://issues.apache.org/jira/browse/YARN-3494
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Jian He
Assignee: Rohith
Attachments: 0001-YARN-3494.patch
Now we have the AM resource limit and user limit shown on the web UI, it
would be useful to expose them in the QueueMetrics as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3494) Expose AM resource limit and user limit in QueueMetrics
[
https://issues.apache.org/jira/browse/YARN-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14502707#comment-14502707
]
Rohith commented on YARN-3494:
--
One self review is move below code to setupQueueConfigs. Will be adding in
next patch
{noformat}
// update static CSQueueMetrics
getMetrics().setAMResouceLimit(getAMResourceLimit());
getMetrics().setUserAMResouceLimit(getUserAMResourceLimit());
{noformat}
Expose AM resource limit and user limit in QueueMetrics
Key: YARN-3494
URL: https://issues.apache.org/jira/browse/YARN-3494
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Jian He
Assignee: Rohith
Attachments: 0001-YARN-3494.patch
Now we have the AM resource limit and user limit shown on the web UI, it
would be useful to expose them in the QueueMetrics as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3494) Expose AM resource limit and user limit in QueueMetrics
[
https://issues.apache.org/jira/browse/YARN-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14502713#comment-14502713
]
Hadoop QA commented on YARN-3494:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12726544/0001-YARN-3494.patch
against trunk revision c17cd4f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/7395//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7395//console
This message is automatically generated.
Expose AM resource limit and user limit in QueueMetrics
Key: YARN-3494
URL: https://issues.apache.org/jira/browse/YARN-3494
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Jian He
Assignee: Rohith
Attachments: 0001-YARN-3494.patch
Now we have the AM resource limit and user limit shown on the web UI, it
would be useful to expose them in the QueueMetrics as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3494) Expose AM resource limit and user limit in QueueMetrics
[ https://issues.apache.org/jira/browse/YARN-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14502635#comment-14502635 ] Rohith commented on YARN-3494: -- Updated the patch adding metric for AMResourceLimit, UserAMResourceLimit and AMUsed per queue. Userlimit metric is already available in QueueMetric. Kindly review the patch Expose AM resource limit and user limit in QueueMetrics Key: YARN-3494 URL: https://issues.apache.org/jira/browse/YARN-3494 Project: Hadoop YARN Issue Type: Improvement Reporter: Jian He Assignee: Rohith Attachments: 0001-YARN-3494.patch Now we have the AM resource limit and user limit shown on the web UI, it would be useful to expose them in the QueueMetrics as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3494) Expose AM resource limit and user limit in QueueMetrics
[ https://issues.apache.org/jira/browse/YARN-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497807#comment-14497807 ] Rohith commented on YARN-3494: -- AM Resource Limit and User Limit are specific to CapacityScheduler. Instead of adding in QueuMetrics, how about adding in CSQueueMetrics? This CSQueueMetrics can have many more CS specific details. Any thoughts? Expose AM resource limit and user limit in QueueMetrics Key: YARN-3494 URL: https://issues.apache.org/jira/browse/YARN-3494 Project: Hadoop YARN Issue Type: Bug Reporter: Jian He Assignee: Rohith Now we have the AM resource limit and user limit shown on the web UI, it would be useful to expose them in the QueueMetrics as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3494) Expose AM resource limit and user limit in QueueMetrics
[ https://issues.apache.org/jira/browse/YARN-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14498306#comment-14498306 ] Sunil G commented on YARN-3494: --- Could we define the metric in QueueMetrics, but the setter for same can be updated from CSQueueUtils. And can be used only by CS for now. Expose AM resource limit and user limit in QueueMetrics Key: YARN-3494 URL: https://issues.apache.org/jira/browse/YARN-3494 Project: Hadoop YARN Issue Type: Bug Reporter: Jian He Assignee: Rohith Now we have the AM resource limit and user limit shown on the web UI, it would be useful to expose them in the QueueMetrics as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3494) Expose AM resource limit and user limit in QueueMetrics
[ https://issues.apache.org/jira/browse/YARN-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14498662#comment-14498662 ] Jian He commented on YARN-3494: --- I think AM resource usage and limit can be in the QueueMetrics, since FS also has the AM resource limit notion. We can put the user usage and limit in CSQueueMetrics, similarly FS has the FSQueueMetrics class for their FS queue metrics specific bits. Expose AM resource limit and user limit in QueueMetrics Key: YARN-3494 URL: https://issues.apache.org/jira/browse/YARN-3494 Project: Hadoop YARN Issue Type: Bug Reporter: Jian He Assignee: Rohith Now we have the AM resource limit and user limit shown on the web UI, it would be useful to expose them in the QueueMetrics as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
