[jira] [Commented] (YARN-3494) Expose AM resource limit and user limit in QueueMetrics
[ https://issues.apache.org/jira/browse/YARN-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14506355#comment-14506355 ] Hudson commented on YARN-3494: -- FAILURE: Integrated in Hadoop-trunk-Commit #7632 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7632/]) YARN-3494. Expose AM resource limit and usage in CS QueueMetrics. Contributed by Rohith Sharmaks (jianhe: rev bdd90110e6904b59746812d9a093924a65e72280) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueMetrics.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java > Expose AM resource limit and user limit in QueueMetrics > > > Key: YARN-3494 > URL: https://issues.apache.org/jira/browse/YARN-3494 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jian He >Assignee: Rohith > Fix For: 2.8.0 > > Attachments: 0001-YARN-3494.patch, 0002-YARN-3494.patch, > 0002-YARN-3494.patch > > > Now we have the AM resource limit and user limit shown on the web UI, it > would be useful to expose them in the QueueMetrics as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3494) Expose AM resource limit and user limit in QueueMetrics
[ https://issues.apache.org/jira/browse/YARN-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14504958#comment-14504958 ] Hadoop QA commented on YARN-3494: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12726861/0002-YARN-3494.patch against trunk revision 8ddbb8d. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7423//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7423//console This message is automatically generated. > Expose AM resource limit and user limit in QueueMetrics > > > Key: YARN-3494 > URL: https://issues.apache.org/jira/browse/YARN-3494 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jian He >Assignee: Rohith > Attachments: 0001-YARN-3494.patch, 0002-YARN-3494.patch, > 0002-YARN-3494.patch > > > Now we have the AM resource limit and user limit shown on the web UI, it > would be useful to expose them in the QueueMetrics as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3494) Expose AM resource limit and user limit in QueueMetrics
[ https://issues.apache.org/jira/browse/YARN-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14504831#comment-14504831 ] Rohith commented on YARN-3494: -- Updated the patch fixing review comment, kindly review it. > Expose AM resource limit and user limit in QueueMetrics > > > Key: YARN-3494 > URL: https://issues.apache.org/jira/browse/YARN-3494 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jian He >Assignee: Rohith > Attachments: 0001-YARN-3494.patch, 0002-YARN-3494.patch > > > Now we have the AM resource limit and user limit shown on the web UI, it > would be useful to expose them in the QueueMetrics as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3494) Expose AM resource limit and user limit in QueueMetrics
[ https://issues.apache.org/jira/browse/YARN-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14504830#comment-14504830 ] Hadoop QA commented on YARN-3494: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12726858/0002-YARN-3494.patch against trunk revision 8ddbb8d. {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7422//console This message is automatically generated. > Expose AM resource limit and user limit in QueueMetrics > > > Key: YARN-3494 > URL: https://issues.apache.org/jira/browse/YARN-3494 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jian He >Assignee: Rohith > Attachments: 0001-YARN-3494.patch, 0002-YARN-3494.patch > > > Now we have the AM resource limit and user limit shown on the web UI, it > would be useful to expose them in the QueueMetrics as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3494) Expose AM resource limit and user limit in QueueMetrics
[ https://issues.apache.org/jira/browse/YARN-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14503751#comment-14503751 ] Jian He commented on YARN-3494: --- Rohith, thanks for working on this ! couple comments: - remove below ? {code} stats.setReservedVCores(getMetrics().getAMResourceLimitMB()); stats.setReservedVCores(getMetrics().getAMResourceLimitVCores()); stats.setReservedVCores(getMetrics().getUsedAMResourceMB()); stats.setReservedVCores(getMetrics().getUsedAMResourceVCores()); stats.setReservedVCores(getMetrics().getUserAMResourceLimitMB()); stats.setReservedVCores(getMetrics().getUserAMResourceLimitVCores()); {code} - usedAMResource for a given user seems not updated. Existing code re-uses the QueueMetrics for userMetrics. Following that, we may not need separate userAMResourceLimitMB/userAMResourceLimitVCores, we can use the AMResourceLimitMB and AMResourceLimitVCores > Expose AM resource limit and user limit in QueueMetrics > > > Key: YARN-3494 > URL: https://issues.apache.org/jira/browse/YARN-3494 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jian He >Assignee: Rohith > Attachments: 0001-YARN-3494.patch > > > Now we have the AM resource limit and user limit shown on the web UI, it > would be useful to expose them in the QueueMetrics as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3494) Expose AM resource limit and user limit in QueueMetrics
[ https://issues.apache.org/jira/browse/YARN-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14502713#comment-14502713 ] Hadoop QA commented on YARN-3494: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12726544/0001-YARN-3494.patch against trunk revision c17cd4f. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7395//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7395//console This message is automatically generated. > Expose AM resource limit and user limit in QueueMetrics > > > Key: YARN-3494 > URL: https://issues.apache.org/jira/browse/YARN-3494 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jian He >Assignee: Rohith > Attachments: 0001-YARN-3494.patch > > > Now we have the AM resource limit and user limit shown on the web UI, it > would be useful to expose them in the QueueMetrics as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3494) Expose AM resource limit and user limit in QueueMetrics
[ https://issues.apache.org/jira/browse/YARN-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14502707#comment-14502707 ] Rohith commented on YARN-3494: -- One self review is move below code to setupQueueConfigs. Will be adding in next patch {noformat} // update static CSQueueMetrics getMetrics().setAMResouceLimit(getAMResourceLimit()); getMetrics().setUserAMResouceLimit(getUserAMResourceLimit()); {noformat} > Expose AM resource limit and user limit in QueueMetrics > > > Key: YARN-3494 > URL: https://issues.apache.org/jira/browse/YARN-3494 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jian He >Assignee: Rohith > Attachments: 0001-YARN-3494.patch > > > Now we have the AM resource limit and user limit shown on the web UI, it > would be useful to expose them in the QueueMetrics as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3494) Expose AM resource limit and user limit in QueueMetrics
[ https://issues.apache.org/jira/browse/YARN-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14502635#comment-14502635 ] Rohith commented on YARN-3494: -- Updated the patch adding metric for AMResourceLimit, UserAMResourceLimit and AMUsed per queue. Userlimit metric is already available in QueueMetric. Kindly review the patch > Expose AM resource limit and user limit in QueueMetrics > > > Key: YARN-3494 > URL: https://issues.apache.org/jira/browse/YARN-3494 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jian He >Assignee: Rohith > Attachments: 0001-YARN-3494.patch > > > Now we have the AM resource limit and user limit shown on the web UI, it > would be useful to expose them in the QueueMetrics as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3494) Expose AM resource limit and user limit in QueueMetrics
[ https://issues.apache.org/jira/browse/YARN-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14498662#comment-14498662 ] Jian He commented on YARN-3494: --- I think AM resource usage and limit can be in the QueueMetrics, since FS also has the AM resource limit notion. We can put the user usage and limit in CSQueueMetrics, similarly FS has the FSQueueMetrics class for their FS queue metrics specific bits. > Expose AM resource limit and user limit in QueueMetrics > > > Key: YARN-3494 > URL: https://issues.apache.org/jira/browse/YARN-3494 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Rohith > > Now we have the AM resource limit and user limit shown on the web UI, it > would be useful to expose them in the QueueMetrics as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3494) Expose AM resource limit and user limit in QueueMetrics
[ https://issues.apache.org/jira/browse/YARN-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14498306#comment-14498306 ] Sunil G commented on YARN-3494: --- Could we define the metric in QueueMetrics, but the setter for same can be updated from CSQueueUtils. And can be used only by CS for now. > Expose AM resource limit and user limit in QueueMetrics > > > Key: YARN-3494 > URL: https://issues.apache.org/jira/browse/YARN-3494 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Rohith > > Now we have the AM resource limit and user limit shown on the web UI, it > would be useful to expose them in the QueueMetrics as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3494) Expose AM resource limit and user limit in QueueMetrics
[ https://issues.apache.org/jira/browse/YARN-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14497807#comment-14497807 ] Rohith commented on YARN-3494: -- AM Resource Limit and User Limit are specific to CapacityScheduler. Instead of adding in QueuMetrics, how about adding in CSQueueMetrics? This CSQueueMetrics can have many more CS specific details. Any thoughts? > Expose AM resource limit and user limit in QueueMetrics > > > Key: YARN-3494 > URL: https://issues.apache.org/jira/browse/YARN-3494 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Rohith > > Now we have the AM resource limit and user limit shown on the web UI, it > would be useful to expose them in the QueueMetrics as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)