[jira] [Commented] (YARN-4285) Display resource usage as percentage of queue and cluster in the RM UI
[ https://issues.apache.org/jira/browse/YARN-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210375#comment-15210375 ] Sunil G commented on YARN-4285: --- Yes [~leftnoteasy] and [~vvasudev] It will be good if we can show them separately as we already showing this resources separately in cluster metrics UI. Also Wangda's comment looks fine, which s to show 2 resource in 2 colors in same queue bar. I making some changes to try this. I will update. > Display resource usage as percentage of queue and cluster in the RM UI > -- > > Key: YARN-4285 > URL: https://issues.apache.org/jira/browse/YARN-4285 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 2.8.0 > > Attachments: YARN-4285.001.patch, YARN-4285.002.patch, > YARN-4285.003.patch, YARN-4285.004.patch > > > Currently, we display the memory and vcores allocated to an app in the RM UI. > It would be useful to display the resources consumed as a %of the queue and > the cluster to identify apps that are using a lot of resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4285) Display resource usage as percentage of queue and cluster in the RM UI
[ https://issues.apache.org/jira/browse/YARN-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210254#comment-15210254 ] Varun Vasudev commented on YARN-4285: - Thanks for the explanation [~leftnoteasy]. So it looks like for internal accounting purposes we should count reserved containers as used resources but from a web UI perspective, we should subtract reserved resources and show them seperately? > Display resource usage as percentage of queue and cluster in the RM UI > -- > > Key: YARN-4285 > URL: https://issues.apache.org/jira/browse/YARN-4285 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 2.8.0 > > Attachments: YARN-4285.001.patch, YARN-4285.002.patch, > YARN-4285.003.patch, YARN-4285.004.patch > > > Currently, we display the memory and vcores allocated to an app in the RM UI. > It would be useful to display the resources consumed as a %of the queue and > the cluster to identify apps that are using a lot of resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4285) Display resource usage as percentage of queue and cluster in the RM UI
[ https://issues.apache.org/jira/browse/YARN-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209449#comment-15209449 ] Wangda Tan commented on YARN-4285: -- [~jianhe], bq. However, the queue's used resource in the UI does include reserved resource too. IIUC, it should queue's used resource should include reserved resources [~vvasudev], bq. it makes sense to remove reserved resources from the used resources, Actually I think we should include reserved resources by used resources, unless we can show them together on UI. See my [#2 comment|https://issues.apache.org/jira/browse/YARN-4678?focusedCommentId=15209365=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15209365]. bq. ...but do we know why we counted reserved resources as part of used resources in the first place? The reason is, if we create a reserved container under a queue, we need to make sure it doesn't go beyond queue's max capacity. In another word, if resource is reserved by someone, nobody else can use that part of resources. >From YARN's perspective, a queue has 99G allocated (not reserved) + 1G >reserved is as same as 1G allocated + 99G reserved. To be more transparent to >users and avoid answer questions like: "why my total allocated resource is >always less than total resources, used resource should be allocated + reserved. > Display resource usage as percentage of queue and cluster in the RM UI > -- > > Key: YARN-4285 > URL: https://issues.apache.org/jira/browse/YARN-4285 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 2.8.0 > > Attachments: YARN-4285.001.patch, YARN-4285.002.patch, > YARN-4285.003.patch, YARN-4285.004.patch > > > Currently, we display the memory and vcores allocated to an app in the RM UI. > It would be useful to display the resources consumed as a %of the queue and > the cluster to identify apps that are using a lot of resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4285) Display resource usage as percentage of queue and cluster in the RM UI
[ https://issues.apache.org/jira/browse/YARN-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207991#comment-15207991 ] Varun Vasudev commented on YARN-4285: - [~jianhe] - it makes sense to remove reserved resources from the used resources, but do we know why we counted reserved resources as part of used resources in the first place? > Display resource usage as percentage of queue and cluster in the RM UI > -- > > Key: YARN-4285 > URL: https://issues.apache.org/jira/browse/YARN-4285 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 2.8.0 > > Attachments: YARN-4285.001.patch, YARN-4285.002.patch, > YARN-4285.003.patch, YARN-4285.004.patch > > > Currently, we display the memory and vcores allocated to an app in the RM UI. > It would be useful to display the resources consumed as a %of the queue and > the cluster to identify apps that are using a lot of resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4285) Display resource usage as percentage of queue and cluster in the RM UI
[ https://issues.apache.org/jira/browse/YARN-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15205650#comment-15205650 ] Sunil G commented on YARN-4285: --- Hi [~jianhe], in YARN-4678, we were trying to take reserved capacity out of used capacity in UI. And given fox for same. Could you also pls take a look. > Display resource usage as percentage of queue and cluster in the RM UI > -- > > Key: YARN-4285 > URL: https://issues.apache.org/jira/browse/YARN-4285 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 2.8.0 > > Attachments: YARN-4285.001.patch, YARN-4285.002.patch, > YARN-4285.003.patch, YARN-4285.004.patch > > > Currently, we display the memory and vcores allocated to an app in the RM UI. > It would be useful to display the resources consumed as a %of the queue and > the cluster to identify apps that are using a lot of resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4285) Display resource usage as percentage of queue and cluster in the RM UI
[ https://issues.apache.org/jira/browse/YARN-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15205262#comment-15205262 ] Jian He commented on YARN-4285: --- [~vvasudev], [~leftnoteasy], looks like the % of queue/cluster for the app does not include reserved resources. However, the queue's used resource in the UI does include reserved resource too. This caused the total amount of resources for all apps in the table does not equal to the total used resource shown in the queue. Should we consider reserved resource too in the % of queue/cluster for an app ? > Display resource usage as percentage of queue and cluster in the RM UI > -- > > Key: YARN-4285 > URL: https://issues.apache.org/jira/browse/YARN-4285 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 2.8.0 > > Attachments: YARN-4285.001.patch, YARN-4285.002.patch, > YARN-4285.003.patch, YARN-4285.004.patch > > > Currently, we display the memory and vcores allocated to an app in the RM UI. > It would be useful to display the resources consumed as a %of the queue and > the cluster to identify apps that are using a lot of resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4285) Display resource usage as percentage of queue and cluster in the RM UI
[ https://issues.apache.org/jira/browse/YARN-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975233#comment-14975233 ] Hudson commented on YARN-4285: -- FAILURE: Integrated in Hadoop-Yarn-trunk #1323 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1323/]) YARN-4285. Display resource usage as percentage of queue and cluster in (wangda: rev 3cc73773eb26f7469c99b25a76814d6fad0be28e) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppsBlock.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebPageUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationResourceUsageReportPBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationResourceUsageReport.java > Display resource usage as percentage of queue and cluster in the RM UI > -- > > Key: YARN-4285 > URL: https://issues.apache.org/jira/browse/YARN-4285 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 2.8.0 > > Attachments: YARN-4285.001.patch, YARN-4285.002.patch, > YARN-4285.003.patch, YARN-4285.004.patch > > > Currently, we display the memory and vcores allocated to an app in the RM UI. > It would be useful to display the resources consumed as a %of the queue and > the cluster to identify apps that are using a lot of resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4285) Display resource usage as percentage of queue and cluster in the RM UI
[ https://issues.apache.org/jira/browse/YARN-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975211#comment-14975211 ] Hudson commented on YARN-4285: -- FAILURE: Integrated in Hadoop-trunk-Commit #8710 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8710/]) YARN-4285. Display resource usage as percentage of queue and cluster in (wangda: rev 3cc73773eb26f7469c99b25a76814d6fad0be28e) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppsBlock.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationResourceUsageReport.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebPageUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationResourceUsageReportPBImpl.java > Display resource usage as percentage of queue and cluster in the RM UI > -- > > Key: YARN-4285 > URL: https://issues.apache.org/jira/browse/YARN-4285 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 2.8.0 > > Attachments: YARN-4285.001.patch, YARN-4285.002.patch, > YARN-4285.003.patch, YARN-4285.004.patch > > > Currently, we display the memory and vcores allocated to an app in the RM UI. > It would be useful to display the resources consumed as a %of the queue and > the cluster to identify apps that are using a lot of resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4285) Display resource usage as percentage of queue and cluster in the RM UI
[ https://issues.apache.org/jira/browse/YARN-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975265#comment-14975265 ] Hudson commented on YARN-4285: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #599 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/599/]) YARN-4285. Display resource usage as percentage of queue and cluster in (wangda: rev 3cc73773eb26f7469c99b25a76814d6fad0be28e) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationResourceUsageReportPBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationResourceUsageReport.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppsBlock.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebPageUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java > Display resource usage as percentage of queue and cluster in the RM UI > -- > > Key: YARN-4285 > URL: https://issues.apache.org/jira/browse/YARN-4285 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 2.8.0 > > Attachments: YARN-4285.001.patch, YARN-4285.002.patch, > YARN-4285.003.patch, YARN-4285.004.patch > > > Currently, we display the memory and vcores allocated to an app in the RM UI. > It would be useful to display the resources consumed as a %of the queue and > the cluster to identify apps that are using a lot of resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4285) Display resource usage as percentage of queue and cluster in the RM UI
[ https://issues.apache.org/jira/browse/YARN-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975338#comment-14975338 ] Hudson commented on YARN-4285: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2477 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2477/]) YARN-4285. Display resource usage as percentage of queue and cluster in (wangda: rev 3cc73773eb26f7469c99b25a76814d6fad0be28e) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebPageUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppsBlock.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationResourceUsageReportPBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationResourceUsageReport.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java > Display resource usage as percentage of queue and cluster in the RM UI > -- > > Key: YARN-4285 > URL: https://issues.apache.org/jira/browse/YARN-4285 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 2.8.0 > > Attachments: YARN-4285.001.patch, YARN-4285.002.patch, > YARN-4285.003.patch, YARN-4285.004.patch > > > Currently, we display the memory and vcores allocated to an app in the RM UI. > It would be useful to display the resources consumed as a %of the queue and > the cluster to identify apps that are using a lot of resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4285) Display resource usage as percentage of queue and cluster in the RM UI
[ https://issues.apache.org/jira/browse/YARN-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975398#comment-14975398 ] Hudson commented on YARN-4285: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2530 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2530/]) YARN-4285. Display resource usage as percentage of queue and cluster in (wangda: rev 3cc73773eb26f7469c99b25a76814d6fad0be28e) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppsBlock.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationResourceUsageReportPBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebPageUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationResourceUsageReport.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java > Display resource usage as percentage of queue and cluster in the RM UI > -- > > Key: YARN-4285 > URL: https://issues.apache.org/jira/browse/YARN-4285 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 2.8.0 > > Attachments: YARN-4285.001.patch, YARN-4285.002.patch, > YARN-4285.003.patch, YARN-4285.004.patch > > > Currently, we display the memory and vcores allocated to an app in the RM UI. > It would be useful to display the resources consumed as a %of the queue and > the cluster to identify apps that are using a lot of resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4285) Display resource usage as percentage of queue and cluster in the RM UI
[ https://issues.apache.org/jira/browse/YARN-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975494#comment-14975494 ] Hudson commented on YARN-4285: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #540 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/540/]) YARN-4285. Display resource usage as percentage of queue and cluster in (wangda: rev 3cc73773eb26f7469c99b25a76814d6fad0be28e) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationResourceUsageReportPBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationResourceUsageReport.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebPageUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppsBlock.java > Display resource usage as percentage of queue and cluster in the RM UI > -- > > Key: YARN-4285 > URL: https://issues.apache.org/jira/browse/YARN-4285 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 2.8.0 > > Attachments: YARN-4285.001.patch, YARN-4285.002.patch, > YARN-4285.003.patch, YARN-4285.004.patch > > > Currently, we display the memory and vcores allocated to an app in the RM UI. > It would be useful to display the resources consumed as a %of the queue and > the cluster to identify apps that are using a lot of resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4285) Display resource usage as percentage of queue and cluster in the RM UI
[ https://issues.apache.org/jira/browse/YARN-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975508#comment-14975508 ] Hudson commented on YARN-4285: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #588 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/588/]) YARN-4285. Display resource usage as percentage of queue and cluster in (wangda: rev 3cc73773eb26f7469c99b25a76814d6fad0be28e) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppsBlock.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebPageUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationResourceUsageReport.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationResourceUsageReportPBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java > Display resource usage as percentage of queue and cluster in the RM UI > -- > > Key: YARN-4285 > URL: https://issues.apache.org/jira/browse/YARN-4285 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 2.8.0 > > Attachments: YARN-4285.001.patch, YARN-4285.002.patch, > YARN-4285.003.patch, YARN-4285.004.patch > > > Currently, we display the memory and vcores allocated to an app in the RM UI. > It would be useful to display the resources consumed as a %of the queue and > the cluster to identify apps that are using a lot of resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4285) Display resource usage as percentage of queue and cluster in the RM UI
[ https://issues.apache.org/jira/browse/YARN-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14974643#comment-14974643 ] Varun Vasudev commented on YARN-4285: - The test failures are not related to the patch. They pass on my machine. > Display resource usage as percentage of queue and cluster in the RM UI > -- > > Key: YARN-4285 > URL: https://issues.apache.org/jira/browse/YARN-4285 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-4285.001.patch, YARN-4285.002.patch, > YARN-4285.003.patch, YARN-4285.004.patch > > > Currently, we display the memory and vcores allocated to an app in the RM UI. > It would be useful to display the resources consumed as a %of the queue and > the cluster to identify apps that are using a lot of resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4285) Display resource usage as percentage of queue and cluster in the RM UI
[
https://issues.apache.org/jira/browse/YARN-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972297#comment-14972297
]
Hadoop QA commented on YARN-4285:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 22m 51s | Pre-patch trunk has 3 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 4 new or modified test files. |
| {color:green}+1{color} | javac | 7m 59s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 22s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 30s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 45s | The applied patch generated 1
new checkstyle issues (total was 10, now 11). |
| {color:green}+1{color} | whitespace | 0m 5s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 7m 39s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 24s | Tests passed in
hadoop-yarn-api. |
| {color:green}+1{color} | yarn tests | 7m 3s | Tests passed in
hadoop-yarn-client. |
| {color:green}+1{color} | yarn tests | 2m 2s | Tests passed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 3m 19s | Tests passed in
hadoop-yarn-server-applicationhistoryservice. |
| {color:green}+1{color} | yarn tests | 0m 26s | Tests passed in
hadoop-yarn-server-common. |
| {color:red}-1{color} | yarn tests | 55m 48s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 124m 30s | |
\\
\\
|| Reason || Tests ||
| Timed out tests |
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices |
| |
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
|
| | org.apache.hadoop.yarn.server.resourcemanager.TestMoveApplication |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12768364/YARN-4285.004.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 15eb84b |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/9554/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-common.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/9554/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9554/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-client test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9554/artifact/patchprocess/testrun_hadoop-yarn-client.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9554/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-applicationhistoryservice test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9554/artifact/patchprocess/testrun_hadoop-yarn-server-applicationhistoryservice.txt
|
| hadoop-yarn-server-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9554/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9554/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9554/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9554/console |
This message was automatically generated.
> Display resource usage as percentage of queue and cluster in the RM UI
> --
>
> Key: YARN-4285
> URL: https://issues.apache.org/jira/browse/YARN-4285
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: YARN-4285.001.patch, YARN-4285.002.patch,
> YARN-4285.003.patch, YARN-4285.004.patch
>
>
> Currently, we display the memory and vcores allocated to an app in the RM UI.
> It would be useful to display the resources consumed as a %of the queue and
> the cluster to identify apps that
[jira] [Commented] (YARN-4285) Display resource usage as percentage of queue and cluster in the RM UI
[
https://issues.apache.org/jira/browse/YARN-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971097#comment-14971097
]
Hadoop QA commented on YARN-4285:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 26m 8s | Pre-patch trunk has 3 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 4 new or modified test files. |
| {color:green}+1{color} | javac | 9m 35s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 54s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 53s | The applied patch generated 1
new checkstyle issues (total was 10, now 11). |
| {color:green}+1{color} | whitespace | 0m 3s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 39s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 36s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 6m 50s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 24s | Tests passed in
hadoop-yarn-api. |
| {color:red}-1{color} | yarn tests | 34m 55s | Tests failed in
hadoop-yarn-client. |
| {color:green}+1{color} | yarn tests | 2m 1s | Tests passed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 3m 51s | Tests passed in
hadoop-yarn-server-applicationhistoryservice. |
| {color:red}-1{color} | yarn tests | 327m 39s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 428m 42s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| | hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart |
| | hadoop.yarn.server.resourcemanager.TestRMHA |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerQueueACLs |
| | hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerUtils |
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerQueueACLs
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerDynamicBehavior
|
| | hadoop.yarn.server.resourcemanager.TestApplicationACLs |
| | hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler |
| | hadoop.yarn.server.resourcemanager.TestContainerResourceUsage |
| | hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
| | hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodeLabelUpdate
|
| | hadoop.yarn.server.resourcemanager.TestRM |
| | hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA |
| |
hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerApplicationAttempt |
| | hadoop.yarn.server.resourcemanager.TestClientRMService |
| | hadoop.yarn.server.resourcemanager.rmapp.TestNodesListManager |
| Timed out tests | org.apache.hadoop.yarn.client.api.impl.TestAMRMClient |
| | org.apache.hadoop.yarn.client.api.impl.TestNMClient |
| | org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup |
| |
org.apache.hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter
|
| | org.apache.hadoop.yarn.server.resourcemanager.TestSignalContainer |
| |
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.TestRMContainerImpl |
| | org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart |
| |
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
|
| |
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationPriority
|
| |
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
|
| |
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing
|
| |
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation
|
| |
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterService |
| |
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
|
| |
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestWorkPreservingRMRestartForNodeLabel
|
| |
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher |
| | org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA
|
| |
[jira] [Commented] (YARN-4285) Display resource usage as percentage of queue and cluster in the RM UI
[ https://issues.apache.org/jira/browse/YARN-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972161#comment-14972161 ] Wangda Tan commented on YARN-4285: -- Tried this patch locally, looks great! Also tried to sort applications based on queue percentage, cluster percentage, all work as expected. Will commit shortly if no objections. > Display resource usage as percentage of queue and cluster in the RM UI > -- > > Key: YARN-4285 > URL: https://issues.apache.org/jira/browse/YARN-4285 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-4285.001.patch, YARN-4285.002.patch, > YARN-4285.003.patch, YARN-4285.004.patch > > > Currently, we display the memory and vcores allocated to an app in the RM UI. > It would be useful to display the resources consumed as a %of the queue and > the cluster to identify apps that are using a lot of resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4285) Display resource usage as percentage of queue and cluster in the RM UI
[
https://issues.apache.org/jira/browse/YARN-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971846#comment-14971846
]
Hadoop QA commented on YARN-4285:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 29m 49s | Pre-patch trunk has 3 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 4 new or modified test files. |
| {color:green}+1{color} | javac | 11m 27s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 12m 17s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 33s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 3m 56s | The applied patch generated 1
new checkstyle issues (total was 10, now 11). |
| {color:green}+1{color} | whitespace | 0m 4s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 49s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 38s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 8m 9s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 24s | Tests passed in
hadoop-yarn-api. |
| {color:green}+1{color} | yarn tests | 7m 11s | Tests passed in
hadoop-yarn-client. |
| {color:green}+1{color} | yarn tests | 2m 3s | Tests passed in
hadoop-yarn-common. |
| {color:red}-1{color} | yarn tests | 3m 7s | Tests failed in
hadoop-yarn-server-applicationhistoryservice. |
| {color:green}+1{color} | yarn tests | 0m 25s | Tests passed in
hadoop-yarn-server-common. |
| {color:red}-1{color} | yarn tests | 0m 19s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 83m 32s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore |
| Failed build | hadoop-yarn-server-resourcemanager |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12768364/YARN-4285.004.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 600ad7b |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/9552/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-common.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/9552/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9552/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-client test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9552/artifact/patchprocess/testrun_hadoop-yarn-client.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9552/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-applicationhistoryservice test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9552/artifact/patchprocess/testrun_hadoop-yarn-server-applicationhistoryservice.txt
|
| hadoop-yarn-server-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9552/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9552/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9552/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9552/console |
This message was automatically generated.
> Display resource usage as percentage of queue and cluster in the RM UI
> --
>
> Key: YARN-4285
> URL: https://issues.apache.org/jira/browse/YARN-4285
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: YARN-4285.001.patch, YARN-4285.002.patch,
> YARN-4285.003.patch, YARN-4285.004.patch
>
>
> Currently, we display the memory and vcores allocated to an app in the RM UI.
> It would be useful to display the resources consumed as a %of the queue and
> the cluster to identify apps that are using a lot of resources.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4285) Display resource usage as percentage of queue and cluster in the RM UI
[
https://issues.apache.org/jira/browse/YARN-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969638#comment-14969638
]
Wangda Tan commented on YARN-4285:
--
Thanks for updating, [~vvasudev]!
Some comments:
1) Since ApplicationResourceUsageReport is public API, I suggest to rename
parameter, getter/setter name from Perc to Percentage. Same as
yarn_protos.proto.
2)
{code}
if (rmContext.getScheduler() instanceof YarnScheduler) {
calc = rmContext.getScheduler().getResourceCalculator();
}
{code}
rmContext.getScheduler() should be always YarnScheduler, correct? This check
maybe not required.
3) Is it better to change int percentage to float, for AppInfo and other APIs.
4) Since this is not a trivial patch, could you add some tests to verify if
SchedulerApplicationAttempt can return percentage properly?
> Display resource usage as percentage of queue and cluster in the RM UI
> --
>
> Key: YARN-4285
> URL: https://issues.apache.org/jira/browse/YARN-4285
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: YARN-4285.001.patch, YARN-4285.002.patch
>
>
> Currently, we display the memory and vcores allocated to an app in the RM UI.
> It would be useful to display the resources consumed as a %of the queue and
> the cluster to identify apps that are using a lot of resources.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4285) Display resource usage as percentage of queue and cluster in the RM UI
[
https://issues.apache.org/jira/browse/YARN-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970172#comment-14970172
]
Hadoop QA commented on YARN-4285:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 38m 19s | Pre-patch trunk has 3 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 11m 13s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 13m 40s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 31s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 3m 13s | The applied patch generated 1
new checkstyle issues (total was 10, now 11). |
| {color:red}-1{color} | whitespace | 0m 2s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 2m 9s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 46s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 9m 16s | The patch appears to introduce 1
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 33s | Tests passed in
hadoop-yarn-api. |
| {color:red}-1{color} | yarn tests | 35m 28s | Tests failed in
hadoop-yarn-client. |
| {color:green}+1{color} | yarn tests | 2m 48s | Tests passed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 3m 59s | Tests passed in
hadoop-yarn-server-applicationhistoryservice. |
| {color:red}-1{color} | yarn tests | 267m 10s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 390m 20s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-server-resourcemanager |
| Failed unit tests | hadoop.yarn.client.api.impl.TestYarnClient |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps |
| | hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart |
| | hadoop.yarn.server.resourcemanager.TestClientRMService |
| | hadoop.yarn.server.resourcemanager.TestRM |
| | hadoop.yarn.server.resourcemanager.rmcontainer.TestRMContainerImpl |
| | hadoop.yarn.server.resourcemanager.TestMoveApplication |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestWorkPreservingRMRestartForNodeLabel
|
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodeLabels |
| | hadoop.yarn.server.resourcemanager.TestContainerResourceUsage |
| | hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler |
| | hadoop.yarn.server.resourcemanager.rmapp.TestRMAppTransitions |
| | hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
| | hadoop.yarn.server.resourcemanager.security.TestAMRMTokens |
| | hadoop.yarn.server.resourcemanager.TestApplicationCleanup |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodeLabelUpdate
|
| | hadoop.yarn.server.resourcemanager.TestRMAdminService |
| | hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerQueueACLs |
| | hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerUtils |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerDynamicBehavior
|
| | hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA |
| | hadoop.yarn.server.resourcemanager.TestApplicationACLs |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerQueueACLs
|
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesFairScheduler |
| | hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler |
| | hadoop.yarn.server.resourcemanager.security.TestClientToAMTokens |
| | hadoop.yarn.server.resourcemanager.rmapp.TestNodesListManager |
| Timed out tests | org.apache.hadoop.yarn.client.api.impl.TestAMRMClient |
| | org.apache.hadoop.yarn.client.api.impl.TestNMClient |
| | org.apache.hadoop.yarn.server.resourcemanager.TestSignalContainer |
| | org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart |
| |
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
|
| |
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationPriority
|
| |
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
|
| |
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCResponseId
|
| |
[jira] [Commented] (YARN-4285) Display resource usage as percentage of queue and cluster in the RM UI
[
https://issues.apache.org/jira/browse/YARN-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14967784#comment-14967784
]
Wangda Tan commented on YARN-4285:
--
[~vvasudev], thanks for working on this patch. This will be very useful!
Some suggestions to the patch:
- Instead of computing % in the web service component, it's better to compute %
in the scheduler module, which we can consider resource calculator, etc. Now we
have a place to do this, see
{{org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.getResourceUsageReport()}}.
After ApplicationResourceUsageReport is created, you can set % values to
{{org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.AppInfo}}.
- And AppAttempt page can get AppInfo as well, may need to modify some
interfaces.
> Display resource usage as percentage of queue and cluster in the RM UI
> --
>
> Key: YARN-4285
> URL: https://issues.apache.org/jira/browse/YARN-4285
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: YARN-4285.001.patch
>
>
> Currently, we display the memory and vcores allocated to an app in the RM UI.
> It would be useful to display the resources consumed as a %of the queue and
> the cluster to identify apps that are using a lot of resources.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
