[jira] [Commented] (YARN-3098) Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues
[ https://issues.apache.org/jira/browse/YARN-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14304915#comment-14304915 ] Hudson commented on YARN-3098: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #94 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/94/]) YARN-3098. Created common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues. Contributed by Wangda Tan (jianhe: rev 21d80b3dd90a8e33e51701887c8d9369ed4ab17d) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueCapacities.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/QueueCapacities.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ResourceUsage.java * hadoop-yarn-project/CHANGES.txt Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues - Key: YARN-3098 URL: https://issues.apache.org/jira/browse/YARN-3098 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler Reporter: Wangda Tan Assignee: Wangda Tan Fix For: 2.7.0 Attachments: YARN-3098.1.patch, YARN-3098.2.patch, YARN-3098.3.patch, YARN-3098.4.patch Similar to YARN-3092, after YARN-796, now queues (ParentQueue and LeafQueue) need to track capacities-label (e.g. absolute-capacity, maximum-capacity, absolute-capacity, absolute-maximum-capacity, etc.). It's better to have a class to encapsulate these capacities to make both better maintainability/readability and fine-grained locking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3098) Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues
[ https://issues.apache.org/jira/browse/YARN-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14305096#comment-14305096 ] Hudson commented on YARN-3098: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2026 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2026/]) YARN-3098. Created common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues. Contributed by Wangda Tan (jianhe: rev 21d80b3dd90a8e33e51701887c8d9369ed4ab17d) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ResourceUsage.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/QueueCapacities.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueCapacities.java Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues - Key: YARN-3098 URL: https://issues.apache.org/jira/browse/YARN-3098 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler Reporter: Wangda Tan Assignee: Wangda Tan Fix For: 2.7.0 Attachments: YARN-3098.1.patch, YARN-3098.2.patch, YARN-3098.3.patch, YARN-3098.4.patch Similar to YARN-3092, after YARN-796, now queues (ParentQueue and LeafQueue) need to track capacities-label (e.g. absolute-capacity, maximum-capacity, absolute-capacity, absolute-maximum-capacity, etc.). It's better to have a class to encapsulate these capacities to make both better maintainability/readability and fine-grained locking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3098) Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues
[ https://issues.apache.org/jira/browse/YARN-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14305123#comment-14305123 ] Hudson commented on YARN-3098: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #91 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/91/]) YARN-3098. Created common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues. Contributed by Wangda Tan (jianhe: rev 21d80b3dd90a8e33e51701887c8d9369ed4ab17d) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ResourceUsage.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/QueueCapacities.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueCapacities.java Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues - Key: YARN-3098 URL: https://issues.apache.org/jira/browse/YARN-3098 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler Reporter: Wangda Tan Assignee: Wangda Tan Fix For: 2.7.0 Attachments: YARN-3098.1.patch, YARN-3098.2.patch, YARN-3098.3.patch, YARN-3098.4.patch Similar to YARN-3092, after YARN-796, now queues (ParentQueue and LeafQueue) need to track capacities-label (e.g. absolute-capacity, maximum-capacity, absolute-capacity, absolute-maximum-capacity, etc.). It's better to have a class to encapsulate these capacities to make both better maintainability/readability and fine-grained locking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3098) Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues
[ https://issues.apache.org/jira/browse/YARN-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14305204#comment-14305204 ] Hudson commented on YARN-3098: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #95 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/95/]) YARN-3098. Created common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues. Contributed by Wangda Tan (jianhe: rev 21d80b3dd90a8e33e51701887c8d9369ed4ab17d) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ResourceUsage.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueCapacities.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/QueueCapacities.java * hadoop-yarn-project/CHANGES.txt Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues - Key: YARN-3098 URL: https://issues.apache.org/jira/browse/YARN-3098 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler Reporter: Wangda Tan Assignee: Wangda Tan Fix For: 2.7.0 Attachments: YARN-3098.1.patch, YARN-3098.2.patch, YARN-3098.3.patch, YARN-3098.4.patch Similar to YARN-3092, after YARN-796, now queues (ParentQueue and LeafQueue) need to track capacities-label (e.g. absolute-capacity, maximum-capacity, absolute-capacity, absolute-maximum-capacity, etc.). It's better to have a class to encapsulate these capacities to make both better maintainability/readability and fine-grained locking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3098) Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues
[ https://issues.apache.org/jira/browse/YARN-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303662#comment-14303662 ] Wangda Tan commented on YARN-3098: -- [~sunilg], Thanks, I think for 2.2, we can make all CS queue related fields read from outside consistent, there're only limited places accessing fields of CS queues. Since it is not so urgent, we can address the problem in following ticket. Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues - Key: YARN-3098 URL: https://issues.apache.org/jira/browse/YARN-3098 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-3098.1.patch, YARN-3098.2.patch, YARN-3098.3.patch, YARN-3098.4.patch Similar to YARN-3092, after YARN-796, now queues (ParentQueue and LeafQueue) need to track capacities-label (e.g. absolute-capacity, maximum-capacity, absolute-capacity, absolute-maximum-capacity, etc.). It's better to have a class to encapsulate these capacities to make both better maintainability/readability and fine-grained locking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3098) Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues
[ https://issues.apache.org/jira/browse/YARN-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303669#comment-14303669 ] Sunil G commented on YARN-3098: --- +1 Yes, that can be addressed in a following ticket. Could I raise a ticket for the same. Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues - Key: YARN-3098 URL: https://issues.apache.org/jira/browse/YARN-3098 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-3098.1.patch, YARN-3098.2.patch, YARN-3098.3.patch, YARN-3098.4.patch Similar to YARN-3092, after YARN-796, now queues (ParentQueue and LeafQueue) need to track capacities-label (e.g. absolute-capacity, maximum-capacity, absolute-capacity, absolute-maximum-capacity, etc.). It's better to have a class to encapsulate these capacities to make both better maintainability/readability and fine-grained locking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3098) Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues
[ https://issues.apache.org/jira/browse/YARN-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303671#comment-14303671 ] Wangda Tan commented on YARN-3098: -- Sure, plz go ahead :) Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues - Key: YARN-3098 URL: https://issues.apache.org/jira/browse/YARN-3098 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-3098.1.patch, YARN-3098.2.patch, YARN-3098.3.patch, YARN-3098.4.patch Similar to YARN-3092, after YARN-796, now queues (ParentQueue and LeafQueue) need to track capacities-label (e.g. absolute-capacity, maximum-capacity, absolute-capacity, absolute-maximum-capacity, etc.). It's better to have a class to encapsulate these capacities to make both better maintainability/readability and fine-grained locking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3098) Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues
[ https://issues.apache.org/jira/browse/YARN-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303865#comment-14303865 ] Hudson commented on YARN-3098: -- SUCCESS: Integrated in Hadoop-trunk-Commit #6991 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6991/]) YARN-3098. Created common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues. Contributed by Wangda Tan (jianhe: rev 21d80b3dd90a8e33e51701887c8d9369ed4ab17d) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueCapacities.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/QueueCapacities.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ResourceUsage.java Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues - Key: YARN-3098 URL: https://issues.apache.org/jira/browse/YARN-3098 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler Reporter: Wangda Tan Assignee: Wangda Tan Fix For: 2.7.0 Attachments: YARN-3098.1.patch, YARN-3098.2.patch, YARN-3098.3.patch, YARN-3098.4.patch Similar to YARN-3092, after YARN-796, now queues (ParentQueue and LeafQueue) need to track capacities-label (e.g. absolute-capacity, maximum-capacity, absolute-capacity, absolute-maximum-capacity, etc.). It's better to have a class to encapsulate these capacities to make both better maintainability/readability and fine-grained locking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3098) Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues
[ https://issues.apache.org/jira/browse/YARN-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302686#comment-14302686 ] Sunil G commented on YARN-3098: --- Hi Wangda Thank you for the detailed explanation. I agree to your points 1) and 2) However my thoughts were on similar lines what you have explained on 2.2. Isnt it a good idea for having a similar lock for all related metric items accessible from queue. But I feel that is not part of this JIRA. And +1 for the patch. Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues - Key: YARN-3098 URL: https://issues.apache.org/jira/browse/YARN-3098 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-3098.1.patch, YARN-3098.2.patch, YARN-3098.3.patch, YARN-3098.4.patch Similar to YARN-3092, after YARN-796, now queues (ParentQueue and LeafQueue) need to track capacities-label (e.g. absolute-capacity, maximum-capacity, absolute-capacity, absolute-maximum-capacity, etc.). It's better to have a class to encapsulate these capacities to make both better maintainability/readability and fine-grained locking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3098) Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues
[ https://issues.apache.org/jira/browse/YARN-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301745#comment-14301745 ] Wangda Tan commented on YARN-3098: -- [~sunilg], any other comments? Thanks, Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues - Key: YARN-3098 URL: https://issues.apache.org/jira/browse/YARN-3098 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-3098.1.patch, YARN-3098.2.patch, YARN-3098.3.patch, YARN-3098.4.patch Similar to YARN-3092, after YARN-796, now queues (ParentQueue and LeafQueue) need to track capacities-label (e.g. absolute-capacity, maximum-capacity, absolute-capacity, absolute-maximum-capacity, etc.). It's better to have a class to encapsulate these capacities to make both better maintainability/readability and fine-grained locking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3098) Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues
[ https://issues.apache.org/jira/browse/YARN-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294961#comment-14294961 ] Sunil G commented on YARN-3098: --- Yes [~leftnoteasy] I understood you point. Sometimes we can take the call to keep a clean interface in top level and I also understood your intention As for the second point, ResourceUsage and QueueCapacities are two new classes. and It have its own separate locks now. Earlier this data is protected with the parent lock alone. SO now lock order is LeafQueue - ResourceUsage. I was worried a scenario where LeafQueue and ParentQueue is invoking these 2 new classes in opposite order, as per my review the locking order is correct. But when future additions happens in LeafQueue/ParentQueue/FiCaScehdulerApp etc, keeping correct lock order is at most priority. On that note, I was thinking that these 2 new locks should not complicate existing locks. This was I meant earlier. Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues - Key: YARN-3098 URL: https://issues.apache.org/jira/browse/YARN-3098 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-3098.1.patch, YARN-3098.2.patch, YARN-3098.3.patch, YARN-3098.4.patch Similar to YARN-3092, after YARN-796, now queues (ParentQueue and LeafQueue) need to track capacities-label (e.g. absolute-capacity, maximum-capacity, absolute-capacity, absolute-maximum-capacity, etc.). It's better to have a class to encapsulate these capacities to make both better maintainability/readability and fine-grained locking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3098) Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues
[ https://issues.apache.org/jira/browse/YARN-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14295583#comment-14295583 ] Wangda Tan commented on YARN-3098: -- Hi [~sunilg], Thanks for thinking about this,. I think two potential issues we need to take care regarding locks: 1) Deadlock caused by the new introduced locks. 2) Data inconsistency. For 1), I think it will be no problem if *a.* we don't try to acquire other locks from methods of QueueCapacities/ResourceUsage. And *b.* methods of QC/RU are pretty simple, all can terminate in finite time. If a/b meet at the same time. No new deadlock will happen. Agree? For 2), Data inconsistency also has two different types, 2.1 Data inconsistency while editing When we edit data within QC/RU, it still under queue's lock. Which has no data inconsistency issue. 2.2 Data inconsistency while reading When we read data in QC/RU from outside, it is possible QC in state1 (e.g. container-1 allocated) but RU in state2 (e.g. container-2 allocated). In the past we have same issue. We cannot avoid this except we wrap such data to a structure like QueueStatusInfo like AppReport. Thoughts? Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues - Key: YARN-3098 URL: https://issues.apache.org/jira/browse/YARN-3098 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-3098.1.patch, YARN-3098.2.patch, YARN-3098.3.patch, YARN-3098.4.patch Similar to YARN-3092, after YARN-796, now queues (ParentQueue and LeafQueue) need to track capacities-label (e.g. absolute-capacity, maximum-capacity, absolute-capacity, absolute-maximum-capacity, etc.). It's better to have a class to encapsulate these capacities to make both better maintainability/readability and fine-grained locking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3098) Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues
[ https://issues.apache.org/jira/browse/YARN-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294354#comment-14294354 ] Wangda Tan commented on YARN-3098: -- [~sunilg], Thanks for review, For #1, I didn't do that because I think add a type as parameter is not clear enough, since they will be used very frequently, it's more important to me to keep interface clean and easy (than less duplications). Sounds reasonable? For #2, Sorry I'm not quite understand your point. Wangda Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues - Key: YARN-3098 URL: https://issues.apache.org/jira/browse/YARN-3098 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-3098.1.patch, YARN-3098.2.patch, YARN-3098.3.patch, YARN-3098.4.patch Similar to YARN-3092, after YARN-796, now queues (ParentQueue and LeafQueue) need to track capacities-label (e.g. absolute-capacity, maximum-capacity, absolute-capacity, absolute-maximum-capacity, etc.). It's better to have a class to encapsulate these capacities to make both better maintainability/readability and fine-grained locking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3098) Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues
[
https://issues.apache.org/jira/browse/YARN-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293858#comment-14293858
]
Sunil G commented on YARN-3098:
---
Hi [~leftnoteasy]
This is a good addition with respect to maintainability. I have couple of small
suggestions, you can share your thoughts.
1.
{noformat}
+ public void setUsedCapacity(String label, float value) {
+_set(label, CapacityType.USED_CAP, value);
+ }
{noformat}
Is it better to have a generic setCapacity method as below.. It can avoid many
setters.
something like
{noformat}
setCapacity(label1, 3.5, CapacityType.USED_CAP)
+ public void setCapacity(String label, float value, int type) {
+_set(label, type, value);
+ }
{noformat}
2. ResourceUsage and QueueCapacities has its own locks now. Hence calling order
is now more important. Leaf Queue and Parent Queue can call these new class to
get resource usages.
How do you feel about same? In another JIRA, if we are breaking the dependency
b/w leaf and parent queue, then this will be fine.
Create common QueueCapacities class in Capacity Scheduler to track
capacities-by-labels of queues
-
Key: YARN-3098
URL: https://issues.apache.org/jira/browse/YARN-3098
Project: Hadoop YARN
Issue Type: Sub-task
Components: capacityscheduler
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-3098.1.patch, YARN-3098.2.patch, YARN-3098.3.patch,
YARN-3098.4.patch
Similar to YARN-3092, after YARN-796, now queues (ParentQueue and LeafQueue)
need to track capacities-label (e.g. absolute-capacity, maximum-capacity,
absolute-capacity, absolute-maximum-capacity, etc.). It's better to have a
class to encapsulate these capacities to make both better
maintainability/readability and fine-grained locking.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3098) Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues
[
https://issues.apache.org/jira/browse/YARN-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14292802#comment-14292802
]
Hadoop QA commented on YARN-3098:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12694659/YARN-3098.4.patch
against trunk revision 6f9fe76.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6427//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6427//console
This message is automatically generated.
Create common QueueCapacities class in Capacity Scheduler to track
capacities-by-labels of queues
-
Key: YARN-3098
URL: https://issues.apache.org/jira/browse/YARN-3098
Project: Hadoop YARN
Issue Type: Sub-task
Components: capacityscheduler
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-3098.1.patch, YARN-3098.2.patch, YARN-3098.3.patch,
YARN-3098.4.patch
Similar to YARN-3092, after YARN-796, now queues (ParentQueue and LeafQueue)
need to track capacities-label (e.g. absolute-capacity, maximum-capacity,
absolute-capacity, absolute-maximum-capacity, etc.). It's better to have a
class to encapsulate these capacities to make both better
maintainability/readability and fine-grained locking.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3098) Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues
[
https://issues.apache.org/jira/browse/YARN-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14292738#comment-14292738
]
Hadoop QA commented on YARN-3098:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12694645/YARN-3098.2.patch
against trunk revision 1f2b695.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6426//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6426//console
This message is automatically generated.
Create common QueueCapacities class in Capacity Scheduler to track
capacities-by-labels of queues
-
Key: YARN-3098
URL: https://issues.apache.org/jira/browse/YARN-3098
Project: Hadoop YARN
Issue Type: Sub-task
Components: capacityscheduler
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-3098.1.patch, YARN-3098.2.patch, YARN-3098.3.patch,
YARN-3098.4.patch
Similar to YARN-3092, after YARN-796, now queues (ParentQueue and LeafQueue)
need to track capacities-label (e.g. absolute-capacity, maximum-capacity,
absolute-capacity, absolute-maximum-capacity, etc.). It's better to have a
class to encapsulate these capacities to make both better
maintainability/readability and fine-grained locking.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3098) Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues
[
https://issues.apache.org/jira/browse/YARN-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14292548#comment-14292548
]
Hadoop QA commented on YARN-3098:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12694617/YARN-3098.1.patch
against trunk revision 21d5599.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6422//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6422//console
This message is automatically generated.
Create common QueueCapacities class in Capacity Scheduler to track
capacities-by-labels of queues
-
Key: YARN-3098
URL: https://issues.apache.org/jira/browse/YARN-3098
Project: Hadoop YARN
Issue Type: Sub-task
Components: capacityscheduler
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-3098.1.patch
Similar to YARN-3092, after YARN-796, now queues (ParentQueue and LeafQueue)
need to track capacities-label (e.g. absolute-capacity, maximum-capacity,
absolute-capacity, absolute-maximum-capacity, etc.). It's better to have a
class to encapsulate these capacities to make both better
maintainability/readability and fine-grained locking.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3098) Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues
[ https://issues.apache.org/jira/browse/YARN-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14292550#comment-14292550 ] Wangda Tan commented on YARN-3098: -- Failed test is not related to this patch. Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues - Key: YARN-3098 URL: https://issues.apache.org/jira/browse/YARN-3098 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-3098.1.patch Similar to YARN-3092, after YARN-796, now queues (ParentQueue and LeafQueue) need to track capacities-label (e.g. absolute-capacity, maximum-capacity, absolute-capacity, absolute-maximum-capacity, etc.). It's better to have a class to encapsulate these capacities to make both better maintainability/readability and fine-grained locking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3098) Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues
[
https://issues.apache.org/jira/browse/YARN-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14292566#comment-14292566
]
Jian He commented on YARN-3098:
---
- maybe a bug ? it's internally using NL
{code}
public void setUsedCapacity(String label, float value) {
try {
writeLock.lock();
Capacities cap = capacitiesMap.get(NL);
{code}
- {{ public float getAbsoluteUsedCapacity() }}, the implementation can be
{{getAbsoluteUsedCapacity(NL)}}
- maybe have a generic getter/setter and try parametrizing the capacity type to
avoid duplication.
Create common QueueCapacities class in Capacity Scheduler to track
capacities-by-labels of queues
-
Key: YARN-3098
URL: https://issues.apache.org/jira/browse/YARN-3098
Project: Hadoop YARN
Issue Type: Sub-task
Components: capacityscheduler
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-3098.1.patch
Similar to YARN-3092, after YARN-796, now queues (ParentQueue and LeafQueue)
need to track capacities-label (e.g. absolute-capacity, maximum-capacity,
absolute-capacity, absolute-maximum-capacity, etc.). It's better to have a
class to encapsulate these capacities to make both better
maintainability/readability and fine-grained locking.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
