[jira] [Commented] (YARN-2824) Capacity of labels should be zero by default
[ https://issues.apache.org/jira/browse/YARN-2824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14203435#comment-14203435 ] Hudson commented on YARN-2824: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1927 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1927/]) YARN-2824. Fixed Capacity Scheduler to not crash when some node-labels are not mapped to queues by making default capacities per label to be zero. Contributed by Wangda Tan. (vinodkv: rev 2ac1be7dec4aef001e3162e364249933b2c4a6c4) * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java Capacity of labels should be zero by default Key: YARN-2824 URL: https://issues.apache.org/jira/browse/YARN-2824 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Wangda Tan Priority: Critical Fix For: 2.6.0 Attachments: YARN-2824-1.patch, YARN-2824-2.patch In existing Capacity Scheduler behavior, if user doesn't specify capacity of label, queue initialization will be failed. That will cause queue refreshment failed when add a new label to node labels collection and doesn't modify capacity-scheduler.xml. With this patch, capacity of labels should be explicitly set if user want to use it. If user doesn't set capacity of some labels, we will treat such labels are unused labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2824) Capacity of labels should be zero by default
[ https://issues.apache.org/jira/browse/YARN-2824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14203460#comment-14203460 ] Hudson commented on YARN-2824: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1951 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1951/]) YARN-2824. Fixed Capacity Scheduler to not crash when some node-labels are not mapped to queues by making default capacities per label to be zero. Contributed by Wangda Tan. (vinodkv: rev 2ac1be7dec4aef001e3162e364249933b2c4a6c4) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java Capacity of labels should be zero by default Key: YARN-2824 URL: https://issues.apache.org/jira/browse/YARN-2824 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Wangda Tan Priority: Critical Fix For: 2.6.0 Attachments: YARN-2824-1.patch, YARN-2824-2.patch In existing Capacity Scheduler behavior, if user doesn't specify capacity of label, queue initialization will be failed. That will cause queue refreshment failed when add a new label to node labels collection and doesn't modify capacity-scheduler.xml. With this patch, capacity of labels should be explicitly set if user want to use it. If user doesn't set capacity of some labels, we will treat such labels are unused labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2824) Capacity of labels should be zero by default
[ https://issues.apache.org/jira/browse/YARN-2824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14202417#comment-14202417 ] Vinod Kumar Vavilapalli commented on YARN-2824: --- Makes sense, this is a much better default behaviour. Summarizing from what I understand - Non-labeled nodes: Default root-capacity is 100% everywhere - Labeled nodes: Default capacity is 0%, which means sum of capacities per label also will be 0 by default. Patch looks good, +1. Checking this in. Capacity of labels should be zero by default Key: YARN-2824 URL: https://issues.apache.org/jira/browse/YARN-2824 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Wangda Tan Priority: Critical Attachments: YARN-2824-1.patch, YARN-2824-2.patch In existing Capacity Scheduler behavior, if user doesn't specify capacity of label, queue initialization will be failed. That will cause queue refreshment failed when add a new label to node labels collection and doesn't modify capacity-scheduler.xml. With this patch, capacity of labels should be explicitly set if user want to use it. If user doesn't set capacity of some labels, we will treat such labels are unused labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2824) Capacity of labels should be zero by default
[ https://issues.apache.org/jira/browse/YARN-2824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14202433#comment-14202433 ] Wangda Tan commented on YARN-2824: -- Hi [~vinodkv], Thanks for review and commit. bq. Non-labeled nodes: Default root-capacity is 100% everywhere Yes, bq. Labeled nodes: Default capacity is 0%, which means sum of capacities per label also will be 0 by default. Yes, default capacity of labels are 0%, and if parent's capacity is 0, sum of children's capacity should be 0 too. Wangda Capacity of labels should be zero by default Key: YARN-2824 URL: https://issues.apache.org/jira/browse/YARN-2824 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Wangda Tan Priority: Critical Fix For: 2.6.0 Attachments: YARN-2824-1.patch, YARN-2824-2.patch In existing Capacity Scheduler behavior, if user doesn't specify capacity of label, queue initialization will be failed. That will cause queue refreshment failed when add a new label to node labels collection and doesn't modify capacity-scheduler.xml. With this patch, capacity of labels should be explicitly set if user want to use it. If user doesn't set capacity of some labels, we will treat such labels are unused labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2824) Capacity of labels should be zero by default
[ https://issues.apache.org/jira/browse/YARN-2824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14202438#comment-14202438 ] Hudson commented on YARN-2824: -- FAILURE: Integrated in Hadoop-trunk-Commit #6481 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6481/]) YARN-2824. Fixed Capacity Scheduler to not crash when some node-labels are not mapped to queues by making default capacities per label to be zero. Contributed by Wangda Tan. (vinodkv: rev 2ac1be7dec4aef001e3162e364249933b2c4a6c4) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java * hadoop-yarn-project/CHANGES.txt Capacity of labels should be zero by default Key: YARN-2824 URL: https://issues.apache.org/jira/browse/YARN-2824 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Wangda Tan Priority: Critical Fix For: 2.6.0 Attachments: YARN-2824-1.patch, YARN-2824-2.patch In existing Capacity Scheduler behavior, if user doesn't specify capacity of label, queue initialization will be failed. That will cause queue refreshment failed when add a new label to node labels collection and doesn't modify capacity-scheduler.xml. With this patch, capacity of labels should be explicitly set if user want to use it. If user doesn't set capacity of some labels, we will treat such labels are unused labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2824) Capacity of labels should be zero by default
[
https://issues.apache.org/jira/browse/YARN-2824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14201351#comment-14201351
]
Hadoop QA commented on YARN-2824:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12679998/YARN-2824-1.patch
against trunk revision 75b820c.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5762//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/5762//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5762//console
This message is automatically generated.
Capacity of labels should be zero by default
Key: YARN-2824
URL: https://issues.apache.org/jira/browse/YARN-2824
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Wangda Tan
Priority: Critical
Attachments: YARN-2824-1.patch
In existing Capacity Scheduler behavior, if user doesn't specify capacity of
label, queue initialization will be failed. That will cause queue refreshment
failed when add a new label to node labels collection and doesn't modify
capacity-scheduler.xml.
With this patch, capacity of labels should be explicitly set if user want to
use it. If user doesn't set capacity of some labels, we will treat such
labels are unused labels.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2824) Capacity of labels should be zero by default
[ https://issues.apache.org/jira/browse/YARN-2824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14201413#comment-14201413 ] Wangda Tan commented on YARN-2824: -- Attached new patch suppressing findbug warnings Capacity of labels should be zero by default Key: YARN-2824 URL: https://issues.apache.org/jira/browse/YARN-2824 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Wangda Tan Assignee: Wangda Tan Priority: Critical Attachments: YARN-2824-1.patch, YARN-2824-2.patch In existing Capacity Scheduler behavior, if user doesn't specify capacity of label, queue initialization will be failed. That will cause queue refreshment failed when add a new label to node labels collection and doesn't modify capacity-scheduler.xml. With this patch, capacity of labels should be explicitly set if user want to use it. If user doesn't set capacity of some labels, we will treat such labels are unused labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2824) Capacity of labels should be zero by default
[
https://issues.apache.org/jira/browse/YARN-2824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14201504#comment-14201504
]
Hadoop QA commented on YARN-2824:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12680048/YARN-2824-2.patch
against trunk revision ae71a67.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5766//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5766//console
This message is automatically generated.
Capacity of labels should be zero by default
Key: YARN-2824
URL: https://issues.apache.org/jira/browse/YARN-2824
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Wangda Tan
Assignee: Wangda Tan
Priority: Critical
Attachments: YARN-2824-1.patch, YARN-2824-2.patch
In existing Capacity Scheduler behavior, if user doesn't specify capacity of
label, queue initialization will be failed. That will cause queue refreshment
failed when add a new label to node labels collection and doesn't modify
capacity-scheduler.xml.
With this patch, capacity of labels should be explicitly set if user want to
use it. If user doesn't set capacity of some labels, we will treat such
labels are unused labels.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
