[jira] [Commented] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores
[ https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14130051#comment-14130051 ] Hudson commented on YARN-2440: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1868 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1868/]) YARN-2440. Enabled Nodemanagers to limit the aggregate cpu usage across all containers to a preconfigured limit. Contributed by Varun Vasudev. (vinodkv: rev 4be95175cdb58ff12a27ab443d609d3b46da7bfa) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/NodeManagerHardwareUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestNodeManagerHardwareUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java > Cgroups should allow YARN containers to be limited to allocated cores > - > > Key: YARN-2440 > URL: https://issues.apache.org/jira/browse/YARN-2440 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 2.6.0 > > Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch, > apache-yarn-2440.2.patch, apache-yarn-2440.3.patch, apache-yarn-2440.4.patch, > apache-yarn-2440.5.patch, apache-yarn-2440.6.patch, > screenshot-current-implementation.jpg > > > The current cgroups implementation does not limit YARN containers to the > cores allocated in yarn-site.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores
[ https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14130028#comment-14130028 ] Hudson commented on YARN-2440: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1893 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1893/]) YARN-2440. Enabled Nodemanagers to limit the aggregate cpu usage across all containers to a preconfigured limit. Contributed by Varun Vasudev. (vinodkv: rev 4be95175cdb58ff12a27ab443d609d3b46da7bfa) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/NodeManagerHardwareUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestNodeManagerHardwareUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java > Cgroups should allow YARN containers to be limited to allocated cores > - > > Key: YARN-2440 > URL: https://issues.apache.org/jira/browse/YARN-2440 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 2.6.0 > > Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch, > apache-yarn-2440.2.patch, apache-yarn-2440.3.patch, apache-yarn-2440.4.patch, > apache-yarn-2440.5.patch, apache-yarn-2440.6.patch, > screenshot-current-implementation.jpg > > > The current cgroups implementation does not limit YARN containers to the > cores allocated in yarn-site.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores
[ https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14129906#comment-14129906 ] Hudson commented on YARN-2440: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #677 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/677/]) YARN-2440. Enabled Nodemanagers to limit the aggregate cpu usage across all containers to a preconfigured limit. Contributed by Varun Vasudev. (vinodkv: rev 4be95175cdb58ff12a27ab443d609d3b46da7bfa) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestNodeManagerHardwareUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/NodeManagerHardwareUtils.java > Cgroups should allow YARN containers to be limited to allocated cores > - > > Key: YARN-2440 > URL: https://issues.apache.org/jira/browse/YARN-2440 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 2.6.0 > > Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch, > apache-yarn-2440.2.patch, apache-yarn-2440.3.patch, apache-yarn-2440.4.patch, > apache-yarn-2440.5.patch, apache-yarn-2440.6.patch, > screenshot-current-implementation.jpg > > > The current cgroups implementation does not limit YARN containers to the > cores allocated in yarn-site.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores
[ https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14129533#comment-14129533 ] Vinod Kumar Vavilapalli commented on YARN-2440: --- This looks good, +1. Checking this in.. > Cgroups should allow YARN containers to be limited to allocated cores > - > > Key: YARN-2440 > URL: https://issues.apache.org/jira/browse/YARN-2440 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch, > apache-yarn-2440.2.patch, apache-yarn-2440.3.patch, apache-yarn-2440.4.patch, > apache-yarn-2440.5.patch, apache-yarn-2440.6.patch, > screenshot-current-implementation.jpg > > > The current cgroups implementation does not limit YARN containers to the > cores allocated in yarn-site.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores
[
https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14129058#comment-14129058
]
Hadoop QA commented on YARN-2440:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667797/apache-yarn-2440.6.patch
against trunk revision cbfe263.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4875//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4875//console
This message is automatically generated.
> Cgroups should allow YARN containers to be limited to allocated cores
> -
>
> Key: YARN-2440
> URL: https://issues.apache.org/jira/browse/YARN-2440
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch,
> apache-yarn-2440.2.patch, apache-yarn-2440.3.patch, apache-yarn-2440.4.patch,
> apache-yarn-2440.5.patch, apache-yarn-2440.6.patch,
> screenshot-current-implementation.jpg
>
>
> The current cgroups implementation does not limit YARN containers to the
> cores allocated in yarn-site.xml.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores
[ https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14129020#comment-14129020 ] Vinod Kumar Vavilapalli commented on YARN-2440: --- The build machine ran into an issue which [~gkesavan] helped fixing on my offline request. Rekicked the build manually.. > Cgroups should allow YARN containers to be limited to allocated cores > - > > Key: YARN-2440 > URL: https://issues.apache.org/jira/browse/YARN-2440 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch, > apache-yarn-2440.2.patch, apache-yarn-2440.3.patch, apache-yarn-2440.4.patch, > apache-yarn-2440.5.patch, apache-yarn-2440.6.patch, > screenshot-current-implementation.jpg > > > The current cgroups implementation does not limit YARN containers to the > cores allocated in yarn-site.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores
[ https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14128843#comment-14128843 ] Vinod Kumar Vavilapalli commented on YARN-2440: --- bq. As I mentioned before, I think most users would rather not use the functionality proposed by this JIRA but instead setup peer cgroups for other systems and set their relative cgroup shares appropriately. With this JIRA the CPUs could sit idle despite demand from YARN containers, while a peer cgroup setup allows CPU guarantees without idle CPUs if the demand is there. [~jlowe], agree with the general philosophy. Though we are not yet there in practice - datanodes, region servers don't yet live in cgroups in many sites. Looking back at this JIRA, I see a good use for this. Having the overall YARN limit will help ensure that apps' containers don't thrash cpu once we start enabling support. The other dimension to this is determinism w.r.t performance. Limiting to allocated cores overall (as well as per container later) helps orgs run workloads and reason about them deterministically. One of the examples is benchmarking apps, but deterministic execution is a desired option beyond benchmarks too. > Cgroups should allow YARN containers to be limited to allocated cores > - > > Key: YARN-2440 > URL: https://issues.apache.org/jira/browse/YARN-2440 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch, > apache-yarn-2440.2.patch, apache-yarn-2440.3.patch, apache-yarn-2440.4.patch, > apache-yarn-2440.5.patch, apache-yarn-2440.6.patch, > screenshot-current-implementation.jpg > > > The current cgroups implementation does not limit YARN containers to the > cores allocated in yarn-site.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores
[
https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14128842#comment-14128842
]
Hadoop QA commented on YARN-2440:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667797/apache-yarn-2440.6.patch
against trunk revision b67d5ba.
{color:red}-1 patch{color}. Trunk compilation may be broken.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4871//console
This message is automatically generated.
> Cgroups should allow YARN containers to be limited to allocated cores
> -
>
> Key: YARN-2440
> URL: https://issues.apache.org/jira/browse/YARN-2440
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch,
> apache-yarn-2440.2.patch, apache-yarn-2440.3.patch, apache-yarn-2440.4.patch,
> apache-yarn-2440.5.patch, apache-yarn-2440.6.patch,
> screenshot-current-implementation.jpg
>
>
> The current cgroups implementation does not limit YARN containers to the
> cores allocated in yarn-site.xml.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores
[
https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127467#comment-14127467
]
Hadoop QA commented on YARN-2440:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667461/apache-yarn-2440.5.patch
against trunk revision 2749fc6.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4860//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4860//console
This message is automatically generated.
> Cgroups should allow YARN containers to be limited to allocated cores
> -
>
> Key: YARN-2440
> URL: https://issues.apache.org/jira/browse/YARN-2440
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch,
> apache-yarn-2440.2.patch, apache-yarn-2440.3.patch, apache-yarn-2440.4.patch,
> apache-yarn-2440.5.patch, screenshot-current-implementation.jpg
>
>
> The current cgroups implementation does not limit YARN containers to the
> cores allocated in yarn-site.xml.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores
[ https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127086#comment-14127086 ] Jason Lowe commented on YARN-2440: -- bq. Specifically in the context of heterogeneous clusters where uniform % configurations can go really bad where the only resort will then be to do per-node configuration - not ideal. Yes, I could see the heterogenous cluster being a case where specifying absolute instead of relative may be desirable. My biggest concern is that it's confusing when trying to combine the absolute and relative concepts -- it's not obvious if one overrides the other or if one is relative to the other. Part of my concern to keep this as simple as possible and the configuration burden to an absolute minimum is that I'm missing the real-world use case. As I mentioned before, I think most users would rather not use the functionality proposed by this JIRA but instead setup peer cgroups for other systems and set their relative cgroup shares appropriately. With this JIRA the CPUs could sit idle despite demand from YARN containers, while a peer cgroup setup allows CPU guarantees without idle CPUs if the demand is there. > Cgroups should allow YARN containers to be limited to allocated cores > - > > Key: YARN-2440 > URL: https://issues.apache.org/jira/browse/YARN-2440 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch, > apache-yarn-2440.2.patch, apache-yarn-2440.3.patch, apache-yarn-2440.4.patch, > screenshot-current-implementation.jpg > > > The current cgroups implementation does not limit YARN containers to the > cores allocated in yarn-site.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores
[
https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14126140#comment-14126140
]
Vinod Kumar Vavilapalli commented on YARN-2440:
---
Just caught up with the discussion. I can get behind an absolute limit too.
Specifically in the context of heterogeneous clusters where uniform %
configurations can go really bad where the only resort will then be to do
per-node configuration - not ideal. Would that be a valid use-case for putting
in the absolute limit? [~jlowe]? Even if it were, I am okay punting that off to
a separate JIRA.
Comments on the patch:
- containers-limit-cpu-percentage ->
{{yarn.nodemanager.resource.percentage-cpu-limit}} to be consistent? Similarly
NM_CONTAINERS_CPU_PERC? I don't like the tag 'resource', it should have been
'resources' but it is what it is.
- You still have refs to YarnConfiguration.NM_CONTAINERS_CPU_ABSOLUTE in the
patch. Similarly the javadoc in NodeManagerHardwareUtils needs to be updated if
we are not adding the absolute cpu config. It should no longer refer to "number
of cores that should be used for YARN containers"
- TestCgroupsLCEResourcesHandler: You can use mockito if you only want to
override num-processors in TestResourceCalculatorPlugin. Similarly in
TestNodeManagerHardwareUtils.
- The tests may fail on a machine with > 4 cores? :)
> Cgroups should allow YARN containers to be limited to allocated cores
> -
>
> Key: YARN-2440
> URL: https://issues.apache.org/jira/browse/YARN-2440
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch,
> apache-yarn-2440.2.patch, apache-yarn-2440.3.patch, apache-yarn-2440.4.patch,
> screenshot-current-implementation.jpg
>
>
> The current cgroups implementation does not limit YARN containers to the
> cores allocated in yarn-site.xml.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores
[
https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14112350#comment-14112350
]
Hadoop QA commented on YARN-2440:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12664653/apache-yarn-2440.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4742//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4742//console
This message is automatically generated.
> Cgroups should allow YARN containers to be limited to allocated cores
> -
>
> Key: YARN-2440
> URL: https://issues.apache.org/jira/browse/YARN-2440
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch,
> apache-yarn-2440.2.patch, apache-yarn-2440.3.patch, apache-yarn-2440.4.patch,
> screenshot-current-implementation.jpg
>
>
> The current cgroups implementation does not limit YARN containers to the
> cores allocated in yarn-site.xml.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores
[
https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14112084#comment-14112084
]
Hadoop QA commented on YARN-2440:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12664596/apache-yarn-2440.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4739//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4739//console
This message is automatically generated.
> Cgroups should allow YARN containers to be limited to allocated cores
> -
>
> Key: YARN-2440
> URL: https://issues.apache.org/jira/browse/YARN-2440
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch,
> apache-yarn-2440.2.patch, apache-yarn-2440.3.patch,
> screenshot-current-implementation.jpg
>
>
> The current cgroups implementation does not limit YARN containers to the
> cores allocated in yarn-site.xml.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores
[ https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14111471#comment-14111471 ] Jason Lowe commented on YARN-2440: -- For the case presented by [~sjlee0] the user has an 8 core system and wants to use at most 6 cores for YARN containers. That can be done by simply setting containers-cpu-percentage to 75. I don't see why we need a separate containers-cpu-cores parameter here, and I think it causes more problems than it solves per my previous comment. If we only want to support whole-core granularity then I can see containers-cpu-cores as a better choice, but otherwise containers-cpu-percentage is more flexible. Also I don't see vcores being relevant for this JIRA. The way vcores map to physical cores is node-dependent, but apps ask for vcores in a node-independent fashion. IIUC this JIRA is focused on simply limiting the amount of CPU all YARN containers on the node can possibly use in aggregate. Changing the vcore-to-core ratio on the node will change how many containers the node might run simultaneously, but it shouldn't impact how much of the physical CPU the user wants reserved for non-container processes. On a related note, it's interesting to step back and see if this is really what most users will want in practice. If the intent is to ensure the NM, DN, and other system processes get enough CPU time then I think a better approach is to put those system processes in a peer cgroup to the YARN containers cgroup and set their relative CPU shares accordingly. Then YARN containers can continue to use any spare CPU if desired (i.e.: no CPU "fragmentation") but the system processes are guaranteed not to be starved out by the YARN containers. Some users may want a hard limit and hence why this feature would be useful for them, but I suspect most users will not want to leave spare CPU lying around when containers need it. bq. How about yarn.nodemanager.all-containers-cpu-cores and yarn.nodemanager.all-containers-cpu-percentage? I'm indifferent on adding "all" as a prefix. Something like yarn.nodemanager.containers-limit-cpu-percentage might be more clear that this is a hard limit and CPUs can go idle even if containers are demanding more from the machine than this limit. > Cgroups should allow YARN containers to be limited to allocated cores > - > > Key: YARN-2440 > URL: https://issues.apache.org/jira/browse/YARN-2440 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch, > apache-yarn-2440.2.patch, screenshot-current-implementation.jpg > > > The current cgroups implementation does not limit YARN containers to the > cores allocated in yarn-site.xml. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores
[ https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14110567#comment-14110567 ] Beckham007 commented on YARN-2440: -- In addition, mesos uses cpuset for default. https://github.com/apache/mesos/blob/master/src/linux/cgroups.cpp > Cgroups should allow YARN containers to be limited to allocated cores > - > > Key: YARN-2440 > URL: https://issues.apache.org/jira/browse/YARN-2440 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch, > apache-yarn-2440.2.patch, screenshot-current-implementation.jpg > > > The current cgroups implementation does not limit YARN containers to the > cores allocated in yarn-site.xml. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores
[ https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14110566#comment-14110566 ] Beckham007 commented on YARN-2440: -- hi, [~vvasudev] “Cpusets provide a mechanism for assigning a set of CPUs and Memory Nodes to a set of tasks.” https://www.kernel.org/doc/Documentation/cgroups/cpusets.txt For a NM has 24 pcores, we can use cpuset subsystem to make hadoop-yarn use cpu core 0-21, and left the others((22,23) for system. And then using cpu.shares to share the pcore 0-21. What's more, we can assign a pcore(such as core 21) to run a long-running container, and other containers only share pcore 0-20. > Cgroups should allow YARN containers to be limited to allocated cores > - > > Key: YARN-2440 > URL: https://issues.apache.org/jira/browse/YARN-2440 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch, > apache-yarn-2440.2.patch, screenshot-current-implementation.jpg > > > The current cgroups implementation does not limit YARN containers to the > cores allocated in yarn-site.xml. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores
[ https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14110556#comment-14110556 ] Varun Vasudev commented on YARN-2440: - [~beckham007] the current implementation of Cgroups uses cpu instead of cpuset, probably due to the flexibility offered(sharing the cores is handled by the kernel). Is there any particular benefit to cpuset? > Cgroups should allow YARN containers to be limited to allocated cores > - > > Key: YARN-2440 > URL: https://issues.apache.org/jira/browse/YARN-2440 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch, > apache-yarn-2440.2.patch, screenshot-current-implementation.jpg > > > The current cgroups implementation does not limit YARN containers to the > cores allocated in yarn-site.xml. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores
[
https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14110555#comment-14110555
]
Varun Vasudev commented on YARN-2440:
-
[~jlowe] the example provided by [~sjlee0] is the one I wanted to address when
I added support for both percentage and absolute cores. Would it make more
sense if I picked the lower value instead of one overriding the other.
Something like -
1. Evaluate cores allocated by yarn.nodemanager.containers-cpu-cores and
yarn.nodemanager.containers-cpu-percentage.
2. Pick the lower of the two values
3. Log a warning/info message that both were specified and that we're picking
the lower value.
{quote}
I'm not thrilled about the name template "containers-cpu-*" since it could
easily be misinterpreted as a per-container thing as well, but I'm currently at
a loss for a better prefix. Suggestions welcome.
{quote}
How about yarn.nodemanager.all-containers-cpu-cores and
yarn.nodemanager.all-containers-cpu-percentage?
{quote}
Does getOverallLimits need to check for a quotaUS that's too low as well?
{quote}
Thanks for catching this; I'll fix it in the next patch.
{quote}
I think minimally we need to log a warning if we're going to ignore setting up
cgroups to limit CPU usage across all containers if the user specified to do so.
{quote}
I'll add in the logging message.
{quote}
Related to the previous comment, I think it would be nice if we didn't try to
setup any limits if none were specified. That way if there's some issue with
correctly determining the number of cores on a particular system it can still
work in the default, "use everything" scenario.
{quote}
Will do.
{quote}
NodeManagerHardwareUtils.getContainerCores should be getContainersCores (the
per-container vs. all-containers confusion again)
{quote}
I'll rename the function.
> Cgroups should allow YARN containers to be limited to allocated cores
> -
>
> Key: YARN-2440
> URL: https://issues.apache.org/jira/browse/YARN-2440
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch,
> apache-yarn-2440.2.patch, screenshot-current-implementation.jpg
>
>
> The current cgroups implementation does not limit YARN containers to the
> cores allocated in yarn-site.xml.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores
[ https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14110227#comment-14110227 ] Beckham007 commented on YARN-2440: -- Hi, all Why not use the cpuset subsystem of cgroups? The cpuset could make container to run on allocated cores, and reserving some cores for system. > Cgroups should allow YARN containers to be limited to allocated cores > - > > Key: YARN-2440 > URL: https://issues.apache.org/jira/browse/YARN-2440 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch, > apache-yarn-2440.2.patch, screenshot-current-implementation.jpg > > > The current cgroups implementation does not limit YARN containers to the > cores allocated in yarn-site.xml. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores
[
https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14109856#comment-14109856
]
Sangjin Lee commented on YARN-2440:
---
It might be good to use several fairly representative scenarios and see how we
can satisfy them with clear configuration.
One scenario I can see pretty common is this (just for illustration):
- 8-core system
- want to use only 6 cores for containers (reserving 2 for NM and DN, etc.)
- want to allocate 1/2 core per container by default
IMO, the simplest config is
{panel}
yarn.nodemanager.resource.cpu-vcores = 60
yarn.nodemanager.containers-cores-to-vcores = 10
each container asks 5 vcores
{panel}
Or I could have
{panel}
yarn.nodemanager.resource.cpu-vcores = 60
yarn.nodemanager.containers-cpu-cores = 6 (core-to-vcore ratio understood as
the ratio of these two)
each container asks 5 vcores
{panel}
I'm not sure how I can use containers-cpu-percentage to describe this
scenario...
Does this help? Are there other types of use cases that we should review this
with?
> Cgroups should allow YARN containers to be limited to allocated cores
> -
>
> Key: YARN-2440
> URL: https://issues.apache.org/jira/browse/YARN-2440
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch,
> apache-yarn-2440.2.patch, screenshot-current-implementation.jpg
>
>
> The current cgroups implementation does not limit YARN containers to the
> cores allocated in yarn-site.xml.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores
[ https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14109801#comment-14109801 ] Jason Lowe commented on YARN-2440: -- Thanks for updating the patch, Varun. I don't see why we need both a containers-cpu-cores and containers-cpu-percentage, and I think it leads to confusion when both exist. At first I did not realize that one overrode the other. Instead I assumed that if you set cpu-cores to X and cpu-percentage to Y then you were requesting Y% of X cores. Then there's the additional question of whether container usage is pinned to those cores, etc. Only having cpu-percentage is a simpler model that still allows the user to specify cores indirectly (e.g.: 25% of an 8 core system is 2 cores). Maybe I'm missing the use case where we really need containers-cpu-cores and the confusing (to me at least) override behavior between the two properties. Other comments on the patch: - I'm not thrilled about the name template "containers-cpu-*" since it could easily be misinterpreted as a per-container thing as well, but I'm currently at a loss for a better prefix. Suggestions welcome. - Does getOverallLimits need to check for a quotaUS that's too low as well? - I think minimally we need to log a warning if we're going to ignore setting up cgroups to limit CPU usage across all containers if the user specified to do so. - Related to the previous comment, I think it would be nice if we didn't try to setup any limits if none were specified. That way if there's some issue with correctly determining the number of cores on a particular system it can still work in the default, "use everything" scenario. - NodeManagerHardwareUtils.getContainerCores should be getContainersCores (the per-container vs. all-containers confusion again) > Cgroups should allow YARN containers to be limited to allocated cores > - > > Key: YARN-2440 > URL: https://issues.apache.org/jira/browse/YARN-2440 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch, > apache-yarn-2440.2.patch, screenshot-current-implementation.jpg > > > The current cgroups implementation does not limit YARN containers to the > cores allocated in yarn-site.xml. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores
[
https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14108751#comment-14108751
]
Hadoop QA commented on YARN-2440:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12664090/apache-yarn-2440.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4714//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4714//console
This message is automatically generated.
> Cgroups should allow YARN containers to be limited to allocated cores
> -
>
> Key: YARN-2440
> URL: https://issues.apache.org/jira/browse/YARN-2440
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch,
> apache-yarn-2440.2.patch, screenshot-current-implementation.jpg
>
>
> The current cgroups implementation does not limit YARN containers to the
> cores allocated in yarn-site.xml.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores
[ https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14108718#comment-14108718 ] Varun Vasudev commented on YARN-2440: - Update title to reflect what this patch is doing. > Cgroups should allow YARN containers to be limited to allocated cores > - > > Key: YARN-2440 > URL: https://issues.apache.org/jira/browse/YARN-2440 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch, > screenshot-current-implementation.jpg > > > The current cgroups implementation does not limit YARN containers to the > cores allocated in yarn-site.xml. -- This message was sent by Atlassian JIRA (v6.2#6252)
