[jira] [Commented] (YARN-600) Hook up cgroups CPU settings to the number of virtual cores allocated
[ https://issues.apache.org/jira/browse/YARN-600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13682267#comment-13682267 ] Hudson commented on YARN-600: - Integrated in Hadoop-Mapreduce-trunk #1456 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1456/]) YARN-600. Hook up cgroups CPU settings to the number of virtual cores allocated. (sandyr via tucu) (Revision 1492365) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1492365 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java > Hook up cgroups CPU settings to the number of virtual cores allocated > - > > Key: YARN-600 > URL: https://issues.apache.org/jira/browse/YARN-600 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager, scheduler >Affects Versions: 2.0.3-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.1.0-beta > > Attachments: YARN-600.patch > > > YARN-3 introduced CPU isolation and monitoring through cgroups. YARN-2 and > introduced CPU scheduling in the capacity scheduler, and YARN-326 will > introduce it in the fair scheduler. The number of virtual cores allocated to > a container should be used to weight the number of cgroups CPU shares given > to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-600) Hook up cgroups CPU settings to the number of virtual cores allocated
[ https://issues.apache.org/jira/browse/YARN-600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13682218#comment-13682218 ] Hudson commented on YARN-600: - Integrated in Hadoop-Hdfs-trunk #1429 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1429/]) YARN-600. Hook up cgroups CPU settings to the number of virtual cores allocated. (sandyr via tucu) (Revision 1492365) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1492365 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java > Hook up cgroups CPU settings to the number of virtual cores allocated > - > > Key: YARN-600 > URL: https://issues.apache.org/jira/browse/YARN-600 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager, scheduler >Affects Versions: 2.0.3-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.1.0-beta > > Attachments: YARN-600.patch > > > YARN-3 introduced CPU isolation and monitoring through cgroups. YARN-2 and > introduced CPU scheduling in the capacity scheduler, and YARN-326 will > introduce it in the fair scheduler. The number of virtual cores allocated to > a container should be used to weight the number of cgroups CPU shares given > to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-600) Hook up cgroups CPU settings to the number of virtual cores allocated
[ https://issues.apache.org/jira/browse/YARN-600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13682114#comment-13682114 ] Hudson commented on YARN-600: - Integrated in Hadoop-Yarn-trunk #239 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/239/]) YARN-600. Hook up cgroups CPU settings to the number of virtual cores allocated. (sandyr via tucu) (Revision 1492365) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1492365 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java > Hook up cgroups CPU settings to the number of virtual cores allocated > - > > Key: YARN-600 > URL: https://issues.apache.org/jira/browse/YARN-600 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager, scheduler >Affects Versions: 2.0.3-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.1.0-beta > > Attachments: YARN-600.patch > > > YARN-3 introduced CPU isolation and monitoring through cgroups. YARN-2 and > introduced CPU scheduling in the capacity scheduler, and YARN-326 will > introduce it in the fair scheduler. The number of virtual cores allocated to > a container should be used to weight the number of cgroups CPU shares given > to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-600) Hook up cgroups CPU settings to the number of virtual cores allocated
[ https://issues.apache.org/jira/browse/YARN-600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13681514#comment-13681514 ] Hudson commented on YARN-600: - Integrated in Hadoop-trunk-Commit #3907 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3907/]) YARN-600. Hook up cgroups CPU settings to the number of virtual cores allocated. (sandyr via tucu) (Revision 1492365) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1492365 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java > Hook up cgroups CPU settings to the number of virtual cores allocated > - > > Key: YARN-600 > URL: https://issues.apache.org/jira/browse/YARN-600 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager, scheduler >Affects Versions: 2.0.3-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.1.0-beta > > Attachments: YARN-600.patch > > > YARN-3 introduced CPU isolation and monitoring through cgroups. YARN-2 and > introduced CPU scheduling in the capacity scheduler, and YARN-326 will > introduce it in the fair scheduler. The number of virtual cores allocated to > a container should be used to weight the number of cgroups CPU shares given > to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-600) Hook up cgroups CPU settings to the number of virtual cores allocated
[ https://issues.apache.org/jira/browse/YARN-600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13681492#comment-13681492 ] Alejandro Abdelnur commented on YARN-600: - Chris, thanks for verifying the patch works in cgroup CPU controller environment. +1 > Hook up cgroups CPU settings to the number of virtual cores allocated > - > > Key: YARN-600 > URL: https://issues.apache.org/jira/browse/YARN-600 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager, scheduler >Affects Versions: 2.0.3-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-600.patch > > > YARN-3 introduced CPU isolation and monitoring through cgroups. YARN-2 and > introduced CPU scheduling in the capacity scheduler, and YARN-326 will > introduce it in the fair scheduler. The number of virtual cores allocated to > a container should be used to weight the number of cgroups CPU shares given > to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-600) Hook up cgroups CPU settings to the number of virtual cores allocated
[
https://issues.apache.org/jira/browse/YARN-600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13681486#comment-13681486
]
Chris Riccomini commented on YARN-600:
--
Hey Sandy,
I just applied your patch to my local YARN, and can verify that it appears to
be working.
{noformat}
$ cat container_1371061837111_0001_01_02/cpu.shares
1024
$ cat container_1371061837111_0002_01_02/cpu.shares
32768
{noformat}
I have 8 of the second type of container (32768 cpu shares) on an 8 core
machine. When running 8 * 32768 and 1 * 1024, I get a top that looks like this:
{noformat}
1404 criccomi 20 0 1022m 108m 13m S 98.1 0.2 2:33.03
/export/apps/jdk/JDK-1_6_0_27/bin/java -Xmx160M -XX:+PrintGCDateStamps
-Xloggc:/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs/userlogs/application_1371061837111_0005/container_1371061837111_0005_01_02/gc.log
-Dlog4j.configuration=file:/tmp/hadoop-cri
3192 criccomi 20 0 1022m 109m 13m S 98.1 0.2 2:25.93
/export/apps/jdk/JDK-1_6_0_27/bin/java -Xmx160M -XX:+PrintGCDateStamps
-Xloggc:/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs/userlogs/application_1371061837111_0009/container_1371061837111_0009_01_02/gc.log
-Dlog4j.configuration=file:/tmp/hadoop-cri
428 criccomi 20 0 1022m 109m 13m S 97.7 0.2 2:36.41
/export/apps/jdk/JDK-1_6_0_27/bin/java -Xmx160M -XX:+PrintGCDateStamps
-Xloggc:/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs/userlogs/application_1371061837111_0004/container_1371061837111_0004_01_02/gc.log
-Dlog4j.configuration=file:/tmp/hadoop-cri
3022 criccomi 20 0 1022m 110m 13m S 97.2 0.2 2:29.74
/export/apps/jdk/JDK-1_6_0_27/bin/java -Xmx160M -XX:+PrintGCDateStamps
-Xloggc:/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs/userlogs/application_1371061837111_0007/container_1371061837111_0007_01_02/gc.log
-Dlog4j.configuration=file:/tmp/hadoop-cri
32443 criccomi 20 0 1022m 109m 13m S 95.1 0.2 2:40.17
/export/apps/jdk/JDK-1_6_0_27/bin/java -Xmx160M -XX:+PrintGCDateStamps
-Xloggc:/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs/userlogs/application_1371061837111_0003/container_1371061837111_0003_01_02/gc.log
-Dlog4j.configuration=file:/tmp/hadoop-cri
2850 criccomi 20 0 1022m 107m 13m S 93.6 0.2 2:31.09
/export/apps/jdk/JDK-1_6_0_27/bin/java -Xmx160M -XX:+PrintGCDateStamps
-Xloggc:/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs/userlogs/application_1371061837111_0006/container_1371061837111_0006_01_02/gc.log
-Dlog4j.configuration=file:/tmp/hadoop-cri
3112 criccomi 20 0 1022m 108m 13m S 93.2 0.2 2:25.54
/export/apps/jdk/JDK-1_6_0_27/bin/java -Xmx160M -XX:+PrintGCDateStamps
-Xloggc:/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs/userlogs/application_1371061837111_0008/container_1371061837111_0008_01_02/gc.log
-Dlog4j.configuration=file:/tmp/hadoop-cri
31038 criccomi 20 0 1022m 109m 13m S 84.5 0.2 3:07.39
/export/apps/jdk/JDK-1_6_0_27/bin/java -Xmx160M -XX:+PrintGCDateStamps
-Xloggc:/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs/userlogs/application_1371061837111_0002/container_1371061837111_0002_01_02/gc.log
-Dlog4j.configuration=file:/tmp/hadoop-cri
29451 criccomi 20 0 1925m 249m 13m S 16.3 0.4 0:33.29
/export/apps/jdk/JDK-1_6_0_27/bin/java -Dproc_nodemanager -Xmx1000m -server
-Dhadoop.log.dir=/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs
-Dyarn.log.dir=/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs
-Dhadoop.log.file=yarn.log -Dyarn.log.file=yarn.lo
30447 criccomi 20 0 1022m 109m 13m S 3.7 0.2 1:28.42
/export/apps/jdk/JDK-1_6_0_27/bin/java -Xmx160M -XX:+PrintGCDateStamps
-Xloggc:/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs/userlogs/application_1371061837111_0001/container_1371061837111_0001_01_02/gc.log
-Dlog4j.configuration=file:/tmp/hadoop-cri
{noformat}
The column that starts with 98 is the CPU column. As you can see,
container_1371061837111_0001_01_02 is only taking 3% CPU, while the other
processes are taking 100%. They're all doing the same thing to burn up CPU, so
it appears the CGroups are throttling my 1024 container as expected.
Cheers,
Chris
> Hook up cgroups CPU settings to the number of virtual cores allocated
> -
>
> Key: YARN-600
> URL: https://issues.apache.org/jira/browse/YARN-600
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager, scheduler
>Affects Versions: 2.0.3-alpha
>Reporter: Sandy Ryza
>Assignee: Sandy Ryza
> Attachments: YARN-600.patch
>
>
> YARN-3 introduced CPU isolation and monitoring through cgroups. YARN-2 and
> introduced CPU scheduling in the capacity scheduler, and YARN-326 will
> introduce it in the fair scheduler. The number of virtual cores allocated to
> a container should be used to
[jira] [Commented] (YARN-600) Hook up cgroups CPU settings to the number of virtual cores allocated
[
https://issues.apache.org/jira/browse/YARN-600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13681460#comment-13681460
]
Hadoop QA commented on YARN-600:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12587466/YARN-600.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1203//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1203//console
This message is automatically generated.
> Hook up cgroups CPU settings to the number of virtual cores allocated
> -
>
> Key: YARN-600
> URL: https://issues.apache.org/jira/browse/YARN-600
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager, scheduler
>Affects Versions: 2.0.3-alpha
>Reporter: Sandy Ryza
>Assignee: Sandy Ryza
> Attachments: YARN-600.patch
>
>
> YARN-3 introduced CPU isolation and monitoring through cgroups. YARN-2 and
> introduced CPU scheduling in the capacity scheduler, and YARN-326 will
> introduce it in the fair scheduler. The number of virtual cores allocated to
> a container should be used to weight the number of cgroups CPU shares given
> to it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-600) Hook up cgroups CPU settings to the number of virtual cores allocated
[ https://issues.apache.org/jira/browse/YARN-600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13681437#comment-13681437 ] Sandy Ryza commented on YARN-600: - Submitted a simple patch. Haven't had a chance to verify it manually yet. > Hook up cgroups CPU settings to the number of virtual cores allocated > - > > Key: YARN-600 > URL: https://issues.apache.org/jira/browse/YARN-600 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager, scheduler >Affects Versions: 2.0.3-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-600.patch > > > YARN-3 introduced CPU isolation and monitoring through cgroups. YARN-2 and > introduced CPU scheduling in the capacity scheduler, and YARN-326 will > introduce it in the fair scheduler. The number of virtual cores allocated to > a container should be used to weight the number of cgroups CPU shares given > to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-600) Hook up cgroups CPU settings to the number of virtual cores allocated
[ https://issues.apache.org/jira/browse/YARN-600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13646436#comment-13646436 ] Sandy Ryza commented on YARN-600: - (by which I mean YARN doesn't give hints to the OS on how and when to assign cores to processes) > Hook up cgroups CPU settings to the number of virtual cores allocated > - > > Key: YARN-600 > URL: https://issues.apache.org/jira/browse/YARN-600 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager, scheduler >Affects Versions: 2.0.3-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > > YARN-3 introduced CPU isolation and monitoring through cgroups. YARN-2 and > introduced CPU scheduling in the capacity scheduler, and YARN-326 will > introduce it in the fair scheduler. The number of virtual cores allocated to > a container should be used to weight the number of cgroups CPU shares given > to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-600) Hook up cgroups CPU settings to the number of virtual cores allocated
[ https://issues.apache.org/jira/browse/YARN-600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13646435#comment-13646435 ] Sandy Ryza commented on YARN-600: - I don't believe anything special happens when it's not enabled. > Hook up cgroups CPU settings to the number of virtual cores allocated > - > > Key: YARN-600 > URL: https://issues.apache.org/jira/browse/YARN-600 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager, scheduler >Affects Versions: 2.0.3-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > > YARN-3 introduced CPU isolation and monitoring through cgroups. YARN-2 and > introduced CPU scheduling in the capacity scheduler, and YARN-326 will > introduce it in the fair scheduler. The number of virtual cores allocated to > a container should be used to weight the number of cgroups CPU shares given > to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-600) Hook up cgroups CPU settings to the number of virtual cores allocated
[ https://issues.apache.org/jira/browse/YARN-600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639090#comment-13639090 ] Timothy St. Clair commented on YARN-600: What happens when a node does not have it enabled? SetAffinity? > Hook up cgroups CPU settings to the number of virtual cores allocated > - > > Key: YARN-600 > URL: https://issues.apache.org/jira/browse/YARN-600 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager, scheduler >Affects Versions: 2.0.3-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > > YARN-3 introduced CPU isolation and monitoring through cgroups. YARN-2 and > introduced CPU scheduling in the capacity scheduler, and YARN-326 will > introduce it in the fair scheduler. The number of virtual cores allocated to > a container should be used to weight the number of cgroups CPU shares given > to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
