[jira] [Commented] (YARN-1775) Create SMAPBasedProcessTree to get PSS information
[ https://issues.apache.org/jira/browse/YARN-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14041740#comment-14041740 ] Steven Wong commented on YARN-1775: --- [~rajesh.balamohan], can you explain why you want to exclude 'the read only shared memory mappings in the process (i.e r--s, r-xs)'? Thanks. Create SMAPBasedProcessTree to get PSS information -- Key: YARN-1775 URL: https://issues.apache.org/jira/browse/YARN-1775 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Fix For: 2.4.0 Attachments: YARN-1775-v2.patch, YARN-1775-v3.patch, YARN-1775-v3.patch, YARN-1775-v4.patch, YARN-1775-v5.patch, yarn-1775-2.4.0.patch Create SMAPBasedProcessTree (by extending ProcfsBasedProcessTree), which will make use of PSS for computing the memory usage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1775) Create SMAPBasedProcessTree to get PSS information
[ https://issues.apache.org/jira/browse/YARN-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944015#comment-13944015 ] Hudson commented on YARN-1775: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #517 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/517/]) YARN-1775. Enhanced ProcfsBasedProcessTree to optionally add the ability to use smaps for obtaining used memory information. Contributed by Rajesh Balamohan. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1580087) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java Create SMAPBasedProcessTree to get PSS information -- Key: YARN-1775 URL: https://issues.apache.org/jira/browse/YARN-1775 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Fix For: 2.4.0 Attachments: YARN-1775-v2.patch, YARN-1775-v3.patch, YARN-1775-v3.patch, YARN-1775-v4.patch, YARN-1775-v5.patch, yarn-1775-2.4.0.patch Create SMAPBasedProcessTree (by extending ProcfsBasedProcessTree), which will make use of PSS for computing the memory usage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1775) Create SMAPBasedProcessTree to get PSS information
[ https://issues.apache.org/jira/browse/YARN-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944068#comment-13944068 ] Hudson commented on YARN-1775: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1709 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1709/]) YARN-1775. Enhanced ProcfsBasedProcessTree to optionally add the ability to use smaps for obtaining used memory information. Contributed by Rajesh Balamohan. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1580087) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java Create SMAPBasedProcessTree to get PSS information -- Key: YARN-1775 URL: https://issues.apache.org/jira/browse/YARN-1775 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Fix For: 2.4.0 Attachments: YARN-1775-v2.patch, YARN-1775-v3.patch, YARN-1775-v3.patch, YARN-1775-v4.patch, YARN-1775-v5.patch, yarn-1775-2.4.0.patch Create SMAPBasedProcessTree (by extending ProcfsBasedProcessTree), which will make use of PSS for computing the memory usage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1775) Create SMAPBasedProcessTree to get PSS information
[ https://issues.apache.org/jira/browse/YARN-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944105#comment-13944105 ] Hudson commented on YARN-1775: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1734 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1734/]) YARN-1775. Enhanced ProcfsBasedProcessTree to optionally add the ability to use smaps for obtaining used memory information. Contributed by Rajesh Balamohan. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1580087) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java Create SMAPBasedProcessTree to get PSS information -- Key: YARN-1775 URL: https://issues.apache.org/jira/browse/YARN-1775 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Fix For: 2.4.0 Attachments: YARN-1775-v2.patch, YARN-1775-v3.patch, YARN-1775-v3.patch, YARN-1775-v4.patch, YARN-1775-v5.patch, yarn-1775-2.4.0.patch Create SMAPBasedProcessTree (by extending ProcfsBasedProcessTree), which will make use of PSS for computing the memory usage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1775) Create SMAPBasedProcessTree to get PSS information
[
https://issues.apache.org/jira/browse/YARN-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13942872#comment-13942872
]
Hadoop QA commented on YARN-1775:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12635971/YARN-1775-v4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3420//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3420//console
This message is automatically generated.
Create SMAPBasedProcessTree to get PSS information
--
Key: YARN-1775
URL: https://issues.apache.org/jira/browse/YARN-1775
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Reporter: Rajesh Balamohan
Assignee: Rajesh Balamohan
Attachments: YARN-1775-v2.patch, YARN-1775-v3.patch,
YARN-1775-v3.patch, YARN-1775-v4.patch, yarn-1775-2.4.0.patch
Create SMAPBasedProcessTree (by extending ProcfsBasedProcessTree), which will
make use of PSS for computing the memory usage.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1775) Create SMAPBasedProcessTree to get PSS information
[ https://issues.apache.org/jira/browse/YARN-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13943690#comment-13943690 ] Karthik Kambatla commented on YARN-1775: The latest patch looks good to me too. Create SMAPBasedProcessTree to get PSS information -- Key: YARN-1775 URL: https://issues.apache.org/jira/browse/YARN-1775 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: YARN-1775-v2.patch, YARN-1775-v3.patch, YARN-1775-v3.patch, YARN-1775-v4.patch, YARN-1775-v5.patch, yarn-1775-2.4.0.patch Create SMAPBasedProcessTree (by extending ProcfsBasedProcessTree), which will make use of PSS for computing the memory usage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1775) Create SMAPBasedProcessTree to get PSS information
[
https://issues.apache.org/jira/browse/YARN-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13943700#comment-13943700
]
Hadoop QA commented on YARN-1775:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12636125/YARN-1775-v5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3428//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3428//console
This message is automatically generated.
Create SMAPBasedProcessTree to get PSS information
--
Key: YARN-1775
URL: https://issues.apache.org/jira/browse/YARN-1775
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Reporter: Rajesh Balamohan
Assignee: Rajesh Balamohan
Attachments: YARN-1775-v2.patch, YARN-1775-v3.patch,
YARN-1775-v3.patch, YARN-1775-v4.patch, YARN-1775-v5.patch,
yarn-1775-2.4.0.patch
Create SMAPBasedProcessTree (by extending ProcfsBasedProcessTree), which will
make use of PSS for computing the memory usage.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1775) Create SMAPBasedProcessTree to get PSS information
[ https://issues.apache.org/jira/browse/YARN-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13941725#comment-13941725 ] Rajesh Balamohan commented on YARN-1775: [~vinodkv] - Sure, it would be great if this can be accommodated in ProcfsBasedProcessTree itself. I will add a config flag to enable/disable smap based computation (default is disabled). I will upload the patch soon. - To get realistic RSS of a process, we need Private_Clean + Private_Dirty + Shared_Dirty of the process (i.e shared_dirty/n where n is the number of processes sharing). If we sum up all Shared_Dirty of memory regions, we would be double counting across processes. If we take PSS, then we would end up counting Shared_Clean as well. So closer approximation of real shared_dirty can be obtained by doing Min(shared_dirty, PSS) = Min (shared_dirty, (shared_dirty+shared_clean)/n); This will fall somewhere between shared_dirty shared_dirty/n. Hence, closer approximation of RSS = Math.min(sharedDirty, pss) + privateDirty + privateClean. [~cnauroth] Also interesting would be confirming that containers still get killed for exceeding the limit with private/non-shared pages. Had a offline discussion with Gopal. Based on the suggestion, I tried to ByteBuffer.allocateDirect() which would create enough PRIVATE_DIRTY in the process. The process got killed once it exceeded the physical memory limits of the container. Create SMAPBasedProcessTree to get PSS information -- Key: YARN-1775 URL: https://issues.apache.org/jira/browse/YARN-1775 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: YARN-1775-v2.patch, yarn-1775-2.4.0.patch Create SMAPBasedProcessTree (by extending ProcfsBasedProcessTree), which will make use of PSS for computing the memory usage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1775) Create SMAPBasedProcessTree to get PSS information
[
https://issues.apache.org/jira/browse/YARN-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13941741#comment-13941741
]
Hadoop QA commented on YARN-1775:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12635783/YARN-1775-v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3405//console
This message is automatically generated.
Create SMAPBasedProcessTree to get PSS information
--
Key: YARN-1775
URL: https://issues.apache.org/jira/browse/YARN-1775
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Reporter: Rajesh Balamohan
Assignee: Rajesh Balamohan
Attachments: YARN-1775-v2.patch, yarn-1775-2.4.0.patch
Create SMAPBasedProcessTree (by extending ProcfsBasedProcessTree), which will
make use of PSS for computing the memory usage.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1775) Create SMAPBasedProcessTree to get PSS information
[
https://issues.apache.org/jira/browse/YARN-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13941944#comment-13941944
]
Hadoop QA commented on YARN-1775:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12635792/YARN-1775-v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3408//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3408//console
This message is automatically generated.
Create SMAPBasedProcessTree to get PSS information
--
Key: YARN-1775
URL: https://issues.apache.org/jira/browse/YARN-1775
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Reporter: Rajesh Balamohan
Assignee: Rajesh Balamohan
Attachments: YARN-1775-v2.patch, YARN-1775-v3.patch,
yarn-1775-2.4.0.patch
Create SMAPBasedProcessTree (by extending ProcfsBasedProcessTree), which will
make use of PSS for computing the memory usage.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1775) Create SMAPBasedProcessTree to get PSS information
[ https://issues.apache.org/jira/browse/YARN-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13941989#comment-13941989 ] Chris Nauroth commented on YARN-1775: - Thanks for the further testing, Rajesh! +1 from me. The patch looks good, but I'm not going to commit it in case Vinod or others have additional feedback. Create SMAPBasedProcessTree to get PSS information -- Key: YARN-1775 URL: https://issues.apache.org/jira/browse/YARN-1775 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: YARN-1775-v2.patch, YARN-1775-v3.patch, yarn-1775-2.4.0.patch Create SMAPBasedProcessTree (by extending ProcfsBasedProcessTree), which will make use of PSS for computing the memory usage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1775) Create SMAPBasedProcessTree to get PSS information
[ https://issues.apache.org/jira/browse/YARN-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13941997#comment-13941997 ] Karthik Kambatla commented on YARN-1775: I ll try to take a look at it today. If I don't get to it today, please feel free to commit it. Create SMAPBasedProcessTree to get PSS information -- Key: YARN-1775 URL: https://issues.apache.org/jira/browse/YARN-1775 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: YARN-1775-v2.patch, YARN-1775-v3.patch, yarn-1775-2.4.0.patch Create SMAPBasedProcessTree (by extending ProcfsBasedProcessTree), which will make use of PSS for computing the memory usage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1775) Create SMAPBasedProcessTree to get PSS information
[
https://issues.apache.org/jira/browse/YARN-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13942631#comment-13942631
]
Karthik Kambatla commented on YARN-1775:
bq. closer approximation of RSS = Math.min(sharedDirty, pss) + privateDirty +
privateClean.
Sounds reasonable to me. While enforcing memory constraints, I wonder if people
would want to use any other definitions of RSS to be more conservative or
aggressive. Do you think it would make sense to provide these options
separately, and have what you have as the default? We can punt this to a
different JIRA, just wanted to bring it up.
Whichever we choose to go, we should definitely the way RSS is computed in
yarn-default.xml, so users understand what they are switching to.
bq.
yarn.nodemanager.container-monitor.procfs-based-proces-tree.smaps-based-rss.enabled
That seems awfully long. May be something between what the patch has and
Vinod's suggestion :)
Other than that, adding to Vinod's cosmetic changes list:
# Rename MEM_INFO to MemInfo to go with other enums in the source?
# Also, may be just list the values line-by-line, so we can add comments
corresponding to values, if any.
# We should probably switch the following two ifs?
{code}
if (isSmapEnabled()) {
return getSmapBasedCumulativeRssmem(olderThanAge);
}
if (PAGE_SIZE 0) {
return 0;
}
{code}
# javadoc error?
{code}
* @param pinfo
* @param procfsDir
*/
private static void constructProcessSMAPInfo(ProcessMemInfo pInfo,
{code}
Create SMAPBasedProcessTree to get PSS information
--
Key: YARN-1775
URL: https://issues.apache.org/jira/browse/YARN-1775
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Reporter: Rajesh Balamohan
Assignee: Rajesh Balamohan
Attachments: YARN-1775-v2.patch, YARN-1775-v3.patch,
yarn-1775-2.4.0.patch
Create SMAPBasedProcessTree (by extending ProcfsBasedProcessTree), which will
make use of PSS for computing the memory usage.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1775) Create SMAPBasedProcessTree to get PSS information
[ https://issues.apache.org/jira/browse/YARN-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13942711#comment-13942711 ] Vinod Kumar Vavilapalli commented on YARN-1775: --- bq. Also, I wonder if we could make this default to true and simply disable it when /proc/PID/smaps is not present or writable. Let's wait for a little while to make sure this is reliable enough and working well. May be, in the 2.5/2.6 releases. Create SMAPBasedProcessTree to get PSS information -- Key: YARN-1775 URL: https://issues.apache.org/jira/browse/YARN-1775 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: YARN-1775-v2.patch, YARN-1775-v3.patch, yarn-1775-2.4.0.patch Create SMAPBasedProcessTree (by extending ProcfsBasedProcessTree), which will make use of PSS for computing the memory usage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1775) Create SMAPBasedProcessTree to get PSS information
[
https://issues.apache.org/jira/browse/YARN-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13941094#comment-13941094
]
Vinod Kumar Vavilapalli commented on YARN-1775:
---
Some comments:
- This new class is just an extension of ProcfsBasedProcessTree, so is better
served by using the same util class (from the admin's point of view) but with
an additional configuration option to better track RSS
- Can you explain why we are doing this
{code}
+total += Math.min(info.sharedDirty, info.pss) + info.privateDirty
++ info.privateClean;
{code}
Test
- Most of the test-code is duplicating testProcfsBasedProcess. Can you avoid
that?
- Reuse at least some of MemoryMappingInfo, ProcessMemInfo etc from the
regular code instead of duplicating in the test?
Lost of white spaces in the patch, mostly empty lines.
Create SMAPBasedProcessTree to get PSS information
--
Key: YARN-1775
URL: https://issues.apache.org/jira/browse/YARN-1775
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Reporter: Rajesh Balamohan
Assignee: Rajesh Balamohan
Attachments: yarn-1775-2.4.0.patch
Create SMAPBasedProcessTree (by extending ProcfsBasedProcessTree), which will
make use of PSS for computing the memory usage.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1775) Create SMAPBasedProcessTree to get PSS information
[
https://issues.apache.org/jira/browse/YARN-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13941120#comment-13941120
]
Chris Nauroth commented on YARN-1775:
-
[~rajesh.balamohan], thank you for explaining the testing. These results sound
very promising! Also interesting would be confirming that containers still get
killed for exceeding the limit with private/non-shared pages.
I wonder then if counting RSS still has some potential advantages in certain
deployments, or if the PSS approach is always superior. Your testing so far
seems to indicate that PSS is always superior. Therefore, should this just be
combined right into the current code? (This echoes Vinod's prior comment about
folding the logic back into {{ProcfsBasedProcessTree}}.) A conservative
approach is to introduce a config flag, try to get some experience running it
in real-world clusters, and then we can flip the default in a later release if
it goes well.
Create SMAPBasedProcessTree to get PSS information
--
Key: YARN-1775
URL: https://issues.apache.org/jira/browse/YARN-1775
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Reporter: Rajesh Balamohan
Assignee: Rajesh Balamohan
Attachments: yarn-1775-2.4.0.patch
Create SMAPBasedProcessTree (by extending ProcfsBasedProcessTree), which will
make use of PSS for computing the memory usage.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1775) Create SMAPBasedProcessTree to get PSS information
[ https://issues.apache.org/jira/browse/YARN-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13938938#comment-13938938 ] Karthik Kambatla commented on YARN-1775: Just barely went over the patch. Can we add documentation on the advantages of SMAPBasedProcessTree over ProcfsBasedProcessTree. Also, orthogonally, would we want to limit the number of ResourceCalculatorProcessTree implementations for any reason? I was looking into a cgroups-based version and was wondering what the implications of this patch would be. Create SMAPBasedProcessTree to get PSS information -- Key: YARN-1775 URL: https://issues.apache.org/jira/browse/YARN-1775 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Rajesh Balamohan Priority: Minor Fix For: 2.5.0 Attachments: yarn-1775-2.4.0.patch Create SMAPBasedProcessTree (by extending ProcfsBasedProcessTree), which will make use of PSS for computing the memory usage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1775) Create SMAPBasedProcessTree to get PSS information
[
https://issues.apache.org/jira/browse/YARN-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13939599#comment-13939599
]
Chris Nauroth commented on YARN-1775:
-
bq. Tested with branch 2.4.0 on 20 node cluster.
Hi, [~rajesh.balamohan]. Can you please share more details on this testing?
What would be particularly interesting is pushing a cluster close to the limits
of its memory capacity and then observing the differences in behavior between
{{ProcfsBasedProcessTree}} and {{SMAPBasedProcessTree}}. The argument is that
{{ProcfsBasedProcessTree}} is effectively over-conservative and may kill
containers too quickly. Did this testing show that {{SMAPBasedProcessTree}}
was able to keep more containers running? If so, were there any negative
performance impacts observed? (I wondered about the potential for introducing
more thrashing as a side effect of killing containers less quickly.)
Thanks for taking on this effort!
Create SMAPBasedProcessTree to get PSS information
--
Key: YARN-1775
URL: https://issues.apache.org/jira/browse/YARN-1775
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Reporter: Rajesh Balamohan
Priority: Minor
Fix For: 2.5.0
Attachments: yarn-1775-2.4.0.patch
Create SMAPBasedProcessTree (by extending ProcfsBasedProcessTree), which will
make use of PSS for computing the memory usage.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1775) Create SMAPBasedProcessTree to get PSS information
[ https://issues.apache.org/jira/browse/YARN-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934829#comment-13934829 ] Rajesh Balamohan commented on YARN-1775: Review request link : https://reviews.apache.org/r/19220/ Create SMAPBasedProcessTree to get PSS information -- Key: YARN-1775 URL: https://issues.apache.org/jira/browse/YARN-1775 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Rajesh Balamohan Priority: Minor Attachments: yarn-1775-2.4.0.patch Create SMAPBasedProcessTree (by extending ProcfsBasedProcessTree), which will make use of PSS for computing the memory usage. -- This message was sent by Atlassian JIRA (v6.2#6252)
