[jira] [Updated] (YARN-1775) Create SMAPBasedProcessTree to get PSS information
[ https://issues.apache.org/jira/browse/YARN-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated YARN-1775: --- Attachment: YARN-1775-v3.patch getSmapBasedCumulativeRssmem() should be private -Fixed When converting #pages to bytes, use PAGE_SIZE instead of hard-coding 1024. -smap information has KB which needs to be converted to bytes. PAGE_SIZE mostly will be 4096 which will give wrong value in getSmapBasedCumulativeRssmem. Move the constant PROCFS_SMAPS_ENABLED to YarnConfiguration -Fixed. Suggestions for renames PROCFS_SMAPS_ENABLED - PROCFS_USE_SMAPS_BASED_RSS yarn.nodemanager.container-monitor.process-tree.smaps.enabled - yarn.nodemanager.container-monitor.procfs-based-proces-tree.smaps-based-rss.enabled. (Did I just say that? ) -Fixed (yarn.nodemanager.container-monitor.procfs-tree.smaps-based-rss.enabled). Still long I believe. ProcessMemInfo - ProcessTreeSmapMemInfo?, MemoryMappingInfo - ProcessSmapMemoryInfo, moduleMemList - memoryInfoList, processSMAPTree should be cleared in every iteration of updating the process-tree -Fixed isSmapEnabled() should be private -Removed this method completely. As a part of setConf() call, smapEnabled is computed. MemoryMappingInfo.updateModuleMemInfo: We should skip everything else when we run into integer parsing issue of the value. Right now you are logging, ignoring and continuing. -Fixed Rename MEM_INFO to MemInfo to go with other enums in the source? -Fixed We should probably switch the following two ifs? -Fixed Javadoc error -Fixed Reformatted the testcase as well. While enforcing memory constraints, I wonder if people would want to use any other definitions of RSS to be more conservative or aggressive. Do you think it would make sense to provide these options separately, and have what you have as the default? We can punt this to a different JIRA, just wanted to bring it up. -This option can be provided as advanced/expert configuration. We can have a separate JIRA to track it separately. Please feel free to open a new JIRA. Create SMAPBasedProcessTree to get PSS information -- Key: YARN-1775 URL: https://issues.apache.org/jira/browse/YARN-1775 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: YARN-1775-v2.patch, YARN-1775-v3.patch, YARN-1775-v3.patch, YARN-1775-v4.patch, yarn-1775-2.4.0.patch Create SMAPBasedProcessTree (by extending ProcfsBasedProcessTree), which will make use of PSS for computing the memory usage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-1775) Create SMAPBasedProcessTree to get PSS information
[ https://issues.apache.org/jira/browse/YARN-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated YARN-1775: --- Attachment: YARN-1775-v4.patch Renaming the patch as v4. Create SMAPBasedProcessTree to get PSS information -- Key: YARN-1775 URL: https://issues.apache.org/jira/browse/YARN-1775 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: YARN-1775-v2.patch, YARN-1775-v3.patch, YARN-1775-v3.patch, YARN-1775-v4.patch, yarn-1775-2.4.0.patch Create SMAPBasedProcessTree (by extending ProcfsBasedProcessTree), which will make use of PSS for computing the memory usage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-1775) Create SMAPBasedProcessTree to get PSS information
[ https://issues.apache.org/jira/browse/YARN-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1775: -- Attachment: YARN-1775-v5.patch Same patch with a few more renames. Will check this in if Jenkins says okay. Create SMAPBasedProcessTree to get PSS information -- Key: YARN-1775 URL: https://issues.apache.org/jira/browse/YARN-1775 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: YARN-1775-v2.patch, YARN-1775-v3.patch, YARN-1775-v3.patch, YARN-1775-v4.patch, YARN-1775-v5.patch, yarn-1775-2.4.0.patch Create SMAPBasedProcessTree (by extending ProcfsBasedProcessTree), which will make use of PSS for computing the memory usage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-1775) Create SMAPBasedProcessTree to get PSS information
[ https://issues.apache.org/jira/browse/YARN-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated YARN-1775: --- Attachment: YARN-1775-v2.patch Create SMAPBasedProcessTree to get PSS information -- Key: YARN-1775 URL: https://issues.apache.org/jira/browse/YARN-1775 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: YARN-1775-v2.patch, yarn-1775-2.4.0.patch Create SMAPBasedProcessTree (by extending ProcfsBasedProcessTree), which will make use of PSS for computing the memory usage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-1775) Create SMAPBasedProcessTree to get PSS information
[ https://issues.apache.org/jira/browse/YARN-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated YARN-1775: --- Attachment: YARN-1775-v3.patch Attaching correct patch. Create SMAPBasedProcessTree to get PSS information -- Key: YARN-1775 URL: https://issues.apache.org/jira/browse/YARN-1775 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: YARN-1775-v2.patch, YARN-1775-v3.patch, yarn-1775-2.4.0.patch Create SMAPBasedProcessTree (by extending ProcfsBasedProcessTree), which will make use of PSS for computing the memory usage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-1775) Create SMAPBasedProcessTree to get PSS information
[ https://issues.apache.org/jira/browse/YARN-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1775: -- Priority: Major (was: Minor) Fix Version/s: (was: 2.5.0) Started looking at the patch. But first up, please don't use fix-version to specify your intention. Target-version is what you should use, fix-version is set by committers at the time of commit. Create SMAPBasedProcessTree to get PSS information -- Key: YARN-1775 URL: https://issues.apache.org/jira/browse/YARN-1775 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: yarn-1775-2.4.0.patch Create SMAPBasedProcessTree (by extending ProcfsBasedProcessTree), which will make use of PSS for computing the memory usage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-1775) Create SMAPBasedProcessTree to get PSS information
[ https://issues.apache.org/jira/browse/YARN-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated YARN-1775: Assignee: Rajesh Balamohan Create SMAPBasedProcessTree to get PSS information -- Key: YARN-1775 URL: https://issues.apache.org/jira/browse/YARN-1775 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Priority: Minor Fix For: 2.5.0 Attachments: yarn-1775-2.4.0.patch Create SMAPBasedProcessTree (by extending ProcfsBasedProcessTree), which will make use of PSS for computing the memory usage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-1775) Create SMAPBasedProcessTree to get PSS information
[ https://issues.apache.org/jira/browse/YARN-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated YARN-1775: --- Attachment: yarn-1775-2.4.0.patch Computes the RSS by reading /proc/pid/smaps. Tested with branch 2.4.0 on 20 node cluster. Create SMAPBasedProcessTree to get PSS information -- Key: YARN-1775 URL: https://issues.apache.org/jira/browse/YARN-1775 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Rajesh Balamohan Priority: Minor Attachments: yarn-1775-2.4.0.patch Create SMAPBasedProcessTree (by extending ProcfsBasedProcessTree), which will make use of PSS for computing the memory usage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-1775) Create SMAPBasedProcessTree to get PSS information
[ https://issues.apache.org/jira/browse/YARN-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated YARN-1775: --- Fix Version/s: 2.5.0 Create SMAPBasedProcessTree to get PSS information -- Key: YARN-1775 URL: https://issues.apache.org/jira/browse/YARN-1775 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Rajesh Balamohan Priority: Minor Fix For: 2.5.0 Attachments: yarn-1775-2.4.0.patch Create SMAPBasedProcessTree (by extending ProcfsBasedProcessTree), which will make use of PSS for computing the memory usage. -- This message was sent by Atlassian JIRA (v6.2#6252)
