[
https://issues.apache.org/jira/browse/YARN-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139318#comment-15139318
]
Chris Nauroth commented on YARN-4681:
-------------------------------------
[~je.ik], thank you for posting a patch. Have you had a chance to try testing
this with the Spark use case that you described on the mailing list?
One important point that you brought up on the mailing list is that the Locked
field in smaps doesn't seem to be universal across all tested kernel versions.
bq. If I do this on an older kernel (2.6.x), the Locked field is missing.
The current patch completely switches the calculation from using Private_Clean
to Locked. For a final version of the patch, we'd want to make sure that the
change doesn't break anything for older kernels that don't show the Locked
field.
You also discussed possibly even more aggressive strategies, like trying to
anticipate that the kernel might free more memory by flushing file-backed dirty
pages. During YARN-1775, there was some discussion about supporting different
kinds of configurability for this calculation. That topic might warrant
further discussion here.
> ProcfsBasedProcessTree should not calculate private clean pages
> ---------------------------------------------------------------
>
> Key: YARN-4681
> URL: https://issues.apache.org/jira/browse/YARN-4681
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
> Affects Versions: 2.6.0, 2.7.0
> Reporter: Jan Lukavsky
> Attachments: YARN-4681.patch
>
>
> ProcfsBasedProcessTree in Node manager calculates memory used by a process
> tree by parsing {{/etc/<pid>/smaps}}, where it calculates {{min(Pss,
> Shared_Dirty) + Private_Dirty + Private_Clean}}. Because not {{mlocked}}
> private clean pages can be reclaimed by kernel, this should be changed to
> calculating only {{Locked}} pages instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
