[
https://issues.apache.org/jira/browse/YARN-5551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437166#comment-15437166
]
Gopal V commented on YARN-5551:
-------------------------------
bq. The storage behind that mapping will not be freed even though the path has
been deleted because this process still has an active mapping against it.
That's exactly the point - these are actually not memory pages, these are pages
borrowed from the buffer-cache. Some of them are dirty and some of them are
clean, which implies that they are not actually memory consumed by the process
if there's any memory pressure.
The ideal mechanism for YARN to react would be to force a dirty flush for the
specific process to reduce its memory footprint instead of always killing the
process when the observed memory footprint is larger - killing a process is not
the only way to reclaim memory from a process.
Operating purely with kill signals is genuinely overkill.
This implementation is trying to be more forgiving of a process which has a
large number of clean pages in memory backed by a disk cache file, which are
available to the process via .read or .map, but the disk buffer pages used by
the OS are counted differently by YARN if it uses .map().
> Ignore deleted file mapping from memory computation when smaps is enabled
> -------------------------------------------------------------------------
>
> Key: YARN-5551
> URL: https://issues.apache.org/jira/browse/YARN-5551
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Priority: Minor
> Attachments: YARN-5551.branch-2.001.patch
>
>
> Currently deleted file mappings are also included in the memory computation
> when SMAP is enabled. For e.g
> {noformat}
> 7f612004a000-7f612004c000 rw-s 00000000 00:10 4201507513
> /dev/shm/HadoopShortCircuitShm_DFSClient_NONMAPREDUCE_-521969216_162_734673185
> (deleted)
> Size: 8 kB
> Rss: 4 kB
> Pss: 2 kB
> Shared_Clean: 0 kB
> Shared_Dirty: 4 kB
> Private_Clean: 0 kB
> Private_Dirty: 0 kB
> Referenced: 4 kB
> Anonymous: 0 kB
> AnonHugePages: 0 kB
> Swap: 0 kB
> KernelPageSize: 4 kB
> MMUPageSize: 4 kB
> 7f6123f99000-7f6163f99000 rw-p 00000000 08:41 211419477
> /grid/4/hadoop/yarn/local/usercache/root/appcache/application_1466700718395_1249/container_e19_1466700718395_1249_01_000003/7389389356021597290.cache
> (deleted)
> Size: 1048576 kB
> Rss: 637292 kB
> Pss: 637292 kB
> Shared_Clean: 0 kB
> Shared_Dirty: 0 kB
> Private_Clean: 0 kB
> Private_Dirty: 637292 kB
> Referenced: 637292 kB
> Anonymous: 637292 kB
> AnonHugePages: 0 kB
> Swap: 0 kB
> KernelPageSize: 4 kB
> {noformat}
> It would be good to exclude these from getSmapBasedRssMemorySize()
> computation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
