[
https://issues.apache.org/jira/browse/YARN-5551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437655#comment-15437655
]
Jason Lowe commented on YARN-5551:
----------------------------------
The more I think about this, the more I feel ignoring deleted files is the
wrong thing to do. I think we all can agree that mappings to deleted files can
still consume memory, and if we skip those mappings then we fail to account for
that memory. For purposes of deciding how much memory will be freed when YARN
kills a process, skipping those sections will make YARN think it can free up
_less_ memory than it really would.
If we go back to the write() vs. mmap'd file which seems to be the origin
behind this idea, the write() case is going to eventually be throttled by the
OS because it will only allow so many dirty buffer cache pages in the system.
I don't believe that's the case for the mmap'd file. If we create a process
that mmap's a large file, deletes it, then spin-loops dirtying the pages, that
significant memory use needs to be associated with that process in the
accounting.
> Ignore deleted file mapping from memory computation when smaps is enabled
> -------------------------------------------------------------------------
>
> Key: YARN-5551
> URL: https://issues.apache.org/jira/browse/YARN-5551
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Priority: Minor
> Attachments: YARN-5551.branch-2.001.patch
>
>
> Currently deleted file mappings are also included in the memory computation
> when SMAP is enabled. For e.g
> {noformat}
> 7f612004a000-7f612004c000 rw-s 00000000 00:10 4201507513
> /dev/shm/HadoopShortCircuitShm_DFSClient_NONMAPREDUCE_-521969216_162_734673185
> (deleted)
> Size: 8 kB
> Rss: 4 kB
> Pss: 2 kB
> Shared_Clean: 0 kB
> Shared_Dirty: 4 kB
> Private_Clean: 0 kB
> Private_Dirty: 0 kB
> Referenced: 4 kB
> Anonymous: 0 kB
> AnonHugePages: 0 kB
> Swap: 0 kB
> KernelPageSize: 4 kB
> MMUPageSize: 4 kB
> 7fbf28000000-7fbf68000000 rw-s 00000000 08:02 11927571
> /tmp/7298569189125604642/arena-1291157252088664681.cache (deleted)
> Size: 1048576 kB
> Rss: 17288 kB
> Pss: 17288 kB
> Shared_Clean: 0 kB
> Shared_Dirty: 0 kB
> Private_Clean: 232 kB
> Private_Dirty: 17056 kB
> Referenced: 17288 kB
> Anonymous: 0 kB
> AnonHugePages: 0 kB
> Swap: 0 kB
> KernelPageSize: 4 kB
> MMUPageSize: 4 kB
> {noformat}
> It would be good to exclude these from getSmapBasedRssMemorySize()
> computation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
