[
https://issues.apache.org/jira/browse/YARN-5551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439246#comment-15439246
]
Jason Lowe commented on YARN-5551:
----------------------------------
bq. Yes, deleted files is a red-herring
OK, cool. That's been my main issue with the JIRA's proposed change.
bq. If I understand page-writeback.c correctly, the blocking operation would be
the page fault on a memory block which is missing in memory.
Good, so it looks like users can't abuse an mmap'd region any more than they
can abuse the buffer cache. Thanks for looking into this.
bq. If YARN counted only the "anonymous" pages as the "will be free'd a kill"
memory, it would give me a better way.
I'm torn on this proposal. My initial reaction to ignoring private dirty pages
is that a user can hide a lot of memory by creating their own, personal swap
file. For example, a user allocates 10G normally and touches it all. I think
we all agree that should count as 10G of usage. If the user changes the app to
back that memory with a private file on disk then it wouldn't count for
anything. That seems wrong since the user is charged differently based on
whether the memory is backed by public swap vs. private swap. Memory pressure
could cause either of them to spill to disk, so it doesn't make sense to me why
we would count them differently.
On the other hand, it would be a different situation if there was no swap
configured on the system. Memory pressure could push out the private dirty
pages but dirty anonymous pages are pinned until the process exits. Then I
think it makes sense to count them differently because they won't behave the
same under memory pressure.
> Ignore deleted file mapping from memory computation when smaps is enabled
> -------------------------------------------------------------------------
>
> Key: YARN-5551
> URL: https://issues.apache.org/jira/browse/YARN-5551
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Priority: Minor
> Attachments: YARN-5551.branch-2.001.patch
>
>
> Currently deleted file mappings are also included in the memory computation
> when SMAP is enabled. For e.g
> {noformat}
> 7f612004a000-7f612004c000 rw-s 00000000 00:10 4201507513
> /dev/shm/HadoopShortCircuitShm_DFSClient_NONMAPREDUCE_-521969216_162_734673185
> (deleted)
> Size: 8 kB
> Rss: 4 kB
> Pss: 2 kB
> Shared_Clean: 0 kB
> Shared_Dirty: 4 kB
> Private_Clean: 0 kB
> Private_Dirty: 0 kB
> Referenced: 4 kB
> Anonymous: 0 kB
> AnonHugePages: 0 kB
> Swap: 0 kB
> KernelPageSize: 4 kB
> MMUPageSize: 4 kB
> 7fbf28000000-7fbf68000000 rw-s 00000000 08:02 11927571
> /tmp/7298569189125604642/arena-1291157252088664681.cache (deleted)
> Size: 1048576 kB
> Rss: 17288 kB
> Pss: 17288 kB
> Shared_Clean: 0 kB
> Shared_Dirty: 0 kB
> Private_Clean: 232 kB
> Private_Dirty: 17056 kB
> Referenced: 17288 kB
> Anonymous: 0 kB
> AnonHugePages: 0 kB
> Swap: 0 kB
> KernelPageSize: 4 kB
> MMUPageSize: 4 kB
> {noformat}
> It would be good to exclude these from getSmapBasedRssMemorySize()
> computation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
