[
https://issues.apache.org/jira/browse/YARN-5551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437858#comment-15437858
]
Gopal V commented on YARN-5551:
-------------------------------
bq. The more I think about this, the more I feel ignoring deleted files is the
wrong thing to do
Yes, deleted files is a red-herring (happens to be how we secure the files away
from other users).
I think the original problem of YARN killing a process needs to be fixed (the
original SMAPS fix was for HDFS Zero Copy read via mmap).
{code}
total +=
Math.min(info.sharedDirty, info.pss) + info.privateDirty
+ info.privateClean;
{code}
If as [~nroberts] suggests, If YARN counted only the "anonymous" pages as the
"will be free'd a kill" memory, it would give me a better way.
bq. the write() case is going to eventually be throttled by the OS because it
will only allow so many dirty buffer cache pages in the system. I don't believe
that's the case for the mmap'd file.
Once you exceed the dirty_ratio, the only way you can avoid a page-fault is by
modifying an existing dirty page over & over again.
If I understand page-writeback.c correctly, the blocking operation would be the
page fault on a memory block which is missing in memory.
bq. that significant memory use needs to be associated with that process in the
accounting.
Accounting isn't the problem, killing processes is the problem.
> Ignore deleted file mapping from memory computation when smaps is enabled
> -------------------------------------------------------------------------
>
> Key: YARN-5551
> URL: https://issues.apache.org/jira/browse/YARN-5551
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Priority: Minor
> Attachments: YARN-5551.branch-2.001.patch
>
>
> Currently deleted file mappings are also included in the memory computation
> when SMAP is enabled. For e.g
> {noformat}
> 7f612004a000-7f612004c000 rw-s 00000000 00:10 4201507513
> /dev/shm/HadoopShortCircuitShm_DFSClient_NONMAPREDUCE_-521969216_162_734673185
> (deleted)
> Size: 8 kB
> Rss: 4 kB
> Pss: 2 kB
> Shared_Clean: 0 kB
> Shared_Dirty: 4 kB
> Private_Clean: 0 kB
> Private_Dirty: 0 kB
> Referenced: 4 kB
> Anonymous: 0 kB
> AnonHugePages: 0 kB
> Swap: 0 kB
> KernelPageSize: 4 kB
> MMUPageSize: 4 kB
> 7fbf28000000-7fbf68000000 rw-s 00000000 08:02 11927571
> /tmp/7298569189125604642/arena-1291157252088664681.cache (deleted)
> Size: 1048576 kB
> Rss: 17288 kB
> Pss: 17288 kB
> Shared_Clean: 0 kB
> Shared_Dirty: 0 kB
> Private_Clean: 232 kB
> Private_Dirty: 17056 kB
> Referenced: 17288 kB
> Anonymous: 0 kB
> AnonHugePages: 0 kB
> Swap: 0 kB
> KernelPageSize: 4 kB
> MMUPageSize: 4 kB
> {noformat}
> It would be good to exclude these from getSmapBasedRssMemorySize()
> computation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
