[
https://issues.apache.org/jira/browse/HDFS-6919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14157086#comment-14157086
]
Colin Patrick McCabe commented on HDFS-6919:
--------------------------------------------
[~cnauroth], you bring up some very good points. Cache pool enforcement is
centralized now on the NameNode, whereas write cache is a per-DataNode thing.
Perhaps pools aren't the best abstraction to use here.
Still, we need some way of allowing users to use both features at once. Right
now, it seems like if a datanode starts using more memory for read cache
(HDFS-4949 cache), the write cache will not take that extra consumption into
account. It will keep trying to put the same number of replicas into the
ramdisk. This could lead to out-of-memory situations.
The simplest way to solve this, and something that I think might work pretty
well in practice, is to have the write cache shrink by the size of the read
cache on the local DataNode. So if the read cache starts using R bytes, and
the write cache is configured to use W bytes, the write cache uses W - R bytes
instead. That will avoid out of memory situations. It also seems appropriate
for the write cache's role as a temporary storage area, that you use if it's
available.
> Enforce a single limit for RAM disk usage and replicas cached via locking
> -------------------------------------------------------------------------
>
> Key: HDFS-6919
> URL: https://issues.apache.org/jira/browse/HDFS-6919
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Arpit Agarwal
> Assignee: Colin Patrick McCabe
> Priority: Blocker
>
> The DataNode can have a single limit for memory usage which applies to both
> replicas cached via CCM and replicas on RAM disk.
> See comments
> [1|https://issues.apache.org/jira/browse/HDFS-6581?focusedCommentId=14106025&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14106025],
>
> [2|https://issues.apache.org/jira/browse/HDFS-6581?focusedCommentId=14106245&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14106245]
> and
> [3|https://issues.apache.org/jira/browse/HDFS-6581?focusedCommentId=14106575&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14106575]
> for discussion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
