[
https://issues.apache.org/jira/browse/HDFS-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13912186#comment-13912186
]
Colin Patrick McCabe commented on HDFS-5957:
--------------------------------------------
Thanks for this data point, Gopal. I agree that allocating your own direct
{{ByteBuffer}} objects will probably be slower than {{mmap}}. It certainly
will cause issues with the Java heap. I was suggesting it as a temporary
workaround until YARN-1747 is fixed, not as a permanent solution.
It seems like a better interim solution would be simply to tweak
{{dfs.client.mmap.cache.size}} (and perhaps some job-specific parameters that
you're using) to ensure that you don't exceed the limits that YARN is placing.
As I mentioned earlier, HDFS's behavior here is completely deterministic and
synchronous... we will immediately {{munmap}} old segments in {{releaseBuffer}}
(according to the LRU policy) when there is no more space for them. It is not
"an eventual {{munmap}} on a background thread." If you set the cache size too
big, that's a problem, but not one that HDFS can solve. Only the administrator
can solve it.
Why don't we close this JIRA and move the discussion to YARN-1747?
> Provide support for different mmap cache retention policies in
> ShortCircuitCache.
> ---------------------------------------------------------------------------------
>
> Key: HDFS-5957
> URL: https://issues.apache.org/jira/browse/HDFS-5957
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
> Affects Versions: 2.3.0
> Reporter: Chris Nauroth
>
> Currently, the {{ShortCircuitCache}} retains {{mmap}} regions for reuse by
> multiple reads of the same block or by multiple threads. The eventual
> {{munmap}} executes on a background thread after an expiration period. Some
> client usage patterns would prefer strict bounds on this cache and
> deterministic cleanup by calling {{munmap}}. This issue proposes additional
> support for different caching policies that better fit these usage patterns.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
