[
https://issues.apache.org/jira/browse/HADOOP-7714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13122003#comment-13122003
]
Nathan Roberts commented on HADOOP-7714:
----------------------------------------
Yes nice improvement. Seems like it's probably the intermediate files staying
resident but yes it would be good to know. Maybe we could add some mincore()
metrics somewhere that would help shed some light. Each time we open a file, we
use mincore to tell us what percentage of that file is resident.
As Cristina mentioned, we'll continue looking into why fadvise isn't clearing
up everything. I think it's racing with readahead and the search for dirty
pages to writeback. One thing I asked Cristina to try was to issue the fadvise
exactly once, just before close to see which pages remain in core.
> Add support in native libs for OS buffer cache management
> ---------------------------------------------------------
>
> Key: HADOOP-7714
> URL: https://issues.apache.org/jira/browse/HADOOP-7714
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 0.24.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hadoop-7714-20s-prelim.txt
>
>
> Especially in shared HBase/MR situations, management of the OS buffer cache
> is important. Currently, running a big MR job will evict all of HBase's hot
> data from cache, causing HBase performance to really suffer. However, caching
> of the MR input/output is rarely useful, since the datasets tend to be larger
> than cache and not re-read often enough that the cache is used. Having access
> to the native calls {{posix_fadvise}} and {{sync_data_range}} on platforms
> where they are supported would allow us to do a better job of managing this
> cache.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
