[
https://issues.apache.org/jira/browse/HADOOP-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152342#comment-13152342
]
Cristina L. Abad commented on HADOOP-7832:
------------------------------------------
At this point I just wanted to share it. It can be very useful i evaluating the
fadvise effect on the page cache. For example, I ran some simple tests and
verified the benefit of using fadvise: for the intermediate files of a simple
terasort test (1 slave node, 10G of input data), the fadvise stuff leads to an
increase of from 44% to 78% of the average of pages (of intermediate files)
already in the page cache. This is an effect of Map input data being flushed
out of the page cache periodically. In this case, I am talking about the
intermediate files that we ideally want to keep cached as long as possible
(before being sent to the reducers) so that the shuffle and sort phase finishes
faster. You already showed how this improves job completion time, and this
patch shows how/why this happens.
> Using mincore to understand the effect of fadvise in the Linux page cache
> -------------------------------------------------------------------------
>
> Key: HADOOP-7832
> URL: https://issues.apache.org/jira/browse/HADOOP-7832
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: io, native, performance
> Reporter: Cristina L. Abad
> Priority: Minor
> Attachments: hadoop-7714-20s-205-mincore.txt
>
>
> This Jira adds a way to log the percentage of pages of a file that were on
> the Linux page cache (when the file is being opened) with the purpose of
> measuring the effect of issuing posix_fadvise (POSIX_FADV_DONTNEED) calls.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
