[
https://issues.apache.org/jira/browse/HBASE-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Otis Gospodnetic updated HBASE-6351:
------------------------------------
Summary: Stop compactions from polluting OS FS cache (was: IO impact
reduction for compaction)
> Stop compactions from polluting OS FS cache
> --------------------------------------------
>
> Key: HBASE-6351
> URL: https://issues.apache.org/jira/browse/HBASE-6351
> Project: HBase
> Issue Type: Bug
> Reporter: Zhihong Ted Yu
>
> The following came from Otis via http://search-hadoop.com/m/MGVqgZJ4Mj2 :
> Lucene 4.0.0-Alpha was recently released. Mike McCandless, sne of the Lucene
> developers, wrote a really nice post about new things in this version of
> Lucene. The part that I think is interesting for HBase, and that HBase devs
> may want to look at (and borrow to use with compactions) is this:
> Reducing merge IO impact
> Merging (consolidating many small segments into a single big one) is a very
> IO and CPU intensive operation which can easily interfere with ongoing
> searches. In 4.0.0 we now have two ways to reduct this impact:
> * Rate-limit the IO caused by ongoing merging, by calling
> FSDirectory.setMaxMergeWriteMBPerSec.
> * Use the new NativeUnixDirectory which bypasses the OS's IO cache
> for all merge IO, by using direct IO. This ensures that a merge won't evict
> hot pages used by searches. (Note that there is also a native
> WindowsDirectory, but it does not yet use direct IO during merging... patches
> welcome!).
> Remember to also set swappiness to 0 on Linux if you want to maximize search
> responsiveness.
> More generally, the APIs that open an input or output file
> (Directory.openInput and Directory.createOutput) now take an IOContext
> describing what's being done (e.g., flush vs merge), so you can create a
> custom Directory that changes its behavior depending on the context.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
