[
https://issues.apache.org/jira/browse/HBASE-23066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jacob LeBlanc updated HBASE-23066:
----------------------------------
Attachment: HBASE-23066.patch
Status: Patch Available (was: Open)
Added patch for master. After I added to TestCacheOnWrite, I ran into errors
with "Too many open files" when running it. It would occur when chmod was run
on one of the region files or a child thread was created on around the 50th
test iteration. Running only the test I added worked fine, as did running all
of the tests after I adjusted my system's open file limit. I'm not sure what
threads/open files/processes are getting left around in this test but my
addition seems to be pushing it over the edge. Is this a problem? Is it
expected to have >1024 open file limit when running large and/or integration
tests?
> Allow cache on write during compactions when prefetching is enabled
> -------------------------------------------------------------------
>
> Key: HBASE-23066
> URL: https://issues.apache.org/jira/browse/HBASE-23066
> Project: HBase
> Issue Type: Improvement
> Components: Compaction, regionserver
> Affects Versions: 1.4.10
> Reporter: Jacob LeBlanc
> Assignee: Jacob LeBlanc
> Priority: Minor
> Fix For: 1.5.0, 2.3.0
>
> Attachments: HBASE-23066.patch, prefetchCompactedBlocksOnWrite.patch
>
>
> In cases where users care a lot about read performance for tables that are
> small enough to fit into a cache (or the cache is large enough),
> prefetchOnOpen can be enabled to make the entire table available in cache
> after the initial region opening is completed. Any new data can also be
> guaranteed to be in cache with the cacheBlocksOnWrite setting.
> However, the missing piece is when all blocks are evicted after a compaction.
> We found very poor performance after compactions for tables under heavy read
> load and a slower backing filesystem (S3). After a compaction the prefetching
> threads need to compete with threads servicing read requests and get
> constantly blocked as a result.
> This is a proposal to introduce a new cache configuration option that would
> cache blocks on write during compaction for any column family that has
> prefetch enabled. This would virtually guarantee all blocks are kept in cache
> after the initial prefetch on open is completed allowing for guaranteed
> steady read performance despite a slow backing file system.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
