[
https://issues.apache.org/jira/browse/HBASE-23066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988783#comment-16988783
]
Jacob LeBlanc commented on HBASE-23066:
---------------------------------------
{quote}There is global config. The global config value can be overridden at a
CF level via
ColumnFamilyDescriptorBuilder#setConfiguration(final String key, final String
value)
Using the same global config name as the Key
BTW - confirmed seeing the code that it is easy to trigger this per
table/family also. Even using shell or the java client.
{quote}
Ah, OK. I wasn't aware there was a free-form way to set configuration at CF
level, as I saw all of the explicit properties listed there. Thanks.
{quote}No. Once compaction is done, the old compacted away files get closed
with evictBlocks = true always. It wont honor this conf then. The conf is more
used while closing of region.
{quote}
OK, I see now in HStore.removeCompactedFiles that evictBlocks is set explicitly
to true every time. Thanks for clarification on that. So with caching compacted
blocks on write there will be a temporary rise in cache size until the
compacted files are evicted after compaction is done. Got it.
{quote}Can you just give some rough numbers on you cache size and the number of
blocks that you always see in your cache? Is there a sporadic raise in your
block count and if so by how much and hope your cache size is good enough to
have them.
{quote}
We have two production environments currently with our largest currently
hosting 25.4 TB of data, with about 14 TB we are trying to keep "hot" in the
cache for quick reading, with about 4 TB of that kept "IN_MEMORY" since we want
fast phoenix searching on the data and care more about keeping it in cache. We
currently are running 48 region servers with 1 TB on-disk bucket cache each,
but that's probably overprovisioned as we've scaled up due to some recent
issues. We can probably shoot for around 24 RS which would mean about 600 GB of
data we are trying to keep in bucket cache. With the number of regions we have,
we have plenty of spare space in bucket cache for handling many large
compactions happening at once with this setting. In reality our used space for
the cache will be a bit higher because other reading is still done beyond the
14 TB we prefetch, but it's OK if LRU blocks get evicted. I haven't done a deep
analysis to see if there are sporadic spikes in cache size during compactions.
1 TB bucket caches may seem large but with the way costs are in AWS, storage is
relatively cheap compared to compute resources so it's more cost effective for
us to host fewer region servers with hosting more data with large caches rather
than smaller caches where we would need more region servers to maintain
performance (our compute nodes are probably 5x more expensive than the 1 TB of
storage per month).
{quote}If you are fine with the latest PR - I can just merge them and work on
the other sub task to make this configuration based on a size so that all the
older files' blocks are not cacheed.
{quote}
I am fine with it. Thanks for the attention on this!
> Allow cache on write during compactions when prefetching is enabled
> -------------------------------------------------------------------
>
> Key: HBASE-23066
> URL: https://issues.apache.org/jira/browse/HBASE-23066
> Project: HBase
> Issue Type: Improvement
> Components: Compaction, regionserver
> Affects Versions: 1.4.10
> Reporter: Jacob LeBlanc
> Assignee: Jacob LeBlanc
> Priority: Minor
> Fix For: 2.3.0, 1.6.0
>
> Attachments: HBASE-23066.patch, performance_results.png,
> prefetchCompactedBlocksOnWrite.patch
>
>
> In cases where users care a lot about read performance for tables that are
> small enough to fit into a cache (or the cache is large enough),
> prefetchOnOpen can be enabled to make the entire table available in cache
> after the initial region opening is completed. Any new data can also be
> guaranteed to be in cache with the cacheBlocksOnWrite setting.
> However, the missing piece is when all blocks are evicted after a compaction.
> We found very poor performance after compactions for tables under heavy read
> load and a slower backing filesystem (S3). After a compaction the prefetching
> threads need to compete with threads servicing read requests and get
> constantly blocked as a result.
> This is a proposal to introduce a new cache configuration option that would
> cache blocks on write during compaction for any column family that has
> prefetch enabled. This would virtually guarantee all blocks are kept in cache
> after the initial prefetch on open is completed allowing for guaranteed
> steady read performance despite a slow backing file system.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
