[
https://issues.apache.org/jira/browse/HBASE-23066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16941073#comment-16941073
]
Jacob LeBlanc commented on HBASE-23066:
---------------------------------------
I've run some performance tests to demonstrate the effectiveness of the patch.
I have not patched our production cluster yet as I'm waiting on confirmation
from AWS service team that I won't be overwriting AWS-specific changes in the
HStore class, but I've done some sampling on a test cluster.
Basic setup is an EMR cluster running 1.4.9 backed by S3 as well as ganglia
installed to capture the metrics. I have a stress tester executing about 1000
scans per second on a 1.5 GB region. Prefetching is enabled, and I have one
region server that is unpatched or has the new configuration setting disabled,
and one region server that is patched and has the new configuration option
enabled. I then execute the following test:
1. Move the region to the desired region server (either patched or unpatched).
2. Wait for prefetching to complete and for mean scan times to normalize.
3. Execute a major compaction on the target region.
4. Check region server UI / logs to see when the compaction completes.
5. Collect data from ganglia.
One issue I identified with my test is the scans aren't as random as they
should be so I believe data after compaction is getting cached on read more
quickly than it otherwise should be on the unpatched server if my scans were
truly random. I can improve the test, but results still validate the patch.
Baseline mean scan time was about 20 - 60 milliseconds. After compaction the
results were:
Trial 1 (unpatched): mean scan time peaked at over 27000 milliseconds, and
stayed above 5000 milliseconds for 3 minutes
Trial 2 (unpatched): mean scan time peaked at over 27000 milliseconds, and
stayed above 5000 milliseconds for 3.5 minutes
Trial 3 (patched): mean scan time peaked to 282 milliseconds for one time sample
Trial 4 (patched): mean scan time peaked at just over 1300 milliseconds and
remained abover 1000 milliseconds for 30 seconds
Trial 5 (patched): no noticable spike in mean scan time
I've attached a picture of a graph of the results.
> Allow cache on write during compactions when prefetching is enabled
> -------------------------------------------------------------------
>
> Key: HBASE-23066
> URL: https://issues.apache.org/jira/browse/HBASE-23066
> Project: HBase
> Issue Type: Improvement
> Components: Compaction, regionserver
> Affects Versions: 1.4.10
> Reporter: Jacob LeBlanc
> Assignee: Jacob LeBlanc
> Priority: Minor
> Fix For: 1.5.0, 2.3.0
>
> Attachments: HBASE-23066.patch, performance_results.png,
> prefetchCompactedBlocksOnWrite.patch
>
>
> In cases where users care a lot about read performance for tables that are
> small enough to fit into a cache (or the cache is large enough),
> prefetchOnOpen can be enabled to make the entire table available in cache
> after the initial region opening is completed. Any new data can also be
> guaranteed to be in cache with the cacheBlocksOnWrite setting.
> However, the missing piece is when all blocks are evicted after a compaction.
> We found very poor performance after compactions for tables under heavy read
> load and a slower backing filesystem (S3). After a compaction the prefetching
> threads need to compete with threads servicing read requests and get
> constantly blocked as a result.
> This is a proposal to introduce a new cache configuration option that would
> cache blocks on write during compaction for any column family that has
> prefetch enabled. This would virtually guarantee all blocks are kept in cache
> after the initial prefetch on open is completed allowing for guaranteed
> steady read performance despite a slow backing file system.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
