[
https://issues.apache.org/jira/browse/HBASE-16417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16191144#comment-16191144
]
Eshcar Hillel commented on HBASE-16417:
---------------------------------------
After a long period of time we experimented with different parameters and
in-memory compaction policies we would like to share with you our main results.
First set of experiments aims at tuning the Basic (default as of 2.0) in memory
compaction policy. We found out that under zipf distribution its optimal write
performance is with active segment size being 2% of the memory components and
with pipeline size bounded to 5 on SSD and 4 on HDD. The experiments are
summarized in the attached pdf file.
Second set of experiments compares a new adaptive compaction policy with the
existing basic and none policies. Bottom line is that there is a tradeoff
between write performance and write volume. We also experimented with mixed
read-write workload which show that with in-memory compaction we are able to
reduce the high percentiles of read latency on HDD machines. On SSD machines we
see a minor degradation in read latency. The experiments and main principle of
the adaptive policy are also summarized in an attached pdf file.
I am attaching a patch for trunk which includes the new adaptive policy as well
as the new optimal parameters for basic.
I would be happy to give more details on the experiment settings and results
and answer any questions you may have.
And also waiting to get your comments on reviewboard :)
Thank you!
> In-Memory MemStore Policy for Flattening and Compactions
> --------------------------------------------------------
>
> Key: HBASE-16417
> URL: https://issues.apache.org/jira/browse/HBASE-16417
> Project: HBase
> Issue Type: Sub-task
> Reporter: Anastasia Braginsky
> Assignee: Eshcar Hillel
> Fix For: 3.0.0
>
> Attachments: HBASE-16417 - Adaptive Compaction Policy - 20171001.pdf,
> HBASE-16417-benchmarkresults-20161101.pdf,
> HBASE-16417-benchmarkresults-20161110.pdf,
> HBASE-16417-benchmarkresults-20161123.pdf,
> HBASE-16417-benchmarkresults-20161205.pdf,
> HBASE-16417-benchmarkresults-20170309.pdf,
> HBASE-16417-benchmarkresults-20170317.pdf, HBASE-16417 - parameter tuning -
> 20171001.pdf, HBASE-16417-V01.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
