[
https://issues.apache.org/jira/browse/HBASE-20390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16467243#comment-16467243
]
Eshcar Hillel commented on HBASE-20390:
---------------------------------------
Following HBASE-20188 we realized in-memory compaction combined with MSLABs may
suffer from heap under-utilization due to internal fragmentation.
For example, setting the active segment threshold to A=0.02 means it stores
0.02*128MB=2.56MB. Each such 2.5MB segment utilizes 2 chunks (spanning *4MB*)
which are carried in the compaction pipeline until the data is flushed to disk.
Each 2.5MB data taking 4MB space means IMC heap utilization is roughly at 65%.
Not ideal.
We therefore experimented with A=0.014, namely active segment of size 1.8MB,
which fits into a single chunk (leaving some space for overflow etc). Running
workloadx+workloada+workloadc show improvement in performance in all these
workloads wrt the default parameters of IMC (results are attached).
We therefore suggest to set the IMC defaults to A=0.014, S=2.
I will make a new patch.
This number is obliviously working well with the current default chunk size.
In parallel we are working on a new solution in HBASE-20542 to circumvent this
problem regardless of the size of chunks, workload distribution or size of data
written.
> IMC Default Parameters for 2.0.0
> --------------------------------
>
> Key: HBASE-20390
> URL: https://issues.apache.org/jira/browse/HBASE-20390
> Project: HBase
> Issue Type: Sub-task
> Reporter: Eshcar Hillel
> Assignee: Eshcar Hillel
> Priority: Major
> Attachments: HBASE-20390-branch-2.0-01.patch,
> HBASE-20390-branch-2.0-01.patch, HBASE-20390.branch-2.0.002.patch, HBase 2.0
> performance evaluation - throughput SSD_HDD.pdf, hits.ihc.png
>
>
> Setting new default parameters for in-memory compaction based on performance
> tests done in HBASE-20188
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
