[
https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16538313#comment-16538313
]
Eshcar Hillel commented on HBASE-20542:
---------------------------------------
TestHStore overrides the internal implementation of CompactingMemStore to
manipulate the timing of the compaction and count the number of times it is
successfully invoked. Since the internals of CompactingMemStore were changed in
this issue the test as well as the handles inside CompactingMemStore needed to
be changed to allow the correct manipulation.
TestCompactingToCellFlatMapMemStore was also affected by the changes in
internal mechanism, specifically the fact that in memory flush is now happening
as part of a write operation, and only the compaction runs in the background.
The fix also helps here.
> Better heap utilization for IMC with MSLABs
> -------------------------------------------
>
> Key: HBASE-20542
> URL: https://issues.apache.org/jira/browse/HBASE-20542
> Project: HBase
> Issue Type: Sub-task
> Components: in-memory-compaction
> Reporter: Eshcar Hillel
> Assignee: Eshcar Hillel
> Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-20542-addendum.master.005.patch,
> HBASE-20542.branch-2.001.patch, HBASE-20542.branch-2.003.patch,
> HBASE-20542.branch-2.004.patch, HBASE-20542.branch-2.005.patch,
> HBASE-20542.master.003.patch, HBASE-20542.master.005-addendum.patch, run.sh,
> workloada, workloadc, workloadx, workloady
>
>
> Following HBASE-20188 we realized in-memory compaction combined with MSLABs
> may suffer from heap under-utilization due to internal fragmentation. This
> jira presents a solution to circumvent this problem. The main idea is to have
> each update operation check if it will cause overflow in the active segment
> *before* it is writing the new value (instead of checking the size after the
> write is completed), and if it is then the active segment is atomically
> swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to
> the compaction pipeline. Later on the IMC deamon will run its compaction
> operation (flatten index/merge indices/data compaction) in the background.
> Some subtle concurrency issues should be handled with care. We next elaborate
> on them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
