[
https://issues.apache.org/jira/browse/HBASE-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14707438#comment-14707438
]
Matteo Bertozzi commented on HBASE-13408:
-----------------------------------------
some general comment on the design:
The doc says something like "we apply memstore compaction only to in-memory
families, because we may end up with compacting something that has no
updates/deletes".
but, in theory it should be easy to know if we have updates and deletes. since
we have a skiplist, when we insert we can check if the previous record is the
same key is an update/delete. so we end up with 2 counters of how many updates
and deletes we have and we can apply a logic like "if (update/delete - insert)
> 40% do a memstore compact otherwise flush on disk.
given the above, we don't really need to have the compaction running randomly,
we can start the compaction on flush if necessary.
one other thing I was expecting was that the compacted version of the memstore
was written as an in-memory hfile, so we can have leverage stuff like
compression and encoding. but from the code looks like the compacted version
(memstore segment?) is just another skiplist.
can you expand more on why you ended up with the current implementation? what
are the benefit and so on.
> HBase In-Memory Memstore Compaction
> -----------------------------------
>
> Key: HBASE-13408
> URL: https://issues.apache.org/jira/browse/HBASE-13408
> Project: HBase
> Issue Type: New Feature
> Reporter: Eshcar Hillel
> Attachments: HBASE-13408-trunk-v01.patch,
> HBASE-13408-trunk-v02.patch,
> HBaseIn-MemoryMemstoreCompactionDesignDocument-ver02.pdf,
> HBaseIn-MemoryMemstoreCompactionDesignDocument.pdf,
> InMemoryMemstoreCompactionEvaluationResults.pdf,
> InMemoryMemstoreCompactionScansEvaluationResults.pdf
>
>
> A store unit holds a column family in a region, where the memstore is its
> in-memory component. The memstore absorbs all updates to the store; from time
> to time these updates are flushed to a file on disk, where they are
> compacted. Unlike disk components, the memstore is not compacted until it is
> written to the filesystem and optionally to block-cache. This may result in
> underutilization of the memory due to duplicate entries per row, for example,
> when hot data is continuously updated.
> Generally, the faster the data is accumulated in memory, more flushes are
> triggered, the data sinks to disk more frequently, slowing down retrieval of
> data, even if very recent.
> In high-churn workloads, compacting the memstore can help maintain the data
> in memory, and thereby speed up data retrieval.
> We suggest a new compacted memstore with the following principles:
> 1. The data is kept in memory for as long as possible
> 2. Memstore data is either compacted or in process of being compacted
> 3. Allow a panic mode, which may interrupt an in-progress compaction and
> force a flush of part of the memstore.
> We suggest applying this optimization only to in-memory column families.
> A design document is attached.
> This feature was previously discussed in HBASE-5311.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
