[
https://issues.apache.org/jira/browse/HBASE-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119208#comment-15119208
]
Anoop Sam John edited comment on HBASE-14918 at 1/27/16 3:04 PM:
-----------------------------------------------------------------
[~eshcar] In one of the subtask (infrastructure refactor I guess) , I have
added a comment that may be we should move the MSLAB copy stuff out of Memstore
impl. Now in the patch it is moved to the Abstract base impl of Memstore and
Allocator stuff passed through the Segment also. (I think I read it that way..
not remembering).. actually this MSLAB stuff should be moved to HStore
level. (This is not a Memstore impl detail).. Not your patch issue. It was
this way from begin and when Memstore is made interface impl way, I missed that
too.. wdyt? I can see how we can move it. (Am already half way in that
patch)
Let me have a go at the attached pdf
was (Author: anoop.hbase):
[~eshcar] In one of the subtask (infrastructure refactor I guess) , I have
added a comment that may be we should move the MSLAB copy stuff out of Memstore
impl. Now in the patch it is moved to the Abstract base impl of Memstore and
Allocator stuff passed through the Segment also. (I think I read it that way..
not remembering).. actually this MSLAB stuff should be moved to HStore
level. (This is not a Memstore impl detail).. Not your patch issue. It was
this way from begin and when Memstore is made interface impl way, I missed that
too.. wdyt? I can see how we can me it.
Let me have a go at the attached pdf
> In-Memory MemStore Flush and Compaction
> ---------------------------------------
>
> Key: HBASE-14918
> URL: https://issues.apache.org/jira/browse/HBASE-14918
> Project: HBase
> Issue Type: Umbrella
> Affects Versions: 2.0.0
> Reporter: Eshcar Hillel
> Assignee: Eshcar Hillel
> Fix For: 0.98.18
>
> Attachments: CellBlocksSegmentDesign.pdf
>
>
> A memstore serves as the in-memory component of a store unit, absorbing all
> updates to the store. From time to time these updates are flushed to a file
> on disk, where they are compacted (by eliminating redundancies) and
> compressed (i.e., written in a compressed format to reduce their storage
> size).
> We aim to speed up data access, and therefore suggest to apply in-memory
> memstore flush. That is to flush the active in-memory segment into an
> intermediate buffer where it can be accessed by the application. Data in the
> buffer is subject to compaction and can be stored in any format that allows
> it to take up smaller space in RAM. The less space the buffer consumes the
> longer it can reside in memory before data is flushed to disk, resulting in
> better performance.
> Specifically, the optimization is beneficial for workloads with
> medium-to-high key churn which incur many redundant cells, like persistent
> messaging.
> We suggest to structure the solution as 4 subtasks (respectively, patches).
> (1) Infrastructure - refactoring of the MemStore hierarchy, introducing
> segment (StoreSegment) as first-class citizen, and decoupling memstore
> scanner from the memstore implementation;
> (2) Adding StoreServices facility at the region level to allow memstores
> update region counters and access region level synchronization mechanism;
> (3) Implementation of a new memstore (CompactingMemstore) with non-optimized
> immutable segment representation, and
> (4) Memory optimization including compressed format representation and off
> heap allocations.
> This Jira continues the discussion in HBASE-13408.
> Design documents, evaluation results and previous patches can be found in
> HBASE-13408.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
