[
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15192967#comment-15192967
]
Anastasia Braginsky commented on HBASE-14921:
---------------------------------------------
Hi Guys,
Long time no speak. Meanwhile, CellBlocksSegment based on simple Cells Array is
ready (as on second picture in the document). This means additional
dereferencing and spending some additional memory on object headers. I'm also
implementing the totally flat CellBlocks, so it would be possible to allocate
it off-heap and it is generally more efficient (third picture in the updated
document). In order to do that I need a toBytes representation of a reference
to Chunk. As far as I know in pure Java you can’t get the address of an object.
So it can be done only by defining an index per Chunk and having a
translation/mapping from some integer index to Chunk and backward (from Chunk
to its index).
Looking closer on HeapMemStoreLAB and MemStoreChunkPool I see that
HeapMemStoreLAB.Chunk can be allocated both directly from JVM Heap and also via
MemStoreChunkPool, because MemStoreChunkPool is sometimes not configured or
can’t be used. It makes much more sense that all Chunk allocations are
centralized and go only through MemStoreChunkPool. It helps in managing the
mapping described above and also in off-heaping. Can MemStoreChunkPool exists
always? MemStoreChunkPool can pre-allocate Chunks as a Pool or allocate single
Chunk on demand as HeapMemStoreLAB currently does.
Then we have clear roles. HeapMemStoreLAB - deals only with reference counting
and when Chunk can be deallocated. It gets/returns chunks from/to
MemStoreChunkPool. While MemStoreChunkPool - deals only with chunks allocation,
pre-allocation and re-allocation.
[~stack], [~anoop.hbase], and everybody, what do you think?
Thanks,
Anastasia
> Memory optimizations
> --------------------
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
> Issue Type: Sub-task
> Affects Versions: 2.0.0
> Reporter: Eshcar Hillel
> Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf,
> CellBlocksSegmentinthecontextofMemStore(1).pdf
>
>
> Memory optimizations including compressed format representation and offheap
> allocations
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
