[
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15392213#comment-15392213
]
Edward Bortnikov commented on HBASE-14921:
------------------------------------------
Suggest we install some order in this discussion, there are really multiple
issues on the table.
TL;DR: Let's get this patch in shape and check it in without over-optimizing;
it's already quite big.
1. Bugs in the current PR. Thanks for reporting. Those must be fixed, period.
We are working on reproducing and fixing.
2. Decoupling In-Memory Flush (with Flattening) from Compaction - either
algorithmic or via configuration. IMHO, this is a matter of optimization,
either approach has its pros and contras. For example, if flattening and
compaction were always coupled, the too-many-open-files problem would not have
emerged. In general, we're in favor of having a smart system with as few
parameter knobs as possible, capable of figuring out the compaction benefits at
a low cost. But again, this is a matter of policy. We suggest to defer it
beyond the current commit.
3. Concurrent development. Currently, there are at least two JIRA's
(HBASE-16003 and HBASE-16229) that try to concurrently handle the same issues
as this JIRA, which creates a lot of friction in the code. The prior consensus
was that HBASE-14921 would be the umbrella for all memory optimizations,
including the ultimate flattening (CellChunkMap). Failing to stick to this
discipline slows us down a lot. [~anoop.hbase] and [~ram_krish], if you feel
you've reached a more advanced stage of flat memory implementation for the
ultimate off-heaping, and prefer to lead that charge - this is perfectly fine
with us. But let us merge the 14921 patch first (it's already heavily
invested), and start optimizing on top of it.
Cheers,
Ed
> Memory optimizations
> --------------------
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
> Issue Type: Sub-task
> Affects Versions: 2.0.0
> Reporter: Eshcar Hillel
> Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf,
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch,
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch,
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch,
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf,
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap
> allocations
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
