[
https://issues.apache.org/jira/browse/HBASE-11165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136860#comment-14136860
]
Mikhail Antonov commented on HBASE-11165:
-----------------------------------------
[~toffer], [~virag] - guys, getting back for a moment to the size of meta
table..there's an interesting thing about it. [~octo47], [~sergey.soldatov] and
myself are doing some prototyping of more compact & fast in-memory
representation of meta, and have noticed interesting thing inspecting the data
structures under the microscope (like
https://code.google.com/p/memory-measurer).
In the discussion about, and in the doc put together by [~stack] it's mentioned
that in best case, for fully compacted meta with single-versioned cells 1 row
in meta takes up 7-10kb (please correct me if I'm wrong).
If I run a minicluster test, create some simple table with regions and then do
MTA.fullScan() to get list of Result, a single Result won't take more than 1Kb,
normally less than that. And the list of Results is not exactly super-compacted
structure, if initial experiments we were able to compact it further quite a
bit.
So I'm curious how exactly the size heap occupied by meta was calculated (was
it some sort of direct sizeOf, like using Unsafe or instrumentation etc), or
default impl provided by HeapSize for hregion, or size of HFiles, or something
else? Also, I'd appreciate a lot if you could share a sample representative row
from your meta, so we can see the typical size of elements in it?
> Scaling so cluster can host 1M regions and beyond (50M regions?)
> ----------------------------------------------------------------
>
> Key: HBASE-11165
> URL: https://issues.apache.org/jira/browse/HBASE-11165
> Project: HBase
> Issue Type: Brainstorming
> Reporter: stack
> Attachments: HBASE-11165.zip, Region Scalability test.pdf,
> zk_less_assignment_comparison_2.pdf
>
>
> This discussion issue comes out of "Co-locate Meta And Master HBASE-10569"
> and comments on the doc posted there.
> A user -- our Francis Liu -- needs to be able to scale a cluster to do 1M
> regions maybe even 50M later. This issue is about discussing how we will do
> that (or if not 50M on a cluster, how otherwise we can attain same end).
> More detail to follow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
