[
https://issues.apache.org/jira/browse/HBASE-11165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14019262#comment-14019262
]
Vladimir Rodionov commented on HBASE-11165:
-------------------------------------------
{quote}
Issues with HDFS when millions of regions. Need 4Gs of heap to list a directory
of millions of regions. Six hours to create a million regions in HDFS.
{quote}
We want to build a race car out of a shit rock, sorry - sheet rock (HDFS). Time
has come to attack the major limitation of HDFS: single-NN design and lack of
scalability. Having ability to keep billions files in a file system w/o any
performance penalty will allow HBase to consider compact-less design, for
example. There is no need for compaction if random IO is cheap (SSD). You won't
need large regions as well because you can afford millions of them and small
regions will improve MTTR significantly.
Region assignment/re-assignment must be done in a completely decentralized
manner with conflict resolution protocol in place to resolve possible rare
conflicts.
> Scaling so cluster can host 1M regions and beyond (50M regions?)
> ----------------------------------------------------------------
>
> Key: HBASE-11165
> URL: https://issues.apache.org/jira/browse/HBASE-11165
> Project: HBase
> Issue Type: Brainstorming
> Reporter: stack
> Attachments: HBASE-11165.zip, Region Scalability test.pdf
>
>
> This discussion issue comes out of "Co-locate Meta And Master HBASE-10569"
> and comments on the doc posted there.
> A user -- our Francis Liu -- needs to be able to scale a cluster to do 1M
> regions maybe even 50M later. This issue is about discussing how we will do
> that (or if not 50M on a cluster, how otherwise we can attain same end).
> More detail to follow.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
