[
https://issues.apache.org/jira/browse/HBASE-11165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114894#comment-14114894
]
stack commented on HBASE-11165:
-------------------------------
bq. I thought that the primary requirement for splittable meta is not really
read-write throughput, but rather the thinking that on large enough cluster
with 1M+ regions, meta may not simply fit in master memory (or JVM would have
hard time keeping process consuming that much memory up)?
It'd be both [~mantonov] Scale reads because pieces of meta served by many
servers and master can write in // so scale writes too (though not all ops will
be //izable).
Yeah, adding features to meta comes up all the time (today it was adding region
flush/compaction history, previous its keeping hfiles per cf in meta, region
replica info, and so on)
bq. I would think that if we want to keep regions as a unit of
assignment/recovery, splittable meta is a the must (so far didn't see an
approach describing how to avoid it)
bq. HBASE-10070 is about region replicas ....
I was thinking that we could have meta region replicas to scale out the read
i/o using hbase-10070 (solving one of the objections to splitting meta
arguments)
bq. HBASE-11467 is about ZK-less client...
That is the issue where client takes a quorum of masters instead of a quorum of
zks, right? I was extrapolating that the endpoint of this issue is a quorum of
masters where we could read from any (or at least some reads could be stale...)
as another means of scaling out the read i/o when master and meta colocated.
bq. ...but it would work fine with current single active-many backup-masters
schema as well.
Good to know [~mantonov]
bq. I'm thinking that multi-masters and partitioned-masters (if we go in these
approaches) need to be discussed closely together and considering each other,
otherwise it'd be really hard to merge them together later on.
Agree. This issue seems to be arriving at single master to serve mllions of
regions. A quorum of masters or partitioning master responsibilities for sure
should be discussed together but I don't think they are soln to this issues
problem (maybe partitioned master but single server soln seems simpler?)
bq. I'd be curious to hear more opinions/assessments on how bad is that when
master is down, and what timeframe various people would consider as "generally
ok", "kind of long, really want it to be faster" and "unacceptably long"?
Yes. Will write something up. In fact I don't even think a client can connect
to the cluster currently if master is down which makes a bit of a farce of the
above notion and needs fixing.
Let me look at HBASE-7767. I got burned by it today. Its annoying.... to say
the least.
Yeah, that seems to be the conclusion that is beginning to prevail here.
> Scaling so cluster can host 1M regions and beyond (50M regions?)
> ----------------------------------------------------------------
>
> Key: HBASE-11165
> URL: https://issues.apache.org/jira/browse/HBASE-11165
> Project: HBase
> Issue Type: Brainstorming
> Reporter: stack
> Attachments: HBASE-11165.zip, Region Scalability test.pdf,
> zk_less_assignment_comparison_2.pdf
>
>
> This discussion issue comes out of "Co-locate Meta And Master HBASE-10569"
> and comments on the doc posted there.
> A user -- our Francis Liu -- needs to be able to scale a cluster to do 1M
> regions maybe even 50M later. This issue is about discussing how we will do
> that (or if not 50M on a cluster, how otherwise we can attain same end).
> More detail to follow.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
