[
https://issues.apache.org/jira/browse/HBASE-4365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101668#comment-13101668
]
Todd Lipcon commented on HBASE-4365:
------------------------------------
One idea that I'd like to propose is the following. We would introduce a
special value for max region size (eg 0 or -1) which configures HBase to use a
heuristic instead of a constant. The heuristic would be:
- on region-open (and periodically, perhaps on flush or compaction) the RS
scans META to find how many regions are in the table. Let's call this _R_.
- the region server also periodically scans ZK or asks the master to determine
how many RS are in the cluster. Let's call this _S_.
- If _R_ < _S_ (ie there are fewer regions than servers), we use a max region
size of perhaps 256MB. This is so that early in a table's lifetime, even if not
pre-split, the table will split rapidly and be able to spread load across the
cluster.
- If _R_ < _5S_, we use a max region size of perhaps 1GB.
- If _R_ > _5S_, we use a max region size of something rather large, like 10GB.
As an example, if we started putting a 1T table on a cluster with 20 nodes,
we'd quickly expand to 20 regions of 256M. The size would switch to 1G, and
we'd get another ~100 regions. The size would switch to 10G and we'd get
another 100 regions. So at the end, we'd have around 200-250 regions, or about
10 per server.
The specific numbers above aren't well thought out, but does this seem to be
more "foolproof" than any predetermined default.
> Add a decent heuristic for region size
> --------------------------------------
>
> Key: HBASE-4365
> URL: https://issues.apache.org/jira/browse/HBASE-4365
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.94.0
> Reporter: Todd Lipcon
>
> A few of us were brainstorming this morning about what the default region
> size should be. There were a few general points made:
> - in some ways it's better to be too-large than too-small, since you can
> always split a table further, but you can't merge regions currently
> - with HFile v2 and multithreaded compactions there are fewer reasons to
> avoid very-large regions (10GB+)
> - for small tables you may want a small region size just so you can
> distribute load better across a cluster
> - for big tables, multi-GB is probably best
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
