[
https://issues.apache.org/jira/browse/HBASE-4365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13214226#comment-13214226
]
Jean-Daniel Cryans commented on HBASE-4365:
-------------------------------------------
Alright I did a macrotest with 100GB.
Configurations:
Good old 15 machines test cluster (1 master), 2xquad, 14GB given to HBase, 4x
SATA.
The table is configured to flush at 256MB, split at 2GB.
40 clients that use a 12MB buffer, collocated on the RS.
Higher threshold for compactions.
Without patch:
1558s
With patch:
1457s
1.07x improvement.
Then what I saw is that once we've split a few times and that the load got
balanced, the performance is exactly the same. That's expected. Also it seems
that my split-after-flush patch also goes into full effect.
I'm running another experiment right now uploading 1TB with flush set at 512MB
and split at 20GB. I assume an even bigger difference. The reason to use 20GB
is that with bigger data sets you need bigger regions, and starting such a load
from scratch is currently horrible but this is what this jira is about.
> Add a decent heuristic for region size
> --------------------------------------
>
> Key: HBASE-4365
> URL: https://issues.apache.org/jira/browse/HBASE-4365
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.1, 0.94.0
> Reporter: Todd Lipcon
> Priority: Critical
> Labels: usability
> Attachments: 4365-v2.txt, 4365.txt
>
>
> A few of us were brainstorming this morning about what the default region
> size should be. There were a few general points made:
> - in some ways it's better to be too-large than too-small, since you can
> always split a table further, but you can't merge regions currently
> - with HFile v2 and multithreaded compactions there are fewer reasons to
> avoid very-large regions (10GB+)
> - for small tables you may want a small region size just so you can
> distribute load better across a cluster
> - for big tables, multi-GB is probably best
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
