[
https://issues.apache.org/jira/browse/HBASE-25302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaolin Ha updated HBASE-25302:
-------------------------------
Attachment: after-split.png
> Fast split regions with stripe store engine
> -------------------------------------------
>
> Key: HBASE-25302
> URL: https://issues.apache.org/jira/browse/HBASE-25302
> Project: HBase
> Issue Type: Improvement
> Reporter: Xiaolin Ha
> Assignee: Xiaolin Ha
> Priority: Major
> Attachments: after-split.png, before-split.png
>
>
> In our company MeiTuan, we have encountered a problem, that is a production
> cluster with about more than 10000 regions switched off the split switch by
> some mistakes. And more than 10000 regions are larger than 600GB before we
> can do something to split them. Because the R+W QPS is higher than 80000 and
> if split all the large regions at a time means lots of IO and CPU resources
> are needed under current splitting method, expecially that all the regions
> need to split more than once.
> Fortunately, we use stripe compaction in most of our production clusters,
> according to the design docs in
> https://issues.apache.org/jira/browse/HBASE-7667 , we have implemented a
> fast split region method using the idea of HFileLink and a hfile movement
> method between same table regions. Actually, this idea was mentioned in
> HBASE-7667 , it said that `region splits become marvelously simple (if we
> could move files between regions, no references would be needed)`.
> This issue is point at the fast split method.
> Details are in the doc,
> [https://docs.google.com/document/d/1hzBMdEFCckw18RE-kQQCe2ArW0MXhmLiiqyqpngItBM/edit?usp=sharing]
> It is very simple and efficiency, we have implement all the ideas described
> in the design doc and used on our production clusters. A region of about 600G
> can be splitted to 75G*8 regions in about five minutes, with less than 5G
> total rewrite size(all are L0) in the whole process, while normal continuous
> split needs 600G*3=1800G. If using movement for same table HFileLinks, the
> rewritten size is less than 50G(two stripe size), because the rebuild of
> HFileLinks to stripes may insert some files to L0.
> I will push two images about a RS before and after splitting all the regions
> using this method.
>
> If there is someone who has interest in this issue, please let me know,
> thanks.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
