[
https://issues.apache.org/jira/browse/HBASE-25302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17242089#comment-17242089
]
Xiaolin Ha commented on HBASE-25302:
------------------------------------
Hi, [~zhangduo], [~stack], [~zghao], [~anoop.hbase] ,
Could you help to review this issue? We have practiced the idea on our
production clusters. Detailed design doc is in the issue description, and
images of before and after splitting are attached to show the result. If you
think this idea is OK, I can contribute all the implement codes to community,
and our company is very happy to do it.
Thank you.
> Fast split regions with stripe store engine
> -------------------------------------------
>
> Key: HBASE-25302
> URL: https://issues.apache.org/jira/browse/HBASE-25302
> Project: HBase
> Issue Type: Improvement
> Reporter: Xiaolin Ha
> Assignee: Xiaolin Ha
> Priority: Major
> Attachments: after-split.png, before-split.png
>
>
> In our company MeiTuan, we have encountered a problem, that is a production
> cluster with about more than 10000 regions switched off the split switch by
> some mistakes. And more than 10000 regions are larger than 600GB before we
> can do something to split them. Because the R+W QPS is higher than 80000 and
> if split all the large regions at a time means lots of IO and CPU resources
> are needed under current splitting method, expecially that all the regions
> need to split more than once.
> Fortunately, we use stripe compaction in most of our production clusters,
> according to the design docs in
> https://issues.apache.org/jira/browse/HBASE-7667 , we have implemented a
> fast split region method using the idea of HFileLink and a hfile movement
> method between same table regions. Actually, this idea was mentioned in
> HBASE-7667 , it said that `region splits become marvelously simple (if we
> could move files between regions, no references would be needed)`.
> This issue is point at the fast split method.
> Details are in the doc,
> [https://docs.google.com/document/d/1hzBMdEFCckw18RE-kQQCe2ArW0MXhmLiiqyqpngItBM/edit?usp=sharing]
> It is very simple and efficiency, we have implement all the ideas described
> in the design doc and used on our production clusters. A region of about 600G
> can be splitted to 75G*8 regions in about five minutes, with less than 5G
> total rewrite size(all are L0) in the whole process, while normal continuous
> split needs 600G*3=1800G. If using movement for same table HFileLinks, the
> rewritten size is less than 50G(two stripe size), because the rebuild of
> HFileLinks to stripes may insert some files to L0.
> I will push two images about a RS before and after splitting all the regions
> using this method.
>
> We are willing to contribute the codes to community. This idea can be not
> only be used in stripe store engine, but also default store engine, and can
> be very benefit to merge regions. If there is someone who has interest in
> this issue, please let me know, thanks.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
