[
https://issues.apache.org/jira/browse/HBASE-25302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17237104#comment-17237104
]
Xiaolin Ha commented on HBASE-25302:
------------------------------------
> Why didn't the region is split until too big? Can we tune that?
Thanks your question, [~Joseph295].
Some problems happened when we used hbck2 to repair the cluster, the
`splitormerge_switch 'SPLIT'` was false before we discovered the huge regions.
The cluster is used for a monitor service in our company, when a region is more
than 600G and has thousands hfiles, it will be reopened in a few of minutes,
and has great bad affect on availability. This cluster is with about 10000+
regions, and splitting all the regions will need a lot of read and write IO. As
a result, we want to find out a method using the least IO to split all the
regions to the normal size. Fortunately, we use stripe store engine, and using
HFileLinks instead of half References to refer a whole hfile can make the split
generate less Reference files, which is helpful to the following splits. At the
end, we can move the parent referred data files to the child regions, and do
not need to rewrite them. How to avoid the file movement affecting snapshots is
referred in the design doc. If there is some problems else, please feel free to
contact me. Thanks.
> Fast continuous split regions with stripe store engine
> ------------------------------------------------------
>
> Key: HBASE-25302
> URL: https://issues.apache.org/jira/browse/HBASE-25302
> Project: HBase
> Issue Type: Improvement
> Reporter: Xiaolin Ha
> Assignee: Xiaolin Ha
> Priority: Major
> Attachments: Fast continuous split regions with stripe store
> engine.pdf
>
>
> We have implemented a fast continuous split region method using HFileLink,
> depending on the stripe store file manager.
> It is very simple and efficiency, we have implement all the ideas described
> in the design doc and used on our production clusters. A region of about 600G
> can be splitted to 75G*8 regions in about five minutes, with less than 5G
> total rewrite size(all are L0) in the whole process, while normal continuous
> split needs 600G*3=1800G. If using movement for same table HFileLinks, the
> rewritten size is less than 50G(two stripe size), because the rebuild of
> HFileLinks to stripes may insert some files to L0.
> Details are in the doc,
> [https://docs.google.com/document/d/1hzBMdEFCckw18RE-kQQCe2ArW0MXhmLiiqyqpngItBM/edit?usp=sharing]
> If there is someone who has interest in this issue, please let me know,
> thanks.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
