[
https://issues.apache.org/jira/browse/HBASE-25302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaolin Ha updated HBASE-25302:
-------------------------------
Description:
In our company MeiTuan, we have encountered a problem, that is a production
cluster with about more than 10000 regions set off the split switch by some
mistakes. And more than 10000 regions are larger than 600GB before we can do
something to split them. Because online real time R+W QPS of each RS is higher
than 80000 (8000+ per region), if we just switch on split, the IO pressure
brought by compactions will influence the availabilities, and all large regions
need repeatedly compactions until the size of child regions are normal.
Different with the normal circumstances, we used stripe store engine in this
cluster, after looking through docs of
https://issues.apache.org/jira/browse/HBASE-7667 , we have implemented a fast
split region method using the similar ideas of HFileLink and move store files
between regions when compacting. Actually, this idea has already been mentioned
in HBASE-7667 , it said that `region splits become marvelously simple (if we
could move files between regions, no references would be needed)`.
We have implemented the ideas and practiced them on that cluster. The fact
results show that, a region of about 600G can be splitted to 75G*8 regions in
about five minutes, with less than 5G total rewrite size(all are L0) in the
whole process, while normal continuous split needs 600G*3=1800G. By using
movement of store files, the rewritten size is less than 50G(two stripe size,
because the rebuild of HFileLinks to stripes may insert some files to L0.).I
have attached two pictures about a RS before and after splitting all the
regions using this method.
This idea can not only be used in stripe store engine, but also default store
engine, and the store movement methods are very benefit to region MERGE.
was:
In our company MeiTuan, we have encountered a problem, that is a production
cluster with about more than 10000 regions switched off the split switch by
some mistakes. And more than 10000 regions are larger than 600GB before we can
do something to split them. Because the R+W QPS is higher than 80000 and if
split all the large regions at a time means lots of IO and CPU resources are
needed under current splitting method, expecially that all the regions need to
split more than once.
Fortunately, we use stripe compaction in most of our production clusters,
according to the design docs in
https://issues.apache.org/jira/browse/HBASE-7667 , we have implemented a fast
split region method using the idea of HFileLink and a hfile movement method
between same table regions. Actually, this idea was mentioned in HBASE-7667 ,
it said that `region splits become marvelously simple (if we could move files
between regions, no references would be needed)`.
This issue is point at the fast split method.
Details are in the doc,
[https://docs.google.com/document/d/1hzBMdEFCckw18RE-kQQCe2ArW0MXhmLiiqyqpngItBM/edit?usp=sharing]
It is very simple and efficiency, we have implement all the ideas described in
the design doc and used on our production clusters. A region of about 600G can
be splitted to 75G*8 regions in about five minutes, with less than 5G total
rewrite size(all are L0) in the whole process, while normal continuous split
needs 600G*3=1800G. If using movement for same table HFileLinks, the rewritten
size is less than 50G(two stripe size), because the rebuild of HFileLinks to
stripes may insert some files to L0.
I will push two images about a RS before and after splitting all the regions
using this method.
We are willing to contribute the codes to community. This idea can not only be
used in stripe store engine, but also default store engine, and can be very
benefit to merge regions. If there is someone who has interest in this issue,
please let me know, thanks.
> Fast split regions and lightweight compaction with stripe store engine
> ----------------------------------------------------------------------
>
> Key: HBASE-25302
> URL: https://issues.apache.org/jira/browse/HBASE-25302
> Project: HBase
> Issue Type: Improvement
> Reporter: Xiaolin Ha
> Assignee: Xiaolin Ha
> Priority: Major
> Attachments: after-split.png, before-split.png
>
>
> In our company MeiTuan, we have encountered a problem, that is a production
> cluster with about more than 10000 regions set off the split switch by some
> mistakes. And more than 10000 regions are larger than 600GB before we can do
> something to split them. Because online real time R+W QPS of each RS is
> higher than 80000 (8000+ per region), if we just switch on split, the IO
> pressure brought by compactions will influence the availabilities, and all
> large regions need repeatedly compactions until the size of child regions are
> normal.
> Different with the normal circumstances, we used stripe store engine in this
> cluster, after looking through docs of
> https://issues.apache.org/jira/browse/HBASE-7667 , we have implemented a
> fast split region method using the similar ideas of HFileLink and move store
> files between regions when compacting. Actually, this idea has already been
> mentioned in HBASE-7667 , it said that `region splits become marvelously
> simple (if we could move files between regions, no references would be
> needed)`.
>
>
> We have implemented the ideas and practiced them on that cluster. The fact
> results show that, a region of about 600G can be splitted to 75G*8 regions
> in about five minutes, with less than 5G total rewrite size(all are L0) in
> the whole process, while normal continuous split needs 600G*3=1800G. By using
> movement of store files, the rewritten size is less than 50G(two stripe size,
> because the rebuild of HFileLinks to stripes may insert some files to L0.).I
> have attached two pictures about a RS before and after splitting all the
> regions using this method.
>
> This idea can not only be used in stripe store engine, but also default store
> engine, and the store movement methods are very benefit to region MERGE.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
