[
https://issues.apache.org/jira/browse/HBASE-25302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaolin Ha updated HBASE-25302:
-------------------------------
Description:
In our company MeiTuan, we have encountered a problem, that is a production
cluster with about more than 10000 regions set off the split switch by some
mistakes. And more than 10000 regions are larger than 600GB before we can do
something to split them. Because online real time R+W QPS of each RS is higher
than 80000 (8000+ per region), if we just switch on split, the IO pressure
brought by compactions will influence the availabilities, and all large regions
need repeatedly compactions until the size of child regions are normal.
Different with the normal circumstances, we used stripe store engine in this
cluster, after looking through docs of
https://issues.apache.org/jira/browse/HBASE-7667 , we have implemented a fast
split region method using the similar ideas of HFileLink and move store files
between regions when compacting. Actually, this idea has already been mentioned
in HBASE-7667 , it said that `region splits become marvelously simple (if we
could move files between regions, no references would be needed)`.
We have implemented the ideas and practiced them on that cluster. The fact
results show that, a region of about 600G can be splitted to 75G*8 regions in
about five minutes, with less than 5G total rewrite size(all are L0) in the
whole process, while normal continuous split needs 600G*3=1800G. By using
movement of store files, the rewritten size is less than 50G(two stripe size,
because the rebuild of HFileLinks to stripes may insert some files to L0.).I
have attached two pictures about a RS before and after splitting all the
regions using this method.
This idea can not only be used in stripe store engine, but also default store
engine, and the store movement methods are very benefit to region MERGE.
was:
In our company MeiTuan, we have encountered a problem, that is a production
cluster with about more than 10000 regions set off the split switch by some
mistakes. And more than 10000 regions are larger than 600GB before we can do
something to split them. Because online real time R+W QPS of each RS is higher
than 80000 (8000+ per region), if we just switch on split, the IO pressure
brought by compactions will influence the availabilities, and all large regions
need repeatedly compactions until the size of child regions are normal.
Different with the normal circumstances, we used stripe store engine in this
cluster, after looking through docs of
https://issues.apache.org/jira/browse/HBASE-7667 , we have implemented a fast
split region method using the similar ideas of HFileLink and move store files
between regions when compacting. Actually, this idea has already been mentioned
in HBASE-7667 , it said that `region splits become marvelously simple (if we
could move files between regions, no references would be needed)`.
We have implemented the ideas and practiced them on that cluster. The fact
results show that, a region of about 600G can be splitted to 75G*8 regions in
about five minutes, with less than 5G total rewrite size(all are L0) in the
whole process, while normal continuous split needs 600G*3=1800G. By using
movement of store files, the rewritten size is less than 50G(two stripe size,
because the rebuild of HFileLinks to stripes may insert some files to L0.).I
have attached two pictures about a RS before and after splitting all the
regions using this method.
This idea can not only be used in stripe store engine, but also default store
engine, and the store movement methods are very benefit to region MERGE.
> Fast split regions and lightweight compaction with stripe store engine
> ----------------------------------------------------------------------
>
> Key: HBASE-25302
> URL: https://issues.apache.org/jira/browse/HBASE-25302
> Project: HBase
> Issue Type: Improvement
> Reporter: Xiaolin Ha
> Assignee: Xiaolin Ha
> Priority: Major
> Attachments: after-split.png, before-split.png
>
>
> In our company MeiTuan, we have encountered a problem, that is a production
> cluster with about more than 10000 regions set off the split switch by some
> mistakes. And more than 10000 regions are larger than 600GB before we can do
> something to split them. Because online real time R+W QPS of each RS is
> higher than 80000 (8000+ per region), if we just switch on split, the IO
> pressure brought by compactions will influence the availabilities, and all
> large regions need repeatedly compactions until the size of child regions are
> normal.
> Different with the normal circumstances, we used stripe store engine in this
> cluster, after looking through docs of
> https://issues.apache.org/jira/browse/HBASE-7667 , we have implemented a
> fast split region method using the similar ideas of HFileLink and move store
> files between regions when compacting. Actually, this idea has already been
> mentioned in HBASE-7667 , it said that `region splits become marvelously
> simple (if we could move files between regions, no references would be
> needed)`.
> We have implemented the ideas and practiced them on that cluster. The fact
> results show that, a region of about 600G can be splitted to 75G*8 regions
> in about five minutes, with less than 5G total rewrite size(all are L0) in
> the whole process, while normal continuous split needs 600G*3=1800G. By using
> movement of store files, the rewritten size is less than 50G(two stripe size,
> because the rebuild of HFileLinks to stripes may insert some files to L0.).I
> have attached two pictures about a RS before and after splitting all the
> regions using this method.
> This idea can not only be used in stripe store engine, but also default store
> engine, and the store movement methods are very benefit to region MERGE.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
