we had this issue. We adjusted tables with split policy to allow for large region in terms of file size. Then altered table with optimizer flag then enabled optimizer and waited. Eventually it will merge small regions into larger ones . Fully automated.
It takes time. On Thu, Aug 30, 2018 at 3:50 PM Austin Heyne <[email protected]> wrote: > I'm using HBase 1.4.4 (AWS/EMR) and I'm looking for an automated > solution because I believe there are going to be a few hundred if not > thousand merges. It's also challenging to find candidate pairs. > > -Austin > > > On 08/30/2018 03:45 PM, Jean-Marc Spaggiari wrote: > > Hi Austin, > > > > Which version are you using? Why not just using the shell merge command? > > > > JMS > > > > Le jeu. 30 août 2018 à 15:41, Austin Heyne <[email protected]> a écrit : > > > >> We're currently sitting at a very high number of regions due to an > >> initially poor value for hbase.regionserver.regionSplitLimit and would > >> like to reign in our region count. Additionally, we have a > >> spatio-temporal key structure and our region pre-splitting was done > >> evenly, without regard to the spatial distribution of our data and thus > >> have a lot of small and empty regions we'd like to clean up. I've found > >> the HMerge class [1], and it seems it would do something reasonable for > >> our used case. However, it's marked as Private and doesn't seem to be > >> used anywhere so I thought I'd ask if anyone knows the status of this > >> class and how safe it is. > >> > >> Thanks, > >> Austin > >> > >> [1] > >> > >> > https://github.com/apache/hbase/blob/branch-1.4/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HMerge.java > >> > >> > > -- > Austin L. Heyne > > -- -- CONFIDENTIALITY NOTICE: The information contained in this email is privileged and confidential and intended only for the use of the individual or entity to whom it is addressed. If you receive this message in error, please notify the sender immediately at 613-729-1100 and destroy the original message and all copies. Thank you.
