HBase will not remove empty regions. It assumes you know what you're doing.
In 1.2 there's a new "Region Normalizer" [0] feature that runs in the Master, which acts a bit like the Balancer, but for region sizes. I think it's still considered experimental, so YMMV. We're still on 1.1, so I haven't tried it in prod yet. I also have this problem in our cluster, because of a combination of TTL and Phoenix's Rowkey Timestamp feature. This results in region boundaries including the timestamp, and data that goes away after a period. Eventually all regions will become empty. I have a script that runs periodically to calculate region sizes and drop empty regions. I volunteered to contribute it [1], but haven't pulled it out of our puppet infra in the form of a patch. Let me revive that thread internally. -n [0]: https://issues.apache.org/jira/browse/HBASE-13103 [1]: https://issues.apache.org/jira/browse/HBASE-15712 On Fri, Nov 18, 2016 at 4:59 PM, Xi Yang <[email protected]> wrote: > Yes, we're still on old release of 0.98.x > > Many of empty region didn't be deleted. Take one for example: > > > > "Namespace_default_table_java_app_logs_region_ > 29d4fa5482d17daf813e12732155e0ee_metric_storeCount" > > : 1, > > > > "Namespace_default_table_java_app_logs_region_ > 29d4fa5482d17daf813e12732155e0ee_metric_storeFileCount" > > : 1, > > > > "Namespace_default_table_java_app_logs_region_ > 29d4fa5482d17daf813e12732155e0ee_metric_memStoreSize" > > : 424, > > > > "Namespace_default_table_java_app_logs_region_ > 29d4fa5482d17daf813e12732155e0ee_metric_storeFileSize" > > : 549, > > > > "Namespace_default_table_java_app_logs_region_ > 29d4fa5482d17daf813e12732155e0ee_metric_compactionsCompletedCount" > > : 4, > > > > "Namespace_default_table_java_app_logs_region_ > 29d4fa5482d17daf813e12732155e0ee_metric_numBytesCompactedCount" > > : 8616506268, > > > > "Namespace_default_table_java_app_logs_region_ > 29d4fa5482d17daf813e12732155e0ee_metric_numFilesCompactedCount" > > : 4, > > > The log about last time major compaction of this region is: > > 2016-11-03 22:45:33,453 INFO org.apache.hadoop.hbase.regionserver.HStore: > > Completed major compaction of 1 (all) file(s) in log of > > java_app_logs,CM-sjcmhpcapp01-1476076988799-135168375- > 178aebc1-4bfc-47c4-907f-952f50270f50-b,1476360804449. > 29d4fa5482d17daf813e12732155e0ee. > > into 47b8ee796fc74ff0b82222bf60318f99(size=549), total size for store is > > 549. This selection was in queue for 0sec, and took 0sec to execute. > > 2016-11-03 22:45:33,454 INFO > > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed > > compaction: Request = > > regionName=java_app_logs,CM-sjcmhpcapp01-1476076988799- > 135168375-178aebc1-4bfc-47c4-907f-952f50270f50-b,1476360804449. > 29d4fa5482d17daf813e12732155e0ee., > > storeName=log, fileCount=1, fileSize=1.3 G, priority=9, > > time=25652433393376299; duration=0sec > > > You can find that its size is only 549 bytes at that time. But HBase didn't > remove it. > > Here is the result I run HFile tool > > $ hbase org.apache.hadoop.hbase.io.hfile.HFile -v -f > > hdfs://xxxxxxx:8020/hbase/data/default/java_app_logs/ > 29d4fa5482d17daf813e12732155e0ee/log/47b8ee796fc74ff0b82222bf60318f99 > > 16/11/18 16:48:25 INFO Configuration.deprecation: hadoop.native.lib is > > deprecated. Instead, use io.native.lib.available > > 16/11/18 16:48:29 INFO Configuration.deprecation: fs.default.name is > > deprecated. Instead, use fs.defaultFS > > Scanning -> > > hdfs://xxxxxxx:8020/hbase/data/default/java_app_logs/ > 29d4fa5482d17daf813e12732155e0ee/log/47b8ee796fc74ff0b82222bf60318f99 > > 16/11/18 16:48:29 INFO hfile.CacheConfig: CacheConfig:disabled > > Scanned kv count -> 0 > > > > 2016-11-17 18:45 GMT-08:00 Ted Yu <[email protected]>: > > > You would be able to find in region server log(s). > > > > Can you do quick inspection of the hfiles using: > > > > http://hbase.apache.org/book.html#hfile_tool > > > > BTW you're still on old release of 0.98.x, right ? > > > > On Thu, Nov 17, 2016 at 6:40 PM, Xi Yang <[email protected]> wrote: > > > > > Thank you for reply. > > > > > > Yes. There are many regions have this problem. And those regions were > > > created several years ago. > > > hbase.hregion.majorcompaction of our cluster is 7 days. > > > Where can I see the last major compaction time? > > > > > > Thanks, > > > Alex > > > > > > 2016-11-17 18:22 GMT-08:00 Ted Yu <[email protected]>: > > > > > > > When was the last time major compaction was performed on this region > ? > > > > > > > > Were you referring to the store files by 'them' in your question ? > > > > > > > > Cheers > > > > > > > > On Thu, Nov 17, 2016 at 6:18 PM, Xi Yang <[email protected]> > > wrote: > > > > > > > > > I have some regions are empty. But HBase doesn't remove them, why? > > > > > > > > > > You can see the storeFileSize is only 549. But I use hbase shell to > > > scan > > > > > this region, the result is empty > > > > > > > > > > "Namespace_default_table_java_app_logs_region_ > > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_storeFileCount" > > > > > : 1, > > > > > "Namespace_default_table_java_app_logs_region_ > > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_memStoreSize" > > > > > : 424, > > > > > "Namespace_default_table_java_app_logs_region_ > > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_storeFileSize" > > > > > : 549, > > > > > "Namespace_default_table_java_app_logs_region_ > > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_compactionsCompletedCount" > > > > > : 4, > > > > > "Namespace_default_table_java_app_logs_region_ > > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_numBytesCompactedCount" > > > > > : 8503271111, > > > > > "Namespace_default_table_java_app_logs_region_ > > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_numFilesCompactedCount" > > > > > : 4, > > > > > > > > > > > > > > >
