Thanks Nick & Ted. Thank you for your help and your patience. "Tool for retiring empty regions" will be an useful feature!
2016-11-18 17:18 GMT-08:00 Nick Dimiduk <[email protected]>: > HBase will not remove empty regions. It assumes you know what you're doing. > > In 1.2 there's a new "Region Normalizer" [0] feature that runs in the > Master, which acts a bit like the Balancer, but for region sizes. I think > it's still considered experimental, so YMMV. We're still on 1.1, so I > haven't tried it in prod yet. > > I also have this problem in our cluster, because of a combination of TTL > and Phoenix's Rowkey Timestamp feature. This results in region boundaries > including the timestamp, and data that goes away after a period. Eventually > all regions will become empty. I have a script that runs periodically to > calculate region sizes and drop empty regions. I volunteered to contribute > it [1], but haven't pulled it out of our puppet infra in the form of a > patch. Let me revive that thread internally. > > -n > > [0]: https://issues.apache.org/jira/browse/HBASE-13103 > [1]: https://issues.apache.org/jira/browse/HBASE-15712 > > On Fri, Nov 18, 2016 at 4:59 PM, Xi Yang <[email protected]> wrote: > > > Yes, we're still on old release of 0.98.x > > > > Many of empty region didn't be deleted. Take one for example: > > > > > > > "Namespace_default_table_java_app_logs_region_ > > 29d4fa5482d17daf813e12732155e0ee_metric_storeCount" > > > : 1, > > > > > > "Namespace_default_table_java_app_logs_region_ > > 29d4fa5482d17daf813e12732155e0ee_metric_storeFileCount" > > > : 1, > > > > > > "Namespace_default_table_java_app_logs_region_ > > 29d4fa5482d17daf813e12732155e0ee_metric_memStoreSize" > > > : 424, > > > > > > "Namespace_default_table_java_app_logs_region_ > > 29d4fa5482d17daf813e12732155e0ee_metric_storeFileSize" > > > : 549, > > > > > > "Namespace_default_table_java_app_logs_region_ > > 29d4fa5482d17daf813e12732155e0ee_metric_compactionsCompletedCount" > > > : 4, > > > > > > "Namespace_default_table_java_app_logs_region_ > > 29d4fa5482d17daf813e12732155e0ee_metric_numBytesCompactedCount" > > > : 8616506268, > > > > > > "Namespace_default_table_java_app_logs_region_ > > 29d4fa5482d17daf813e12732155e0ee_metric_numFilesCompactedCount" > > > : 4, > > > > > > The log about last time major compaction of this region is: > > > > 2016-11-03 22:45:33,453 INFO org.apache.hadoop.hbase. > regionserver.HStore: > > > Completed major compaction of 1 (all) file(s) in log of > > > java_app_logs,CM-sjcmhpcapp01-1476076988799-135168375- > > 178aebc1-4bfc-47c4-907f-952f50270f50-b,1476360804449. > > 29d4fa5482d17daf813e12732155e0ee. > > > into 47b8ee796fc74ff0b82222bf60318f99(size=549), total size for store > is > > > 549. This selection was in queue for 0sec, and took 0sec to execute. > > > 2016-11-03 22:45:33,454 INFO > > > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed > > > compaction: Request = > > > regionName=java_app_logs,CM-sjcmhpcapp01-1476076988799- > > 135168375-178aebc1-4bfc-47c4-907f-952f50270f50-b,1476360804449. > > 29d4fa5482d17daf813e12732155e0ee., > > > storeName=log, fileCount=1, fileSize=1.3 G, priority=9, > > > time=25652433393376299; duration=0sec > > > > > > You can find that its size is only 549 bytes at that time. But HBase > didn't > > remove it. > > > > Here is the result I run HFile tool > > > > $ hbase org.apache.hadoop.hbase.io.hfile.HFile -v -f > > > hdfs://xxxxxxx:8020/hbase/data/default/java_app_logs/ > > 29d4fa5482d17daf813e12732155e0ee/log/47b8ee796fc74ff0b82222bf60318f99 > > > 16/11/18 16:48:25 INFO Configuration.deprecation: hadoop.native.lib is > > > deprecated. Instead, use io.native.lib.available > > > 16/11/18 16:48:29 INFO Configuration.deprecation: fs.default.name is > > > deprecated. Instead, use fs.defaultFS > > > Scanning -> > > > hdfs://xxxxxxx:8020/hbase/data/default/java_app_logs/ > > 29d4fa5482d17daf813e12732155e0ee/log/47b8ee796fc74ff0b82222bf60318f99 > > > 16/11/18 16:48:29 INFO hfile.CacheConfig: CacheConfig:disabled > > > Scanned kv count -> 0 > > > > > > > > 2016-11-17 18:45 GMT-08:00 Ted Yu <[email protected]>: > > > > > You would be able to find in region server log(s). > > > > > > Can you do quick inspection of the hfiles using: > > > > > > http://hbase.apache.org/book.html#hfile_tool > > > > > > BTW you're still on old release of 0.98.x, right ? > > > > > > On Thu, Nov 17, 2016 at 6:40 PM, Xi Yang <[email protected]> > wrote: > > > > > > > Thank you for reply. > > > > > > > > Yes. There are many regions have this problem. And those regions were > > > > created several years ago. > > > > hbase.hregion.majorcompaction of our cluster is 7 days. > > > > Where can I see the last major compaction time? > > > > > > > > Thanks, > > > > Alex > > > > > > > > 2016-11-17 18:22 GMT-08:00 Ted Yu <[email protected]>: > > > > > > > > > When was the last time major compaction was performed on this > region > > ? > > > > > > > > > > Were you referring to the store files by 'them' in your question ? > > > > > > > > > > Cheers > > > > > > > > > > On Thu, Nov 17, 2016 at 6:18 PM, Xi Yang <[email protected]> > > > wrote: > > > > > > > > > > > I have some regions are empty. But HBase doesn't remove them, > why? > > > > > > > > > > > > You can see the storeFileSize is only 549. But I use hbase shell > to > > > > scan > > > > > > this region, the result is empty > > > > > > > > > > > > "Namespace_default_table_java_app_logs_region_ > > > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_storeFileCount" > > > > > > : 1, > > > > > > "Namespace_default_table_java_app_logs_region_ > > > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_memStoreSize" > > > > > > : 424, > > > > > > "Namespace_default_table_java_app_logs_region_ > > > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_storeFileSize" > > > > > > : 549, > > > > > > "Namespace_default_table_java_app_logs_region_ > > > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_ > compactionsCompletedCount" > > > > > > : 4, > > > > > > "Namespace_default_table_java_app_logs_region_ > > > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_numBytesCompactedCount" > > > > > > : 8503271111, > > > > > > "Namespace_default_table_java_app_logs_region_ > > > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_numFilesCompactedCount" > > > > > > : 4, > > > > > > > > > > > > > > > > > > > > >
