Yes, we're still on old release of 0.98.x Many of empty region didn't be deleted. Take one for example:
> "Namespace_default_table_java_app_logs_region_29d4fa5482d17daf813e12732155e0ee_metric_storeCount" > : 1, > > "Namespace_default_table_java_app_logs_region_29d4fa5482d17daf813e12732155e0ee_metric_storeFileCount" > : 1, > > "Namespace_default_table_java_app_logs_region_29d4fa5482d17daf813e12732155e0ee_metric_memStoreSize" > : 424, > > "Namespace_default_table_java_app_logs_region_29d4fa5482d17daf813e12732155e0ee_metric_storeFileSize" > : 549, > > "Namespace_default_table_java_app_logs_region_29d4fa5482d17daf813e12732155e0ee_metric_compactionsCompletedCount" > : 4, > > "Namespace_default_table_java_app_logs_region_29d4fa5482d17daf813e12732155e0ee_metric_numBytesCompactedCount" > : 8616506268, > > "Namespace_default_table_java_app_logs_region_29d4fa5482d17daf813e12732155e0ee_metric_numFilesCompactedCount" > : 4, The log about last time major compaction of this region is: 2016-11-03 22:45:33,453 INFO org.apache.hadoop.hbase.regionserver.HStore: > Completed major compaction of 1 (all) file(s) in log of > java_app_logs,CM-sjcmhpcapp01-1476076988799-135168375-178aebc1-4bfc-47c4-907f-952f50270f50-b,1476360804449.29d4fa5482d17daf813e12732155e0ee. > into 47b8ee796fc74ff0b82222bf60318f99(size=549), total size for store is > 549. This selection was in queue for 0sec, and took 0sec to execute. > 2016-11-03 22:45:33,454 INFO > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed > compaction: Request = > regionName=java_app_logs,CM-sjcmhpcapp01-1476076988799-135168375-178aebc1-4bfc-47c4-907f-952f50270f50-b,1476360804449.29d4fa5482d17daf813e12732155e0ee., > storeName=log, fileCount=1, fileSize=1.3 G, priority=9, > time=25652433393376299; duration=0sec You can find that its size is only 549 bytes at that time. But HBase didn't remove it. Here is the result I run HFile tool $ hbase org.apache.hadoop.hbase.io.hfile.HFile -v -f > hdfs://xxxxxxx:8020/hbase/data/default/java_app_logs/29d4fa5482d17daf813e12732155e0ee/log/47b8ee796fc74ff0b82222bf60318f99 > 16/11/18 16:48:25 INFO Configuration.deprecation: hadoop.native.lib is > deprecated. Instead, use io.native.lib.available > 16/11/18 16:48:29 INFO Configuration.deprecation: fs.default.name is > deprecated. Instead, use fs.defaultFS > Scanning -> > hdfs://xxxxxxx:8020/hbase/data/default/java_app_logs/29d4fa5482d17daf813e12732155e0ee/log/47b8ee796fc74ff0b82222bf60318f99 > 16/11/18 16:48:29 INFO hfile.CacheConfig: CacheConfig:disabled > Scanned kv count -> 0 2016-11-17 18:45 GMT-08:00 Ted Yu <[email protected]>: > You would be able to find in region server log(s). > > Can you do quick inspection of the hfiles using: > > http://hbase.apache.org/book.html#hfile_tool > > BTW you're still on old release of 0.98.x, right ? > > On Thu, Nov 17, 2016 at 6:40 PM, Xi Yang <[email protected]> wrote: > > > Thank you for reply. > > > > Yes. There are many regions have this problem. And those regions were > > created several years ago. > > hbase.hregion.majorcompaction of our cluster is 7 days. > > Where can I see the last major compaction time? > > > > Thanks, > > Alex > > > > 2016-11-17 18:22 GMT-08:00 Ted Yu <[email protected]>: > > > > > When was the last time major compaction was performed on this region ? > > > > > > Were you referring to the store files by 'them' in your question ? > > > > > > Cheers > > > > > > On Thu, Nov 17, 2016 at 6:18 PM, Xi Yang <[email protected]> > wrote: > > > > > > > I have some regions are empty. But HBase doesn't remove them, why? > > > > > > > > You can see the storeFileSize is only 549. But I use hbase shell to > > scan > > > > this region, the result is empty > > > > > > > > "Namespace_default_table_java_app_logs_region_ > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_storeFileCount" > > > > : 1, > > > > "Namespace_default_table_java_app_logs_region_ > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_memStoreSize" > > > > : 424, > > > > "Namespace_default_table_java_app_logs_region_ > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_storeFileSize" > > > > : 549, > > > > "Namespace_default_table_java_app_logs_region_ > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_compactionsCompletedCount" > > > > : 4, > > > > "Namespace_default_table_java_app_logs_region_ > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_numBytesCompactedCount" > > > > : 8503271111, > > > > "Namespace_default_table_java_app_logs_region_ > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_numFilesCompactedCount" > > > > : 4, > > > > > > > > > >
