Hi,
We met a weird issue in our production environment.
Region compaction is always failing with following errors,
1.
2017-01-10 02:19:10,427 | ERROR |
regionserver/RS-HOST/RS-IP:PORT-longCompactions-1483858654825 | Compaction
failed Request = regionName=XXXX., storeName=XYZ, fileCount=6, fileSize=100.7 M
(3.2 M, 20.8 M, 15.1 M, 20.9 M, 21.0 M, 19.7 M), priority=-5,
time=1747414906352088 |
org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:562)
java.io.IOException: ScanWildcardColumnTracker.checkColumn ran into a column
actually smaller than the previous column: XXXXXXX
at
org.apache.hadoop.hbase.regionserver.ScanWildcardColumnTracker.checkVersions(ScanWildcardColumnTracker.java:114)
at
org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.match(ScanQueryMatcher.java:457)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:551)
at
org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:328)
at
org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:104)
at
org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:133)
at
org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1243)
at
org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1895)
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:546)
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:583)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecuto
2.
2017-01-10 02:33:53,009 | ERROR |
regionserver/RS-HOST/RS-IP:PORT-longCompactions-1483686810953 | Compaction
failed Request = regionName=YYYYYY, storeName=ABC, fileCount=6, fileSize=125.3
M (20.9 M, 20.9 M, 20.9 M, 20.9 M, 20.9 M, 20.9 M), priority=-68,
time=1748294500157323 |
org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:562)
java.io.IOException: Non-increasing Bloom keys: XXXXXXXXXXXXXXXXXXXXXX after
XXXXXXXXXXXX
at
org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:911)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:947)
at
org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:337)
at
org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:104)
at
org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:133)
at
org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1243)
at
org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1895)
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:546)
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:583)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
HBase version : 1.0.2
We have verified all the HFiles in the store using HFilePrettyPrinter with "k"
(checkrow), all report is normal. Full scan is also successful.
We don't have the access to the actual data and may be customer wont agree to
share that .
Have anyone faced this issue, any pointers will be much appreciated.
Thanks & Regards,
Pankaj
