Thanks Ted for replying.
Actually issue happened in production environment and there are many HFiles in
that store (can't get the file). As we don't log the file name which is
corrupted, Is there anyway to get the corrupted file name?
Block encoding is "NONE", table schema has bloom filter as "ROW", compression
type is "Snappy" and durability is SKIP_WAL.
Regards,
Pankaj
-----Original Message-----
From: Ted Yu [mailto:[email protected]]
Sent: Friday, January 13, 2017 10:30 PM
To: [email protected]
Cc: [email protected]
Subject: Re: Region comapction failed
In the second case, the error happened when writing hfile. Can you track down
the path of the new file so that further investigation can be done ?
Does the table use any encoding ?
Thanks
> On Jan 13, 2017, at 2:47 AM, Pankaj kr <[email protected]> wrote:
>
> Hi,
>
> We met a weird issue in our production environment.
>
> Region compaction is always failing with following errors,
>
> 1.
> 2017-01-10 02:19:10,427 | ERROR |
> regionserver/RS-HOST/RS-IP:PORT-longCompactions-1483858654825 | Compaction
> failed Request = regionName=XXXX., storeName=XYZ, fileCount=6, fileSize=100.7
> M (3.2 M, 20.8 M, 15.1 M, 20.9 M, 21.0 M, 19.7 M), priority=-5,
> time=1747414906352088 |
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:562)
> java.io.IOException: ScanWildcardColumnTracker.checkColumn ran into a column
> actually smaller than the previous column: XXXXXXX
> at
> org.apache.hadoop.hbase.regionserver.ScanWildcardColumnTracker.checkVersions(ScanWildcardColumnTracker.java:114)
> at
> org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.match(ScanQueryMatcher.java:457)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:551)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:328)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:104)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:133)
> at
> org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1243)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1895)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:546)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:583)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecuto
>
> 2.
> 2017-01-10 02:33:53,009 | ERROR |
> regionserver/RS-HOST/RS-IP:PORT-longCompactions-1483686810953 | Compaction
> failed Request = regionName=YYYYYY, storeName=ABC, fileCount=6,
> fileSize=125.3 M (20.9 M, 20.9 M, 20.9 M, 20.9 M, 20.9 M, 20.9 M),
> priority=-68, time=1748294500157323 |
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:562)
> java.io.IOException: Non-increasing Bloom keys: XXXXXXXXXXXXXXXXXXXXXX after
> XXXXXXXXXXXX
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:911)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:947)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:337)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:104)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:133)
> at
> org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1243)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1895)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:546)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:583)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> HBase version : 1.0.2
>
> We have verified all the HFiles in the store using HFilePrettyPrinter with
> "k" (checkrow), all report is normal. Full scan is also successful.
> We don't have the access to the actual data and may be customer wont agree to
> share that .
>
> Have anyone faced this issue, any pointers will be much appreciated.
>
> Thanks & Regards,
> Pankaj
