File attachments won't work on the mailing list. Can you put the files on
some hosting service?
Can you reproduce the problem on hbase 2.2? HBase 2.1 has been EOM since
May.
On Fri, Jul 3, 2020, 18:20 Mohamed Meeran <meeran.gladiat...@gmail.com>
wrote:
> Hi,
>
> We are using HBase-2.1.9 (Hadoop-3.1.3) in our setup. In the logs, we see
> major compaction failed for some of the regions with the following error
> logs.
>
> Caused by: java.io.IOException: Could not iterate
> StoreFileScanner[HFileScanner for reader
> reader=hdfs://TestCluster/hbasedata/data/Test/Test/6472f3839fc9b0a1d4b64e182043bc52/hb/2ec37243628b4a03ae3d937da4c27081,
> compression=none, cacheConf=blockCache=LruBlockCache{blockCount=332,
> currentSize=485.88 MB, freeSize=333.32 MB, maxSize=819.20 MB,
> heapSize=485.88 MB, minSize=778.24 MB, minFactor=0.95, multiSize=389.12 MB,
> multiFactor=0.5, singleSize=194.56 MB, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
> cacheDataCompressed=false, prefetchOnOpen=false,
> firstKey=Optional[10259783_1010157000000008129/hb:B/1490097103780/Put/seqid=0],
> lastKey=Optional[10260211_1009658000000470017/hb:H/1490097295354/Put/seqid=0],
> avgKeyLen=43, avgValueLen=213357, entries=10134, length=2163318554,
> cur=10259783_1010157000000008851/hb:B/1490097148981/Put/vlen=16591695/seqid=0]
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:217)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:120)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:654)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:153)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:6593)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6757)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:6527)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3158)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3407)
> ... 5 more
> Caused by: java.io.IOException: Invalid onDisksize=-969694035: expected to
> be at least 33 and at most 2147483647, or -1
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.checkAndGetSizeAsInt(HFileBlock.java:1673)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1746)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1610)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1496)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.readNextDataBlock(HFileReaderImpl.java:931)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.isNextBlock(HFileReaderImpl.java:1064)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.positionForNextBlock(HFileReaderImpl.java:1058)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl._next(HFileReaderImpl.java:1076)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.next(HFileReaderImpl.java:1097)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:208)
> ... 13 more
>
> We analysed a file using the hfile tool. Attaching the output for
> printblocks and printblockheaders.
>
> Any help to fix this would be greatly appreciated.
> --
> Thanks in advance,
> Meeran
>
