can you try to read the file by using: org.apache.hadoop.hbase.io.hfile.HFile -f FILE_PATH -s -e
to verify if the problem is reading the block from cache or the data on disk is corrupted in some way? The stacktrace is showing a file size > 255GB and a Key of 16M which I doubt being correct. Matteo On Tue, Mar 18, 2014 at 5:26 PM, Ted Yu <[email protected]> wrote: > The stack trace doesn't reveal too much other than the error occurred at > the beginning of a scan. > > Other experts would have more insights than I do. > > Cheers > > > On Tue, Mar 18, 2014 at 10:10 AM, Sudarshan Kadambi (BLOOMBERG/ 731 LEXIN) > < > [email protected]> wrote: > > > Thanks Ted. No, we don't use data block encoding. This is 0.96. It > appears > > multiple HFiles are affected. Does the stack trace give some context on > > what is that it's attempting to do at the lowest level? > > > > > > ----- Original Message ----- > > From: [email protected] > > To: Sudarshan Kadambi (BLOOMBERG/ 731 LEXIN) <[email protected]>, > > [email protected] > > At: Mar 18 2014 13:03:20 > > > > Are you using Data Block Encoding for this table ? > > > > Can you sideline the HFile and see if other HFiles were affected ? > > > > I assume you use 0.94.x > > > > Cheers > > > > > > On Tue, Mar 18, 2014 at 7:14 AM, Sudarshan Kadambi (BLOOMBERG/ 731 LEXIN) > > <[email protected]> wrote: > > > >> Hi Folks: > >> > >> We've a table that we're unable to make online. A scan on this table > >> yields the following stack trace. hbck doesn't report any > inconsistencies. > >> I went ahead and did a repair, which didn't make a difference. Any tips? > >> > >> ERROR: java.io.IOException: Invalid currKeyLen 16777280 or currValueLen > >> 64. Block offset: 274609471488, block length: 16415, position: 0 > (without > >> header). > >> at > org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2213) > >> at > >> org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1879) > >> Caused by: java.lang.IllegalStateException: Invalid currKeyLen 16777280 > >> or currValueLen 64. Block offset: 274609471488, block length: 16415, > >> position: 0 (without header). > >> at > >> > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.readKeyValueLen(HFileReaderV2.java:820) > >> at > >> > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.updateCurrBlock(HFileReaderV2.java:787) > >> at > >> > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:749) > >> at > >> > org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:232) > >> at > >> > org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:145) > >> at > >> > org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:168) > >> at > >> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1642) > >> at > >> > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:3460) > >> at > >> > org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1768) > >> at > >> > org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1760) > >> at > >> > org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1737) > >> at > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:2996) > >> at > >> > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26929) > >> at > org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2175) > >> ... 1 more > >> > >> > > >
