(What version of hbase?)
Unfortunately the exception does not print the file name. Can you figure
which file or region it is? Could try reading content w/ hfile tool to see
if it complains also.
Problem is:
On-disk size without header provided is
36901, but block header contains 859321901. Block offset: 440164791, data
starts with: 1-9d66-4386-83bc-48b335630d0e\x88\x01\x01
at org.apache.hadoop.hbase.io.hfile.HFileBlock.
validateOnDiskSizeWithoutHeader(HFileBlock.java:521)
I see the above exception in HBASE-13830 where Nils perhaps fixes it by
upgrading hbase (see end comment).
St.Ack
On Wed, Jul 5, 2017 at 1:00 AM, mukund murrali <[email protected]>
wrote:
> Hi,
>
> Recently we faced an issue in our HBase Cluster and following is the
> exception trace in client. Any idea what could be the cause and how to
> rectify this?
>
> RpcRetryingCaller{globalStartTime=1499165478610, pause=100, retries=35},
> java.io.IOException: java.io.IOException: Could not reseek
> StoreFileScanner[org.apache.hadoop.hbase.io.HalfStoreFileReader$1@2e508d2b
> ,
> cur=xxxxxxx/dataCF:5f1e00cd-cbea-45d6-8542-72b8b00960ff/
> 1498885978888/Maximum/vlen=0/seqid=0] to key xxxxxxx/dataCF:5f1e00cd-cbea-
> 45d6-8542-72b8b00960ff/LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> reseek(StoreFileScanner.java:217)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> enforceSeek(StoreFileScanner.java:408)
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
> pollRealKV(KeyValueHeap.java:363)
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
> next(KeyValueHeap.java:123)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.
> next(StoreScanner.java:588)
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
> next(KeyValueHeap.java:150)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
> populateResult(HRegion.java:5731)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
> nextInternal(HRegion.java:5894)
> at org.apache.hadoop.hbase.regionserver.HRegion$
> RegionScannerImpl.nextRaw(HRegion.java:5668)
> at org.apache.hadoop.hbase.regionserver.HRegion$
> RegionScannerImpl.next(
> HRegion.java:5645)
> at org.apache.hadoop.hbase.regionserver.HRegion$
> RegionScannerImpl.next(
> HRegion.java:5631)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6829)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6807)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.
> get(RSRpcServices.java:2049)
> at org.apache.hadoop.hbase.protobuf.generated.
> ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(
> RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: On-disk size without header provided is
> 36901, but block header contains 859321901. Block offset: 440164791, data
> starts with: 1-9d66-4386-83bc-48b335630d0e\x88\x01\x01
> at org.apache.hadoop.hbase.io.hfile.HFileBlock.
> validateOnDiskSizeWithoutHeader(HFileBlock.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileBlock.access$700(
> HFileBlock.java:88)
> at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.
> readBlockDataInternal(HFileBlock.java:1665)
> at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.
> readBlockData(HFileBlock.java:1532)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(
> HFileReaderV2.java:452)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.
> loadDataBlockWithScanInfo(HFileBlockIndex.java:271)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$
> AbstractScannerV2.seekTo(HFileReaderV2.java:649)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$
> AbstractScannerV2.reseekTo(HFileReaderV2.java:629)
> at org.apache.hadoop.hbase.io.HalfStoreFileReader$1.
> reseekTo(HalfStoreFileReader.java:293)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> reseekAtOrAfter(StoreFileScanner.java:288)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> reseek(StoreFileScanner.java:200)
> ... 19 more
>
> ThanksU
> Regards,
>
> Mukund Murrali
>
