The exception came from FSInputChecker.verifySum(). Can you check Namenode log around the time the error happened to find more information ? What hadoop version do you use ?
BTW 0.90.6 is really old. I suggest upgrading to 0.92 or 0.94 Cheers On Sat, Dec 15, 2012 at 4:13 PM, Stanislav Orlenko <[email protected]>wrote: > Hello > We use HBase 0.90.6 and have the problem: > > org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact > region server marv.site:56463 for region > > webpage,pl.allegro:http/stare-rozdzielacze-i2839591826.html,1355478823557.80e5ed6af22f12d45eabae89537f8602., > row 'pl.allegro:http/stare-rozdzielacze-i2839591826.html', but failed after > 10 attempts. > Exceptions: > java.io.IOException: java.io.IOException: Could not iterate > StoreFileScanner[HFileScanner for reader > > reader=file:/home/stas/pricex/hbase/webpage/80e5ed6af22f12d45eabae89537f8602/f/5763919690181742358, > compression=none, inMemory=false, > > firstKey=pl.allegro:http/stare-rozdzielacze-i2839591826.html/f:bas/1355311168483/Put, > lastKey=pl.gumtree.lodz:http/p-TermsAndConditions/f:ts/1355360972551/Put, > avgKeyLen=83, avgValueLen=2880, entries=64207, length=191013462, > > cur=pl.allegro:http/swiat-mebli-komoda-srebrna-61980-k14-i2787252255.html/f:cnt/1355480654644/Put/vlen=62169] > at > > org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:89) > at > > org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:82) > at > > org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:262) > at > > org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114) > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2469) > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2425) > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2442) > at > > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1863) > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039) > Caused by: org.apache.hadoop.fs.ChecksumException: Checksum error: > > file:/home/stas/pricex/hbase/webpage/80e5ed6af22f12d45eabae89537f8602/f/5763919690181742358 > at 28099584 > at > org.apache.hadoop.fs.FSInputChecker.verifySum(FSInputChecker.java:277) > at > > org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:241) > at > org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189) > at > org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158) > at java.io.DataInputStream.read(DataInputStream.java:132) > at > > org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:105) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:256) > at java.io.BufferedInputStream.read(BufferedInputStream.java:317) > at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102) > at > org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094) > at > org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036) > at > org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1276) > at > > org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:87) > ... 12 more > > java.io.IOException: java.io.IOException: > java.lang.IllegalArgumentException > at > > org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:997) > at > > org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:986) > at > > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1887) > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039) > Caused by: java.lang.IllegalArgumentException > at java.nio.Buffer.position(Buffer.java:218) > at > org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1266) > at > > org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:87) > at > > org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:82) > at > > org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:262) > at > > org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114) > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2469) > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2425) > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2442) > at > > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1863) > ... 5 more > > We do not use special configuration. Only hbase.rootdir and > hbase.zookeeper.property.dataDir are specified in hbase-site. > What can be a reason for such behavior? > > Thanks in advance >
