I have been trying to run my scanner jobs and sometimes they fail due to DFS
errors in one of the storefiles:
I looked at the namenode logs and the file that caused the problem was in the
process of getting fixed by the namenode but by then the scanner failed.. (I
tried copying the file after the failure and it did without any problems)..
I just wanted to confirm with you guys if this is a common problem..
Vidhya
The MR exceptions:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server b3130504.yst.yahoo.net:60020 for region
DocDB,0000000190009985,1282947505610.532732ba4420906a085e22489a897bea., row
'0000000190009985', but failed after 10 attempts.
Exceptions:
java.io.IOException: java.io.IOException: Could not seek
StoreFileScanner[HFileScanner for reader
reader=hdfs://b3130080.yst.yahoo.net:4600/hbase/DocDB/532732ba4420906a085e22489a897bea/bigColumn/4660977350540113930,
compression=none, inMemory=false,
firstKey=0000000190009985/bigColumn:CONTENT/1282909598083/Put,
lastKey=0000000190143983/bigColumn:CONTENT/1282910525475/Put, avgKeyLen=44,
avgValueLen=32000, entries=133999, length=4295200267, cur=null]
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:104)
at org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:73)
at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1201)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.(HRegion.java:2333)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateInternalScanner(HRegion.java:1205)
at
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1197)
at
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1178)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1814)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:576)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:919)
Caused by: java.io.IOException: Could not obtain block:
blk_4731742382812534502_1129302
file=/hbase/DocDB/532732ba4420906a085e22489a897bea/bigColumn/4660977350540113930
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1787)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1615)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1742)
at java.io.DataInputStream.read(DataInputStream.java:132)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:105)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100)
at org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1060)
at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1007)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1373)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:125)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:96)
... 12 more
