The RS logs is filled with exceptions like the one I have specified below..
Vidhya
RS log:
2010-09-01 18:23:55,883 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
java.io.IOException: Could not seek StoreFileScanner[HFileScanner for reader
reader=hdfs://b3130080.yst.yahoo.net:4600/hbase/DocDB/912a67889c51bee4a6252f289b0ad95e/bigColumn/149303798533194752,
compression=none, inMemory=false,
firstKey=0000003568669995/bigColumn:CONTENT/1282901111530/Put,
lastKey=0000003568803993/bigColumn:CONTENT/1282902034907/Put, avgKeyLen=44,
avgValueLen=32000, entries=133999, length=4295200267, cur=null]
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:104)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:73)
at
org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1201)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.<init>(HRegion.java:2333)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateInternalScanner(HRegion.java:1205)
at
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1197)
at
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1178)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1814)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:576)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:919)
Caused by: java.io.IOException: Could not obtain block:
blk_4305541000700762499_1100131
file=/hbase/DocDB/912a67889c51bee4a6252f289b0ad95e/bigColumn/149303798533194752
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1787)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1615)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1742)
at java.io.DataInputStream.read(DataInputStream.java:132)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:105)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1060)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1007)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1373)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:125)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:96)
... 12 more
2010-09-01 18:23:56,926 WARN org.apache.hadoop.hdfs.DFSClient: DFS Read:
java.io.IOException: Could not obtain block: blk_4305541000700762499_1100131
file=/hbase/DocDB/912a67889c51bee4a6252f289b0ad95e/bigColumn/149303798533194752
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1787)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1615)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1742)
at java.io.DataInputStream.read(DataInputStream.java:132)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:105)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1060)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1007)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1373)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:125)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:96)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:73)
at
org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1201)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.<init>(HRegion.java:2333)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateInternalScanner(HRegion.java:1205)
at
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1197)
at
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1178)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1814)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:576)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:919)
On 9/1/10 12:06 PM, "Todd Lipcon" <[email protected]> wrote:
Hi Vidhya,
Problems like this used to be more frequent, but then we did a bunch
of DFS bug fixes in the hadoop-0.20-append branch that resolved a lot
of them. I imagine you're using YDH which doesn't have all the fixes,
but I couldn't say exactly what issue this is.
Could you grep both the NN and RS logs for blk_4731742382812534502 and
pastebin what you find?
Thanks
-Todd
On Wed, Sep 1, 2010 at 11:58 AM, Vidhyashankar Venkataraman
<[email protected]> wrote:
> I have been trying to run my scanner jobs and sometimes they fail due to DFS
> errors in one of the storefiles:
>
> I looked at the namenode logs and the file that caused the problem was in the
> process of getting fixed by the namenode but by then the scanner failed.. (I
> tried copying the file after the failure and it did without any problems)..
>
> I just wanted to confirm with you guys if this is a common problem..
>
> Vidhya
>
> The MR exceptions:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
> region server b3130504.yst.yahoo.net:60020 for region
> DocDB,0000000190009985,1282947505610.532732ba4420906a085e22489a897bea., row
> '0000000190009985', but failed after 10 attempts.
> Exceptions:
> java.io.IOException: java.io.IOException: Could not seek
> StoreFileScanner[HFileScanner for reader
> reader=hdfs://b3130080.yst.yahoo.net:4600/hbase/DocDB/532732ba4420906a085e22489a897bea/bigColumn/4660977350540113930,
> compression=none, inMemory=false,
> firstKey=0000000190009985/bigColumn:CONTENT/1282909598083/Put,
> lastKey=0000000190143983/bigColumn:CONTENT/1282910525475/Put, avgKeyLen=44,
> avgValueLen=32000, entries=133999, length=4295200267, cur=null]
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:104)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:73)
> at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1201)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.(HRegion.java:2333)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateInternalScanner(HRegion.java:1205)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1197)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1178)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1814)
> at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:576)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:919)
> Caused by: java.io.IOException: Could not obtain block:
> blk_4731742382812534502_1129302
> file=/hbase/DocDB/532732ba4420906a085e22489a897bea/bigColumn/4660977350540113930
> at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1787)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1615)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1742)
> at java.io.DataInputStream.read(DataInputStream.java:132)
> at
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:105)
> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100)
> at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1060)
> at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1007)
> at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1373)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:125)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:96)
> ... 12 more
>
>
--
Todd Lipcon
Software Engineer, Cloudera
