Hi Dru,
There are some issues related it is believed to HBase keeping
open references to files in its DFS client. This saves the
latency of round trips to the master for block locations, but
seems fragile.
I have found that increasing the number of datanode handlers
via dfs.datanode.handler.count improves DFS/DFSClient
stability under heavy load.
- Andy
> From: Dru Jensen
> Subject: Missing file
> java.io.IOException: java.io.IOException: Cannot open
> filename
> /hbase/webmaps/743469791/header/mapfiles/4735713346568547172/data
> at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1395)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1720)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1536)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1663)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1593)
> at
> java.io.DataInputStream.readInt(DataInputStream.java:370)
> at
> org.apache.hadoop.hbase.io.SequenceFile$Reader.readRecordLength(SequenceFile.java:1909)
> at
> org.apache.hadoop.hbase.io.SequenceFile$Reader.next(SequenceFile.java:1939)
> at
> org.apache.hadoop.hbase.io.SequenceFile$Reader.next(SequenceFile.java:1844)
> at
> org.apache.hadoop.hbase.io.MapFile$Reader.seekInternal(MapFile.java:472)
> at
> org.apache.hadoop.hbase.io.MapFile$Reader.getClosest(MapFile.java:567)
> at
> org.apache.hadoop.hbase.io.MapFile$Reader.getClosest(MapFile.java:550)
> at
> org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.getClosest(BloomFilterMapFile.java:112)
> at
> org.apache.hadoop.hbase.regionserver.HStore.get(HStore.java:1485)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:975)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1504)
> at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
> Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
>
> The MR process needs to check to make sure the row is the
> latest before inserting it so it calls htable.get(key) in
> the mr process. This is where the process fails.
>
> I also tried performing a get 'table',
> 'key' in the shell with the same error. I verified
> that the file is missing in hadoop. It looks like HBase did
> not update its reference after a split/region assignment.
> Is anyone else experiencing this?
>
> using 732094 from trunk.
>
> Thanks for your help,
> Dru
