Re: Missing file

Tue, 13 Jan 2009 10:19:48 -0800 

Thanks Andy.  I will try it and let you know if it helps.
On Jan 12, 2009, at 5:08 PM, Andrew Purtell wrote:


Hi Dru,

There are some issues related it is believed to HBase keeping
open references to files in its DFS client. This saves the
latency of round trips to the master for block locations, but
seems fragile.

I have found that increasing the number of datanode handlers
via dfs.datanode.handler.count improves DFS/DFSClient
stability under heavy load.

  - Andy


From: Dru Jensen
Subject: Missing file



java.io.IOException: java.io.IOException: Cannot open
filename
/hbase/webmaps/743469791/header/mapfiles/4735713346568547172/data
        at
org.apache.hadoop.hdfs.DFSClient $DFSInputStream.openInfo(DFSClient.java:1395) 
        at
org.apache.hadoop.hdfs.DFSClient $DFSInputStream.chooseDataNode(DFSClient.java:1720) 
        at
org.apache.hadoop.hdfs.DFSClient $DFSInputStream.blockSeekTo(DFSClient.java:1536) 
        at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java: 1663) 
        at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java: 1593) 
        at
java.io.DataInputStream.readInt(DataInputStream.java:370)
        at
org.apache.hadoop.hbase.io.SequenceFile $Reader.readRecordLength(SequenceFile.java:1909) 
        at
org.apache.hadoop.hbase.io.SequenceFile $Reader.next(SequenceFile.java:1939) 
        at
org.apache.hadoop.hbase.io.SequenceFile $Reader.next(SequenceFile.java:1844) 
        at
org.apache.hadoop.hbase.io.MapFile$Reader.seekInternal(MapFile.java: 472) 
        at
org.apache.hadoop.hbase.io.MapFile$Reader.getClosest(MapFile.java: 567) 
        at
org.apache.hadoop.hbase.io.MapFile$Reader.getClosest(MapFile.java: 550) 
        at
org.apache.hadoop.hbase.io.BloomFilterMapFile $Reader.getClosest(BloomFilterMapFile.java:112) 
        at
org.apache.hadoop.hbase.regionserver.HStore.get(HStore.java:1485)
        at
org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:975)
        at
org .apache .hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1504) 
        at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
Source)
        at
sun .reflect .DelegatingMethodAccessorImpl .invoke(DelegatingMethodAccessorImpl.java:25) 
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
        at
org.apache.hadoop.hbase.ipc.HBaseServer $Handler.run(HBaseServer.java:894) 

The MR process needs to check to make sure the row is the
latest before inserting it so it calls htable.get(key) in
the mr process.  This is where the process fails.

I also tried performing a get 'table',
'key' in the shell with the same error.  I verified
that the file is missing in hadoop. It looks like HBase did
not update its reference after a split/region assignment.
Is anyone else experiencing this?

using 732094 from trunk.

Thanks for your help,
Dru

Reply via email to