I ran into the same issue when using the default settings for dfs.data.dir, which is under the /tmp directory. Files in this directory will be cleaned our periodically as needed by the OS, which will break HDFS.
On Thu, Jul 2, 2009 at 7:01 AM, Gross, Danny <[email protected]>wrote: > Hello Johnson, > > I have observed similar error messages when my system ran out of disk > space on an HDFS node, or in hadoop.tmp.dir. > > Hope it helps. > > Best regards, > > Danny > > -----Original Message----- > From: Johnson Chen [mailto:[email protected]] > Sent: Thursday, July 02, 2009 3:47 AM > To: [email protected] > Subject: Block not found > > Hi , > > My hadoop program stop at 66% Reduce phrase > > 09/07/02 16:41:37 INFO mapred.JobClient: map 0% reduce 0% > 09/07/02 16:41:48 INFO mapred.JobClient: map 50% reduce 0% > 09/07/02 16:41:51 INFO mapred.JobClient: map 100% reduce 0% > 09/07/02 16:41:58 INFO mapred.JobClient: map 100% reduce 66% > > And I found it pop up a lot of error message in namenode log . > > here is the error messages : > > 2009-07-02 16:43:44,461 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: > commitBlockSynchronization(lastblock=blk_-7800669485846603924_144754, > newgenerationstamp=0, newlength=0, newtargets=[], closeFile=false, > deleteBlock=true) > 2009-07-02 16:43:44,461 INFO org.apache.hadoop.ipc.Server: IPC Server > handler 3 on 50040, call > commitBlockSynchronization(blk_-7800669485846603924_144754, 0, 0, false, > true, [Lorg.apache.hadoop.hdfs.protocol.DatanodeID;@2df2888) from > 192.168.151.231:39976: error: java.io.IOException: Block > (=blk_-7800669485846603924_144754) not found > java.io.IOException: Block (=blk_-7800669485846603924_144754) not found > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.commitBlockSynchroni > zation(FSNamesystem.java:1906) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.commitBlockSynchronizati > on(NameNode.java:410) > at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor > Impl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894) > 2009-07-02 16:43:44,967 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: > commitBlockSynchronization(lastblock=blk_-8293900342000823254_748370, > newgenerationstamp=0, newlength=0, newtargets=[], closeFile=false, > deleteBlock=true) > 2009-07-02 16:43:44,967 INFO org.apache.hadoop.ipc.Server: IPC Server > handler 0 on 50040, call > commitBlockSynchronization(blk_-8293900342000823254_748370, 0, 0, false, > true, [Lorg.apache.hadoop.hdfs.protocol.DatanodeID;@8ddfa31) from > 192.168.151.232:45283: error: java.io.IOException: Block > (=blk_-8293900342000823254_748370) not found > java.io.IOException: Block (=blk_-8293900342000823254_748370) not found > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.commitBlockSynchroni > zation(FSNamesystem.java:1906) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.commitBlockSynchronizati > on(NameNode.java:410) > at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor > Impl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894) > > > Can anyone help me ? Thanks. > > -- > Best wishes, > Johnson Chen >
