Thanks Brian. Killing an orphan process on one of the nodes resolved the issue (I did not capture a stack trace)
Zeev On Wed, Sep 16, 2009 at 5:47 AM, Brian Bockelman <[email protected]>wrote: > Hey Zeev, > > This is caused by a misbehaving client stuck in an infinite loop. When you > restart the NN, the client resumes causing these errors. > > I suspect there is an infinite loop in the DFSClient for HDFS 0.19.x as you > are the third person to report this. If you can recover a stack trace from > the misbehaving client, it would be very interesting to C - I trigger this > often, but I access via libhdfs, so getting a stack trace is very difficult. > > Find the client, kill it, and you will see these messages go away > > Brian > > > On Sep 15, 2009, at 5:52 PM, Zeev Milin wrote: > > I see lots of errors (below) in the namenode.log file. Fsck is clean and >> restart did not help. Hadoop version is 0.19.1. If anyone has run into >> this >> and resolved it already please let me know. >> >> Thanks! >> >> >> 2009-09-15 18:13:58,783 INFO >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: >> commitBlockSynchronization(lastblock=blk_-1745012098763993903_2457112, >> newgenerationstamp=0, newlength=0, newtargets=[]) >> 2009-09-15 18:13:58,783 INFO org.apache.hadoop.ipc.Server: IPC Server >> handler 18 on 8080, call >> commitBlockSynchronization(blk_-1745012098763993903_2457112, 0, 0, false, >> true, [Lorg.apache.hadoop.hdfs.protocol.DatanodeID;@6fa0b772) from >> 10.60.32.172:35019: error: java.io.IOException: Block >> (=blk_-1745012098763993903_2457112) not found >> java.io.IOException: Block (=blk_-1745012098763993903_2457112) not found >> at >> >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.commitBlockSynchronization(FSNamesystem.java:1898) >> at >> >> org.apache.hadoop.hdfs.server.namenode.NameNode.commitBlockSynchronization(NameNode.java:410) >> at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source) >> at >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452) >> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:892) >> > >
