Re: namenode errors

[Brian Bockelman]

Hey Zeev,






Find the client, kill it, and you will see these messages go away

Brian

On Sep 15, 2009, at 5:52 PM, Zeev Milin wrote:

This is caused by a misbehaving client stuck in an infinite loop. When you restart the NN, the client resumes causing these errors. I suspect there is an infinite loop in the DFSClient for HDFS 0.19.x as you are the third person to report this. If you can recover a stack trace from the misbehaving client, it would be very interesting to C - I trigger this often, but I access via libhdfs, so getting a stack trace is very difficult.

I see lots of errors (below) in the namenode.log file. Fsck is clean and restart did not help. Hadoop version is 0.19.1. If anyone has run into this

and resolved it already please let me know.

Thanks!


2009-09-15 18:13:58,783 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
commitBlockSynchronization(lastblock=blk_-1745012098763993903_2457112,
newgenerationstamp=0, newlength=0, newtargets=[])
2009-09-15 18:13:58,783 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 18 on 8080, call

true, [Lorg.apache.hadoop.hdfs.protocol.DatanodeID;@6fa0b772) from
10.60.32.172:35019: error: java.io.IOException: Block
(=blk_-1745012098763993903_2457112) not found

   at

   at

   at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source)
   at

   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:892)

commitBlockSynchronization(blk_-1745012098763993903_2457112, 0, 0, false, java.io.IOException: Block (=blk_-1745012098763993903_2457112) not found org.apache.hadoop.hdfs.server.namenode.FSNamesystem.commitBlockSynchronization (FSNamesystem.java:1898) org.apache.hadoop.hdfs.server.namenode.NameNode.commitBlockSynchronization (NameNode.java:410) sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:25)

smime.p7s
Description: S/MIME cryptographic signature