"this file is already being created by NN_Recovery" error

[Agarwal, Saurabh]

Hi,

I am getting the following error and hbase shell is hanging and we are not able 
to connect hbase. Hadoop filesystem is fine and map reduce jobs are running ok.

Here is the warning message we are getting in the master log file.
for lease recovery on 
hdfs://<servername>:54310/user/root/hbase/.logs/<servername>,60020,1329596937708/<servername>%3A60020.1329607742925:org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException:
 failed to create file 
/user/root/hbase/.logs/<>,60020,1329596937708/<servername>%3A60020.1329607742925
 for DFSClient_hb_m_:60000_1329688357813 on client <>, because this file is 
already being created by NN_Recovery on <>
        at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:1202)
        at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:1157)
        at 
org.apache.hadoop.hdfs.server.namenode.NameNode.recoverLease(NameNode.java:404)
        at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:961)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:957)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:955)

Any help is appreciated,
Thanks,
Saurabh.
-----Original Message-----
From: NNever [mailto:nnever...@gmail.com]
Sent: Monday, February 20, 2012 9:04 AM
To: user@hbase.apache.org
Subject: HBase0.92: In Coprocessor's prePut method, muilt put on same HTable 
cause HBase crash

Hi~

On HBase0.92.

I write a coprosessor on table 'File'.In the prePut method,  I put several
new rows on File HTable instance. As u know , this op will trigger prePut
again.
I use this logic to realize something like "copy a File and auto copy its
all level's subfiles" .
The code seems no wrong, but everytime i trigger this coprocessor, the
HBase stuck and couldn't shutdown.
(through the log I found the coprocssor's code fall into dead-loop, but the
prePut's code don't have any place may cause dead-loop)

I try lot of times and finally get the reason: I cache the File's HTable
instance and use it like this

*private HTableInterface htFile = null;*
*...*
*@Override*
*public void prePut(...){*

*...*
*if(null == htFile){*
*   htFile = e.getEnvironment().getTable(Bytes.toBytes("File"));*
*}*
*...*
*htFile.put(some list of new puts);*
*...*

*}*

if I change the code and get HTable's instance everytime, the problem fixed.
But I want to know why* using the same Instance of HTable to put puts in
same prePut method may cause coprocessor dead-loop*?
(it just trigger prePut with same put again and again util the HBase
stuck...)

Thanks~