On Wed, Sep 8, 2010 at 6:00 PM, Matthew LeMieux <[email protected]> wrote: > 2010-09-09 00:54:58,406 WARN org.apache.hadoop.hbase.util.FSUtils: Waited > 69188ms for lease recovery on > hdfs://domU-12-31-39-18-12-05.compute-1.internal:9000/hbase/.logs/domU-12-31-39-0C-38-31.compute-1.internal,60020,1283905848540/10.215.59.191%3A60020.1283905909298:org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: > failed to create file > /hbase/.logs/domU-12-31-39-0C-38-31.compute-1.internal,60020,1283905848540/10.215.59.191%3A60020.1283905909298 > for DFSClient_hb_m_10.104.37.247:60000 on client 10.104.37.247 because > current leaseholder is trying to recreate file. >
This is the master trying to take over the lease on a file that a regionserver had open so it can split its logs. It never succeeds? If you have CDH3b2 and the below noted version of hbase, dfs append should be on by default. Was 10.104.37.247 down for sure? What if you grep that file in namenode log, whats it show? > > 2010-09-09 00:53:49,111 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Telling master at > 10.104.37.247:60000 that we are up > They are wating for the master to come up. Its busy splitting files (or stuck splitting files as per above). > I've been using this version for a little under a week without incident > (http://people.apache.org/~jdcryans/hbase-0.89.20100830-candidate-1/ ). > New candidate coming out soon. Keep an eye out for it. It fixes bugs though nothing in the area you are currently suffering. St.Ack > The HDFS comes from CDH3. > > Does anybody have any ideas on what I can do to get back up and running? > > Thank you, > > Matthew > >
