Hi Jack, Do you have a rack topology script set up for HDFS?
-Todd On Fri, Jan 7, 2011 at 6:32 PM, Jack Levin <[email protected]> wrote: > Greetings all. I have been observing some interesting problems that > sometimes making hbase start/restart very hard to achieve. Here is a > situation: > > Power goes out of a rack, and kills some datanodes, and some regionservers. > > We power things back on, HDFS reports all datanodes back to normal, > and we cold restart hbase. > Obviously we have some log files in the /hbase/.logs directory on > HDFS. So, when master starts, it scans that dir and attempts to > replay the logs and insert all the data into the region files, so far > so good... > > Now at some instances, we get this message: > > hbase-root-master-rdag1.prod.imageshack.com.log.2011-01-02:2011-01-02 > 20:47:37,343 WARN org.apache.hadoop.hbase.util.FSUtils: Waited > 121173ms for lease recovery on > hdfs://namenode:9000/hbase/.logs/mtae6.prod.imageshack.com > ,60020,1293990443946/10.103.5.6 > %3A60020.1294005303076:org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: > failed to create file > /hbase/.logs/mtae6.prod.imageshack.com,60020,1293990443946/10.103.5.6 > %3A60020.1294005303076 > for DFSClient_hb_m_10.101.7.1:60000_1294029805305 on client > 10.101.7.1, because this file is already being created by NN_Recovery > on 10.101.7.1 > > Those messages (in master.log), will spew continuously and hbase will > not start. My understanding that namenode or maybe some datanode is > holding a lease on a file, and master is unable to process it. Left > by itself, the problem will not go away. The only way to resolve it, > is to shutdown the master, do > > hadoop fs -cp /hbase/.logs/* /tmp/.logs > hadoop fs -rm /hbase/.logs/* > hadoop fs -mv /tmp/.logs/* /hbase/.logs/ > > Start master, and things are back to normal (all logs replay, master > starts). > So, a question -- is there some sort of HDFS setting (are we hitting a > bug), to instruct the lease to be removed automatically? A timer > maybe? Can master be granted an authority maybe to copy a file into a > new name, and then replay it? It seems silly that master shouldn't be > able to do that, after all, its an hbase log file anyway. > > Next, there is this situation: > > 2011-01-02 20:56:58,219 WARN org.apache.hadoop.hdfs.DFSClient: Error > Recovery for block blk_-1736208949609845257_8228359 failed because > recovery from primary datanode 10.101.1.6:50010 failed 6 times. > Pipeline was 10.101.6.1:50010, 10.103.5.8:50010, 10.103.5.6:50010, > 10.101.1.6:50010. Marking primary datanode as bad. > > Here /hbase/.logs/log_name exists, but the data is missing completely. > It seems this empty file persists after hbase/hdfs crash. The only > solution is to perform the above (cp, rm, mv), or simply delete those > files by hand. Now, is it possible that master would do that? > Master should be able to detect invalid files in the .log/ dir and get > rid of them without operators interaction, is there is some sort of > design element that I am simply missing? > > Thanks. > > -Jack > -- Todd Lipcon Software Engineer, Cloudera
