Hi folks, One of our RS/DN/TT nodes went down dirty (kernel panic). Users contacted about reports failing, and i saw wierd logs in the jobtracker for "pending shutdown" etc.
Proceeded to stop jobtracker/tt nodes, hbase, and hdfs. On attempting to turn back on the NN + DN's our namenode is failing to start, complaining about being unable to move one of our hbase region files because source file does exist. Has anyone come across this/have suggestions?? 2013-12-07 01:20:43,156 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 149098688 loaded in 14 seconds. 2013-12-07 01:20:43,253 WARN org.apache.hadoop.hdfs.StateChange: DIR* FSDirectory.unprotectedRenameTo: failed to rename /hbase_prod/prod_transactions/d3fdbd9b9883b2f8efe6d0f8b7790f41/c/df40b2db7d9345ab9384a77a2c573de0 to /hbase_prod/.archive/prod_transactions/d3fdbd9b9883b2f8efe6d0f8b7790f41/c/df40b2db7d9345ab9384a77a2c573de0 because source does not exist 2013-12-07 01:20:43,348 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.lang.NullPointerException at org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedSetTimes(FSDirectory.java:1343) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedSetTimes(FSDirectory.java:1332) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:774) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:1008) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:826) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:362) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:97) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:386) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:360) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:276) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:496) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1288) 2013-12-07 01:20:43,349 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at namenode0.hadoop.compe.tv/10.10.11.10 Thank you, -- iain <http://www.labctsi.org/> This email message is confidential, intended only for the recipient(s) named above and may contain information that is privileged, exempt from disclosure under applicable law. If you are not the intended recipient, do not disclose or disseminate the message to anyone except the intended recipient. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender by return email, and delete all copies of this message.
