What version of Hadoop / HBase are you using ? Cheers
On Sat, Dec 7, 2013 at 9:38 AM, iain wright <[email protected]> wrote: > Hi folks, > > One of our RS/DN/TT nodes went down dirty (kernel panic). Users contacted > about reports failing, and i saw wierd logs in the jobtracker for "pending > shutdown" etc. > > Proceeded to stop jobtracker/tt nodes, hbase, and hdfs. > > On attempting to turn back on the NN + DN's our namenode is failing to > start, complaining about being unable to move one of our hbase region files > because source file does exist. > > Has anyone come across this/have suggestions?? > > 2013-12-07 01:20:43,156 INFO org.apache.hadoop.hdfs.server.common.Storage: > Image file of size 149098688 loaded in 14 seconds. > 2013-12-07 01:20:43,253 WARN org.apache.hadoop.hdfs.StateChange: DIR* > FSDirectory.unprotectedRenameTo: failed to rename > > /hbase_prod/prod_transactions/d3fdbd9b9883b2f8efe6d0f8b7790f41/c/df40b2db7d9345ab9384a77a2c573de0 > to > > /hbase_prod/.archive/prod_transactions/d3fdbd9b9883b2f8efe6d0f8b7790f41/c/df40b2db7d9345ab9384a77a2c573de0 > because source does not exist > 2013-12-07 01:20:43,348 ERROR > org.apache.hadoop.hdfs.server.namenode.NameNode: > java.lang.NullPointerException > at > > org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedSetTimes(FSDirectory.java:1343) > at > > org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedSetTimes(FSDirectory.java:1332) > at > > org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:774) > at > > org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:1008) > at > > org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:826) > at > > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:362) > at > > org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:97) > at > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:386) > at > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:360) > at > > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:276) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:496) > at > > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279) > at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1288) > > 2013-12-07 01:20:43,349 INFO > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: > /************************************************************ > SHUTDOWN_MSG: Shutting down NameNode at > namenode0.hadoop.compe.tv/10.10.11.10 > > Thank you, > > -- > iain > > <http://www.labctsi.org/> > This email message is confidential, intended only for the recipient(s) > named above and may contain information that is privileged, exempt from > disclosure under applicable law. If you are not the intended recipient, do > not disclose or disseminate the message to anyone except the intended > recipient. If you have received this message in error, or are not the named > recipient(s), please immediately notify the sender by return email, and > delete all copies of this message. >
