Can you attach initialization part of NameNode?

thanks,
Raghu.

C G wrote:
We've recently upgraded from 0.15.0 to 0.16.4.  Two nights ago we had a problem 
where DFS nodes could not communicate. After not finding anything obviously 
wrong we decided to shut down DFS and restart.  Following restart I was seeing 
a corrupted system with significant amounts of missing data.  Further checking 
showed that DataNodes on all slaves did not start due to what looks like a 
version skew issue.
Our distribution is a straight 0.16.4 dist, so I'm having difficulty understanding what's causing this issue. Note that we haven't finalized the upgrade yet. Any help understanding this problem would be very much appreciated. We have several TB of data in our system and reloading from scratch would be a big problem. Here is the log from one of the failed nodes: /************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG:   host = worker9/10.2.0.9
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 0.16.4
STARTUP_MSG:   build = 
http://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.16 -r 652614; 
compiled by 'hadoopqa' on Fri May  2 00:18:12 UTC 2008
************************************************************/
2008-05-23 08:10:47,196 FATAL org.apache.hadoop.dfs.DataNode: Incompatible 
build versions: namenode BV = 2182; datanode BV = 652614
2008-05-23 08:10:47,202 ERROR org.apache.hadoop.dfs.DataNode: 
java.io.IOException: Incompatible build versions: namenode BV = 2182; datanode 
BV = 652614
        at org.apache.hadoop.dfs.DataNode.handshake(DataNode.java:342)
        at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:213)
        at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:162)
        at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:2512)
        at org.apache.hadoop.dfs.DataNode.run(DataNode.java:2456)
        at org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:2477)
        at org.apache.hadoop.dfs.DataNode.main(DataNode.java:2673)
  2008-05-23 08:10:47,203 INFO org.apache.hadoop.dfs.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at worker9/10.2.0.9
************************************************************/


Reply via email to