Hi Experts, According to section 'Upgrading Non-Federated Clusters' of http://hadoop.apache.org/docs/r2.4.0/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html, I tried to upgrade hadoop 2.2.0 to hadoop 2.4.1. However, I failed on step 2.2 'Start NN2 as standby with the "-rollingUpgrade started" option.'. The exception is as below. Any comments from you will be appreciated!
14/09/01 11:54:22 INFO namenode.FSImage: Reading org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream@cb7c0ebc expect ing start txid #3353 14/09/01 11:54:22 INFO namenode.FSImage: Start loading edits file http://host74-public:8480/getJournal?jid=BICluster&s egmentTxId=3353&storageInfo=-56%3A1518542315%3A0%3ACID-9513e950-061b-4e96-924b-3faab87520d5, http://host71-public:8480 /getJournal?jid=BICluster&segmentTxId=3353&storageInfo=-56%3A1518542315%3A0%3ACID-9513e950-061b-4e96-924b-3faab87520d5 14/09/01 11:54:22 INFO namenode.EditLogInputStream: Fast-forwarding stream ' http://host74-public:8480/getJournal?jid=B ICluster&segmentTxId=3353&storageInfo=-56%3A1518542315%3A0%3ACID-9513e950-061b-4e96-924b-3faab87520d5, http://host71-public.cn.ib m.com:8480/getJournal?jid=BICluster&segmentTxId=3353&storageInfo=-56%3A1518542315%3A0%3ACID-9513e950-061b-4e96-924b-3faab87520d5' t o transaction ID 3353 14/09/01 11:54:22 INFO namenode.EditLogInputStream: Fast-forwarding stream ' http://host74-public:8480/getJournal?jid=B ICluster&segmentTxId=3353&storageInfo=-56%3A1518542315%3A0%3ACID-9513e950-061b-4e96-924b-3faab87520d5' to transaction ID 3353 14/09/01 11:54:22 ERROR namenode.EditLogInputStream: caught exception initializing http://host74-public:8480/getJourna l?jid=BICluster&segmentTxId=3353&storageInfo=-56%3A1518542315%3A0%3ACID-9513e950-061b-4e96-924b-3faab87520d5 org.apache.hadoop.hdfs.server.namenode.TransferFsImage$HttpGetFailedException: Fetch of http://host74-public:8480/getJ ournal?jid=BICluster&segmentTxId=3353&storageInfo=-56%3A1518542315%3A0%3ACID-9513e950-061b-4e96-924b-3faab87520d5 failed with statu s code 403 Response message: This node has storage info '-47:1518542315:0:CID-9513e950-061b-4e96-924b-3faab87520d5' but the requesting node expected '-56:151854 2315:0:CID-9513e950-061b-4e96-924b-3faab87520d5' at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:472) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:460) at java.security.AccessController.doPrivileged(AccessController.java:366) at javax.security.auth.Subject.doAs(Subject.java:572) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1558) at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:448) at org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(SecurityUtil.java:442) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java:459) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:145) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:196) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:254) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:180) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:133) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:805) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:665) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:272) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:891) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:638) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:503) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:559) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:724) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:708) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1358) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1424) 14/09/01 11:54:22 ERROR namenode.EditLogInputStream: Got error reading edit log input stream http://host74-public:8480 /getJournal?jid=BICluster&segmentTxId=3353&storageInfo=-56%3A1518542315%3A0%3ACID-9513e950-061b-4e96-924b-3faab87520d5; failing ove r to edit log http://host71-public:8480/getJournal?jid=BICluster&segmentTxId=3353&storageInfo=-56%3A1518542315%3A0%3AC ID-9513e950-061b-4e96-924b-3faab87520d5 org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 3352; e xpected file to go up to 3354 at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:180) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:133) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:805) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:665) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:272) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:891) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:638) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:503) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:559) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:724) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:708) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1358) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1424) ... ... 14/09/01 11:54:32 INFO mortbay.log: Stopped SelectChannelConnector@host73-public:50070 14/09/01 11:54:32 INFO impl.MetricsSystemImpl: Stopping NameNode metrics system... 14/09/01 11:54:32 INFO impl.MetricsSystemImpl: NameNode metrics system stopped. 14/09/01 11:54:32 INFO impl.MetricsSystemImpl: NameNode metrics system shutdown complete. 14/09/01 11:54:32 FATAL namenode.NameNode: Exception in namenode join org.apache.hadoop.hdfs.server.namenode.EditLogInputException: Error replaying edit log at offset 0. Expected transaction ID was 33 53 at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:193) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:133) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:805) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:665) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:272) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:891) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:638) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:503) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:559) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:724) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:708) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1358) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1424) Caused by: org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at t xid 3352; expected file to go up to 3354 at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:180) ... 12 more 14/09/01 11:54:32 INFO util.ExitUtil: Exiting with status 1 14/09/01 11:54:32 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at host73-public/9.11.64.232 ************************************************************/
