Alejandro Fernandez created AMBARI-13755:
--------------------------------------------
Summary: Express Upgrade: Downgrade fails at Namenode restart
Key: AMBARI-13755
URL: https://issues.apache.org/jira/browse/AMBARI-13755
Project: Ambari
Issue Type: Bug
Components: ambari-server
Affects Versions: 2.2.0, 2.1.3
Reporter: Alejandro Fernandez
Assignee: Alejandro Fernandez
Fix For: 2.2.0, 2.1.3
Setup:
# Express Upgrade on HDP 2.2.6.0 -> HDP 2.3.0.0
# Services: HDFS + ZK
# Upgrade completes successfully
# Downgrade after reaching Finalize Upgrade
# Downgrade blocked at Namenode restart
Logs:
{noformat}
2015-10-30 01:27:29,563 WARN namenode.FSNamesystem
(FSNamesystem.java:loadFromDisk(750)) - Encountered exception loading fsimage
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
version of storage directory /hadoop/hdfs/namenode. Reported: -63. Expecting =
-60.
at
org.apache.hadoop.hdfs.server.common.StorageInfo.setLayoutVersion(StorageInfo.java:178)
at
org.apache.hadoop.hdfs.server.common.StorageInfo.setFieldsFromProperties(StorageInfo.java:131)
at
org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:610)
at
org.apache.hadoop.hdfs.server.namenode.NNStorage.readProperties(NNStorage.java:639)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:326)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:202)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1032)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:748)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:538)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:597)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:764)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:748)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1441)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1507)
2015-10-30 01:27:29,585 INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped
HttpServer2$SelectChannelConnectorWithSafeStartup@jay-cfg-1.c.pramod-thangali.internal:50070
2015-10-30 01:27:29,587 WARN http.HttpServer2
(HttpServer2.java:isRunning(460)) - HttpServer Acceptor: isRunning is false.
Rechecking.
2015-10-30 01:27:29,587 WARN http.HttpServer2
(HttpServer2.java:isRunning(469)) - HttpServer Acceptor: isRunning is false
2015-10-30 01:27:29,595 INFO impl.MetricsSystemImpl
(MetricsSystemImpl.java:stop(210)) - Stopping NameNode metrics system...
2015-10-30 01:27:29,596 INFO impl.MetricsSystemImpl
(MetricsSystemImpl.java:stop(216)) - NameNode metrics system stopped.
2015-10-30 01:27:29,596 INFO impl.MetricsSystemImpl
(MetricsSystemImpl.java:shutdown(605)) - NameNode metrics system shutdown
complete.
2015-10-30 01:27:29,596 FATAL namenode.NameNode (NameNode.java:main(1512)) -
Failed to start namenode.
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
version of storage directory /hadoop/hdfs/namenode. Reported: -63. Expecting =
-60.
at
org.apache.hadoop.hdfs.server.common.StorageInfo.setLayoutVersion(StorageInfo.java:178)
at
org.apache.hadoop.hdfs.server.common.StorageInfo.setFieldsFromProperties(StorageInfo.java:131)
at
org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:610)
at
org.apache.hadoop.hdfs.server.namenode.NNStorage.readProperties(NNStorage.java:639)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:326)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:202)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1032)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:748)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:538)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:597)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:764)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:748)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1441)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1507)
2015-10-30 01:27:29,600 INFO util.ExitUtil (ExitUtil.java:terminate(124)) -
Exiting with status 1
2015-10-30 01:27:29,605 INFO namenode.NameNode (StringUtils.java:run(659)) -
SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at
jay-cfg-1.c.pramod-thangali.internal/10.240.2.108
************************************************************/
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
