@Nirmal, You need to run NameNode with upgrade option e.g. $ /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade
2013/12/3 Nirmal Kumar <[email protected]> > Hi All, > > > > I am doing a test migration from Apache Hadoop-1.2.0 to Apache > Hadoop-2.0.6-alpha on a single node environment. > > > > I did the following: > > · Installed Apache Hadoop-1.2.0 > > · Ran word count sample MR jobs. The jobs executed successfully. > > · I stop all the services in Apache Hadoop-1.2.0 and then was > able to start all services again. > > · The previous submitted jobs are visible after the stop/start in > the job tracker url. > > > > Next I installed Apache Hadoop-2.0.6-alpha alongside. > > I used the SAME data directory locations that were in Apache Hadoop-1.2.0 > in the configuration files namely: > > core-site.xml > > ---------------- > > $hadoop.tmp.dir > /home/cloud/hadoop_migration/hadoop-data/tempdir > > > > hdfs-site.xml > > ----------------- > > $dfs.data.dir > /home/cloud/hadoop_migration/hadoop-data/data > > $dfs.name.dir > /home/cloud/hadoop_migration/hadoop-data/name > > > > I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha > installation I am getting the error: > > > > 2013-12-03 18:28:23,941 INFO > org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from > hadoop-metrics2.properties > > 2013-12-03 18:28:24,080 INFO > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot > period at 10 second(s). > > 2013-12-03 18:28:24,081 INFO > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system > started > > 2013-12-03 18:28:24,576 WARN > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage > directory (dfs.namenode.name.dir) configured. Beware of dataloss due to > lack of redundant storage directories! > > 2013-12-03 18:28:24,576 WARN > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace > edits storage directory (dfs.namenode.edits.dir) configured. Beware of > dataloss due to lack of redundant storage directories! > > 2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader: > Refreshing hosts (include/exclude) list > > 2013-12-03 18:28:24,749 INFO > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: > dfs.block.invalidate.limit=1000 > > 2013-12-03 18:28:24,762 INFO > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: > dfs.block.access.token.enable=false > > 2013-12-03 18:28:24,762 INFO > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: > defaultReplication = 1 > > 2013-12-03 18:28:24,762 INFO > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: > maxReplication = 512 > > 2013-12-03 18:28:24,762 INFO > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: > minReplication = 1 > > 2013-12-03 18:28:24,763 INFO > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: > maxReplicationStreams = 2 > > 2013-12-03 18:28:24,763 INFO > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: > shouldCheckForEnoughRacks = false > > 2013-12-03 18:28:24,763 INFO > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: > replicationRecheckInterval = 3000 > > 2013-12-03 18:28:24,763 INFO > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: > encryptDataTransfer = false > > 2013-12-03 18:28:24,771 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = > cloud (auth:SIMPLE) > > 2013-12-03 18:28:24,771 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = > supergroup > > 2013-12-03 18:28:24,771 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = > true > > 2013-12-03 18:28:24,771 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false > > 2013-12-03 18:28:24,776 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true > > 2013-12-03 18:28:25,230 INFO > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names > occuring more than 10 times > > 2013-12-03 18:28:25,243 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: > dfs.namenode.safemode.threshold-pct = 0.9990000128746033 > > 2013-12-03 18:28:25,244 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: > dfs.namenode.safemode.min.datanodes = 0 > > 2013-12-03 18:28:25,244 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: > dfs.namenode.safemode.extension = 30000 > > 2013-12-03 18:28:25,288 INFO org.apache.hadoop.hdfs.server.common.Storage: > Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired > by nodename [email protected] > > 2013-12-03 18:28:25,462 INFO > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode > metrics system... > > 2013-12-03 18:28:25,462 INFO > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system > stopped. > > 2013-12-03 18:28:25,473 INFO > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system > shutdown complete. > > 2013-12-03 18:28:25,474 FATAL > org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join > > org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected > version of storage directory /home/cloud/hadoop_migration/hadoop-data/name. > Reported: -41. Expecting = -40. > > at > org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079) > > at > org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887) > > at > org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583) > > at > org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918) > > at > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304) > > at > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200) > > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627) > > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469) > > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403) > > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437) > > at > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609) > > at > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594) > > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169) > > at > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235) > > 2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting with > status 1 > > 2013-12-03 18:28:25,481 INFO > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: > > /************************************************************ > > SHUTDOWN_MSG: Shutting down NameNode at > Impetus-942.impetus.co.in/192.168.41.106 > > ************************************************************/ > > > > Independently both the installations(Apache Hadoop-1.2.0 and Apache > Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on > both the installations independently though. > > But I aim to migrate the data and jobs submitted from Apache Hadoop-1.2.0 > to Apache Hadoop-2.0.6-alpha. > > > > Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache > Hadoop-2.0.6-alpha? > > > > Thanks, > > -Nirmal > > > > *From:* Nirmal Kumar > *Sent:* Wednesday, November 27, 2013 2:56 PM > *To:* [email protected]; [email protected] > *Subject:* RE: Any reference for upgrade hadoop from 1.x to 2.2 > > > > Hello Sandy, > > > > The post was useful and gave an insight of the migration. > > > > I am doing a test migration from Apache Hadoop-1.2.0 to Apache > Hadoop-2.0.6-alpha on a single node environment. > > I am having the Apache Hadoop-1.2.0 up and running. > > > > Can you please let me know the steps that one should follow for the > migration? > > I am thinking of doing something like: > > · Install Apache Hadoop-2.0.6-alpha alongside the existing Apache > Hadoop-1.2.0 > > · Use the same HDFS locations > > · Change the various required configuration files > > · Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha > > · Verify all the services are running > > · Test via mapreduce (test MRv1 and MRv2 examples) > > · Check Web UI Console and verify the MRv1 and MRv2 jobs > > > > These above steps needs to be performed on all the nodes in a cluster > environment. > > > > The translation table mapping old configuration to new would be definitely > *very* useful. > > > > Also the existing Hadoop ecosystem components needs to be considered: > > · Hive Scripts > > · Pig Scripts > > · Oozie Workflows > > Their compatibility and version support would need to be checked. > > > > Also thinking of any risks like Data Loss, others that one should keep in > mind. > > > > Also I found: > http://strataconf.com/strata2014/public/schedule/detail/32247 > > > > Thanks, > > -Nirmal > > > > *From:* Robert Dyer [mailto:[email protected] <[email protected]>] > *Sent:* Friday, November 22, 2013 9:08 PM > *To:* [email protected] > *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2 > > > > Thanks Sandy! These seem helpful! > > > > "MapReduce cluster configuration options have been split into YARN > configuration options, which go in yarn-site.xml; and MapReduce > configuration options, which go in mapred-site.xml. Many have been given > new names to reflect the shift. ... *We’ll follow up with a full > translation table in a future post.*" > > > This type of translation table mapping old configuration to new would be > *very* useful! > > > > - Robert > > On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <[email protected]> > wrote: > > For MapReduce and YARN, we recently published a couple blog posts on > migrating: > > > http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/ > > > http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/ > > > > hope that helps, > > Sandy > > > > On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <[email protected]> > wrote: > > Hi All, > > > > I am also looking into migrating\upgrading from Apache Hadoop 1.x to > Apache Hadoop 2.x. > > I didn’t find any doc\guide\blogs for the same. > > Although there are guides\docs for the CDH and HDP migration\upgradation > from Hadoop 1.x to Hadoop 2.x > > Would referring those be of some use? > > > > I am looking for similar guides\docs for Apache Hadoop 1.x to Apache > Hadoop 2.x. > > > > I found something on slideshare though. Not sure how much useful that is > going to be. I still need to verify that. > > http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade > > > > Any suggestions\comments will be of great help. > > > > Thanks, > > -Nirmal > > > > *From:* Jilal Oussama [mailto:[email protected]] > *Sent:* Friday, November 08, 2013 9:13 PM > *To:* [email protected] > *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2 > > > > I am looking for the same thing if anyone can point us to a good direction > please. > > Thank you. > > (Currently running Hadoop 1.2.1) > > > > 2013/11/1 YouPeng Yang <[email protected]> > > Hi users > > Are there any reference docs to introduce how to upgrade hadoop from > 1.x to 2.2. > > > > Regards > > > > > ------------------------------ > > > > > > > > NOTE: This message may contain information that is confidential, > proprietary, privileged or otherwise protected by law. The message is > intended solely for the named addressee. If received in error, please > destroy and notify the sender. Any use of this email is prohibited when > received in error. Impetus does not represent, warrant and/or guarantee, > that the integrity of this communication has been maintained nor that the > communication is free of errors, virus, interception or interference. > > > ------------------------------ > > > > > > > > NOTE: This message may contain information that is confidential, > proprietary, privileged or otherwise protected by law. The message is > intended solely for the named addressee. If received in error, please > destroy and notify the sender. Any use of this email is prohibited when > received in error. Impetus does not represent, warrant and/or guarantee, > that the integrity of this communication has been maintained nor that the > communication is free of errors, virus, interception or interference. > > ------------------------------ > > > > > > > NOTE: This message may contain information that is confidential, > proprietary, privileged or otherwise protected by law. The message is > intended solely for the named addressee. If received in error, please > destroy and notify the sender. Any use of this email is prohibited when > received in error. Impetus does not represent, warrant and/or guarantee, > that the integrity of this communication has been maintained nor that the > communication is free of errors, virus, interception or interference. >
