@Nirmal, And later, you need to make a decision to finalize the upgrade or rollback.
2013/12/3 Adam Kawa <[email protected]> > @Nirmal, > > You need to run NameNode with upgrade option e.g. > $ /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade > > > 2013/12/3 Nirmal Kumar <[email protected]> > >> Hi All, >> >> >> >> I am doing a test migration from Apache Hadoop-1.2.0 to Apache >> Hadoop-2.0.6-alpha on a single node environment. >> >> >> >> I did the following: >> >> · Installed Apache Hadoop-1.2.0 >> >> · Ran word count sample MR jobs. The jobs executed successfully. >> >> · I stop all the services in Apache Hadoop-1.2.0 and then was >> able to start all services again. >> >> · The previous submitted jobs are visible after the stop/start >> in the job tracker url. >> >> >> >> Next I installed Apache Hadoop-2.0.6-alpha alongside. >> >> I used the SAME data directory locations that were in Apache Hadoop-1.2.0 >> in the configuration files namely: >> >> core-site.xml >> >> ---------------- >> >> $hadoop.tmp.dir >> /home/cloud/hadoop_migration/hadoop-data/tempdir >> >> >> >> hdfs-site.xml >> >> ----------------- >> >> $dfs.data.dir >> /home/cloud/hadoop_migration/hadoop-data/data >> >> $dfs.name.dir >> /home/cloud/hadoop_migration/hadoop-data/name >> >> >> >> I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha >> installation I am getting the error: >> >> >> >> 2013-12-03 18:28:23,941 INFO >> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from >> hadoop-metrics2.properties >> >> 2013-12-03 18:28:24,080 INFO >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot >> period at 10 second(s). >> >> 2013-12-03 18:28:24,081 INFO >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system >> started >> >> 2013-12-03 18:28:24,576 WARN >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage >> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to >> lack of redundant storage directories! >> >> 2013-12-03 18:28:24,576 WARN >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace >> edits storage directory (dfs.namenode.edits.dir) configured. Beware of >> dataloss due to lack of redundant storage directories! >> >> 2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader: >> Refreshing hosts (include/exclude) list >> >> 2013-12-03 18:28:24,749 INFO >> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: >> dfs.block.invalidate.limit=1000 >> >> 2013-12-03 18:28:24,762 INFO >> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: >> dfs.block.access.token.enable=false >> >> 2013-12-03 18:28:24,762 INFO >> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: >> defaultReplication = 1 >> >> 2013-12-03 18:28:24,762 INFO >> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: >> maxReplication = 512 >> >> 2013-12-03 18:28:24,762 INFO >> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: >> minReplication = 1 >> >> 2013-12-03 18:28:24,763 INFO >> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: >> maxReplicationStreams = 2 >> >> 2013-12-03 18:28:24,763 INFO >> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: >> shouldCheckForEnoughRacks = false >> >> 2013-12-03 18:28:24,763 INFO >> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: >> replicationRecheckInterval = 3000 >> >> 2013-12-03 18:28:24,763 INFO >> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: >> encryptDataTransfer = false >> >> 2013-12-03 18:28:24,771 INFO >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = >> cloud (auth:SIMPLE) >> >> 2013-12-03 18:28:24,771 INFO >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = >> supergroup >> >> 2013-12-03 18:28:24,771 INFO >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = >> true >> >> 2013-12-03 18:28:24,771 INFO >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false >> >> 2013-12-03 18:28:24,776 INFO >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true >> >> 2013-12-03 18:28:25,230 INFO >> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names >> occuring more than 10 times >> >> 2013-12-03 18:28:25,243 INFO >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: >> dfs.namenode.safemode.threshold-pct = 0.9990000128746033 >> >> 2013-12-03 18:28:25,244 INFO >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: >> dfs.namenode.safemode.min.datanodes = 0 >> >> 2013-12-03 18:28:25,244 INFO >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: >> dfs.namenode.safemode.extension = 30000 >> >> 2013-12-03 18:28:25,288 INFO >> org.apache.hadoop.hdfs.server.common.Storage: Lock on >> /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by >> nodename [email protected] >> >> 2013-12-03 18:28:25,462 INFO >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode >> metrics system... >> >> 2013-12-03 18:28:25,462 INFO >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system >> stopped. >> >> 2013-12-03 18:28:25,473 INFO >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system >> shutdown complete. >> >> 2013-12-03 18:28:25,474 FATAL >> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join >> >> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: >> Unexpected version of storage directory >> /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting = >> -40. >> >> at >> org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079) >> >> at >> org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887) >> >> at >> org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583) >> >> at >> org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918) >> >> at >> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304) >> >> at >> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200) >> >> at >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627) >> >> at >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469) >> >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403) >> >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437) >> >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609) >> >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594) >> >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169) >> >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235) >> >> 2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting >> with status 1 >> >> 2013-12-03 18:28:25,481 INFO >> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: >> >> /************************************************************ >> >> SHUTDOWN_MSG: Shutting down NameNode at >> Impetus-942.impetus.co.in/192.168.41.106 >> >> ************************************************************/ >> >> >> >> Independently both the installations(Apache Hadoop-1.2.0 and Apache >> Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on >> both the installations independently though. >> >> But I aim to migrate the data and jobs submitted from Apache >> Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha. >> >> >> >> Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache >> Hadoop-2.0.6-alpha? >> >> >> >> Thanks, >> >> -Nirmal >> >> >> >> *From:* Nirmal Kumar >> *Sent:* Wednesday, November 27, 2013 2:56 PM >> *To:* [email protected]; [email protected] >> *Subject:* RE: Any reference for upgrade hadoop from 1.x to 2.2 >> >> >> >> Hello Sandy, >> >> >> >> The post was useful and gave an insight of the migration. >> >> >> >> I am doing a test migration from Apache Hadoop-1.2.0 to Apache >> Hadoop-2.0.6-alpha on a single node environment. >> >> I am having the Apache Hadoop-1.2.0 up and running. >> >> >> >> Can you please let me know the steps that one should follow for the >> migration? >> >> I am thinking of doing something like: >> >> · Install Apache Hadoop-2.0.6-alpha alongside the existing >> Apache Hadoop-1.2.0 >> >> · Use the same HDFS locations >> >> · Change the various required configuration files >> >> · Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha >> >> · Verify all the services are running >> >> · Test via mapreduce (test MRv1 and MRv2 examples) >> >> · Check Web UI Console and verify the MRv1 and MRv2 jobs >> >> >> >> These above steps needs to be performed on all the nodes in a cluster >> environment. >> >> >> >> The translation table mapping old configuration to new would be >> definitely *very* useful. >> >> >> >> Also the existing Hadoop ecosystem components needs to be considered: >> >> · Hive Scripts >> >> · Pig Scripts >> >> · Oozie Workflows >> >> Their compatibility and version support would need to be checked. >> >> >> >> Also thinking of any risks like Data Loss, others that one should keep in >> mind. >> >> >> >> Also I found: >> http://strataconf.com/strata2014/public/schedule/detail/32247 >> >> >> >> Thanks, >> >> -Nirmal >> >> >> >> *From:* Robert Dyer [mailto:[email protected] <[email protected]>] >> *Sent:* Friday, November 22, 2013 9:08 PM >> *To:* [email protected] >> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2 >> >> >> >> Thanks Sandy! These seem helpful! >> >> >> >> "MapReduce cluster configuration options have been split into YARN >> configuration options, which go in yarn-site.xml; and MapReduce >> configuration options, which go in mapred-site.xml. Many have been given >> new names to reflect the shift. ... *We’ll follow up with a full >> translation table in a future post.*" >> >> >> This type of translation table mapping old configuration to new would be >> *very* useful! >> >> >> >> - Robert >> >> On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <[email protected]> >> wrote: >> >> For MapReduce and YARN, we recently published a couple blog posts on >> migrating: >> >> >> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/ >> >> >> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/ >> >> >> >> hope that helps, >> >> Sandy >> >> >> >> On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <[email protected]> >> wrote: >> >> Hi All, >> >> >> >> I am also looking into migrating\upgrading from Apache Hadoop 1.x to >> Apache Hadoop 2.x. >> >> I didn’t find any doc\guide\blogs for the same. >> >> Although there are guides\docs for the CDH and HDP migration\upgradation >> from Hadoop 1.x to Hadoop 2.x >> >> Would referring those be of some use? >> >> >> >> I am looking for similar guides\docs for Apache Hadoop 1.x to Apache >> Hadoop 2.x. >> >> >> >> I found something on slideshare though. Not sure how much useful that is >> going to be. I still need to verify that. >> >> http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade >> >> >> >> Any suggestions\comments will be of great help. >> >> >> >> Thanks, >> >> -Nirmal >> >> >> >> *From:* Jilal Oussama [mailto:[email protected]] >> *Sent:* Friday, November 08, 2013 9:13 PM >> *To:* [email protected] >> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2 >> >> >> >> I am looking for the same thing if anyone can point us to a good >> direction please. >> >> Thank you. >> >> (Currently running Hadoop 1.2.1) >> >> >> >> 2013/11/1 YouPeng Yang <[email protected]> >> >> Hi users >> >> Are there any reference docs to introduce how to upgrade hadoop from >> 1.x to 2.2. >> >> >> >> Regards >> >> >> >> >> ------------------------------ >> >> >> >> >> >> >> >> NOTE: This message may contain information that is confidential, >> proprietary, privileged or otherwise protected by law. The message is >> intended solely for the named addressee. If received in error, please >> destroy and notify the sender. Any use of this email is prohibited when >> received in error. Impetus does not represent, warrant and/or guarantee, >> that the integrity of this communication has been maintained nor that the >> communication is free of errors, virus, interception or interference. >> >> >> ------------------------------ >> >> >> >> >> >> >> >> NOTE: This message may contain information that is confidential, >> proprietary, privileged or otherwise protected by law. The message is >> intended solely for the named addressee. If received in error, please >> destroy and notify the sender. Any use of this email is prohibited when >> received in error. Impetus does not represent, warrant and/or guarantee, >> that the integrity of this communication has been maintained nor that the >> communication is free of errors, virus, interception or interference. >> >> ------------------------------ >> >> >> >> >> >> >> NOTE: This message may contain information that is confidential, >> proprietary, privileged or otherwise protected by law. The message is >> intended solely for the named addressee. If received in error, please >> destroy and notify the sender. Any use of this email is prohibited when >> received in error. Impetus does not represent, warrant and/or guarantee, >> that the integrity of this communication has been maintained nor that the >> communication is free of errors, virus, interception or interference. >> > >
