When i run namenode with upgrade option i get the following error and and namenode dosent start... 2013-12-24 14:48:38,595 INFO org.apache.hadoop.hdfs.StateChange: STATE* Network topology has 0 racks and 0 datanodes 2013-12-24 14:48:38,595 INFO org.apache.hadoop.hdfs.StateChange: STATE* UnderReplicatedBlocks has 0 blocks 2013-12-24 14:48:38,631 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2013-12-24 14:48:38,632 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9000: starting 2013-12-24 14:48:38,633 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode RPC up at: 192.168.1.2/192.168.1.2:9000 2013-12-24 14:48:38,633 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services required for active state 2013-12-24 14:50:50,060 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: RECEIVED SIGNAL 15: SIGTERM 2013-12-24 14:50:50,062 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at localhost.localdomain/127.0.0.1 ************************************************************/
On 12/24/13, Sitaraman Vilayannur <[email protected]> wrote: > Found it, > I get the following error on starting namenode in 2.2 > 10/contrib/capacity-scheduler/*.jar:/usr/local/Software/hadoop-0.23.10/contrib/capacity-scheduler/*.jar:/usr/local/Software/hadoop-0.23.10/contrib/capacity-scheduler/*.jar > STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common > -r 1529768; compiled by 'hortonmu' on 2013-10-07T06:28Z > STARTUP_MSG: java = 1.7.0_45 > ************************************************************/ > 2013-12-24 13:25:48,876 INFO > org.apache.hadoop.hdfs.server.namenode.NameNode: registered UNIX > signal handlers for [TERM, HUP, INT] > 2013-12-24 13:25:49,042 INFO > org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from > hadoop-metrics2.properties > 2013-12-24 13:25:49,102 INFO > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot > period at 10 second(s). > 2013-12-24 13:25:49,102 INFO > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics > system started > 2013-12-24 13:25:49,232 WARN org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using > builtin-java classes where applicable > 2013-12-24 13:25:49,375 INFO org.mortbay.log: Logging to > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via > org.mortbay.log.Slf4jLog > 2013-12-24 13:25:49,410 INFO org.apache.hadoop.http.HttpServer: Added > global filter 'safety' > (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) > 2013-12-24 13:25:49,412 INFO org.apache.hadoop.http.HttpServer: Added > filter static_user_filter > (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) > to context hdfs > 2013-12-24 13:25:49,412 INFO org.apache.hadoop.http.HttpServer: Added > filter static_user_filter > (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) > to context static > 2013-12-24 13:25:49,412 INFO org.apache.hadoop.http.HttpServer: Added > filter static_user_filter > (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) > to context logs > 2013-12-24 13:25:49,422 INFO org.apache.hadoop.http.HttpServer: > dfs.webhdfs.enabled = false > 2013-12-24 13:25:49,432 INFO org.apache.hadoop.http.HttpServer: Jetty > bound to port 50070 > 2013-12-24 13:25:49,432 INFO org.mortbay.log: jetty-6.1.26 > 2013-12-24 13:25:49,459 WARN org.mortbay.log: Can't reuse > /tmp/Jetty_0_0_0_0_50070_hdfs____w2cu08, using > /tmp/Jetty_0_0_0_0_50070_hdfs____w2cu08_2787234685293301311 > 2013-12-24 13:25:49,610 INFO org.mortbay.log: Started > [email protected]:50070 > 2013-12-24 13:25:49,611 INFO > org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: > 0.0.0.0:50070 > 2013-12-24 13:25:49,628 WARN > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image > storage directory (dfs.namenode.name.dir) configured. Beware of > dataloss due to lack of redundant storage directories! > 2013-12-24 13:25:49,628 WARN > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one > namespace edits storage directory (dfs.namenode.edits.dir) configured. > Beware of dataloss due to lack of redundant storage directories! > 2013-12-24 13:25:49,668 INFO > org.apache.hadoop.hdfs.server.namenode.HostFileManager: read includes: > HostSet( > ) > 2013-12-24 13:25:49,669 INFO > org.apache.hadoop.hdfs.server.namenode.HostFileManager: read excludes: > HostSet( > ) > 2013-12-24 13:25:49,670 INFO > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: > dfs.block.invalidate.limit=1000 > 2013-12-24 13:25:49,672 INFO org.apache.hadoop.util.GSet: Computing > capacity for map BlocksMap > 2013-12-24 13:25:49,672 INFO org.apache.hadoop.util.GSet: VM type = > 64-bit > 2013-12-24 13:25:49,673 INFO org.apache.hadoop.util.GSet: 2.0% max > memory = 889 MB > 2013-12-24 13:25:49,673 INFO org.apache.hadoop.util.GSet: capacity > = 2^21 = 2097152 entries > 2013-12-24 13:25:49,677 INFO > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: > dfs.block.access.token.enable=false > 2013-12-24 13:25:49,677 INFO > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: > defaultReplication = 1 > 2013-12-24 13:25:49,677 INFO > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: > maxReplication = 512 > 2013-12-24 13:25:49,677 INFO > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: > minReplication = 1 > 2013-12-24 13:25:49,677 INFO > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: > maxReplicationStreams = 2 > 2013-12-24 13:25:49,677 INFO > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: > shouldCheckForEnoughRacks = false > 2013-12-24 13:25:49,677 INFO > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: > replicationRecheckInterval = 3000 > 2013-12-24 13:25:49,677 INFO > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: > encryptDataTransfer = false > 2013-12-24 13:25:49,681 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner > = sitaraman (auth:SIMPLE) > 2013-12-24 13:25:49,681 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup > = supergroup > 2013-12-24 13:25:49,681 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: > isPermissionEnabled = true > 2013-12-24 13:25:49,681 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false > 2013-12-24 13:25:49,682 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: > true > 2013-12-24 13:25:49,801 INFO org.apache.hadoop.util.GSet: Computing > capacity for map INodeMap > 2013-12-24 13:25:49,801 INFO org.apache.hadoop.util.GSet: VM type = > 64-bit > 2013-12-24 13:25:49,801 INFO org.apache.hadoop.util.GSet: 1.0% max > memory = 889 MB > 2013-12-24 13:25:49,801 INFO org.apache.hadoop.util.GSet: capacity > = 2^20 = 1048576 entries > 2013-12-24 13:25:49,802 INFO > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names > occuring more than 10 times > 2013-12-24 13:25:49,804 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: > dfs.namenode.safemode.threshold-pct = 0.9990000128746033 > 2013-12-24 13:25:49,804 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: > dfs.namenode.safemode.min.datanodes = 0 > 2013-12-24 13:25:49,804 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: > dfs.namenode.safemode.extension = 30000 > 2013-12-24 13:25:49,805 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache on > namenode is enabled > 2013-12-24 13:25:49,805 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache will > use 0.03 of total heap and retry cache entry expiry time is 600000 > millis > 2013-12-24 13:25:49,807 INFO org.apache.hadoop.util.GSet: Computing > capacity for map Namenode Retry Cache > 2013-12-24 13:25:49,807 INFO org.apache.hadoop.util.GSet: VM type = > 64-bit > 2013-12-24 13:25:49,807 INFO org.apache.hadoop.util.GSet: > 0.029999999329447746% max memory = 889 MB > 2013-12-24 13:25:49,807 INFO org.apache.hadoop.util.GSet: capacity > = 2^15 = 32768 entries > 2013-12-24 13:25:49,816 INFO > org.apache.hadoop.hdfs.server.common.Storage: Lock on > /usr/local/Software/hadoop-0.23.10/data/hdfs/namenode/in_use.lock > acquired by nodename [email protected] > 2013-12-24 13:25:49,861 INFO org.mortbay.log: Stopped > [email protected]:50070 > 2013-12-24 13:25:49,964 INFO > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode > metrics system... > 2013-12-24 13:25:49,965 INFO > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics > system stopped. > 2013-12-24 13:25:49,965 INFO > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics > system shutdown complete. > 2013-12-24 13:25:49,965 FATAL > org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode > join > java.io.IOException: > File system image contains an old layout version -39. > An upgrade to version -47 is required. > Please restart NameNode with -upgrade option. > at > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:221) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:787) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:568) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:443) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:491) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:684) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:669) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1254) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1320) > 2013-12-24 13:25:49,967 INFO org.apache.hadoop.util.ExitUtil: Exiting > with status 1 > 2013-12-24 13:25:49,968 INFO > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: > /************************************************************ > SHUTDOWN_MSG: Shutting down NameNode at localhost.localdomain/127.0.0.1 > ************************************************************/ > > On 12/24/13, Sitaraman Vilayannur <[email protected]> wrote: >> The line beginning with ulimit that i have appended below, i thought >> was the log file? >> /usr/local/Software/hadoop-2.2.0/logs/hadoop-sitaraman-namenode-localhost.localdomain.out >> Sitaraman >> On 12/24/13, Nitin Pawar <[email protected]> wrote: >>> Without log, very hard to guess what's happening. >>> >>> Can you clean up the log directory and then start over and check for the >>> logs again. >>> >>> >>> On Tue, Dec 24, 2013 at 11:44 AM, Sitaraman Vilayannur < >>> [email protected]> wrote: >>> >>>> Hi Nitin, >>>> I moved to the release 2.2.0 on starting node manager it remains >>>> silent without errors but nodemanager dosent start....while it does in >>>> the earlier 0.23 version >>>> >>>> >>>> ./hadoop-daemon.sh start namenode >>>> starting namenode, logging to >>>> >>>> /usr/local/Software/hadoop-2.2.0/logs/hadoop-sitaraman-namenode-localhost.localdomain.out >>>> Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library >>>> /usr/local/Software/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0 which >>>> might have disabled stack guard. The VM will try to fix the stack >>>> guard now. >>>> It's highly recommended that you fix the library with 'execstack -c >>>> <libfile>', or link it with '-z noexecstack'. >>>> [sitaraman@localhost sbin]$ jps >>>> 13444 Jps >>>> [sitaraman@localhost sbin]$ vi >>>> >>>> /usr/local/Software/hadoop-2.2.0/logs/hadoop-sitaraman-namenode-localhost.localdomain.out >>>> >>>> >>>> ulimit -a for user sitaraman >>>> core file size (blocks, -c) 0 >>>> data seg size (kbytes, -d) unlimited >>>> scheduling priority (-e) 0 >>>> file size (blocks, -f) unlimited >>>> pending signals (-i) 135104 >>>> max locked memory (kbytes, -l) 32 >>>> max memory size (kbytes, -m) unlimited >>>> open files (-n) 1024 >>>> pipe size (512 bytes, -p) 8 >>>> POSIX message queues (bytes, -q) 819200 >>>> real-time priority (-r) 0 >>>> stack size (kbytes, -s) 10240 >>>> cpu time (seconds, -t) unlimited >>>> max user processes (-u) 135104 >>>> virtual memory (kbytes, -v) unlimited >>>> file locks (-x) unlimited >>>> >>>> >>>> On 12/24/13, Nitin Pawar <[email protected]> wrote: >>>> > For now you can ignore this warning, >>>> > it was your first program so you can try building other things and >>>> > slowly >>>> > run the commands mentioned the log message to fix these small >>>> > warnings. >>>> > >>>> > >>>> > On Tue, Dec 24, 2013 at 10:07 AM, Sitaraman Vilayannur < >>>> > [email protected]> wrote: >>>> > >>>> >> Thanks Nitin, That worked, >>>> >> When i run the Pi example, i get the following warning at the end, >>>> >> what must i do about this warning....thanks much for your help. >>>> >> Sitaraman >>>> >> inished in 20.82 seconds >>>> >> Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library >>>> >> /usr/local/Software/hadoop-0.23.10/lib/native/libhadoop.so.1.0.0 >>>> >> which >>>> >> might have disabled stack guard. The VM will try to fix the stack >>>> >> guard now. >>>> >> It's highly recommended that you fix the library with 'execstack -c >>>> >> <libfile>', or link it with '-z noexecstack'. >>>> >> 13/12/24 10:05:19 WARN util.NativeCodeLoader: Unable to load >>>> >> native-hadoop library for your platform... using builtin-java >>>> >> classes >>>> >> where applicable >>>> >> Estimated value of Pi is 3.14127500000000000000 >>>> >> [sitaraman@localhost mapreduce]$ >>>> >> >>>> >> On 12/23/13, Nitin Pawar <[email protected]> wrote: >>>> >> > Can you try starting the process as non root user. >>>> >> > Give proper permissions to the user and start it as a different >>>> >> > user. >>>> >> > >>>> >> > Thanks, >>>> >> > Nitin >>>> >> > >>>> >> > >>>> >> > On Mon, Dec 23, 2013 at 2:15 PM, Sitaraman Vilayannur < >>>> >> > [email protected]> wrote: >>>> >> > >>>> >> >> Hi, >>>> >> >> When i attempt to start nodemanager i get the following error. >>>> >> >> Any >>>> >> >> help >>>> >> >> appreciated. I was able to start resource manager datanode, >>>> namenode >>>> >> >> and >>>> >> >> secondarynamenode, >>>> >> >> >>>> >> >> >>>> >> >> ./yarn-daemon.sh start nodemanager >>>> >> >> starting nodemanager, logging to >>>> >> >> >>>> >> >>>> /usr/local/Software/hadoop-0.23.10/logs/yarn-root-nodemanager-localhost.localdomain.out >>>> >> >> Unrecognized option: -jvm >>>> >> >> Error: Could not create the Java Virtual Machine. >>>> >> >> Error: A fatal exception has occurred. Program will exit. >>>> >> >> [root@localhost sbin]# emacs >>>> >> >> >>>> >> >>>> /usr/local/Software/hadoop-0.23.10/logs/yarn-root-nodemanager-localhost.localdomain.out >>>> >> >> & >>>> >> >> [4] 29004 >>>> >> >> [root@localhost sbin]# jps >>>> >> >> 28402 SecondaryNameNode >>>> >> >> 30280 Jps >>>> >> >> 28299 DataNode >>>> >> >> 6729 Main >>>> >> >> 26044 ResourceManager >>>> >> >> 28197 NameNode >>>> >> >> >>>> >> > >>>> >> > >>>> >> > >>>> >> > -- >>>> >> > Nitin Pawar >>>> >> > >>>> >> >>>> > >>>> > >>>> > >>>> > -- >>>> > Nitin Pawar >>>> > >>>> >>> >>> >>> >>> -- >>> Nitin Pawar >>> >> >
