Not a lot of interest I suppose, but the problem is believed to have been caused by not following the recommended upgrade procedure when going from 1.0.1 to 1.0.2.
On 5/4/12 9:56 AM, Terry Healy wrote: > Running Apache 1.0.2 Primary NN on Ubuntu 11.10, 8 datanodes. When I > came in today I ran a stop-all.sh since one of the datanodes was not > showing on the dfsnodelist.jsp?whatNodes=LIVE status page. I then ran > start-all.sh, and the NN dies on a Null Pointer Exception. I looked at > the source but cannot determine what is going. Below is the trace - any > explanation of cause or suggestions to get the system back up would be > appreciated. > > ** I have removed the prefix "org.apache.hadoop." from the list below in > the interest of saving space. ** > > -Terry > > 2012-05-04 09:45:02,046 INFO hdfs.server.namenode.NameNode: STARTUP_MSG: > /************************************************************ > STARTUP_MSG: Starting NameNode > STARTUP_MSG: host = xxxx/xxxxxxxx > STARTUP_MSG: args = [] > STARTUP_MSG: version = 1.0.2 > STARTUP_MSG: build = > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0.2 -r > 1304954; compiled by 'hortonfo' on Sat Mar 24 23:58:21 UTC 2012 > ************************************************************/ > 2012-05-04 09:45:02,229 INFO metrics2.impl.MetricsConfig: loaded > properties from hadoop-metrics2.properties > 2012-05-04 09:45:02,244 INFO metrics2.impl.MetricsSourceAdapter: MBean > for source MetricsSystem,sub=Stats registered. > 2012-05-04 09:45:02,246 INFO metrics2.impl.MetricsSystemImpl: Scheduled > snapshot period at 10 second(s). > 2012-05-04 09:45:02,246 INFO metrics2.impl.MetricsSystemImpl: NameNode > metrics system started > 2012-05-04 09:45:02,483 INFO metrics2.impl.MetricsSourceAdapter: MBean > for source ugi registered. > 2012-05-04 09:45:02,488 WARN metrics2.impl.MetricsSystemImpl: Source > name ugi already exists! > 2012-05-04 09:45:02,494 INFO metrics2.impl.MetricsSourceAdapter: MBean > for source jvm registered. > 2012-05-04 09:45:02,496 INFO metrics2.impl.MetricsSourceAdapter: MBean > for source NameNode registered. > 2012-05-04 09:45:02,530 INFO hdfs.util.GSet: VM type = 64-bit > 2012-05-04 09:45:02,530 INFO hdfs.util.GSet: 2% max memory = 17.77875 MB > 2012-05-04 09:45:02,530 INFO hdfs.util.GSet: capacity = 2^21 = > 2097152 entries > 2012-05-04 09:45:02,530 INFO hdfs.util.GSet: recommended=2097152, > actual=2097152 > 2012-05-04 09:45:02,560 INFO hdfs.server.namenode.FSNamesystem: > fsOwner=thealy > 2012-05-04 09:45:02,560 INFO hdfs.server.namenode.FSNamesystem: > supergroup=supergroup > 2012-05-04 09:45:02,560 INFO hdfs.server.namenode.FSNamesystem: > isPermissionEnabled=true > 2012-05-04 09:45:02,567 INFO hdfs.server.namenode.FSNamesystem: > dfs.block.invalidate.limit=100 > 2012-05-04 09:45:02,567 INFO hdfs.server.namenode.FSNamesystem: > isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), > accessTokenLifetime=0 min(s) > 2012-05-04 09:45:02,749 INFO hdfs.server.namenode.FSNamesystem: > Registered FSNamesystemStateMBean and NameNodeMXBean > 2012-05-04 09:45:02,765 INFO hdfs.server.namenode.NameNode: Caching file > names occuring more than 10 times > 2012-05-04 09:45:02,775 INFO hdfs.server.common.Storage: Number of files > = 11 > 2012-05-04 09:45:02,783 INFO hdfs.server.common.Storage: Number of files > under construction = 0 > 2012-05-04 09:45:02,783 INFO hdfs.server.common.Storage: Image file of > size 11087 loaded in 0 seconds. > 2012-05-04 09:45:02,785 ERROR hdfs.server.namenode.NameNode: > java.lang.NullPointerException > at hdfs.server.namenode.FSDirectory.addChild(FSDirectory.java:1094) > at hdfs.server.namenode.FSDirectory.addChild(FSDirectory.java:1106) > at hdfs.server.namenode.FSDirectory.addNode(FSDirectory.java:1009) > at > hdfs.server.namenode.FSDirectory.unprotectedAddFile(FSDirectory.java:208) > at hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:626) > at hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:1015) > at hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:833) > at hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:372) > at hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:100) > at hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:388) > at hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:362) > at hdfs.server.namenode.NameNode.initialize(NameNode.java:276) > at hdfs.server.namenode.NameNode.<init>(NameNode.java:496) > at hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279) > at hdfs.server.namenode.NameNode.main(NameNode.java:1288) -- Terry Healy / the...@bnl.gov Cyber Security Operations Brookhaven National Laboratory Building 515, Upton N.Y. 11973
