Thanks Harsh. I am using 0.20.2, I see on the Jira this issue was fixed for 0.23?
I will try out your suggestions and get back. On May 14, 2012, at 1:22 PM, Harsh J <ha...@cloudera.com> wrote: > Your fsimage seems to have gone bad (is it 0-sized? I recall that as a > known issue long since fixed). > > The easiest way is to fall back to the last available good checkpoint > (From SNN). Or if you have multiple dfs.name.dirs, see if some of the > other points have better/complete files on them, and re-spread them > across after testing them out (and backing up the originals). > > Though what version are you running? Cause AFAIK most of the recent > stable versions/distros include NN resource monitoring threads which > should have placed your NN into safemode the moment all its disks ran > near to out of space. > > On Mon, May 14, 2012 at 10:50 PM, Prashant Kommireddi > <prash1...@gmail.com> wrote: >> Hi, >> >> I am seeing an issue where Namenode does not start due an EOFException. The >> disk was full and I cleared space up but I am unable to get past this >> exception. Any ideas on how this can be resolved? >> >> 2012-05-14 10:10:44,018 INFO >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=hadoop >> 2012-05-14 10:10:44,018 INFO >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: >> isPermissionEnabled=false >> 2012-05-14 10:10:44,023 INFO >> org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics: >> Initializing FSNamesystemMetrics using context >> object:org.apache.hadoop.metrics.file.FileContext >> 2012-05-14 10:10:44,024 INFO >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered >> FSNamesystemStatusMBean >> 2012-05-14 10:10:44,047 INFO org.apache.hadoop.hdfs.server.common.Storage: >> Number of files = 205470 >> 2012-05-14 10:10:44,844 ERROR >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem >> initialization failed. >> java.io.EOFException >> at java.io.DataInputStream.readFully(DataInputStream.java:180) >> at org.apache.hadoop.io.UTF8.readFields(UTF8.java:106) >> at >> org.apache.hadoop.hdfs.server.namenode.FSImage.readString(FSImage.java:1578) >> at >> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:880) >> at >> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:807) >> at >> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:364) >> at >> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87) >> at >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311) >> at >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:292) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:201) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965) >> 2012-05-14 10:10:44,845 INFO org.apache.hadoop.ipc.Server: Stopping server >> on 54310 >> 2012-05-14 10:10:44,845 ERROR >> org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.EOFException >> at java.io.DataInputStream.readFully(DataInputStream.java:180) >> at org.apache.hadoop.io.UTF8.readFields(UTF8.java:106) >> at >> org.apache.hadoop.hdfs.server.namenode.FSImage.readString(FSImage.java:1578) >> at >> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:880) >> at >> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:807) >> at >> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:364) >> at >> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87) >> at >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311) >> at >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:292) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:201) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965) >> >> 2012-05-14 10:10:44,846 INFO >> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: >> /************************************************************ >> SHUTDOWN_MSG: Shutting down NameNode at >> gridforce-1.internal.salesforce.com/10.0.201.159 >> ************************************************************/ > > > > -- > Harsh J
