Your fsimage seems to have gone bad (is it 0-sized? I recall that as a known issue long since fixed).
The easiest way is to fall back to the last available good checkpoint (From SNN). Or if you have multiple dfs.name.dirs, see if some of the other points have better/complete files on them, and re-spread them across after testing them out (and backing up the originals). Though what version are you running? Cause AFAIK most of the recent stable versions/distros include NN resource monitoring threads which should have placed your NN into safemode the moment all its disks ran near to out of space. On Mon, May 14, 2012 at 10:50 PM, Prashant Kommireddi <prash1...@gmail.com> wrote: > Hi, > > I am seeing an issue where Namenode does not start due an EOFException. The > disk was full and I cleared space up but I am unable to get past this > exception. Any ideas on how this can be resolved? > > 2012-05-14 10:10:44,018 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=hadoop > 2012-05-14 10:10:44,018 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: > isPermissionEnabled=false > 2012-05-14 10:10:44,023 INFO > org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics: > Initializing FSNamesystemMetrics using context > object:org.apache.hadoop.metrics.file.FileContext > 2012-05-14 10:10:44,024 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered > FSNamesystemStatusMBean > 2012-05-14 10:10:44,047 INFO org.apache.hadoop.hdfs.server.common.Storage: > Number of files = 205470 > 2012-05-14 10:10:44,844 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed. > java.io.EOFException > at java.io.DataInputStream.readFully(DataInputStream.java:180) > at org.apache.hadoop.io.UTF8.readFields(UTF8.java:106) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.readString(FSImage.java:1578) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:880) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:807) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:364) > at > org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:292) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:201) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965) > 2012-05-14 10:10:44,845 INFO org.apache.hadoop.ipc.Server: Stopping server > on 54310 > 2012-05-14 10:10:44,845 ERROR > org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.EOFException > at java.io.DataInputStream.readFully(DataInputStream.java:180) > at org.apache.hadoop.io.UTF8.readFields(UTF8.java:106) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.readString(FSImage.java:1578) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:880) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:807) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:364) > at > org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:292) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:201) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965) > > 2012-05-14 10:10:44,846 INFO > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: > /************************************************************ > SHUTDOWN_MSG: Shutting down NameNode at > gridforce-1.internal.salesforce.com/10.0.201.159 > ************************************************************/ -- Harsh J
