Yes Harsh, I haven't set dfs.namenode.name.dir anywhere in config files. My name node has again gone into safe mode today while it was idle. I shall try setting this value to something other than /tmp
On Tue, Jul 16, 2013 at 6:39 AM, Harsh J <ha...@cloudera.com> wrote: > > 2013-07-12 11:04:26,002 WARN > org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space > available on volume 'null' is 0, which is below the configured reserved > amount 104857600 > > This is interesting. Its calling your volume "null", which may be more > of a superficial bug. > > What is your dfs.namenode.name.dir set to? From > /tmp/hadoop-dsadm/dfs/name I'd expect you haven't set it up and /tmp > is being used off of the out-of-box defaults. Could you try to set it > to a specific directory thats not on /tmp? > > On Mon, Jul 15, 2013 at 2:43 PM, Krishna Kishore Bonagiri > <write2kish...@gmail.com> wrote: > > I don't have it in my hdfs-site.xml, in which case probably the default > > value is taken.. > > > > > > On Mon, Jul 15, 2013 at 2:29 PM, Azuryy Yu <azury...@gmail.com> wrote: > >> > >> please check dfs.datanode.du.reserved in the hdfs-site.xml > >> > >> On Jul 15, 2013 4:30 PM, "Aditya exalter" <adityaexal...@gmail.com> > wrote: > >>> > >>> Hi Krishna, > >>> > >>> Can you please send screenshots of namenode web UI. > >>> > >>> Thanks Aditya. > >>> > >>> > >>> On Mon, Jul 15, 2013 at 1:54 PM, Krishna Kishore Bonagiri > >>> <write2kish...@gmail.com> wrote: > >>>> > >>>> I have had enough space on the disk that is used, like around 30 Gigs > >>>> > >>>> Thanks, > >>>> Kishore > >>>> > >>>> > >>>> On Mon, Jul 15, 2013 at 1:30 PM, Venkatarami Netla > >>>> <venkatarami.ne...@cloudwick.com> wrote: > >>>>> > >>>>> Hi, > >>>>> pls see the available space for NN storage directory. > >>>>> > >>>>> Thanks & Regards > >>>>> > >>>>> Venkat > >>>>> > >>>>> > >>>>> On Mon, Jul 15, 2013 at 12:14 PM, Krishna Kishore Bonagiri > >>>>> <write2kish...@gmail.com> wrote: > >>>>>> > >>>>>> Hi, > >>>>>> > >>>>>> I am doing no activity on my single node cluster which is using > >>>>>> 2.1.0-beta, and still observed that it has gone to safe mode by > itself after > >>>>>> a while. I was looking at the name node log and see many of these > kinds of > >>>>>> entries.. Can anything be interpreted from these? > >>>>>> > >>>>>> 2013-07-12 09:06:11,256 INFO > >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log > segment at > >>>>>> 561 > >>>>>> 2013-07-12 09:07:11,290 INFO > >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log > from > >>>>>> 9.70.137.114 > >>>>>> 2013-07-12 09:07:11,290 INFO > >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs > >>>>>> 2013-07-12 09:07:11,290 INFO > >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log > segment 561 > >>>>>> 2013-07-12 09:07:11,291 INFO > >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of > transactions: 2 > >>>>>> Total time for transactions(ms): 1 Number of transactions batched > in Syncs: > >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 14 > >>>>>> 2013-07-12 09:07:11,292 INFO > >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of > transactions: 2 > >>>>>> Total time for transactions(ms): 1 Number of transactions batched > in Syncs: > >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 15 > >>>>>> 2013-07-12 09:07:11,293 INFO > >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: > Finalizing edits > >>>>>> file > /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000561 > >>>>>> -> > >>>>>> > /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000561-0000000000000000562 > >>>>>> 2013-07-12 09:07:11,294 INFO > >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log > segment at > >>>>>> 563 > >>>>>> 2013-07-12 09:08:11,397 INFO > >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log > from > >>>>>> 9.70.137.114 > >>>>>> 2013-07-12 09:08:11,398 INFO > >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs > >>>>>> 2013-07-12 09:08:11,398 INFO > >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log > segment 563 > >>>>>> 2013-07-12 09:08:11,399 INFO > >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of > transactions: 2 > >>>>>> Total time for transactions(ms): 2 Number of transactions batched > in Syncs: > >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 11 > >>>>>> 2013-07-12 09:08:11,400 INFO > >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of > transactions: 2 > >>>>>> Total time for transactions(ms): 2 Number of transactions batched > in Syncs: > >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12 > >>>>>> 2013-07-12 09:08:11,402 INFO > >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: > Finalizing edits > >>>>>> file > /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000563 > >>>>>> -> > >>>>>> > /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000563-0000000000000000564 > >>>>>> 2013-07-12 09:08:11,402 INFO > >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log > segment at > >>>>>> 565 > >>>>>> 2013-07-12 09:09:11,440 INFO > >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log > from > >>>>>> 9.70.137.114 > >>>>>> 2013-07-12 09:09:11,440 INFO > >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs > >>>>>> 2013-07-12 09:09:11,440 INFO > >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log > segment 565 > >>>>>> 2013-07-12 09:09:11,440 INFO > >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of > transactions: 2 > >>>>>> Total time for transactions(ms): 0 Number of transactions batched > in Syncs: > >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 13 > >>>>>> 2013-07-12 09:09:11,441 INFO > >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of > transactions: 2 > >>>>>> Total time for transactions(ms): 0 Number of transactions batched > in Syncs: > >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 13 > >>>>>> > >>>>>> > >>>>>> And after sometime it said: > >>>>>> > >>>>>> 2013-07-12 11:03:19,799 INFO > >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log > segment at > >>>>>> 795 > >>>>>> 2013-07-12 11:04:19,826 INFO > >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log > from > >>>>>> 9.70.137.114 > >>>>>> 2013-07-12 11:04:19,826 INFO > >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs > >>>>>> 2013-07-12 11:04:19,827 INFO > >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log > segment 795 > >>>>>> 2013-07-12 11:04:19,827 INFO > >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of > transactions: 2 > >>>>>> Total time for transactions(ms): 0 Number of transactions batched > in Syncs: > >>>>>> 0 Number of syncs: 2 SyncTimes(ms): 12 > >>>>>> 2013-07-12 11:04:19,827 INFO > >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of > transactions: 2 > >>>>>> Total time for transactions(ms): 0 Number of transactions batched > in Syncs: > >>>>>> 0 Number of syncs: 3 SyncTimes(ms): 12 > >>>>>> 2013-07-12 11:04:19,829 INFO > >>>>>> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: > Finalizing edits > >>>>>> file > /tmp/hadoop-dsadm/dfs/name/current/edits_inprogress_0000000000000000795 > >>>>>> -> > >>>>>> > /tmp/hadoop-dsadm/dfs/name/current/edits_0000000000000000795-0000000000000000796 > >>>>>> 2013-07-12 11:04:19,829 INFO > >>>>>> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log > segment at > >>>>>> 797 > >>>>>> 2013-07-12 11:04:26,002 WARN > >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: > Space > >>>>>> available on volume 'null' is 0, which is below the configured > reserved > >>>>>> amount 104857600 > >>>>>> 2013-07-12 11:04:26,003 WARN > >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on > >>>>>> available disk space. Entering safe mode. > >>>>>> 2013-07-12 11:04:26,004 INFO org.apache.hadoop.hdfs.StateChange: > >>>>>> STATE* Safe mode is ON. > >>>>>> Resources are low on NN. Please add or free up more resources then > >>>>>> turn off safe mode manually. NOTE: If you turn off safe mode > before adding > >>>>>> resources, the NN will immediately return to safe mode.. > >>>>>> 2013-07-12 11:04:31,004 WARN > >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: > Space > >>>>>> available on volume 'null' is 0, which is below the configured > reserved > >>>>>> amount 104857600 > >>>>>> 2013-07-12 11:04:31,005 WARN > >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on > >>>>>> available disk space. Already in safe mode. > >>>>>> 2013-07-12 11:04:31,005 INFO org.apache.hadoop.hdfs.StateChange: > >>>>>> STATE* Safe mode is ONResources are low on NN. Please add or free > up more > >>>>>> resources then turn off safe mode manually. NOTE: If you turn off > safe > >>>>>> mode before adding resources, the NN will immediately return to > safe mode.. > >>>>>> 2013-07-12 11:04:36,005 WARN > >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: > Space > >>>>>> available on volume 'null' is 0, which is below the configured > reserved > >>>>>> amount 104857600 > >>>>>> > >>>>>> > >>>>>> Thanks, > >>>>>> Kishore > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> N Venkata Rami Reddy > >>>>> Hadoop Admin > >>>>> Cloudwick Technologies > >>>> > >>>> > >>> > > > > > > -- > Harsh J >
