RE: No edits files in dfs.namenode.edits.dir

[sky88088]

Hi, Harsh. 
The "dfs.namenode.edits.dir" from "hdfs getconf" is 
"file:///data1/hadoop/name", the same as "dfs.namenode.name.dir".
However, in the NameNode web UI("namenode:8088/conf"), none of 
"dfs.namenode.edits.dir" or "dfs.namenode.name.dir" is set. 
In both source, the "hadoop.tmp.dir" is set as "/data1/hadoop/tmp".
It seems strange because "dfs.namenode.name.dir" should be 
"file://${hadoop.tmp.dir}/dfs/name" by default, which means 
"/data1/hadoop/tmp/dfs/name", instead of the path value from "hdfs 
getconf"("file:///data1/hadoop/name").
However, only "file:///data1/hadoop/name" directionary can be found in my 
cluster.

I think Ravi's suggestion make sense.  If NameNode actually persisting edits, 
it could by found by "lsof -grep edit", right?
By the way, the version of the cluster is hadoop 2.2.0. I have no idea if this 
information helpful.

Thanks

From: ha...@cloudera.com
Date: Mon, 23 May 2016 03:33:17 +
Subject: Re: No edits files in dfs.namenode.edits.dir
To: sky880883...@hotmail.com; gurmukh.dhil...@yahoo.com; ravihad...@gmail.com
CC: user@hadoop.apache.org

Are you absolutely certain you are looking at the right directory? The NameNode 
is designed to crash if it cannot persist edits (transactions) durably.
The "hdfs getconf" utility checks local command classpaths but your service may 
be running over a different configuration directory. If you have access to your 
NameNode web UI, try visiting its /conf URL and you can see the actual 
directories configured for the NameNode's images and edit log.
P.s. The config dfs.namenode.edits.dir usually also defaults to the value of 
dfs.namenode.name.dir.
On Mon, 23 May 2016 at 11:13 sky88088 <sky880883...@hotmail.com> wrote:




I use "hdfs -getconf" to check all the relate dirs, and still no edits files 
found.

Thanks Ravi's suggestion. I tried "lsof -grep edit" and "lsof -grep hadoop", 
but still nothing meaningful found.

If really no edits persist to disk, any idea about how to fix this problem?

thanks
Subject: Re: No edits files in dfs.namenode.edits.dir
To: ravihad...@gmail.com; sky880883...@hotmail.com
CC: user@hadoop.apache.org
From: gurmukh.dhil...@yahoo.com.INVALID
Date: Sun, 22 May 2016 21:45:50 +1000


  

  
  
Have you explicitly configured the parameter
"dfs.namenode.edits.dir" ? else it default to dfs.namenode.name.dir







On 20/05/16 4:46 AM, Ravi Prakash
  wrote:



  No! You are probably writing the edits file
somewhere still. An `lsof` on the namenode process may be more
revealing. Obviously this depends on configuration, but unless
you have some really crazy settings, I'm pretty sure the edits
would be persisted to disk.



  
  

On Wed, May 18, 2016 at 2:47 AM,
  sky88088 <sky880883...@hotmail.com>
  wrote:

  

  Hi, 
I found that there is no edits file in my
  cluster's dfs.namenode.edits.dir.
  



  It is a 20-nodes cluster without secondary
namenode, and it runs well for a long while
(more than one year).




However, it's found that no edits file in the 
dfs.namenode.edits.dir
and the the fsimage file doesn't update. 
  
  


  Is it meaning that
  all the metadata of name node only stay in
the memory?
  

  
  Is there a way to fix it? Is there any
configuration to control the persistence?
  

  
  Thanks!

  

  



  



-- 
--
Thanks and Regards

Gurmukh Singh 

Re: No edits files in dfs.namenode.edits.dir

[Harsh J]

Are you absolutely certain you are looking at the right directory? The
NameNode is designed to crash if it cannot persist edits (transactions)
durably.

The "hdfs getconf" utility checks local command classpaths but your service
may be running over a different configuration directory. If you have access
to your NameNode web UI, try visiting its /conf URL and you can see the
actual directories configured for the NameNode's images and edit log.

P.s. The config dfs.namenode.edits.dir usually also defaults to the value
of dfs.namenode.name.dir.

On Mon, 23 May 2016 at 11:13 sky88088 <sky880883...@hotmail.com> wrote:

>
> I use "hdfs -getconf" to check all the relate dirs, and still no edits
> files found.
>
> Thanks Ravi's suggestion. I tried "lsof -grep edit" and "lsof -grep
> hadoop", but still nothing meaningful found.
>
> If really no edits persist to disk, any idea about how to fix this problem?
>
> thanks
> ------
> Subject: Re: No edits files in dfs.namenode.edits.dir
> To: ravihad...@gmail.com; sky880883...@hotmail.com
> CC: user@hadoop.apache.org
> From: gurmukh.dhil...@yahoo.com.INVALID
> Date: Sun, 22 May 2016 21:45:50 +1000
>
>
> Have you explicitly configured the parameter "dfs.namenode.edits.dir" ?
> else it default to dfs.namenode.name.dir
>
>
>
> On 20/05/16 4:46 AM, Ravi Prakash wrote:
>
> No! You are probably writing the edits file somewhere still. An `lsof` on
> the namenode process may be more revealing. Obviously this depends on
> configuration, but unless you have some really crazy settings, I'm pretty
> sure the edits would be persisted to disk.
>
>
> On Wed, May 18, 2016 at 2:47 AM, sky88088 <sky880883...@hotmail.com>
> wrote:
>
> Hi,
> I found that there is no edits file in my
> cluster's dfs.namenode.edits.dir.
>
>
> It is a 20-nodes cluster without secondary namenode, and it runs well for
> a long while (more than one year).
>
> However, it's found that no edits file in the dfs.namenode.edits.dir and
> the the fsimage file doesn't update.
>
> Is it meaning that all the metadata of name node only stay in the memory?
>
> Is there a way to fix it? Is there any configuration to control the
> persistence?
>
> Thanks!
>
>
>
> --
> --
> Thanks and Regards
>
> Gurmukh Singh
>
>

RE: No edits files in dfs.namenode.edits.dir

[sky88088]

I use "hdfs -getconf" to check all the relate dirs, and still no edits files 
found.

Thanks Ravi's suggestion. I tried "lsof -grep edit" and "lsof -grep hadoop", 
but still nothing meaningful found.

If really no edits persist to disk, any idea about how to fix this problem?

thanks

Subject: Re: No edits files in dfs.namenode.edits.dir
To: ravihad...@gmail.com; sky880883...@hotmail.com
CC: user@hadoop.apache.org
From: gurmukh.dhil...@yahoo.com.INVALID
Date: Sun, 22 May 2016 21:45:50 +1000


  

  
  
Have you explicitly configured the parameter
"dfs.namenode.edits.dir" ? else it default to dfs.namenode.name.dir







On 20/05/16 4:46 AM, Ravi Prakash
  wrote:



  No! You are probably writing the edits file
somewhere still. An `lsof` on the namenode process may be more
revealing. Obviously this depends on configuration, but unless
you have some really crazy settings, I'm pretty sure the edits
would be persisted to disk.



  
  

On Wed, May 18, 2016 at 2:47 AM,
  sky88088 <sky880883...@hotmail.com>
  wrote:

  

  Hi, 
I found that there is no edits file in my
  cluster's dfs.namenode.edits.dir.
  



  It is a 20-nodes cluster without secondary
namenode, and it runs well for a long while
(more than one year).




However, it's found that no edits file in the 
dfs.namenode.edits.dir
and the the fsimage file doesn't update. 
  
  


  Is it meaning that
  all the metadata of name node only stay in
the memory?
  

  
  Is there a way to fix it? Is there any
configuration to control the persistence?
  

  
  Thanks!

  

  



  



-- 
--
Thanks and Regards

Gurmukh Singh 

Re: No edits files in dfs.namenode.edits.dir

[Gurmukh Singh]

Have you explicitly configured the parameter "dfs.namenode.edits.dir" ? 
else it default to dfs.namenode.name.dir




On 20/05/16 4:46 AM, Ravi Prakash wrote:
No! You are probably writing the edits file somewhere still. An `lsof` 
on the namenode process may be more revealing. Obviously this depends 
on configuration, but unless you have some really crazy settings, I'm 
pretty sure the edits would be persisted to disk.



On Wed, May 18, 2016 at 2:47 AM, sky88088 > wrote:


Hi,
I found that there is no edits file in my
cluster's dfs.namenode.edits.dir.


It is a 20-nodes cluster without secondary namenode, and it runs
well for a long while (more than one year).

However, it's found that noedits file in the
dfs.namenode.edits.dir and the the fsimage file doesn't update.

Is it meaning that all the metadata of name node only stay in the
memory?

Is there a way to fix it? Is there any configuration to control
the persistence?

Thanks!




--
--
Thanks and Regards

Gurmukh Singh

Re: No edits files in dfs.namenode.edits.dir

[Ravi Prakash]

No! You are probably writing the edits file somewhere still. An `lsof` on
the namenode process may be more revealing. Obviously this depends on
configuration, but unless you have some really crazy settings, I'm pretty
sure the edits would be persisted to disk.


On Wed, May 18, 2016 at 2:47 AM, sky88088  wrote:

> Hi,
> I found that there is no edits file in my cluster's dfs.namenode.edits.dir.
>
>
> It is a 20-nodes cluster without secondary namenode, and it runs well for
> a long while (more than one year).
>
> However, it's found that no edits file in the dfs.namenode.edits.dir and
> the the fsimage file doesn't update.
>
> Is it meaning that all the metadata of name node only stay in the memory?
>
> Is there a way to fix it? Is there any configuration to control the
> persistence?
>
> Thanks!
>

No edits files in dfs.namenode.edits.dir

[sky88088]

Hi, I found that there is no edits file in my cluster's dfs.namenode.edits.dir.

It is a 20-nodes cluster without secondary namenode, and it runs well for a 
long while (more than one year).
However, it's found that no edits file in the dfs.namenode.edits.dir and the 
the fsimage file doesn't update. 
Is it meaning that all the metadata of name node only stay in the memory?
Is there a way to fix it? Is there any configuration to control the persistence?
Thanks!