Re: Secondary namenode fsimage concept

[Kai Voigt]

Hi,

yes, the secondary namenode is actually a badly named piece of software, as 
it's not a namenode at all. And it's going to be renamed to checkpoint node.

To prevent metadata loss when your namenode fails, you should write the 
namenode files to a local RAID and also a networked storage (NFS, SAN, DRBD). 
It's not the secondary namenode's task to make the metadata available.

Kai

Am 06.10.2011 um 09:04 schrieb shanmuganathan.r:

> Hi Kai,
> 
>      There is no datas stored  in the secondarynamenode related to the Hadoop 
> cluster . Am I correct?
> If it correct means If we run the secondaryname node in separate machine then 
> fetching , merging and transferring time is increased if the cluster has 
> large data in the namenode fsimage file . At the time if fail over occurs , 
> then how can we recover the nearly one hour changes in the HDFS file ? 
> (default check point time is one hour)
> 
> Thanks R.Shanmuganathan  
> 
> 
> 
> 
> 
> 
> ---- On Thu, 06 Oct 2011 12:20:28 +0530 Kai Voigt<k...@123.org> wrote 
> ---- 
> 
> 
> Hi, 
> 
> the secondary namenode only fetches the two files when a checkpointing is 
> needed. 
> 
> Kai 
> 
> Am 06.10.2011 um 08:45 schrieb shanmuganathan.r: 
> 
> > Hi Kai, 
> > 
> > In the Second part I meant 
> > 
> > 
> > Is the secondary namenode also contain the FSImage file or the two 
> files(FSImage and EdiltLog) are transferred from the namenode at the 
> checkpoint time. 
> > 
> > 
> > Thanks 
> > Shanmuganathan 
> > 
> > 
> > 
> > 
> > 
> > ---- On Thu, 06 Oct 2011 11:37:50 +0530 Kai 
> Voigt<k...@123.org> wrote ---- 
> > 
> > 
> > Hi, 
> > 
> > you're correct when saying the namenode hosts the fsimage file and the 
> edits log file. 
> > 
> > The fsimage file contains a snapshot of the HDFS metadata (a filename to 
> blocks list mapping). Whenever there is a change to HDFS, it will be appended 
> to the edits file. Think of it as a database transaction log, where changes 
> will not be applied to the datafile, but appended to a log. 
> > 
> > To prevent the edits file growing infinitely, the secondary namenode 
> periodically pulls these two files, and the namenode starts writing changes 
> to a new edits file. Then, the secondary namenode merges the changes from the 
> edits file with the old snapshot from the fsimage file and creates an updated 
> fsimage file. This updated fsimage file is then copied to the namenode. 
> > 
> > Then, the entire cycle starts again. To answer your question: The 
> namenode has both files, even if the secondary namenode is running on a 
> different machine. 
> > 
> > Kai 
> > 
> > Am 06.10.2011 um 07:57 schrieb shanmuganathan.r: 
> > 
> > > 
> > > Hi All, 
> > > 
> > > I have a doubt in hadoop secondary namenode concept . Please 
> correct if the following statements are wrong . 
> > > 
> > > 
> > > The namenode hosts the fsimage and edit log files. The 
> secondary namenode hosts the fsimage file only. At the time of checkpoint the 
> edit log file is transferred to the secondary namenode and the both files are 
> merged, Then the updated fsimage file is transferred to the namenode . Is it 
> correct? 
> > > 
> > > 
> > > If we run the secondary namenode in separate machine , then 
> both machines contain the fsimage file . Namenode only contains the editlog 
> file. Is it true? 
> > > 
> > > 
> > > 
> > > Thanks R.Shanmuganathan 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > 
> > -- 
> > Kai Voigt 
> > k...@123.org 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> 
> -- 
> Kai Voigt 
> k...@123.org 
> 
> 
> 
> 
> 
> 

-- 
Kai Voigt
k...@123.org