Alright this is from my note while i were playing with this. NFS4 ==== would be pretty straight forward.
@NFS server /data *(rw,fsid=0) !!! don't forget fsid=0 @Client mount -t rpc_pipefs sunrpc /var/lib/nfs/rpc_pipefs/ service rpcidmapd start mount -t nfs4 nfs_host:/ /hadoop/backup !!!! NOT mount -t nfs4 nfs_host:/data /hadoop/backup NFS3 ==== $ ps -ef |grep rpc rpc 1509 1 0 17:28 ? 00:00:00 portmap root 1592 1 0 17:28 ? 00:00:00 rpc.idmapd rpcuser 1704 1 0 17:29 ? 00:00:00 rpc.statd # service rpcidmapd start Starting RPC idmapd: Error: RPC MTAB does not exist. Fixed by mount -t rpc_pipefs sunrpc /var/lib/nfs/rpc_pipefs/ hth P On Mon, Oct 10, 2011 at 7:55 PM, Harsh J <[email protected]> wrote: > Generally you just gotta ensure that your rpc.lockd service is up and > running on both ends, to allow for locking over NFS. > > On Tue, Oct 11, 2011 at 8:16 AM, Uma Maheswara Rao G 72686 > <[email protected]> wrote: > > Hi, > > > > It looks to me that, problem with your NFS. It is not supporting locks. > Which version of NFS are you using? > > Please check your NFS locking support by writing simple program for file > locking. > > > > I think NFS4 supports locking ( i did not tried). > > > > http://nfs.sourceforge.net/ > > > > A6. What are the main new features in version 4 of the NFS protocol? > > *NFS Versions 2 and 3 are stateless protocols, but NFS Version 4 > introduces state. An NFS Version 4 client uses state to notify an NFS > Version 4 server of its intentions on a file: locking, reading, writing, and > so on. An NFS Version 4 server can return information to a client about what > other clients have intentions on a file to allow a client to cache file data > more aggressively via delegation. To help keep state consistent, more > sophisticated client and server reboot recovery mechanisms are built in to > the NFS Version 4 protocol. > > *NFS Version 4 introduces support for byte-range locking and share > reservation. Locking in NFS Version 4 is lease-based, so an NFS Version 4 > client must maintain contact with an NFS Version 4 server to continue > extending its open and lock leases. > > > > > > Regards, > > Uma > > ----- Original Message ----- > > From: Shouguo Li <[email protected]> > > Date: Tuesday, October 11, 2011 2:31 am > > Subject: Re: Secondary namenode fsimage concept > > To: [email protected] > > > >> hey parick > >> > >> i wanted to configure my cluster to write namenode metadata to > >> multipledirectories as well: > >> <property> > >> <name>dfs.name.dir</name> > >> <value>/hadoop/var/name,/mnt/hadoop/var/name</value> > >> </property> > >> > >> in my case, /hadoop/var/name is local directory, > >> /mnt/hadoop/var/name is NFS > >> volume. i took down the cluster first, then copied over files from > >> /hadoop/var/name to /mnt/hadoop/var/name, and then tried to start > >> up the > >> cluster. but the cluster won't start up properly... > >> here's the namenode log: http://pastebin.com/gmu0B7yd > >> > >> any ideas why it wouldn't start up? > >> thx > >> > >> > >> On Thu, Oct 6, 2011 at 6:58 PM, patrick sang > >> <[email protected]>wrote: > >> > I would say your namenode write metadata in local fs (where your > >> secondary> namenode will pull files), and NFS mount. > >> > > >> > <property> > >> > <name>dfs.name.dir</name> > >> > <value>/hadoop/name,/hadoop/nfs_server_name</value> > >> > </property> > >> > > >> > > >> > my 0.02$ > >> > P > >> > > >> > On Thu, Oct 6, 2011 at 12:04 AM, shanmuganathan.r < > >> > [email protected]> wrote: > >> > > >> > > Hi Kai, > >> > > > >> > > There is no datas stored in the secondarynamenode related > >> to the > >> > > Hadoop cluster . Am I correct? > >> > > If it correct means If we run the secondaryname node in > >> separate machine > >> > > then fetching , merging and transferring time is increased if > >> the cluster > >> > > has large data in the namenode fsimage file . At the time if > >> fail over > >> > > occurs , then how can we recover the nearly one hour changes in > >> the HDFS > >> > > file ? (default check point time is one hour) > >> > > > >> > > Thanks R.Shanmuganathan > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > ---- On Thu, 06 Oct 2011 12:20:28 +0530 Kai Voigt<[email protected]> > >> wrote> > ---- > >> > > > >> > > > >> > > Hi, > >> > > > >> > > the secondary namenode only fetches the two files when a > >> checkpointing is > >> > > needed. > >> > > > >> > > Kai > >> > > > >> > > Am 06.10.2011 um 08:45 schrieb shanmuganathan.r: > >> > > > >> > > > Hi Kai, > >> > > > > >> > > > In the Second part I meant > >> > > > > >> > > > > >> > > > Is the secondary namenode also contain the FSImage file or > >> the two > >> > > files(FSImage and EdiltLog) are transferred from the namenode > >> at the > >> > > checkpoint time. > >> > > > > >> > > > > >> > > > Thanks > >> > > > Shanmuganathan > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > ---- On Thu, 06 Oct 2011 11:37:50 +0530 Kai > >> Voigt&lt;[email protected]> &gt; > >> > > wrote ---- > >> > > > > >> > > > > >> > > > Hi, > >> > > > > >> > > > you're correct when saying the namenode hosts the fsimage > >> file and > >> > the > >> > > edits log file. > >> > > > > >> > > > The fsimage file contains a snapshot of the HDFS metadata (a > >> > filename > >> > > to blocks list mapping). Whenever there is a change to HDFS, it > >> will be > >> > > appended to the edits file. Think of it as a database > >> transaction log, > >> > where > >> > > changes will not be applied to the datafile, but appended to a > >> log.> > > > >> > > > To prevent the edits file growing infinitely, the > >> secondary namenode > >> > > periodically pulls these two files, and the namenode starts > >> writing> changes > >> > > to a new edits file. Then, the secondary namenode merges the > >> changes from > >> > > the edits file with the old snapshot from the fsimage file and > >> creates an > >> > > updated fsimage file. This updated fsimage file is then copied > >> to the > >> > > namenode. > >> > > > > >> > > > Then, the entire cycle starts again. To answer your > >> question: The > >> > > namenode has both files, even if the secondary namenode is > >> running on a > >> > > different machine. > >> > > > > >> > > > Kai > >> > > > > >> > > > Am 06.10.2011 um 07:57 schrieb shanmuganathan.r: > >> > > > > >> > > > &gt; > >> > > > &gt; Hi All, > >> > > > &gt; > >> > > > &gt; I have a doubt in hadoop secondary namenode > >> concept . > >> > Please > >> > > correct if the following statements are wrong . > >> > > > &gt; > >> > > > &gt; > >> > > > &gt; The namenode hosts the fsimage and edit log > >> files. The > >> > > secondary namenode hosts the fsimage file only. At the time of > >> checkpoint> > the edit log file is transferred to the secondary > >> namenode and the both > >> > > files are merged, Then the updated fsimage file is transferred > >> to the > >> > > namenode . Is it correct? > >> > > > &gt; > >> > > > &gt; > >> > > > &gt; If we run the secondary namenode in separate > >> machine , then > >> > > both machines contain the fsimage file . Namenode only contains > >> the> editlog > >> > > file. Is it true? > >> > > > &gt; > >> > > > &gt; > >> > > > &gt; > >> > > > &gt; Thanks R.Shanmuganathan > >> > > > &gt; > >> > > > &gt; > >> > > > &gt; > >> > > > &gt; > >> > > > &gt; > >> > > > &gt; > >> > > > > >> > > > -- > >> > > > Kai Voigt > >> > > > [email protected] > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > -- > >> > > Kai Voigt > >> > > [email protected] > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > >> > > > > > > -- > Harsh J >
