On Thursday 07 December 2006 20:25, Sridharan Ramaswamy (srramasw) wrote: > Hi, > > Has anyone used Lustre client as root filesystem for diskless nodes? > > I saw this old mail on using Lustre as root filesystem, > > https://mail.clusterfs.com/pipermail/lustre-discuss/2005-May/000702.html > > Is the HOWTO document mentioned in there available someplace? Seems it > is supposed to work w/ few known problem(may be there are fixed in > recent releases) > > Appreciate any help on this. >
Just to give a "me too", we're running lustre as a root filesystem for 130 nodes. It's a "mostly shared root". Though bear in mind, AFAIK such things are so not a Clusterfs supported configuration. Our setup is quite similar to other people's described in this thread. PXE Boot a (large) initrd (but you can get away with quite large initrds when tftping them :-) ). Initrd is made by letting redhat tools build a normal initrd using mkinitrd --preload, and mutilating it by uncompressing it, and rpm2cpio installing certain rpms to it, and using a different init script. Note that RHEL4 includes a "system-config-netboot" package for autobuilding NFS-root clients- this was very helpful in working out what parts of the FS were safe to share and what parts should be per-host, and how various initscripts should be modified. The initrd sets up networking, uses the client-side zero-configuration support of lustre 1.4.7.x to mount lustre without bloating the initrd too much, then pivot_roots to a bind-mounted subdir of the lustre fs (similar to other users supporting multiple roots). When control is transferred to that root (i.e. the exec /sbin/init...), a script /etc/rc.bindmounts is called from very early on in the init process. This bind mounts /perhost/currenthost to /perhost/<hostname>, then bind mounts the various host-specific parts (like, say, /var/spool/) to subdirs of /perhost/currenthost This makes admin mostly a breeze, though I have definite qualms about the client shutdown "process" I use* and the large numbers of locks sharedroot clients are holding. * Hard shutdown, basically. As reported by others, for system level stuff the fs is kind of slow (as system level stuff means lots of small files and metadata changes I guess, which lustre doesn't currently excel at), but has little impact on user level stuff. But pay close attention to cron jobs and such if you go this route - only _one_ node needs to update the locatedb or rpmdb or whatever, and you cause horrible load spikes at best, crashes at worst, if you mess cron jobs up, all nodes e.g. trying to "yum update" the same FS at the one time would be messy. _______________________________________________ Lustre-discuss mailing list [email protected] https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
