It took me a few hours (mainly because it has been some years that I
played around with netboot and such), but I got it:
It needs some cleanup and still gives some errors during the boot
process, but I can boot a clean, fresh debian lenny via PXE directly
from a glusterfs Storage as a root-filesystem.
Of course, you need a dhcpd and tftpd running somewhere so the
netboot-server can get its ip, kernel and initrd.
this is a very short explanation, as I said it still needs some clean-up
and I will hopefully end up with a simple script or debian-package that
doeas all the work for you:
you need a debian installation to boot:
debootstrap lenny $INSTALLDIR http://ftp.de.debian.org/debian
chroot $INSTALLDIR
echo "deb http://www.backports.org/debian lenny-backports contrib main
non-free" >>/etc/apt/sources.list
apt-get update
apt-get install glusterfs-client debian-backports-keyring vim rsync
linux-image-amd64 dnsutils
Add a hook to have initramfs copy all necessary stuff to the initrd:
/etc/initramfs-tools/hooks/glusterfs
---cut---
#!/bin/sh
PREREQ=""
prereqs()
{
echo "$PREREQ"
}
case $1 in
prereqs)
prereqs
exit 0
;;
esac
. /usr/share/initramfs-tools/hook-functions
# Begin real processing below this line
copy_exec /sbin/mount.glusterfs
copy_exec /usr/sbin/glusterfs
copy_exec /sbin/ifconfig
copy_exec /usr/bin/host
test -d ${DESTDIR}/usr/lib/glusterfs/ || mkdir -p
${DESTDIR}/usr/lib/glusterfs/
cp -ar /usr/lib/gluster* ${DESTDIR}/usr/lib/
cp /lib/libnss_* /lib/libresolv* /lib/ld-linux-* /lib/libc.* ${DESTDIR}/lib
---cut---
Now, add a short script to the initrd that will be called automatically
when the boot=glusterfs kernel parameter is set (via pxe)
/etc/initramfs-tools/scripts/glusterfs
---cut---
mountroot()
{
echo 192.168.200.206 testserv1 >>/etc/hosts
echo 192.168.200.207 linux-netboot >>/etc/hosts
echo 192.168.200.206 >>/etc/resolv.conf
hostname linux-netboot
configure_networking
mkdir -p /var/log/glusterfs
sh /sbin/mount.glusterfs ${NFSROOT} ${rootmnt}
}
---cut---
run (in the chroot):
update-initramfs -u
to create a new initrd.
PXE-config on the server:
prompt 1
default lenny
timeout 10
label lenny
kernel vmlinuz
append vga=794 initrd=initrd boot=glusterfs nfsroot=192.168.200.206 ip=dhcp
(the kernel and initrd are symlinks to the glusterfs share that holds
the root-filesystem:)
lrwxrwxrwx 1 root root 24 18. Jun 20:29 /var/lib/tftpboot/initrd ->
/data/netboot/initrd.img
lrwxrwxrwx 1 root root 21 18. Jun 20:29 /var/lib/tftpboot/vmlinuz ->
/data/netboot/vmlinuz
Setting it up this way makes it even possible to do even kernel-updates
via apt-get on the netboot-client without having to touch anything on
the dhcp/pxe-server.
more to come... I need to finish some other important work first, but I
will hopefully be able to clean everything up by the end of next week
and have a howto and some script or package ready for others that are
interested.
Jan
Am 18.06.2010 09:57, schrieb Jan:
Hello Benjamin,
I think there should be more elegant ways to build the initrd.
I set up diskless servers that got their root-directory via nfs a few
years ago.
They shared the same installation (rootdir) and we had scripts in the
initrd to mount special directories under /var based on the hostname.
It was a little messy doing kernel-updates and updating everything.
Debian builds its initrd automatically after the installation of a new
kernel, based on your hardware and configuration. The tool used for
that (initramfs) has a lot of hooks to add your own modules or scripts.
I am sure it is possible to build a simple debian-package that adds
those scripts to your installation. Afterwards, you should even be
able to make a kernel-upgrade on the server although it is a net install.
Setting up a new installation should be as simple as starting
debootstrap (sets up a blank debian-installation (any architecture) in
a subfolder, can also install custom packages listed on the command
line), and then configure the tftp-server.
On the diskless server, you should have all possibilities (including
installation of packages, kernel-updates) that you have on any other
server that has its own disk.
I hope I'll have the time to have a closer look at this next week.
Jan
Am 17.06.2010 17:16, schrieb Benjamin Hudgens:
Here were the notes for us in Debian. Since these notes are mildly out
of context hopefully they simply give you a starting point. Rsync is
the tool we use to slurp the debian image. You manage gluster as you
would on a normal install. However, you manage the configs and such on
a copy at (X) location which is then slurped into ram on a boot. For
our dumb storage boxes we dynamically detect drives that need
filesystems, format, then add to gluster configs. Thus our server
configs get generated on-the-fly. This allows us to maintain a single
master instance of the OS image and deploy storage boxes quickly.
##############################
Summary:
We want to run a slim install of Debian in ram. Controlling how we
mount Gluster is simple. You manage this as you would a regular
install. Reboot any machine and the changes take effect. You can blow
away a machine at any time since the OS on our storage machines are
irrelevant and pull from our master copy.
Below remember that ${rootmnt} is pooled from kernel args for the PXE
boot.
Unpackage the initrd:
gzip -dc< ../initrd.img-2.6.30-2-686.netboot | cpio -i
Edit ./init
Modifications start at line 205:
------------
## Make a ramdisk
mkdir /ram
mount -t tmpfs -o size=2G tmpfs /ram
##Move the libraries over since rsync was failing initially. This just
grabs all the libraries from root
mount ${rootmnt}/lib /lib
mount ${rootmnt}/usr/lib /usr/lib
##Copy the filesystem into ram
${rootmnt}/usr/bin/rsync -rav ${rootmnt}/ /ram/
##Change the rootmnt point to our new location - the rest of the script
will take care of booting us up
rootmnt=/ram
------
Now re-create the initrd image:
find . -print0 | cpio -0 -H newc -ov | gzip -c>
../initrd.img-2.6.30-2-686.netboot
################################
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Jan
Sent: Thursday, June 17, 2010 6:30 AM
To: [email protected]
Subject: Re: [Gluster-users] Netboot / PXE-Boot from glusterfs?
Am 17.06.2010 13:15, schrieb Benjamin Hudgens:
Hello Jan,
Our company took the approach of slurping our OS into a ram drive and
then mounting file system points from Gluster. The OS becomes
expendable. In our case (large amounts of dumb storage machines) this
is okay. We were itching to get away from NFS. Boot time is slow
while
it reads directly from network -> ram. However, the final result is
an
OS that is extremely fast and no NFS dependency.
Obviously this approach is only applicable in certain situations.
Hello Benjamin,
Thanks, that's an interesting idea - as long as the root filesystem is
not too big.
But even then, one might be able to split it up in the most important
files and directories that are needed to boot and mount the
glusterfs-directories.
This needs some time to figure out what are the minimum files needed to
boot until the glusterfs-directories are available - and a minor change
in the initrd (mount NFS, create ramdisk, copy files). But definitely
worth a try.
thanks
Jan
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
