-----BEGIN PGP SIGNED MESSAGE----- Hash: RIPEMD160 Hello, listserv!
A while ago I promised to post a lengthy description of a design I use in our z/VM environment, which is based on SLES9 and makes it possible to have z/VM users sharing complete root and /boot partitions and "own" as little as two MDISKs (or more, two is minimum), a 70-cylinder /srv containing /etc, and a 660 cylinder /var. While I somehow managed to not opt in for complete oblivion from friends and family yet by having stayed away from NSS, PAV and EIP (so far :)), I already do have a working setup with several clones of a master image which only run with a private /etc and /var partition. Not having done any measurements yet, I can only say that apart from DASD savings and more efficient cloning, the prime benefit of such a setup is faster recovery from major upgrades and z/VM related maintenance, but I'm sure you can think of additional benefits (do alert me if you can, anyways :)). The scripts come in two flavours, one for SLES9 GA and the other for SP3, since the meantime between these two release levels was spent on other projects, and the z/Linux not nearing production anywhere soon did not mandate installing the other two service packs. If the SP3 script is missing, there is no need for it to be fixed specially for that release level. I'm also attaching a preliminary document on concepts, maintenance-related issues and cloning procedures, which grew out of my on-the-fly documentation, and is nowhere near being complete yet - an entire section on user directory planning and organization is missing, as well as some sort of a final touch which would make it appropriate for beginners in the area of z/Linux as well, but for those of you willing to experiment a bit, it should more than do the trick. I'm open for any kinds of comments, suggestions on how to improve both the implementation as well as the concepts involved, and don't refrain from mindless rants about pointlessness of such barbaric acts against the corpus of a fine product like SLES9 either. :) In the hope it will assist you all in managing your penguin farms by at least providing some useful insight, here's my $0.1. Cheers, - -- ~ Grega Bremec ~ gregab at p0f dot net -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (GNU/Linux) iD8DBQFD07aLfu4IwuB3+XoRA28yAJ4+wPPMJCqqSzd0W8HETAxgXp6ImACeIGXW V1ARZ6shF3t2eSbecAmyDxg= =bSCq -----END PGP SIGNATURE----- ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
etc.tar.gz
Description: application/gunzip
******************************************************************************
* *
* Modifying SLES9: adding capability to boot from dislocated /etc. *
* *
******************************************************************************
------------------------------------------------------------------------------
0. Quickstart guide
------------------------------------------------------------------------------
TODO: z/VM user directory layout.
Bring the LNXMAINT user to single-user mode. LOGOFF all cloned users.
Copy /etc to /srv/etc or create a fresh MDISK that will only contain
/etc. Then remove everything but the following files from /etc:
- /etc/blkid.tab
- /etc/fstab
- /etc/group
- /etc/init.d/boot
- /etc/inittab
- /etc/ld.so.cache
- /etc/pam.d/login
- /etc/pam.d/run_init
- /etc/passwd
- /etc/rc.status
- /etc/shadow
- /etc/shells
- /etc/sysconfig/boot
Create a symlink on the root partition:
- /home -> /var/homes
Copy /etc/init.d/boot from the accompanying package to both root and
/etc partitions.
Copy the rest of the files to the per-user /etc partition.
Create copies of /etc and /var partitions for clones, LINKing to them and
activating them as necessary, then deactivating and detaching from them to
enable users' R/W access upon IPL.
Modify /etc/fstab on the cloned users' root, /boot and /etc partitions so
it contains a square zero in both fifth and sixth field (dump and passno).
Modify the following files to contain the cloned user hostnames and IP
addresses:
/etc/HOSTNAME
/etc/install.inf
/etc/sysconfig/network/ifcfg-hsi-bus-ccw-0.0.0a00
/etc/sysconfig/network/ifcfg-qeth-bus-ccw-0.0.0700
Remove the following files from guests' /etc partitions:
/srv/etc/ssh/ssh_host_key
/srv/etc/ssh/ssh_host_key.pub
/srv/etc/ssh/ssh_host_dsa_key
/srv/etc/ssh/ssh_host_dsa_key.pub
/srv/etc/ssh/ssh_host_rsa_key
/srv/etc/ssh/ssh_host_rsa_key.pub
IPL guests. Voila.
------------------------------------------------------------------------------
1. Concepts and functionality
------------------------------------------------------------------------------
A) USER DIRECTORY ORGANIZATION AND LAYOUT, VM-RELATED OPTIMIZATIONS
A.1) The LNXMAINT user
There has to be a VM user set up in order to hold the ownership of the
shared volumes, with the exclusive write right (MRV), which is only
taken advantage of when there is a need to do upgrades, while the rest
of the "penguin farm" are accessing those volumes in read-only mode
(RR).
In accordance with the sg246695 IBM RedBook (*), this user is called
LNXMAINT, is a descendant of the TCPCMSU profile with the addition of
the link to TCPMAINT's 0x592 minidisk (FTP and other commands). Local
addition to this RedBook concept is a private CMS profile minidisk
(0x191, label LNXCMS, mode A, MRV), which is used to store all the
z/Linux init files (VMRDR, INITRD and PARMFILE) for various releases
of z/Linux, which can be handy in installing new guests from scratch
(LINK LNXMAINT 191 191 RR), as well as some utility REXX EXECs.
LNXMAINT has the minimum required storage capacity (64M to 128M) to
both allow it to IPL z/Linux for basic administrative tasks and at the
same time keep storage overhead for running such a guest to a minimum,
as it is, essentially, only intended to ever run CMS.
LNXMAINT is a member of privilege classes B and G, and has options
MAINTCCW and RMCHINFO enabled in user directory, in order to be able
to perform some VM-related administrative tasks.
On another point, LNXMAINT's 0x150 and 0x151 MDISKs, which are being
shared with (or between) other users, have minidisk cache enabled. The
rest of the MDISKs in LNXMAINT have MDC disabled explicitly.
NOTE: DUE TO THE FACT MINIDISK CACHING IS BEING EMPLOYED IN THIS
SCENARIO, UTMOST CARE NEEDS TO BE TAKEN TO KEEP USER STORAGE
ALLOCATION AT A LEVEL WHICH STILL PROVIDES ENOUGH FOR MDC
AS AN EXAMPLE MEASURE:
SET MDCACHE SYSTEM ON
SET MDCACHE STORAGE 0M 128M
SET MDCACHE XSTORE 0M 256M
WITH THOSE VALUES, ALWAYS HAVE 128M OF REAL STORAGE AVAILABLE!
TODO: PROFILE EXEC for IPLing z/Linux guests so they may automatically
format swap V-DISKs at 0x140 and do other cool stuff.
* - The sg246695 RedBook, a.k.a.
z/VM and Linux on zSeries:
From LPAR to Virtual Servers in Two Days
A.2) The LNXUPGR user
As detailed in subsection D below, there will be an occasional need to
perform a massive upgrade of all packages on the system, which is, of
course, quite over the top with tens of clones running off of a shared
partition, as they would all have to be shut down and stay that way
during the entire upgrade process, have their private storage updated,
configurations fixed and then be reIPL'd one by one just to stay on
the safe side of the Linux filesystem cache.
Hence, when planning for such a directory layout, it is extremely
advisable to plan in a near-replica of the LNXMAINT user, LNXUPGR,
which can be used as both the subject of upgrade and a stand-in for
LNXMAINT in such upgrade scenarios, in order to mitigate the problems
outlined above.
See subsection D below for more details on this.
A.3) The "penguin farm"
The so-called "penguin farm" may vary wildly in shared DASD patterns,
however, the most important common point remains: all virtual device
addresses affected by shared DASD scheme, connectivity devices and the
common elements of z/Linux IPL process MUST BE THE SAME across entire
farm of servers.
In other words, the following devices defined in the user directory
MUST adhere to the scheme set forth below:
a. To be able to IPL CMS
CMS CODE DISKS LINK MAINT 0190 0190 RR
LINK MAINT 019D 019D RR
LINK MAINT 019E 019E RR
CMS PROFILE DISK LINK LNXMAINT 0191 0191 RR
b. Due to shared DASD and CMS PROFILE requirements
SWAP V-DISK MDISK 140 FB-512 V-DISK ${blksize} WRV
ROOT FILESYSTEM LINK LNXMAINT 0150 0150 RR
/boot FILESYSTEM LINK LNXMAINT 0151 0151 RR
c. Since a shared DASD root doesn't provide for per-guest configuration
/srv FILESYSTEM MDISK 152 3390 ${begin} 0070 ${volume} MRV ALL
WPASS MPASS
(containing etc/, ftp/, root/ and www/)
d. To simplify system management and cloning
/var FILESYSTEM MDISK 153 3390 ${begin} 0660 ${volume} MRV ALL
WPASS MPASS
(containing homes/, among others)
/tmp FILESYSTEM --bind mounted from /dev/shm, if sufficient
space is allocated in /dev/shm
ETH OSA INTERFACE DEDICATE 700 ${raddr1}
DEDICATE 701 ${raddr2}
DEDICATE 702 ${raddr3}
HSI OSA INTERFACE DEDICATE A00 ${raddr1}
DEDICATE A01 ${raddr2}
DEDICATE A02 ${raddr3}
B) CLONING PRIVATE STORAGE FOR A NEW GUEST
Choose a cloned system you trust or have the fixed configuration files
somewhere handy on LNXMAINT storage, so you can overwrite LNXMAINT's
configuration files that you've just replicated to a new clone.
Assuming the private MDISKs for a new clone have already been declared
in the user directory, link to them in WR mode, either from CP or from
z/Linux using /sbin/hcp:
$ /sbin/hcp LINK TO CLONE 0152 0252 WR <WPASS>
$ /sbin/hcp LINK TO CLONE 0153 0253 WR <WPASS>
CMS FORMAT, CMS RESERVE or dasdfmt(8) the devices, if not yet done so.
Activate the newly attached devices in /proc/dasd/devices:
$ echo 1 > /sys/bus/ccw/devices/0.0.0252/online
$ echo 1 > /sys/bus/ccw/devices/0.0.0253/online
or, having s390-tools installed,
$ /sbin/chccwdev -e 0.0.0252
Setting device 0.0.0252 online
Done
$ /sbin/chccwdev -e 0.0.0253
Setting device 0.0.0253 online
Done
Just what /dev/dasd* device it is can be established from the sysctl
file called /proc/dasd/devices using this pipe of commands:
$ grep 0.0.0252 /proc/dasd/devices | sed 's/^.* is //; s/ .*//'
dasdf
$ grep 0.0.0253 /proc/dasd/devices | sed 's/^.* is //; s/ .*//'
dasde
Create a filesystem on both new devices' first partition, if not yet
done so.
Migrate /etc and /var to the new clone's devices:
$ mkdir /tmp/srv /tmp/var
$ mount /dev/dasdf1 /tmp/srv
$ mount /dev/dasdg1 /tmp/var
$ tar cC /srv/ ./ | tar xpC /tmp/srv
$ tar cC /var/ ./ | tar xpC /tmp/var
Modify the configuration as outlined in subsection C below.
Unmount the devices, deactivate them, and detach from them in CP.
IPL the new guest, et voila!
C) CLONING-RELATED MODIFICATION OF NEW GUESTS
If the aforementioned guidelines for defining guests in user directory
have been followed, the only files in urgent need of modification on
guests' new /etc filesystem (in order to prevent hostname and IP
conflicts) are the following ones:
/srv/etc/fstab
/srv/etc/HOSTNAME
/srv/etc/install.inf
/srv/etc/sysconfig/network/ifcfg-hsi-bus-ccw-0.0.0a00
/srv/etc/sysconfig/network/ifcfg-qeth-bus-ccw-0.0.0700
Most of these files simply need to have hostnames and associated IP
information modified so they contain values assigned by the admin.
As mentioned in section 3 below though, /etc/fstab is in a rather
desperate need to have shared storage (that is, / and /boot) fsck
passno and dump fields set to zero, so they aren't checked during the
IPL process, as the cloned user only ever has read access to these
MDISKs, which would have hence inevitably caused otherwise harmless,
but annoying and quite nasty-looking error messages.
The following files need to be removed so they can be recreated upon
first boot of the new guest (they MUST stay distinct to provide for
host authentication):
/srv/etc/ssh/ssh_host_key
/srv/etc/ssh/ssh_host_key.pub
/srv/etc/ssh/ssh_host_dsa_key
/srv/etc/ssh/ssh_host_dsa_key.pub
/srv/etc/ssh/ssh_host_rsa_key
/srv/etc/ssh/ssh_host_rsa_key.pub
Depending on the packages installed, other files might need to be
modified as well:
/srv/etc/postfix/main.cf
In general though, the above lot is all that needs to be modified to
ensure the first IPL will render the cloned guest available from a
remote location, which then in turn makes it possible to use YaST2 for
all additional configuration.
D) UPGRADING LNXMAINT WITH A FIXPACK
Having LNXMAINT undergo a major upgrade, not just a small number of
packages but tens or hundreds of them, such as installing a fixpack or
a major software package to be used by all guests, is best done by
creating a copy of the entire disk space assigned to LNXMAINT to a
new user such as LNXUPGR. LNXMAINT and/or its descendants may then be
left running in an unchanged state until the upgrade process has been
verified to succeed with the LNXUPGR user. This also helps avoid
massive downtime while it still allows for a graceful transition of
individual guests in a relatively short time.
After having performed the upgrade successfully, one can simply diff
both users' /etc and /var directories and perform the necessary change
using a script in the rest of the farm, on a one-by-one basis, linking
the descending VM users' root and boot partitions to the new LNXUPGR's
MDISKs, reIPLing them from LNXUPGR's shared storage until finished
with the entire upgrade process, and then finally copying the contents
of LNXUPGR's fresh partitions to the now unused LNXMAINT's assigned
space, and abandoning the LNXUPGR user until next major upgrade.
A different cool strategy is to have all modified files symlinked to
a directory lying underneath /etc, for example /etc/local.d, which
contains only the modified files, which may then simplify the process
of migrating to a new version, as one may hold it for certain that the
rest of /etc needs absolutely no modification, making it possible to
just rsync(1) /etc directories (for example over a HSI link) with the
appropriate exclude option (in order for local.d not to be overridden)
and then compare the important files for any significant changes that
need to be resolved manually.
------------------------------------------------------------------------------
2. Files being modified:
------------------------------------------------------------------------------
The following files have had their contents altered in order to make IPL
from a dislocated /etc directory possible:
INITIAL RAMDISK (/boot/initrd)
- /linuxrc
Nothing much changed, there's just a fix for "umount -n" of /proc
and /sys, which made the process of giving control over to the
kernel hiccup; the "-n" option, which only suits the purpose of
not writing to /etc/mtab and which the initrd version of mount(8)
did not understand, was removed (this was actually a bugfix for
the GA version of SLES9, not an adjustment specific to this
alteration).
PS1: v9sp3: The initrd does not need any modification AT ALL, finally.
The only nuisance, the above -n with umount, was taken care
of. Rejoice, rejoice! :)
ORIGINAL (BAREBONES) /etc ON THE ROOT PARTITION
*AND*
CUSTOMIZED (SHADOW) /etc ON THE /etc PARTITION
- /etc/init.d/boot
This is where the main thing goes on. Don't fix it (I already
did that for you :)), because it's probably going to break. It's
just too arcane. Or at least read section 1 above prior to
attempting to do so.
This is where the support for additional zipl.conf arguments is
implemented, and this is the script responsible for a private
/etc appearing where it should in the first place.
PS1: v9sp3: The upgrade does not touch /etc/init.d/boot in any way.
CUSTOMIZED (SHADOW) /etc ON THE /etc PARTITION
- /etc/init.d/boot.shm
Adding an extra --bind to mount /dev/shm on /tmp as well, or lots
of things will break, because /tmp is on rootfs, which is mounted
read-only.
PS1: v9sp3: Reverts boot.shm to the original version. Just a new copy
of the modified script is needed.
- /etc/init.d/boot.rootfsck
*sigh* Yes, SLES9 assumes the root filesystem is always going to
be mounted read-write. :-( So these fixes, while they go to the
customized /etc mounted AFTER root is attached, ensure you don't
end up with a situation completely different from one specified in
your "shadow" /etc/fstab mounted afterwards.
Also, the manner in which the removing of the traces of root
filesystem manipulations taking place during rootfsck is done,
also removes all trace of any filesystems being mounted prior to
it. The script simply removes /etc/mtab in the original version.
GRRR. Fixed though.
Another fix being done in boot.rootfsck is taking care that the
root filesystem is not checked on clones, that is, on systems with
fsck_passno field of root filesysteme in /etc/fstab set to zero.
There is no point in doing fsck on a device linked to a user in
RR mode.
PS1: v9sp3: There are some changes between v9ga and v9sp3 versions of
boot.rootfsck, namely enabling swap files so that swapon
does not complain about non-existant devices, adding some
stuff for VOLID mounts (blkid) and moving ROOTFS_BLKDEV to
/dev/shm/root; either patch manually or using a diff, but
beware of using too much context as >3 lines will b0rk.
- /etc/init.d/boot.localfs
Again, removing traces of all sorts of filesystem manipulations
taking place during localfs removes all trace of any filesystems
being mounted prior to it. Whoever thought removing mtab is just
swell ought to be shot. GRRR. Fixed thanks to /proc/mounts. :)
Also, procfs, tmpfs and devpts are attempted at being mounted
again here; as most of this had already been done though, and all
this script wants to do is just a vain attempt of making up for
the ruthless removal of /etc/mtab, we kill this attempt if there's
no need to mount those (i.e. /etc/init.d/boot and boot.shm).
PS1: v9sp3: blkid handling has changed, as well as the failed fsck
message emitted by this script. Also, sysfs is now mounted
unconditionally. Patch manually as sysfs breaks diffs.
- /etc/zipl.conf
Adding the new boot configuration, a menu, and three new options:
etc=<block-special>[,<mountpoint>]
etc=<mountpoint>
This must either be a device specified in a form which
mount(8) will understand, or, it may also be an existing
mount point from fstab(5), which the script will access
(without writing to /etc/mtab, if necessary) and mount in
/etc using the --bind option of mount(8).
A special case is the full first form of the argument,
which explains to mount(8) what block special device to
mount where, and then --bind /etc from there onwards.
Useful for getting around the chore of fstab manipulation.
etcopt=<mnt-options>
A comma-separated list of mount options you want the
filesystem to be mounted with. It also applies to mount
with --bind, as a complement, and can be used to override
the standard fstab-specified options (in the short form of
the etc= parameter).
etcfst=<fstype>
Be careful with this one, as it can really hose your
system to no avail. It was added to complement the long
etc= form of the argument, where mount(8) will only be
passed a block special device and a mount point, and will
hence complain that "you didn't specify a filesystem type"
and fall back to ext3, yet show up as "unknown" in
/etc/mtab; if you don't like that, specify a fstype here,
but make sure it's the right one or everything will break.
IPLing the user is now being performed in a slightly different
sequence than before, namely, because of the menu. One option is
this:
IPL <vdev> LOADPARM <x>
where <vdev> is the virtual device address you want to IPL from
and <x> is the sequence number of the predefined boot config for
ZIPL. The other option of IPLing the machine is to first just
start off the same way as always:
IPL <vdev>
and then, when the boot menu appears, follow it by the CP VINPUT
command:
#CP VINPUT VMSG <x>
or
#CP VI VMSG <x>
PS1: v9sp3: The upgrade does not touch /etc/zipl.conf in any way.
------------------------------------------------------------------------------
3. Mandatory "barebones" /etc files:
------------------------------------------------------------------------------
After the initial ramdisk had given control over to init(8), the following
extra files will need to be present in the "barebones" /etc if you want to
ensure a successful boot, because they're being accessed and used by tools
PRIOR to the "real" /etc actually being mounted:
- /etc/blkid.tab (*)
- /etc/fstab (**)
- /etc/group (***)
- /etc/init.d/boot
- /etc/inittab
- /etc/ld.so.cache
- /etc/pam.d/login (***)
- /etc/pam.d/run_init (***)
- /etc/pam.d/su (***)
- /etc/pam.d/sudo (***)
- /etc/passwd (***)
- /etc/rc.status
- /etc/security/pam_unix2.conf (***)
- /etc/shadow (***)
- /etc/shells (***)
- /etc/sudoers (***)
- /etc/sysconfig/boot
- /home -> /var/homes
These files have not in any way been modified by this conversion process.
* blkid.tab will usually be regenerated by the blkid program, if enabled
in /etc/sysconfig/boot by setting DISABLE_BLKID="no"; probably its
new contents will be stored into the new /etc, but no harm is done
if the old file is left lingering around just in case the boot script
needs it for some perverse task.
** if you want to use the short etc=<mpoint> form of --bind version to
overlay /etc with the contents of some other mountpoint (or its etc/
subdirectory), you will need an /etc/fstab record on the *old* /etc,
revealing to mount(8) the whereabouts of this mount point; the other
option is to use the etc=<blkdev>,<mpoint> form of the boot parameter
IMPORTANT NOTE:
IF YOU'RE USING THE SHORT etc=<mpoint> FORM OF --bind VERSION, OR YOU
HAVE ANY INFORMATION ON YOUR LONG etc=<blkdev>,<mpoint> DEVICE PRESENT
IN THE *NEW* /etc/fstab, MAKE SURE THE FILESYSTEM IS MARKED AS NOCHECK
*AND* noauto IN THE *NEW* /etc/fstab!
ANOTHER IMPORTANT NOTE: :)
/etc/fstab ON CLONED IMAGES OF THE MASTER IMAGE **MUST** HAVE THE DUMP
AND FSCK PASSNO FIELDS FOR THE SHARED PARTITIONS **DISABLED**; EXPECT
ALL SORTS OF WEIRD ERRORS UPON IPL OTHERWISE.
*** it might be handy to have at least the basic super-user information
present in the login files in case something goes wrong; the boot
process will probably revert to /sbin/sulogin in that case, and you
*will* want to login to fix stuff. :) Another case where these files
come in handy is inadvertently unmounting /etc from an unprivileged
user, using su(1) or sudo(8); there is no way back, if these files
are missing.
----------------------------------------------------------------------
For LINUX-390 subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
