> Have anyone of you experienced a crash of the server that holds most
> of the RW volumes? Does the AFS volume backups include the "acls" of each
> volumes and subdirectories? or you have to set them manually after restoring
> a backup? Have any one use ADSM with AFS?
> I'm not in that situation right now, but I want to make sure we are
> prepare. Your comments are appreciated
I'm sure most AFS sites have had server crashes. Even if all
things AFS-wise are sane, power glitches, OS bugs, human mistakes,
or other external factors generally ensure a system crash sooner or
later... Generally, a computer crash does not necessitate restoring
a disk - fsck & salvage generally suffice. Occasionally, something
bad will have happened, and it will be found that the salvager has
savaged one or two volumes, leaving only a few shreds behind.
This can be detected by reviewing the SalvageLog's after every
restart. This is relatively rare, but it can happen.
Occasionally, of course, "something worse" happens - and by
your concern over "restoring a backup", I presume you are most
worried about a head crash. Ie, something that renders the
data completely and permamently inaccessible on the disk.
Indeed, this is more painful. Generally, it means the data
on the disk isn't fully accessible for at least a day.
It's safe to say that any large AFS site has had this happen
often enough to have it down to routine - an annoying thing
that may even happen as often as once every few months,
depending on the size, activity, & quality of disks purchased.
With a small new site, you're talking about a mean time between
failure of likely several years - long enough that the first
crash may be a distant future worry, or perhaps even an
exciting excuse to upgrade machines.
An AFS backup of a volume (either made with "vos backup" or
on an AFS backup tape as made by the AFS backup utilities),
contains all the file acl's and all other information associated
with the volume. Nearly the same mechanism is used to move
volumes between file servers ("vos move") so you can be
nearly positive that *all* of the information contained in
the volume will be restored exactly as it was.
I don't know what ADSM is. Perhaps some background information
on how AFS files are stored will help. AFS uses Unix inodes
to store files, but it does NOT use the Unix directory structure.
It also pulls one other "trick"; in most implementations, it
"steals" some unused fields in inodes to store supplemental information
on the file. This is necessarily OS dependendent. Of
course, in order to access the partition with the inodes, it
must be mounted. In AFS, these are called "vice" partitions,
and once mounted, will appear as /vicepa, /vicepb, etc. If viewed with
"ls", it will appear to have one large root directory, with many
files named V and a large number (the volume #), and
that will be it. The V### files contain the volume header,
which only contains per-volume information. All the other
files and directories that belong to the volume aren't named
in any Unix directory, and cannot be seen from Unix.
The fileserver uses a special "iopen" system call that is
patched into the kernel, in order to access these extra
inodes. Since these inodes don't have Unix filenames,
if you were to run an unmodified Unix "fsck" on the disk,
they would all be freed, & their storage reclaimed, which
is why you instead run a special "AFS" fsck or fsck helper
to check the consistency of the disk.
A Unix backup program that goes "inode-by-inode" and stores *all* of
the data of the inode *May* store sufficient information to do
a backup of the disk. Programs such as Unix restore, tar, cpio,
which *rely* on Unix pathnames, will not work properly with an AFS
vice partition. Any program which works either at the inode or
partition level, will not know enough about AFS volumes to be
very "useful", unless it has been trained to know about AFS; as
it will not allow AFS volume by AFS volume restores. I've
heard of several replacmenets for the AFS backup suite that
claim to do a "better" job of managing backups, I don't know
how good any of them are.
So much for AFS fileservers. AFS database servers have
special and separate backup requirements. This includes
the kerberos, pt, and vl databases. These are NOT stored
in weird inodes, but as perfectly ordinary files in /usr/afs/db/,
and yes, they SHOULD be backed up, on a regular basis. The
backups made of the kerberos database should be treated
with special paranoia. These backups contain *every*
kerberos key in your cell, and if a bad person gained access
to these keys, they could make life *very* miserable for
you and every user in your cell. Any backups of
database or fileservers that contain /usr/afs/etc/KeyFile
also have the same security concern - anyone who knows
the key of afs in your cell can forge afs service tickets
for any user in your cell, including admin. Tapes of these files
should probably be kept locked up, just as you (should) be keeping
your database and file servers physically secure and out of reach of all
but a select few trusted individuals. Before making a backup
copy of these files, you should probably use "bos" to stop the
server instances - if you make a backup copy of the running files,
you run the risk that if a change is made to the files in the
middle, you might get an inconsistent copy of the files.
Alternatively, you could use "udebug" before & after to
verify that the version# hasn't changed. The tools to repair
these files are somewhere between not very good and non-existant,
so you shouldn't want to save any inconsistent copies.
Other things worth mentioning:
If you have to restore AFS incrementals, you should take
care to tell users not to change their data "in the middle"
after the old full is restored, but before the incremental(s)
are applied. Otherwise, bad things can happen.
If you have a large cell, the AFS backup database (the database
that says where all the AFS backups of a volume live) can become
corrupt and need to be restarted.
It is worthwhile to replicate the top levels of your file
hierarchy, if at all possible. You can spread multiple RO
copies of your top level volumes amongst several servers, and
thus continue to have access to much of your data even if one
server crashes.
It is always worthwhile to practice and plan *before* the real thing.
Consider, for instance, having a backup database server, backup
fileserver, and backup disks, ready to sling into production.
Definitely have lots of extra blank tapes. Test things out ahead of
time - some weekend, turn off a database server, push it into a corner,
and see if you can put the backup database server into service. Try
using "vos backup" to save & restore a test volume. Play with
the "backup" utilities to learn how many on-tape copies of your home
directory exist. See if you can find the tapes too. Before putting
commissioning a new fileserver, try populating a partition with test
data, back it up to tape, then turn if off, pretend it crashed, & see
if you can restore the data elsewhere. If you have multiple brands
of fileservers, can you restore data on one brand that was
backed up on the other? Worry about what happens if your data center
catches fire and burns to the ground. Do you have any off-site backup
storage? Consider renting a bank safety deposit box - 8mm tapes are
awfully compact, after all.
-Marcus Watts
UM ITD PD&D Umich Systems Group
