On Friday 04 Jan 2008, Barton C Massey spake thus:
> That's very helpful---thanks much! I imagine I'll go ahead
> and just make dirvish-expire do the right thing here,
On Sat, Jan 05, 2008 at 02:08:56PM +0100, Eric Mountain wrote:
> I disagree with your perception of the "right thing". See Paul's responses
> on
> this thread - they are spot on.
Perhaps there is something that Eric and Paul (and myself!) are
missing about Bart's and Tony's usage. That may be because that the
particular application is a poor match for dirvish/rsync capabilities,
or that dirvish/rsync is configured sub-optimally.
The two features of rsync that make dirvish worthwhile are its ability
to efficiently move data based on a pre-existing data set, and its
ability to efficiently use backup space to store multiple copies of
the data set. If one is merely making one-of copies, and replacing
them each time, rsync is the wrong tool. dirvish-expire is often the
wrong tool - it uses precious backup server time to delete inexpensive
copies. I don't use dirvish-expire myself - I have all my backup
copies back to the first day I used dirvish, years ago, on about 10
large disks (mostly in fireproof storage). But mine is an extreme case.
The important issue is to make sure you have plenty of backup space
(the docs suggest 3x what is being saved) and to store giant rapidly
changing files (like databases, mail spools, and vmware images) some
other way. Use branches to to store duplicates. The files that are
left pack very nicely - I use 500GB Seagate PATA backup drives, and
back up about 200GB of unique data from about 400GB of sources with
them. At current consumption rates on actively updated machines, I
can get about 300 images on those drives, without expiring. That
costs me about 30 cents US per day. Compared to what I once spent on
backup tapes ...
With expires, I could put images on the backup drives indefinitely,
until they fail. It is better to retire them while they are still
working. The backup drives get a lot more thrashing than the source
client drives, so they do fail more often.
I do backups primarily to retain data that could be accidentally or
maliciously deleted or modified. Sometimes this takes a while to
detect. So I want many copies over time. Even though one copy a
night, forever, is more than necessary, it does permit me to pinpoint
the precise day that something got modified, and I can easily
correlate that modification to the activity of that particular day;
automated upgrades, visiting users, security announcements, etc.
Sometimes this is months in the past.
For example, I don't look at all the files on the dirvish.org wiki
every day. Sometimes the wiki is vandalized, weeks in the past,
before I learn about it. But I have a daily image of the site all
the way back to the first day it was set up, so I can look at changes
to wiki pages in context, and restore legitimate changes made (for
example) 40 days ago, just before some idiot vandalized them 38 days
ago. If I did frequent expires, I could not do that.
So, before designing your dirvish backup system with aggressive
use of dirvish-expire, consider instead being clever with what you
back up, and how. Use enough backup disk - it is getting ridiculously
cheap - and plan on archiving backup drives before they fail. If you
have dirvish/rsync set up properly, and adequate disk space, you should
have tens or hundreds of images on your backup drives, and the loss of
one of those images should not make a big difference to you. If you
schedule expires properly, and write your expire rules correctly
(somebody please write up a wiki page about using dirvish-expire!) then
you will always have adequate recent and archival copies of your systems.
If you feel the expire tool needs to change (and I do), consider the
global purpose of expire: making more backup drive space, or making
image sets smaller and easier to copy. IMHO, deleting old images should
never be done by a daily script, it should be done with an interactive
tool that can help an experienced sysadmin understand the cost and benefit
of each deleted image. The easier-to-copy need is because existing file
system copy tools suck. So from a high level perspective, tweaking
on dirvish-expire to make it slightly less abusable is like putting
lipstick on a pig; it focuses attention on the wrong things. Worst
of all, if changes to dirvish-expire are inadequately tested, they
could do enormous damage to existing backups or to the backup server.
dirvish rule number one: "first, do no harm".
So, action items:
1) A better writeup on using dirvish-expire properly.
2) An interactive image-delete tool that estimates space saved with
each delete, and helps the sysadmin understand what may be lost.
3) Safe ways of thoroughly testing (2), and dirvish tools in general.
Keith
--
Keith Lofstrom [EMAIL PROTECTED] Voice (503)-520-1993
KLIC --- Keith Lofstrom Integrated Circuits --- "Your Ideas in Silicon"
Design Contracting in Bipolar and CMOS - Analog, Digital, and Scan ICs
_______________________________________________
Dirvish mailing list
[email protected]
http://www.dirvish.org/mailman/listinfo/dirvish
