Re: dump(8), incremental backups, Tower of Hanoi sequence, don't get it

2005-08-25 Thread Scott Mitchell
On Wed, Aug 24, 2005 at 02:37:08PM -0700, Danny Howard wrote:
 All this great discussion got me researchinng.  I haven't tried this
 out but it looks like rsnapshot integrates a lot of features like this
 into a single configurable, cronable script.  It is in ports as well.
 
 A lot of systems make use of cp -al ... well, for us FreeBSD people that
 means gcp from coreutils.
 
 rsnapshot looks like a lightweight, OS/FS-portable method of building
 rotating filesystem-wide snapshots via hardlinks, but can be made to
 operate on limited sets of directories, etc.  It can create local
 snaphots of remote directories, but not, apparently, remote copies of
 local directories.  One trick I gleaned from
 http://burd.info/gary/2003/03/snapshot-backup-using-rsync-and-ssh.html
 is to invoke rsync with --rsync-path which points to a script which
 performs maintenence functions and then passes off to rsync proper, so
 you could probably set up a client-triggered rsnapshot configuration if
 you were, say, doing backups of a Windows laptop client. :)
 
 -danny

That sounds pretty cool - probably better than my homebrewed scripts :-)
I will have to check it out.

Cheers,

Scott

-- 
===
Scott Mitchell   | PGP Key ID | Eagles may soar, but weasels
Cambridge, England   | 0x54B171B9 |  don't get sucked into jet engines
scott at fishballoon.org | 0xAA775B8B |  -- Anon
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: dump(8), incremental backups, Tower of Hanoi sequence, don't get it

2005-08-24 Thread Scott Mitchell
On Tue, Aug 23, 2005 at 11:35:35AM -0700, Philip Hallstrom wrote:
 2) If all you have to deal with are static files and a
 not-super-giant-filesystem, use rsync.  rsync -avz --delete once a night
 will mirror your data between drives or between machines without any
 trouble.  The only disadvantage is there is no file retention if you
 want to restore a corrupt / deleted file after the fact.
 
 Actually there is... sort of...  The trick is to use the --backup and
 --backup-dir options:
 
 -b, --backup
 With  this  option, preexisting destination files are renamed as each
 file is transferred or deleted.  You can control where  the backup  file
 goes  and what (if any) suffix gets appended using the --backup-dir and
 --suffix options.
 
 --backup-dir=DIR
 In combination with the --backup option,  this  tells rsync  to store
 all backups in the specified directory. This is very use- ful for
 incremental backups.  You  can  additionally specify  a backup  suffix
 using  the  --suffix option (otherwise the files backed up in the
 specified directory will  keep  their original filenames).

Another useful rsync option is --link-dest:

--link-dest=DIR
  This  option  behaves  like --copy-dest, but unchanged files are
  hard linked from DIR to the destination  directory.   The  files
  must be identical in all preserved attributes (e.g. permissions,
  possibly  ownership)  in  order  for  the  files  to  be  linked
  together. 

I use this to rsync each backup into a new directory on the backup volume,
named for the date of the backup.  The result is a directory for each backup
run, apparently containing every file from the source tree - but most of them
are just hard links to some previous backup.  This saves a *lot* of disk
space as you only ever copy files that have changed since the last backup.
You also have the complete file tree for each backup, so it's trivial to
restore the version of a file that existed on any particular date.

Cheers,

Scott

-- 
===
Scott Mitchell   | PGP Key ID | Eagles may soar, but weasels
Cambridge, England   | 0x54B171B9 |  don't get sucked into jet engines
scott at fishballoon.org | 0xAA775B8B |  -- Anon
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: dump(8), incremental backups, Tower of Hanoi sequence, don't get it

2005-08-24 Thread Danny Howard
On Wed, Aug 24, 2005 at 12:32:38PM +0100, Scott Mitchell wrote:

 Another useful rsync option is --link-dest:
 
 --link-dest=DIR
   This  option  behaves  like --copy-dest, but unchanged files are
   hard linked from DIR to the destination  directory.   The  files
   must be identical in all preserved attributes (e.g. permissions,
   possibly  ownership)  in  order  for  the  files  to  be  linked
   together. 
 
 I use this to rsync each backup into a new directory on the backup volume,
 named for the date of the backup.  The result is a directory for each backup
 run, apparently containing every file from the source tree - but most of them
 are just hard links to some previous backup.  This saves a *lot* of disk
 space as you only ever copy files that have changed since the last backup.
 You also have the complete file tree for each backup, so it's trivial to
 restore the version of a file that existed on any particular date.

All this great discussion got me researchinng.  I haven't tried this
out but it looks like rsnapshot integrates a lot of features like this
into a single configurable, cronable script.  It is in ports as well.

A lot of systems make use of cp -al ... well, for us FreeBSD people that
means gcp from coreutils.

rsnapshot looks like a lightweight, OS/FS-portable method of building
rotating filesystem-wide snapshots via hardlinks, but can be made to
operate on limited sets of directories, etc.  It can create local
snaphots of remote directories, but not, apparently, remote copies of
local directories.  One trick I gleaned from
http://burd.info/gary/2003/03/snapshot-backup-using-rsync-and-ssh.html
is to invoke rsync with --rsync-path which points to a script which
performs maintenence functions and then passes off to rsync proper, so
you could probably set up a client-triggered rsnapshot configuration if
you were, say, doing backups of a Windows laptop client. :)

-danny

-- 
http://dannyman.toldme.com/
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: dump(8), incremental backups, Tower of Hanoi sequence, don't get it

2005-08-23 Thread Danny Howard
On Fri, Aug 19, 2005 at 05:15:35PM +0300, Ilari Laitinen wrote:
 Hello.
 
 Lately I have been getting more and more worried about data on my
 FreeBSD box at home. I am forming a real small-scale backup policy
 with two different big USB harddrives (yet to buy) storing regular
 incremental backups (yet to figure out). The idea is to have those
 harddrives mirror each other for extra security.

Ilari,

Three suggestions:

1) Use gmirror to mirror the hard drives, instead of manually mirroring
them.

2) If all you have to deal with are static files and a
not-super-giant-filesystem, use rsync.  rsync -avz --delete once a night
will mirror your data between drives or between machines without any
trouble.  The only disadvantage is there is no file retention if you
want to restore a corrupt / deleted file after the fact.  Maybe you
could rsync between the USB disks prior to rsyncing your backups, then
you have two iterations of data ...

3) Look into using AMANDA, which I think can be run without a tape
device, if it regards your disks as long-term holding disks.  It is
probably overkill for you, but at least then you get to learn a more
powerful backup package which you might want to have experience with for
the future, and you won't have to worry about Towers of Hanoi or
anything because AMANDA is smart enough to figure this out itself.

Though, honestly, probably all that you need is to do 0 1 1 1 or such.

Or possibly, mount filesystem snapshots and if you get acceptible
performance just dump level 0 every night.

Or ... really it depends what you want and what you've got.  Do you want
to be able to restore deleted files from the week prior?  If not, then
you can get away with nightly full dump or nightly full mirror
strategies.  Otherwise, yes, you need to rig up some multiple-dumplevel
magic.

Cheers,
-danny


-- 
http://dannyman.toldme.com/
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: dump(8), incremental backups, Tower of Hanoi sequence, don't get it

2005-08-23 Thread Philip Hallstrom

2) If all you have to deal with are static files and a
not-super-giant-filesystem, use rsync.  rsync -avz --delete once a night
will mirror your data between drives or between machines without any
trouble.  The only disadvantage is there is no file retention if you
want to restore a corrupt / deleted file after the fact.


Actually there is... sort of...  The trick is to use the --backup and
--backup-dir options:

-b, --backup
With  this  option, preexisting destination files are renamed as each
file is transferred or deleted.  You can control where  the backup  file
goes  and what (if any) suffix gets appended using the --backup-dir and
--suffix options.

--backup-dir=DIR
In combination with the --backup option,  this  tells rsync  to store
all backups in the specified directory. This is very use- ful for
incremental backups.  You  can  additionally specify  a backup  suffix
using  the  --suffix option (otherwise the files backed up in the
specified directory will  keep  their original filenames).

I use this to maintain a one day full archive and a two week archive of 
dailys.


---
#
# manage daily directory cleanup
#
rm -rf $dailysDir-old
if [ -d $dailysDir ]; then
mv $dailysDir $dailysDir-old
fi

#
# rsync command
#
rsync=rsync
rsync=$rsync --archive# archive mode, equivalent to -rlptgoD
rsync=$rsync --delete# delete files that don't exist on sender
rsync=$rsync --delete-after# delete after transferring, not before
rsync=$rsync --delete-excluded# also delete excluded files on receiver
rsync=$rsync --devices# preserve devices (root only)
rsync=$rsync --group# preserve group
rsync=$rsync --links# copy symlinks as symlinks
rsync=$rsync --owner# preserve owner (root only)
rsync=$rsync --perms# preserve permissions
rsync=$rsync --recursive# recurse into directories
rsync=$rsync --relative# use relative path names
rsync=$rsync --safe-links# ignore links outside the destination tree
rsync=$rsync --sparse# handle sparse files efficiently
rsync=$rsync --stats# give some file transfer stats
rsync=$rsync --times# preserve times
rsync=$rsync --whole-file# copy whole files, no incremental checks

$rsync \
--compress \
--files-from=files/bravo.files \
--exclude-from=files/bravo.exclude \
--backup --backup-dir $dailysDir/bravo \
bravo.mydomain.com:/ $backupDir/bravo

$rsync \
--compress \
--files-from=files/foxtrot.files \
--exclude-from=files/foxtrot.exclude \
--backup --backup-dir $dailysDir/foxtrot \
foxtrot.mydomain.com:/ $backupDir/foxtrot

---

files/bravo.files looks like this:

/etc
/local/home
/root
/usr/local/etc
/var/cron
/fs

and files/bravo.exclude looks like this:

/fs/tmp/
/fs/software/


This gives me a backup directory that looks like this:

[EMAIL PROTECTED]:/backups
% ll data
drwxr-xr-x  8 root  wheel  512 Mar 19  2004 bravo/
drwxr-xr-x  7 root  wheel  512 Oct 12  2004 foxtrot/

[EMAIL PROTECTED]:/backups
% ll dailys/
drwx--  7 root  wheel 512 Aug 19 03:37 Fri/
drwx--  7 root  wheel 512 Aug 12 03:43 Fri-old/
drwx--  7 root  wheel 512 Aug 22 03:37 Mon/
drwx--  6 root  wheel 512 Aug 15 03:37 Mon-old/
drwx--  7 root  wheel 512 Aug 20 03:36 Sat/
drwx--  7 root  wheel 512 Aug 13 03:42 Sat-old/
drwx--  6 root  wheel 512 Aug 21 03:36 Sun/
drwx--  6 root  wheel 512 Aug 14 03:36 Sun-old/
drwx--  7 root  wheel 512 Aug 18 03:36 Thu/
drwx--  7 root  wheel 512 Aug 11 03:39 Thu-old/
drwx--  8 root  wheel 512 Aug 23 03:36 Tue/
drwx--  7 root  wheel 512 Aug 16 03:36 Tue-old/
drwx--  7 root  wheel 512 Aug 17 03:37 Wed/
drwx--  7 root  wheel 512 Aug 10 03:39 Wed-old/

[EMAIL PROTECTED]:/backups
% ll dailys/Fri

drwx--  3 root  wheel  512 Aug 19 03:30 bravo/
drwx--  3 root  wheel  512 Aug 19 03:30 foxtrot/
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: dump(8), incremental backups, Tower of Hanoi sequence, don't get it

2005-08-22 Thread Alex Zbyslaw

Ilari Laitinen wrote:


On Fri, Aug 19, 2005 at 04:22:20PM +0100, Alex Zbyslaw wrote:
 


This pretty much cleared it up. Now that I read the manual page again,
enlightened, it seems quite easy to follow. Nice.

Using the algorithm above I get the following:

SequenceDumps needed
0 3 0 3
0 3 2   0 2
0 3 2 5 0 2 5
0 3 2 5 4   0 2 4
0 3 2 5 4 7 0 2 4 7
0 3 2 5 4 7 6   0 2 4 6
0 3 2 5 4 7 6 9 0 2 4 6 9
0 3 2 5 4 7 6 9 8   0 2 4 6 8
0 3 2 5 4 7 6 9¹8 9²0 2 4 6 8 9²

Am I doing this right? Every time a dump of level N is, eh, taken,
earlier tapes of level N become obsolete and are free to go(*). In this
case, that happens every other time.

(*) Unless one would like to have those file versions around for a
   longer time, of course.
 

Yes, that looks correct.  Like I said, for a lightly used computer you 
might want to keep it simpler.  It really depends how *big* the files 
which change are and how big your backup disk is compared to your real 
one.  You can figure out what was dumped on each backup using restore 
-ivf {path_to_backup} and typing ls* at the prompt.  A simpler scheme 
makes it easier to find a random single file which you deleted by 
accident, as opposed to recovering your whole disk.


If you are backing up to disk, then you could also consider compressing 
you backups.  Instead of doing


dump -NuaL -f [path_to_backup]

you could do

dump -Nual -f - | gzip -9  [path_to_backup].gz


Then to check what is on it:

gunzip -9 [path_to_backup].gz | restore -ivf -

(*) Actually, that will just show the top level directories.  What you 
get is a *very* simple shell with ls, cd and a few restore specific 
options.  Depressingly, restore isn't set up to use the readline 
library, so you get no command editing.



clip

I would also consider doing your backups daily, not weekly as your 
example suggests.  The timing of full backups depends on how busy your 
machine is.  Anything from weekly to quarterly.
   



Well, I am the only active user on this computer. And I know when there
is something to back up, so it will be a bit irregular in reality. If I
only surf the Net all weekend long, there is nothing to worry about. Or
if I am not physically around, the computer will have no power to mess
with.
 

That's fine.  Certainly nothing changes when the machine is off :-)  
Beware of I was just surfing the net though, as you may have had email 
coming in, and have bookmarked somewhere that you'll never manage to 
find again :-)  Regrettably, even trivial sessions can generate 
important data.  But like you say, that's your call.  From experience I 
know that there is a strong tendency for laziness to kick in.


--Alex


___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


dump(8), incremental backups, Tower of Hanoi sequence, don't get it

2005-08-19 Thread Ilari Laitinen
Hello.

Lately I have been getting more and more worried about data on my
FreeBSD box at home. I am forming a real small-scale backup policy
with two different big USB harddrives (yet to buy) storing regular
incremental backups (yet to figure out). The idea is to have those
harddrives mirror each other for extra security.

Handbook reads dump(8) is the best backup program there is. So I am
giving it a try - only to find out that I don't understand at all the
meaning of that modified Tower of Hanoi algorithm descibed in the
manual page and elsewhere. The manual page says it is an efficient
method of staggering incremental dumps to minimize the number of
tapes. I just don't get the picture here.

So, could somebody please give an idiot-proof explanation why 3 2 5
4 7 6 9 8 9 9 is such a tape-number-minimizing dump level sequence
(with helpful examples, if at all possible)? How does it work?

Am I relatively safe doing level 0 dump every two months and
increasing dump level for weekly backups like the following, given
two separate harddrives storing them?

DateDump level
2005-09-01  0
2005-09-08  1
2005-09-15  2
...
2005-10-27  8
2005-11-03  0

Thanks.

Ilari Laitinen,
dumb dump newbie lost in this big world of ever-so-failing disks

-- 
Ilari Laitinen - [EMAIL PROTECTED] - http://iki.fi/ilari.laitinen/


pgpIxoGSMrqv8.pgp
Description: PGP signature


Re: dump(8), incremental backups, Tower of Hanoi sequence, don't get it

2005-08-19 Thread Alex Zbyslaw

Ilari Laitinen wrote:


Handbook reads dump(8) is the best backup program there is. So I am
giving it a try - only to find out that I don't understand at all the
meaning of that modified Tower of Hanoi algorithm descibed in the
manual page and elsewhere. The manual page says it is an efficient
method of staggering incremental dumps to minimize the number of
tapes. I just don't get the picture here.

So, could somebody please give an idiot-proof explanation why 3 2 5
4 7 6 9 8 9 9 is such a tape-number-minimizing dump level sequence
(with helpful examples, if at all possible)? How does it work?

Am I relatively safe doing level 0 dump every two months and
increasing dump level for weekly backups like the following, given
two separate harddrives storing them?

DateDump level
2005-09-01  0
2005-09-08  1
2005-09-15  2
...
2005-10-27  8
2005-11-03  0
 

No, your sequence is the worst possible.  If you have a crash on  
2005-10-27 then you will need to recover files from *every* dump  from 
your last level 0.


A level 0 dumps everything. 


A level 1 everything since the last 0

a level 2, everything since the last 0 or 1

a level 3 everything since the last 0, 1 or 2

A level 4, everything since the last 0, 1, 2, or 3

etc.

The idea is is to make the numbers rise and fall to minimise the number 
of backups needed to do a full restore.  Write yourself some sequences 
and figure out for yourself which ones you would need for a full 
backup.  Try to figure out for each backup whether the same files will 
be dumped by a later backup.  They will, if a later backup number is 
*lower*.


The algorithm your aiming to create is:
   Start with a level 0 and ignore everything before.
   from end of list, find the lowest number before you reach the 
starting dump.  You'll need this backup.  Make it the new start of list.
   from end of list, find the lowest number before you reach the 
starting dump.  You'll need this backup.  Make it the new start of list.

   etc.

E.g. Given 0 3 2 5 4 7 6 9

To restore everything you need the 0, 2, 4 and 6.  I.e. every second 
dump.  You'll see that wherever you stop in that sequence, no more than 
3 backups are required to recover everything.


I have never liked the 9 9 9 trailing off to infinity.  If you are not 
doing a full backup by then, then you should be restarting your sequence 
from the dump after the full backup.  E.g. 3 2 5 4 7 6 9 8 then back to 
3 2 5 4 7 6 9 8 and repeat.


Personally I use a much shorter sequence: 0 1 3 2 1 3 2 1 3 2.  But 
that's because I don't usually change vast amounts of data.


I would also consider doing your backups daily, not weekly as your 
example suggests.  The timing of full backups depends on how busy your 
machine is.  Anything from weekly to quarterly.


--Alex

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: dump(8), incremental backups, Tower of Hanoi sequence, don't get it

2005-08-19 Thread Jerry McAllister
 
 Hello.
 
 Lately I have been getting more and more worried about data on my
 FreeBSD box at home. I am forming a real small-scale backup policy
 with two different big USB harddrives (yet to buy) storing regular
 incremental backups (yet to figure out). The idea is to have those
 harddrives mirror each other for extra security.

It really depends on the size of your file systems to back up, the size 
of your media to receie the backup files, the speed of the backup device
and how much data changes during any given interval between backups.

If your media is big enough to hold a complete backup and there is 
not a lot of data change between backups, then doing a full (leval 0)
dump once per week and a level 1 on other days would be fine.

If your filesystem is much bigger than your media or your data changes
a lot between backup, then you will want to implement a more complex
scheme, such as a modified version of the plan in the handbook.

Someone else has pointed out that level 0 gets everythingm level 1
gets everything that has changed since the last level 0, level 2 gets
everything since the last level 1, etc. 

My suggestion is to plan your backups so that you do a level 0 at a 
specific interval, such as once per week - oftener if you have critical 
stuff changing frequently.   It is nice to have media of a size that one 
unit will handle the whole level 0 dump, but that may not be possible.   
You may need to use multiple tapes/DVDs/disk for it.

Then schedule higher level (1...5) dumps so that the lowest level possible 
just fills one unit of your media.  You may even be able to use a smaller
size media unit for the intervening higher level dumps (we call them change
dumps).It is even possible that you can fit more than one change
dump on a single unit of media.   (If you are using sequential media (tape)
be careful with managing your skip forwards [mt -f /dev/nsa0 1 or 2 or 3, etc] 
to make sure you use the no rewind device (nra0) or you may overwrite
something).

Another consideration when you choose change dump levels is how
many levels you have to restore to completely rebuild a mangled file
system,   Using fewer levels - maybe just doing level 0 with only 
level 1's in between, makes restore the most simple, but can make
for unnecessarily large level 1 dumps as the interval between level 0
dumps grows longer and more files are changed and created.   

If your 'change' dumps are larger than one media unit, then your problem
is, of necessity, larger and more complex than the simple scheme in the
handbook covers well.   You will have to think it out, still using the
basic concerns I mention above, eg a periodic full dump, intervened by
a scheme of progressively higher level change dumps to minimize the
number of media units your must use for each.   

Remember that just as higher level dumps can intervene for a leval 0 dump,
dumps higher than 1 can intervene level 1 dumps - eg another way of saying
that level 2 or higher dumps only backup files that have changed since
the last level 1 dump, level 3 only backs up data that has changed since
the most recent level 2 or 1 or 0 dump, whichever was most recent, etc.

It is a little mind fogging to just say it.   It can help to draw it
out on paper with circles and arrows.  But it does work.   

This doesn't specifically explain the towers of Hanoi scheme in
the handbook and man page, but I hope it provides some background
to help understand what they are trying to accomplish and that this
will help clarify it.

jerry


 Handbook reads dump(8) is the best backup program there is. So I am
 giving it a try - only to find out that I don't understand at all the
 meaning of that modified Tower of Hanoi algorithm descibed in the
 manual page and elsewhere. The manual page says it is an efficient
 method of staggering incremental dumps to minimize the number of
 tapes. I just don't get the picture here.
 
 So, could somebody please give an idiot-proof explanation why 3 2 5
 4 7 6 9 8 9 9 is such a tape-number-minimizing dump level sequence
 (with helpful examples, if at all possible)? How does it work?
 
 Am I relatively safe doing level 0 dump every two months and
 increasing dump level for weekly backups like the following, given
 two separate harddrives storing them?
 
 DateDump level
 2005-09-01  0
 2005-09-08  1
 2005-09-15  2
 =2E..
 2005-10-27  8
 2005-11-03  0
 
 Thanks.
 
 Ilari Laitinen,
 dumb dump newbie lost in this big world of ever-so-failing disks
 
 --=20
 Ilari Laitinen - [EMAIL PROTECTED] - http://iki.fi/ilari.laitinen/
 
 --huq684BweRXVnRxX
 Content-Type: application/pgp-signature
 Content-Disposition: inline
 
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v1.4.1 (FreeBSD)
 
 iD8DBQFDBemHQj4nNFSfK+YRAjX0AKCf/Ut/gyvfvKXEUkH69Q0jSV5yQwCfYqOT
 d5wx5asYHJqZju/cA2pnrJc=
 =lGaX
 -END PGP SIGNATURE-
 
 --huq684BweRXVnRxX--
 

___
freebsd-questions@freebsd.org mailing 

Re: dump(8), incremental backups, Tower of Hanoi sequence, don't get it

2005-08-19 Thread Charles Swiger

On Aug 19, 2005, at 11:22 AM, Alex Zbyslaw wrote:
[ ...a good explanation snipped... ]
Personally I use a much shorter sequence: 0 1 3 2 1 3 2 1 3 2.  But  
that's because I don't usually change vast amounts of data.


I would also consider doing your backups daily, not weekly as your  
example suggests.  The timing of full backups depends on how busy  
your machine is.  Anything from weekly to quarterly.


Start with something even easier:

Do a full-level-0 once per week, or once per month.
Do level-1 incrementals daily.

Change this to something fancier if you're using too much tape, but  
this is easy to understand


--
-Chuck

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: dump(8), incremental backups, Tower of Hanoi sequence, don't get it

2005-08-19 Thread Ilari Laitinen
On Fri, Aug 19, 2005 at 04:22:20PM +0100, Alex Zbyslaw wrote:
 Ilari Laitinen wrote:
 
 Handbook reads dump(8) is the best backup program there is. So I am
 giving it a try - only to find out that I don't understand at all the
 meaning of that modified Tower of Hanoi algorithm descibed in the
 manual page and elsewhere. The manual page says it is an efficient
 method of staggering incremental dumps to minimize the number of
 tapes. I just don't get the picture here.
 
 So, could somebody please give an idiot-proof explanation why 3 2 5
 4 7 6 9 8 9 9 is such a tape-number-minimizing dump level sequence
 (with helpful examples, if at all possible)? How does it work?
 
 Am I relatively safe doing level 0 dump every two months and
 increasing dump level for weekly backups like the following, given
 two separate harddrives storing them?
 
 DateDump level
 2005-09-01  0
 2005-09-08  1
 2005-09-15  2
 ...
 2005-10-27  8
 2005-11-03  0
  
 
 No, your sequence is the worst possible.  If you have a crash on  
 2005-10-27 then you will need to recover files from *every* dump  from 
 your last level 0.
 
 A level 0 dumps everything. 
 
 A level 1 everything since the last 0
 
 a level 2, everything since the last 0 or 1
 
 a level 3 everything since the last 0, 1 or 2
 
 A level 4, everything since the last 0, 1, 2, or 3
 
 etc.
 
 The idea is is to make the numbers rise and fall to minimise the number 
 of backups needed to do a full restore.  Write yourself some sequences 
 and figure out for yourself which ones you would need for a full 
 backup.  Try to figure out for each backup whether the same files will 
 be dumped by a later backup.  They will, if a later backup number is 
 *lower*.
 
 The algorithm your aiming to create is:
Start with a level 0 and ignore everything before.
from end of list, find the lowest number before you reach the 
 starting dump.  You'll need this backup.  Make it the new start of list.
from end of list, find the lowest number before you reach the 
 starting dump.  You'll need this backup.  Make it the new start of list.
etc.
 
 E.g. Given 0 3 2 5 4 7 6 9
 
 To restore everything you need the 0, 2, 4 and 6.  I.e. every second 
 dump.  You'll see that wherever you stop in that sequence, no more than 
 3 backups are required to recover everything.

This pretty much cleared it up. Now that I read the manual page again,
enlightened, it seems quite easy to follow. Nice.

Using the algorithm above I get the following:

SequenceDumps needed
0 3 0 3
0 3 2   0 2
0 3 2 5 0 2 5
0 3 2 5 4   0 2 4
0 3 2 5 4 7 0 2 4 7
0 3 2 5 4 7 6   0 2 4 6
0 3 2 5 4 7 6 9 0 2 4 6 9
0 3 2 5 4 7 6 9 8   0 2 4 6 8
0 3 2 5 4 7 6 9¹8 9²0 2 4 6 8 9²

Am I doing this right? Every time a dump of level N is, eh, taken,
earlier tapes of level N become obsolete and are free to go(*). In this
case, that happens every other time.

(*) Unless one would like to have those file versions around for a
longer time, of course.

 clip

 I would also consider doing your backups daily, not weekly as your 
 example suggests.  The timing of full backups depends on how busy your 
 machine is.  Anything from weekly to quarterly.

Well, I am the only active user on this computer. And I know when there
is something to back up, so it will be a bit irregular in reality. If I
only surf the Net all weekend long, there is nothing to worry about. Or
if I am not physically around, the computer will have no power to mess
with.

Thank you, Alex, and others who replied (Jerry, Charles)! Now I only
have to buy those harddrives to start my new, shiny life with less fear
for random data loss. :)

Ilari

-- 
Ilari Laitinen - [EMAIL PROTECTED] - http://iki.fi/ilari.laitinen/


pgpqktrrGrKSr.pgp
Description: PGP signature


Re: dump(8), incremental backups, Tower of Hanoi sequence, don't get it

2005-08-19 Thread David Israelsson
Ilari Laitinen [EMAIL PROTECTED] writes:

 Handbook reads dump(8) is the best backup program there is. So I am
 giving it a try - only to find out that I don't understand at all the
 meaning of that modified Tower of Hanoi algorithm descibed in the
 manual page and elsewhere. The manual page says it is an efficient
 method of staggering incremental dumps to minimize the number of
 tapes. I just don't get the picture here.

 So, could somebody please give an idiot-proof explanation why 3 2 5
 4 7 6 9 8 9 9 is such a tape-number-minimizing dump level sequence
 (with helpful examples, if at all possible)? How does it work?

I have asked myself the same question, why is the 3 2 5 4 6 5 9 8 9 9
... considered a good algorithm for storing backups?  We will get
exactly the same result by storing the dumps like 2 2 3 3 4 4 5 5 ...

I did ask this question elsewhere, and got the answer that it is
likely that somebody got it slightly wrong when rewriting some ancient
dump man page (of course I cannot find that man page right now).  The
algorithm describes labels on the tapes and which tapes to keep for
how long time, rather than dump levels.  This makes sense to me, can
someone supply a link to the man page I am talking about?
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]