Incremental backup solution. was: What logs etc do I need tocheckfrequently?
This solution sounds nice, I can even imagine setting up an additional machine (on the same location though) to have a somewhat galvanic isolation between the disks. Only fire, earthquake and a neutronbomb would affect such a backup solution. However, I could use a push in the right direction when it comes to how to configure and what software to use for achieving the incremental backup tasks. Could you hint me in how your system is doing this in a more detailed way? Cheers, Joachim --- | On Sun, 2003-12-28 at 10:27, Robert Huff wrote: | There are systems that will put 160 GB (uncompressed) on a | single tape ... they'll just run you $3000-3500. | If, on the other hand, you think of it as a yearly full dump | (split over multiple tapes) plus monthly incrementals then a DLT | 8000 ($1000 ??) at 40 GB (uncompressed) will do just fine. | | | Robert Huff | | I'd like to throw in my (home) solution here. | | I have had a dedicated file server on my home network for years. It | serves out files to clients on the network via SMB and HTTP. This | machine stores all of my permanent (and not so permanent) data and has | two large identical disks. Only the first is used. The other is used | strictly to back up the information on the first. A cron job runs a | script at 7AM every morning which powers up the backup disk, mounts it, | performs an incremental backup and then powers down the backup disk | again until the next morning. | | The moral: Buy double the amount of disk space that you think you'll | need or settle for half of what you can afford. Then force yourself to | use one half only to back up the other half. Disk-to-disk backup is | probably the best way to go for the home user. It's cheap and it's easy, | but it won't break the bank. Reliability is probably significantly less | than a $3k tape solution, but careful monitoring of the system and quick | response to potential problems can mitigate this to a large degree. | | Pretty soon I plan to move the backup disk to a separate machine on the | network that gets powered up each day by some kind of external timer. | The machine will power up, contact the file server, do an incremental | backup, then shut itself off. This would put me just one step short of a | complete daily off-site backup, all with hardware that is considered by | most to be obsolete. ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Incremental backup solution. was: What logs etc do I need tocheckfrequently?
On Mon, Dec 29, 2003 at 10:35:49AM +0100, Joachim Dagerot wrote: This solution sounds nice, I can even imagine setting up an additional machine (on the same location though) to have a somewhat galvanic isolation between the disks. Only fire, earthquake and a neutronbomb would affect such a backup solution. Before certain events in New York, we used to talk about hypothetical jumbo jets when considering our disaster plans. Secure off-site backups are a necessity. Take care thought that the off-site location really is secure. I did hear that some of the businesses in the World Trade Center had considered the other tower as a suitable location for their off-site backups. Cheers, Matthew -- Dr Matthew J Seaman MA, D.Phil. 26 The Paddocks Savill Way PGP: http://www.infracaninophile.co.uk/pgpkey Marlow Tel: +44 1628 476614 Bucks., SL7 1TH UK pgp0.pgp Description: PGP signature
Re: Incremental backup solution. was: What logs etc do I need tocheckfrequently?
| Before certain events in New York, we used to talk about hypothetical | jumbo jets when considering our disaster plans. Secure off-site | backups are a necessity. Take care thought that the off-site location | really is secure. I did hear that some of the businesses in the World | Trade Center had considered the other tower as a suitable location | for their off-site backups. I know a company whom's(?) office burned down to the ground. They where saved by the secretary who forgot to put the backup-tape in the safety box, instead she brought it in her handbag. ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Incremental backup solution. was: What logs etc do I need tocheckfrequently?
On Mon, 2003-12-29 at 04:35, Joachim Dagerot wrote: This solution sounds nice, I can even imagine setting up an additional machine (on the same location though) to have a somewhat galvanic isolation between the disks. Only fire, earthquake and a neutronbomb would affect such a backup solution. However, I could use a push in the right direction when it comes to how to configure and what software to use for achieving the incremental backup tasks. Could you hint me in how your system is doing this in a more detailed way? Cheers, Joachim I'd be glad to. First, it's actually a Linux system, though there's nothing particularly Linux-specific about it except the device names and the method of spinning down the backup disk after the job. The cornerstone of the solution is the rdiff-backup program (http://rdiff-backup.stanford.edu/ or in ports at /sysutils/rdiff-backup). rdiff-backup is a python script that mirrors one directory to another. It can do incremental backups and it can do them either locally or remotely. It's really a slick piece of software and I'm continually surprised that it doesn't get more publicity. First, there's the (trivial) script /usr/local/sbin/backup-share.sh. This is run by a daily cron job to backup directories on the disk that contain Important Data. Mine is very specific to my system. It is *not* pretty and I plan to overhaul it sometime soon to include error handling and an external config file. #!/bin/bash # script to automatically back up the important stuff on /nfs/share prog=/usr/local/bin/rdiff-backup src=/nfs/share dst=/backup/share budirs=code emu images media music school software text webpage mount /backup for dir in $budirs do $prog $src/$dir $dst/$dir done umount /backup # put backup drive in sleep mode since we won't be needing # it again for the next 24 hours or so hdparm -qY /dev/hdd A note about the last line: it appears that FreeBSD can only spin-down SCSI disks on command. (See camcontrol(8).) The best way to power down IDE disks seems to be just setting a suspend timeout in the power management section of your BIOS. Once the disk is unmounted, FreeBSD won't touch it thereafter and the system should put it in suspend mode automatically. The crontab entry looks like this: # backup selected dirs in /nfs/share @ 0730 daily 30 07 * * * sh /usr/local/sbin/backup-share.sh That's really about it. Like I said before, moving the backup disk to a separate machine would be trivial. If there are any questions, I'd be glad to answer them. Charles Ulrich -- http://bityard.net ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: What logs etc do I need tocheckfrequently?
| As you with good memories know, I lost 3000 pictures of my first sons | first year this month. I did have a RAID-5 system with fresh disks, | however, shit happens and I have a feeling that this could have been | avoided if I read my log files better. | | I'm sorry that you lost data. | | While you may have been able to notice the problem with the RAID-5 array in | time to do something, what you ought to do to avoid losing more data sometime | in the future involves making good backups-- not poring over the system log | files, not configuring RAID. I realise you are right. The thing is that this is a home system and I have (had!) around 230 GB of data that was non-replicable. I am not aware of a deasent backup system that can handle that amount of data. I will post another question about this later on. Thanks for your answers! ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: What logs etc do I need tocheckfrequently?
Joachim Dagerot writes: I realise you are right. The thing is that this is a home system and I have (had!) around 230 GB of data that was non-replicable. I am not aware of a deasent backup system that can handle that amount of data. There are systems that will put 160 GB (uncompressed) on a single tape ... they'll just run you $3000-3500. If, on the other hand, you think of it as a yearly full dump (split over multiple tapes) plus monthly incrementals then a DLT 8000 ($1000 ??) at 40 GB (uncompressed) will do just fine. Robert Huff ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: What logs etc do I need tocheckfrequently?
On Sunday 28 December 2003 10:27 am, Robert Huff wrote: Joachim Dagerot writes: I realise you are right. The thing is that this is a home system and I have (had!) around 230 GB of data that was non-replicable. I am not aware of a deasent backup system that can handle that amount of data. There are systems that will put 160 GB (uncompressed) on a single tape ... they'll just run you $3000-3500. Even better: the LTO2 will do 200GB uncompressed quite quickly. Here's one on e-bay for $3k: http://cgi.ebay.com/ws/ eBayISAPI.dll?ViewItemitem=2775174028category=3756 Nicholas If, on the other hand, you think of it as a yearly full dump (split over multiple tapes) plus monthly incrementals then a DLT 8000 ($1000 ??) at 40 GB (uncompressed) will do just fine. Robert Huff ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: What logs etc do I need tocheckfrequently?
On Sun, 2003-12-28 at 10:27, Robert Huff wrote: There are systems that will put 160 GB (uncompressed) on a single tape ... they'll just run you $3000-3500. If, on the other hand, you think of it as a yearly full dump (split over multiple tapes) plus monthly incrementals then a DLT 8000 ($1000 ??) at 40 GB (uncompressed) will do just fine. Robert Huff I'd like to throw in my (home) solution here. I have had a dedicated file server on my home network for years. It serves out files to clients on the network via SMB and HTTP. This machine stores all of my permanent (and not so permanent) data and has two large identical disks. Only the first is used. The other is used strictly to back up the information on the first. A cron job runs a script at 7AM every morning which powers up the backup disk, mounts it, performs an incremental backup and then powers down the backup disk again until the next morning. The moral: Buy double the amount of disk space that you think you'll need or settle for half of what you can afford. Then force yourself to use one half only to back up the other half. Disk-to-disk backup is probably the best way to go for the home user. It's cheap and it's easy, but it won't break the bank. Reliability is probably significantly less than a $3k tape solution, but careful monitoring of the system and quick response to potential problems can mitigate this to a large degree. Pretty soon I plan to move the backup disk to a separate machine on the network that gets powered up each day by some kind of external timer. The machine will power up, contact the file server, do an incremental backup, then shut itself off. This would put me just one step short of a complete daily off-site backup, all with hardware that is considered by most to be obsolete. Charles Ulrich -- http://bityard.net ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: What logs etc do I need tocheckfrequently?
On Sun, 28 Dec 2003 16:14:47 -0500 C. Ulrich [EMAIL PROTECTED] wrote: I have had a dedicated file server on my home network for years. It serves out files to clients on the network via SMB and HTTP. This machine stores all of my permanent (and not so permanent) data and has two large identical disks. Only the first is used. The other is used strictly to back up the information on the first. A cron job runs a script at 7AM every morning which powers up the backup disk, mounts it, performs an incremental backup and then powers down the backup disk again until the next morning. So why not use a cheap IDE RAID controller and do RAID1 ? I think it would be much safer, and reduce the amount of time needed to restore the system once a hard drive fails. We use RAID1 with a spare drive on our web and email servers here, and there's no downtime each time a drive fails, having put all the drives on hot-swap bays on a promise fasttrack controller. Greetings -- Stucchi Massimiliano | Gruppo Utenti FreeBSD Italia WillyStudios.com | http://www.gufi.org [EMAIL PROTECTED] | [EMAIL PROTECTED] People who make no mistakes do not usually make anything pgp0.pgp Description: PGP signature
Re: What logs etc do I need tocheckfrequently?
On Sun, 2003-12-28 at 16:30, Massimiliano Stucchi wrote: So why not use a cheap IDE RAID controller and do RAID1 ? I think it would be much safer, and reduce the amount of time needed to restore the system once a hard drive fails. We use RAID1 with a spare drive on our web and email servers here, and there's no downtime each time a drive fails, having put all the drives on hot-swap bays on a promise fasttrack controller. Greetings You could, and it would definitely give you an increase in the availability of the machine as you mention. One disk goes down, simply mount the good one and press on. What it doesn't give you, however, is a proper backup solution. Backups protect not only against disk failures but also mitigate the following: * Accidental or hasty deletions * Security compromise * Catastrophic overall system failure (such as a power surge that takes out everything connected to the motherboard). This, of course, depends on the backups being isolated from the system, which mine currently are not. RAID will not help you in any of these situations. My incremental backup solution allows me to retrieve an exact copy of the contents of the file server as it looked at any point in time, from when the very first backup was made (about two years ago) to the present. This has saved me more times than I care to remember. Charles Ulrich -- http://bityard.net ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
