Re: [SLUG] Backup notes from Mary's talk (28 Nov)
Hi, On Wed, Dec 10, 2008 at 12:01 AM, Sridhar Dhanapalan srid...@dhanapalan.com wrote: Are there any good tools to backup/restore databases, with versioning, delta storage and so on just like you get with rdiff-backup? I'm currently running mysqldump and pg_dump in a shell script via cron job, but surely there's a better way. I've seen a few tools, but what do people on the list recommend? If you are using LVM - http://lenz.homelinux.org/mylvmbackup/ (15gb db snapshot takes roughly 2-3 seconds) I believe there is a similar script floating around for PgSQL, however I have not used it. http://www.postgresqlforums.com/wiki/Backup__Recovery also has some docs on doing point in time recovery. -- Paul De Audney -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Backup notes from Mary's talk (28 Nov)
2008/11/28 Mary Gardiner [EMAIL PROTECTED]: If you're backing up databases (eg, MySQL for your blog or whatever) make sure to dump them before backing up: live database backups seldom restore well, you need dumps. Are there any good tools to backup/restore databases, with versioning, delta storage and so on just like you get with rdiff-backup? I'm currently running mysqldump and pg_dump in a shell script via cron job, but surely there's a better way. I've seen a few tools, but what do people on the list recommend? -- Bring choice back to your computer. http://www.linux.org.au/linux -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Backup notes from Mary's talk (28 Nov)
Sridhar Dhanapalan wrote: Are there any good tools to backup/restore databases, with versioning, delta storage and so on just like you get with rdiff-backup? I'm currently running mysqldump and pg_dump in a shell script via cron job, but surely there's a better way. I've seen a few tools, but what do people on the list recommend? the closest we have is backupninja - its just a shell wrapper that saves having a million different shells scripts. So you make a dump with it and then rdiff that dump directory. That works pretty well for us (having had to do lots of restores using it). http://riseuplabs.org/backupninja/ Backupninja allows you to coordinate system backup by dropping a few simple configuration files into /etc/backup.d/. Most programs you might use for making backups don't have their own configuration file format. Backupninja provides a centralized way to configure and schedule many different backup utilities. I know thats not what you were after, but this is the better way to manage backups we are using now... thanks dave -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Backup notes from Mary's talk (28 Nov)
2008/12/10 Sridhar Dhanapalan [EMAIL PROTECTED]:
2008/11/28 Mary Gardiner [EMAIL PROTECTED]:
If you're backing up databases (eg, MySQL for your blog or whatever)
make sure to dump them before backing up: live database backups seldom
restore well, you need dumps.
Are there any good tools to backup/restore databases, with versioning,
delta storage and so on just like you get with rdiff-backup? I'm
currently running mysqldump and pg_dump in a shell script via cron
job, but surely there's a better way. I've seen a few tools, but what
do people on the list recommend?
I've heard about database data and schema versioning tools over the
years but unfortunately can't remember anything off the top of my head
right now. It's worth googling - there is stuff to be found (e.g.
{mysql,postgresql} incremental backup comes up with interesting
links).
(Looking through my google bookmarks I found the following which might
help versioning database schemas:
http://search.cpan.org/~crakrjack/DBIx-Migration-Directories/)
One thing I remember (but haven't yet got around to implement myself)
is to backup the database logs - that way you get full point-in-time
restoration capability as well as relatively more efficient backup
(both in time and in space), and it's contineous so can be used to
recover up to the point that disaster struck.
Here is a specific link from the MySQL docs:
http://dev.mysql.com/doc/refman/5.0/en/backup.html
and from the PostgresQL docs:
http://www.postgresql.org/docs/8.2/interactive/continuous-archiving.html
Cheers,
--Amos
--
SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/
Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Backup notes from Mary's talk (28 Nov)
On Saturday 29 November 2008 10:00:05 [EMAIL PROTECTED] wrote: the partial inspiration for the talk), although rsync doesn't save older data, which I definitely recommend. Very minor nitpick, rsync can save older data, and put them in whatever dir you want. before rdiff-backup i used to use rsync --backup --backup-dir=$date- based-dir I'd really appreciate anybody explaining why rdiff does a better job than rsync in my situation: I honestly can't see anything at all: My server (tigger) runs 24/7 The other important machines do a bios wake at 7am At 7:05 the backup machine (elsewhere in the house) via cron does an rsync to all the other machines, then shutsdown. Every month or so I run a version that uses rsync --delete, so all old files are kept on the backup machine until manually cleared Only new files are added, old files are kept until manually deleted, and the only possible flaw I perceive is that ImportantFile is backed up, trashed and tomorrow the trashed version is saved blotting out the original. The daily backup is quick. Having pondered the doco I cannot see any benefit other than saving ImportantFile at the cost of quite a lot more complexity. What have I missed? Thanks James -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Backup notes from Mary's talk (28 Nov)
On Sat, Nov 29, 2008 at 06:41:44PM +0900, jam wrote: On Saturday 29 November 2008 10:00:05 [EMAIL PROTECTED] wrote: the [snip] Having pondered the doco I cannot see any benefit other than saving ImportantFile at the cost of quite a lot more complexity. What have I missed? can't tell if you use --link-dest, which allows for a history of changes Thanks James -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html -- I want to thank you for coming to the White House to give me an opportunity to urge you to work with these five senators and three congressmen, to work hard to get this trade promotion authority moving. The power that be, well most of the power that be, sits right here. - George W. Bush 06/18/2001 Washington, DC signature.asc Description: Digital signature -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Backup notes from Mary's talk (28 Nov)
Yay Mary, Was the last time that a backup talk was given my one in 1997? I see the basic principles are the same, but disc technology has moved on: I didn't even consider backing up to an external hard drive ten years ago, because a 4G DAT was cheaper than a 4G hard drive; and discs were around 2G in general. We now have main memory bigger than that! (In 1997 I ran Linux 0.98 on a 50MHz 486 with 1G disc and 1M memory) I'm now using rsnapshot; really important data gets burned to CD and stored off-site. Really important means the best of the photos, audio recordings, etc., and stuff there's legal requirements on (7 years for data for the tax return, etc). There's a lot less really important stuff than there is stuff on the disc. -- Dr Peter Chubb http://www.gelato.unsw.edu.au peterc AT gelato.unsw.edu.au http://www.ertos.nicta.com.au ERTOS within National ICT Australia A university is a non-profit organisation only in the sense that it spends everything it gets ... Luca Turin. -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Backup notes from Mary's talk (28 Nov)
On Fri, Nov 28, 2008 at 10:56 AM, Mary Gardiner [EMAIL PROTECTED] wrote: Hi all, As promised here are the backup notes from my talk at SLUG on 28 Nov. Materials related to this talk are at http://users.puzzling.org/users/mary/Presentations/SLUG2008/ (including a version of these notes). A note about style: this is a set of recommendations purely based on the fact that I have both backed my home data up AND recovered it. And having some working backup regime is better than none. I don't claim this is the One Best Way, merely One of the Adequate Ways That Isn't Entirely Maddening. [..snip..] Hi Mary, Thanks for sharing this with us. I had purchased a 500GB drive with an external SATA connector just for this purpose. I installed Bacula and started configuring it, then left it not-fully-setup and basically the drive had been sitting there for some time, simply because Bacula is awesome but it's a little too much for my needs. I think rdiff-backup is what I want. Just wondering: while I was running rdiff-backup, I noticed a lot of files were backed up that I could either delete or exclude from the next backup. Can I just simply go into my /media/disk and delete them? I guess I should get rid of any incremental files, if they exist. The idea was to backup a windows laptop too so I'll see how I go doing that. There is a native Win32 version of rsync-backup so I'm going to give that a shot. Failing that I'm going to try backing up via samba (using smbmount). Cheers Gonzalo -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
[SLUG] Backup notes from Mary's talk (28 Nov)
Hi all, As promised here are the backup notes from my talk at SLUG on 28 Nov. Materials related to this talk are at http://users.puzzling.org/users/mary/Presentations/SLUG2008/ (including a version of these notes). A note about style: this is a set of recommendations purely based on the fact that I have both backed my home data up AND recovered it. And having some working backup regime is better than none. I don't claim this is the One Best Way, merely One of the Adequate Ways That Isn't Entirely Maddening. A note about me: I am sadly short of time at the moment and will not participate in the thread following this post (should there be one), and I can't give one-on-one help to design your personal backup regime. Sorry about that: hopefully slug@slug.org.au can help you out. The talk was on backups for home users. It doesn't cover mission-critical or business-grade backups. --- The magical 10 second version --- If you don't have backups, you should. Here's how: 1. Go out, right now, and buy an external hard drive as big as, or bigger than your main hard drive. Yep, there is no free lunch. 2. Install the program called rdiff-backup. 3. Plug in the drive and run: sudo rdiff-backup --exclude-other-filesystems ::/ ::/media/disk (/media/disk being the place your external drive mounted, under Ubuntu, substitute as needed) 4. Run that as often as you can. (Every so often, run sudo rdiff-backup --force --remove-older-than 60D ::/media/disk or similar to delete very old backups.) See http://jwz.livejournal.com/801607.html for someting similar (and the partial inspiration for the talk), although rsync doesn't save older data, which I definitely recommend. --- More about rdiff-backup --- Do check out the webpage and man rdiff-backup for full details. http://www.nongnu.org/rdiff-backup/ In summary, it's 'reverse' increments, if you will. That is, you can get the most recent backup just by looking at the filesystem under /media/disk. Older versions are recovered by rdiff-backup applying older and older chnages incrementally to the files, and are recovered like so: sudo rdiff-backup -r 1D /media/disk/path-to-file [destination you'd like to restore to] --- Why you need backups --- You may not want to protect against all of these things: some of them are expensive or time-consuming to protect against. But consider these risks when deciding on your backup regime. 1. Accidental deletion: very common. An on-site backup is good enough to recover from this. 2. Media failure (dead hard drive). This will happen to you, sooner or later. You may or may not get any warning. An on-site backup *on a different disk* is good enough to recover from this. Not a different partition, a different *disk*. This is the only one RAID helps with too provided that (a) you have a full mirror on the other disk(s) in the array and (b) you don't stuff it up somehow and set the new empty disk as the master. RAID is not a substitute for backups. *Not* a substitute for backups. It won't help with 1, 3, 4 or 5. 3. Software failure. Say some bit of code, from the drive firmware to the filesystem to the end user software (eg GIMP) has a bug in it and writes out your data incorrectly. In most cases this is rather like accidental deletion, but if the bug is very low level (kernel) it may affect the backup too. At the very least, have your backup drive be not the same manufacturer and model as your main drive. This makes them less likely to share the same bugs and less likely to fail at the same time. 4. Provider failure. You have uploaded your valuble data to Flickr, LiveJournal, WordPress.com etc etc. They go bust, and their creditors swoop in, turn their machines off and sell them for scrap parts. This really happens, see http://blogs.zdnet.com/digitalcameras/?p=362 for an example. Smaller examples are the occasional data loss that a lot of web services, up to and including those run by Google, have. 5. Massive local failure. Flood, fire, surge: we had victims of two of these at the meeting. And someone who had had all their computer equipment stolen in one go. To recover from these you need an(other) backup, as far away from your main data store as you can. At least in a different suburb. Another country is entirely possible these days, if you have broadband. --- Media recommendations --- For home users, get another hard drive and backup to that. Optical media: no. CDs and DVDs are too small for most people now. You will have to insert at least 5 of the things for a full backup cycle. So that's boring and dull, so you'll never do it. Also, they have a short-ish lifespan and testing their backup goodness is even *more* boring and dull, so you definitely won't ever do that. Solid state media: no. Consumer grade SSDs are really really unreliable right now. You need to back them up, not use them *for* your backups! See
Re: [SLUG] Backup notes from Mary's talk (28 Nov)
Mary Gardiner wrote: 2. Install the program called rdiff-backup. All excellent suggestions Mary. Sorry I missed the talk. As an additions to rdiff-backup, people might benefit from using backupninja. Its a front end to rdiff-backup and other backup tools (like mysqldump). That makes it easier to configure. http://riseuplabs.org/backupninja/ Also make sure to check out the rdiff-backup wiki. there is a ton of scripts and hints and tips on there to make using rdiff-backup easier. The rdiff-backup list has been working hard to make rdiff-backup work on windows natively now too. We have used winsshd to pull backups off windows boxes with rdiff-backup successfully. http://www.bitvise.com/winsshd (not Free) CAUTION 2: The remote backup space is more than a little crowded right now. Keep an eye on your provider, some are undoubtedly headed for failure. These guys have been active on the rdiff-backup mailing lists in the past http://rsync.net/ and support rdiff-backup as a backup target. I don't actually use them, but they seem to have their stuff together when it comes to offsite backups. Again, I second all the recommendations in this talk - it matches my commercial and personal experience very closely :) thanks Dave -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Backup notes from Mary's talk (28 Nov)
On 28/11/2008, at 11:56 PM, Mary Gardiner wrote: See http://jwz.livejournal.com/801607.html for someting similar (and the partial inspiration for the talk), although rsync doesn't save older data, which I definitely recommend. Very minor nitpick, rsync can save older data, and put them in whatever dir you want. before rdiff-backup i used to use rsync --backup --backup-dir=$date- based-dir -- http://chesterton.id.au/blog/ http://barrang.com.au/ -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
