Re: [ubuntu-uk] Simple backup script
On Thu, 2 Jun 2011 17:28:57 +0100, Chris Rowson wrote: I've been tinkering with backups and backup rotation today and I have come across many wierd and wonderful backup scripts of varying complexity. snip I use s3sync [0] to just sync my whole (chosen) directory/directories to an Amazon bucket once a week via a cron job. [0] http://s3sync.net/wiki -- Jon Reynolds (j0nr) http://www.jcrdevelopments.com -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.ubuntu.com/UKTeam/
Re: [ubuntu-uk] Simple backup script
On 02/06/11 17:28, Chris Rowson wrote: I've been tinkering with backups and backup rotation today and I have come across many wierd and wonderful backup scripts of varying complexity. Is there anything wrong with using something simple like this? (except of course for the lack of validation). Nothing wrong with that. From a research perspective I would recommend that you take a look at a rather old (but still functional script) I use called automysqlbackup[1]. It has some great demonstrations of various tricks to use in bash scripting... Cheers Al [1]http://sourceforge.net/projects/automysqlbackup/ -- The Open Learning Centre http://www.theopenlearningcentre.com -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.ubuntu.com/UKTeam/
Re: [ubuntu-uk] Simple backup script
On 02/06/11 17:28, Chris Rowson wrote: I've been tinkering with backups and backup rotation today and I have come across many wierd and wonderful backup scripts of varying complexity. Is there anything wrong with using something simple like this? (except of course for the lack of validation). Basically, let's delete anything over 7 days old and then make a new backup. Hi, Chris. I use this ETOH (Extended Towers of Hanoi) backup that I contributed to NEBC Bio-Linux. It provides a baseline level 0 dump and three months of incremental backups. You can get it as a deb from NEBC: http://nebc.nerc.ac.uk/tools/bio-linux/ Bye, Tony. -- Dr. A.J.Travis, University of Aberdeen, Rowett Institute of Nutrition and Health, Greenburn Road, Bucksburn, Aberdeen AB21 9SB, Scotland, UK tel +44(0)1224 712751, fax +44(0)1224 716687, http://www.rowett.ac.uk mailto:a.tra...@abdn.ac.uk, http://bioinformatics.rri.sari.ac.uk #!/bin/bash #Fail on errors! set -e set -u ### # /etc/cron.daily/backup - a script to backup one hd to # # another on the same machine, using dump # # (see the dump man page for details, and # # http://envgen.nox.ac.uk/envgen/software/archives/000501.html#auto # # # # Written by MIT, 20/6/2002 # # Modified by Dan Swan for Bio-Linux 2.0 17/3/2003# # Updated for Bio-Linux 5 by Tim Booth 1/8/2008 # # Updated to perform Quarterly ETOH backups by Tony Travis 17/7/2009 # # Updated to dump ext4 fileystems by Tony Travis 19/5/2010# ### # Source configuration file CONFIG=/etc/default/backup if [ -r $CONFIG ]; then . $CONFIG fi #Use basic or ETOH mode? backup_mode=${MODE:-etoh} #Where to backup backup_destination=${DESTINATION:-/backups} #What to backup backup_filesystems=(`echo ${FILESYSTEMS:-/ /home /var /usr}`) #Compression scheme. For large drives bzip2 is probably too slow. compression_mode=${COMPRESSION:-z} # get the date day=`date +%a` month=`date +%b` dom=`date +%d` #Dump level can be set on command line or by todays date. if [ -n $* ]; then level=$1 elif [ $backup_mode == etoh ] ; then # Quarterly Enhanced Towers of Hanoi if [ $day == Sun ]; then if [ $dom -le 7 ]; then case $month in Jan | Apr | Jul | Oct) level=0 ;; *) level=1 ;; esac elif [ $dom -le 14 ]; then level=3 elif [ $dom -le 21 ]; then level=2 elif [ $dom -le 28 ]; then level=4 elif [ $dom -le 31 ]; then level=3 fi else case $day in Mon) level=6 ;; Tue) level=5 ;; Wed) level=8 ;; Thu) level=7 ;; Fri) level=9 ;; Sat) level=8 ;; esac fi elif [ $backup_mode == basic ] ; then # dump is 9 any day but Sunday if [[ $day == Sun ]]; then level=0 else level=9 fi else echo ERROR: Unknown backup mode and no level specified exit 1 fi # dirs to back up - used to be: # backups=(hda1 hda2 hda5 hda6) #But now I need to be a bit more sophisticated. #List filesystems to back up and use mount to find where they are. backups=() for fs in ${backup_filesystems[@]} ; do #DEBUG #echo Trying $fs device=`mount | egrep [^[:space:]]+ on $fs type ext[234] | awk '{print $1}'` if [ $device != ]; then backups[${#backups[@]}]=$device fi done #Did I find anything to backup? true ${backups:?No partitions found to backup - from paths ${backup_filesystems[@]}} #DEBUG # echo ${backups[@]} $level $compression_mode # exit # if backup drive is already mounted then try to unmount it first. # if this is in use the step will fail and exit /bin/mount | grep -q /backups /bin/umount /backups 2/dev/null # mount necessary drive - must be listed # in /etc/fstab in order to be mounted /bin/mount $backup_destination 2/dev/null || exit 0 # dump each directory, one at a time for item in ${backups[@]}; do basename=`basename $item` #sync or swim sync /sbin/dump -$level -$compression_mode -n -u -f $backup_destination/$basename.bak.$level $item done # umount drive /bin/umount $backup_destination -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.ubuntu.com/UKTeam/
Re: [ubuntu-uk] Simple backup script
Thanks so much for all of your suggestions guys. I ended up with something like this which resulted in part from a need to get my backups onto a Windows Server and to get an email when it happens (or if it doesn't). #!/bin/bash BACKUPDIR=/home/USERNAME/backup WHATTOBACKUP=/var/www SERVERNAME=SERVERNAME BACKUPADMIN=em...@domain.com MESSAGE=/tmp/message.txt if [ -d $BACKUPDIR -a -d $WHATTOBACKUP ] #make sure the source dest dirs exist then #backup the directory /bin/tar -cpzf $BACKUPDIR/`date +%a-%d-%b-%Y-`backup.tar.gz $WHATTOBACKUP #then remove anything over 7 days old find $BACKUPDIR/*.tar.gz -mtime +7 -exec rm -f {} \; #And let us know what happened SUBJECT=Backup Completed TO=$BACKUPADMIN echo Backup of $SERVERNAME completed $MESSAGE echo Result of backup $MESSAGE echo `ls -alt` $BACKUPDIR $MESSAGE /usr/bin/mail -s $SUBJECT $TO $MESSAGE rm $MESSAGE else #if backup dir does not exist, tell us SUBJECT=Backup Failure TO=$BACKUPADMIN echo Error backing up $SERVERNAME $MESSAGE echo One of the following directories is missing: $BACKUPDIR $WHATTOBACKUP $MESSAGE echo Date: `date` $MESSAGE /usr/bin/mail -s $SUBJECT $TO $MESSAGE rm $MESSAGE fi -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.ubuntu.com/UKTeam/
Re: [ubuntu-uk] Simple backup script
I promise to stop spamming the list now! Final draft. I know the email component should be put into a function but I'm a bad man and didn't do it! Just added an extra section that checks to make sure that tar didn't throw an error code for some reason when it exited. Chris BACKUPDIR=/home/username/backup WHATTOBACKUP=/var/www SERVERNAME=servername BACKUPADMIN=em...@domain.com MESSAGE=/tmp/message.txt if [ -d $BACKUPDIR -a -d $WHATTOBACKUP ] *#make sure the source dest dirs exist* then *#backup the directory* /bin/tar -cpzf $BACKUPDIR/`date +%a-%d-%b-%Y-`backup.tar.gz $WHATTOBACKUP RESULT=$? *#grab the tar exit code* if [ $RESULT -eq 0 ] *#tar will return non-zero if an error occurs. If we have a 0 exit code continue* then * #then remove anything over 7 days old* find $BACKUPDIR/*.tar.gz -mtime +7 -exec rm -f {} \; * #And let us know what happened* SUBJECT=Backup Completed TO=$BACKUPADMIN echo Backup of $SERVERNAME completed $MESSAGE echo Result of backup $MESSAGE echo `ls -alt` $BACKUPDIR $MESSAGE /usr/bin/mail -s $SUBJECT $TO $MESSAGE rm $MESSAGE exit else * #send a mail if tar failed* SUBJECT=Backup Failure TO=$BACKUPADMIN echo Backup of $SERVERNAME failed $MESSAGE echo Unable to create tar archive. Tar exit code was $RESULT $MESSAGE /usr/bin/mail -s $SUBJECT $TO $MESSAGE rm $MESSAGE exit fi else * #if backup dir does not exist, tell us* SUBJECT=Backup Failure TO=$BACKUPADMIN echo Error backing up $SERVERNAME $MESSAGE echo One of the following directories is missing: $BACKUPDIR $WHATTOBACKUP $MESSAGE echo Date: `date` $MESSAGE /usr/bin/mail -s $SUBJECT $TO $MESSAGE rm $MESSAGE fi -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.ubuntu.com/UKTeam/
Re: [ubuntu-uk] Simple backup script
We always used to do a daily backup, but Sundays we did a weekly one. Daily backups (ie. Mon-Sat) were kept 7 days, Sunday backups were on a 4-week cycle. And, of course, we actually backed up to somewhere else... not to the same system, as your script appears to do. You could make it 8-weekly or something, if you think it will take you time to discover problems, but we tended to find 4 weeks was sufficient. Sean -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.ubuntu.com/UKTeam/
Re: [ubuntu-uk] Simple backup script
And, of course, we actually backed up to somewhere else... not to the same system, as your script appears to do. It does. But read the rest of the thread and you'll see why, and what happens next ;-) Chris -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.ubuntu.com/UKTeam/
Re: [ubuntu-uk] Simple backup script
Chris Rowson wrote: Unfortunately I'm backing the Linux box up to a Windows server. This in turn gets backed up by a centralised backup system. I'd be tempted, in that case, to make things simple(r) for yourself if you can and just rsync what would need backing up onto some share on the Windows box, and join in the general backup regimen, unless there's something peculiar to this data that means you need to back it up in some way that the general backup scheme can't. In general, the simpler the backup, the simpler the restore. -- Avi -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.ubuntu.com/UKTeam/
Re: [ubuntu-uk] Simple backup script
Chris, Sorry if this has been mentioned before but have you tried sbackup? Regards, Tony. On 03/06/11 14:49, Chris Rowson wrote: I promise to stop spamming the list now! Final draft. I know the email component should be put into a function but I'm a bad man and didn't do it! Just added an extra section that checks to make sure that tar didn't throw an error code for some reason when it exited. Chris BACKUPDIR=/home/username/backup WHATTOBACKUP=/var/www SERVERNAME=servername BACKUPADMIN=em...@domain.com mailto:em...@domain.com MESSAGE=/tmp/message.txt if [ -d $BACKUPDIR -a -d $WHATTOBACKUP ] *#make sure the source dest dirs exist* then *#backup the directory* /bin/tar -cpzf $BACKUPDIR/`date +%a-%d-%b-%Y-`backup.tar.gz $WHATTOBACKUP RESULT=$? *#grab the tar exit code* if [ $RESULT -eq 0 ] *#tar will return non-zero if an error occurs. If we have a 0 exit code continue* then *#then remove anything over 7 days old* find $BACKUPDIR/*.tar.gz -mtime +7 -exec rm -f {} \; * #And let us know what happened* SUBJECT=Backup Completed TO=$BACKUPADMIN echo Backup of $SERVERNAME completed $MESSAGE echo Result of backup $MESSAGE echo `ls -alt` $BACKUPDIR $MESSAGE /usr/bin/mail -s $SUBJECT $TO $MESSAGE rm $MESSAGE exit else *#send a mail if tar failed* SUBJECT=Backup Failure TO=$BACKUPADMIN echo Backup of $SERVERNAME failed $MESSAGE echo Unable to create tar archive. Tar exit code was $RESULT $MESSAGE /usr/bin/mail -s $SUBJECT $TO $MESSAGE rm $MESSAGE exit fi else *#if backup dir does not exist, tell us* SUBJECT=Backup Failure TO=$BACKUPADMIN echo Error backing up $SERVERNAME $MESSAGE echo One of the following directories is missing: $BACKUPDIR $WHATTOBACKUP $MESSAGE echo Date: `date` $MESSAGE /usr/bin/mail -s $SUBJECT $TO $MESSAGE rm $MESSAGE fi -- Tony Arnold,Tel: +44 (0) 161 275 6093 Head of IT Security,Fax: +44 (0) 705 344 3082 University of Manchester, Mob: +44 (0) 773 330 0039 Manchester M13 9PL. Email: tony.arn...@manchester.ac.uk -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.ubuntu.com/UKTeam/
Re: [ubuntu-uk] Simple backup script
Chris, Sorry if this has been mentioned before but have you tried sbackup? Regards, Tony. Hi Tony, That's a GUI tool isn't it? This is to backup a Linux Server up to a Windows Server (only you can't see the Windows Server bits). Chris -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.ubuntu.com/UKTeam/
Re: [ubuntu-uk] Simple backup script
Chris Rowson wrote: Unfortunately I'm backing the Linux box up to a Windows server. This in turn gets backed up by a centralised backup system. I'd be tempted, in that case, to make things simple(r) for yourself if you can and just rsync what would need backing up onto some share on the Windows box, and join in the general backup regimen, unless there's something peculiar to this data that means you need to back it up in some way that the general backup scheme can't. In general, the simpler the backup, the simpler the restore. -- Avi Hi Avi, You're pretty much right. Centralised backup handles Win Server but I need to backup the data on the Linux box too, hence shifting data. ATM the Win box uses a script to 'pull' archive files from the Linux box using pscp.exe. I looked briefly at rsync but it seemed to require cygwin deps? Chris -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.ubuntu.com/UKTeam/
Re: [ubuntu-uk] Simple backup script
Chris Rowson wrote: Chris Rowson wrote: You're pretty much right. Centralised backup handles Win Server but I need to backup the data on the Linux box too, hence shifting data. ATM the Win box uses a script to 'pull' archive files from the Linux box using pscp.exe. I looked briefly at rsync but it seemed to require cygwin deps? Yeah if you want to run rsync on the Win box you need cygwin or similar. But you can rsync on the Linux box from one volume to another (such as one mounted from the Windows box). Windows' xcopy, though, is roughly analogous to an rsync that's not talking to an rsyncd. I'd do one of two things, depending on which is simpler and what your network looks like. I'd want to install Unix Services for Windows on the win server, and mount over NFS the bit of the Linux box to be backed up. If that's not possible - if the network's not isolated enough, or whoever looks after the Win server doesn't want to add stuff to it or whatever - perhaps configure Samba on the Linux box and do same over Samba. -- Avi -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.ubuntu.com/UKTeam/
Re: [ubuntu-uk] Simple backup script
Chris Rowson wrote: You're pretty much right. Centralised backup handles Win Server but I need to backup the data on the Linux box too, hence shifting data. ATM the Win box uses a script to 'pull' archive files from the Linux box using pscp.exe. I looked briefly at rsync but it seemed to require cygwin deps? Yeah if you want to run rsync on the Win box you need cygwin or similar. But you can rsync on the Linux box from one volume to another (such as one mounted from the Windows box). Windows' xcopy, though, is roughly analogous to an rsync that's not talking to an rsyncd. I'd do one of two things, depending on which is simpler and what your network looks like. I'd want to install Unix Services for Windows on the win server, and mount over NFS the bit of the Linux box to be backed up. If that's not possible - if the network's not isolated enough, or whoever looks after the Win server doesn't want to add stuff to it or whatever - perhaps configure Samba on the Linux box and do same over Samba. In this instance I could mount a share from the Win box and backup there. Unfortunately it's not always possible - dependent on as you say - network security. -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.ubuntu.com/UKTeam/
Re: [ubuntu-uk] Simple backup script
Chris, On 03/06/11 18:54, Chris Rowson wrote: Chris, Sorry if this has been mentioned before but have you tried sbackup? Regards, Tony. Hi Tony, That's a GUI tool isn't it? This is to backup a Linux Server up to a Windows Server (only you can't see the Windows Server bits). I missed that detail as I came into the thread late. But at the end of the day, sbackup is just a Python script that gets run by anacron (usually). There is a GUI to configure it, but if you can hack the config file you might be in business. I'm kind of thinking out loud here and I might be totally off the mark but there may be some mileage here. Regards, Tony. -- Tony Arnold,Tel: +44 (0) 161 275 6093 Head of IT Security,Fax: +44 (0) 705 344 3082 University of Manchester, Mob: +44 (0) 773 330 0039 Manchester M13 9PL. Email: tony.arn...@manchester.ac.uk -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.ubuntu.com/UKTeam/
Re: [ubuntu-uk] Simple backup script
On 2 June 2011 17:28, Chris Rowson christopherrow...@gmail.com wrote: I've been tinkering with backups and backup rotation today and I have come across many wierd and wonderful backup scripts of varying complexity. Is there anything wrong with using something simple like this? (except of course for the lack of validation). Basically, let's delete anything over 7 days old and then make a new backup. Chris #!/bin/bash find /home/username/backup/*.tar.gz -mtime +7 -exec rm -f {} \; /bin/tar -cpzf /home/username/backup/`date +%a-%d-%b-%y-`backupfile.tar.gz /var/www Nope, that's fine for something simple. Half the world probably backs up like that ;) s/ -- Twitter: @sfgreenwood Is this your sanderling? -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.ubuntu.com/UKTeam/
Re: [ubuntu-uk] Simple backup script
I would say it depends on what you mean by 'wrong' I handle the backups for a local government, so yes, that is wrong, very wrong. But it depends what you need it for. If you are happy with the retention that that gives you, then that's fine. The only suggestion I would make is that you should create the backup first, and only if the backup is created successfully should you delete the old one Hth, Bodsda Sent from my BlackBerry® wireless device -Original Message- From: Chris Rowson christopherrow...@gmail.com Sender: ubuntu-uk-boun...@lists.ubuntu.com Date: Thu, 2 Jun 2011 17:28:57 To: British Ubuntu Talkubuntu-uk@lists.ubuntu.com Reply-To: UK Ubuntu Talk ubuntu-uk@lists.ubuntu.com Subject: [ubuntu-uk] Simple backup script I've been tinkering with backups and backup rotation today and I have come across many wierd and wonderful backup scripts of varying complexity. Is there anything wrong with using something simple like this? (except of course for the lack of validation). Basically, let's delete anything over 7 days old and then make a new backup. Chris #!/bin/bash find /home/username/backup/*.tar.gz -mtime +7 -exec rm -f {} \; /bin/tar -cpzf /home/username/backup/`date +%a-%d-%b-%y-`backupfile.tar.gz /var/www -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.ubuntu.com/UKTeam/ -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.ubuntu.com/UKTeam/
Re: [ubuntu-uk] Simple backup script
On Jun 2, 2011 5:50 PM, bod...@googlemail.com wrote: I would say it depends on what you mean by 'wrong' I handle the backups for a local government, so yes, that is wrong, very wrong. I know where you're coming from. I think the list has quite a few public sector members in various capacities ;-) But it depends what you need it for. If you are happy with the retention that that gives you, then that's fine. Sorry. I've probably given you the wrong impression. Data from the backup directory would be backed up too another box. That box in turn is backed up elsewhere. 7 days is the local retention only. The only suggestion I would make is that you should create the backup first, and only if the backup is created successfully should you delete the old one Again, my fault, but the final script would run validation checks. I was basically asking if people use find rather than over complicating things as I've seen elsewhere. I wouldn't use that script as is. Chris -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.ubuntu.com/UKTeam/
Re: [ubuntu-uk] Simple backup script
On 2 June 2011 18:07, Chris Rowson christopherrow...@gmail.com wrote: On Jun 2, 2011 5:50 PM, bod...@googlemail.com wrote: I would say it depends on what you mean by 'wrong' I handle the backups for a local government, so yes, that is wrong, very wrong. I know where you're coming from. I think the list has quite a few public sector members in various capacities ;-) But it depends what you need it for. If you are happy with the retention that that gives you, then that's fine. Sorry. I've probably given you the wrong impression. Data from the backup directory would be backed up too another box. That box in turn is backed up elsewhere. 7 days is the local retention only. The only suggestion I would make is that you should create the backup first, and only if the backup is created successfully should you delete the old one Again, my fault, but the final script would run validation checks. I was basically asking if people use find rather than over complicating things as I've seen elsewhere. I wouldn't use that script as is. Chris I've started using rsnapshot. It does something similar with the local retention, but each file which is identical (between retained backups) is hard-linked rather than taking multiple disk blocks. I set up the config file once (which took 2 attempts and maybe 2.5 hours in total), set up cron and it just runs. The problem I find with hand-rolled backup scripts (which I've used in the past) is that you're the one maintaining them... Neil. -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.ubuntu.com/UKTeam/
Re: [ubuntu-uk] Simple backup script
I've started using rsnapshot. It does something similar with the local retention, but each file which is identical (between retained backups) is hard-linked rather than taking multiple disk blocks. Neil. Unfortunately I'm backing the Linux box up to a Windows server. This in turn gets backed up by a centralised backup system. Chris -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.ubuntu.com/UKTeam/
Re: [ubuntu-uk] Simple backup script
On 02/06/11 17:28, Chris Rowson wrote: I've been tinkering with backups and backup rotation today and I have come across many wierd and wonderful backup scripts of varying complexity. Is there anything wrong with using something simple like this? (except of course for the lack of validation). Nothing at all wrong with that. I used something very similar as the first step on a set of production servers a few years ago. I've recently discovered Déjà Dup, which is a simple front-end to duplicity(1), which in turn has an impressive range of options. It may or may not be suitable for your purposes. JT -- ---+ James Tait, BSc|xmpp:jayte...@wyrddreams.org Programmer and Free Software advocate |Tel: +44 (0)870 490 2407 ---+ -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.ubuntu.com/UKTeam/