Hello Steve,
In order to really wrap my head around rdiff-backup and avoid remembering it's usage and localized arguments, I've written a set of scripts to automate it's usage. Refer to the output after my sig. There is a longer tutorial/quick start in the README file. Feel free to post a link on the Related Pages page. http://www.physics.wisc.edu/~rader/rdiff-backup-tools/README http://www.physics.wisc.edu/~rader/rdiff-backup-tools/
thank you for creating and sharing 'rdiff-backup-tools' shell-scripts. I changed some parts to work/added some features and use them daily via cronjobs on several PCs/Servers/Virtual Machines with Linux and Windows Cygwin. Included are the diffs in case you want to put (some of) the changes into a new release. Do you consider converting your useful scripts into Python language, so all rdiff-backup users can benefit from them, not only users on Unix/Windows-Cygwin with BASH-shell? And what is the software license you use, GPL? ## CHANGES ## - BUG FIXED: the scripts broke in several places on filenames with spaces in them - BUG FIXED: RDB-recover: usage function told to use '--at', but '--as-of' was required - BUG FIXED: RDB-info: specifying '--saveset' did not work - BUG FIXED: RDB-report: problem with filenames with spaces in them - ADDED: custom configuration-file-support for all RDBT-scripts via commandline option - ADDED: in config: a common/custom path to rdiff-backup can be specified, referenced from the shell-scripts, no more editing all scripts when the path is different - ADDED: in config: added common options to pass to rdiff-backup commandline in all scripts - ADDED: in RDB-recover/-save: if destination directory-structure was not existing, the scripts aborted, now there is a '--force', resp. '--initial-run' option to create the destination dir-structure. - ADDED: in cronjob: consumes less system resources, if processes are 'nice'd - some typos fixed ## SUGGESTIONS ## - converting to Python or - change to be compatible with other shells than BASH: http://en.wikipedia.org/wiki/Comparison_of_command_shells - provide the allowed user names via central config-file instead of hard-coded in each script: if [ `whoami` != "root" ]; then echo "permission denied" exit 1 fi - evg. also add a 'run how often' (daily, each two days) option and a default retention policy (e.g. '2Y') for using with RDB-expire if not specified in each save-set explicitly - RDBT(-recover): maybe by convention, there should be a save-set "0" for recovering ALL configured change sets - RDBT-report could store an incremental file (with marker showing processed till which date) with change-history itself, so it does not need to walk the directories and parse the same output each time. Especially for RDB-repos with many changes/longer version history this takes quite long! Maybe this could be directly implemented in rdiff-backup, some sort of 'history/statistics-cache'? <nitpicking> - and just an idea: the name 'rdiff-backup-restore' would better suit the script currently named 'rdiff-backup-recover' (IMO recovery is more than restoring backups ...) and rdiff-backup uses '--restore-as-of' itself ... </nitpicking> And here are the changes: <code> diff U3 rdiff-backup-expire rdiff-backup-expire --- rdiff-backup-expire Thu Nov 06 13:40:50 2008 +++ rdiff-backup-expire Sun Dec 07 13:04:54 2008 @@ -3,7 +3,7 @@ # $Id: rdiff-backup-expire,v 1.10 2008/11/06 02:26:07 rader Exp $ # -if [ `whoami` != "root" ]; then +if [ `whoami` != "root" ]; then ## Maybe specify allowed users via config-file echo "permission denied" exit fi @@ -17,9 +17,12 @@ echo " --age TIMESPEC expire saves older than TIMESPEC (date or interval)" echo " --force expire multiple saves" echo " --debug debug output" + echo " --config PATH use the specified configuation file instead of default" } dosave=1 +Conf='/etc/rdiff-backup-tools.conf' + while [ $# -gt 0 ]; do arg=$1 @@ -79,12 +82,22 @@ shift continue fi + if [ "$arg" == '--config' ] || [ "$arg" == '-c' ]; then + shift + Conf=$1 + if [ $debug ]; then + echo "Using custom configuration file '$Conf'" + fi + shift + continue + fi echo "Unrecognized option \"$arg\"" usage exit done -Conf=/usr/local/etc/rdiff-backup-tools.conf + +#Conf=/etc/rdiff-backup-tools.conf if [ ! -f "$Conf" ]; then echo "Error: $Conf: no such file" @@ -120,7 +133,7 @@ continue fi if [ $debug ]; then - echo "rdiff-backup ${args}--remove-older-than "$date" $dst" + echo "${PathRDB} ${args}--remove-older-than "$date" $dst" fi if [ $verbose ]; then echo "==================================================================" @@ -130,7 +143,7 @@ date fi if [ $dosave ]; then - rdiff-backup ${args}--remove-older-than "$date" $dst 2>&1 + ${PathRDB} ${args}--remove-older-than "$date" $dst 2>&1 fi done diff U3 rdiff-backup-info rdiff-backup-info --- rdiff-backup-info Thu Nov 06 13:40:50 2008 +++ rdiff-backup-info Sun Dec 07 16:03:53 2008 @@ -3,7 +3,7 @@ # $Id: rdiff-backup-info,v 1.14 2008/11/06 12:22:12 rader Exp $ # -if [ `whoami` != "root" ]; then +if [ `whoami` != "root" ]; then ## Maybe specify allowed users via config-file echo "permission denied" exit fi @@ -13,6 +13,7 @@ echo " --saveset N report on save set number N" echo " --path PATH report on save of PATH" echo " --debug debug output" + echo " --config PATH use the specified configuation file instead of default" echo " list list sources and destinations" echo " files list files" echo " ls list files with ls -l" @@ -20,10 +21,12 @@ } mode=none +Conf='/etc/rdiff-backup-tools.conf' + while [ $# -gt 0 ]; do arg=$1 - if [ "$arg" == '--debug' ] || [ "$arg" == '-d' ]; then + if [ "$arg" == '--debug' ] || [ "$arg" == '-d' ] || [ "$arg" == '--verbose' ] || [ "$arg" == '-v' ]; then debug=1 echo Debug mode is on shift @@ -34,9 +37,10 @@ exit fi if [ "$arg" == '--saveset' ] || [ "$arg" == '-s' ]; then + shift saveset=$1 if [ $debug ]; then - echo "Save set is $saveset" + echo "Save set is '$saveset'" fi shift continue @@ -50,6 +54,15 @@ shift continue fi + if [ "$arg" == '--config' ] || [ "$arg" == '-c' ]; then + shift + Conf=$1 + if [ $debug ]; then + echo "Using custom configuration file '$Conf'" + fi + shift + continue + fi if [ "$arg" == 'list' ]; then mode=list shift @@ -85,7 +98,7 @@ exit fi -Conf=/usr/local/etc/rdiff-backup-tools.conf +#Conf=/etc/rdiff-backup-tools.conf if [ ! -f "$Conf" ]; then echo "rdiff-backup-info: $Conf: no such file" @@ -109,65 +122,65 @@ if [ $debug ]; then echo "Source is $src" fi - if [ $saveset ] && [ $saveset != $i ] && [ $saveset != $src ]; then + if [ $saveset ] && [ $saveset != $i ] && [ $saveset != "$src" ]; then if [ $debug ]; then - echo Skipping save set $i + echo "Skipping save set '$i'" fi continue; fi if [ "${Destination[$i]}" == '' ]; then - echo "Error: cannot report on $src: missing Destination[$i] definition" + echo "Error: cannot report on '$src': missing Destination[$i] definition" continue fi if [ ! -d "${Destination[$i]}" ]; then - echo "Error: cannot report on $src: ${Destination[$i]}: no such directory" + echo "Error: cannot report on '$src': '${Destination[$i]}': no such directory" continue fi case $mode in list) - echo "Save set $i: save $src to $dst" + echo "Save set '$i': save '$src' to '$dst'" ;; files) time=`date +%s` - echo Listing files in the current mirror of $src: - if [ $debug ]; then - echo "rdiff-backup $mode_arg $time $dst \\" + echo "Listing files in the current mirror of '$src':" + if [ $debug ]; then + echo "${PathRDB} $mode_arg '$time' '$dst' \\" echo " | sed \"s:^:$dst/:\"" fi - rdiff-backup $mode_arg $time $dst | sed "s:^:$dst/:" + ${PathRDB} $mode_arg $time "$dst" | sed "s:^:$dst/:" ;; ls) if [ $debug ]; then - echo "find $dst -print | xargs ls -ld | grep -v $dst/rdiff-backup-data " + echo "find '$dst' -print0 | xargs --null ls -ld | grep -v '$dst/rdiff-backup-data'" fi - find $dst -print | xargs ls -ld | grep -v $dst/rdiff-backup-data +#WAS: find $dst -print | xargs ls -ld | grep -v $dst/rdiff-backup-data ## breaks on files with spaces (' ') in them. + find "$dst" -print0 | xargs --null ls -ld | grep -v "$dst/rdiff-backup-data" ## works fine ;; changed) if [ $debug ]; then echo "Getting last save time via..." - echo "rdiff-backup --list-increments $dst \\ " + echo "${PathRDB} --list-increments '$dst' \\ " echo " | tail -2 | head -1 | sed 's/.*dir[ ]*//'" fi - time=`rdiff-backup --list-increments $dst | tail -2 | head -1 | sed 's/.*dir[ ]*//'` + time=`${PathRDB} --list-increments "$dst" | tail -2 | head -1 | sed 's/.*dir[ ]*//'` if [ $debug ]; then - echo "Last save was \"$time\"" + echo "Last save was '$time'" fi if [ "$time" == "Found 0 increments:" ]; then echo There are no changes to the mirror $src continue fi - echo Listing of changes in to the mirror $src at $time: + echo "Listing of changes to the mirror '$src' since '$time':" if [ $debug ]; then - echo "rdiff-backup $mode_arg "$time" $dst | ..." + echo "${PathRDB} $mode_arg '$time' '$dst' ..." fi - rdiff-backup $mode_arg "$time" $dst | \ + ${PathRDB} $mode_arg "$time" "$dst" | \ sed "s:new :new $src/:" | \ sed "s:changed :changed $src/:" | \ sed "s:deleted :deleted $src/:" ;; esac - + done exit - diff U3 rdiff-backup-recover rdiff-backup-recover --- rdiff-backup-recover Thu Nov 06 13:40:50 2008 +++ rdiff-backup-recover Sun Dec 07 13:04:16 2008 @@ -3,21 +3,25 @@ # $Id: rdiff-backup-recover,v 1.9 2008/11/05 23:51:43 rader Exp $ # -if [ `whoami` != "root" ]; then +if [ `whoami` != "root" ]; then ## Maybe specify allowed users via config-file echo "permission denied" exit fi function usage { - echo "usage: rdiff-backup-recover [args] --at <DATE> <SOURCE> <DESTINATION>" + echo "usage: rdiff-backup-recover [args] --as-of <DATE> <SOURCE> <DESTINATION>" echo " --as-of TIMESPEC recover from mirror as of TIMESPEC (date or interval)" echo " --dry-run no recover (also -n)" echo " --force overwrite DESTINATION" echo " --debug debug output" + echo " --config PATH use the specified configuation file instead of default" } dosave=1 saveset=1 +force=0 +Conf='/etc/rdiff-backup-tools.conf' + while [ $# -gt 0 ]; do arg=$1 @@ -48,8 +52,9 @@ fi if [ "$arg" == "--force" ] || [ "$arg" == "-f" ]; then args="$1 " + force=1 if [ $debug ]; then - echo "Added $args" + echo "Added $args - forcing destination dir(s) creation and overwrite" fi shift continue @@ -70,6 +75,15 @@ shift continue fi + if [ "$arg" == '--config' ] || [ "$arg" == '-c' ]; then + shift + Conf=$1 + if [ $debug ]; then + echo "Using custom configuration file '$Conf'" + fi + shift + continue + fi echo "Unrecognized option \"$arg\"" usage exit @@ -87,7 +101,7 @@ exit fi -Conf=/usr/local/etc/rdiff-backup-tools.conf +#Conf=/etc/rdiff-backup-tools.conf if [ ! -f "$Conf" ]; then echo "Error: $Conf: no such file" @@ -101,12 +115,21 @@ ddst=${Destination[$saveset]} if ( echo $src | egrep ^$dsrc > /dev/null ) ; then + + if [ $force ] && [ ! -e "$dst" ]; then + mkdir -p "$dst" + + if [ $debug ]; then + echo "Forcing overwrite and/or destination dir(s) creation (if not existing already) for '$dst': " + fi + fi + if [ $debug ]; then - echo "rdiff-backup ${args}--restore-as-of \"$date\" $ddst $dst" + echo "${PathRDB} ${args}--restore-as-of \"$date\" $ddst $dst" fi if [ $dosave ]; then echo "Starting recover of $ddst to $dst... this may take awhile..." - rdiff-backup ${args}--restore-as-of "$date" $ddst $dst + ${PathRDB} ${args}--restore-as-of "$date" "$ddst" "$dst" fi else echo "Error: path prefix \"$src\" does not match prefix of save set $saveset (\"$dsrc\")" diff U3 rdiff-backup-report rdiff-backup-report --- rdiff-backup-report Thu Nov 06 13:40:50 2008 +++ rdiff-backup-report Sun Dec 07 16:09:25 2008 @@ -4,7 +4,7 @@ # # -if [ `whoami` != "root" ]; then +if [ `whoami` != "root" ]; then ## Maybe specify allowed users via config-file echo "permission denied" exit fi @@ -14,9 +14,12 @@ echo " --path PATH only report on save set of PATH" echo " --saveset N only report on save set number N" echo " --debug debug output" + echo " --config PATH use the specified configuation file instead of default" } save=1 +Conf='/etc/rdiff-backup-tools.conf' + while [ $# -gt 0 ]; do arg=$1 @@ -48,11 +51,20 @@ shift continue fi + if [ "$arg" == '--config' ] || [ "$arg" == '-c' ]; then + shift + Conf=$1 + if [ $debug ]; then + echo "Using custom configuration file '$Conf'" + fi + shift + continue + fi usage exit done -Conf=/usr/local/etc/rdiff-backup-tools.conf +#Conf=/etc/rdiff-backup-tools.conf if [ ! -f "$Conf" ]; then echo "rdiff-backup-report: $Conf: no such file" @@ -69,7 +81,7 @@ let t=$t+1 for (( i = 1; i < $t; i++ )) do src=${Source[$i]} - if [ $saveset ] && [ $saveset != $i ] && [ $saveset != $src ]; then + if [ $saveset ] && [ $saveset != $i ] && [ $saveset != "$src" ]; then if [ $debug ]; then echo Skipping save set $i fi @@ -85,7 +97,7 @@ fi dst=${Destination[$i]} echo - echo "Save set $i: save $src to $dst" + echo "Save set '$i': save '$src' to '$dst'" for f in `( cd $dst/rdiff-backup-data; find * -name 'session_statistics*' -print | sort)`; do adds=`grep NewFiles $dst/rdiff-backup-data/$f | awk '{print $2}'` if [ "$adds" = "" ]; then adds=0; fi diff U3 rdiff-backup-save rdiff-backup-save --- rdiff-backup-save Thu Nov 06 13:40:50 2008 +++ rdiff-backup-save Mon Dec 08 15:17:06 2008 @@ -3,41 +3,52 @@ # $Id: rdiff-backup-save,v 1.27 2008/11/06 12:28:20 rader Exp $ # -if [ `whoami` != "root" ]; then +## Maybe provide the allowed users also via config-file? +if [ `whoami` != "root" ]; then ## Maybe specify allowed users via config-file echo "permission denied" exit 1 fi function usage { echo "usage: rdiff-backup-save [args]" - echo " --path PATH only backup PATH" - echo " --saveset N only backup save set number N" - echo " --dry-run no save" - echo " --verbose verbose output (prints summary)" - echo " --debug debug output" + echo " -p / --path PATH only backup PATH" + echo " -s / --saveset N only backup save set number N" + echo " -c / --config PATH use the specified configuation file instead of default" + echo " -n / --dry-run only test, not performing a real backup" + echo " -i / --initial-run create destination directory-structure (uses RDB '--force')" + echo " -v / --verbose verbose output (prints summary)" + echo " -d / --debug debug output" } save=1 +Conf='/etc/rdiff-backup-tools.conf' +force=0 while [ $# -gt 0 ]; do arg=$1 - if [ $arg == '--debug' ] || [ $arg == '-d' ]; then + if [ "$arg" == '--debug' ] || [ "$arg" == '-d' ]; then debug=1 echo Debug mode is on shift continue fi - if [ $arg == '--help' ] || [ $arg == '-h' ]; then + if [ "$arg" == '--help' ] || [ "$arg" == '-h' ]; then usage exit 1 fi - if [ "$arg" == '--dry-run' ] || [ $arg == '-n' ]; then + if [ "$arg" == '--dry-run' ] || [ "$arg" == '-n' ]; then save='' echo No save mode is on shift continue fi - if [ $arg == '--verbose' ] || [ $arg == '-v' ]; then + if [ "$arg" == '--initial-run' ] || [ "$arg" == '-i' ]; then + force=1 + echo "Initial run, creating backup-destination directories if non-existant." + shift + continue + fi + if [ $arg == '--verbose' ] || [ "$arg" == '-v' ]; then verbose=1 if [ $debug ]; then echo Verbose mode is on @@ -63,15 +74,24 @@ shift continue fi + if [ "$arg" == '--config' ] || [ "$arg" == '-c' ]; then + shift + Conf=$1 + if [ $debug ]; then + echo "Using custom configuration file '$Conf'" + fi + shift + continue + fi echo "rdiff-backup-save: unknown option \"$arg\"" usage exit 1 done -Conf=/usr/local/etc/rdiff-backup-tools.conf +#Conf=/etc/rdiff-backup-tools.conf if [ ! -f "$Conf" ]; then - echo "rdiff-backup-save: $Conf: no such file" + echo "rdiff-backup-save: '$Conf': no such file" exit 1 fi @@ -107,7 +127,7 @@ log=$Log.$$.$i src=${Source[$i]} dst=${Destination[$i]} - if [ $saveset ] && [ $saveset != $i ] && [ $saveset != $src ]; then + if [ $saveset ] && [ $saveset != $i ] && [ $saveset != "$src" ]; then if [ $debug ]; then echo Skipping save set $i fi @@ -115,7 +135,7 @@ fi now=`date` msg "==================================================================" - msg "Saving $src to $dst..." + msg "Saving '$src' to '$dst'..." if [ $debug ]; then echo Doing save number $i echo Source is $src @@ -134,26 +154,40 @@ fi ls=`ssh $host ls -ld $dir 2>&1` if ( echo $ls | grep 'No such file' > /dev/null 2>&1 ); then - msg "Error: save set number $i: $dst no such directory" - cat $log >> $Log.$$; rm -f $log - continue + if [ $force == 1 ]; then + msg "Notice: save set number '$i': '$dst' not existant, creating directory-structure" +# BackupCommonOptions="${BackupCommonOptions} --force" ## OR use below 'mkdir -p' + mkdir -p "$dst" + else + msg "Error: save set number '$i': '$dst' no such directory" + cat $log >> $Log.$$; rm -f $log + continue + fi fi - else + else if [ ! -d "$dst" ]; then - msg "Error: save set number $i: $dst no such directory" - cat $log >> $Log.$$; rm -f $log - continue + if [ $force == 1 ]; then + msg "Notice: save set number '$i': '$dst' not existant, creating directory-structure" +# BackupCommonOptions="${BackupCommonOptions} --force" ## OR use below 'mkdir -p' + mkdir -p "$dst" + else + msg "Error: save set number '$i': '$dst' no such directory" + cat $log >> $Log.$$; rm -f $log + continue + fi fi fi msg StartTime $now - if [ $debug ]; then - echo rdiff-backup --print-statistics ${Excludes[$i]} $src $dst + if [ $debug ]; then + echo ${PathRDB} ${BackupCommonOptions} --print-statistics ${Excludes[$i]} "$src" "$dst" fi - if [ $save ]; then - if [ $debug ]; then - rdiff-backup --print-statistics ${Excludes[$i]} $src $dst | tee -a $log 2>&1 - else - rdiff-backup --print-statistics ${Excludes[$i]} $src $dst >> $log 2>&1 + if [ $save ]; then + if [ $debug ]; then + ${PathRDB} ${BackupCommonOptions} --print-statistics ${Excludes[$i]} "$src" "$dst" | tee -a $log 2>&1 + chmod -R u+rX "$dst" ## The capital '+X' changes only directory-permissions! + else + ${PathRDB} ${BackupCommonOptions} --print-statistics ${Excludes[$i]} "$src" "$dst" >> $log 2>&1 + chmod -R u+rX "$dst" ## The capital '+X' changes only directory-permissions! fi if [ $verbose ]; then egrep "$SummaryRegex" $log diff U3 rdiff-backup-tools.conf rdiff-backup-tools.conf --- rdiff-backup-tools.conf Thu Nov 06 13:40:50 2008 +++ rdiff-backup-tools.conf Tue Dec 09 13:56:22 2008 @@ -2,6 +2,11 @@ Log="/var/log/rdiff-backup.log" Summary="/var/log/rdiff-backup.summary" +PathRDB='/usr/bin/rdiff-backup' +PathOverrideCharsAndFSAbilities='/etc/rdiff-backup_CharsAndFSAbilities.override' +BackupCommonOptions="--use-utc --use-compatible-timestamps --override-quote-chars-and-fsabilities-from-file ${PathOverrideCharsAndFSAbilities}" +RestoreCommonOptions='' + SummaryRegex="Error|ElapsedTime|done" SummaryRegex="$SummaryRegex|SourceFileSize|MirrorFileSize" SummaryRegex="$SummaryRegex|TotalDestinationSizeChange" diff U3 rdiff-backup.cron rdiff-backup.cron --- rdiff-backup.cron Thu Nov 06 13:40:50 2008 +++ rdiff-backup.cron Tue Dec 09 14:01:27 2008 @@ -1,5 +1,19 @@ +#!/bin/sh +# Uncomment the next line for testing/debugging +#set -x -( echo; /usr/local/sbin/rdiff-backup-save --verbose ; \ - echo; /usr/local/sbin/rdiff-backup-expire --verbose --force ; \ - echo ) 2>&1 | mail -s "rdiff-backup report for $HOSTNAME" root +## $Id: own/backup/rdiff-backup/rdiff-backup.cron,v 6db197774474 2008/11/14 17:57:50 trinity-rt $ ## + +PATH_PREFIX_RDBTOOLS='/usr/local/sbin' +BE_NICE='/usr/bin/nice -n 19' +COMMON_OPTIONS='--config /etc/rdiff-backup-tools.conf' + +TEST_ONLY='' ## --dry-run' + + ( echo "######### Performing Backup .... #########";echo; $BE_NICE ${PATH_PREFIX_RDBTOOLS}/rdiff-backup-save ${COMMON_OPTIONS} --verbose ${TEST_ONLY}; \ + echo "######### Expiring Old Backups #########";echo; $BE_NICE ${PATH_PREFIX_RDBTOOLS}/rdiff-backup-expire ${COMMON_OPTIONS} --verbose --force ; \ + echo "######### rdiff-backup-info #########"; echo; $BE_NICE ${PATH_PREFIX_RDBTOOLS}/rdiff-backup-info ${COMMON_OPTIONS} changed ; \ + echo "######### rdiff-backup-report #########"; echo; $BE_NICE ${PATH_PREFIX_RDBTOOLS}/rdiff-backup-report ${COMMON_OPTIONS}; \ + echo ) 2>&1 | mail -s "rdiff-backup report for $HOSTNAME" root +#OR USE:# >> /var/log/anacron.log </code> Works like a charm for me, thanks again :-)! Cheers, Oliver _______________________________________________ rdiff-backup-users mailing list at [email protected] http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
