[rdiff-backup-users] Force a regression of rdiff-backup archive
I have a problem that a repository which 3 days ago occupied 26G has mushroomed to 44G. The reason is that one night the backup seems to have gone badly wrong, thinking the source data was missing, then the next night it was all okay again - so it generated lots of diffs for files that had gone and then I guess they all got stored in the clear again because they had reappeared as 'new' files. I know that these days 18G is not such a big deal but it happens to be a problem for me with my present media. What I would like to do is remove these last few days of backups but without losing my earlier backups (which are valuable). The logical thing seems to be to force a regression of the backup (twice, say, to get back to the state before the first problem occurred), but there does not seem to be any way to force this to happen? Can anyone suggest anything? Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] What dirs can be excluded on a full linux image?
bart wrote: What directories can be excluded when doing a full linux backup image which could be restored and run on a new blank disk? A more appropriate package for this purpose is mondo - http://www.mondorescue.org - if it supports your distro. Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] still the problem to backup linux system to windows
Wei, Xiaohai wrote: I want to backup a whole linux system to windows for restoring the system from windows to linux later. I successfully installed native rdiff-backup on windows xp and it can backup linux data to windows folder. But it seems failed if there is symlink in the linux system. Error information attached at the end. I want to know: 1. I want to backup all the information of the linux system to windows and so i can restore the OS(not only data) to linux later. I think the information includes user/group/permission etc. At first i want to use rsync to do so but found rsync can not do it. and I found that rdiff-backup will use meta-data to keep such information, so I assume it can do so. correct me if I am wrong. 2. If i can get what I need with rdiff-backup, what should I do? Thanks, James I don't believe rdiff-backup is able to do this on its own; rdiff-backup suits a situation in which you want to backup data (not whole system) and to do so repeatedly (e.g. day after day), and be able to recover data from earlier backups as well as from the latest one. Even if rdiff-backup can backup your whole system it cannot easily restore it. To restore a system you need to be able to boot up in a temporary OS, do the partitioning and formatting of the (new) system disk etc etc (and then copy all the system files and data), rdiff-backup is not designed to do this. If you want to backup a whole linux system for easy restoring, then I suggest you look at a package designed for this purpose. I use mondo http://www.mondorescue.org - and it doesn't need Windows. You can create a backup on a flash drive, take this to a different machine, boot it and restore your old system to the new hardware. It offers differential backups, but not the 'history' magic that is rdiff-backup's special trick. I suggest you use both: mondo to backup your system (excluding your own data files, because these may be too big for a flash disk), and then rdiff-backup [over a network to another machine] for your data files. mondo only needs to be run occasionally, rdiff-backup you should run frequently (e.g. as daily cron job). When disaster strikes, use mondo to recover your system and then rdiff-backup to recover your data. And of course if you need the data from one week or one month or even one year ago, rdiff-backup will oblige. HTH Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] still the problem to backup linux system to windows
Wei, Xiaohai wrote: Dominic, thanks for your reply. I have a solution to recover from disaster which can restore the system to factory default. After that, I want to restore the system to latest or earlier version. I can achieve it in linux - linux way. I can back up the whole system to a Linux host and then restore later. It works. But came to Windows, problems happened. As the linux system I am working for is a mobile device, and users need to backup the device to a host machine, which usually is a windows machine. So I am seeking a solution to backup the linux system to windows. If there are only data files, it is relatively easy, there are many solutions for it. but I want to design the feature similar with Apple's Time Machine. I want to restore the system to earlier state, not only the data, but also application/application config etc. So I need to backup/restore the whole Linux system files to/from Windows system. Thansk James I don't know how to do this, I admit, so maybe others can help. But I think for rdiff-backup you should try first with remote linux host, because this is the best-tested version of rdiff-backup. Then if this works you can see if it can work also for rdiff-backup.exe (the windows version). This way problems may be easier to diagnose. Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] unable to recover from interrupted backup
Ben Tucker wrote: I had a machine fail in the midst of backing up a couple weeks ago and ever since I have been unable to get it to successfully backup. rdiff-backup --check-destination-dir runs without error, but then running the actual backup command results in an exception. I'm running 1.2.8 stock deb install on Ubuntu jaunty. The client machine is also running 1.2.8 (also ubuntu). These are the only versions of rdiff-backup with have been used. I'd greatly appreciate any pointers on how to resolve this. Thanks! Did you get this sorted? It sounds an alarming problem for any of us who have important repositories of data going back over a long period; the whole thing could get trashed by one failed backup! I give below a way (posted here by Steven a few weeks ago, all credit to him) to force a regression. This might undo the 'bad' backup(s) that you have and then allow you to make further backups. I haven't had the need to try it but it might help you: Dominic While rdiff-backup is making a backup it creates a new current_mirror.$timestamp.data file in the rdiff-backup-data destination directory. Once it's done making the backup, it deletes the old current_mirror file. In other words, while the backup is running there are two current_mirror files. In order to undo a backup, you'll need to to create a fake current_mirror file with the timestamp of the previous backup and run rdiff-backup -v6 --check-destination-dir $dest. rdiff-backup will be fooled into thinking the latest backup failed and revert it. I do not know of a way to regress more than one backup at once, but this method should work multiple times. P.S. A full example in case my description isn't clear: $ mkdir source $ echo 1 source/a $ echo 2 source/b $ rdiff-backup source dest $ ls dest a b rdiff-backup-data $ rm source/a $ rdiff-backup source dest $ ls dest b rdiff-backup-data $ ls dest/rdiff-backup-data/mirror* dest/rdiff-backup-data/mirror_metadata.2009-07-03T18:47:33-06:00.diff.gz dest/rdiff-backup-data/mirror_metadata.2009-07-03T18:48:12-06:00.snapshot.gz $ echo PID 99 \ dest/rdiff-backup-data/current_mirror.2009-07-03T18:47:33-06:00.diff.gz $ rdiff-backup -v6 --check-destination-dir dest Using rdiff-backup version 1.2.7 ... snip ... Regressing to Fri Jul 3 18:47:33 2009 ... snip ... Regressing file a ... snip ... Cleaning up $ ls dest a b rdiff-backup-data ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] A bug in rdiff-backup with changing files
Josh Nisly wrote: One of my clients encountered an interesting bug in rdiff-backup - under certain conditions, if a file changes while it is being backed up, rdiff-backup throws an unhandled exception... I can't help on the main problem, but as a workaround (and a better approach anyway), users should backup from a snapshot - or, in Windows terminology, a 'Volume Shadow'. This is the approach I take in my TimeDicer script. That way files can't change or be locked during a backup. (Note that it might still be possible for files to be in an inconsistent state, though if the application manipulating them is 'volume shadow aware', they shouldn't be.) Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] A bug in rdiff-backup with changing files
Josh Nisly wrote: One of my clients encountered an interesting bug in rdiff-backup - under certain conditions, if a file changes while it is being backed up, rdiff-backup throws an unhandled exception... I can't help on the main problem, but as a workaround (and a better approach anyway), users should backup from a snapshot - or, in Windows terminology, a 'Volume Shadow'. This is the approach I take in my TimeDicer script. That way files can't change or be locked during a backup. (Note that it might still be possible for files to be in an inconsistent state, though if the application manipulating them is 'volume shadow aware', they shouldn't be.) Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] unable to recover from interrupted backup
Ben Tucker wrote: I had a machine fail in the midst of backing up a couple weeks ago and ever since I have been unable to get it to successfully backup. rdiff-backup --check-destination-dir runs without error, but then running the actual backup command results in an exception. I'm running 1.2.8 stock deb install on Ubuntu jaunty. The client machine is also running 1.2.8 (also ubuntu). These are the only versions of rdiff-backup with have been used. I'd greatly appreciate any pointers on how to resolve this. Thanks! Did you get this sorted? It sounds an alarming problem for any of us who have important repositories of data going back over a long period; the whole thing could get trashed by one failed backup! I give below a way (posted here by Steven a few weeks ago) to force a regression. This might undo the 'bad' backup(s) that you have and then allow you to make further backups. I haven't had the need to try it but it might help you: Dominic While rdiff-backup is making a backup it creates a new current_mirror.$timestamp.data file in the rdiff-backup-data destination directory. Once it's done making the backup, it deletes the old current_mirror file. In other words, while the backup is running there are two current_mirror files. In order to undo a backup, you'll need to to create a fake current_mirror file with the timestamp of the previous backup and run rdiff-backup -v6 --check-destination-dir $dest. rdiff-backup will be fooled into thinking the latest backup failed and revert it. I do not know of a way to regress more than one backup at once, but this method should work multiple times. P.S. A full example in case my description isn't clear: $ mkdir source $ echo 1 source/a $ echo 2 source/b $ rdiff-backup source dest $ ls dest a b rdiff-backup-data $ rm source/a $ rdiff-backup source dest $ ls dest b rdiff-backup-data $ ls dest/rdiff-backup-data/mirror* dest/rdiff-backup-data/mirror_metadata.2009-07-03T18:47:33-06:00.diff.gz dest/rdiff-backup-data/mirror_metadata.2009-07-03T18:48:12-06:00.snapshot.gz $ echo PID 99 \ dest/rdiff-backup-data/current_mirror.2009-07-03T18:47:33-06:00.diff.gz $ rdiff-backup -v6 --check-destination-dir dest Using rdiff-backup version 1.2.7 ... snip ... Regressing to Fri Jul 3 18:47:33 2009 ... snip ... Regressing file a ... snip ... Cleaning up $ ls dest a b rdiff-backup-data ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Re: Remote encrypted backup with slow connection.
Piotr Karbowski wrote: On Mon, Sep 21, 2009 at 2:02 PM, Matthew Miller mat...@mattdm.org wrote: On Mon, Sep 21, 2009 at 01:58:11PM +0200, Piotr Karbowski wrote: local rdiff-backup dir with remote server but how? If I will use for example rsync it still need to check whole files for changes (read, download it) and upload only new. I hope you will understand what I need and help me. rsync won't check whole files unless you give the -c flag. Otherwise, it just compares metadata. I don't know if that's also the case with rdiff-backup, but I assume so. So I need to know how rdiff default compares data, if by size and mod-time, it will not be so painful but still it will download changed files to generate diff. Rdiff-backup is designed to be ultra-efficient at this activity. It only sends the changes in a file over the wire, not the whole file. To do this it uses the librsync library which is effectively the same as rsync. You can read more about the technique at http://en.wikipedia.org/wiki/Rsync. rdiff-backup does not use file times to determine whether to do backups. It can backup very large files with small changes very quickly. Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Re: Remote encrypted backup with slow connection.
Piotr Karbowski wrote: On Mon, Sep 21, 2009 at 8:21 PM, Dominic Raferd domi...@timedicer.info wrote: Piotr Karbowski wrote: On Mon, Sep 21, 2009 at 2:02 PM, Matthew Miller mat...@mattdm.org wrote: On Mon, Sep 21, 2009 at 01:58:11PM +0200, Piotr Karbowski wrote: local rdiff-backup dir with remote server but how? If I will use for example rsync it still need to check whole files for changes (read, download it) and upload only new. I hope you will understand what I need and help me. rsync won't check whole files unless you give the -c flag. Otherwise, it just compares metadata. I don't know if that's also the case with rdiff-backup, but I assume so. So I need to know how rdiff default compares data, if by size and mod-time, it will not be so painful but still itefficient will download changed files to generate diff. Rdiff-backup is designed to be ultra-efficient at this activity. It only sends the changes in a file over the wire, not the whole file. To do this it uses the librsync library which is effectively the same as rsync. You can read more about the technique at http://en.wikipedia.org/wiki/Rsync. rdiff-backup does not use file times to determine whether to do backups. It can backup very large files with small changes very quickly. Dominic You dont understand me, rdiff-backup is efficient, but to make diff it must read WHOLE file, on remote nfs or sshfs it is SLW and painful Sorry I get it now. But I think rdiff-backup and rsync require a separate computer at the remote end in order to optimise transfers, so if you are just accessing a remote share using sshfs or similar then they can still work of course but as you realise they will be slow. I guess it is not possible for you to run rdiff-backup (or rsync) at the remote end as well? You could run rdiff-backup locally to create a backup store and then mirror this store to the remote share using rcp. Still it will be slow because rdiff-backup always stores the latest copy of each file in full and so if this changes even slightly then the whole file will must be transferred by rcp. Duplicity http://duplicity.nongnu.org/ might work better for you, because it uses forward diffs. Also its archives are secure. Although not directly relevant I found a page here http://www.psc.edu/networking/projects/hpn-ssh/ which provides a patch to greatly speed up OpenSSH in some situations. ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Mozy Online Back-Up Problems
simsam wrote: My experience with Mozy was not good. I installed MOZY FREE Backup software on my computer as per their instructions. After installation, my computer kept freezing and it CRASHED my computer. Thanks god I have one additional backup of data. I need storage space, but a more organized approach. Does anybody provide with automated backups? For individual users it looks as if Windows 7 comes with some pretty reasonable MS backup software (try googling windows 7 backup); it can do scheduled backups to network shares too. Vista also had similar capability, but much less flexible. Windows XP's built-in backup is pretty useless, but there are many other choices. Rdiff-backup is the engine behind TimeDicer, http://www.timedicer.info/, a script specifically aimed at Windows XP backups to a Linux machine. ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Re: Remote encrypted backup with slow connection.
Piotr Karbowski wrote: On Tue, Sep 22, 2009 at 10:51 AM, Dominic Raferd domi...@timedicer.info wrote: Piotr Karbowski wrote: On Mon, Sep 21, 2009 at 8:21 PM, Dominic Raferd domi...@timedicer.info wrote: Piotr Karbowski wrote: On Mon, Sep 21, 2009 at 2:02 PM, Matthew Miller mat...@mattdm.org wrote: On Mon, Sep 21, 2009 at 01:58:11PM +0200, Piotr Karbowski wrote: local rdiff-backup dir with remote server but how? If I will use for example rsync it still need to check whole files for changes (read, download it) and upload only new. I hope you will understand what I need and help me. rsync won't check whole files unless you give the -c flag. Otherwise, it just compares metadata. I don't know if that's also the case with rdiff-backup, but I assume so. So I need to know how rdiff default compares data, if by size and mod-time, it will not be so painful but still itefficient will download changed files to generate diff. Rdiff-backup is designed to be ultra-efficient at this activity. It only sends the changes in a file over the wire, not the whole file. To do this it uses the librsync library which is effectively the same as rsync. You can read more about the technique at http://en.wikipedia.org/wiki/Rsync. rdiff-backup does not use file times to determine whether to do backups. It can backup very large files with small changes very quickly. Dominic You dont understand me, rdiff-backup is efficient, but to make diff it must read WHOLE file, on remote nfs or sshfs it is SLW and painful Sorry I get it now. But I think rdiff-backup and rsync require a separate computer at the remote end in order to optimise transfers, so if you are just accessing a remote share using sshfs or similar then they can still work of course but as you realise they will be slow. I guess it is not possible for you to run rdiff-backup (or rsync) at the remote end as well? You could run rdiff-backup locally to create a backup store and then mirror this store to the remote share using rcp. Still it will be slow because rdiff-backup always stores the latest copy of each file in full and so if this changes even slightly then the whole file will must be transferred by rcp. Duplicity http://duplicity.nongnu.org/ might work better for you, because it uses forward diffs. Also its archives are secure. Although not directly relevant I found a page here http://www.psc.edu/networking/projects/hpn-ssh/ which provides a patch to greatly speed up OpenSSH in some situations. Duplicity is interesing project. What you think about using rdiff-backup to create local backup, for example in /backups and then send this /backups to remote server by duplicity? As far as I know duplicity is encrypted so I DONT need using encfs, dmcrypt or other - only ssh access is needed (realy I dont need duplicity on remote server?). I just want be able to send _ENCRYPTED_ backups to remote server where I have only ssh access (sftp/scp work). I have not used duplicity myself, I use rdiff-backup. But I am not sure you need to run rdiff-backup first, I think duplicity may make its own local copies of backup increments so that it can send future increments without having to access the earlier increments from the remote share. And yes duplicity sends encrypted files so you don't need other encryption. Because duplicity uses forward diffs you have to keep all backups forever, and if there is any corruption of a file you lose all backups that occurred *after* the date of this file. Rdiff-backup uses reverse diffs so corruption, if it occurs, affects backups *before* the date of the corrupted backup. With rdiff-backup you can delete backups before a certain date (though in my experience the storage is so efficient it is not usually worth bothering), which is not possible with duplicity. Still it sounds like duplicity would better suit your needs. ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Re: Remote encrypted backup with slow connection.
Piotr Karbowski wrote: So best will be using duplicity to make local backup and run rsync to send new files (diffs) to remote server. and if backup will be TOO big just mv backup backup_old and start new backup (every 4 weeks for example) I think so. There is a duplicity mailing list where you could probably get more help: http://lists.nongnu.org/mailman/listinfo/duplicity-talk ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Error
prateekmoturi wrote: Hi i am getting the following error messages when i am trying to backup. I am using the rdiff-backup 1.2.8. Can anyone please help me to find the solution for the following error... Traceback (most recent call last): File /usr/bin/rdiff-backup, line 30, in ? rdiff_backup.Main.error_check_Main(sys.argv[1:]) File /usr/lib/python2.4/site-packages/rdiff_backup/Main.py, line 304, in error_check_Main try: Main(arglist) File /usr/lib/python2.4/site-packages/rdiff_backup/Main.py, line 324, in Main take_action(rps) File /usr/lib/python2.4/site-packages/rdiff_backup/Main.py, line 280, in take_action elif action == backup: Backup(rps[0], rps[1]) File /usr/lib/python2.4/site-packages/rdiff_backup/Main.py, line 337, in Backup backup_final_init(rpout) File /usr/lib/python2.4/site-packages/rdiff_backup/Main.py, line 501, in backup_final_init checkdest_if_necessary(rpout) File /usr/lib/python2.4/site-packages/rdiff_backup/Main.py, line 920, in checkdest_if_necessary dest_rp.conn.regress.Regress(dest_rp) File /usr/lib/python2.4/site-packages/rdiff_backup/regress.py, line 71, in Regress for rf in iterate_meta_rfs(mirror_rp, inc_rpath): ITR(rf.index, rf) File /usr/lib/python2.4/site-packages/rdiff_backup/regress.py, line 197, in iterate_meta_rfs for raw_rf, metadata_rorp in collated: File /usr/lib/python2.4/site-packages/rdiff_backup/rorpiter.py, line 100, in Collate2Iters try: relem2 = riter2.next() File /usr/lib/python2.4/site-packages/rdiff_backup/metadata.py, line 274, in iterate for record in self.iterate_records(): File /usr/lib/python2.4/site-packages/rdiff_backup/metadata.py, line 283, in iterate_records next_pos = self.get_next_pos() File /usr/lib/python2.4/site-packages/rdiff_backup/metadata.py, line 266, in get_next_pos newbuf = self.fileobj.read(self.blocksize) File /usr/lib/python2.4/gzip.py, line 225, in read self._read(readsize) File /usr/lib/python2.4/gzip.py, line 273, in _read self._read_eof() File /usr/lib/python2.4/gzip.py, line 309, in _read_eof raise IOError, CRC check failed IOError: CRC check failed This is what Andrew Ferguson (maintainer of rdiff-backup) wrote in response to a similar problem a couple of years ago: Given this error, and especially since one client is failing and three are working, I would be suspicious of the failing client's hardware (probably RAM or hard disk). Are all of your clients backing up to one server? What is the setup here? CRC is the Cyclic Redundancy Check and is a way of checking the integrity of a data stream. Were there any other messages logged from rdiff-backup? Perhaps in the rdiff-backup-data/error_log files? And here is what someone else wrote about this error: This problem has come up many times on the mailing list, and so far there is not a good fix for it... usually it appears when one of the .gz files was incomplete. Try this: find rdiff-backup-data/ -name '*.gz | while read a; do echo -n $a: gzip --test $a done This will tell you which files are bad. In our case, we had a problem with the mirror_metadata files. Search the archive for specifics, but basically I copied a good mirror_metadata over the bad mirror_metadata and crossed my fingers. It worked for us, though I do not know the implications here. After doing this surgery, you might try a restore of the past few backup dates and make sure it is successful. Worst case, you rm-rf rdiff-backup-data, though you will loose your increments. HTH, Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Problem with Detection of Multiple rdiff-backup instances
Jakob Unterwurzacher wrote: in bash: #!/bin/bash for i in `seq 1 $((RANDOM%100))`; do /bin/true; done But you should make sure that $RANDOM assumes different values after a reboot (same for perl's rand(100), which is likely more advanced and less likely to suffer from that problem). Jakob this can be done by prefixing your 'for' line with: RANDOM=$(($(date +%s) % 32768)) this seeds the random number generator based on the number of seconds since 1/1/70, so is likely to avoid repeats. Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Verify times increasing
listserv.traf...@sloop.net wrote: I'm not aware, so if I'm wrong perhaps someone could correct me, but I'd like a command to, in essence, do a comprehensive --verify-all-files-in-the-archive. [I'm pretty sure such a thing doesn't exist, at least I never saw it in the docs.] This would apply all deltas to *all* files (back to the oldest copy) and compare the stored hashes at the time of backup to the rebuilt file. [Note all the files, not just those in a particular target date/delta.] This wouldn't verify that every file would be correct in every delta version, but it would, I think, get as close as one might come to that. I think it is an excellent proposal (as in, how come rdiff-backup doesn't already offer that?) But development of rdiff-backup seems to have gone quiet recently, I guess Andrew has been busy? Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Failover to warm standby and restore
Jakob Unterwurzacher wrote:
steve.du...@pgcmls.info schrieb:
Dear all,
A friend recommended that I use rdiff-backup to maintain a back up of an
inconveniently large, reasonably slowly changing file repository.
I would like to keep the back up copy on a warm standby system, so that if the
primary host fails, I can reconfigure the standby to replace it.
I understand that modifying the back up files will invalidate the rdiff
snapshots.
Is there a sane way of keeping the snapshots valid that does not involve
maintaining a second back-up copy?
I think you will need LVM snapshots for this.
When the standby goes hot, create a read/write snapshot of the backup,
make that snapshot available to the clients.
When the primary host comes back up, are there any extraordinary considerations
for updating its repository ?
You will have to push the changes in the snapshot somehow to the primary
(rsync -au ?).
Then you can destroy the snapshot and do business as usual.
Very neat tip from Jakob, and makes me glad I use LVM! Depending on
circumstances it might even be possible to make this an automatic
failover solution, which would be even neater.
The physical volumes in the LVM volume group must be in LVM2 metadata
format to create a read/write snapshot ('vgdisplay' will tell you).
[LVM2 (try 'lvm version') uses LVM2 format by default, I think, but you
can use 'pvcreate' with '-M2' switch to ensure it does. You can convert
an existing LVM1 vg with 'vgconvert -M2'.]
Allocate plenty of space for the snapshot - if it runs out of space you
lose it entirely. Although a short-lived snapshot is likely to require
little space compared with the original logical volume (it is
effectively a diff of it I guess), the only absolutely safe size is the
same as the original logical volume.
Dominic
___
rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] locking the rdiff-backup destination
Jakob Unterwurzacher wrote: I guess you will have to use a wrapper script for your rsync job and your rdiff-backup job that does the locking (and obeys the lock). If you start both jobs at the same machine, it's easy: Put this at the start of each script - it will then exit when another instance is running. # #!/bin/bash exec 200 /tmp/lockfileXYZ flock -n 200 || echo 'Could not get lock!' exit 1 # Nice tip! An alternative which doesn't require flock or a temporary lock file is (again for a bash script): ## #!/bin/bash [ -n `ps h -C rdiff-backup` ] echo `basename $0` aborting - rdiff-backup is running exit 1 ## Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] rdiff-backup on USB distro
Sorry I don't know of any at the moment. I use Devil-Linux (http://www.devil-linux.org/), and have requested that rdiff-backup be added. Let me know whether you find an alternative. If no other USB-stick distros support it then that alone is a good reason to add rdiff-backup to Devil-Linux. Dominic Renato E. Silva wrote: Hi all, I was wondering, is there any distribution that is bootable from an USB stick and has rdiff-backup for use? I'm trying to mount a backup server but my boss would like me to check if there's a simple distribution so we could run from an USB stick. I've been looking around on DSL, Puppy and even FreeNAS but no luck though. I appreciate any help or directions. Renato E. Silva ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Unable to backup due to remote system closing connection
Hi Alex There is a long thread about this issue at http://www.howtoforge.com/forums/showthread.php?t=3618. But the problem in the end seemed to be (in this case) that the user had generated the authorized_keys file with nano and had not switched off hard wrapping, so that when the file was saved nano broke up the line. If you are using nano prior to version 2.1.11, use the -w switch on the command line: nano -wc /mnt/backup/.ssh/authorized_keys If you have the latest and greatest nano, you can use soft-wrapping which is *much* nicer: nano -$c /mnt/backup/.ssh/authorized_keys If you are using a different editor, have a look for the same problem i.e. what you intend as a single line is being saved as multiple lines Dominic Alex Pounds wrote: Hello, I'm trying to set up backups from one machine (alan) to a backup host (gertie) using rdiff-backup. Unfortunately, I haven't been able to get it working. I'm sure I'm making a mistake somewhere, but reading the documentation and experimentation hasn't got me any closer. Hopefully someone on the mailing list will be able to help... On the backup server, gertie, I created a user 'rdiffbackup'. From /etc/passwd: rdiffbackup:x:118:1001:Automated backups:/mnt/backup:/bin/false In /mnt/backup/.ssh/authorized_keys, I have a line like this (with no linebreaks): command=/usr/bin/rdiff-backup -v9 --server,no-X11-forwarding,no-port-forwarding,no-pty ssh-rsa abde[long ssh key]abcde r...@alan On alan, I run the following command to perform the backup, and get the following output: alan ~$ sudo rdiff-backup --print-statistics --exclude-globbing-filelist /root/bin/excludes.txt / rdiffbac...@gertie::/mnt/backup Connection closed by 10.5.2.2 Fatal Error: Truncated header string (problem probably originated remotely) Couldn't start up the remote connection by executing ssh -C rdiffbac...@gertie rdiff-backup --server Remember that, under the default settings, rdiff-backup must be installed in the PATH on the remote system. See the man page for more information on this. This message may also be displayed if the remote version of rdiff-backup is quite different from the local version (1.2.8). Both servers are running Ubuntu 9.10, and have rdiffbackup 1.2.8 on them. Running the ssh command 'by hand' with the -vv option produces the following debug output, after a straightforward auth procedure: debug1: Authentications that can continue: publickey,password debug1: Next authentication method: publickey debug1: Trying private key: /root/.ssh/identity debug1: Offering public key: /root/.ssh/id_rsa debug2: we sent a publickey packet, wait for reply debug1: Remote: Forced command: /usr/bin/rdiff-backup -v9 --server debug1: Remote: X11 forwarding disabled. debug1: Remote: Port forwarding disabled. debug1: Remote: Pty allocation disabled. debug1: Server accepts key: pkalg ssh-rsa blen [num] debug2: input_userauth_pk_ok: fp [fingerprint] debug1: read PEM private key done: type RSA debug1: Remote: Forced command: /usr/bin/rdiff-backup -v9 --server debug1: Remote: X11 forwarding disabled. debug1: Remote: Port forwarding disabled. debug1: Remote: Pty allocation disabled. Connection closed by 10.5.2.2 Any suggestions as to what I've misconfigured, or how I can produce some helpful logs/debugging output, are gratefully received. Many thanks, ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Excluding directories for restore
Hi Dominik Although it is not an answer to your present predicament, I would say that rdiff-backup is not advisable over an unstable connection. A better strategy is to run rdiff-backup locally (preferably to a different machine) and then use rsync to mirror the rdiff-backup repository to an offsite machine using rsync with the --link-dest option. I recently posted on this mailing list an example of how to do this. Problems with the offsite backup process are minimised and you have a primary and secondary backup. Dominic Dominik Sandjaja wrote: Hi, I have a setup of a server being backed up regularly over a DSL line to my server at home. After quite a bad crash I am in the process of restoring the data... Thanks Dominik ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] native VSS (Shadow Copy) support
On 06/04/2010 17:03, Greg Freemyer wrote: On Tue, Apr 6, 2010 at 11:27 AM, Randy Syringrsyr...@inteli-com.com wrote: Josh Nisly wrote: Two things: 1) The rdiff-backup project likely won't accept patches for features that are platform specific. There are exceptions for OS-specific filesystem metadata, but VSS doesn't fall into that category. Now that there is consideration of restarting development, any chance we could reconsider this? Any backup solution has to wrestle with the Windows user base and the key issue on Windows is whether or not a backup solution uses VSS. If rdiff-backup supported VSS, its uptake might be huge. At very best, maybe a plugin system so that VSS can be added easily without needing to touch the core code? I don't post here often, but I strongly agree that snapshot technology should be leveraged by the rdiff-backup environment. I'm not as positive that it needs to be in the tool itself. I can't remember the name of the package, but I believe there is a linux package that wraps rdiff-backup and uses LVM snapshots from which it runs rdiff-backup. As to VSS, it is supported in Windows XP SP3, 2003, Vista, 2008, etc. So it is a long term technology that it here to stay. To me the only thing to really discuss is how vss support should be integrated into a windows rdiff-backup environment, not if it should be. 2) I've written a python extension module in C++ to implement VSS. Is that something that you'd be interested in? Very, please share. Have you integrated this with rdiff-backup at all? The great thing about C code, etc. (as opposed to just scripts calling the Vista provided CLI commands) is that VSS CLI commands are not included in XP SP3 iirc. And also with XP SP3 you only have 20 or 30 seconds to mount the VSS snapshot after you initialize it. (again, iirc) So having it in a real program as opposed to just a script makes it easier to have the error / retry logic built in. === And for those that don't know much about VSS snapshots, one really cool feature is that services like Exchange, MS-SQL, etc. have vss knowledge integrated into them. They register themselves with the vss service and whenever a snapshot is created, the registered services are notified to quiesce themselves prior to the snapshot being made. I don't know if people are backing up that class of service via rdiff-backup, but if they are it is pretty critical that vss snapshots be used in order to ensure you have quiesced data. vss also solves the open file issue that is so problematic with windows backups. Thanks for reading Greg I have a Windows utility called TimeDicer (www.timedicer.co.uk) which is a wrapper for rdiff-backup, backing up from Windows to Linux, and it uses VSS. So it is not essential to have VSS built into rdiff-backup, even though it would be nice. Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Stable API and Non-recursive list-at-time
On 22/06/2010 19:28, Darren Hart wrote: On Tue, Jun 22, 2010 at 10:07 AM, Anthony Toolearto...@gmail.com wrote: Hi Darren, Are you aware that there is already a rdiff-backup FUSE plugin? http://code.google.com/p/archfs/ I had no idea! This is excellent, I'll have a look and see how it does. Thanks for the link - I'm glad I posted early ;-) When I last used archfs (at v 0.5.3) it worked fine for small repositories (say 100 files) but for large ones (say 5000 files) it was very slow and liable to crash the server. Has anyone got any better experience with it? It's a really neat concept and if someone could make it faster and reliable, it would be great. Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Stable API and Non-recursive list-at-time
On 28/06/2010 14:22, Darren Hart wrote: ...What I'm looking to do is occasionally mount the backup repository, browse for a single file, and restore it from some time in the past, then unmount the repo. I think my approach of building the directory listings from the responses provided by the rdiff-backup API is a fine way to do that. Each command has some latency involved, but it should be adequate. So my original questions stands - what do the rdiff-backup developers think about the API, is it likely to remain relatively stable? What about the recursive option I asked about, is such a patch likely to be accepted? -- Darren Not answering your direct question (because I can't), but if you want to recover individual files the best I have found is Josh's rdiffWeb http://www.rdiffweb.org/. As the name implies it uses a web interface and although the initial configuration can be a bit tricky, once set up it works well and makes retrieving individual files, including versions from the past, a snap. Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Crash on incremental backup after regression
It looks to me as if the problem maybe relates to the special characters
in the filenames '\xf3' and similar. And I see at the bottom mention of
a win_acl. My suggestions are:
- try using option --override-chars-to-quote - or
- try using option --no-acls - or
- try rdiff-backup version 1.2.8. [1.3.3 is experimental.] - or
- rename the offending files
Dominic
On 26/11/2010 13:11, Luciano Leveroni wrote:
Hi,
Im using rdiff-backup version 1.3.3 on a Windows 2000 server here and
the software is crashing on performing a backup. As the previous
backups have failed, rdiff-backup makes a regression which seems
successful and the, on making the backup itself, the script crashes
with the following output:
# rdiff-backup.exe -v5 P:/Documentos Z:/Documentos
Z:/rdiff-last-output.txt 21
...
Processing changed file Contaduria/ANA/santander rio solicitud de
chequeras.doc
Incrementing mirror file Z:/Documentos/Contaduria/ANA/santander rio
solicitud de chequeras.doc
Processing changed file Contaduria/ANA/standard bank pedido de cheque
rechazado.doc
Incrementing mirror file Z:/Documentos/Contaduria/ANA/standard bank
pedido de cheque rechazado.doc
Processing changed file Contaduria/ANA/wlsetup-custom.exe
Incrementing mirror file Z:/Documentos/Contaduria/ANA/wlsetup-custom.exe
Processing changed file Contaduria/ANA/wlsetup-web.exe
Incrementing mirror file Z:/Documentos/Contaduria/ANA/wlsetup-web.exe
Processing changed file Contaduria/Ana
Incrementing mirror file Z:/Documentos/Contaduria/Ana
UpdateError Claudio/Konzert 2010/rdiff-backup.tmp.85491 [Errno 2] No
such file or directory:
'Z:/Documentos/rdiff-backup-data/increments/Claudio/Konzert 2010/DWG -
VOXPOP y sus voces en el Bicentenario en la Embajada de Alemania 2010
- Invitaci\xf3n para el Sr_ Carlos Brady Alet y Sra_ (Alchouron,
Berisso, Brady Alet Fernandez
Pelayo).eml.2010-09-15T23-00-01-03-00.missing'
UpdateError Claudio/Konzert 2010/rdiff-backup.tmp.85492 [Errno 2] No
such file or directory:
'Z:/Documentos/rdiff-backup-data/increments/Claudio/Konzert 2010/DWG -
VOXPOP y sus voces en el Bicentenario en la Embajada de Alemania 2010
- Invitaci\xf3n para el Sr_ Cosme Roberto Smiraglia y Sra_ (MAN
FERROSTAAL ARGENTINA S_A_).eml.2010-09-15T23-00-01-03-00.missing'
UpdateError Claudio/Konzert 2010/rdiff-backup.tmp.85493 [Errno 2] No
such file or directory:
'Z:/Documentos/rdiff-backup-data/increments/Claudio/Konzert 2010/DWG -
VOXPOP y sus voces en el Bicentenario en la Embajada de Alemania 2010
- Invitaci\xf3n para el Sr_ Mart\xedn Moncayo von Hase y Sra_ (ZANG,
BERGEL y VI\xd1ES Abogados).eml.2010-09-15T23-00-01-03-00.missing'
UpdateError Claudio/MLK/Friedrich und Brigitte
Werner/rdiff-backup.tmp.85502 [Errno 2] No such file or directory:
'Z:/Documentos/rdiff-backup-data/increments/Claudio/MLK/Friedrich und
Brigitte Werner/Fw_ MLK Notwendigkeits- Projekte - Wunschliste,
Erg\xe4nzungen Re_ Stiftung Brigitte und Friedrich Werner
---Fw_ Heutiger Besuch im MLK.eml.2010-09-15T23-00-01-03-00.missing'
UpdateError Claudio/MLK/Friedrich und Brigitte
Werner/rdiff-backup.tmp.85503 [Errno 2] No such file or directory:
'Z:/Documentos/rdiff-backup-data/increments/Claudio/MLK/Friedrich und
Brigitte Werner/Re_ MLK Notwendigkeits- Projekte - Wunschliste,
Erg\xe4nzungen Re_ Stiftung Brigitte und Friedrich Werner -
Comentarios sobre valuaci\xf3n.eml.2010-09-15T23-00-01-03-00.missing'
Exception 'QuotedPath:
Z:/Documentos/rdiff-backup-data/increments/Contaduria/Ana.2010-09-15T23-00-01-03-00.dir
Index: ('Contaduria', 'Ana.2010-09-15T23-00-01-03-00.dir')
Data: {'win_acl': u'# file:
Contaduria/Ana.2010-09-15T23-00-01-03-00.dir\nO:BAG:DUD:AI(A;ID;FA;;;WD)\n',
'uid': 0, 'perms': 438, 'type': 'reg', 'gname': None, 'ctime':
1290745523L, 'devloc': 0, 'uname': None, 'nlink': 0, 'gid': 0,
'mtime': 1284475197L, 'atime': 1290745523L, 'inode': 0L, 'size': 0L}'
raised of class 'type 'exceptions.AssertionError'':
File rdiff_backup\Main.pyc, line 306, in error_check_Main
File rdiff_backup\Main.pyc, line 326, in Main
File rdiff_backup\Main.pyc, line 282, in take_action
File rdiff_backup\Main.pyc, line 345, in Backup
File rdiff_backup\backup.pyc, line 51, in Mirror_and_increment
File rdiff_backup\backup.pyc, line 251, in patch_and_increment
File rdiff_backup\rorpiter.pyc, line 284, in __call__
File rdiff_backup\backup.pyc, line 732, in start_process
File rdiff_backup\increment.pyc, line 41, in Increment
File rdiff_backup\increment.pyc, line 103, in makedir
File rdiff_backup\increment.pyc, line 123, in get_inc
Traceback (most recent call last):
File rdiff-backup, line 30, in module
File rdiff_backup\Main.pyc, line 306, in error_check_Main
File rdiff_backup\Main.pyc, line 326, in Main
File rdiff_backup\Main.pyc, line 282, in take_action
File rdiff_backup\Main.pyc, line 345, in Backup
File rdiff_backup\backup.pyc, line 51, in Mirror_and_increment
File rdiff_backup\backup.pyc, line 251, in
Re: [rdiff-backup-users] Daemon VS ssh
On 26/11/2010 11:14, Valerio Pachera wrote: 2010/11/25 Jacob Anawaltjlanaw...@gmail.com: Look into the remote-schema option, search around for rdiff-backup and netcat, and there are ssh cipher options. I found an article http://osdir.com/ml/sysutils.backup.rdiff-backup.general/2006-02/msg00088.html I read about --remote.schema but, honestly, I didn't digested it so well. Could you please tell me if in the point 2 of the article, ssh is involved or not? Most of all, I'm wondering if there's an easy way to have a central backup server that contact windows and linux clients to back up their data. You can certainly do this with a package such as (for Windows) TimeDicer http://www.timedicer.co.uk. This uses rdiff-backup with ssh connection managed by plink. But this is for 'push' backups - the clients contact the server and send backups. Works well with a primary server on the local network, and offsite mirror server for belt'n'braces protection. Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: AW: [rdiff-backup-users] Rdiff backup still chokes if read-only files are found at target
David: Following your posting, I have tested the behaviour of our system backing up read-only folders and files with rdiff-backup, and it works correctly, even when the read-only files change. We are backing up from Windows machines but to a Linux server (using TimeDicer as wrapper for rdiff-backup), I suspect this is where your problem lies. So a possible solution is to move your backup repository to a Linux server, this could even be a VM hosted on your Windows machine. Dominic On 29/11/2010 10:04, D. Kriesel wrote: Anybody here? ;-) The list seems to be a bit quiet. If this is the wrong place for this report, someone please tell me the correct one :-) I feel I did not provide enough information to get rid of the below bug. So, here you go: 1.I am using rdiff 1.3.3 for a local backup on a windows server 2003 32bit machine. 2.I am using the windows release that can be downloaded on rdiff-backup.nongnu.org as rdiff-backup-1.3.3-win32.zip. 3.My rdiff call is: rdiff-backup --force --no-acls --no-hard-links --print-statistics --include D:\\/home --include D:\\/profile --exclude D:\\/share/ElRv-Daten --include D:\\/share --exclude D:\\/** D:\/ D:\/rd/data 4.Zugriff verweigert in the below message is german for permission denied. When checking the folders, I found that the ACL entries are okay, but the read-only flag on the Eigene Bilder folder was set. So all in all, *it**seems to me that rdiff cannot remove this flag from folders*. Below linked you find a similar problem with files. Next, *windows**prevents rdiff from deleting a folder that is still marked read-only*. Consecuently, Rdiff crashes. To support my concern, I can tell you that you find quite a number of „read only folders“ on windows systems, because that windows itself uses the read only flag on folders differently than on files: see http://support.microsoft.com/kb/256614 . Additionally, the attribute setting for folders may be handled different (for example, you can’t remove read only attribute on folders just using the explorer). Any help is appreciated, really... I like rdiff and don’t want to chance the backup system (again)... Cheers, David -Ursprüngliche Nachricht- Von: rdiff-backup-users-bounces+mail=dkriesel@nongnu.org [mailto:rdiff-backup-users-bounces+mail=dkriesel@nongnu.org] Im Auftrag von D. Kriesel Gesendet: Sonntag, 28. November 2010 01:52 An: Lista rdiff-backup Betreff: [rdiff-backup-users] Rdiff backup still chokes if read-only files are found at target Hi guys, I have rdiff crashing when read-only files are in the rdiff repository. It gives the following Exceptions: Exception '[Error 5] Zugriff verweigert: 'D:\\/rd/data/home/kwasny/Eigene Dateie n/Eigene Bilder'' raised of class 'type 'exceptions.WindowsError'': File rdiff_backup\Main.pyc, line 306, in error_check_Main File rdiff_backup\Main.pyc, line 326, in Main File rdiff_backup\Main.pyc, line 282, in take_action File rdiff_backup\Main.pyc, line 345, in Backup File rdiff_backup\backup.pyc, line 51, in Mirror_and_increment File rdiff_backup\backup.pyc, line 251, in patch_and_increment File rdiff_backup\rorpiter.pyc, line 277, in __call__ File rdiff_backup\rorpiter.pyc, line 229, in finish_branches File rdiff_backup\backup.pyc, line 676, in end_process File rdiff_backup\rpath.pyc, line 993, in rmdir Traceback (most recent call last): File rdiff-backup, line 30, in module File rdiff_backup\Main.pyc, line 306, in error_check_Main File rdiff_backup\Main.pyc, line 326, in Main File rdiff_backup\Main.pyc, line 282, in take_action File rdiff_backup\Main.pyc, line 345, in Backup File rdiff_backup\backup.pyc, line 51, in Mirror_and_increment File rdiff_backup\backup.pyc, line 251, in patch_and_increment File rdiff_backup\rorpiter.pyc, line 277, in __call__ File rdiff_backup\rorpiter.pyc, line 229, in finish_branches File rdiff_backup\backup.pyc, line 676, in end_process File rdiff_backup\rpath.pyc, line 993, in rmdir WindowsError: [Error 5] Zugriff verweigert: 'D:\\/rd/data/home/kwasny/Eigene Dat eien/Eigene Bilder' Sounds similar to me like the old read-only file issue. http://www.mail-archive.com/rdiff-backup- http://www.mail-archive.com/rdiff-backup-users@nongnu.org/msg03549.html us...@nongnu.org/msg03549.html http://www.mail-archive.com/rdiff-backup-users@nongnu.org/msg03549.html Any advice? Cheers, David ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL:
Re: [rdiff-backup-users] rdiff-backup fails, no gzipped file
See Andrew Ferguson's response regarding this error message here: http://www.backupcentral.com/phpBB2/two-way-mirrors-of-external-mailing-lists-3/rdiff-backup-23/regression-fails-ioerror-not-a-gzipped-file-93222/ Andrew is the current maintainer of rdiff-backup so his word is law! What version of rdiff-backup are you using? 2.8-6 is not a valid version number, the current stable version is 1.2.8. Check with: rdiff-backup -V Dominic On 05/12/10 17:12, ~D wrote: Hi, I did setup a backup system like here, but then the other way around, backup my desktop to the server. http://www.howtoforge.com/linux_rdiff_backup I have two scripts on my server. The first one seems to run fine, the second not atm (till a week ago it went well, I didn't change a thing on a the server side) ... I have version 2.8-6 on both sides. On the server I installed the version of Debian testing on Lenny (Stable), which is on the server. I do run Debian testing on my desktop. See the messages below. Thanks in advance, ~D $ ./laptop_backup Previous backup seems to have failed, regressing destination now. * Exception 'Not a gzipped file' raised of class 'type 'exceptions.IOError'': ... File /usr/lib/python2.5/gzip.py, line 263, in _read self._read_gzip_header() File /usr/lib/python2.5/gzip.py, line 164, in _read_gzip_header raise IOError, 'Not a gzipped file' IOError: Not a gzipped file Fatal Error: Lost connection to the remote system ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] rdiff-backup fails, no gzipped file
On 07/12/2010 14:42, ~D wrote: On 12/07/2010 03:26 PM, D. Kriesel wrote: Do you prevent a shutdown or reboot when rdiff-backup is running? How? Since rdiff-backup does not like backup interruptions (and therefore is not usable on slow or unreliable connections) I rdiff to a local repository on the server (the server usually doesn't reboot) and rsync the entire repository to the remote destination. My backup only runs when there is a internet connection. I use my laptop at home most of the time. I have to find a way to check wether rdiff-backup is running and to reboot after it shutdown. I can use htop of course, but I am not sure if rdiff-backup is runnin also local on my laptop or only on my server. Maybe there is a more advanced way to do this? You can test for running rdiff-backup locally thus: [ -n `ps -o pid --no-heading -C rdiff-backup` ] echo running || echo not running and you can query remote server thus: [ -n `ssh u...@remote.server ps -o pid --no-heading -C rdiff-backup 2/dev/null` ] echo running || echo not running ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] rdiff-backup fails, no gzipped file
On 07/12/10 18:54, ~D wrote: On 12/07/2010 05:09 PM, Dominic Raferd wrote: On 07/12/2010 14:42, ~D wrote: On 12/07/2010 03:26 PM, D. Kriesel wrote: Do you prevent a shutdown or reboot when rdiff-backup is running? How? Since rdiff-backup does not like backup interruptions (and therefore is not usable on slow or unreliable connections) I rdiff to a local repository on the server (the server usually doesn't reboot) and rsync the entire repository to the remote destination. My backup only runs when there is a internet connection. I use my laptop at home most of the time. I have to find a way to check wether rdiff-backup is running and to reboot after it shutdown. I can use htop of course, but I am not sure if rdiff-backup is runnin also local on my laptop or only on my server. Maybe there is a more advanced way to do this? You can test for running rdiff-backup locally thus: [ -n `ps -o pid --no-heading -C rdiff-backup` ] echo running || echo not running and you can query remote server thus: [ -n `ssh u...@remote.server ps -o pid --no-heading -C rdiff-backup 2/dev/null` ] echo running || echo not running . Ah Thanks! So it should be possible to make some script which checks if it's is running, if no, shutdown, if yes, wait 15 minuts and check again etc... right? ~D yes you could write a bash script and put it as a job in your crontab to run every 15 minutes, say... My mirror backup server (not my primary) is updated from my primary by a script which uses rdiff. The script runs on the primary and starts by waking up the mirror server (over the internet), then runs rsync, then after checking all is well, powers down the mirror. Works well with rsync - as David pointed out earlier, it is not a good idea to run rdiff-backup over an unstable connection. Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] rdiff-backup fails, no gzipped file
On 07/12/2010 22:40, ~D wrote: On 12/07/2010 09:33 PM, Dominic Raferd wrote: On 07/12/10 18:54, ~D wrote: On 12/07/2010 05:09 PM, Dominic Raferd wrote: On 07/12/2010 14:42, ~D wrote: On 12/07/2010 03:26 PM, D. Kriesel wrote: Do you prevent a shutdown or reboot when rdiff-backup is running? How? Since rdiff-backup does not like backup interruptions (and therefore is not usable on slow or unreliable connections) I rdiff to a local repository on the server (the server usually doesn't reboot) and rsync the entire repository to the remote destination. My backup only runs when there is a internet connection. I use my laptop at home most of the time. I have to find a way to check wether rdiff-backup is running and to reboot after it shutdown. I can use htop of course, but I am not sure if rdiff-backup is runnin also local on my laptop or only on my server. Maybe there is a more advanced way to do this? You can test for running rdiff-backup locally thus: [ -n `ps -o pid --no-heading -C rdiff-backup` ] echo running || echo not running and you can query remote server thus: [ -n `ssh u...@remote.server ps -o pid --no-heading -C rdiff-backup 2/dev/null` ] echo running || echo not running . Ah Thanks! So it should be possible to make some script which checks if it's is running, if no, shutdown, if yes, wait 15 minuts and check again etc... right? ~D yes you could write a bash script and put it as a job in your crontab to run every 15 minutes, say... My mirror backup server (not my primary) is updated from my primary by a script which uses rdiff. The script runs on the primary and starts by waking up the mirror server (over the internet), then runs rsync, then after checking all is well, powers down the mirror. Works well with rsync - as David pointed out earlier, it is not a good idea to run rdiff-backup over an unstable connection. I was thinking about a script on my laptop for shutdown my machine, instead of running sudo shutdown -h now, run a script which only make the laptop shutdown when rdiff-backup is not runnning. I'm using fluxbox, so I'm used to shutdown my machine via terminal. ~D Here's a script 'auto-shutdown.sh' I use which shuts down a machine if there is no ssh connection and certain other programs are not running: if [ -z `netstat -t|grep :ssh.*ESTABLISHED$` ]; then [ `ps -A|grep -Ec cp$|rsync$|mv$|rm$` = 0 ] shutdown -h now fi add a line to /etc/crontab thus, to run the script hourly: 01 ** * * root/opt/auto-shutdown.sh ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] rdiff-backup fails, no gzipped file
yes you could write a bash script and put it as a job in your crontab to run every 15 minutes, say... My mirror backup server (not my primary) is updated from my primary by a script which uses rdiff. The script runs on the primary and starts by waking up the mirror server (over the internet), then runs rsync, then after checking all is well, powers down the mirror. Hmm I just have a small NAS server (qnap 109), no 'mirror backup server' here. What do you mean by 'waking up the mirror server'? ~D Wiki Well our main backup is done by all our machines to a server on our LAN using rdiff-backup - this I call the 'primary' backup server. Then each night this primary machine runs a script which uses rsync to copy its contents to an offsite machine, which therefore is maintained as a 'mirror' of the primary. The mirror machine is normally switched off. To switch it on, the script sends a 'magic packet' to the mirror machine, thus turning it on. Then the script runs rsync, and when that is over, it shuts the mirror machine down again. In a simple case, you can use the wakonlan utility to wake the remote machine: see http://gsd.di.uminho.pt/jpo/software/wakeonlan/. If it isn't already available on your machine and you use Debian or Ubuntu you can add it with apt-get I think. If the remote machine is behind another router then waking it may be more complicated: I have a special script to pass through Netgear DG834G router, for example (see http://www.timedicer.co.uk/dg834g.) Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
[rdiff-backup-users] librsync.so and _librsync.so
Can anyone explain the relationship between /usr/lib/librsync.so, and /usr/local/lib/python2.6/dist-packages/rdiff_backup/_librsync.so? They are not the same: /usr/lib/librsync.so is symlinked to /usr/lib/librsync.so.1.0.2 which is 46612 bytes, and I think came via apt-get install librsync-dev /usr/local/lib/python2.6/dist-packages/rdiff_backup/_librsync.so is 30110 bytes, is located in my rdiff-backup installation location, and seems to have come with rdiff-backup in the tarfile Both Rdiff.py and robust.py import librsync [i.e. librsync.py?], and librsync.py imports _librsync. When I built rdiff-backup I specified --librsync-dir=/usr/lib. Which file is rdiff-backup using: librsync.so or _librsync.so? Are they alternatives and if so which one should it use? Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Getting a comprehensive file listing of a directory
On 21/12/2010 22:56, Greg Freemyer wrote: All, I've got a directory that seems to be missing some files. I don't recall their names or exactly when they existed. rdiff-backup -l --list-increment-sizespath-to-backup-dir shows me that I have about 10 instances of that directory backup up. Is there some way to get a list of every file that I've ever backed up for that directory tree? Even if I see the same file 10 times, I'm happy. I just want to see all the files. Thanks Greg If you don't mind using a web application rather than CLI, have you considered rdiffWeb? In each directory this shows all files that ever existed as well as the currently-existing ones, and you can click'n'choose which revision to retrieve. I don't know how rdiffWeb does this 'under the hood' but I guess it calls some of rdiff-backup's internal functions (rdiffWeb, like rdiff-backup, is a python app). Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] [PATCH] Sparse file support
On 03/01/2011 20:27, Eric Wheeler wrote: On Sun, Jan 02, 2011 at 08:11:00PM -0800, Eric Wheeler wrote: Feedback and comments are appreciated! Original Message- From: Matthew Miller How will this interact with existing backups? rdiff-backup re-writes changed destination files, so if a source file changes, rdiff-backup will write the whole file after calculating a reverse-diff for the increment tree. Existing backup trees will become sparse over time, as files change. -Eric So I guess it is forward-compatible, but not backward-compatible? Existing repositories can be read fine with rdiff-backup patched for sparse file support, but repositories created with sparse file support cannot be read by unpatched rdiff-backup. That's okay, but it might make most of us feel that we would rather not use this patch unless we specifically need it, until it is part of mainstream rdiff-backup. Sadly, development of rdiff-backup has gone silent for a while now. I think these patches are great (as were Daniel Miller's a while ago, which provided a --verify-full option) but no one from the development side seems to be updating the code at the moment. (I would love to be proved wrong.) - Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] [PATCH] Sparse file support
AFAIK Andrew's last posting on this newsgroup was March 2009, and his last entry in CVS was January 2010, which is indeed the last entry by anyone. Josh (who also created the excellent rdiffWeb GUI front-end) was last seen here April 2010. You could try requesting to become a project member http://savannah.nongnu.org/my/groups.php?words=rdiff-backup#searchgroup? Dominic On 04/01/2011 15:38, D. Kriesel wrote: I would also love you to be proved wrong, but some weeks ago i asked the head developer if he is still active in the project and did not even get an answer. Dominic Raferddomi...@timedicer.co.uk schrieb: On 03/01/2011 20:27, Eric Wheeler wrote: On Sun, Jan 02, 2011 at 08:11:00PM -0800, Eric Wheeler wrote: Feedback and comments are appreciated! Original Message- From: Matthew Miller How will this interact with existing backups? rdiff-backup re-writes changed destination files, so if a source file changes, rdiff-backup will write the whole file after calculating a reverse-diff for the increment tree. Existing backup trees will become sparse over time, as files change. -Eric So I guess it is forward-compatible, but not backward-compatible? Existing repositories can be read fine with rdiff-backup patched for sparse file support, but repositories created with sparse file support cannot be read by unpatched rdiff-backup. That's okay, but it might make most of us feel that we would rather not use this patch unless we specifically need it, until it is part of mainstream rdiff-backup. Sadly, development of rdiff-backup has gone silent for a while now. I think these patches are great (as were Daniel Miller's a while ago, which provided a --verify-full option) but no one from the development side seems to be updating the code at the moment. (I would love to be proved wrong.) - Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: AW: [rdiff-backup-users] [PATCH] Sparse file support
On 08/01/11 10:31, D. Kriesel wrote: Hi Dominic, this is certainly a cool idea, only I have no Python developer experience und therefore you for sure don't want me as a project member :-). Currently, I just try to help out people on the mailinglist from my experience as a computer scientist and rdiff-backup user. ditto (except the scientist bit) Anyway, there is hope. I am currently preparing my PhD project, and plan to use python for some scripting. Once I feel my understanding of python and the internals of rdiff-backup is good enough, I will file a request as you said. Thanks. In the meantime, if there are any python programmers who use and value rdiff-backup and could offer their services to the project, the rest of us would be very grateful I'm sure. Dominic AFAIK Andrew's last posting on this newsgroup was March 2009, and his last entry in CVS was January 2010, which is indeed the last entry by anyone. Josh (who also created the excellent rdiffWeb GUI front-end) was last seen here April 2010. You could try requesting to become a project member http://savannah.nongnu.org/my/groups.php?words=rdiff- backup#searchgroup? Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Re: What happens if you add a --exclude to an existing rdiff-backup?
Thanks David, that is helpful. It would be good if there was a way of removing a subset of data from the entire repository. So let's say I put a 500GB folder in /home by accident and it has gone into the repository and is bloating it. I can exclude it from my future rdiff-backup runs but the folder will still be held as snapshot[s]. If I run --remove-older-than it will remove all data older than whenever, but I want to keep all the other stuff and just remove this folder (and its contents). Quite a common scenario but rdiff-backup can't handle it (AFAIK) and I don't know of a reliable workaround (apart from: get a bigger disk for the repository). Dominic On 08/01/11 10:15, D. Kriesel wrote: Hi, according to rdiff-backups doc, excluded files are just treated as if they would not exist. This means that a snapshot of such a file will be created in the metadata once a backup run with the exclusion is performed, and the file will be deleted from the mirror data.llyu Cheers, david Dominic Raferddomi...@timedicer.co.uk schrieb: I agree that makes sense in terms of the question in the body of your posting. But the subject of your posting was a slightly different question: 'What happens if you add a --exclude to an existing rdiff-backup?' If a week ago you added --exclude /home/fred to your rdiff-backup line backing up /home, will /home/fred now be removed from the destination by a --remove-older-than 5D run? In other words, if you add exclusion criteria to an existing rdiff-backup run, are the copies of the newly-excluded files removed from the main repository and placed in the increments folder [in which case they *would* be removed by a subsequent --remove-older-than command], or are they just left where they were [in which case they *wouldn't* be]? I don't know the answer, but if someone does I would be interested. Dominic On 07/01/11 21:31, Chris G wrote: On Fri, Jan 07, 2011 at 02:38:45PM -0500, cov...@ccs.covici.com wrote: When the files are deleted, they are copied to the increments folder and kept till they are removed by --remove-older-than. That makes sense, thank you. Chris Gc...@isbd.net wrote: If you delete files/directories from the 'source' of an rdiff-backup will they get removed from the destination with an appropriate --remove-older-than run? For example if rdiff-backup has been backing up a hierarchy with a directory called 'tmp' for a while and then the 'tmp' directory is removed can one get rdiff-backup to remove the 'tmp' backups 7 days later by --remove-older-than 7D. From the man page it sounds as if deleted files *will* be removed:- Note that snapshots of deleted files are covered by this opera- tion. Thus if you deleted a file two weeks ago, backed up imme- diately afterwards, and then ran rdiff-backup with --remove- older-than 10D today, no trace of that file would remain. Finally, file selection options such as --include and --exclude don't affect --remove-older-than. But this bit from the examples section of the documentation worries me slightly:- Note that an existing file which hasn't changed for a year will still be preserved. But a file which was deleted 15 days ago cannot be restored after this command is run. -- Chris Green ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] external hard drive for backups with multiple file systems
On 12/01/11 00:55, Thomas Evangelidis wrote: I have an external hard drive where I would like to save my Windows files and also create incremental backups for Linux. The problem is that incremental backups cannot be created in the default HPFS/NTFS file system of the hard drive. Is it possible to format the drive to a file system (i.e. VFAT or FAT32) which would be read by Windows but could also save Linux incremental backups? Which file system is that? The program I use for backing up is rdiff-backup. I don't know of any reason why rdiff-backup under Linux could not back up to FAT32, but I have never tried it and personally I would stick to a standard Linux format such as ext4. If there isn't any such file system, in what way could I create 2 partitions just for backing up (no OS will be installed), one of which will be mounted by Windows (i.e. NTFS) and the other by Linux (i.e. ext4) using a partitioning program like GParted? No problem to do this, but make the first partition on the drive NTFS, because this is the only one that will be seen by Windows, and then the second can be for Linux. You can probably even create the first partition under Windows. I would greatly appreciate any advice! If both machines (Windows and Linux) are in the same place and switched on at the same time, then you could leave the external drive attached to the Linux machine (and formatted for Linux), and run rdiff-backup from the Windows machine to the Linux machine, saving the data on the external drive. Using rdiff-backup from Windows to Linux is well tested. Dominic http://www.timedicer.co.uk ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Backup on Windows with exclude-globbing-filelist and spaces in path
I think for exclude-globbing-filelist you don't need that dash prefix.
Here is an extract from mine, which works under Windows:
ignorecase:**/printhood/**
ignorecase:**settings/temp**
ignorecase:**NTUser.dat**
ignorecase:**Deleted Items.dbx**
ignorecase:**Old.dbx**
ignorecase:**/Kiesoft/**
ignorecase:**.ldb**
ignorecase:**Tecom Bluetooth Dongle Application setup guide.pdf**
ignorecase:**/Temp/**
ignorecase:**/Temporary Internet Files/**
ignorecase:**{CE77444D-FC14-4BDA-991A-BDE3A17E85D1}**
ignorecase:**Trash**
So:
- remove the prefixed dash
- you are right to use unescaped forward slashes
- try using ignorecase: as a prefix and see if this helps
- no problem to exclude files or folders with spaces in the names
Dominic
http://www.timedicer.co.uk
[With TimeDicer you provide an excludelist in Windows format and it
*nixifies it 'under the hood' to work with rdiff-backup.]
On 18/01/2011 13:38, ABC DEF wrote:
Hello,
I've been trying to make a backup with rdiff-backup from Windows Vista
to a remote Linux box via ssh.
I'm trying to achieve something like this:
S:\Program Files\rdiff-backup\rdiff-backup.exe
--exclude-globbing-filelist S:\Program
Files\rdiff-backup\my_exclude.txt --remote-schema s:\Program
Files\putty2\plink.exe -i privatekey.ppk %%s rdiff-backup --server
s:\Temp\doc u...@example.com::/home/user/.rdiff-backup
File my_exclude.txt looks like this:
- s:/Temp/doc/test
I've tried other variations, like S:\\/Temp\\/doc\\/testor
S:\Temp\docwhich was suggested in Windows howto, but nothing works and
it throws this:
Fatal Error: Fatal Error: The file specification
's:/Temp/doc/test'
cannot match any files in the base directory
's:\Temp\doc'
So my goal is to exclude test directory from S:\Temp\doc. I am
planning on backuping my Documents after I success with this, so I
would also like to exclude some directories with spaces in their name
(like My Games).
Is it possible to do this with rdiff-backup on Windows?
Thanks and regards,
GroundZero
___
rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Force a regression of rdiff-backup archive
This must be almost a record snail-wise for a newsgroup correspondence (my original posting June 2009, Janne's reply Feb 2010, this reply Jan 2011), but by way of a very belated thank you to Janne for his suggestion, and using his technique, I attach a bash script which automates the process of forcibly regressing an rdiff-backup repository. Help is built in, just run the script without parameters to see. It does a single step regression, it can be run multiple times if you need to regress the repository back further. All the usual caveats apply. Dominic On 23/02/2010 10:39, Janne Peltonen wrote: Hi! I don't know if you solved your problem already some another way, but I thought it'd be nice to tell what I do in situation like this, anyway. Without looking at the source, it appears that rdiff-backup decides whether a regression is in order by checking whether there are two current_mirror stamps (one for the supposedly failed backup, the other for the previous, successful backup). So you can force a regression to the previous version by creating a new current_mirror stamp file in the destdir/rdiff-backup-data directory. (The file contains the word PID, a space, and the pid of the rdiff-backup process that created the version, but the contents aren't that interesting in this case; you can just copy the existing current_mirror file.) The timestamp in the file name of the current_mirror file must match the timestamp of the previous backup, the one you want to regress to (you can only regress to the previous backup, don't try to skip versions). An example might be in order: --clip-- varma:/backup/data/aulis/rdiff-backup-data# ls -ltr mirror_metadata*|tail -n2 -rw--- 1 root root7243 21.2. 02:49 mirror_metadata.2010-02-20T01:00:20+02:00.diff.gz -rw--- 1 root root 5453359 23.2. 12:07 mirror_metadata.2010-02-21T01:00:21+02:00.snapshot.gz varma:/backup/data/aulis/rdiff-backup-data# cp current_mirror.2010-02-21T01:00:21+02:00.data current_mirror.2010-02-20T01:00:20+02:00.data varma:/backup/data/aulis/rdiff-backup-data# rdiff-backup --check-destination-dir /backup/data/aulis --clip-- Depending on your version of rdiff-backup and the exact situation, you might get an error like this: --clip-- Exception 'RPath instance has no attribute 'inc_compressed'' raised of class 'exceptions.AttributeError': File /var/lib/python-support/python2.4/rdiff_backup/Main.py, line 295, in error_check_Main try: Main(arglist) File /var/lib/python-support/python2.4/rdiff_backup/Main.py, line 315, in Main take_action(rps) File /var/lib/python-support/python2.4/rdiff_backup/Main.py, line 273, in take_action elif action == check-destination-dir: CheckDest(rps[0]) File /var/lib/python-support/python2.4/rdiff_backup/Main.py, line 781, in CheckDest dest_rp.conn.regress.Regress(dest_rp) File /var/lib/python-support/python2.4/rdiff_backup/regress.py, line 69, in Regress regress_rbdir(manager) File /var/lib/python-support/python2.4/rdiff_backup/regress.py, line 124, in regress_rbdir if has_meta_diff and not has_meta_snap: recreate_meta(meta_manager) File /var/lib/python-support/python2.4/rdiff_backup/regress.py, line 145, in recreate_meta writer = metadata.MetadataFile(temprp, 'w', check_path = 0) File /var/lib/python-support/python2.4/rdiff_backup/metadata.py, line 378, in __init__ if compress and not rp_base.isinccompressed(): File /var/lib/python-support/python2.4/rdiff_backup/rpath.py, line 1087, in isinccompressed return self.inc_compressed Traceback (most recent call last): File /usr/bin/rdiff-backup, line 23, in ? rdiff_backup.Main.error_check_Main(sys.argv[1:]) File /var/lib/python-support/python2.4/rdiff_backup/Main.py, line 295, in error_check_Main try: Main(arglist) File /var/lib/python-support/python2.4/rdiff_backup/Main.py, line 315, in Main take_action(rps) File /var/lib/python-support/python2.4/rdiff_backup/Main.py, line 273, in take_action elif action == check-destination-dir: CheckDest(rps[0]) File /var/lib/python-support/python2.4/rdiff_backup/Main.py, line 781, in CheckDest dest_rp.conn.regress.Regress(dest_rp) File /var/lib/python-support/python2.4/rdiff_backup/regress.py, line 69, in Regress regress_rbdir(manager) File /var/lib/python-support/python2.4/rdiff_backup/regress.py, line 124, in regress_rbdir if has_meta_diff and not has_meta_snap: recreate_meta(meta_manager) File /var/lib/python-support/python2.4/rdiff_backup/regress.py, line 145, in recreate_meta writer = metadata.MetadataFile(temprp, 'w', check_path = 0) File /var/lib/python-support/python2.4/rdiff_backup/metadata.py, line 378, in __init__ if compress and not rp_base.isinccompressed(): File /var/lib/python-support/python2.4/rdiff_backup/rpath.py, line 1087, in isinccompressed return self.inc_compressed AttributeError: RPath instance has no attribute
Re: [rdiff-backup-users] Slowdown from a few mins to 8 hours backup time due to one large log file
That seems extraordinarily slow for rdiff-backup to backup a text file
to a local disk, even a file that is 3.9GB. Was the file changing
(additional log entries, maybe?) while rdiff-backup was trying to back
it up?
Dominic
On 01/02/2011 06:33, Patrick Nagel wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Hi,
I have rdiff-backup running nightly on my home server. It keeps a versioned
backup of all relevant data on the second HDD.
I didn't monitor the backup too closely, because I didn't get any alarm that
the backup had failed. Today I looked at the logs, and noticed, that the
backup time went up from a few minutes a while back, to8 hours.
2010-11-23T03:00: ElapsedTime 277.51 (4 minutes 37.51 seconds)
2010-11-24T03:00: ElapsedTime 438.69 (7 minutes 18.69 seconds)
2010-11-25T03:00: ElapsedTime 321.99 (5 minutes 21.99 seconds)
2010-11-26T03:00: ElapsedTime 2848.57 (47 minutes 28.57 seconds)
2010-11-27T03:00: ElapsedTime 3358.68 (55 minutes 58.68 seconds)
2010-11-28T03:00: ElapsedTime 2876.65 (47 minutes 56.65 seconds)
2010-11-29T03:00: ElapsedTime 2834.06 (47 minutes 14.06 seconds)
2010-11-30T03:00: ElapsedTime 3553.56 (59 minutes 13.56 seconds)
2010-12-01T03:00: ElapsedTime 3549.67 (59 minutes 9.67 seconds)
2010-12-02T03:00: ElapsedTime 3555.71 (59 minutes 15.71 seconds)
[...]
2010-12-10T03:00: ElapsedTime 8015.67 (2 hours 13 minutes 35.67 seconds)
2010-12-11T03:00: ElapsedTime 9066.51 (2 hours 31 minutes 6.51 seconds)
[...]
2011-01-30T03:00: ElapsedTime 28665.77 (7 hours 57 minutes 45.77 seconds)
2011-01-31T03:00: ElapsedTime 31093.79 (8 hours 38 minutes 13.79 seconds)
2011-02-01T03:00: ElapsedTime 31489.30 (8 hours 44 minutes 49.30 seconds)
By tracking what rdiff-backup was doing ('strace -prdiff-backup's PID' on
another terminal), I quickly found out that it spent huge amounts of time
diffing one big file. The file was a log file, written to by a script I
wrote just around the time the backup time started to increase. I guess I
overdid it with the logging a bit, since the log file had 3.9 GB by now.
After removing the oversized log file, the backup times dropped back to
normal again:
2011-02-01T12:22: ElapsedTime 685.87 (11 minutes 25.87 seconds)
2011-02-01T13:57: ElapsedTime 468.25 (7 minutes 48.25 seconds)
Maybe the diff-algorithm used can be improved to not take so long on large
text files, especially since this one only grew into one direction, and the
other content never changed.
In any case, I'm glad the backup doesn't take the whole night any more.
Patrick.
___
rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Slowdown from a few mins to 8 hours backup time due to one large log file
Yes, backing up from LVM snapshots is the best way (or, for Windows, use VSS), but at least you found a workaround... Dominic http://www.timedicer.co.uk On 01/02/2011 09:42, Patrick Nagel wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Dominic, On 2011-02-01 17:14, Dominic Raferd wrote: That seems extraordinarily slow for rdiff-backup to backup a text file to a local disk, even a file that is 3.9GB. Was the file changing (additional log entries, maybe?) while rdiff-backup was trying to back it up? Yes, I think so, and I don't use LVM snapshots on this server. That's probably the real trigger then. Patrick. ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Removing intermediate snapshot
I'm afraid that the short answer is: no In most situations there should be no need to remove an intermediate backup because there would be little space saving. However in some situations it would be very helpful - say if you backed up the wrong stuff into an existing repository on one occasion (and then subsequently corrected it). Still, there is no easy way to fix this. What can be done is to regress a repository day-by-day to get back beyond the 'bad' backup and then start anew from there. If the backup(s) you wish to remove are reasonably recent, and you don't mind losing any changes since then, this might be appropriate, and I have a script which can help (which I posted here a few weeks ago). Dominic On 04/02/2011 12:00, Alex Schuster wrote: Hi there! I'm a happy rdiff-backup user, this utility is really excellent. Great work! But some backup partitions are getting full. Is there a possibility to remove not only the oldest backups(s), but some backups in between? For my /home directory, the last backup is most important of course, but I would also like to be able to restore the very first backup I did make. All other backups in between are not that important, and I would like to remove some. But rdiff-backup only has a --remove-older-thandate option, not something like --remove-betweendate1 date2. Is there some workaround, or do I want the impossible? I'm expecially concerned about one backup that has very much data in it that was only temporary and is no longer needed. I read I could remove those specific files if I am careful, but I still think there is an important feature missing. Wonko ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] ANN: rdiff-backup-fs 1.0.0
excellent! I look forward to trying it... Dominic On 08/02/2011 11:22, Filip Gruszczyński wrote: I'd like to announce first stable release of rdiff-backup-fs filesystem (formerly known as archfs). rdiff-backup-fs is a filesystem that provides easy access and browsing of rdiff-backup archives. New version of filesystem implements on demand revision data retrieval and caching, so large rdiff-backup archives can be accessed (which was not possible with archfs). Home page: http://code.google.com/p/rdiff-backup-fs/ ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Why using --force while restoring is dangerous?
My reading is that using --force on a restore will overwrite existing files with the same name - so you may lose previous data at the restore destination. In general if you are restoring a directory (or a complete repository) it is logical to use a clean destination, in which case it shouldn't be a problem. Anyone know different? Dominic On 09/02/2011 12:40, Filip Gruszczyński wrote: I have read in manual, that --force can be dangerous, when restoring files from backup. I am using --force, when calling rdiff-backup in my filesystem to restore a file. I can't yet remove it, because otherwise it fails to restore file. Is it really dangerous or maybe I can keep it? ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Re: Why using --force while restoring is dangerous?
On 09/02/2011 14:48, Robert Nichols wrote: On 02/09/2011 07:32 AM, Dominic Raferd wrote: My reading is that using --force on a restore will overwrite existing files with the same name - so you may lose previous data at the restore destination. In general if you are restoring a directory (or a complete repository) it is logical to use a clean destination, in which case it shouldn't be a problem. Anyone know different? Dominic On 09/02/2011 12:40, Filip Gruszczyński wrote: I have read in manual, that --force can be dangerous, when restoring files from backup. I am using --force, when calling rdiff-backup in my filesystem to restore a file. I can't yet remove it, because otherwise it fails to restore file. Is it really dangerous or maybe I can keep it? When you are restoring a directory, --force will not only overwrite existing files (which is probably what you intended, anyway), but it will also _delete_ any files or even entire subdirectories that were not present in the backup. It will restore your directory to exactly the state it was on the backup, nothing more, nothing less. That might be a nasty surprise. Thanks for the info. That is logical, but I wasn't aware of it. Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] rdiff-backup streaming?
I used rdiff.exe briefly before moving to rdiff-backup. Certainly rdiff-backup does your 2-4 'under the hood' and fast. It is highly optimised both for speed of transfer and for storage space - but does require a reliable connection between source and destination. It does not do your 1, you can achieve this best by backing up from an LVM snapshot (if your source machine is Linux) or VSS (if it is Windows). I am pretty sure that rdiff-backup does not use rdiff, instead it uses a python module built from the librsync library (which is also called, I think, by rdiff). Dominic http://www.timedicer.co.uk/ On 17/02/2011 17:41, Mark Price wrote: Question - We use rdiff (not rdiff-backup) to do our incremental file backups. We do: 1. Copy the file to a staging area (so the file won't disappear or be modified while we work on it) 2. Hash the original file, and computes an rdiff signature (used for delta differencing) 3. Comput an rdiff delta difference (if we have no prior version, this step is skipped) 4. Compress encrypt the resulting delta difference Our problem is all of these things happen in separate phases, distinctly one from the other. This means it takes a long time to do its job. What I am wondering is if rdiff-backup does all of these things in one read/write file pass in a streaming manner, or if it simply calls to the standard rdiff.exe (which isn't working for us)? I didn't quite see information concerning this in the docs or wiki. We have considered using xdelta (because it operates in a streaming manner) but the problem with xdelta is that it stores double copies of the deltas and kills storage space. Any help on this matter would be great! ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] rdiff-backup --verify 1.2.8 running around in circles
Marc: I can't help with the python. But first step, unless the most recent backup is critical to you, is to try regressing the repository to get it back to a clean state, with the --check-destination-dir switch. If this refuses to regress, claiming that there is no corruption, I have a bash script which can 'force' a regression. You can repeat this as many times as necessary to get past (hopefully) the original corruption. Secondly, rdiff-backup --verify switch gives only a partial verification of the repository. Daniel Miller wrote a patch, available for rdiff-backup 1.2.8, which adds --verify-full and --verify-full-at-time switches which can perform a full verification of a repository. I use this patched version in TimeDicer Server for this purpose only in preference to the standard rdiff-backup program. Dominic Http://www.timedicer.co.uk On 20/02/11 12:23, Marc Haber wrote: Hi, I have an rdiff-backup directory that was corrupted by a file system crash. As I don't want to lose the backup, I tried rdiff-backup -v9 --verify on the repository. But already after a few seconds of running, output stops at: Sun Feb 20 08:28:51 2011 Verified SHA1 digest of etc/kde4/kdm/backgroundrc Sun Feb 20 08:28:51 2011 Verified SHA1 digest of etc/kde4/kdm/kdm.options Sun Feb 20 08:28:51 2011 Verified SHA1 digest of etc/kde4/kdm/kdmrc Sun Feb 20 08:28:51 2011 Verified SHA1 digest of etc/kde4/kdm/kdmrc.dpkg-old Sun Feb 20 08:28:51 2011 Verified SHA1 digest of etc/kde4/khotnewstuff.knsrc Sun Feb 20 08:28:51 2011 Verified SHA1 digest of etc/kde4/kshorturifilterrc Sun Feb 20 08:28:51 2011 Verified SHA1 digest of etc/kernel/postinst.d/initrSun Feb 20 08:28:51 2011 Warning: Could not restore file etc/kernel/postinst.d/initramfs-tools! A regular file was indicated by the metadata, but could not be constructed from existing increments because last increment had type None. Instead of the actual file's data, an empty length file will be created. This error is probably caused by data loss in the rdiff-backup destination directory, or a bug in rdiff-backup Sun Feb 20 08:28:51 2011 Warning: Computed SHA1 digest of etc/kernel/postinst.d/initramfs-tools da39a3ee5e6b4b0d3255bfef95601890afd80709 doesn't match recorded digest of e69689ccfb34f74c4ba64d1f1e842df083020b31 Your backup repository may be corrupted! and rdiff-backup starts taking 100 % CPU and stopped accessing the disk. This was not the first case of an increment of type None being found and flagged as such. From a cursory look at an strace output, rdiff-backup seems to be running around in the following loop... ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Re: rdiff-backup --verify 1.2.8 running around in circles
On 21/02/11 00:22, Robert Nichols wrote: On 02/20/2011 02:51 PM, Marc Haber wrote: On Sun, Feb 20, 2011 at 06:11:19PM +, Dominic Raferd wrote: Secondly, rdiff-backup --verify switch gives only a partial verification of the repository. Daniel Miller wrote a patch, available for rdiff-backup 1.2.8, which adds --verify-full and --verify-full-at-time switches which can perform a full verification of a repository. Debian's version does not seem to be patched. What does --verify omit? --verify only checks the latest version of data in the repository, and because rdiff-backup works with reverse diffs this is far from confirming that the entire repository is okay. --verify-at-time allows you to check earlier date, but even checking a very early date (e.g. before the first backup) does not guarantee integrity of subsequent backups e.g. for files that did not exist at the earlier time. Without the patch, the only safe way is to run --verify-at-time for every time a backup set has been updated in the repository, but this is often impractical. If you read the last message from Dan Miller in that New feature: --verify-full thread you see why versions including that patch have not been distributed. http://lists.nongnu.org/archive/html/rdiff-backup-users/2010-02/msg00054.html Yes. This version cannot for instance create new repositories. Daniel had backported it from his original patch (for rdiff-backup in CSV), and there are some glitches. So I only use it for the --verify-full and --verify-full-at-time options - I build but don't install it. For all other purposes I use standard rdiff-backup 1.2.8. And, it wouldn't help for your case anyway. The patch file contains the following: +WARNING this will not detect if your repository has already been corrupted. +It will create integrity signatures with the files as they exist when you +run the script. Any corruption that happened before that point can only +be detected with --verify-at-time. However, after you run this script you +can use --verify-full to verify that nothing has changed since you ran this +script. I think this warning refers to the integrify.py module not to the patch as a whole. The comment for the whole patch is: +Added --verify-full, which verifies the integrity of an entire repository +including all increments and metadata. This should be much faster than +--verify-at-time=date in the past, and is more comprehensive in terms of +what is verified. The --verify and --verify-at-time now options are OK for checking the current mirror (aside from problems with files having multiple hard links, which rdiff-backup stumbles with in other places too). The big problem is not having a way to verify the entire archive. The suggestion on the wiki site to pick a date that is at least as old as the oldest rdiff session is very wrong. Only the content that existed on that oldest date will be checked. Many forms of corruption for subsequent dates will go unnoticed. ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] rdiff-backup --verify 1.2.8 running around in circles
On 20/02/2011 20:51, Marc Haber wrote: On Sun, Feb 20, 2011 at 06:11:19PM +, Dominic Raferd wrote: first step, unless the most recent backup is critical to you, is to try regressing the repository to get it back to a clean state, with the --check-destination-dir switch Fatal Error: Destination dir scyw00225 does not need checking If this refuses to regress, claiming that there is no corruption, I have a bash script which can 'force' a regression. You can repeat this as many times as necessary to get past (hopefully) the original corruption. So it'll be a force regression, try --verify, until --verify runs through, losing one full, probably uncorrupted increment per iteration? Exactly. Best to take a backup of the repository before trying it. Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Backup not identifying changed file, even though hash is different
On 24/03/2011 00:07, Patrick Nagel wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, On 2011-03-24 06:29, Scott Jilek wrote: Is there any way to force Rdiff to use a hash compare for difference triggering? I looked over the manual, and nothing seems to be what I'm looking for. I've tried adding the --compare-hash switch. The MAN page isn't specific about the function of --compare-hash, however it appears to be only for VERIFICATION and not for candidate IDENTIFICATION, as causes rdiff to simply show a report telling me this file is not in sync. When I use that switch no backups occur from what I can tell as no increments are created. Is there some way to explicitly force Rdiff to create a backup of specific files? What am I missing, because this seems like a huge gap in functionality if it's only using file timestamps. I guess it would be nice to have an option to make rdiff-backup identify changed files by checksum/hash. That being said, I think I wouldn't want to use it for anything, because it would be extremely slow. It would need to read all files from start to end, while hashing them, at each backup run. For a large backup repository with, say, 500 GB worth of files, the backup would take more than an hour, even if not a single file changed (assuming 100 MB/s read and hashing speed). By the way, rsync also wouldn't consider that truecrypt container as changed, by default - you would need to add the '-c' option. And maybe that's the best way to handle this: Exclude the truecrypt container(s) from your rdiff-backup, and instead use rsync -c for that. Alternatively you could 'touch' the truecrypt container just before the backup, so that the modification timestamp gets changed, and rdiff-backup will pick it up (but then it would create a diff every time, which would take even longer than the 'rsync -c' method - but maybe you really want the history of that file...). Patrick. I would echo the suggestion to touch the file. You can download a free touch.exe program from the net. I use it with -m option in a similar case to force rdiff-backup (via TimeDicer) to pick up a vmdk file. Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Backup not identifying changed file, even though hash is different
On 24/03/2011 17:31, Scott Jilek wrote: Unfortunately, I can't just touch the file because it's in use and locked by the truecrypt process. Are you sure? I tried this and it worked for me with a truecrypt file in use. Is there any plan to add the rsync -c option to rdiff-backup to force always checksum for files like this. Obviously it wouldn't be used in most cases, but for known special files like this where timestamps aren't used, it really is the only option. rdiff-backup development is stalled, sorry I could try to install one of the windows rsync variants in this case, but the whole idea of a practical backup process to me is getting multiple snapshots in time, whereas rsync only gives me efficient mirroring. I'm not too keen on mirroring as a backup strategy, because corrupted files completely destroy the usefulness of mirroring. If it's bad in the source, it becomes bad in the singular backup as soon as the backup process runs and then you have no recovery option. With Rdiff, I can go back in time to just before the corruption occurred and restore the file to the last know working time. As far as I'm concerned, simple mirroring is not a safe method of backup I agree, rdiff-backup is practically perfect. But when you hit a limitation you will have to workaround. Dominic http://www.timedicer.co.uk ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Problem with rdiff-backup
The problem appears to be with some non-ASCII characters. What filesystems are you backing up to/from? Try with the switches --null-separator and/or --override-chars-to-quote, or if you are already using them, try without! Dominic http://www.timedicer.co.uk On 03/04/2011 16:28, Benjamin wrote: Hello everyone! Since this weekend (and i also remember that problem to have occured earlier) i have a problem with rdiff-backup. When I try to backup my files (with the same command a always used) i get the following error (see below). At first a few notes on my behavior: I normally backup all my files with the command see below. Because that did not work with the following response: [root@Banjo Benji]# rdiff-backup '/home/Benji' '/media/Banjos_Wissen/Backup-Banjo' Previous backup seems to have failed, regressing destination now. ^[[D^[[DSpecialFileError .gdesklets/sockets/%3A0.0 Socket error: AF_UNIX path too long ListError .gtk-bookmarks/.gvfs [Errno 13] Permission denied: '/home/Benji/.gvfs' UpdateError Downloads/rdiff-backup.tmp.64562 [Errno 84] Invalid or incomplete multibyte or wide character: '/media/Banjos_Wissen/Backup-Banjo/rdiff-backup-data/increments/Downloads/330 - Der Kampf der Strohh\x81tte.avi.2011-02-18T00:21:06+01:00.missing' ... I then tried the following 15. from: http://www.nongnu.org/rdiff-backup/FAQ.html#regress_failure But also that did not help. I then tried to make a whole new backup of my files with the following response: [root@Banjo Desktop]# rdiff-backup '/home/Benji' '/media/Banjos_Wissen/Banjos_Backup' ListError .gtk-bookmarks/.gvfs [Errno 13] Permission denied: '/home/Benji/.gvfs' OSError while renaming /media/Banjos_Wissen/Banjos_Backup/Downloads/rdiff-backup.tmp.40180 to /media/Banjos_Wissen/Banjos_Backup/Downloads/330 - Der Kampf der Strohh�tte.avi UpdateError Downloads/rdiff-backup.tmp.40180 [Errno 84] Invalid or incomplete multibyte or wide character ... That is the present problem, but I really like the simplicity of rdiff-backup and don´t wanna change so please help me. Thx. to you - Benjamin D. Arch: Fedora 14 Kernel Linux 2.6.35.11-83.fc14.i686 ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] New GUI for rdiff-backup: JBackpack
I can run rdiff-backup 1.2.8 fine on Windows 7 64-bit. Edited from http://www.nongnu.org/rdiff-backup/: On Windows, rdiff-backup requires the Visual C++ 2008 redistributable. Download DLL package http://download.savannah.gnu.org/releases/rdiff-backup/Microsoft.VC90.zip and copy the four files into the same directory as rdiff-backup.exe. Regards Dominic On 03/04/2011 21:43, Ronny Standtke wrote: Hi Sorry too for the late response. Now it's my turn again to repeat the apology. :-) It took a while, but I finally got my hands on a Win 7 64 bit system. The strange thing is that I even fail installing rdiff-backup manually. When I start rdiff-backup on the command line I only get the following error message: - C:\Users\Ronnyrdiff-backup Traceback (most recent call last): File C:\Python26\lib\site-packages\py2exe\boot_common.py, line 92, in module ImportError: No module named linecache Traceback (most recent call last): File install zipextimporter, line 1, inmodule ImportError: No module named zipextimporter Traceback (most recent call last): File rdiff-backup, line 20, inmodule ImportError: No module named rdiff_backup.Main - How did you successfully install and start rdiff-backup on a Win 7 64 bit system!? Regards Ronny ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] IOError: [Errno 75] Value too large for defined data type
Bruno: Do you need to use NTFS for the external drive? I suggest you try again using a Linux format such as ext3 and see if you have more success. My impression is that a high proportion of problems with rdiff-backup relate to backups to NTFS volumes (backup *from* NTFS generally works fine). Dominic On 10/05/11 16:21, Bruno Basilio wrote: Hi, I'm new at using rdiff-backup but already fascinated with the simple principle. Hope it will not stop to be maintained. I'm running Ubuntu 10.04, installed the rdiff-backup 1.2.8 and tried to backup 80GB of pictures and movies using to USB external drives with NTFS, at around 30GB, it stopped with following error: $ rdiff-backup -v9 --print-statistics --no-hard-links /media/BBasilioHomeBackup/fotos/ /media/Elements/rdiff-backup/fotos/ Mon May 9 22:23:34 2011 Using rdiff-backup version 1.2.8 Mon May 9 22:23:34 2011 Found interrupted initial backup. Removing... Mon May 9 22:23:34 2011 Determining if directory contains files: /media/Elements/rdiff-backup/fotos/rdiff-backup-data/increments Mon May 9 22:23:34 2011 Determining if directory contains files: /media/Elements/rdiff-backup/fotos/rdiff-backup-data/increments/2009 Mon May 9 22:23:34 2011 Determining if directory contains files: /media/Elements/rdiff-backup/fotos/rdiff-backup-data/increments/2009/2009_08_08 Mon May 9 22:23:34 2011 Determining if directory contains files: /media/Elements/rdiff-backup/fotos/rdiff-backup-data/increments/2009/2009_08_08/2009_08_08 - Bruno Mon May 9 22:23:34 2011 Deleting /media/Elements/rdiff-backup/fotos/rdiff-backup-data/increments Mon May 9 22:23:34 2011 Removing directory /media/Elements/rdiff-backup/fotos/rdiff-backup-data/increments Mon May 9 22:23:34 2011 Deleting /media/Elements/rdiff-backup/fotos/rdiff-backup-data/backup.log Mon May 9 22:23:34 2011 Deleting /media/Elements/rdiff-backup/fotos/rdiff-backup-data/chars_to_quote Mon May 9 22:23:34 2011 Deleting /media/Elements/rdiff-backup/fotos/rdiff-backup-data/file_statistics.2011-05-09T22:20:28+01:00.data.gz Mon May 9 22:23:34 2011 Deleting /media/Elements/rdiff-backup/fotos/rdiff-backup-data/mirror_metadata.2011-05-09T22:20:28+01:00.snapshot.gz Mon May 9 22:23:34 2011 POSIX ACLs not supported by filesystem at /media/BBasilioHomeBackup/fotos Mon May 9 22:23:34 2011 Unable to import win32security module. Windows ACLs not supported by filesystem at /media/BBasilioHomeBackup/fotos Mon May 9 22:23:34 2011 escape_dos_devices not required by filesystem at /media/BBasilioHomeBackup/fotos Mon May 9 22:23:34 2011 - Detected abilities for source (read only) file system: Access control lists Off Extended attributes On Windows access control lists Off Case sensitivity On Escape DOS devices Off Escape trailing spaces Off Mac OS X style resource forksOff Mac OS X Finder information Off - Mon May 9 22:23:34 2011 Making directory /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0 Mon May 9 22:23:34 2011 Touching /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/5-_ a.snapshot.gz Mon May 9 22:23:34 2011 Deleting /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/5-_ a.snapshot.gz Mon May 9 22:23:34 2011 Touching /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/uniᄉ Mon May 9 22:23:34 2011 Deleting /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/uniᄉ Mon May 9 22:23:34 2011 Touching /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/:\ Mon May 9 22:23:34 2011 Deleting /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/:\ Mon May 9 22:23:34 2011 Touching /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/A Mon May 9 22:23:34 2011 Deleting /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/A Mon May 9 22:23:34 2011 Touching /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/foo Mon May 9 22:23:34 2011 Deleting /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/foo Mon May 9 22:23:34 2011 Making directory /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/hl Mon May 9 22:23:34 2011 Touching /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/hardlinked_file1 Mon May 9 22:23:34 2011 Hard linking /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/hl/hardlinked_file2 to /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/hardlinked_file1 Mon May 9 22:23:34 2011 POSIX ACLs not supported by filesystem at
Re: [rdiff-backup-users] Q) Is rdiff backup being maintained ?
Alex, I believe Daniel Miller is working on a new project inspired by rdiff-backup, I think he will post here when it is ready for others to try. I don't think its archives will be compatible with rdiff-backup. For most of us rdiff-backups works and works very well indeed. Users like myself would really appreciate some generous volunteer (who understands the code, unlike me!) creating and then maintaining a fork (which in due course could become rdiff-backup2?), so that ongoing bugs can be addressed. I'm not sure whether it would be best to start from 1.2.8 (stable), 1.3.3 (unstable, but I haven't heard of many problems) or CVS. Dominic http://www.timedicer.co.uk/ On 28/04/2011 01:33, Alexander Samad wrote: Hi Wondering if the application is being maintained - bug fixes etc Alex ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] [Errno 13] Permission denied: '/home/username/.gvfs'
Andreas: I have never used the --include-globbing-filelist, only the --exclude-globbing-filelist, but here are some suggestions: Try: run the command with sudo [for cron: put in /etc/crontab] Or: add '- /home/username/.gvfs' at the *top* of your filelist (if you haven't already tried it there) Or: put a symlink inside /etc/ to point to /home/username/.ssh/config and then just backup /etc? Or: do 2 runs of rdiff-backup, one for /etc and one for /home/username/.ssh/config What has fuse got to do with it? Fuse is not normally used by rdiff-backup. However I do successfully use rdiff-backup to backup a remote /home directory mounted using sshfs, so it can work. Dominic http://www.timedicer.co.uk/ On 14/05/11 23:45, Andreas Herrmann wrote: Hello everyone, I'm trying to create a backup of my system configuration using rdiff-backup. The idea is to have a backup of `/etc' and a few dotfiles in `$HOME' just in case I screw something up. Ideally this backup should be performed by a cron job. rdiff-backup is called as rdiff-backup --include-globbing-filelist file_list \ / $HOME/.backup/local where `file_list' looks like - /**~ - /.*.swp - /**/.*.swp /etc /home/username/.ssh/config - / The problem: As soon as I put any file inside home into `file_list' the following error message appears and the backup fails. [Errno 13] Permission denied: '/home/username/.gvfs' Adding `- /home/username/.gvfs' to the file list doesn't help. Googling around revealed, that this is a known problem and has to do with fuse [1,2]. [1] even says this issue would be solved in the cvs version. I've tried 1.2.8, 1.3.3 and the latest cvs version. All produce the same error message. The exact error message is attached to the mail. I had a look at the code myself, but it's not clear to me where to handle this case. Non readable files should produce an error if they are listed as source or destination. But, I do _not_ want to backup `.gvfs' so it should _not_ produce an error... A quick hack around this is given in [2]. But, it's definitely not an option for a cron job. Best, Andreas [1] http://www.backupcentral.com/phpBB2/two-way-mirrors-of-external-mailing-lists-3/rdiff-backup-23/rdiff-backup-trying-to-access-gvfs-even-though-it-is-exc-93726/ [2] http://ubuntuforums.org/showthread.php?t=783935 ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] [Errno 13] Permission denied: '/home/username/.gvfs'
There is a fuse-fs for browsing rdiff-backup repositories [1]. Passing options to fuse is going to be enabled in the next release. With allow_user set I should be able to mount my config backup as root, but browse it as a user. There's also a nice way to automatically exclude filesystems that do not correspond to block devices listed in /dev. E.g. proc, tmpfs, but also fuse. Haven't tried it yet, but this should work: rdiff-backup --exclude-filelist(grep -v '\(^/dev/\|^rootfs\)' /proc/mounts | cut -d \ -f 2 | uniq) \ other options Best, Andreas [1] http://code.google.com/p/rdiff-backup-fs/ Yes I am aware of rdiff-backup-fs (but others may not be, so thanks for the reminder). It is the successor to archfs. I am waiting for it get beyond 1.0.0 too. I use rdiffweb (0.6.3) which works well. Thanks for the tip re excluding filesystems. Dominic http://www.timedicer.co.uk ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Intermittent connection failures
On 04/07/2011 14:48, Maarten Bezemer wrote: ...My money would be on the network stability. Maybe rdiff-backup could somehow be made more robust, but I doubt the current design can be patched to support reconnection attempts when disconnected unexpectedly... I agree. IMO rdiff-backup should never be run over an internet connection, only on a stable local network. If you want to get the rdiff-backup data offsite, create your rdiff-backup repository locally and then use rsync to mirror it to the offsite location. This is the technique used for TimeDicer primary/mirror servers. Dominic http://www.timedicer.co.uk ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Unable to restore interrupted backup
On 08/07/2011 13:30, abschiedsstein wrote:
Hi,
I used rdiff-backup for a while and everything was fine. Backups were fast and
restoring was no problem.
Then, after an interrupted backup, the data was broken. Neither restoring nor
fixing with --check-destination-dir was possible.
I guess there is somekind of a standard routine to fix this, but I was not able
to figure it out until now.
Before I get into details, here are some information about my machine.
OS: Ubuntu 10.10
rdiff-backup: rdiff-backup 1.2.8
The backup directory is on an external usb drive.(NTFS)
When I tried to restore a file I got the message that the last backup was interrupted and that I
should run --check-destination-dir (so far, so good). I did this. There were many messeges like
... metadata, but could not be constructed from existing increments because last increment
had type ... . When I now run --check-destination-dir again, I get : Fatal Error:
Destination dir /media/TREKSTOR/Sicherungen/neodata-backup/home does not need checking. BUT
restoring is still not possible:
rdiff-backup -r now /media/TREKSTOR/backupdir/Desktop/protokoll.pdf
Desktop/proto.pdf
Warning: Could not restore file .!
A regular file was indicated by the metadata, but could not be
constructed from existing increments because last increment had type
dir. Instead of the actual file's data, an empty length file will be
created. This error is probably caused by data loss in the
rdiff-backup destination directory, or a bug in rdiff-backup
I would like to know a way to fix the backup. It would be completely sufficient
if everything since the last successful backup would be deleted.
I hope someone can help me. Thanks in advance.
I attach a bash script which automates the process of forcibly regressing an
rdiff-backup repository. You can use this to override the 'does not need
checking' message.
Help is built in, just run the script without parameters to see. It does
a single step regression, it can be run multiple times if you need to
regress the repository back further. All the usual caveats apply.
Dominic
#!/bin/bash
THIS=`basename $0`
COLUMNS=$(stty size 2/dev/null||echo 80); COLUMNS=${COLUMNS##* }
while getopts :qfh optname; do
case $optname in
f)FORCE=y;;
q)QUIET=y; RDIFFBACKUPOPTIONS=--terminal-verbosity 0;;
h)HELP=y;;
?)echo Unknown option $OPTARG; exit 1;;
:)echo No argument value for option $OPTARG; exit 1;;
*) # Should not occur
echo Unknown error while processing options; exit 1;;
esac
done
shift $(($OPTIND-1))
if [ -z $QUIET -o -n $HELP -o -z $1 ]; then
echo -en \n$THIS v0.1216 by Dominic domi...@timedicer.co.uk
[ -z $HELP -a -n $1 ] echo -n (-h for help)
echo -e \n${THIS//?/=}\n
fi
if [ -n $HELP -o -z $1 ]; then
echo -e Forces regression of an rdiff-backup archive even if natural
regression does not occur. Can be useful if a repository is corrupted and
regression is neither automatic nor can be initiated with
--check-destination-dir.\n\nUsage: \t$THIS [options] archive-path\nExample:
\t$THIS -f /home/fred/backup\nOptions:\tf - Force, proceed with no
prompt\n\t\th - Help, this help text\n\t\tq - Quiet, no output unless
error\n|fold -s -w $COLUMNS
exit
fi
if [ ! -d $1 ]; then
echo Cannot find directory \$1\, aborting...
exit 1
fi
ARCHIVE=$1
if [ ! -d $ARCHIVE/rdiff-backup-data ]; then
echo $ARCHIVE does not appear to be a valid rdiff-backup repository,
aborting...
exit 1
fi
[ -z $QUIET ] echo Using repository: $ARCHIVE
WHENLAST=`ls $ARCHIVE/rdiff-backup-data/current_mirror*|sed
's/.*current_mirror\.\([^.]*\).*/\1/'`
NUMCURRENT=`echo $WHENLAST|awk '{print NF}'`
if [ $NUMCURRENT -ne 2 ]; then
if [ $NUMCURRENT -ne 1 ]; then
echo $NUMCURRENT current_mirror files, aborting...
exit 1
else
[ -z $QUIET ] echo rdiff-backup does not recognise this
repository as damaged
fi
else
[ -z $QUIET ] echo rdiff-backup recognises this repository as
damaged
fi
PREVRUN=`ls $ARCHIVE/rdiff-backup-data/mirror_metadata*|tail -n2|head -n1|sed
's/.*mirror_metadata\.\([^.]*\).*/\1/'`
if [ -z $FORCE ]; then
read -n 1 -t 30 -p About to regress $ARCHIVE repository from $WHENLAST
to $PREVRUN: ok (y/-)?
[ $REPLY != y ] echo - aborting... exit 1
fi
[ -z $QUIET ] echo -e \nStarted `date`
if [ $NUMCURRENT -eq 1 ]; then
cp -a $ARCHIVE/rdiff-backup-data/current_mirror.$WHENLAST.data
$ARCHIVE/rdiff-backup-data/current_mirror.$PREVRUN.data
[ -z $QUIET ] echo Copied current_mirror
fi
[ -z $QUIET ] echo Regressing, this may take a long time, please be
patient...
rdiff-backup $RDIFFBACKUPOPTIONS --check-destination-dir $ARCHIVE
[ -z $QUIET ] echo Ended `date`
[ -z $QUIET ] [ $? -eq 0 ] echo Regression completed successfully,
current mirror is now $PREVRUN || echo There was an
Re: [rdiff-backup-users] Post-setup questions
On 18/08/11 22:47, Grant wrote: And, looking at the whole subject from a different angle: pushing also has the large drawback that in case your laptop is stolen/lost/whatever, and you use an ssh key for rdiff-backup to connect to your backup server, you risk not only losing your 'real' systems, but the backup server can also be compromised it an attacker starts using that key. If I were to set up a separate user for each pushed backup and one of the systems were compromised, only the compromised system's backups would be accessible to the attacker (read/write unfortunately). Practically this seems to me a pretty good solution. Push your backup from your laptop to a dedicated user on the (GNU/Linux) backup server, ensuring that its users cannot access (read or write) other users' data: To ensure privacy for any new users on the backup server, in /etc/adduser.conf add/alter: DIR_MODE=0700 For existing users on the backup server, run: sudo chmod 700 /home/* When you create a user on the backup server, don't set a password i.e. login is only by public/private key authentication. Use a dedicated private key on the laptop for the backup, and put its public key on the backup server at (and only at) /home/[user]/.ssh/authorized_keys. You need to be happy that there is nothing readable (or writeable) by a non-privileged user on the backup server that you would not want a hacker to see e.g. at /var/ or /opt/. For tighter security on the backup server you can limit the user to running a single command e.g. http://www.cmdln.org/2008/02/11/restricting-ssh-commands/. If the laptop is stolen, remove the corresponding public key from the backup server so that the laptop can no longer get access. And even if the thief moved fast enough to get into the backup server before you could remove the public key, she could only access the backup history for that laptop, and she has the laptop's data already... Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] [from] Re: Memory usage during regressions
On 28/10/2011 11:03, john wrote: Hi The same problem as described here strucked me, too: When regressing a backup, rdiff-backup starts instantly consuming memory at quite a high rate, until it reaches 3GB. Then it dies with a memory allocation error, because it is running on a 32bit system. As this means that I cannot make any new backups now, without starting from scratch, I searched quite a while for a solution to this. But only found this mailing list thread. Does this mean it is a rare error condition? The system to backup is not that big: 2,2 Million files, 400GB. Any hints for debugging? I tested rdiff-backup V1.2.8-r1 and V1.3.3. cu John How much space is there in your /tmp folder? Try specifying a different tmp folder with --local-temp or --remote-temp (i.e. one with massive storage available). Or try increasing the size of your swap location? (rdiff-backup is supposed to use temp folders as specified here under 'tempdir' but I think it doesn't follow this logic, or at least not reliably. However forcing a specific temp location with --local-temp or --remote-temp does seem to work.) Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Prevent rdiff-backup from deleting?
I think (but haven't tried) you could alter the rdiff-backup option text like this (this is under Ubuntu 10.04, the location might differ with another OS): sed -i 's/remove-older-than/remove-older-thax/g' /usr/share/pyshared/rdiff_backup/*.py So unless an infiltrator knew the new command name (remove-older-thax in the example above), they couldn't use it. Dominic On 11/11/2011 20:51, Grant wrote: I'm using rdiff-backup in an automated push arrangement with access to the backup server provided via SSH keys and restricted to the rdiff-backup command like command=rdiff-backup --server. I think an infiltrator could delete a compromised machine's backups from the backup server like this: rdiff-backup --remove-older-than 1s backup@12.34.56.78::/path/to/backup Is there any way to prevent something like that from happening? - Grant ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Is Rdiff-backup still developed?
On 14/11/2011 14:21, Filip Gruszczyński wrote: Do you have any plans to release an update to rdiff-backup-fs 1.0.0? Yes. I have a lot on my head recently (changed job and country too), but I have some changes in SCM waiting to be deployed. I will try to get to that and release a new version with some fixes that people were asking for. I will look forward to it! Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Hash doesn't match recorded hash
On 15/11/2011 00:23, Alex Schuster wrote: I need to restore an old VMware image, but I am getting LOTS of these errors: Error reading /backup/weird/home/wonko/os/vmware/xp/winXPPro-0.log, substituting empty file. Warning: Hash da39a3ee5e6b4b0d3255bfef95601890afd80709 of winXPPro-0.log doesn't match recorded hash db9943d0284382131b553cef380aae99871b0f2e! Any idea what has happened? rdiff-backup --check-destination-dir --force says nothing has to be checked, and option --verify does not find errors. The hard drive seems to be okay, no errors in the syslog, and a long SMART selftest also does not find any errors. I can restore some other directories I tried, but I need the backup of this VMware machine. Vmware files for a running vm may change while being backed up by rdiff-backup, in which case you don't have a consistent backup. Maybe this is why you have this inconsistent hash warning. Before using rdiff-backup on a VMware vm, my suggestion is to pause or suspend or stop the vm (e.g. with vmrun). Alternatively use the VMware snapshot facility and backup the snapshot rather than the running vm. None of this is much help if you have already lost your vm and are trying to recover it from backup. You might try recovering older versions of the vm (from your rdiff-backup repository), maybe you will find a consistent and usable vm there. Also the file you mention is only a log file and not critical. The critical files I think are *.vmdk, *.vmx and *.nvram. Regards Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Hash doesn't match recorded hash
I note there was a similar plea last March on this list, you can see it here: http://www.backupcentral.com/phpBB2/two-way-mirrors-of-external-mailing-lists-3/rdiff-backup-23/restore-fails-110831/. Unfortunately there was no published resolution. I wonder if the problem relates to the temporary folder being used on the server or (possibly) on the local machine? Try running the restore with --remote-tempdir and --tempdir pointed at locations with bags of space. Is it only this log file that gives the error message? What about the vmdk file? Dominic On 15/11/2011 12:27, Alex Schuster wrote: Maarten Bezemer writes: On Tue, 15 Nov 2011, Dominic Raferd wrote: On 15/11/2011 00:23, Alex Schuster wrote: Warning: Hash da39a3ee5e6b4b0d3255bfef95601890afd80709 of winXPPro-0.log doesn't match recorded hash db9943d0284382131b553cef380aae99871b0f2e! Any idea what has happened? Vmware files for a running vm may change while being backed up by rdiff-backup, in which case you don't have a consistent backup. Maybe this is why you have this inconsistent hash warning. If the contents changed during the rdiff-backup run, it bails out and doesn't save the (inconsistent) increment. So that shouldn't happen. I didn't know that, but I wonder if it always works like that or whether in extreme cases (such as huge vmdk files or corrupted source drive) it can break down. And I amusing a backup script which takes a LVM snapshot of my /home partition before rdiff-backupping it. Good thinking, but I doubt this guarantees that the vmdk file is consistent. Restoring other VMs seems to work, although I did not test many. But I cannot restore ANY backup of this special VM I need, I tried about six of my twenty backups. Well, it would be really nice to be able to restore this VM, but if not, I can (and have to) live with that. But I really would like to know why this happened, and if I can trust other backups I do. I successfully restored this VM in the past already, but that was an earlier snapshot I have deleted meanwhile. Yes I think we would all like an explanation for your problem, and a workaround. It could be our problem one day... Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] restoring only changed files
On 08/12/2011 05:12, Chris wrote: I'm doing some testing to learn how to utilise rdiff-backup. I'm hoping some one can help answer a few questions. I have created two directories. One is a directory called root where I have created two files. I back those files up to a directory called backup. When I run rdiff-backup -r now backup root it tells me Fatal Error: Restore target root already exists, specify --force to overwrite. If I add --force it continues to restore all the files. The problem is I don't need to restore all the files if some of the files haven't changed. I only want it to restore files that I have changed. Restoring even the files that haven't changed increases the time it takes to revert to an older backup. My goal is to be able rdiff-backup on a 'root partition' (root) and then restore the files from a backup in the event my system crashes during an apt-get upgrade. I am working with an already slow medium. I also know that rdiff-backup is not the fastest solution. It is probably the best solution though. I need to 'undo' changes relatively frequently too so this is of a great concern to me if it takes a long time to do a restore. The USP of rdiff-backup is that it keeps multiple previous versions of a dataset. But for your purpose I think rsync would be easier - and it can do the type of restore that you seek. Alternatively if you had LVM on your root partition and a newish Linux kernel it might be possible to take an LVM snapshot and then revert back to it if your system crashes - some info here: http://www.thegoldfish.org/2011/09/reverting-to-a-previous-snapshot-using-linux-lvm/. Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] rdiff-backup failed: error 31
On 13/12/2011 14:31, Jean Pierre Dentone wrote: Hi I'm trying to get a backup of a 3TB partition and it failed with this error Exception '[Errno 31] Too many links: '/rdiff/newarsvrfs01/htdocs/chase.archinetonline.com/chasedata/project_89381'' raised of class 'exceptions.OSError': This has been working ok for a while and then we had a problem with the RAID being degrated. we fixed the raid issue and the backups never worked again. I have even tried to do a fresh backup on a new partition/location but we get the same error any ideas what the problem could be? we havent changed anything on our backup server. PS: Im using rdiff-backup 1.2.8 Try running rdiff-backup with --no-eas option (unless you need extended attributes), and see if this helps. Dominic http://www.timedicer.co.uk: File Recovery from Whenever ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] KeyError while backing up to sshfs
Jochen:
what if you change /etc/backup.list by removing the final slash from
/backup/dev/ i.e. /backup/dev
I think that with your backup list it tries to backup /backup/dev
(though not its contents) and maybe this causes the failure?
Dominic
-
TimeDicer: Free File
Recovery from Whenever
On 25/01/2012 14:45, Jochen wrote:
Hi,
I get a (repeatable) KeyError while backing up to sshfs using
rdiff-backup on linux (either version 1.2.8 or 1.3.3). My backups ran
normally for about a year, then this error came, and they don't work
since.
Any help would be greatly appreciated!
Regards,
Jochen
Command
===
rdiff-backup --no-hard-links --include-globbing-filelist
/etc/backup.list --exclude-other-filesystems -v9 /backup /mnt/backup/one
Source directory /backup is a snapshot of /, /mnt/backup is a mounted
sshfs, /dev does not exist in the target directory.
/etc/backup.list
===
- **/*~
- **/rdiff-backup.tmp.*
- **/#*#
- **/rdiff-backup.tmp.*
- /backup/usr/portage/
- /backup/tmp/
- /backup/var/tmp/
- /backup/sys/
- /backup/root/
- /backup/var/log/
- /backup/var/cache/
- /backup/var/run/
- /backup/dev/
Error
===
Wed Jan 25 15:32:48 2012 Setting time of /mnt/backup/one/data to 1326994981
Wed Jan 25 15:32:48 2012 Exception '('dev',)' raised of class 'type
'exceptions.KeyError'':
File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
306, in error_check_Main
try: Main(arglist)
File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
326, in Main
take_action(rps)
File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
282, in take_action
elif action == "backup": Backup(rps[0], rps[1])
File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
345, in Backup
backup.Mirror_and_increment(rpin, rpout, incdir)
File "/usr/lib64/python2.7/site-packages/rdiff_backup/backup.py",
line 51, in Mirror_and_increment
DestS.patch_and_increment(dest_rpath, source_diffiter, inc_rpath)
File "/usr/lib64/python2.7/site-packages/rdiff_backup/backup.py",
line 251, in patch_and_increment
ITR(diff.index, diff)
File "/usr/lib64/python2.7/site-packages/rdiff_backup/rorpiter.py",
line 280, in __call__
if last_branch.can_fast_process(*args):
File "/usr/lib64/python2.7/site-packages/rdiff_backup/backup.py",
line 524, in can_fast_process
mirror_rorp = self.CCPP.get_mirror_rorp(index)
File "/usr/lib64/python2.7/site-packages/rdiff_backup/backup.py",
line 479, in get_mirror_rorp
except KeyError: return self.get_parent_rorps(index)[1]
File "/usr/lib64/python2.7/site-packages/rdiff_backup/backup.py",
line 461, in get_parent_rorps
raise KeyError(index)
Traceback (most recent call last):
File "/usr/bin/rdiff-backup-2.7", line 30, in module
rdiff_backup.Main.error_check_Main(sys.argv[1:])
File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
306, in error_check_Main
try: Main(arglist)
File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
326, in Main
take_action(rps)
File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
282, in take_action
elif action == "backup": Backup(rps[0], rps[1])
File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
345, in Backup
backup.Mirror_and_increment(rpin, rpout, incdir)
File "/usr/lib64/python2.7/site-packages/rdiff_backup/backup.py",
line 51, in Mirror_and_increment
DestS.patch_and_increment(dest_rpath, source_diffiter, inc_rpath)
File "/usr/lib64/python2.7/site-packages/rdiff_backup/backup.py",
line 251, in patch_and_increment
ITR(diff.index, diff)
File "/usr/lib64/python2.7/site-packages/rdiff_backup/rorpiter.py",
line 280, in __call__
if last_branch.can_fast_process(*args):
File "/usr/lib64/python2.7/site-packages/rdiff_backup/backup.py",
line 524, in can_fast_process
mirror_rorp = self.CCPP.get_mirror_rorp(index)
File "/usr/lib64/python2.7/site-packages/rdiff_backup/backup.py",
line 479, in get_mirror_rorp
except KeyError: return self.get_parent_rorps(index)[1]
File "/usr/lib64/python2.7/site-packages/rdiff_backup/backup.py",
line 461, in get_parent_rorps
raise KeyError(index)
KeyError: ('dev',)
___
rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] KeyError while backing up to sshfs
I guess you already considered if you made any changes to your
system or to /etc/backup.list that might have caused the error to
occur?
You could try one or more of options --exclude-special-files
--exclude-sockets --exclude-other-filesystems...
Dominic
On 26/01/2012 17:27, Jochen wrote:
Dominic, thanks for the tip, alas no luck. Didn't
work with the changed backup.list either.
- **/*~
- **/rdiff-backup.tmp.*
- **/#*#
- **/rdiff-backup.tmp.*
- /backup/usr/portage
- /backup/tmp
- /backup/var/tmp
- /backup/sys
- /backup/root
- /backup/var/log
- /backup/var/cache
- /backup/var/run
- /backup/dev
Regards,
Jochen
On 2012-01-26 13:09:17 +, Dominic Raferd said:
Jochen:
what if you change /etc/backup.list by removing the
final slash from /backup/dev/ i.e. /backup/dev
I think that with your backup list it tries to
backup /backup/dev (though not its contents) and maybe this
causes the failure?
Dominic
-
TimeDicer: Free File
Recovery from Whenever
On 25/01/2012 14:45, Jochen wrote:
Hi,
I get a (repeatable) KeyError while backing up to
sshfs using
rdiff-backup on linux (either version 1.2.8 or
1.3.3). My backups ran
normally for about a year, then this error came, and
they don't work
since.
Any help would be greatly appreciated!
Regards,
Jochen
Command
===
rdiff-backup --no-hard-links
--include-globbing-filelist
/etc/backup.list --exclude-other-filesystems -v9
/backup /mnt/backup/one
Source directory /backup is a snapshot of /,
/mnt/backup is a mounted
sshfs, /dev does not exist in the target directory.
/etc/backup.list
===
- **/*~
- **/rdiff-backup.tmp.*
- **/#*#
- **/rdiff-backup.tmp.*
- /backup/usr/portage/
- /backup/tmp/
- /backup/var/tmp/
- /backup/sys/
- /backup/root/
- /backup/var/log/
- /backup/var/cache/
- /backup/var/run/
- /backup/dev/
Error
===
Wed Jan 25 15:32:48 2012 Setting time of
/mnt/backup/one/data to 1326994981
Wed Jan 25 15:32:48 2012 Exception '('dev',)'
raised of class 'type
'exceptions.KeyError'':
File
"/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
306, in error_check_Main
try:
Main(arglist)
File
"/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
326, in Main
take_action(rps)
File
"/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
282, in take_action
elif
action == "backup": Backup(rps[0], rps[1])
File
"/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
345, in Backup
backup.Mirror_and_increment(rpin,
rpout, incdir)
File
"/usr/lib64/python2.7/site-packages/rdiff_backup/backup.py",
line 51, in Mirror_and_increment
DestS.patch_and_increment(dest_rpath,
source_diffiter, inc_rpath)
File
"/usr/lib64/python2.7/site-packages/rdiff_backup/backup.py",
line 251, in patch_and_increment
ITR(diff.index,
diff)
File
"/usr/lib64/python2.7/site-packages/rdiff_backup/rorpiter.py",
line 280, in __call__
if
last_branch.can_fast_process(*args):
File
"/usr/lib64/python2.7/site-packages/rdiff_backup/backup.py",
line 524, in can_fast_process
mirror_rorp
= self.CCPP.get_mirror_rorp(index)
File
"/usr/lib64/python2.7/site-packages/rdiff_backup/backup.py",
line 479, in get_mirror_rorp
except
KeyError: return self.get_parent_rorps(index)[1]
File
"/usr/lib64/python2.7/site-packages/rdiff_backup/backup.py",
line 461, in get_parent_rorps
raise
KeyError(index)
Traceback (most recent call last):
File
"/usr/bin/rdiff-backup-2.7", line 30, in module
rdiff_backup.Main.error_check_Main(sys.argv[1:])
File
"/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
306, in error_check_Main
try:
Main(arglist)
File
"/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line
326, in Main
take_action(rps)
File
&qu
Re: [rdiff-backup-users] how to push rdiff-backup
On 10/03/12 08:00, Nicolas Jungers wrote: On 2012-03-10 04:07, Willem Buitendyk wrote: I'm a little perplexed. My scenario is that I have data loggers in the field, each with a 3g usb modem on board. I want to have the data loggers rdiff back to my server on amazon ec2 - or to push the backup. My data loggers have dynamic ip addresses and the telco company keeps them hidden. In other words I can't have my backup amazon ec2 server poll my data logger. Again, the need to push the backup. I'm not really finding any information on how to accomplish this. Any suggestions would be greatly appreciated. The way I do it is to establish an openVPN tunnel. Beware that rdiff-backup is very sensitive to the link quality. Agreed about the link quality, and so I would suggest that rather than try to run rdiff-backup over a 3g connection, you run rsync, which is much better at recovering from broken connections, especially with the --partial and --link-dest switches (and, occasionally, --checksum). Each machine in the field could rsync back to their dedicated folder on the backup machine and then the backup machine can run rdiff-backup locally. By combining the 2 programs in this way you can still get the benefit of rdiff-backup's versioning '4D' backup. I also recommend creating a snapshot on the source machine and backing up from this rather than from the original data (which might change while the backup is in progress). For this you want LVM (for Linux) or VSS (for Windows) - I don't know what the equivalent is for Apples. Dominic TimeDicer http://www.timedicer.co.uk - Windows Backup and File Recovery from Whenever ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] [m] help using rdiff-backup on Windows
If you want to carry on running native rdiff-backup.exe directly, I suggest you remove backslashes and just use a single forward slash as the directory separator in your specifications (including in your exclude-list file). It is possible that you will hit further problems because you are backing up via SMB, but the best way to find out is to try it... (Regardless of source, the most reliable *destination* for rdiff-backup repositories IMO is a Linux machine/filesystem.) Alternatively have a look at my TimeDicer utility which uses rdiff-backup Windows native client and is tailored specifically for backups from Windows: http://www.timedicer.co.uk. This allows you to specify files and directories in windows style (with the normal Windows backslashes and ignoring case sensitivity) and backs up to a 'TimeDicer Server' which is a dedicated Linux machine (real or virtual). It also uses VSS so it can backup locked files. Dominic On 12/05/2012 02:42, Teddy Brown wrote: Hey all, I'm having difficulty running rdiff-backup on Windows, both under Cygwin and natively. This program is clearly geared towards real Unix based systems so I haven't been able to find many docs to help out. What I'd like to be able to do is create an include list of files to back up. My source directories are across 3 local hard drives and my target is a NAS connected with SMB (actually an Apple Time Capsule). I simply can't get Cygwin to work, at all to the network drive, it gives Python exceptions. In the native Windows client I get a little further: C:\Users\Teddyrdiff-backup --include-filelist backup_list.txt --exclude ** C:\\/ \\teddys-time-cap\Data\Backups\win7-desktop ListError $INPLACE.~TR [Error 5] Access is denied: 'C:/$INPLACE.~TR/*.*' ListError $Recycle.Bin/S-1-5-20 [Error 5] Access is denied: 'C:/$Recycle.Bin /S-1-5-20/*.*' Warning: file specification 'E:\\/Digital\ Camera\ Pics' in filelist backup_list .txt doesn't start with correct prefix C:\\. Ignoring. Warning: file specification 'F:\\/Users' in filelist backup_list.txt doesn't start with correct prefix C:\\. Ignoring. Fatal Error: Fatal Error: The file specification 'C:\\/**' cannot match any files in the base directory 'C:\\' Useful file specifications begin with the base directory or some pattern (such as '**') which matches the base directory. C:\Users\Teddyrdiff-backup --include-filelist backup_list.txt --exclude C:\\/* * C:\\/ \\teddys-time-cap/Data/Backups/win7-desktop Found interrupted initial backup. Removing... ListError $INPLACE.~TR [Error 5] Access is denied: 'C:/$INPLACE.~TR/*.*' ListError $Recycle.Bin/S-1-5-20 [Error 5] Access is denied: 'C:/$Recycle.Bin /S-1-5-20/*.*' Warning: file specification 'E:\\/Digital\ Camera\ Pics' in filelist backup_list .txt doesn't start with correct prefix C:\\. Ignoring. Warning: file specification 'F:\\/Users' in filelist backup_list.txt doesn't start with correct prefix C:\\. Ignoring. Fatal Error: Fatal Error: The file specification 'C:\\/**' cannot match any files in the base directory 'C:\\' Useful file specifications begin with the base directory or some pattern (such as '**') which matches the base directory. backup_list.txt contains: E:\\/Digital\ Camera\ Pics F:\\/Users C:\\/Users Any help? ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] resuming initial backup
Not that I know of. Because of the complexity of the underlying archive, rdiff-backup does not like a failed or interrupted previous backup attempt at all and tries to remove one if it finds it. Otherwise the risk would be that you corrupt the archive and lose your data history. Although it might be possible in theory for rdiff-backup to continue a previously-interrupted session, the code to do this doesn't exist and isn't likely to be written. rsync is good at recovering from broken sessions but then it is not trying to do anything so complex. But there are some backup tools based around rsync such as rsnapshot which might work better for you, although they lack the power of rdiff-backup. Dominic On 20/05/2012 12:24, tim wrote: hi, i'm using a synology ds411 nas as backup target. the processor on this device is very slow, so i don't get a transfer rate of more than 5 mb/s when accessing it via ssh (cpu-bound). this makes backups for my 500GB data partition rather painful. i'd therefore like to interrupt and later resume the initial backup. however rdiff-backup tells me: Found interrupted initial backup. Removing... ... and then starts copying the files ... is there any way to resume an initial backup? thanks, tim ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] resuming initial backup
What exactly needs to be deleted (i.e. the rdiff metadata)? What happens to the data history (held in the rdiff-backup directory), is it still all recoverable after you have followed this procedure? Dominic On 20/05/2012 12:34, D. Kriesel wrote: PS.: You might have to --force the backup, not sure, but I know it works ... -Ursprüngliche Nachricht- Von: rdiff-backup-users-bounces+mail=dkriesel@nongnu.org [mailto:rdiff- backup-users-bounces+mail=dkriesel@nongnu.org] Im Auftrag von D. Kriesel Gesendet: Sonntag, 20. Mai 2012 13:34 An: 'tim'; rdiff-backup-users@nongnu.org Betreff: Re: [rdiff-backup-users] resuming initial backup Rdiff-backup takes any existing data as a base in a target dir that has to be initialized, so just remove the rdiff-metadata from the target, while keeping the data that has already been copied :-) Cheers David ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] resuming initial backup
On 20/05/2012 13:02, D. Kriesel wrote: The whole history will be lost when following this procedure, which renders it inappropriate for backup repositories whose history actually exists. However, if I get Tim right, he is talking about an interrupted _initial_ backup run. In this case, giving up the entire history obviously does not do any harm. Without any metadata (i.e. the rdiff-backup-data subfolder), afaik a forced rdiff-backup run will take whatever there is in the target folder as a base and won't retransfer existing parts. So my solution might be the right thing to do in this special case. Ah yes, in that case it might work. Thanks for the explanation Dominic ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] rdiff-backup file consistency
@David: if you try obnam please let us know how you get on with it. I guess one could store rdiff-backup repositories on a deduplication file system such as lessfs or zfs and get the de-duplication benefits. And using the --no-compression switch might or might not bring further space savings (as well as, presumably, faster backups/restores/verifies). But I am not sure if any deduplication file systems are reliable enough for backup? On 09/06/2012 09:37, D. Kriesel wrote: If I get it right, there is no such thing as deltas. Every generation seems to virtually stand for its own. However, double chunks of data (no matter if being caused by different machines or generations or whatever) are deduplicated in a generalized way. Nice design! As check for duplicate data made before transmission, this could be also very efficient (however, one could still transmit with rsync and than use obnam). I'll stop spamming this list now with obnam features, however the delta question is the one that arises for sure when other backup systems are mentioned here ;) Cheers David -- Timedicer - File Recovery from Whenever ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] rdiff-backup vs. Back-In-Time
Thanks for your posting Kshitij, you give us yet another backup program to consider - back-in-time. If it ain't broke, don't fix it: If your backup routine works for you, why change to rdiff-backup? Back-in-time (according to the website) is a GUI and under the hood it uses rsync, diff, cp and hardlinks to take snapshots. To rdiff-backup users this sounds a bit like reinventing the wheel and it may well be much less space-efficient than rdiff-backup. It's not clear to me whether a small change in a large file leads back-in-time to store the whole file twice; if so, these types of files (databases, mailboxes) could eat up a huge amount of space (in theory 24 x their original size per day). rdiff-backup only stores the diffs or deltas which in such cases are comparatively small. But if you have plenty of space or are happy to delete snapshots older then a certain limited period (1 week? 1 month?) then even in this scenario the space considerations may not be important. bug-free = not yet thoroughly tested: I do think however that anyone reading this mailing list should bear in mind that people post here when they have difficulties. For the most part rdiff-backup works very well and has proved a reliable backup for many years - for me and for others. I recently had to recover an old version of a database file from about 9 months ago (i.e. about 230 reverse-diffs) - rdiff-backup worked perfectly (albeit the recovery was quite slow). I mention this not because it is any kind of record but as a real-world situation where rdiff-backup saved my bacon. For users who find the command line too alarming but still like the power of rdiff-backup, there is jBackup http://sourceforge.net/projects/jbck/, while for Windows users I would naturally say that TimeDicer http://www.timedicer.co.uk (which I maintain) is the way to go. For restoring, rdiffWeb http://www.timedicer.co.uk/rdiffweb/index * works nicely from your browser. Dominic * static version of the wiki; the dynamic one is prone to spammer attacks On 10/06/12 03:28, Kshitij Kotak wrote: Dear rdiff-backup Experts Pardon my naïve query, but need to understand what is the difference between rdiff-backup approach and the following steps: 1) we take the remote sync of primary data store on a mirror server using rsync. This is automated for every 1 hour using cron. 2) to get a point-in-time restore, we use back-in-time on mirror server to get the data stored locally in time slots to recover. My back-in-time runs once every day and is cronned using Back-In-time internal switch that allows me to define schedule. This approach has worked flawless (so far). Plus back-in-time has a fantastic GUI which, for a non-expert like me is a great relief. From what gather on this group, rdiff-backup saves much larger amount of space than my approach. Is that correct? Considering the complexities of command line approach, restore issues and the kind of problems you guys report... I am petrified to try out rdiff-backup. Does rdiff-backup offer me any significant benefits over my novice approach? If so, is there a better, less complex, more reliable way to implement backup (and guarantee restore :D ) for a novice like me? Await your replies... Cheers Kshitiz Sent from BlackBerry® Xcuze typos if N E ___ rdiff-backup-users mailing list atrdiff-backup-us...@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL:http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Possible to Restore Single File
I recommend rdiffweb to restore single files from an rdiff-backup repository. On 22/08/12 22:54, wardrop wrote: Any other means for that'll work on Windows. I have a combination of Windows, OS X and a Busybox QNAP NAS that all need rdiff and possible restore functionality; it's mainly the clients (Windows, OS X) that will need to restore single files. +-- |This was sent by t...@tomwardrop.com via Backup Central. |Forward SPAM to ab...@backupcentral.com. +-- ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Winsdows 7/Vista ACLs not restored correctly?
Leland, have you tried doing the backup from your Windows machine to a Linux host and then restoring from there? Or, if that is not possible, using the linux version of rdiff-backup on your Windows machine through cygwin? Dominic *TimeDicer - Windows Backup and File Recovery from Whenever http://www.timedicer.co.uk/* On 07/09/12 22:56, Leland Best wrote: Hello All, I've tried Googling this problem as well as searching the mailing list archives with no luck. Also there was a Wiki about Windows - Linux backups but that seems to have been down for quite a while now. So, my apologies if this has been answered somewhere and I just haven't found it. Anyway, ... The short version is: When I back up then restore using the rdiff-backup win32 binaries from http://www.nongnu.org/rdiff-backup/ the permissions/ACLs are not restored correctly. I've tried many different combinations of things but even in the simplest cases (i.e. source, backup, and restore directories all on c:\ someplace) the restored permissions/ACLs are incorrect. I need to know if this is a known issue or whether I'm missing something. Here are some more details. On both Windows 7 64-bit and Windows Vista 32-bit I've done the following: 1) Download and unzip rdiff-backup-1.2.8-win32.zip (I've also tried rdiff-backup-1.3.3-win32.zip). 2) Copied the resulting rdiff-backup.exe file to C:\Program Files \Savannah (which I created manually) and added this directory to the end of the system-wide PATH environment variable. 3) Reboot (just to be sure the new PATH is picked up by everything). 4) Log in as, say, my normal (administrative) user which we'll call 'lbest'. 5) Click 'Start', enter 'cmd' in the search box, right-click 'cmd.exe' and select Run as administrator. 6) mkdir C:\Backup 7) rdiff-backup -v 5 c:/Users/Leland c:/Backup/Lelandc:/Backup/Leland_backup.log 21 (all as one command obviously) where 'Leland' is another administrative user I created just for testing. After a little bit this completes without significant errors. 8) rdiff-backup -v 5 -r now c:/Backup/Leland c:/Users/Leland.restoredc:/Backup/Leland_restore.log 21 This also appears to complete without errors. However, when I compare the Properties - Security (in Windows Explorer or whatever) for C: \Users\Leland and C:\Users\Leland.restored they are quite different. For example, 'Leland' does not inherit from C:\Users but 'Leland.restored' does. 'Leland' only shows permissions for users 'SYSTEM', 'Leland', and 'Administrators' while 'Leland.restored' also has permissions for 'EVERYONE', and 'Users'. The specifics of the permissions for each user differ too. So, am I doing something wrong? If so, what? If not, then how on earth are other folks using rdiff-backup to back up Windows machines? Any help, pointers, etc. would be greatly appreciated. If posting log files or additional info will help just let me know. BTW (and slightly off topic) I've used rdiff-backup under Debian GNU/Linux for years and it has worked well. I've done bare metal restores given a Debian Live CD with rdiff-backup installed. I've even used ntfsclone to image an entire Windows partition and put _that_ in rdiff-backup. It all works like a champ! Unfortunately, none of these are acceptable for my current application. :( Thanks in advance. Leland ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Winsdows 7/Vista ACLs not restored correctly?
Sorry Leland, I have always explicitly excluded acls from my Windows backups. But in any case for ACLs rdiff-backup requires the pylibacl and pyxattr modules. My standard rdiff-backup installation (under Ubuntu 12.04) does not have pylibacl or pyxattr. You can check whether you have them in your Debian with: pydoc modules|egrep -o "(pylibacl|pyxattr)" Could this be your problem? Dominic On 09/09/2012 02:56, Leland Best wrote: Hello Dominic, Thank you for taking the time to reply. As I said, I've tried quite a few combinations of things but figured my initial post would be long enough without going through them all. Also, I figured the case I outlined was the simplest one (i.e. least number of variables to worry about). But since you ask ... On Sat, 2012-09-08 at 09:26 +0100, Dominic Raferd wrote: Leland, have you tried doing the backup from your Windows machine to a Linux host and then restoring from there? Since Linux is my OS of choice, this was actually the first thing I tried. My Linux boxes run Debian GNU/Linux. I have one running "squeeze/stable" and one running "wheezy/testing". Only rdiff-backup 1.2.8 is available as an official Debian package though I also tried removing the Debian package and building/installing 1.3.3 from source. In each case the corresponding win32 native binary was installed on the Windows side. Also, on the Windows/client side I tried using both the Cygwin OpenSSH and PuTTY to make the connection to the Linux box. In all cases, as best I can remember, the result was the same as in the simple case I outlined originally (i.e. no errors but restored permissions/ACLs were wrong). Or, if that is not possible, using the linux version of rdiff-backup on your Windows machine through cygwin? [...] [rest of original post deleted] I assume you mean using the Cygwin version of rdiff-backup on the Windows machine to backup and restore to/from the linux box? I've tried this too and the results are (mostly) worse. For example, I removed the win32 native rdiff-backup, installed Cygwin's version (1.2.8), and did (in a Cygwin rxvt/bash window): prompt rdiff-backup -v 5 /cygdrive/c/Users/Leland lbest@harry::tmp/Backup/Leland ./Leland_backup.log 21 prompt rdiff-backup -v 5 -r now lbest@harry::tmp/Backup/Leland /cygdrive/c/Users/Leland.restore ./Leland_restore.log 21 After this, here are the permissions on C:\Users\Leland and C:\Users \Leland.restored C:\Users\Leland (the original source directory): SYSTEM: not inherited, Full Control, This folder, subfolders and files Leland: not inherited, Full Control, This folder, subfolders and files Administrators: not inherited, Full Control, This folder, subfolders and files and on the restored C:\Users\Leland.restored directory: lbest: not inherited, Special, This folder only None: not inherited, Special, This folder only Everyone: not inherited, Special, This folder, subfolders and files SYSTEM: not inherited, Full Control, This folder, subfolders and files Administrator: not inherited, Full Control, This folder, subfolders and files Users: not inherited, Read Execute, This folder, subfolders and files CREATOR OWNER: not inherited, Special, subfolders and files only CREATOR GROUP: not inherited, Special, subfolders and files only The only _good_ thing is that it gets "not inherited" right. Everything else is all bollixed up. The original owner (Leland) has no permissions at all! BTW, I also tried this using the Cygwin rdiff-backup entirely locally on the Windows machine with exactly the same results. At least it's consistent. Is the format of the windows metadata and ACL files (stored in the rdiff-backup-data directory) documented anywhere? Then maybe I could at least determine whether the permissions and ACLs were being stored correctly in the backup. I take it nobody else has had this problem? If not, then it must be something weird in my setup. But I'm clueless what it could be. Locale maybe? I dunno. Thanks again for your time. Cheers Leland -- TimeDicer: Free File Recovery from Whenever ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Winsdows 7/Vista ACLs not restored correctly?
Leland: Plaudits for your detailed research. Here is a developer thread from the time when the Windows ACL code was being developed for rdiff-backup: http://www.backupcentral.com/phpBB2/two-way-mirrors-of-external-mailing-lists-3/rdiff-backup-23/patch-backing-up-windows-acls-90543/ (June 2008). Sadly rdiff-backup is no long under development, the last maintainer seemed to lose interest after March 2009 and no one has picked up the baton. I still find it very powerful and reliable for backing up data from Windows machines to a Linux backup server - I'm not bothered about permissions, though. (Note that the rdiff-backup option --no-acls works both for Posix and Windows systems.) Dominic *TimeDicer* http://www.timedicer.co.uk: Free File Recovery from Whenever On 25/09/2012 03:31, Leland Best wrote: Dominic, Sorry for the delay in replying. Anyway, ... On Wed, 2012-09-19 at 17:53 +0100, Dominic Raferd wrote: Sorry Leland, I have always explicitly excluded acls from my Windows backups. Hmmm, well there's something I haven't tried. However, as I understand it (which may well be wrong of course) the rdiff-backup options about ACLs (e.g. --no-acls) all apply to POSIX ACLs. Windows ACLs are not POSIX ACLs and, in fact, are handled by a different Python module. I also believe that under Windows the ACLs are basically the _only_ security mechanism so to disable them would loose all permissions info. But in any case for ACLs rdiff-backup requires the pylibacl and pyxattr modules. My standard rdiff-backup installation (under Ubuntu 12.04) does not have pylibacl or pyxattr. You can check whether you have them in your Debian with: pydoc modules|egrep -o (pylibacl|pyxattr) 'pydoc' spits out a bunch of errors for me but: lbest@harry:~ dpkg -s python-pylibacl | grep Status Status: install ok installed lbest@harry:~ dpkg -s python-pyxattr | grep Status Status: install ok installed This extract from a linux-linux backup is suggestive: /usr/lib/pymodules/python2.6/rdiff_backup/SetConnections.py:148: DeprecationWarn ing: os.popen2 is deprecated. Use the subprocess module. stdin, stdout = os.popen2(remote_cmd) Unable to import win32security module. Windows ACLs not supported by filesystem at /buckbeak/root/home escape_dos_devices not required by filesystem at /buckbeak/root/home Using rdiff-backup version 1.2.8 Executing ssh -C root@thestral rdiff-backup --server - Detected abilities for source (read only) file system: Access control lists On Extended attributes On Windows access control lists Off Case sensitivity On Escape DOS devices Off Escape trailing spaces Off Mac OS X style resource forksOff Mac OS X Finder information Off - Unable to import win32security module. Windows ACLs not supported by filesystem at home/rdiff-backup-data/rdiff-backup.tmp.0 escape_dos_devices not required by filesystem at home/rdiff-backup-data/rdiff-backup.tmp.0 - Detected abilities for destination (read/write) file system: Ownership changing On Hard linking On fsync() directories On Directory inc permissionsOn High-bit permissions On Symlink permissions Off Extended filenames On Windows reserved filenames Off Access control lists On Extended attributes On Windows access control lists Off Case sensitivity On Escape DOS devices Off Escape trailing spaces Off Mac OS X style resource forksOff Mac OS X Finder information Off - First, note that both source and destination show Access control lists and Extended attributes as On. Presumably if pylibacl and/or pyxattr were not installed they would be Off. Second, and more to the point, are these lines: Unable to import win32security module. Windows ACLs not supported by filesystem at /buckbeak/root/home [...] Windows access control lists Off I believe _Windows_ ACLs are handled in module win32security. Now the log files from the windows native client doing a windows-linux backup shows Windows access control lists as On on the source (windows) filesystem but Off on the destination filesystem. This is okay though because rdiff-backup stores the windows ACL data
Re: [rdiff-backup-users] [a2] Please help getting my backups running.
Hi Gary From what I can tell this doesn't seem to be a problem with rdiff-backup but some sort of issue with ssh? Anyway... ssh-copy-id -i /home/clientrdiff/.ssh/id_rsa.pub serverrd...@server.server.com mailto:serverrd...@server.server.com as both root and clientrdiff returns ssh: connect to host server.server.com http://server.server.com port 22: Connection refused If I add -p or use the Host from config I get: /usr/bin/ssh-copy-id: ERROR: No identities found I am not familiar with ssh-copy-id but checking on man page it looks as if it does not support the -p option and I guess you are using a non-standard ssh port? So don't use ssh-copy-id, just add the public key manually; you only have to do it once for each remote user (root and clientrdiff). Do you know how to do this? /etc/fstab sshfs#ser...@server.com:/home/serverrdiff/ToBeBackedUp /home/clientrdiff/mnt/server.com:/home/clientrdiff/ToBeBackedUp fuse user,noauto,ro 0 0 mount /home/clientrdiff/mnt/server.server.com:/home/serverrdiff/ToBeBackedUp returns: mount: can't find /home/clientrdiff/mnt/ser...@server.com:/home/serverrdiff/ToBeBackedUp in /etc/fstab or /etc/mtab Your second problem seems to be with sshfs, your syntax looks strange to me. To mount the remote directory 'ToBeBackedUp' at say /home/clientrdiff/remotebackup I think it should be like this: $ mkdir -p /home/clientrdiff/mnt/remotebackup $ sshfs ser...@server.com:/home/serverrdiff/ToBeBackedUp /home/clientrdiff/mnt/remotebackup -o workaround=rename But why are you using sshfs at all? You don't need this for rdiff-backup, unless rdiff-backup cannot be installed on either client and server. Using sshfs will make backups slower and use more bandwidth. Here is an example of rdiff-backup doing a push backup (i.e. run on client) using a non-standard ssh port 9222: $ rdiff-backup --remote-schema ssh -C -p9222 %s rdiff-backup --server /home/clientrdiff/ToBeBackedUp serverrd...@server.server.com::/home/serverrdiff/ToBeBackedUp BTW both your machines are using a very old version of rdiff-backup (1.0.5), you should use 1.2.8 (the last stable version) or 1.3.3 (the last unstable, but seems to work fine). Regards, Dominic *TimeDicer - Windows Backup and File Recovery from Whenever http://www.timedicer.co.uk/* On 30/10/12 19:21, Gary Rickert wrote: I have spent the last week googling to figure this out but have failed. I have 2 servers, what I am going to call client, the system that will back up some directories on the server. I can ssh w/ no password by both client-root user and clientrdiffbackup and connect to a user with full sudo priveledges. The info following shows What, client, server. uname -r 3.4.2-x86_64-linode25 3.0.18-x86_64-linode24 yum info rdiff-backup Installed Packages Installed Packages Name : rdiff-backup Name : rdiff-backup Arch : x86_64 Arch : x86_64 Version: 1.0.5 Version: 1.0.5 Release: 2.el5Release: 2.el5 yum info fuse-sshfs Version: 2.4Version: 2.4 Users root and CLIENTrdiffSERVERrdiff ssh-copy-id -i /home/clientrdiff/.ssh/id_rsa.pub serverrd...@server.server.com mailto:serverrd...@server.server.com as both root and clientrdiff returns ssh: connect to host server.server.com http://server.server.com port 22: Connection refused If I add -p or use the Host from config I get: /usr/bin/ssh-copy-id: ERROR: No identities found /etc/fstab sshfs#ser...@server.com:/home/serverrdiff/ToBeBackedUp /home/clientrdiff/mnt/server.com:/home/clientrdiff/ToBeBackedUp fuse user,noauto,ro 0 0 mount /home/clientrdiff/mnt/server.server.com:/home/serverrdiff/ToBeBackedUp returns: mount: can't find /home/clientrdiff/mnt/ser...@server.com:/home/serverrdiff/ToBeBackedUp in /etc/fstab or /etc/mtab Same symptom when if fstab and command is configured to use root. This is what I hope is salient, but I will be most happy to answer any questions. ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Please help getting my backups running.
Well Gary I'm glad you succeeded in the end, and pleased to be of service. Yes normally the build location is /usr/src (absolute location) but you seemed to be using ./usr/src that is why I suggested you stick with that. If it has worked for you with /usr/src then that's better IMHO. I have now uploaded my rdiff-backup installer program rdiff-backup-install.sh to the web - help and download page at http://www.timedicer.co.uk/programs/help/rdiff-backup-install.sh. I've tweaked and renumbered it v0.3. If you use it again please use the latest version from there just so that I can check it is bug-free. Now that you have rdiff-backup working you might want to install rdiffweb which provides a web interface for retrieving files from the rdiff-backup server; install program is at http://www.timedicer.co.uk/programs/help/rdiffweb-install.sh. Regards, Dominic -- TimeDicer: Free File Recovery from Whenever On 02/11/2012 13:50, Gary Rickert wrote: Well Dominic, I sure hope you have had as much fun as I have:) All kidding aside, it looks like this script did it. I installed on my backup system, ran and got the version mis-match message, as expected. Ran rdiff-backup-install.sh on my other system, ran and all again looks OK. Thank you, thank you. So, what was the challenge? I will be doing a bunch more testing/bench-marking again in a few days, but I have some catching up to do after spending most of the last week emersed in this. I haven't looked in detail at the following install output, or tried to figure out the install script, but I believe the destination address in the install command line (/usr/src) should be absolute, not relative. y/n? My hope is that you don't hear from me again, but hope I haven't worn out my welcome if I do have more issues. It is not often that I see this level of responsiveness, even from paid vendors. Thanks again Gary ** root ~/install#./rdiff-backup-install.sh 1.2.8 /usr/src rdiff-backup-install.sh v0.2 [02 Nov 12] by Dominic (-h for help) === Searching for librsync.so*: /usr/lib Searching for Python.h: /usr/include/python2.4/Python.h Download rdiff-backup-1.2.8.tar.gz [y/-]: y Untar the rdiff-backup-1.2.8.tar.gz [y/-]: Build the downloaded program [y/-]: y running build ** On Fri, Nov 2, 2012 at 3:30 AM, Dominic Raferd domi...@timedicer.co.uk wrote: Here is a further-improved version of my script, and hopefully it can now use yum to install any missing dependencies. Notice that its name has changed to rdiff-backup-install.sh (i.e. with '.sh' added at the end). Try this (v0.2) instead of the one I sent a couple of hours ago. Dominic On 02/11/2012 00:50, Gary Rickert wrote: I hope "without any warranties" doesn't mean without assistance:} When I run the script I get: root ~/install/rdiff-backup-1.2.8 # installrdiff -pnq 1.2.8 /opt Unable to locate librsync.a. Please install librsync-dev and try again. root ~/install/rdiff-backup-1.2.8 # installrdiff -pnq rdiff-backup-1.2.8 /opt Unable to locate librsync.a. Please install librsync-dev and try again. The first thing I noticed was the package name installed: Package librsync-devel-0.9.7-13.el5.x86_64 already installed and latest version Package librsync-devel-0.9.7-13.el5.i386 already installed and latest version Name : librsync Arch : x86_64 Version : 0.9.7 Release : 13.el5 Hope this search may help: ./root/install/rdiff-backup-1.2.8/build/lib.linux-x86_64-2.4/rdiff_backup/librsync.py ./root/install/rdiff-backup-1.2.8/rdiff_backup/librsync.py ./usr/lib64/librsync.so ./usr/lib64/librsync.so.1.0.2 ./usr/lib64/librsync.so.1 ./usr/share/man/man3/librsync.3.gz ./usr/share/doc/librsync-0.9.7 ./usr/include/librsync-config.h ./usr/include/
Re: [rdiff-backup-users] WindowsError: [Error 5]
Here is my 2 cents: Long file names are not a problem for rdiff-backup 'Foreign' characters should not be a problem provided they are properly supported on your Windows command line. If you can view the file name correctly from the command line then it should work for rdiff-backup. If not, you may need to add some additional options to Windows... Maybe you don't have sufficient permissions to write to the destination (g:/_rdiffBAK)? Especially likely if it is a network drive. I would not use rdiff-backup to backup Windows c:, I regard rdiff-backup (for Windows) as a 'data' backup tool not a 'system' backup tool. I would focus on backing up %USERPROFILE% with some exclusions to prevent backup of temporary data. I would also suggest that it is easier for ongoing maintenance point of view to have several smaller repositories rather than one huge one, even if this means running rdiff-backup more than once to complete a backup session. In general, rdiff-backup works well *from* Windows, but not so well *to* Windows. If the above suggestions don't help, consider backup to a Linux-based machine - for instance with TimeDicer, which I wrote and which uses rdiff-backup. Dominic TimeDicer: Free File Recovery from Whenever On 06/11/2012 09:13, Slavko Kocjancic wrote: Hello... I tryed to use rdiff-backup in Windows XP. (In my ubuntu works prefectly) When I start backup I got WindowsError: [Error 5] Acess denied in destination name/path. And other problem seems to be long file names and foregin characters in filename. How to proceed. (I realy like rdiff backup reverse diff scheme and can't find some other similar software for backup in win) Slavko... Here is terminal output: G:\g:/_rdiff/rdiff-backup --no-acls -v4 --include-globbing-filelist list.txt c: g:/_rdiffBAK ... ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Please help getting my backups running.
Hi Gary Are you running rdiff-backup as root? Even if you are, backing up mysql databases needs some extra work to ensure that you get a consistent backup. For my linux machines I have mysql databases (and all my data) on LVM volumes, so I briefly lock the mysql server, take a snapshot of the volume (which happens quickly) and unlock the mysql server, and then I can take a backup from the snapshot: # ... simplified code clip ... MYVG="myvg" # name of LVM volume group MYLV="mylv" # name of LVM logical volume # lock mysql, create snapshot, unlock mysql mysql --execute="FLUSH TABLES WITH READ LOCK;" lvcreate -L528M -s -n "$MYLV-snap" "/dev/$MYVG/$MYLV" mysql --execute="UNLOCK TABLES;" # mount snapshot mkdir -p "/mnt/$MYLV-snap" mount "/dev/$MYVG/$MYLV-snap" "/mnt/$MYLV-snap" # backup $MYLV-snap rdiff-backup "/mnt/$MYLV-snap" ... # unmount and delete snapshot umount "/mnt/$MYLV-snap" rm -r "/mnt/$MYLV-snap" lvremove -f "/dev/$MYVG/$MYLV-snap" Without LVM you can't use snapshots, but you could lock the mysql databases and then run rdiff-backup and unlock afterwards. But the mysql databases would remain locked while the rdiff-backup operation is running - which may not be ideal. If you google 'rdiff-backup mysql' you will see a few other possible ways to backup mysql databases (e.g. using mysqldump). BTW, I find it very strange that no Linux filesystems have built-in stable snapshot support except by putting them on top of LVM, which is usually a bespoke configuration - Windows has the advantage here. Dominic On 06/11/2012 14:26, Gary Rickert wrote: Hello again Dominic, One more quick (I hope) question you may have an idea of how to resolve. I am trying to rdiff- the mysql .ibd's, and I get a permissions error. I have tried adding my rdiff user to the mysql group, and was already in sudo, but that did not resolve. The error I am receiving follows: Exception '[Errno 13] Permission denied: '/var/lib/mysql/PFA_production/*.ibd'' raised of class 'exceptions.OSError': File "/usr/lib64/python2.4/site-packages/rdiff_backup/Main.py", line 304, in error_check_Main try: Main(arglist) File "/usr/lib64/python2.4/site-packages/rdiff_backup/Main.py", line 321, in Main rps = map(SetConnections.cmdpair2rp, cmdpairs) File "/usr/lib64/python2.4/site-packages/rdiff_backup/SetConnections.py", line 78, in cmdpair2rp return rpath.RPath(conn, filename).normalize() File "/usr/lib64/python2.4/site-packages/rdiff_backup/rpath.py", line 884, in __init__ else: self.setdata() File "/usr/lib64/python2.4/site-packages/rdiff_backup/rpath.py", line 908, in setdata self.data = ""> File "/usr/lib64/python2.4/site-packages/rdiff_backup/connection.py", line 450, in __call__ return apply(self.connection.reval, (self.name,) + args) File "/usr/lib64/python2.4/site-packages/rdiff_backup/connection.py", line 370, in reval if isinstance(result, Exception): raise result Traceback (most recent call last): File "/usr/bin/rdiff-backup", line 30, in ? rdiff_backup.Main.error_check_Main(sys.argv[1:]) File "/usr/lib64/python2.4/site-packages/rdiff_backup/Main.py", line 304, in error_check_Main try: Main(arglist) File "/usr/lib64/python2.4/site-packages/rdiff_backup/Main.py", line 321, in Main rps = map(SetConnections.cmdpair2rp, cmdpairs) File "/usr/lib64/python2.4/site-packages/rdiff_backup/SetConnections.py", line 78, in cmdpair2rp return rpath.RPath(conn, filename).normalize() File "/usr/lib64/python2.4/site-packages/rdiff_backup/rpath.py", line 884, in __init__ else: self.setdata() File "/usr/lib64/python2.4/site-packages/rdiff_backup/rpath.py", line 908, in setdata self.data = ""> File "/usr/lib64/python2.4/site-packages/rdiff_backup/connection.py", line 450, in __call__ return apply(self.connection.reval, (self.name,) + args) File "/usr/lib64/python2.4/site-packages/rdiff_backup/connection.py", line 370, in reval if isinstance(result, Exception): raise result OSError: [Errno 13] Permission denied: '/var/lib/mysql/PFA_production/*.ibd' Fatal Error: Lost connection to the remote system On Fri, Nov 2, 2012 at 10:48 AM, Dominic Raferd domi...@timedicer.co.uk
Re: [rdiff-backup-users] WindowsError: [Error 5]
On 06/11/2012 14:53, Slavko Kocjancic wrote: On 11/06/2012 12:28 PM, Dominic Raferd wrote: I would not use rdiff-backup to backup Windows c:, I regard rdiff-backup (for Windows) as a 'data' backup tool not a 'system' backup tool. I would focus on backing up %USERPROFILE% with some exclusions to prevent backup It's meant to do data backup and not system. Maybe I just used wrong parameters? for backup I just executed the batch file with this content cd c:\ gf:/_rdiff/rdiff-backup --no-acls -v4 --include-globbing-filelist list.txt c: g:/_rdiffBAK @IF %ERRORLEVEL% EQU 0 (Echo Koncal brez napak) ELSE (Echo Koncal z NAPAKAMI!!!) @pause and list.txt have: c:/aaa/ c:/aaa_fotke/ c:/Documents and Settings/Slavko/Application Data/Thunderbird/ - ** And seem's to have problem with filenames longer than 256 characters too! I suggest you run rdiff-backup 3 times (having first created the backup subdirectories and made sure Thunderbird is not running): gf:/_rdiff/rdiff-backup --no-acls -v4 c:/aaa g:/_rdiffBAK/aaa gf:/_rdiff/rdiff-backup --no-acls -v4 c:/aaa_fotke g:/_rdiffBAK/aaa_fotke gf:/_rdiff/rdiff-backup --no-acls -v4 "c:/Documents and Settings/Slavko/Application Data/Thunderbird/" g:/_rdiffBAK/tbird The problem with filenames longer than 256 characters is a limitation of Windows, not a fault in rdiff-backup. You may be able to work around it - some info here. Dominic -- TimeDicer: Free File Recovery from Whenever ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] WindowsError: [Error 5]
On 07/11/2012 14:43, Slavko Kocjancic wrote:
I tryed just one directory with few dirs under and afret few backups got
same problem.
cd c:/_Skeni
f:/_rdiff/rdiff-backup -v6 c:/_Skeni f:/_Skeni_Rdiff/
snip...snip
Regular copying ('SP1950', 'SP1950_5PRA_revizija_01', 'packet.ico') to
f:/_Skeni
_Rdiff/rdiff-backup-data/increments/SP1950/SP1950_5PRA_revizija_01/packet.ico.20
12-11-07T07;05851;05847+01;05800.snapshot.gz
Processing changed file SP1950/SP1950_5PRA_revizija_01/temp
Regular copying ('SP1950', 'SP1950_5PRA_revizija_01', 'temp') to
f:/_Skeni_Rdiff
/SP1950/SP1950_5PRA_revizija_01/rdiff-backup.tmp.25243
Incrementing mirror file f:/_Skeni_Rdiff/SP1950/SP1950_5PRA_revizija_01/temp
Making directory
f:/_Skeni_Rdiff/rdiff-backup-data/increments/SP1950/SP1950_5PRA
_revizija_01/temp
Processing changed file SP1950/SP1950_5PRA_revizija_02
Removing directory f:/_Skeni_Rdiff/SP1950/SP1950_5PRA_revizija_01/temp
Removing directory f:/_Skeni_Rdiff/SP1950/SP1950_5PRA_revizija_01
Exception '[Error 5] Dostop je zavrnjen:
'f:/_Skeni_Rdiff/SP1950/SP1950_5PRA_rev
izija_01'' raised of class 'type 'exceptions.WindowsError'':
File "rdiff_backup\Main.pyc", line 304, in error_check_Main
File "rdiff_backup\Main.pyc", line 324, in Main
File "rdiff_backup\Main.pyc", line 280, in take_action
File "rdiff_backup\Main.pyc", line 343, in Backup
File "rdiff_backup\backup.pyc", line 51, in Mirror_and_increment
File "rdiff_backup\backup.pyc", line 243, in patch_and_increment
File "rdiff_backup\rorpiter.pyc", line 277, in __call__
File "rdiff_backup\rorpiter.pyc", line 229, in finish_branches
File "rdiff_backup\backup.pyc", line 672, in end_process
File "rdiff_backup\rpath.pyc", line 993, in rmdir
Traceback (most recent call last):
File "rdiff-backup", line 30, in module
File "rdiff_backup\Main.pyc", line 304, in error_check_Main
File "rdiff_backup\Main.pyc", line 324, in Main
File "rdiff_backup\Main.pyc", line 280, in take_action
File "rdiff_backup\Main.pyc", line 343, in Backup
File "rdiff_backup\backup.pyc", line 51, in Mirror_and_increment
File "rdiff_backup\backup.pyc", line 243, in patch_and_increment
File "rdiff_backup\rorpiter.pyc", line 277, in __call__
File "rdiff_backup\rorpiter.pyc", line 229, in finish_branches
File "rdiff_backup\backup.pyc", line 672, in end_process
File "rdiff_backup\rpath.pyc", line 993, in rmdir
WindowsError: [Error 5] Dostop je zavrnjen:
'f:/_Skeni_Rdiff/SP1950/SP1950_5PRA_
revizija_01'
The problem seems to be that it is unable to remove a subdirectory
on your destination USB drive. Try running it using a different
destination on drive C:, as a test, and see if that works okay.
Are you running rdiff-backup under Cygwin? If so you might try the
rdiff-backup.exe program instead...
Dominic
--
TimeDicer: Free
File Recovery from Whenever
___
rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Failling to get started
Hi Luc Why not just use this command: rdiff-backup /cygdrive/t/data luc@jana::/smb/backup/armand If this doesn't work, maybe there is a problem running rdiff-backup to a samba-mounted volume on the rdiff-backup server? Try backing up to e.g. luc@jana::/home/armand/backup Also if the destination is Windows-based this can cause problems - generally rdiff-backup is more reliable backing up to Linux. I use the rdiff-backup.exe program (which doesn't require Cygwin, though I do use Cygwin too) for backups from Windows. Dominic -- TimeDicer: Free File Recovery from Whenever On 05/02/2013 07:45, Luc Saffre wrote: Hi, as a beginner, I'm probably missing something... I'm trying to backup data under t:\data from my cygwin windows machine (called "armand") to a Debian server (called "jana"). I created a file `rdiff-backup-to-jana.sh` with one line: rdiff-backup --include-globbing-filelist include.lst \ /cygdrive luc@jana::/smb/backup/armand Another file `include.lst` contains two lines: /cygdrive/t/data/** - ** I run it like this from a Windows command prompt: T:\data\pathbash rdiff-backup-to-jana.sh /usr/lib/python2.6/site-packages/rdiff_backup/SetConnections.py:148: DeprecationWarning: os.popen2 is deprecated. Use the subprocess module. stdin, stdout = os.popen2(remote_cmd) luc@jana's password: T:\data\path That is, it asks for a password, then works some time and everything seems okay. On jana it has created a directory /smb/backup/armand containing one subdirectory rdiff-backup-data. But I cannot believe that this is a backup of my data because it is only 68KB luc@jana:/smb/backup$ du -h armand 4.0Karmand/rdiff-backup-data/increments 64K armand/rdiff-backup-data 68K armand luc@jana:/smb/backup$ I guess that it is just the extra reverse diffs. But where is the copy of my data? What am I missing? Thanks in advance for any hints. Luc ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] New user questions.
Hello Jason and welcome to rdiff-backup! On 06/02/2013 20:07, Jason Sauders wrote: Hello! I'm a new user to rdiff-backup and have been experimenting with it for the better part of two days now. It was recommended to me by another user who wondered why I was using rsync exclusively for backing my systems up. Considering rdiff-backup seems to keep an rsync-esque mirror of current data + old version archives it stands for good reason as to why this user questioned my current backup procedures. I've since done a lot of reading on rdiff-backup, and so far I like what I see. On the main pagehereI see that there's a notation saying:A remote incremental backup of all your files could be as easy as "rdiff-backup / host.net::/target-dir" This is interesting, to say the least. I do have to wonder, even when you take into account what the disclaimer says about recommended excludes, is it actually "safe" to run rdiff-backup against a currently running root file system? Is it accurate to say that I can SSH into my Ubuntu Server 12.04.1 install and as root run rdiff-backup --exclude /media / /media/external_hdd and presto, my root drive is backed up (while ignoring media) and I can run rdiff-backup -r on a totally different HDD and bingo - I can be up and running? I just want to make sure I understand the context before I try something that might be detrimental. I have never tried that, I have always used rdiff-backup for data backup, not for systems. The versioning that rdiff-backup offers is particularly valuable for data files that change over time. I kinda doubt it would work 'out of the box', too. Question 2... rdiff-backup has to be installed on both the client and the server... I get that... but is it important that they be the same identical version? Will I run into issues if I'm trying to rdiff-backup an Ubuntu 12.10 systems to an Ubuntu 12.04 server? It is advisable to have the same version, in practice you should use 1.2.8 (as I do) or 1.3.3. I am not sure if they will work together but why make it difficult by trying? Question 3... I see that there hasn't been much development on rdiff-backup for a few years now. Is that due to the 70 updated releases fixing all of the bugs? Is that to say that the current version is rock solid stable? The fact that there's an experimental version that is seemingly untouched is food for thought to contradict that, though... Development of rdiff-backup is dead because the previous maintainer went away and no one took over. Both 1.2.8 (stable) and 1.3.3 (unstable) are in practice stable and effective in many scenarios. There are situations where problems can arise, many of them involving a Windows destination, which are best avoided (you can backup data *from* Windows). Last question - With my backup procedure, I only wanted to back up Documents and Pictures to my server. Since rdiff-backup includes everything by default, it appears as if the only way to make it work was to run --exclude '**' as part of the command. Is that exclude entry essentially telling the command "Exclude everything EXCEPT these two includes Documents + Pictures"? Or am I misunderstanding it? I just didn't see too much highlighted info about --exclude '**' so I wanted to run that by this list and see what you folks thought. I'm not clear whether you mean you want to back up everything in your Documents and Pictures folders, or just some particular file types (docx, jpg etc)? If you just want to backup particular folders I suggest you create separate repositories for each folder. The --exclude-globbing-filelist option might help you too. -- TimeDicer: Free File Recovery from Whenever ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Have rdiff-backup not preserve permissions?
Thanks Dave for reporting back that solution, so we all learned something! Dominic On 22/03/2013 22:29, Dave Potts wrote: Hi, Problem solved! Dominic identified the problem correctly, but not the cure. The thing that fixed my problems was editing the Cygwin /etc/fstab file as advised here: http://stackoverflow.com/questions/5828037/cygwin-sets-file-permission-to-000 I've written up the full details here if you are interested: http://dadhacker.blogspot.co.uk/2013/03/getting-rdiff-backup-to-work-with.html Thanks for the help! Dave. On 21 March 2013 17:18, Dominic Raferd domi...@timedicer.co.uk mailto:domi...@timedicer.co.uk wrote: Which version of windows? There is a known bug in Cygwin which causes big problem under Windows 8 (but also exists under Windows 7) by which the 'group' of the cygwin user is set incorrectly. To fix this for any future files: - Look up the group ID of the Users group in /etc/group: cat /etc/group|egrep '^Users:'|cut -f3 -d':' - Edit your /etc/passwd file. Locate the record for your user. The 4th colon-delimited field is the primary group for the user, incorrectly set to a non-existent group. Change that number to the number you found above and save the file. - Close the cygwin terminal and reopen. Create a new file. It should have group Users and you should be able to change its permissions as desired. To correct existing files with wrong group settings use a recursive chgrp like this (for all files in user's home): chgrp -R Users ~ It might help? Dominic On 19/03/2013 22:31, Dave Potts wrote: Hi, I've exactly the same problems as Timothy Stella. I'm running cygwin on windows to debain linux. The first backup works fine. The second fails with permission issues on the directory I'm backing up to. Does anyone have a solution? Thanks, Dave. === From: Timothy Stella Subject: [rdiff-backup-users] Have rdiff-backup not preserve permissions? Date: Fri, 1 Mar 2013 09:57:28 -0800 Hello, I'm currently having a problem with rdiff-backup backing up some data. It's running into really weird permissions issues. I am using rdiff-backup 1.2.8 on CentOS pulling from rdiff-backup 1.2.8 on Cygwin (winxp) like so: backup@ centos$rdiff-backup -v6 address@hidden::/cygdrive/c/files/ /mnt/backups/winserver/files/ I have it backing up a directory -- the first time the backup runs, everything is fine. The second time, however, I get permission issues with the local copy (the directory I am backing up to). I tried to remedy this by adding a chmod -R u+rwx at the end of my script, but it doesn't seem to help. I also tried to set the directory to 777 just to see if it'd work, and it still fails. However, running the backup as root does work. It is also worth noting that other directories (separate rdiff-backup runs) for that same server are working without issue. I tried to look up if I can have rdiff-backup just ignore the remote (in this case, winserver) permissions but it doesn't seem possible. Is there any way for me to get around this other than running that one directory as root? Thanks! ___ rdiff-backup-users mailing list atrdiff-backup-us...@nongnu.org mailto:rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL:http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki -- *TimeDicer* http://www.timedicer.co.uk: Free File Recovery from Whenever ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org mailto:rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki -- *TimeDicer* http://www.timedicer.co.uk: Free File Recovery from Whenever ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Maintenance
Ned, I think it would be great it you wanted to be maintainer, if you have the time to do it. Andrew went off to pastures new a while ago and isn't seen around here now, so the project has been unmaintained for a while. I might be able to get you an email address for him though. Failing that I guess you could create a fork. There is a wish-list somewhere (quite a long and old one I'm afraid). But it might be easier to look back at bug-related issues that have arisen on the mailing list over the last couple of years. Or I dare say if you ask here you will get suggestions (too many maybe!) Some of the recurrent issues relate to backups to/on Windows filesystems or saving/recovering Windows ACLs. However rdiff-backup is stable in my experience for backups *from* Windows onto Linux fs and back to Windows, ignoring ACLs. I am not sure about how well it works on Macs these days, but I haven't seen many issues raised here... One serious failing in rdiff-backup is its verification procedures which are rather inadequate. My suggestion would be to focus on fundamental bug issues and functionality rather than 'front-end' because others can always wrap rdiff-backup with tools that make it easy to use. The bigger problem is if it is, or is perceived to be, unreliable. I created and maintain TimeDicer (http://www.timedicer.co.uk) which is a (free) front-end for rdiff-backup on Windows (among other things it handles snapshots), but I don't do python so I can't help with coding - sorry. Someone else looked at the rdiff-backup codebase a while back and said it was very untidy and repetitious and they lost interest in updating it. Just warning you before you get stuck in! But you would be performing a great service... Regards Dominic On 28/04/13 14:35, Edward Ned Harvey wrote: Hey everyone - Andrew here? Anybody else a current maintainer? I'd like to offer some assistance. While I don't have a lot of time, I'm very good as a sysadmin, and pretty good at python and C. (Currently a half-time IT person, and half-time C# / .Net developer, and in a past life, have had roles as fulltime java and python developer.) The very first thing I'd like to address is the broken links on the rdiff-backup page. Does a wiki still exist? If so, I'd like to correct that link. And if not, remove it. I'd like to correct the link to the mailing list. After that, I'd like to discuss moving off CVS, perhaps go to svn or git. I would bias toward svn just cuz it's simpler, and this is a very low activity project... Git would win hands-down if it were a huge rapidly developing project with a lot of maintainers. But if you want to use git, I can go along with that, and for that matter ... If you really don't want to get off CVS, so be it. But I want to move to subversion. I would like to add two really simple features: #1 Ability to store comments, when you send a backup. Similar to a versioning system commit message. I presume the argument syntax would be like -m message goes here (Heck, it would be nice to store a list of changed files in each backup rev too, or create the ability to easily display differences from rev X to rev Y, but that might be a little tougher. Create the --show-log option and --diff to compare two backups against each other.) #2 A config file that allows user to specify named backup sets. So you don't need to specify the complete source destination path every time you want to send a newly updated backup. You just say something like: sudo rdiff-backup etc -m 'updated ssl certs due to expiration date' ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki -- *TimeDicer - Windows Backup and File Recovery from Whenever http://www.timedicer.co.uk/* ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Maintenance
Ned, re the verification issue, I suggest you look back in mailing list archives to a post entitled 'Verify times increasing' by Daniel Miller on 20 November 2009, and the follow-up posts. He then started a thread 'Implementing new features' on 3 February 2010 and then a thread 'New feature: --verify-full' on 11 February 2010. Lastly a thread 'Restarting development ... or starting over' on 5 April 2010. Daniel coded a modified version of rdiff-backup which allowed full subsequent verification, but then started a new project to match and improve on the features of rdiff-backup which AFAIK was never published. You might be able to reach him for more info. I know nothing about the internal coding of rdiff-backup, I'm afraid, maybe someone else here does? Dominic On 28/04/2013 18:34, Edward Ned Harvey wrote: From: Dominic Raferd [mailto:domi...@timedicer.co.uk] I might be able to get you an email address for him though. Failing that I guess you could create a fork. Thanks, I was able to reach Ben Escoto, who gave me an address for Andrew @ princeton.edu. Ben has long since been a non-maintainer, but he at least still has admin access to the project, so if Andrew proves to be unavailable for some reason, we should be able to at least able to resurrect access without being forced into the fork. That was only today, so the fact that I don't have a response yet is irrelevant. The fact that I don't have a bounce yet is highly relevant. ;-) Still, if you've got another address for Andrew, that would be appreciated. There is a wish-list somewhere (quite a long and old one I'm afraid). That's the thing about volunteer effort. People are motivated to work on whatever they care about. ;-) Some of the recurrent issues relate to backups to/on Windows filesystems or saving/recovering Windows ACLs. Unfortunately, I think I'm unlikely to focus much on the windows side of things... I personally pay $17 to goodsync once every couple of years and I'm happy with that for windows. One serious failing in rdiff-backup is its verification procedures which are rather inadequate. That does sound important. Could you expand? (I haven't dug into source much yet.) Someone else looked at the rdiff-backup codebase a while back and said it was very untidy and repetitious and they lost interest in updating it. Just warning you before you get stuck in! But you would be performing a great service... Yeah, in the quick look I had already at the source code, there seemed to be a lot of indirection ... Which is confusing without a map, but if there's a good block diagram illustrating the interfaces between the zillion tiny little classes etc, that goes a long way toward making it all clear. -- TimeDicer: Free File Recovery from Whenever ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Is rdiff-backup outdated?
Hello Ans, I have been using rdiff-backup for about 4 1/2 years and have found it very stable and reliable. I created and maintain a Windows wrapper for rdiff-backup called TimeDicer http://www.timedicer.co.uk/index and of course use it (and therefore rdiff-backup) every day. There are two versions of rdiff-backup in common use: 1.2.8 is the 'stable' version (which I use), 1.3.3 is 'unstable' but is actually stable too by all accounts. No significant difference in functionality I think. I have repositories (archives) going back 4 1/2 years and can retrieve versions of files that have since changed and been backed up daily going back to the beginning of that time. Although I have never needed to do that, I have on several occasions needed to recover files from the past (i.e. earlier than the most recent backup) and rdiff-backup has delivered the goods and been a life-saver (OK not quite literally). I wrote a page about backup technologies when I was - as you now are - researching them, and it might help you - here: http://www.timedicer.co.uk/finding_a_backup_solution Weaknesses of rdiff-backup? 1. It was not designed with security in mind - indeed the most recent backup is stored 'in the clear' - a related tool 'duplicity' (which however uses forward deltas instead of reverse deltas) addresses this if you need it. 2. Some reports of problems when backing up *to* a Windows file system destination, and there are some unconfirmed problems with storing Windows ACLs; however I have found it totally reliable for non-ACL Windows file backup to Linux machine/file system (and recovery therefrom). 3. It lacks a really robust and usable verification system (it does allow verification but I understand it does not do a 100% job, which is a bit of an oxymoron) - however my experience is that it backup repositories are reliable nevertheless. For TimeDicer I wrote a script http://www.timedicer.co.uk/programs/help/timedicer-verify.sh.php which can do 100% verification (by repeated runs of rdiff-backup with --verify-at-time). 4. Not great over unstable connections (e.g. internet) - it works but is not recommended (repeated failures might lead to repository corruption). I always run rdiff-backup to a destination on our local LAN (via TimeDicer) and then use rsync to backup the repositories offsite (via timedicer-mirror http://www.timedicer.co.uk/programs/help/timedicer-mirror.sh.php). 5. Although you can remove 'old history' for all files in a repository (e.g. all backups older than six months, say), there is no easy way to remove specific files or directories from a backup repository without removing the whole repository. Because all previous history is retained, if you inadvertently backup a source which contains data you don't want to keep (and might bloat the backup), it is no good just correcting the backup for next time, because rdiff-backup will keep the unwanted files in its 'history'. Workaround is to regress the whole repository back to the time before the mistake occurred (losing all subsequent changes) and then resume backups with the now-corrected source specification. I wrote a script http://www.timedicer.co.uk/programs/help/rdiff-backup-regress.sh.php which can help with this. 6. No maintainer - fixed! Thanks Ned! In summary, rdiff-backup is not outdated and remains a fantastic and practical tool - and it is free and open-source. HTH Dominic On 13/05/13 21:41, Ans Alghamdi wrote: Hi, I just find it a bit odd that this powerful tool does not have any bug fix (if there are any) since 2009! So is it outdated, or its so powerful that there is no need for any update? I've already posted this quastion on serverfault at: http://serverfault.com/questions/507259/is-rdiff-backup-outdated? (P.S. The reason behind this is that I'm looking for a backup tool. rdiff-backup really suits my needs but the lack of bug fixes/active development is what is keeping me away) Best Ans ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki -- *TimeDicer - Windows Backup and File Recovery from Whenever http://www.timedicer.co.uk/* ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Curly braces in file names
Kevin:
Just to rule something out, try running it via a script, this way
you can get rid of the sudo on the command line. I have known cases
where sudo (under Ubuntu) causes strange behaviour with wildcards,
not involving rdiff-backup but it is worth checking.
So save the command line without sudo as a file, make it executable
and run it with sudo. At least it will rule out one possible
cause...
Dominic
On 17/05/2013 10:21, KP wrote:
I neglected to include the Python error messages, see below.
My assumption about braces stemmed from advice read in other threads, that the last pathname prior to the error spill is generally worth checking out.
No other pathnames with curly braces were found in the log file, which contains about 1300 entries prior to the error.
Regards,
Kevin Prichard
...
On May 17, 2013, at 2:10 AM, KP wrote:
Hello all,
I am trying to use rdiff-backup. Client is OS X 10.7.5, using v 1.2.8. Backup host is NAS4Free, accessed via AFP, with ZFS raidz2 storage.
The trouble occurs when backing up /. rdiff-backup stops on the following pathname-
Applications/Adobe/AdobePatchFiles/ZipExceptions/{6944077E-F929-4772-B00E-E96C49B55DBA}/030d3aadcd37ad6606cf8a1e71ba150e
Is there a cure for this? I searched extensively before mailing the list, and saw other discussions regarding non-ASCII chars -- but didn't find one on curly braces (ASCII).
I'm invoking with the following-
$ sudo rdiff-backup --preserve-numerical-ids --include-special-files \
--exclude-other-filesystems --exclude-sockets --include-symbolic-links \
--exclude '/proc/*' --exclude '/cores/*' --exclude '/sys/*' --exclude '/tmp/*' \
--exclude '/.DocumentRevisions-V100/*' --exclude '/.Spotlight-V100/*' \
--exclude '/Users/kev/PicturesNew/*' --exclude '/Volumes/*' -v5 \
--exclude '/.Trashes/*' --exclude '/.file/*' --exclude '/.fseventsd/*' \
--exclude '/.vol/*' \
--include-regexp '[0-9a-zA-Z-_\.\(\){} \[\]]+' --exclude-regexp '[.]+' \
/ /Volumes/backups/main/ /tmp/rdiff7.log 21
Also, a sanity check to ensure ZFS filenames allow '{}' did work.
Regards,
Kevin Prichard
--
TimeDicer: Free
File Recovery from Whenever
___
rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Curly braces in file names
I'm sorry Kevin I don't take OS X (or any Apple sauce) so I'm out of
suggestions...
Dominic
On 17/05/2013 20:09, KP wrote:
Dominic,
I added--exclude-regexp '[{}]+' after checking Spotlight to
see if any vital files would be affected (not at the moment, but
future files named with{}may be). That worked, for the narrow
test case of /Applications/Adobe.
Then I modified the backup script with that exclude, to
target the root again, and upped -v to 9. It skipped the {}
file fine, but then a new problem: it hits a symlink and
terminates (on /Applications/Adobe Bridge CS6/Adobe Bridge
CS6.app/Contents/Frameworks/AdobeACE.framework/AdobeACE ).
After reading other rdiff-backup-users threads on symlinks,
I double-checked the timestamps of the symlink and its target
file- identical,so it's not the future timestamp problem
mentioned elsewhere.
Is there a cure for this symlink problem?
Kevin
--
TimeDicer: Free
File Recovery from Whenever
___
rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Offsite mirror of backup
Paul Why not backup all the rdiff-backup data too? I use my TimeDicer package to do same as you and then I use timedicer-mirror (http://www.timedicer.co.uk/programs/help/timedicer-mirror.sh.php) to backup the fileserver (Primary TimeDicer Server) to offsite (Mirror TimeDicer Server) each night. It wakes up the remote machine, runs the mirroring, if (and only if) it completes ok it converts the data on the remote machine to match that on the source and then it shuts down the remote machine. For an archive set of 300GB it transfers typically about 450MB and takes 2 hours. You are of course welcome to look at the code of timedicer-mirror and see if it can be adapted to fit your situation. For offsite backup it uses rsync with these options (among others): -a --fuzzy --delete-after -z --link-dest. link-dest is used to protects the destination from a failed session. You could add --exclude option to ensure that the rdiff-backup-data directory is excluded. I don't think it contains any important info if you are only backing up data files. If you are backing up ACLs or system data I am less sure. Best and safest IMO is to include the rdiff-backup-data directory in the offsite backup. Dominic On 22/05/2013 22:27, Paul Novotny wrote: I run rdiff-backup on my desktop to my fileserver. I would like to move a full-backup offsite every month or so. Instead of doing this directly from my desktop, it would be nice to do this from the fileserver that contains the rdiff-backup files, but I don't want the increments to be part of this. I could just ignore the rdiff-backup-data directory, but I think some of those files contain useful stuff like file ownership and permissions. This is useful information if my filesever went down and all I had was the offsite backup. So what should files do I need to include in this offsite mirror for rdiff-backup to do a full restore? And what files pertain to the increments and I can ignore. Or is it not that simple? -Paul ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki -- TimeDicer: Free File Recovery from Whenever ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Information
Bonjour Ibrahim rdiff-backup does not have any built-in encryption. Also based on my experience rdiff-backup should not be run directly over the internet, it should be run on a lan (or of course a single machine) and then the rdiff-backup repository (archive) mirrored over the internet using rsync. This works very well for maintaining offsite data backup and the internet transmission is secured with ssh. This approach is used by my TimeDicer package (http://www.timedicer.co.uk). However this assumes that both source and destination are secure. It is possible to wrap rdiff-backup in some encryption e.g. with encfs or ecryptfs. Another option is to look at project duplicity http://duplicity.nongnu.org/. Duplicity grew out of rdiff-backup, however: -rdiff-backup's archives are meant to be as easy to view as possible, while duplicity's are as hard to view as possible and can be encrypted with GnuPG - duplicity saves data in the more conventional full+forward delta format instead of rdiff-backup's mirror+reverse deltas - rdiff-backup requires another copy of rdiff-backup on the remote destination, while duplicity currently supports local file storage, scp/ssh, ftp, rsync, HSI, WebDAV, Tahoe-LAFS, and Amazon S3 Note: the use of forward delta/diff format by duplicity means that recovering data from the most recent incremental backup requires a complete undamaged set of incremental backups from the original full backup until the most recent date; in order to reduce this dependency and the associated risk many duplicity users carry out new full backups every month or so, but this means you lose all your older backups (or if you retain them and create a parallel new archive you lose most of the advantage of the delta storage). rdiff-backup, by contrast, stores the most recent backup 'in the clear' so it is always easy to retrieve (not even requiring the use of rdiff-backup itself), and the backup increments apply in reverse order, so most recent backups are inherently safer and older backups more vulnerable. Dominic On 04/06/2013 19:25, Ibrahim Dembele wrote: Good morning Sir, I am working on a project which consist to use rdiff-backup to make backup of data on line. So looking on internet i don't find anyway if rdiff-backup can make encryption component of data send Reason why i contact you, thank you for yor help. Cordialement, ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Fatal Error: Lost connection to the remote system
Are you running this over the internet? I wouldn't advise this precisely because of these sorts of problems. rdiff-backup really needs a reliable connection, rsync is better for backup over the internet. If you need the extra functionality of rdiff-backup, either run rdiff-backup locally on the remote system (or its lan) and then rsync the resulting repository to your laptop, or rsync the data from remote to your laptop and then use rdiff-backup locally on your laptop. I do the former. Quoting Grant emailgr...@gmail.com: I'm trying to back up from a remote system to my laptop but I get: Previous backup seems to have failed, regressing destination now. And then after a few hours: Write failed: Broken pipe Fatal Error: Lost connection to the remote system My connection to the remote system seems fine the entire time and during this process there seems to be long periods of time with no data being transferred between my laptop and the remote system and no disk activity on my laptop's backup disk which seems strange. How can I get this working again? - Grant ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Re: [rdiff-backup-users] Fatal Error: Lost connection to the remote system
The best source for info about rdiff-backup is presently this mailing list! The online documentation is quite old, although we now have a new maintainer in Ned so things are looking up. Yes, you should be able to run rdiff-backup locally on the server with --check-destination-dir to regress the archive (repository) to a previous stable condition. If this doesn't work I have a script at http://www.timedicer.co.uk/programs/help/rdiff-backup-regress.sh.php which forces it to happen, but I doubt you will need this (it is mostly for situations where you want to regress an undamaged archive). Dominic http://www.timedicer.co.uk Quoting Grant emailgr...@gmail.com: Thank you, I didn't realize that was a best practice. Is this documented anywhere? Is there any way to execute the regressing destination now operation on the server without involving the client? - Grant Are you running this over the internet? I wouldn't advise this precisely because of these sorts of problems. rdiff-backup really needs a reliable connection, rsync is better for backup over the internet. If you need the extra functionality of rdiff-backup, either run rdiff-backup locally on the remote system (or its lan) and then rsync the resulting repository to your laptop, or rsync the data from remote to your laptop and then use rdiff-backup locally on your laptop. I do the former. I'm trying to back up from a remote system to my laptop but I get: Previous backup seems to have failed, regressing destination now. And then after a few hours: Write failed: Broken pipe Fatal Error: Lost connection to the remote system My connection to the remote system seems fine the entire time and during this process there seems to be long periods of time with no data being transferred between my laptop and the remote system and no disk activity on my laptop's backup disk which seems strange. How can I get this working again? - Grant ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki ___ rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
