Re: rsync mirror solution: how to prevent accidental mirror deletion
Hi Rahul, I have two suggestions: 1. Use --max-delete option to limit the amount of damage that can be done when this happens. 2. Add some redundancy to your backup by keeping multiple past backups and rotating between them. You can use --link-dest to allow sharing of unchanged files between backups. A set of perl scripts called rsnapshot (http://www.rsnapshot.org/) makes this easy to set up. David On 30 July 2010 10:41, Rahul Nabar rpna...@gmail.com wrote: I had a recent disaster scenario with rsync. I was wondering if there were any suggestions to guard against in the future: I used to maintain mirror backups of the /home dir on our production_server using rsync to a backup_server. The primary server had a rsyncd daemon running and the backup_server had this line in the crontab: 10 01 * * * rsync -av --delete r...@production_server::home /production_server_home_bkup Things worked fine and whenever we tested backup_server it always had a faithful image of the production_server. Unfortunately, one day the RAID array on primary_server had a failure. As a result /home mount was lost temporarily. This happened over the weekend and by the time we got around to checking, the backup_server had a chance to run its cron job. This job merrily erased every file on the backup since it thought /home was now supposed to be empty when it synced the two. Luckily we could recover the RAID so all is well now. But how should I change our procedures to guard against this (or similar) outcomes? Is there a better way out for a mirror via rsync? -- Rahul -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
source-filter_dest-filter patch and --no-whole-file
Hi, I am looking at using the source-filter_dest-filter patch with rsync 3.0.7. The patch sets the --whole-file option (i.e. disables the rsync algorithm) when either --source-filter or --dest-filter are given. I can't see any reason why these new features won't work with --no-whole-file. I tried commenting out the code in options.c that sets --whole-file and it seemed to work ok. Does anyone know why the --whole-file requirement was added to this patch, and what I might need to watch out for if I remove it? David -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Prepared meta directory for one to many sync
Hi Luke, 2009/5/6 Lukas Macura mac...@opf.slu.cz: I know our scenario is similar to some rsync download mirror and maybe it is solved somewhere. But I did not find any solution yet. Maybe it would be great if rsync have some meta directory, where it would prepare all infos about stored files. Like hashes, precompressed files, etc. This meta directory could be created or refreshed by some options. After creating and pointing to that directory, entire mirroring process could be much more faster. Maybe: $ rsync -r --prepare-meta /tmp/meta ./ $ rsync -r --meta-dir /tmp/meta/ ./ rsync://somewhere/ Please is it theoreticaly possible ? Or is there some problem to precompress and prehash files? Have a look at --write-batch and --read-batch. David -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Disaster recovery option for file server
Hi Jeff, 2009/4/18 Jeff Boyce jbo...@meridianenv.com: Would I have to run cygwin on the Vista box? I would prefer not to if possible. Can anyone give me some basic guidance on how I might do this? All the discussions I see about using rsync is either between two Linux boxes, or from a Windows box using cygwin to a Linux server. What I am considering doing is neither of those cases. Thanks. Running rysnc on Windows requires Cygwin, but that doesn't mean you need a full Cygwin installation. The best option, IMHO, is to install the cwRsyncServer package from http://www.itefix.no/i2/node/10650. This provides just what you need to run rsync. It also makes it easy to run the rsync daemon as a Windows service. If you want to run rsync over ssh you can install the copSSH package from the same site. David -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: itemized option (-ii) with --log-file and --log-file format
Pipe the output through sed? 2009/4/6 Jignesh Shah jignesh.shah1...@gmail.com: Thanks Wayne for reply. I want to use log-file support only so that i need not to read stdout and put output into the file. I have to use -ii so that it print all the statistic info about what files/dirs synched and what are not. Please refer below output: with -i only. --- 2009/04/06 10:38:41 [27475] building file list 2009/04/06 10:38:41 [27475] sent 88 bytes received 12 bytes 200.00 bytes/sec 2009/04/06 10:38:41 [27475] total size is 1003012 speedup is 10030.12 with -ii - 2009/03/31 15:15:17 [4537] building file list 2009/03/31 15:15:29 [4537] created directory Mydocs1 2009/03/31 15:15:29 [4537] CONTENTS: cd+ Mydocs/. 4096 send 0 2009/03/31 15:15:29 [4537] CONTENTS: f+ Mydocs/1.pl 394 send 437 2009/03/31 15:15:29 [4537] CONTENTS: f+ Mydocs/CHANGELOG.txt 2588 send 2631 . . I want the output of -ii but dont want cryptic output in it (e.g. cd+ , f+ etc). If we remove it then it will become similar output as 2.6.* version output. Is it possible? Thanks, Jignesh On Thu, Apr 2, 2009 at 8:14 PM, Wayne Davison way...@samba.org wrote: On Wed, Apr 01, 2009 at 10:50:13AM +0530, Jignesh Shah wrote: I noticed that the output format for this command is different from rsync V2.6.* . Is there any way I can make this output similar to rsync V2.6.*? What difference are you referring to? The use of send instead of recv for a local transfer? That is a more accurate representation of what rsync is doing, but you can change it by forcing rsync to pull files from localhost. See the support/lsh script for a way to pull from localhost w/o using a remote shell. If that's not the issue, please elaborate. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync Over SSH (Windows XP)
Hi Henri, 2009/3/14 henri he...@stmargarets.school.nz: In essence I would like to determine if rsync a good choice when it comes to the backup of files on a Windows XP machine? Suggestions such as the following will all be warmly welcomed : (1) Do not run rsync over ssh, keep it simple. (2) Do not use rsync on Windows XP. (3) Try xyz commercial solution/service instead, it works great! (4) Use xyz open source solution it is fantastic! (4) It works but it is difficult, prepare your self for punishment. (5) No problem, just follow these instructions you will be running in 1 day. (6) It works, just make sure you install the xyz patch. (7) Not enough information, it depends on various other factors. ...etc I am aware that there is the popular Cygwin project which provides a super simple way of installing both rsync and OpenSSH on Windows systems. There's no native port of rsync on Windows so using the Cygwin port is the way to go. If you don't want a full Cygwin installation there is an excellent package called cwRsync which contains just what you need at http://www.itefix.no/i2/node/10650. The same site provides an OpenSSH package called CopSSH which is designed to work well with cwRsync. Both packages come with Windows-style installers that set up the SSH/rsync daemons as Windows services. If you want a commercial backup solution, you might be interested in BackupAssist (www.backupassist.com) which now supports backup via rsync as well as other methods. (Disclaimer: I work for the company that develops BackupAssist.) BackupAssist provides various pre-defined schemes for regular scheduled backups with history. No need to fiddle around with scripts and rsync options, just enter your server details, select a scheme then select which files/directories you want to backup. There's a 30 day free trial period if you want to try it out. Other features: - Supports rsync daemon, rsync over ssh, and rsync daemon via ssh tunnel - Backup history uses hard links for files that haven't changed between backups, similar to rsnapshot. (Our marketing department calls this single instance store.) - Support for Volume Shadow Copy service (VSS) for Windows 2003 and later (not XP unfortunately). I hope this doesn't sound too much like an ad, but you did ask about commercial solutions. Cheers, David -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --link-dest option with the destination directory containing old files.
2008/12/23 Robert Bell robert.b...@csiro.au: It seems to us that there is a good case for the enhanced functionality. When a file is found in the destination which should be replaced by one in the source, look in the --link-dest directory first for a candidate, and hard-link that in preference to doing a copy from source to destination. I would also very much like to see this feature. Indeed, this seems far more logical than the current --link-dest behaviour and it's what I assumed --link-dest would do until I read the man page thoroughly (you have to follow the references back from --link-dest to --copy-dest and then --compare-dest and even then the only mention of the actual behaviour is a parenthesised comment (if the files are missing in the destination directory)). I be interested to know what use cases the current behaviour was designed for, because I can't see any advantage to not making use of the --link-dest file if it's available. Providing the proposed alternative behaviour as an extra option, if not the default for --link-dest, would be very useful. David -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Feature I'd love to see: --move-dest
Hi, I'm curious as to why you need a new option rather than just using --link-dest. Both --link-dest and --copy-dest already allow the two very common restor scenarios that you mention and --link-dest uses hardly any extra disk space than your --move-dest suggestion. You also get the added advantage that each previous backup continues to look like a full snapshot. David 2008/12/9 Josh Hanson [EMAIL PROTECTED]: Looking over rsync's --compare-dest, --copy-dest, and --link-dest options, there's one thing I really wish I could do, and I've been looking for a few years now for a program that could do it: Instead of copying or hard-linking from the compare directory to the target directory, I'd like to move the files. That is, whenever the file already exists in the compare directory, it is moved forward into the target directory; but if the existing file is different or doesn't exist, then a new file is made in the target directory. The result would be that, after several backups (each to a new directory, perhaps named with the current date) the most recent directory would be a snapshot of the source, and each previous directory would have previous versions of files or copies of deleted files. (This is effectively the opposite result of --compare-dest, which gives you a snapshot of the first version of the files, with each successive directory storing only the changed files.) This makes two very common restore scenarios very simple: restoring the entire directory tree to the most recent version after a drive failure, and restoring a previous version of a single file after it was accidentally over-written, even if the modified version has since been backed up. I looked over the rsync source to see if I could add this functionality myself, but I'm really not a very good programmer and it's definitely beyond me. To someone familiar with the source, however, I have to imagine it would be a fairly straightforward feature to add... just add replace the copy operation in --copy-dest or the link operation in --link-dest with a move operation. -- Josh -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Looking for a Comprehensive list of error codes
Hi, You could look at the EXIT CODES section of the rsync(1) man page, e.g. at http://www.samba.org/ftp/rsync/rsync.html. However, this only details exit codes from rsync itself, it doesn't mention exit code 255 which I think is actually coming from ssh. The ssh(1) man page says that ssh will return 255 if an error occurred and I have seen this exit code returned by rsync when there is an ssh error. Can anyone confirm whether rsync is actually passing through the error code from ssh, or if there is some other undocumented reason for it returning 255? Are there any other codes that might be returned that aren't in the man page? David 2008/11/19 John Macon [EMAIL PROTECTED]: Good Afternoon, I have been looking for a comprehensive list of error codes that are returned by rsync for a script that I am writing. I have searched Ggoogle and Yahoo! and I have found out the following: 4 = action not supported 5 = wrong password 10 = socket IO error 11 = file IO error 12 = broken connection 23 = file(s) couldn't be transferred 255 = unspecified error Is there anywhere that I can find other error codes for rsync? I am writing a script that will upload/sync all files/folders from a particular domain to a CDN with a click of a button. I need to build as much intelligence into the script as possible and I am hoping to be able to translate all error codes to an end-user readable format so that the end-user can provide ample information if/when something goes wrong (at least more than It doesn't work). Thanks for your help. John Color coding for safety: Windows Live Hotmail alerts you to suspicious email. Sign up today. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: link-dest storage saving on backup server not on usb hard drive
Hi, On 26/06/2008, fglos [EMAIL PROTECTED] wrote: I have setup rsync to backup another linux computer with the link-dest option and realized storage savings. 1st backup is 31gb, daily backups only take approx 4gb. But when I rsync with link-dest to usb external drive all backups take up 31gb. Following is 1st the one that works and 2nd with no storage savings. Any ideas? Thanks, Frank Is it possible that the external disk is formatted with a file system that doesn't support hard links? If it is formatted with a Linux/Unix fs such as ext3 it should be ok. I think NTFS should work too. But if it is formatted for FAT32 or some variant then you cannot create hard links so there can be no saving from --link-dest. David -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
