--relative clobbers --delete
Hi, We back up a lot of machines with varying version of rsync installed on them. The oldest machine (hopefully soon to be retired) is running rsync version 2.5.7 (protocol version 26). We back this machine up from / onwards with a few excludes in there. I noticed today that whenever we use the --relative and --delete flags together, the --delete flag seems to have no affect. This only happens when when we are backing up /. Backing up other paths like /tmp etc, appear to work just fine. I tested the second oldest machine, which is running rsync version 2.6.2 (protocol version 28), and this issue does not appear. Tests on recent versions of rsync (3.0 and greater) also show that the issue does not appear there either. Does anyone remember this issue (google and the rsync bug database do not appear to)? If so, do I need to worry about it with any rsync version 2.5.7 and 2.6.2? Does it magically reappear anywhere else?? Thank you, ..Ch:W.. -- An idea does not gain truth as it gains followers. Amanda Bloom -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --relative clobbers --delete
On Tue, 2 Aug 2011, Kevin Korb wrote: 1. Update rsync. You are MANY years out of date. Updating rsync is not going to break whatever obsolete other stuff you are running. Not an option in this case, but thank you for the suggestion anyway :) 2. If you are rsyncing from / --relative doesn't do anything. True, but normally it does not hurt to specify it. When everything is scripted and / can be changed to /foo/bar quite easily, it is one less thing to remember to add --relative to the list of arguments. 3. --delete is aborted if almost any error is encountered. Check the full rsync output for any error or warning. If you run without - --verbose, --progress, --itemize-changes, and --stats anything rsync says will be an error that must be corrected before --delete will work right. We use those arguments and there are never any warnings or errors. I really think this may be some obscure issue with 2.5.7 and I am hoping that it may jog someone's memory. The device will be gone soon anyway and my interest is only where the problem may crop up elsewhere. That is all. ..Ch:W.. -- An idea does not gain truth as it gains followers. Amanda Bloom -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Multi-Threading?
On Tue, 17 May 2011, cac...@quantum-sci.com wrote: In researching this I find that a change to multi-threaded goodness would require a massive rewrite, and would only be considered for an rsync replacement. Abstracting the core functionality into a librsync.so would be really spiffy too... ..Ch:W.. -- An idea does not gain truth as it gains followers. Amanda Bloom -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Multi-Threading?
On Tue, 17 May 2011, Matt McCutchen wrote: On Tue, 2011-05-17 at 10:54 -0700, Chuck Wolber wrote: Abstracting the core functionality into a librsync.so would be really spiffy too... All easy to say, harder to do (and maintain). I'm thankful that rsync meets my needs right now, and that Wayne has gotten as many bugs out of it as he has over the last few years. Agreed. I mostly meant that as tongue-in-cheek. Creating librsync.so seems about as hard (or harder) as multi-threading. ..Ch:W.. -- An idea does not gain truth as it gains followers. Amanda Bloom -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: compression of source and target files
On Fri, 21 Sep 2007, Kenneth Simpson wrote: Hi - there's a flag for rsync to compress the files in transit - is it possible to compress one side (target) with gzip and have rsync still work correctly? It'll still work correctly, but compressing a compressed file can actually make it slightly bigger and wastes CPU cycles in the process. ..Chuck.. -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The idea that any one of us [presidential candidates] can bring about this change is a fantasy, it is not the truth! We need you to bring about the change on all these issues, we need you involved, we need you taking responsibility! --John Edwards -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Access Database
One of our servers is a CentOS 4 Linux/Samba filesharing server that serves an access database (via a mapped drive) to a client application running on Windows 98. Ever since we started rsync'ing it to back it up every night, it required samba to be restarted after the backup or the clients had trouble accessing the database. Recently though, the client application is complaining when more than one client attempts to connect to the database. I suspect all of this is related somehow. Is there some sort of lethal interaction between samba, rsync and access databases that I am blissfully ignorant of? I thought rsync only opened a target file for reading (which should only update the atime), which shouldn't affect what's going on in samba-land. ..Chuck.. -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR 09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0 -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Compression error?
On Thu, 19 Jul 2007, WCSL wrote: Seems to work fine without compression -z flag omitted. Is this going through a Watchguard firewall at any point? ..Chuck.. -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR 09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0 -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rysnc Schedule
On Wed, 18 Jul 2007, sparty2809 wrote: What I want to do is use rsync to back up a folder to an external drive. I would like to keep 30 days worth. For example: I have backups of June 1 - June 30. Once July 1 comes along, I want to keep June 2 - July 1 and delete June 1, and so forth. Any ideas how I can accomplish this? Do the backups with rsync and write a shell script that runs every night that deletes the 31st (and greater) backup. ..Chuck.. -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR 09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0 -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rysnc Schedule
On Wed, 18 Jul 2007, sparty2809 wrote: I understand how to do the rsync part, but how would I delete the files? In linux, you delete files with the rm command... In windows you'd probably want to use deltree... Or, you could get really fancy and use directed charges of thermite, although that might make it difficult to re-use portions of the disk... ..Chuck.. -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR 09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0 -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--hard-links performance
Have the hard-links optimizations that were described here been implemented? http://lists.samba.org/archive/rsync/2004-January/008137.html In any case, what's the general consensus behind using the --hard-links option on large (100GB and above) images? Does it still use a ton of memory? Or has that situation been alleviated? ..Chuck.. -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR 09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0 -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --hard-links performance
On Tue, 5 Jun 2007, Paul Slootman wrote: In any case, what's the general consensus behind using the --hard-links option on large (100GB and above) images? Does it still use a ton of memory? Or has that situation been alleviated? The size of the filesystem isn't relevant, the number of hard-linked files is. It still uses a certain amount of memory for each hard-linked file, but the situation is a lot better than with earlier rsync versions. (As always, make sure you use the newest version.) In our case, we store images as hardlinks and would like an easy way to migrate images from one backup server to another. We currently do it with a script that does a combination of rsync'ing and cp -al. Our layout is similar to: image_dir | -- img1 | -- img2 (~99% hardlinked to img1) | -- img3 (~99% hardlinked to img2) . . . ` -- imgN (~99% hardlinked to img(N-1)) Each image in image_dir is hundreds of thousands of files. It seems to me that even a small amount of memory for each hardlinked file is going to clobber even the most stout of machines (at least by 2007 standards) if I tried a wholesale rsync of image_dir using --hard-links. No? If so, then is a hard link rich environment an assumption that can be used to make an optimization of some sort? ..Chuck.. -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR 09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0 -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: overflow error
On Fri, 4 May 2007, Wayne Davison wrote: On Tue, Apr 03, 2007 at 02:22:46PM -0500, David Harfst wrote: overflow: flags=0x69 l1=110 12=1633837871 lastname= ERROR: buffer overflow in receive_file_entry rsync error: error allocating core memory buffers (code 22) at util.c(147) That's indicates that something in the protocol got corrupted. You might try upgrading to 2.6.9. If you're using -z (--compress), try leaving that option off and see if the error goes away. We had that exact same thing happen after a customer upgraded their checkpoint firewall. We were never able to figure out what the firewall upgrade versions were, but turning off -z (--compress) fixed the problem. We were using rsync 2.6.2 at the time. We no longer back up this customer's data so are unable to determine if a newer version resolves the problem. ..Chuck.. -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR 09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0 -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: tcl/tk rsync wrapper planned
On Tue, 17 Oct 2006, [EMAIL PROTECTED] wrote: we did some research and we would like to try to write a Tcl/Tk wrapper around rsync. Any chance you'd be willing to consider doing it in Java? There's already at least one, now unmaintained, Java class wrapper to rsync that I know of (not to be confused with the Java implementation of rsync, also currently unmaintained). You'd be able to pick up where they left off. I would consider helping on a project like that. ..Chuck.. -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: tcl/tk rsync wrapper planned
On Tue, 17 Oct 2006, [EMAIL PROTECTED] wrote: Any chance you'd be willing to consider doing it in Java? There's already at least one, now unmaintained, Java class wrapper to rsync that I know of (not to be confused with the Java implementation of rsync, also currently unmaintained). You'd be able to pick up where they left off. I would consider helping on a project like that. I spent some time on the Java and Tcl IRC channels today and both agree that Tcl/Tk is better (lighter, faster) for this specific purpose. Sure. Given a specific purpose, it's always easier to dial in certain variables. But the main goal is to have the tool and not to do it in a specific language (as long as the language serves the purpose) - so if there is more collaboration around the Java idea - sure! I've always wanted to have an api like that, but I've always been wary of chasing protocol changes. A wrapper is the simplest way, but it's still a bad approach IMHO. librsync is sort of a good idea too, but the idea still chases the protocols, and it looks pretty dead to boot. How do we get rsync itself to expose a language independent programmable API that's cleaner than the fork/exec/commandline model? What would such a thing even look like? ..Chuck.. -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: tcl/tk rsync wrapper planned
On Tue, 17 Oct 2006, [EMAIL PROTECTED] wrote: Chuck Wolber wrote: I've always wanted to have an api like that, but I've always been wary of chasing protocol changes. A wrapper is the simplest way, but it's still a bad approach IMHO. librsync is sort of a good idea too, but the idea still chases the protocols, and it looks pretty dead to boot. How do we get rsync itself to expose a language independent programmable API that's cleaner than the fork/exec/commandline model? What would such a thing even look like? Actually what we thought was to call rsync on the command line. I do not know if this is considered bad style. But it is better than replicating the rsync know-how into another piece of code - I would consider this as madness. I don't consider calling rsync from the commandline a clean approach, but I do consider it the only realistic option currently. I would never consider replicating rsync know-how in another piece of code. That's what I mean when I say chasing protocols. It's a waste of time. The rsync client has a hard enough time as it is syncing protocols with itself. We are thinking more about GUI wrapper which is nice for end-users and is specifically created for the service we mentioned in the beginning - easy creation of remote backup services. This means - somebody with server architecture can simply give this application to their clients (may be with hard coded server address) - so the only thing they need to do is to choose what to backup and to be able to browse the backup in order to restore. Is this different from what you initially imagined? Yes, completely different. Good luck! And yes, I agree that a Tcl/tk implementation is probably the right way to go for what you're doing. ..Chuck.. -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: backup system files (and permissions) over ssh
On Thu, 12 Oct 2006, [EMAIL PROTECTED] wrote: Quote: Without root: Use a non-root account and use passwordless sudo in the script. sounds interesting, can you explain how am i suppose to do that ? who runs the script ? when ? Check the man page for the sudoers file: man 5 sudoers. Specifically you're looking for the NOPASSWD tag. From there, you'd be able to use the sudo -c (along with some other arguments) to run your script as root. ..Chuck.. -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Cygwin rsync to RH rsync server
On Fri, 23 Jun 2006, Jason Staudenmayer wrote: So you have it working with cygwin(svr) to cygwin(cli) transfers but not linux(svr) to cygwin(cli) No, we have it working where we have to set up a tunnel (via ssh) to the cygwin machine and then set up rsync as a daemon with paths pre-specified in rsyncd.conf. We want it to work where we can connect to the cygwin sshd service from a linux machine and start rsyncing an arbitrary path. ..Chuck.. -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Cygwin rsync to RH rsync server
On Tue, 20 Jun 2006, Jason Staudenmayer wrote: Ok, so it's a Cygwin issue. Might I ask if anyone on the list has a working cygwin/rsync combo they might be able to share. I've tried downloading some old cygwin versions but so far nothing has been working. We've gotten it to work, but only if the Cygwin side is serving up rsync shares (through rsyncd.conf). We still can't aribtrarily connect to an sshd running under cygwin and sync according to a path we specify. ..Chuck.. -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Data Encryption
On Mon, 12 Jun 2006, Brad Farrell wrote: Is there a way with rsync to encrypt data at the source before transmitting? Not talking about the actually transmission, but the data itself. I've got a few department heads that want their data secured before it leaves their computer so that no one in the office can access the data except for them. The -e ssh actually encrypts the data and everything else *BEFORE* it leaves their computer. Would that not work for you? ..Chuck.. -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Deleted Data
Greetings, When using the --delete argument, is there any way to get rsync to tell me how much data it deleted after a run? ..Chuck.. -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: proposed new parameter: --time-limit 5:30
On Wed, 15 Jun 2005, ches wrote: I find that rsync is an excellent tool for backing up my large partitions, often over slower links on the Internet. I run a cron job every night to update backup mirrors. But occasionally I make large changes to the source disk, and the rsync update takes more thanb 24 ours to complete. Then I can have two or more rsyncs working over the same source and destination directories. What I really want is to tell rsync to desist the mirroring after a certain amount of time. It's ok with me if it takes a few nights to bring the mirrors back into alignment. That may be out of scope for rsync, but that's just me. I solved that problem by obtaining an flock() on a file in /var/lock. If a process starts up and can't attain the lock, it quietly dies (or does whatever I want it to do). Some people have tried goofy stuff like creating a lockfile and then checking for its existance. Aside from race conditions, if your process dies prematurely, it never gets deleted. I managed to write a perl library to manage my flock'ing. I can post it if anyone is interested. ..Chuck.. -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--one-file-system problem
rsync commandline: /usr/bin/rsync -e /usr/bin/ssh --archive --compress --sparse --verbose --stats --delete --numeric-ids --partial --relative --one-file-system target.host:/ /destination/path/ target rsync version: 2.6.3 destination rsync version: 2.6.2 The server we're trying to synchronize contains directories within / that are mounted to other locations within /. When the sync occurs, the mounted directories get copied, despite the fact that we use the --one-file-system argument. Is this a bug, or have I misunderstood the use of the --one-file-system argument? I can wrap my mind around the fact that the mounted directory is actually a part of the filesystem that it is mounted to, and thus can't be divorced from the concept of one file system. If that's truly the case, is it worth my time to come up with a --really-one-file-system patch? -Chuck -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Inflate Error?
On Tue, 7 Dec 2004, Wayne Davison wrote: On Mon, Dec 06, 2004 at 09:38:14AM -0800, Chuck Wolber wrote: inflate (token) returned -5 You should be able to avoid this error by turning off the --compress option. Yup, that solved it. To figure out why it is occurring would require someone to debug a failure case. If you can narrow it down to a particular set of files that always causes the error and make them available, it would allow some debugging to occur. I haven't seen any compression failures in recent versions, so I can't debug it without seeing a failure. I understand. The files are long gone, so we'll have to let this fall into the yeah I heard about that a while ago category. FWIW: The files in question were sleepycat (bdb) data files. This only started ocurring when the customer upgraded from a Watchguard Firebox III 700 to a Firebox X 1000. _According to them_, the software revision (7.2) did not change across the hardware upgrade. -Chuck -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync seems to hang when --delete option is used.
On Thu, 9 Dec 2004, Tony Shum wrote: When I use --delete option, it seems to hang for over 8 hours on 4.5Gb of total data. But when I remove the --delete option, I can see it starts rsyncing. Has anyone encounter this problem before? What type of filesystem is your data on (ext3, xfs, etc)? Is it on an NFS mount or is it local? Regardless of where it is, is the storage medium a single disk or a disk cluster (RAID)? -Chuck -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Inflate Error?
I've been getting this error message for a few days from a customer's server and we can't seem to decipher it. I googled for it and only found references to it with respect to much older versions of rsync. We're running the Debian packaged version of rsync version 2.6.2-3. The customer side is running 2.6.3 compiled from source. inflate (token) returned -5 rsync error: error in rsync protocol data stream (code 12) at token.c(487) rsync: connection unexpectedly closed (31461294 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(343) Command exited with non-zero status 12 Timing information on the run: real 875.26 user 45.04 sys 66.38 Exact (redacted) commandline used: /usr/bin/time -p /usr/bin/rsync -e /usr/bin/ssh --archive --compress --sparse --stats --delete --numeric-ids --partial --exclude /dev --exclude /proc --exclude /sys --exclude /mnt remote.server.org:/ /local/destination/path Does anyone have any insights? -Chuck -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Dynamic bwlimit
It would be useful to be able to send a signal to an active rsync process to get it to throttle (up or down) the bwlimit parameter. Is this a pandora's box, or is it useful for an interested party to put some effort into this? -Chuck -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Re: rsync 2.6.2 hang (was rsync 2.6.2 crash)
On Tue, 28 Sep 2004, jim wrote: Hey, that's pretty funny. Actually, I would much prefer a Linux environment, but I don't always get what I want. We are primarily a Windows and HP-UX shop, so it's not so easy get the powers that be to buy into a Linux box. Bummer, I feel your pain. As for your halting problem, that's a known issue. We solved it by pulling from (Li|U)n(u|i)x rather than pushing from Windows. Not sure if that's an option for you. -Chuck -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Anyone syncing database files?
On Fri, 20 Aug 2004, Loukinas, Jeremy wrote: Anyone syncing say 18-20gb Oracle dbf files..? We're syncing well over 185 GB of sleepycat BDB files. I'm not sure how well they relate to Oracle files. -Chuck -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Help: rsync queries
On Fri, 16 Jul 2004, Swapnil wrote: 1. I want to know whether rsync server and client are available for * Novell servers * Windows 2003 * Windows 2000 server * Windows NT * Windows XP * Xservers * Mac OS X * Mac OS 9 I am unaware about Novell, but I do know that you can run rsync under Windows by using Cygwin. As for Mac OS X, it's Unix based, so grab the source and compile. I am unsure about Mac OS 9. 5. I want to know If rsync Server is receiving the data of around 1 GB every day what is harware configuration recommended for the better performance. We maintain just under 1TB per server with 20 or 30 backup targets and don't see the load average (admittedly a poor measure of performance) exceed 4.0. 7. For backing data across different clients on different platform (as I mentioned earlier) do we need data to be backup should be availble on shares(NFS/CIFS) ? or what rsync supports ? There are some weird hang bugs during certain scenarios when backing up a Windows client to a Linux server. We found (and so have others) that using a push, rather than a pull seems to clear up the problem. -Chuck -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Help: rsync queries
On Fri, 16 Jul 2004, Chuck Wolber wrote: 5. I want to know If rsync Server is receiving the data of around 1 GB every day what is harware configuration recommended for the better performance. We maintain just under 1TB per server with 20 or 30 backup targets and don't see the load average (admittedly a poor measure of performance) exceed 4.0. Sorry, I doubt that the above is terribly useful without a rundown of what's under the hood. Our servers are all single processor Athlon XP boxes. Speeds range from 1700+ to 2200+. RAM is usually in the neighborhood of 768MB to 1GB. -Chuck -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Berkley DB
We are rsync'ing large (hundreds of GB) and constantly changing Berkley DB (aka Sleepycat) datasets (the RPM database uses the same thing, but its dataset is extremely small). When a change occurs (insert, update, delete, etc) in a BDB it has a tendency to propagate through the binary database files such that rsync has to re-download a great quantity of old data (much like the recent example of why you shouldn't gzip large files before rsync'ing them). Are there any known methods for making rsync backups of these databases more efficient? -Chuck -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Backup script
On Thu, 8 Jul 2004, spivkid wrote: I guess I need a full copy of the data that is need to be backup from the remote box on my local box so rsync can compare the file different each date? Not really. Having no data on the remote box means that there's more to transfer across from the source box. In other words, if rsync doesn't find a file to compare with, it transfers the file in its entirety. -Chuck -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Realtime rsyncing
On Mon, 5 Jul 2004, Henry van der Beek wrote: Hi, I am trying to synchronise two directories on separate machines across a network using rsync, but I want to apply rsync whenever the contents of the directories are altered, rather than using a cron job. Does anyone have any ideas? We are running Redhat9.0. Thanks Henry You might want to look at dnotify: http://www.student.lu.se/~nbi98oli/dnotify.html It's likely that rsync isn't the right tool for the job you're describing IMHO. -Chuck -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: faq-o-matic gone
On Thu, 27 May 2004, John Taylor wrote: If someone were to start maintaining a FAQ, I might be interested in helping out. Also, I really hate faq-o-matic. I like the plain ole flat file FAQ in html. Amen! -Chuck -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
New Error
I've never seen this one before. I'm using rsync 2.6.2 on both ends. Server (Athlon, 768MB RAM) checks out just fine. The only anomoly appears to be that this process has been running for 248268.43 seconds (this is not unexpected with this particular host we're backing up). Commandline: /usr/bin/time -p /usr/bin/rsync -e /usr/bin/ssh --archive --compress --sparse --stats --delete --numeric-ids --partial--exclude /proc --exclude /mnt xxx.example.com:/ /storage/xxx.example.com/ 12 Error message: rsync: writefd_unbuffered failed to write 4092 bytes: phase unknown: Broken pipe rsync: connection unexpectedly closed (7054382421 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(836) rsync error: error in rsync protocol data stream (code 12) at io.c(342) Command exited with non-zero status 12 real 248268.43 user 464.11 sys 124.48 -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Would this patch be useful...
On Fri, 14 May 2004, Wayne Davison wrote: On Tue, May 11, 2004 at 11:41:02AM -0400, John Taylor wrote: I am considering a patch that would exclude compressing a given list of file extensions when the -z option is used. Well, since the daemon code has this, it makes sense to at least consider adding this to the normal remote-shell code. I'm not sure how much this option speeds things up -- have you done any measuring? An addition to this might be to reference /etc/mime-magic rather than rely on a list to determine known formats that don't compress efficiently with -z. -Chuck -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync and Perl programming
On Wed, 5 May 2004, Jim Salter wrote: Hi everybody - I'm trying to write a Perl wrapper for some rsync tasks that need doing. Problem is, there's some sort of odd interaction going on between Perl and the daemon mode communication for the rsync client, and I'm at my wit's end in trying to figure it out. Try unbuffering your I/O in the script. Add the following line somewhere towards the top: $| = 1; Note: that's the pipe symbol after the dollar sign. -Chuck -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Feature Request
Just built 2.6.1 and started testing it. Nice job guys. I especially love the --progress and hardlink tweaks. Quite often, while I have systems backing up out of cron, I'd love to be able to see the --progress. Unfortunately, it's just not practical to crank up the verbosity like that on a regular basis (some of these systems have a few million files). A neat feature I'd like to see is the ability to attach to a FIFO or something like that, that would (while the process is running) allow me to get that information. I'm sure that begs the question, why not just redirect to a file and parse out what you want when the backup is done? First of all, that's sort of ugly. What I have in mind is something a lot simpler. How about something *ROUGHLY* analagous to this: # cat /proc/$PID/prev (4643, 13.2% of 1027477) # cat /proc/$PID/curr usr/local/rep/3ECBF9879808410/dictdb_3ECBFB89878410OX.db # cat /proc/$PID/curr_rate 80703888 8% 65.37kB/s3:51:30 Or even better is just to update the environment, so a simple cat /proc/$PID/environ could let you parse out $PREV, $CURR and $CURR_RATE from the environment string. I'm not much of a C coder, otherwise I'd offer up a patch. Wayne *DID* ask for neato features though, so this is my 2c worth. -Chuck -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [PATCH] time limit
On Tue, 20 Apr 2004, Daemian Mack wrote: That's more or less the ugly hack referred to previously. It does work as expected, but personally, I'd rather there were a native rsync method of achieving this. Less kludgy. What might be even better is if there were a cleaner API that rsync exposed (A /proc like interface perhaps???). One-off features like this could be implemented based on this API rather than be implemented in code. -Chuck -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [PATCH] time limit
On Mon, 19 Apr 2004, Wayne Davison wrote: On Mon, Apr 19, 2004 at 04:28:08PM -0400, John Taylor wrote: I have written a patch for rsync-2.6.1pre-2 which adds a --time-limit=T option. Thanks -- I've added it to the patches dir for now. If folks think it is useful, it will eventually get added to the main codebase. I would find it useful. Consider this a vote in favor of the patch. If it gets applied to a release version, I can get rid of some ugly hacks :) -Chuck -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
link-dest
System is current Debian stable with rsync version: * rsync version 2.6.0 protocol version 27 * Capabilities: 64-bit files, socketpairs, hard links, symlinks, * batchfiles, IPv6, 64-bit system inums, 64-bit internal inums * * Modified for Debian to have --bwlimit-mod, a variation on the --bwlimit * algorithm. I have noticed that --link-dest only seems to work for me if I'm using an absolute path for DIR. None of the examples seem to show this, so I gather it's something I'm doing wrong. Here some example code: DIR='/tmp/test/0' rm -rf /tmp/test mkdir /tmp/test rsync -a /bin/ /tmp/test/0 rsync -a --link-dest=$DIR /bin/ /tmp/test/1 cd /tmp/test stat 0/ls | grep Inode stat 1/ls | grep Inode In the example above the Inodes match, meaning the right thing happened. Now, if I re-run the above code and change $DIR to a relative directory from CWD (such as ./0 or 0 if my CWD is /tmp/test) the inodes do not match up. Any ideas? -Chuck -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Full backup
On Fri, 6 Feb 2004, Thorsten Schacht wrote: Hey guys, I'd like to take a full backup of our email server. Is it possible to clone the current server (postfix, spamassassin, qpopper...) to another clean system to have it ready if the current one fails? Yes. In fact we offer that to our customers as a service, in the event of disaster or machine failure. You'll want to use chroot to run the backed up instance. -Chuck -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: AW: Full backup
On Fri, 6 Feb 2004, Thorsten Schacht wrote: Sounds goot but I do not need 'one' backup folder. Not an iso. What I actually mean is that there is a identical clone of the current mail server. The backups are made into a folder, but then you chroot into that folder and run the environment as if you booted. We have had great success with this. -Chuck -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync's Speed
During my initial download for my home directory backup, it took rsync over 6 hours to do the initial backup, but I can FTP the stuff in about 30 Mins. Is Rsync usualy this slow? I have compression turned on, and its across a 100 MB/S network, Anyone had this problem before? In this case, I believe compression is actually slowing you down. -Chuck -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Buffer Overflow?
Maybe it is time to upgrade. Yes, that's the plan. I just figured the list was interested in the message. -Chuck -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Buffer Overflow?
Just got this from our nightly backup rsync: overflow: flags=0x6e l1=99 l2=1952984691 lastname=var/www/manual/mod/mod_php4/de/function.get-exte ERROR: buffer overflow in receive_file_entry rsync error: error allocating core memory buffers (code 22) at util.c(238) Command exited with non-zero status 22 real 5.15 user 1.00 sys 0.88 Rsync version information: [EMAIL PROTECTED] root]# rsync --version rsync version 2.5.4 protocol version 26 Copyright (C) 1996-2002 by Andrew Tridgell and others http://rsync.samba.org/ Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, IPv6, 64-bit system inums, 64-bit internal inums -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync backup
I've setup SSH for auto login. It seems I can just do rsync -e ssh -aupg 10.10.10.24:/home/MYDOMAIN /home fine Do I still need /etc/rsyncd.conf on the server? Nope. I do the same thing, and I've never needed it. -Chuck -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Shorten long lines in man page options summary
One thing that's bugged me is that some of the man page lines in the options summary are longer than 79 chars and wrap onto the next line. % SNIP $ SNIP Comments? Suggestions? I second your point. It's a real pain in the butt to be in a noisy datacenter at 4am with nothing but a console when you're trying to decipher a man page that's wrapping. -Chuck -- Quantum Linux Laboratories - ACCELERATING Business with Open Technology * Education | -=^ Ad Astra Per Aspera ^=- * Integration| http://www.quantumlinux.com * Support| [EMAIL PROTECTED] You know what you get after putting 30 years into a company? You're looking at it. -Downsized CIO of a major insurance carrier. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Permissions Problems
Here's my command copied from a shell script:\ rsync --verbose --progress --stats --compress --rsh=/usr/bin/ssh --recursive --times --perms --links \ /home/* [EMAIL PROTECTED]:/remotebackups/ It looks like you don't have the --delete in there (which you should have to keep identical copies on both sides, but I don't know your situation so take that with a grain of salt), so based on that, I would guess you *might* have a full disk on the machine that you are backing up to. The script is being run from the root account. My understanding is that running as root will eliminate the permission errors, but apparently that's not so. You'll get the permission problems because a full disk will prevent you from adding anymore files onto it. -Chuck -- Quantum Linux Laboratories - ACCELERATING Business with Open Technology * Education | -=^ Ad Astra Per Aspera ^=- * Integration| http://www.quantumlinux.com * Support| [EMAIL PROTECTED] You know what you get after putting 30 years into a company? You're looking at it. -Downsized CIO of a major insurance carrier. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: partial transfer
I am attempting to use rsync to backup a Win98 laptop to a FreeBSD 4.8 backup server. I have experienced the same problem at roughly the same point in the process on two occations. The laptop contains ~2.7Gb of data. On the first attempt we received this error at 2.3Gb and on the second at 2.4Gb. rsync error: partial transfer (code 23) at main.c(575) Would love to have a full backup of the laptop without error. Any assistance greatly appreciated. Which version of rsync are you using on both sides? -Chuck -- Quantum Linux Laboratories - ACCELERATING Business with Open Technology * Education | -=^ Ad Astra Per Aspera ^=- * Integration| http://www.quantumlinux.com * Support| [EMAIL PROTECTED] You know what you get after putting 30 years into a company? You're looking at it. -Downsized CIO of a major insurance carrier. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
