Re: reproducible problem w/ rsync asymmetric routes
On Fri, Jun 22, 2001 at 04:09:19PM -0500, Dave Dykstra wrote: Those versions are recent enough. Try Wayne Davison's nohang patch at http://www.clari.net/~wayne/rsync-nohang.patch I just applied the patch and recompiled. No change in results. Presumably the transfer is hanging for a period of time before it times out. If Wayne's patch doesn't fix it, check the state of the TCP queues during the hang by running netstat on both sides and looking to see if there are things sitting in the send queue on one side or the other for that connection. There are 5839 bytes waiting in the SendQ on the sending side for each connection. 64.71.162.66.56108 206.26.162.146.22 6432 5839 24820 0 CLOSE_WAIT 64.71.162.66.56111 206.26.162.146.22 6432 5839 24820 0 CLOSE_WAIT 64.71.162.66.56114 206.26.162.146.22 6432 5839 24820 0 ESTABLISHED Running with rsync -ave ssh -v gets me this far: [...] debug1: Received RSA challenge from server. debug1: Sending response to host key RSA challenge. debug1: Remote: RSA authentication accepted. debug1: RSA authentication accepted by server. debug1: Sending command: rsync --server -vlogDtpr . /etc/tinydns debug1: Entering interactive session. building file list ... done [hang] I created a script /tmp/rsync, on the receiving side, which looks like this: #! /bin/sh strace /usr/bin/rsync $@ 2/tmp/rsync.log and ran truss rsync -ave ssh -v --rsync-path=/tmp/rsync /etc/tinydns/root [EMAIL PROTECTED]:/etc/tinydns 2/tmp/rsync-truss-sender.txt on the sending side. I've posted the complete truss/strace outputs at http://flounder.net/rsync-truss-sender.txt http://flounder.net/rsync-strace-receiver.txt --Adam -- Adam McKenna [EMAIL PROTECTED] | No matter how much it changes, http://flounder.net/publickey.html | technology's just a bunch of wires GPG: 17A4 11F7 5E7E C2E7 08AA| connected to a bunch of other wires. 38B0 05D0 8BF7 2C6D 110A| Joe Rogan, _NewsRadio_ 11:31pm up 19 day(s), 23:34, 8 users, load average: 0.07, 0.08, 0.07
Re: Anti-hang comments?
On Mon, 25 Jun 2001, Andrew Tridgell wrote: I've applied your simple nohang patch. Cool. That's the one that affects the most people. Instead we need a way of reproducing the bug and see if we can find a solution without a buffer. You can minimize the buffer usage by applying my move-files patch. It constantly reads the redo pipe during the generator's main loop and marks the redo items with a flag in the existing files struct (and also forwards the delete indicators on to the sender). This ensures that this buffer doesn't expand much at all. (With both patches applied I haven't seen it reallocate except when I tested the buffer code with a 16-byte realloc size.) Alternately, it might not be too hard to remove the buffer and have the low-level code take a more direct role in interpreting the data, but I'd have to look at this more closely to see for sure. One way to reproduce this hang is to modify the receiver code to redo every file that is processed in the first phase. Also, my move-files patch puts enough extra data down the sender-to-generator pipe that it should hang up without difficulty if you disable the buffer and use the --move-files option. ..wayne..
Re: 2-way rsync with delete
On Monday, June 25, 2001 03:17:18 PM -0400 Kovalev, Ivan [EMAIL PROTECTED] wrote: +-- | I am doing a poor man cluster using rsync to synchronize content of 2 | servers each of which has its own directly attached storage. Since it is a | cluster (load balancer on top of these 2 servers), new additions as well | as deletions might appear on any of the 2 servers. [...] | The problem would be solved if rsync had a way to do updates compare to | the known system state. I mean if I could direct it to replicate only the | changes that were done after time T. +-X8 What you are describing is unison http://www.cis.upenn.edu/~bcpierce/unison/. /Michael -- This space intentionally left non-blank.
Re: Not able to use server port with auth..
On 30 Jul 2001, J.Saravanan [EMAIL PROTECTED] wrote: Hi, bash$ rsync -a [EMAIL PROTECTED]::test test Password: @ERROR: auth failed on module test I did try this command, But the result is same as you can see above.. Any other suggestion?.. What is in the server error log now? -- Martin VA Linux SystemsGnuPG encrypted email preferred
Re: New protocol to avoid EOF?
On 25 Jun 2001, Wayne Davison [EMAIL PROTECTED] wrote: I was wondering if the protocol should be updated to avoid ever assuming that an EOF on the socket was OK. The only case I know of where this allowed is when we're listing modules from an rsync server. If we modified the protocol to have the daemon rsync send an EOF token (such as @RSYND: EOF) at the end of the list, this would allow the rsync client to always report an unexpected EOF as an error. I've got a patch that implements this, but because it's built on top of my last bogus-module-list-error-removal patch, I'll only work up a diff if this is something that the maintainers want to do. Yes, that sounds good. I think I applied the patch that clears up the is eof OK? flag, so please go ahead and send the new one. We have the technology to apply mismatched patches anyhow :-) virtual beer -- Martin
Re: question on switches and their order
No, you misunderstood him: he meant that --delete would delete things, not the lack of it. Of course if that's what you want then you need to put it in. Yes, your command line should work. Note, however, that when copying between two filesystems mounted on the local machine, people have seen hangs more often than remote, and Wayne Davison's nohang patch (now in the rsync CVS tree) has helped a lot of people. Also for improved performance definitely use -W to disable the rsync rolling checksum algorithm, and don't use -z. The rsync algorithm, and rsync's primary reason for existence, is to optimize the network traffic between two endpoints, but in your case the two endpoints are on the same machine as far as rsync knows. - Dave Dykstra On Mon, Jun 25, 2001 at 02:52:19PM -0700, Jeff Kennedy wrote: Rally? I had no idea that the dest would delete anything not in source. If that's the case, what is --delete for then? Thanks. ~JK Wilson, Mark - MST wrote: Leave out the --delete. It will delete everything on the destination that is not present on the source, except for .snapshot -if present. If you want to play with --delete use the -n option as well. This will tell you what rsync is going to do without actually doing anything. Suggest you try rsync -avW --exclude=.snapshot /mail-data/ /mnt Cheers Mark -Original Message- From: Jeff Kennedy [mailto:[EMAIL PROTECTED]] Sent: Tuesday, 26 June 2001 09:19 To: RSync List (E-mail) Subject: Re: question on switches and their order So, if I understand you correctly, it should work fine (with improvements) like this: rsync --exclude=.snapshot --delete -avW /mail-data/ /mnt Is this correct? Thanks. CAUTION - This message may contain privileged and confidential information intended only for the use of the addressee named above. If you are not the intended recipient of this message you are hereby notified that any use, dissemination, distribution or reproduction of this message is prohibited. If you have received this message in error please notify Air New Zealand immediately. Any views expressed in this message are those of the individual sender and may not necessarily reflect the views of Air New Zealand. _ For more information on the Air New Zealand Group, visit us online at http://www.airnewzealand.com or http://www.ansett.com.au _ -- = Jeff Kennedy Unix Administrator AMCC [EMAIL PROTECTED]
Re: Reducing rsync memory usage for file lists
Yes, that sounds like a pretty good plan for (say) rsync 3.0. We all seem to be more or less on the same track as to how the protocol should look. Here are my feelings about the way to get there. I would be happy to have holes picked in them: * rsync 2.x works well, but is too crufty to be a good base for future development. If we want to do anything other than bug fixes and small feature additions we should start with a clean slate. * Running over ssh is an excellent way of pushing the work of security and authentication onto somebody else who can do it well. Modules over raw TCP is also a very good way of exporting public or high-speed resources. Unfortunately people get a little confused at first about the differences between the two mechanisms. * The protocol should not assume the structure of the program. For example, it should be possible to send and receive files at different times across the same socket, or to hold a socket open and idle for a while waiting for human commands. * The protocol must be pipelined (or streaming, or call it what you will). Basically we must not block waiting for responses if there is something else useful we could be doing, including issuing more responses. * (This is the most controversial.) All other things being equal, it is better not to invent a new network protocol if there is an existing protocol that will do the job well. The asynchronous RPC mechanism in NFSv4 is a good foundation, and has most of what we need except for actually computing and applying deltas, which could straightforwardly be added. I am playing with a userspace NFSv4 client and server, which can already retrieve files and list directories. In a little while I might try a (very simple-minded) mirroring program built on top, to see how it works out. -- Martin VA Linux SystemsGnuPG encrypted email preferred
Re: Anti-hang comments?
On Mon, 25 Jun 2001, Andrew Tridgell wrote: see if we can find a solution without a buffer. Here's a solution with a non-growing buffer. This code keeps the receiver-generator pipe clear by reading the ints and setting redo flags in a character array (of flist-count elements). I'm avoiding setting flags in the actual flist structure since it is shared memory between 2 forked processes, and this might cause a lot of memory to become unshared (if the OS supports copy on write for fork). Since read_int() is a fairly high-level call, I had to manually ensure that a flush doesn't happen and to ensure that reading the redo_fd doesn't try to read the io_error_fd (both to avoid nested read attempts on the redo_fd). I have done some simple testing of this with my usual redo all files testing tweak and it is working fine, but the code is still pretty young. If you want to test this, be sure to unapply my previous no-hang patch or start fresh from the CVS version. The new patch is here: http://www.clari.net/~wayne/rsync-nohang2.patch I think it will also work to start from 2.4.6, but you should also apply the other no-hang fix I made (that was recently committed to CVS): http://www.clari.net/~wayne/rsync-nohang1.patch You'll need to use patch -p1 to apply the new patches (unlike the previous one, which used -p0) since I had a request for the top-level directory to be included in the file names. [FYI, I have not yet ported my move-files patch to use this code.] ..wayne..
Can rysnc place diff files in a different folder?
How can I setup rsync to do a comparison between two folders or servers and then place the files that have differences in a seperate folder? (..a folder other than the two I am comparing, need this for versioning purposes...) All suggestions welcome!! Ned Alcorn Sr. Systems Analyst Continental Airlines - HQJSZ 713-324-8733 888-785-7646
multi-rsync
Hi, I am looking for multi-rsync to transfer files from a master machines to several other machines at the same time. (ie. using broadcast mechanism.) rsync does not have this capability at this moment. (1) Is the next release going to include this capability? If yes, when is it going to be released? (2) Any idea where I can find multi-rsync at other places? hp
Re: multi-rsync
On Wed, Jun 27, 2001 at 12:30:24AM -0400, HP Wei wrote: I am looking for multi-rsync to transfer files from a master machines to several other machines at the same time. (ie. using broadcast mechanism.) The rsync+ patch may help here. I updated it to apply against rsync version 2.4.6 and posted it to this list last week. For more information ob rsync+, go to: http://www.ils.unc.edu/i2dsi/unc_rsync+.html -- Jos Backus _/ _/_/_/Santa Clara, CA _/ _/ _/ _/ _/_/_/ _/ _/ _/_/ [EMAIL PROTECTED] _/_/ _/_/_/use Std::Disclaimer;
