rsync hangs in select on linux 2.4.5
I'm seeing the following bizarre behavior...
- rsync -av from one local fs to another local fs (local disk on both)
- rsync gets to very near completion then hangs indefinitely
- attach strace to rsync process doing the 'copy from' and
it starts up again and finishes!
This is on a linux system with kernel 2.4.5-ac9. I haven't had a chance
to try another kernel yet... I will try to do so tonight. It does sound
like a kernel select/poll bug, I supose, but I thought I'd check the
list to see if anyone has any ideas.
Here's the output of the strace... the rsync has been hung for 10 minutes.
First I'm attaching an strace to the 'copy to' process, which is just
sitting in select, then I ^C out of that and attach to the 'copy from'
process and it starts right up:
[root@nova /u]# strace -p 16354
select(7, NULL, [6], NULL, {26, 15} unfinished ...
^C
[root@nova /u]# strace -p 15409
select(2, NULL, [1], NULL, {53, 61}) = 1 (out [1], left {53, 58})
write(1, %\0\0\tusr/X11R6/lib/X11/locale/iso..., 41) = 41
select(6, [3 5], NULL, NULL, {60, 0}) = 1 (in [5], left {60, 0})
select(6, [5], NULL, NULL, {60, 0}) = 1 (in [5], left {60, 0})
read(5, $\0\0\t, 4) = 4
select(6, [5], NULL, NULL, {60, 0}) = 1 (in [5], left {60, 0})
read(5, usr/X11R6/lib/X11/locale/iso8859..., 36) = 36
select(2, NULL, [1], NULL, {60, 0}) = 1 (out [1], left {60, 0})
write(1, $\0\0\tusr/X11R6/lib/X11/locale/iso..., 40) = 40
select(6, [3 5], NULL, NULL, {60, 0}) = 1 (in [5], left {60, 0})
...more of the same... following are the last few lines before
process exits...
select(6, [5], [1], NULL, {60, 0}) = 2 (in [5], out [1], left {60, 0})
select(6, [5], NULL, NULL, {60, 0}) = 1 (in [5], left {60, 0})
read(5, \24\0\0\t, 4) = 4
select(6, [5], NULL, NULL, {60, 0}) = 1 (in [5], left {60, 0})
read(5, var/cache/man/cat7/\n, 20)= 20
select(2, NULL, [1], NULL, {60, 0}) = 1 (out [1], left {60, 0})
write(1, \24\0\0\tvar/cache/man/cat7/\n, 24) = 24
write(1, \4\0\0\7\377\377\377\377, 8 unfinished ...
--- SIGHUP (Hangup) ---
--- SIGCONT (Continued) ---
... write resumed ) = -1 ENOSYS (Function not implemented)
rt_sigaction(SIGUSR2, {SIG_IGN}, {0x8050590, [USR2], SA_RESTART|0x400}, 8)
= 0
getpid()= 15409
kill(16354, SIGUSR1)= 0
_exit(20) = ?
Any thoughts?
--jurgen
rsyncd log file option and logrotate
Greets. I'm running rsyncd out of xinetd and will be logging to /var/log/rsyncd.log. Is there anything special I need to do with rsync if I use logrotate to manage the rsyncd.log files? Do I need to 'restart' the rsyncd process if one is running and the log gets rotated? -- Regards, Scott Russell ([EMAIL PROTECTED]) Linux Technology Center, System Admin, RHCE. T/L 441-9289 / External 919-543-9289 http://bzimage.raleigh.ibm.com/webcam
Re: rsyncd log file option and logrotate
On Wed, Jun 13, 2001 at 04:03:11PM -0400, Scott Russell wrote: Greets. I'm running rsyncd out of xinetd and will be logging to /var/log/rsyncd.log. Is there anything special I need to do with rsync if I use logrotate to manage the rsyncd.log files? Do I need to 'restart' the rsyncd process if one is running and the log gets rotated? I haven't used logrotate, but modified rsync some time ago to have it close and re-open the log file on every connection, to allow the log file to be moved away when running as a stand-alone background daemon. When running from xinetd you're re-starting rsync completely on every connection so it would have worked even without my modification. Each individual connection is pretty short lived so you shouldn't have to worry about any running rsync daemon processes. Are you having a problem? - Dave Dykstra
Re: rsyncd log file option and logrotate
David - No, no problem, yet. Just trying to avoid problems. :) I think the question really boils down to what happens if the logfile rsyncd is writing to gets moved out from under it? Maybe I should just run rsyncd in standalone mode and use /sbin/kill -HUP to restart it after moving the logfile. -- Scott On Wed, Jun 13, 2001 at 03:32:49PM -0500, Dave Dykstra wrote: On Wed, Jun 13, 2001 at 04:03:11PM -0400, Scott Russell wrote: Greets. I'm running rsyncd out of xinetd and will be logging to /var/log/rsyncd.log. Is there anything special I need to do with rsync if I use logrotate to manage the rsyncd.log files? Do I need to 'restart' the rsyncd process if one is running and the log gets rotated? I haven't used logrotate, but modified rsync some time ago to have it close and re-open the log file on every connection, to allow the log file to be moved away when running as a stand-alone background daemon. When running from xinetd you're re-starting rsync completely on every connection so it would have worked even without my modification. Each individual connection is pretty short lived so you shouldn't have to worry about any running rsync daemon processes. Are you having a problem? - Dave Dykstra -- Regards, Scott Russell ([EMAIL PROTECTED]) Linux Technology Center, System Admin, RHCE. T/L 441-9289 / External 919-543-9289 http://bzimage.raleigh.ibm.com/webcam
Re: rsyncd log file option and logrotate
On Wed, Jun 13, 2001 at 04:38:52PM -0400, Scott Russell wrote: David - No, no problem, yet. Just trying to avoid problems. :) I think the question really boils down to what happens if the logfile rsyncd is writing to gets moved out from under it? If it is during an open connection, it will keep the old file open from wherever it is moved to and write out there. That's assuming it's the same filesystem; if it's a different filesystem, the move will unlink it from the old filesystem and rsync will keep the inode open until finished and the inode will go away when rsync is done. Anything rsync writes out to the log file after the move (which actually did a copy) would get lost. If the rsync daemon and the process that moves the log file are on different machines sharing over NFS, then rsync would probably get a write error because the inode would have disappeared. Maybe I should just run rsyncd in standalone mode and use /sbin/kill -HUP to restart it after moving the logfile. Why do that? A kill -HUP will not restart rsync --daemon automatically, it will just kill it. - Dave Dykstra
Re: rsyncd log file option and logrotate
On Wed, Jun 13, 2001 at 03:54:31PM -0500, Dave Dykstra wrote:
On Wed, Jun 13, 2001 at 04:38:52PM -0400, Scott Russell wrote:
David -
No, no problem, yet. Just trying to avoid problems. :) I think the question
really boils down to what happens if the logfile rsyncd is writing to gets
moved out from under it?
If it is during an open connection, it will keep the old file open from
wherever it is moved to and write out there. That's assuming it's the
same filesystem;
This is good news. Based on this a logrotate script like should work:
/var/log/rsyncd.log {
rotate 6
monthly
compress
missingok
}
--
Regards,
Scott Russell ([EMAIL PROTECTED])
Linux Technology Center, System Admin, RHCE.
T/L 441-9289 / External 919-543-9289
http://bzimage.raleigh.ibm.com/webcam
RE: librsync and network functions
Martin To tridge and I, rsync 2.x is essentially in maintenance mode at the moment: it doesn't seem like a good investment to add more features or functionality to the codebase. A better way to improve it is to build a new, cleaner protocol. Does this mean there is a version 3.x on its way? If so when is the likely release date? Cheers Mark CAUTION - This message may contain privileged and confidential information intended only for the use of the addressee named above. If you are not the intended recipient of this message you are hereby notified that any use, dissemination, distribution or reproduction of this message is prohibited. If you have received this message in error please notify Air New Zealand immediately. Any views expressed in this message are those of the individual sender and may not necessarily reflect the views of Air New Zealand. _ For more information on the Air New Zealand Group, visit us online at http://www.airnewzealand.com or http://www.ansett.com.au _
Re: rsync hangs in select on linux 2.4.5
On Wed, 13 Jun 2001, Jurgen Botz wrote: I'm seeing the following bizarre behavior... - rsync -av from one local fs to another local fs (local disk on both) - rsync gets to very near completion then hangs indefinitely - attach strace to rsync process doing the 'copy from' and it starts up again and finishes! This is the second of two hangs that my recent nohang patch fixes. The patch is relative to the CVS version, but it applies fine to version 2.4.6 as well (with offsets, but no fuzz). I have not looked at 2.4.5, though. You can get the patch from here: http://www.clari.net/~wayne/rsync-nohang.patch ..wayne..
Bogus error listing modules
The CVS version of rsync outputs an error on exit after listing the
modules from an rsync daemon (running rsync host::). The following
patch fixes this. (If you haven't applied my nohang patch, you'll see
an offset for this patch hunk.)
..wayne..
---8--8--8--8---cut here---8--8--8--8---
--- new/io.cWed Jun 13 18:45:12 2001
+++ io.cWed Jun 13 18:40:46 2001
@@ -246,12 +246,12 @@
if (n == 0) {
- if (eof_error) {
- rprintf(FERROR,
-%s: connection to server unexpectedly closed
- (%.0f bytes read so far)\n,
-RSYNC_NAME, (double)stats.total_read);
- }
+ if (!eof_error)
+ exit_cleanup(0);
+ rprintf(FERROR,
+ %s: connection to server unexpectedly closed
+(%.0f bytes read so far)\n,
+ RSYNC_NAME, (double)stats.total_read);
exit_cleanup(RERR_STREAMIO);
}
---8--8--8--8---cut here---8--8--8--8---
RE: rsync hangs in select on linux 2.4.5
Wayne Umm, errr, I have to confess that I don't know how to apply your hang patch... I did try the patch command, but to no avail. A quick explanation would be appreciated. I have: -downloaded the rsync-2.4.6 source and unpacked it. -am using Solaris 2.6. -am using Solaris compilers and tools. Also the patch you list below (link) is different to one I saved of one of your earlier emails. Is the one below the latest version of the anti hang patch? Cheers Mark -Original Message- From: Wayne Davison [mailto:[EMAIL PROTECTED]] Sent: Thursday, 14 June 2001 14:02 To: Jurgen Botz Cc: [EMAIL PROTECTED] Subject: Re: rsync hangs in select on linux 2.4.5 On Wed, 13 Jun 2001, Jurgen Botz wrote: I'm seeing the following bizarre behavior... - rsync -av from one local fs to another local fs (local disk on both) - rsync gets to very near completion then hangs indefinitely - attach strace to rsync process doing the 'copy from' and it starts up again and finishes! This is the second of two hangs that my recent nohang patch fixes. The patch is relative to the CVS version, but it applies fine to version 2.4.6 as well (with offsets, but no fuzz). I have not looked at 2.4.5, though. You can get the patch from here: http://www.clari.net/~wayne/rsync-nohang.patch ..wayne.. CAUTION - This message may contain privileged and confidential information intended only for the use of the addressee named above. If you are not the intended recipient of this message you are hereby notified that any use, dissemination, distribution or reproduction of this message is prohibited. If you have received this message in error please notify Air New Zealand immediately. Any views expressed in this message are those of the individual sender and may not necessarily reflect the views of Air New Zealand. _ For more information on the Air New Zealand Group, visit us online at http://www.airnewzealand.com or http://www.ansett.com.au _
Re: librsync and network functions
On 13 Jun 2001, Deven Phillips [EMAIL PROTECTED] wrote: Hi, I am in the process of gathing up information for the creation of an rsync/srsync io slave for KDE. I'm happy to hear you're interested in working on this. I am wondering if I could get your input on the best approach. I was thinking about taking rsync, and creating a library out of it, and then using that library to handle the network/comparison/deltas of the files; but your librsync only handles the delta. Yes, librsync does only handle the delta, and it doesn't particularly aim to be compatible with the standalone rsync program. The rsync protocol is a product of evolution, and is frankly pretty ugly. For example, there's no way to do more than one transfer, or to move files over a single network connection. Similarly, the protocol (probably) requires both parties to read the entire list of files in to memory and sort it before the transfer begins. The protocol is also fairly tightly tied to the code of the current implementation. Similarly, error handling is quite hard to get right -- the recent patch looks like a big improvement in this regard. To tridge and I, rsync 2.x is essentially in maintenance mode at the moment: it doesn't seem like a good investment to add more features or functionality to the codebase. A better way to improve it is to build a new, cleaner protocol. I'm quite interested in using the NFS4 protocol, of which we have the start of a userspace implementation at http://n4.samba.org/. This gives us a standards-based asynchronous RPC protocol that covers all the basic file operations except for actual delta and patch operations. If we compatibly extend the protocol to add those routines, and take their implementation from librsync, then we should be in good shape. Of course, this is free software so if you disagree, you're free to do something different! So, to be practical, my advice would be that the IO slave should simply popen() rsync, and keep all the code wrapped up inside the program. I think we'd certainly be open to patches that make the output format more easily parsed by programs: separating files in the listing by nul characters, and similar stuff. Essentially the protocol doesn't *let* you do anything more flexible than invoking the program to do one transfer, so I'm not sure that a library could offer an interface more flexible than that to its callers. From a brief browse through the KIO documentation it looks like this might work out. Perhaps you can write libcallrsync or something, so that it can later be reused to make a GNOME-VFS method. After thinking about it for a while, there should be libs for rsh and ssh that I could link in under the librsync for the network portions. I may sound stupid, but it has been a while for me with C/C++ (about 8 years), so please don't flame me for trying. If you have a few minutes, perhaps you can send my some ideas from the perspective of understanding rsync quite well (which I haven't had time to become familiar with yet). Hope that helps, -- Martin
