Re: problems encountered in 2.4.6
On Fri, May 25, 2001 at 02:19:59PM -0500, Dave Dykstra wrote: ... Use the -W option to disable the rsync algorithm. We really ought to make that the default when both the source and destination are local. I went ahead and submitted a change to the rsync CVS to automatically turn on -W when the source and destination are both on the local machine. - Dave Dykstra
Re: 2.4.6 Hang on BSDI 4.2/4.0.1
On Tue, May 29, 2001 at 11:36:11AM -0400, Tuc wrote: I don't have any experience with BSDI, but what's the full command line of the rsync command you're trying to run? /usr/local/bin/rsync -vrlptgozx --partial --delete --force --rsync-path=/usr/local/bin/rsync --exclude='stats/' --exclude='archive/' --include='*/' --blocking-io /usr/home/directory/subdirectory/ [EMAIL PROTECTED]:/usr/home/directory/subdirectory/ I don't know what to tell you. You seem be doing the right things, although --blocking-io is default if you don't set the transport command with -e to something other than the default, which is remsh if it existed at configure time, otherwise rsh. Perhaps it is a BSDI-specific problem. You could try turning off blocking-io by setting the -e command to a full path to rsh, because the test for turning on blocking-io compares exactly to the string without the full path. Can you reliably cause the problem to go away when you go back to 2.3.1 and re-appear when you go to 2.4.6? The main difference between the two versions is that a buffering hack was removed in version 2.4.0 that was supposed to only affect ssh, and instead non-blocking I/O was implemented for use with ssh. Rsync tends to greatly stress TCP implementations, and has brought to light bugs in many of them. When rsync is hung, we often ask that people run netstat on both sides to look at the TCP queues for the rsync connection. If there is data on the send queue on one or both sides while it is hung, TCP is at fault because it is TCP's responsibility to get data from the send queue on one side to the receive queue on the other. - Dave Dykstra
Re: problems encountered in 2.4.6
On Tue, May 29, 2001 at 12:02:41PM -0500, Phil Howard wrote: Dave Dykstra wrote: On Fri, May 25, 2001 at 02:19:59PM -0500, Dave Dykstra wrote: ... Use the -W option to disable the rsync algorithm. We really ought to make that the default when both the source and destination are local. I went ahead and submitted a change to the rsync CVS to automatically turn on -W when the source and destination are both on the local machine. So how do I revert that on the command line? I've been trying with -W doing my disk to disk backups, and I've had to go back to not using -W. Will -c do that? There's currently no way to revert it. I thought it wouldn't be necessary, and I'm not sure how to do it cleanly, because there's currently no precedent in rsync for a general undoing of options that have different defaults depending on the situation. Another one that comes to mind is --block-io. The latest rsync in CVS is now using the popt package to process options intead of getopt. Does anybody know if that package has a standard way to negate options, for example prefixing a no (like --no-block-io) or something like that? I took a quick look through the man page and it wasn't obvious. The reason is the load on the machine gets so high, nothing else can run. This is not CPU load, but rather, buffering/swapping load. CPU load just slows other things down. But buffering/swapping load brings other things to a grinding halt. I suspect Linux's tendency to want to keep everything that anything writes in RAM, even if that means swapping out all other processes, is impacted by this. So I'll need a way to not have the effect of -W to use rsync for disk to disk backups. Wow. Rsync is just going too fast for it I guess. The -W makes it do a lot of unnecessary disk I/O which must be enough to throttle its progress. Sure seems like leaving out -W is the wrong solution. Maybe -W has to turn off more of rsync's pipelining since it is no longer performing the rsync algorithm. The fact that rsync loads so much into VM probably makes the problem a bit worse in this case. I saw 1 process at 35M and 2 processes at 70M (total 175M used by rsync, in addition to all the buffered writes). Does -W have an impact on that? I would think that if anything -W would lessen that effect. I'm wondering if rsync is even a good choice for disk to disk backup duty. Is there some option I missed that disables pre-loading all the file names into memory? Maybe it isn't. There is no such option. I also tried the --bwlimit option and it had no effect, not even on the usual download syncronizing over a dialup that I do. I could not get it to pace the rate below the dialup speed no matter what I would specify. I haven't used the --bwlimit option and don't really know how it works. I remember when somebody contributed it that I was skeptical about how well it could work. I'm especially not surprised that it has no impact on local-to-local transfers. - Dave Dykstra
Re: unexpected EOF in read_timeout
On Wed, May 30, 2001 at 02:05:33AM -0500, Phil Howard wrote: I'm a novice at rsync, so I can't tell you all the details, but, in general, the unexpected EOF in read_timeout (usually) means that something not so good happened on the server side. On what server I was connecting to, I believe that the preprocessing by the server made some watchdog on the server side decide that the process was dead -- it then killed it, and then I got the error message. I never proved this completely. I posted a while back with this problem and someone answered that the message existed in ssh and not in rsync. I never verified it. No, that's not correct. That message is in rsync. The message that you're thinking about that is in ssh is Write failed: Cannot allocate memory and that resulted in rsync printing out the unexpected EOF message, which can come anytime the connection to the server is lost. - Dave Dykstra
Re: rsync stopped synchronizing
On Tue, May 29, 2001 at 05:30:13PM -0400, Frank Artusa wrote: I have a few web servers running Redhat 6.2 which synchronize via two scripts to an rsync server. They run every 5 and 15 minutes through a cron job. For many months this worked great until a few days ago when it just stopped working. If I run these scripts manually it works fine, which leads me to believe its a cron problem. But, cron logs report that these scripts execute fine with no errors. Even the rsync server log reports the systems establish a connection just fine. Except for a cant find /etc/pwd.db - no such file which has been in the logs for months as well (i just took over this responsibility). I am pretty new to rsync, but i have looked through more logs than I can count with no luck. By the way, lock files do not exist either. I need help to troubleshoot this problem desperately. Thanks, What is the evidence that it has stopped working? I suggest putting in an strace -f -o /tmp/somefile ahead of the rsync command in the cron entry and examing the output file to see what it is doing. - Dave
Re: File compression
On Wed, May 30, 2001 at 11:23:56AM -0400, Brian Johnson wrote: I'm using rsync to backup my main hard drive to a second hard drive. Although it is currently a local hard drive, I intend to switch to backup to a remote location. My question is: It seems that rsync can compress the data for transmission, but is there a method to leave the files compressed on the backup hard drive. I know I don't want one large zip file or a windows style compressed drive because restoring damaged data would be mcuh more difficult (possibly impossible). No there isn't, although it was talked about a long time ago. It would make rsync's timestamp and filesize comparisons tough, although somebody said that the size information at least is stored in the header of a gzipped file. - Dave Dykstra
Re: rsync exit codes
On Wed, May 30, 2001 at 10:30:07AM -0400, Neil Schellenberger wrote: Content-Description: message body text John == John L Allen Allen writes: John But I do agree it would be nice if the man page documented John them. Or, better still, if rsync itself simply told you what they meant ;-) The source currently in rsync's CVS already does this. Martin put that in. ... In concert with that, I'd also tried to ensure that all error messages always get logged somewhere (i.e. never just get lost, as could currently happen for certain server errors if the sibling is unreachable). I refactored the logging code to try to make it more clear where the messages were to be routed. In the process, it is possible that I may have unknowingly/inadvertently violated some design intentions (e.g. I seem to remember Dave Dykstra mentioning that certain server errors were not logged to the client for security reasons - I may have busted that, sorry). Anyway, here is the patch (which is totally separate from the first one: you can safely opt-in to the first and opt-out from this one). Martin Tridge will need to decide about that patch. I think Tridge will probably not want all the messages sent to the client of a daemon. - Dave Dykstra
Re: Option --one-machine-only ?
On Thu, May 31, 2001 at 03:32:27PM +0200, [EMAIL PROTECTED] wrote: This'll give you what you want without a mod to rsync. ++ #!/bin/sh mount |grep -v nfs|while read fs crud do rsync -ax $fs remotehost::remotepath/`basename $fs` done ++ I wrote it into the mail, so test it first. If such a mod WERE to be made to rsync, perhaps a -fstype option would be more appropriate. rsync -a -fstype=nonfs / remotehost::remotepath The corresponding option in find is -local. Something like this might work: find / -local | rsync -a --include-from /dev/fd/0 --exclude '*' / host:path - Dave Dykstra
Re: incomplete local directory syncronisation
On Thu, May 31, 2001 at 03:48:08PM +1000, Colin Nathan wrote: Hi all, I continue to have difficulties when attempting to synchronize between two local directories. One of them is a NetWare volume mounted using NCP. The catalogue builds and the synchronization of the two directory's starts. For a reason unknown to me, the process will stop completely when it reaches a certain file. When the file in question is excluded, the synchronization will continue on past the location of that file until it possibly reaches another file that it will not transfer and stop completely on that as well. I am unsure as to why rsync stops on these particular files, I can find nothing in common with any of them. I have gone and viewed some of the files in question, and they are fine. I started with the following: rsync -ravu --stats --exclude *.gho /mnt/bmhsnlc-s4/usr /replication/bmhsnlc-s4/ and then as suggested added the -W or --whole-file switch. This does not ? seem to make any difference to the files that will not copy. The files range in size from 8k to 52mb. The amount of data that is being sync'd in any one attempt is about 4GB. The NetWare 5 server is on the same subnet as our Linux box, and is 100Mbit switched. I am at a loss to try and explain what is occurring. Any idea's??? Do they fail when you copy just those files by themselves? I once had a circumstance where particular data inside a file could cause data to get dropped by some network element; even doing a straight rcp on the file would fail. It may help to examine an strace. I'm assuming the /mnt device is over NFS, probably UDP, so it won't even help to use netstat to examine queues or use tcpdump. - Dave Dykstra
Re: Does rsync close open files?
On Thu, May 31, 2001 at 03:29:42PM -0700, Adam McKenna wrote: Background: Our web application picks up files from a repository and processes them -- the repository is accessed by clients using SFTP. There is an rsync script running once per minute to sync up the client's data on the SFTP server with the real repository on our application servers. The Problem: Since the script runs once a minute, it is possible for the rsync to start while a file is still being transferred to us. What appears to happen at this time, is that something happens and the file doesn't finish transferring properly. So, is there any way to have rsync skip certain files, if they are not complete? There doesn't appear to be any obvious way to do this. Could the rsync process be the reason that these transfers are not finishing correctly? Thanks for any and all input. CC's would be appreciated as I am not a list subscriber -- but I will be checking the archive as well. There is really no way around that problem with rsync. Many other people have tried to do similar things and the wisdom on the mailing list has always been that rsync via cron is the wrong tool for applications that have data that changes frequently and needs to be replicated very soon after it changes. You need something more like a distributed database or a replicating filesystem, or at the very least call rsync in a synchronous manner right after a file has been changed and before the file can change again. - Dave Dykstra
Re: rsync questions
On Tue, Jun 05, 2001 at 09:25:12AM +0100, Gonçalo Gomes wrote: Dear rsync users, i'd like to ask some questions about rsync, 1) i'm going to use rsync to upload backups to backup machine, that machine will hold only backups. - does it keeps the ownership of the files if the users/groups, they don't exist in the remote machine (backup machines) or it keeps them in a database and restores them when replacing the backups ? If you run as root and the users/groups don't exist on the backup machine it will typically keep the original user id and group ids. I'm not sure what happens if you're using an rsync --daemon as the destination, though, perhaps it will just keep whatever uid is specified (defaults to nobody). 2) how to choose where a path (within the actual module name) to where upload the files without having to add a new module ? Paths are chosen either through the module name if you're using rsync --daemon mode or whatever path you specify otherwise. I usually discourage people from using --daemon mode for writing, as it was really mostly intended for reading. Ssh is the best access method. 3) the --delete options means that, if we have a dir with index.htm and main.htm in the backup server, and then we remove main.htm from the client, when we're uploading the newbackup it will remove main.htm from the backup server? Right. - Dave Dykstra
Re: New --with-rsync-name option
On Sun, Jun 10, 2001 at 03:01:26AM -0500, [EMAIL PROTECTED] wrote: Patch to set RSYNC_NAME in rsync.h. It adds a new option to configure, --with-rsync-name, that takes a string that is used to replace RSYNC_NAME. The default is rsync. I kind of hate to pre-process all of rsync.h for just that one variable. Isn't there some way to get autoheader to stick it into config.h.in? Alternatively it could go into a -D as part of @CFLAGS@ in Makefile.in but I'm not sure I care for that much either. - Dave Dykstra
Re: can't bind to port 873
On Mon, Jun 11, 2001 at 12:35:33PM -0300, Valdomiro Kornetz wrote: Hy everyone, I'm trying to put a rsync service to work, via inetd, on my FreeBSD 4.3 stable, but all I got was an (from rsyncd.log) 2001/06/11 12:09:46 [20857] rsyncd version 2.4.6 starting 2001/06/11 12:09:46 [20857] bind failed on port 873 2001/06/11 12:09:46 [20857] transfer interrupted (code 10) at socket.c(254) It's odd cause I can actually start rsync without adding it to services or inetd.conf, just by tiping rsync --daemon at the prompt. Any guesses?? You don't mention if you've eliminated the obvious guesses: Guess 1: something else is already listening on the port. Could be you had it in inetd.conf, then deleted it, but didn't send a kill signal to inetd. Check netstat. Guess 2: you're not running as root. - Dave Dykstra
Re: rsyncd log file option and logrotate
On Wed, Jun 13, 2001 at 04:03:11PM -0400, Scott Russell wrote: Greets. I'm running rsyncd out of xinetd and will be logging to /var/log/rsyncd.log. Is there anything special I need to do with rsync if I use logrotate to manage the rsyncd.log files? Do I need to 'restart' the rsyncd process if one is running and the log gets rotated? I haven't used logrotate, but modified rsync some time ago to have it close and re-open the log file on every connection, to allow the log file to be moved away when running as a stand-alone background daemon. When running from xinetd you're re-starting rsync completely on every connection so it would have worked even without my modification. Each individual connection is pretty short lived so you shouldn't have to worry about any running rsync daemon processes. Are you having a problem? - Dave Dykstra
Re: rsyncd log file option and logrotate
On Wed, Jun 13, 2001 at 04:38:52PM -0400, Scott Russell wrote: David - No, no problem, yet. Just trying to avoid problems. :) I think the question really boils down to what happens if the logfile rsyncd is writing to gets moved out from under it? If it is during an open connection, it will keep the old file open from wherever it is moved to and write out there. That's assuming it's the same filesystem; if it's a different filesystem, the move will unlink it from the old filesystem and rsync will keep the inode open until finished and the inode will go away when rsync is done. Anything rsync writes out to the log file after the move (which actually did a copy) would get lost. If the rsync daemon and the process that moves the log file are on different machines sharing over NFS, then rsync would probably get a write error because the inode would have disappeared. Maybe I should just run rsyncd in standalone mode and use /sbin/kill -HUP to restart it after moving the logfile. Why do that? A kill -HUP will not restart rsync --daemon automatically, it will just kill it. - Dave Dykstra
Re: reproducible problem w/ rsync asymmetric routes
That level of the network is completely outside of rsync's control; if you're using -e ssh, do you have problems when you use ssh by itself? - Dave Dykstra On Thu, Jun 21, 2001 at 12:29:13PM -0700, Adam McKenna wrote: I have a reproducible problem with rsync -- I have two systems, on in Fremont, CA and the other in New York, NY. I use djbdns on these systems and use rsync to move the data.cdb file between machines. For some reason, rsync is unreliable (actually, it doesn't work at all) when the routing is asymmetric between the two machines. Currently the route to my secondary machine goes directly through a peering point (NYIIX) to the other network: traceroute to athlon.flounder.net (206.26.162.146), 30 hops max, 40 byte packets 1 64.71.128.37 (64.71.128.37) 1.074 ms 0.766 ms 0.561 ms 2 gige-f6-0-0.mae-west.he.net (216.218.130.4) 1.770 ms 1.841 ms 1.988 ms 3 64.71.128.22 (64.71.128.22) 90.471 ms 101.314 ms 92.560 ms 4 nyiix.villageworld.com (198.32.160.58) 88.548 ms 88.580 ms 90.550 ms 5 fe0-0.60hudson-1.villageworld.com (204.194.106.65) 92.773 ms 88.878 ms 90.034 ms 6 206.201.222.105 (206.201.222.105) 88.809 ms 94.679 ms 90.006 ms 7 fe1-0.brentwood-1.villageworld.com (204.194.106.33) 94.097 ms 89.310 ms 95.620 ms 8 brentwood-s2-7.backbone.bohemia-1.villageworld.com (206.201.222.33) 107.999 ms 107.901 ms 108.646 ms 9 athlon.flounder.net (206.26.162.146) 110.543 ms 109.124 ms 110.888 ms But the return route goes through ATT + ABOV: ceroute to sunfish.linuxis.net (64.71.162.66), 30 hops max, 38 byte packets 1 206.26.162.145 (206.26.162.145) 0.755 ms 0.647 ms 1.107 ms 2 12.127.249.37 (12.127.249.37) 2.182 ms 2.249 ms 2.132 ms 3 gbr2-a31s1.n54ny.ip.att.net (12.127.0.18) 2.493 ms 4.017 ms 2.348 ms 4 gbr3-p00.n54ny.ip.att.net (12.122.5.246) 2.760 ms 11.442 ms 24.122 ms 5 gbr3-p10.wswdc.ip.att.net (12.122.3.54) 25.009 ms * 27.312 ms 6 * gbr1-p70.wswdc.ip.att.net (12.122.1.158) 26.233 ms 7.882 ms 7 gr1-p340.wswdc.ip.att.net (12.123.8.218) 8.873 ms 9.350 ms 8.716 ms 8 iad-att.iad.above.net (216.200.254.97) 10.856 ms 10.691 ms 10.721 ms 9 core4-core3-oc48.iad1.above.net (208.185.0.142) 17.150 ms 18.667 ms 16.500 ms 10 dca2-iad1-oc192.dca2.above.net (208.184.233.125) 11.317 ms * 52.792 ms 11 ord2-dca2-oc48.ord2.above.net (64.125.31.46) 48.723 ms 33.874 ms 31.911 ms 12 core2-core1-oc192.ord2.above.net (208.185.0.186) 31.844 ms 31.620 ms 31.553 ms 13 sjc2-ord2-oc48.sjc2.above.net (208.184.233.45) 105.576 ms 126.420 ms 106.266 ms 14 core1-core4-oc48.sjc2.above.net (208.184.102.201) 105.930 ms 105.349 ms 114.270 ms 15 core6-sjc2-oc48.sjc1.above.net (216.200.0.177) 105.258 ms 106.499 ms 107.282 ms 16 core2-sjc1-oc3.sjc6.above.net (207.126.96.106) 125.541 ms 119.512 ms 107.034 ms 17 100tx-f6-1.mae-west.he.net (207.126.96.98) 106.085 ms 109.762 ms 107.697 ms 18 gige-g0-0.gsr12008.fmt.he.net (216.218.130.6) 107.394 ms 109.165 ms 110.643 ms If Villageworld turns off the announcement of that netblock at NYIIX, the route from sunfish - athlon goes thru ABOV + ATT, and rsync works fine. If it is asymmetric (as shown above), the following happens: Read from remote host athlon.flounder.net: Connection timed out unexpected EOF in read_timeout Does anyone have any suggestions (command line flags, etc.) that I can add to my scripts to make this transfer work? Right now it is failing _every_ time. Since rsync runs over ssh which is a TCP service, it seems that even dropped packets should be corrected. I don't understand why the transfers are failing. Any ideas? --Adam
Re: configure --with-rsh=ssh
On Fri, Jun 22, 2001 at 05:26:52PM -0700, Wayne Davison wrote: On Fri, 22 Jun 2001, Dave Dykstra wrote: The default value of the non-blocking IO is not affected by this change -- instead rsync only sets non-blocking IO by default if the RSYNC_RSH value is rsh or (if remsh is around) remsh. This is a very tricky thing to do right, and probably the main reason why Martin gave up on doing it. Your patch doesn't do it right (see below). The current code is also broken, and I didn't want to try to solve the deeper problem in a patch that was just attempting to add a configure option. I think my patch is just slightly less broken in this area than what is currently in cvs, but not by much (since the man page says that rsh defaults to blocking IO, but the current code will not default to blocking IO with -e rsh if remsh is the default remote shell). I specifically left off any path-based matching so that people could specify an rsh without blocking IO. Yes, I see what you mean and now I think your way is better. I think a better method would be for rsync to have a default blocking setting for the default remote shell (perhaps configurable along with what the remote shell is), and then let the user override the proper remote shell and blocking behavior for their transport. How is that different from the way it is now? The biggest complicating factor with this is the RSYNC_RSH environment variable, since there is no way to specify the blocking mode associated with this value. For instance, if some sysadmin were to globally set RSYNC_RSH to /foo/rsh and it needed to use blocking IO, all users would be required to manually specify the --blocking-io option. Perhaps we need some kind of a config file setup like ssh uses? It would be possible to specify a different shell/blocking combination as the default, perhaps even based on hostname matching (again, like ssh supports), and the user could specify their own overrides, as needed. This would allow things like using rsh for local systems, and ssh for remote ones. Perhaps a ~/.rsync or ~/.rsync/config and /etc/rsync.conf like ssh will be useful someday to specify any default option, even host-based, but I wouldn't do it because of this issue because I don't think it will help much with solving the problem of selecting the right blocking-io setting. I don't think it's a good idea to change the default, because of the surprise factor as you change from machine to machine and because there are a number of significant differences between rsh and ssh. For me I don't even have rsh installed at home, and it is disabled at work, so an rsync that doesn't function (because it defaults to using rsh) is a bigger surprise factor for a novice user. I edited the source to use ssh as the default when I installed it at work, and will continue to do this if rsync doesn't support a configure option in the future. Good point. I now think your patch should be accepted, but it will be up to Martin or Tridge to put it in. - Dave
Rsync maintainers
For those of you who haven't been on the rsync mailing list long, you may be wondering about who maintains what parts of rsync. The history is that rsync was primarily written by Andrew Tridge Tridgell, and that a couple years ago he asked for somebody else to maintain it while he finished his PhD thesis (which was about rsync). I volunteered for about 6 months and then when he finished his thesis I turned maintainership back to him, and around the same time he began to work for Linuxcare. A while later, one of Tridge's co-workers at Linuxcare, Martin Pool, took over as the official maintainer. Tridge is still the one who knows the most about rsync, especially when it comes to networking and the main rsync algorithm. He still does some of the maintaining of rsync but he is extremely busy on a lot of projects (he's also the primary author of Samba). Tridge Martin are now both at VA Linux. I still have update access to the CVS and occasionally make changes, but only minor ones and only ones that are important to me personally and if I'm not sure if Tridge or Martin will get to them. A while back I encouraged Tridge to expand the rsync maintainer team like he did with Samba, and he was open to the idea and in fact I think one person did get access but he never did any updates. If you want to get update access to the rsync CVS, contact Tridge and Martin. Tridge's home page is http://samba.org/~tridge - Dave Dykstra
Re: 2-way rsync with delete
In general Britton is correct. The only thing that might help Ivan is the -u option, which works strictly on file modification time. He didn't mention having discovered it. - Dave Dykstra On Mon, Jun 25, 2001 at 12:01:00PM -0800, Britton wrote: I do this sort of thing between my home and work machines, but only one system changes at a time, and I do a sync in the appropriate direction at the end of each session. I think what you want may not be possible, since rsync doesn't maintain any database about the files it handles and deleted files have no date stamps associated with them. I think you need full blown distributed file system. Britton Kerin __ GNU GPL: The Source will be with you... always. On Mon, 25 Jun 2001, Kovalev, Ivan wrote: I am doing a poor man cluster using rsync to synchronize content of 2 servers each of which has its own directly attached storage. Since it is a cluster (load balancer on top of these 2 servers), new additions as well as deletions might appear on any of the 2 servers. Newly added files are replicated just fine, but I need list's wisdom on how to replicate deletions. Suppose I have a new file on server1 and it is pooling from server2 with --delete option. Then my new file will be deleted because it is not present on server2. If I do push from server1 to server2 without --delete as a first step (to make sure new file is there by the time I will pool), then I will also push old files from server1 which were deleted on server2 since last synchronization. This will effectively kill replication of deletions. The problem would be solved if rsync had a way to do updates compare to the known system state. I mean if I could direct it to replicate only the changes that were done after time T. I did not find anything like this in man pages or list's archives for last few months, but the question seem to be so obvious, that I am probably missing something. Ivan Kovalev Thomson Financial, BFMG / IBES 195 Broadway, 6-th floor New York, NY 10007-3100 [EMAIL PROTECTED] tel. (646)822-2939 fax.(646)822-2800
Re: question on switches and their order
No, you misunderstood him: he meant that --delete would delete things, not the lack of it. Of course if that's what you want then you need to put it in. Yes, your command line should work. Note, however, that when copying between two filesystems mounted on the local machine, people have seen hangs more often than remote, and Wayne Davison's nohang patch (now in the rsync CVS tree) has helped a lot of people. Also for improved performance definitely use -W to disable the rsync rolling checksum algorithm, and don't use -z. The rsync algorithm, and rsync's primary reason for existence, is to optimize the network traffic between two endpoints, but in your case the two endpoints are on the same machine as far as rsync knows. - Dave Dykstra On Mon, Jun 25, 2001 at 02:52:19PM -0700, Jeff Kennedy wrote: Rally? I had no idea that the dest would delete anything not in source. If that's the case, what is --delete for then? Thanks. ~JK Wilson, Mark - MST wrote: Leave out the --delete. It will delete everything on the destination that is not present on the source, except for .snapshot -if present. If you want to play with --delete use the -n option as well. This will tell you what rsync is going to do without actually doing anything. Suggest you try rsync -avW --exclude=.snapshot /mail-data/ /mnt Cheers Mark -Original Message- From: Jeff Kennedy [mailto:[EMAIL PROTECTED]] Sent: Tuesday, 26 June 2001 09:19 To: RSync List (E-mail) Subject: Re: question on switches and their order So, if I understand you correctly, it should work fine (with improvements) like this: rsync --exclude=.snapshot --delete -avW /mail-data/ /mnt Is this correct? Thanks. CAUTION - This message may contain privileged and confidential information intended only for the use of the addressee named above. If you are not the intended recipient of this message you are hereby notified that any use, dissemination, distribution or reproduction of this message is prohibited. If you have received this message in error please notify Air New Zealand immediately. Any views expressed in this message are those of the individual sender and may not necessarily reflect the views of Air New Zealand. _ For more information on the Air New Zealand Group, visit us online at http://www.airnewzealand.com or http://www.ansett.com.au _ -- = Jeff Kennedy Unix Administrator AMCC [EMAIL PROTECTED]
Re: Rsync problem
On Mon, Jul 02, 2001 at 02:37:00PM +0800, Ganbaa wrote: Dear Sir, I'm using rsync over ssh between 2 hosts. Sometimes getting following error message: system /usr/local/bin/rsync -av --timeout=0 -e ssh user@host:/usr/home/user/backup.tar.gz /usr/home/user/ failed: 3072; What is it mean? Can you help me? Best regards Ganbaa It looks like that message is coming from the surrounding software that is calling rsync, not rsync itself. There is no message that looks like that in rsync source code. What is invoking rsync? - Dave Dykstra
Re: rsync FAQ
On Wed, Jul 04, 2001 at 03:35:00PM +1000, Martin Pool wrote: On 28 Jun 2001, Kevin Faust [EMAIL PROTECTED] wrote: I am running rsync on lots of Win2K machines. It was built from the rsync 2.4.6 sources using Cygwin. One issue, the rsyncd.conf file requires a use chroot=false for the deamon to run on Windows. The 'use chroot' flag is a fairly reliable source of confusion, because it fails under two important circumstances: running as non-root, running on Win32 (or possibly others). On the other hand, if you are running as root, then it might be better to always use chroot. So, perhaps if we can't chroot we should just issue a warning, unless a require chroot flag is set, in which case we fail. I implemented the use chroot = false option (before that it always used to use chroot no matter what), and I don't think it's a good idea to add yet another rsyncd.conf flag because I think that would only increase confusion. We could change the default on Win32 to true. We could also change it to true if not running as root on Unix, but I would hesitate to do so because I don't have complete confidence in the robustness of the security checking when running without chroot. It would be fine to add to the error message about chroot failing to suggest to people to look at the use chroot = false option; in fact, I thought that's what it was doing but I don't see it in there and I went back into the cvsweb archive and it looks like it never was there. - Dave
Re: Anti-hang comments?
On Thu, Jul 05, 2001 at 12:38:00AM -0500, Phil Howard wrote:
Wayne Davison wrote:
We certainly do need to be careful here, since the interaction between
the various read and write functions can be pretty complex. However, I
think that the data flow of my move-files patch stress-tests this code
fairly well, so once we've done some more testing I feel that we will
not leave rsync worse off than it was before the patch.
Along those lines, I've been testing the new code plus I ported a
version of my move-files patch on top of it. The result has a couple
fewer bugs and seems to be working well so far.
The latest non-expanding-buffer-nohang patch is in the same place:
http://www.clari.net/~wayne/rsync-nohang2.patch
and the new move-files patch that works with nohang2 is here:
http://www.clari.net/~wayne/rsync-move-files2.patch
I'll keep banging on it. Let me know what you think.
So far it is working for me. Now I can kill my client side and know
that my daemon side will properly close down and exit and not leave
a dangling lock.
But the problem I still have (not quite as bad as before because of
no more hangs) is that the locks to control the number of daemons is
still working wrong. It's still locking the whole lock file instead
of the first lockable 4 byte record. I still don't know if it is
rsync or Linux causing the problem. The code in both looks right to
me. But lslk shows:
SRC PID DEV INUM SZ TY M ST WH END LEN NAME
rsyncd 24401 3,5 44 0 w 0 0 0 0 0 /tmp/rsyncd.lock
(note, I've been moving the lock file around to see if it might be
sensitive to filesystem mounting options I'm using, etc).
I'd like to find a way to start rsync in daemon mode AND leave it
in the foreground so I can run it via strace and maybe see if the
syscall is being done right.
You shouldn't have to have it be in the foreground in order for strace -f
to work. I just wrote a test program that verified it:
main()
{
if (fork() == 0) {
printf(child\n);
setsid();
sleep(10);
printf(bye bye\n);
}
}
strace on that waits until the child process has exitted.
If you really want it to stay in the foreground, edit become_daemon in
socket.c.
- Dave Dykstra
Re: Problem with --compare-dest=/
On Thu, Jul 05, 2001 at 10:19:33PM -0400, David Bolen wrote: We do a bunch of distribution to remote machines (under Windows NT) using compare-dest to make use of existing files on those machines when possible. Up until now, the comparison directory has always been at least one level down in the directory hierarchy, but we've just started doing full recursive distributions of files that have a mirror structure to our C: drive itself, and I think I've run into a buglet with --compare-dest. The code that uses the compare-dest value blindly builds up the comparison using %s/%s (with compare-dest and filename), so if you set --compare-dest=/, you end up with filenames like //dir/name - the leading double slash may not matter under most Unixes, but it does under Windows (Cygwin still uses it as a UNC path) and I think POSIX permits such a path to be system-specific (// inside a path must be ignored, but at the start can do special stuff). Anyway, a quick workaround appears to be to use /. rather than just / for compare-dest (or actually, since it's Windows, using C: works too), but I'm guessing it's something that should probably be handled inside of rsync better? Perhaps it should be using clean_fname(). Please try making a fix using clean_fname() or some other way if it looks better, test it out, and submit a patch. I wrote that option so I'll make sure the patch gets in if I think it looks good. - Dave Dykstra
Re: Instructions on how rsync uses ssh to run remotely?
On Tue, Jul 10, 2001 at 02:24:57AM -0700, Ben Escoto wrote: About two weeks ago I asked about a way to use rsync to backup (I wanted to make something in-between a mirror and an incremental backup). I wrote a preliminary version of a program that does this (see http://www.stanford.edu/~bescoto/rdiff-backup), and am now considering adding remote filesystem support through ssh the way rsync does. Conceptually this may not be too hard, but there are probably issues that will arise that I don't foresee now. So if anyone knows of a description of how rsync does the ssh/rsh client/server bit, or maybe something more generic on how to use ssh in the way rsync does, I would appreciate a reference. I could look at the rsync source, but it doesn't seem to be overly commented, and I'm not exactly fluent in C and thought that this post might save me time in any event. Thanks. All that rsync uses ssh for is to open a bi-directional data pipe and run a command (another rsync program) on the remote side. It then ignores ssh (or rsh, the default) and does its own protocol over the pipe between the two sides. - Dave Dykstra
Re: Instructions on how rsync uses ssh to run remotely?
On Tue, Jul 10, 2001 at 10:58:21AM -0700, Ben Escoto wrote: DD == Dave Dykstra [EMAIL PROTECTED] wrote the following on Tue, 10 Jul 2001 10:34:53 -0500 DD All that rsync uses ssh for is to open a bi-directional data DD pipe and run a command (another rsync program) on the remote DD side. It then ignores ssh (or rsh, the default) and does its DD own protocol over the pipe between the two sides. Hmm, doesn't rsync run two copies of itself on the remote side? Maybe it uses three (?) unidirectional pipes instead of one bi-directional one? And either way, it would be nice to have a primer on designing a protocol like this, since I'm sure many people have already figured out the best way to handle latency, errors, etc. rsync forks itself into two copies on the receiver side, one process for generating checksums and one for receiving updates. It's still a single bi-directional pipe, it's just that on the receiver side one process is writing and one is mainly reading. I've never read it myself, but I imagine that Andrew Tridgell's PhD thesis covers the protocol design. I don't think it's written down anywhere else. You can find it on his home page, http://samba.org/~tridge. - Dave Dykstra
Re: Include/Exclude Problem
On Fri, Jul 20, 2001 at 05:17:30PM -0500, Perry Hoekstra wrote: I am trying to rsync a list of files from one list to another. I went back about five months on the mailing list archive and found this message: On Tue, Mar 20, 2001 at 03:21:54PM It is possible to build your own complete list of files to copy and give them all to rsync, by building a --include list and doing '--exclude *' at the end. Currently you need to also either have --include '*/' or explicitly list all parent directories above the files you want included or else the exclude '*' will exclude the whole directories. There's been talk of adding a --files-from option which would remove this last restriction, and I even offered to implement it, but I'm still waiting for anybody to give performance measurements (using rsync 2.3.2 which had an include optimization that did something similar if there were no wildcards) to show what the performance impact would be. - Dave Dykstra Taking that message as a template, I created the following command to try to move two files: rsync -vaR --include 'devel/a.html' --include 'devel/b.html' --include '*/' --exclude '*' somebox.foo.com::staging/Incremental/07-20-2001_16:00:27/www.foo.com I get the following displayed: receiving file list ... done drwxrwxrwx 512 2001/07/20 17:04:06 Incremental drwxrwxrwx 512 2001/07/20 17:04:06 Incremental/07-20-2001_16:00:27 drwxrwxrwx 512 2001/07/20 17:04:06 Incremental/07-20-2001_16:00:27/www.foo.com wrote 190 bytes read 187 bytes 251.33 bytes/sec total size is 0 speedup is 0.00 Return code = 0 but the files are not moved. What have I screwed up in my include/exclude patterns? Or have I misinterpreted the email? You're getting that ls -l type output because you did not specify a destination directory. Try adding a '.' to the end of your command line. - Dave Dykstra
Re: Backup with rsync and after that with tar
On Tue, Jul 24, 2001 at 05:37:58PM +0100, RoBSD wrote: Hello rsync list, Sorry if my response is already in the archives, I tried to find it but I failed! I have 5 computers running linux (rsync and ssh) remotely and I started to make backups with the following command: rsync -cavzRq -e /usr/local/sbin/ssh but I think (maybe I'm wrong) that this mess the symbolic links. I want to use tar and gzip to make backups from the HDD to tapes and I don't know if the archives will be correct or not because of the symbolic links. Thank you for your time and for this wonderful tool! Radu That command should preserve symbolic links. Some comments on your options: -c is almost always not needed, and it's a lot of overhead -v and -q are opposites, why include both? - Dave Dykstra
Re: rsync and SSL
On Thu, Sep 13, 2001 at 07:41:19PM -0500, Phil Howard wrote: I'm finding even less on rsync and SSL. I would have imagined someone would have done something with this already, but apparently not. So I guess I need to ask and see for sure: has anyone worked on issues of using rsync via SSL, such as with stunnel? I'm sorry, I didn't read this message before my reply. I see you've already covered everything in my reply, so you can ignore it. I want to have encrypted access, either anonymous or authenticated, but without granting any SSH access to anyone (e.g. the rsync users won't be in the /etc/passwd user space). If stunnel doesn't work, how about this idea: what if you hand out an unencrypted SSH private key to all users, and put in a .ssh/authorized_keys on the server with a forced command that restricts what the users can do to specific rsync commands? That will still encrypt the connection, and even though the authentication key will be well-known it should be safe because the authentication key is independent of the encryption key. - Dave Dykstra
Re: rsync and SSL
On Fri, Sep 14, 2001 at 10:29:37PM -0500, Phil Howard wrote: Dave Dykstra wrote: If stunnel doesn't work, how about this idea: what if you hand out an unencrypted SSH private key to all users, and put in a .ssh/authorized_keys on the server with a forced command that restricts what the users can do to specific rsync commands? That will still encrypt the connection, and even though the authentication key will be well-known it should be safe because the authentication key is independent of the encryption key. My concern with SSH is making it function with an authentication space different than the /etc/passwd space, and absolutely ensuring that there is no way anyone accessing via SSH could execute any other command. I'm quite confident rsync will work over stunnel. But I don't know if there is any effort to standardize a different port number for rsync over ssl. No, there hasn't. Is 874 available? In a separate project I'm developing a new POP3 server, and will be looking at integrating SSL, probably with code from stunnel, so the logic of the server operates with the direct knowledge of where the connection comes from. One way that I might do this is for an SSL connection, to launch an additional process to handle the SSL layer just like stunnel, perhaps actually running that code. For rsync, this might also be a way to do it. Integrating it a client could be even more useful. This has been talked about before but never done. See for instance the thread starting at http://lists.samba.org/pipermail/rsync/2000-October/003041.html Nobody has mentioned trying rsync with with stunnel according to my saved rsync-related email. Somebody made an rsync SASL patch but I really don't know if or how that's related to SSL. That posting is at http://lists.samba.org/pipermail/rsync/1999-July/001250.html - Dave Dykstra
Re: Problem with transfering large files.
On Thu, Sep 20, 2001 at 08:52:26AM -0500, Carey Jung wrote: Not so. --timeout=TIMEOUT This option allows you to set a maximum IO timeout in seconds. If no data is transferred for the specified time then rsync will exit. The default is 0, which means no timeout. The 2400 second timeout means that the connection must be _idle_ for 2400 seconds. It does not mean that the session can not last for more than 2400 seconds total. My experience is that this is broken in at least 2.4.7pre1. Whatever timeout one sets seems to be interpreted as 'total runtime' for the rsync invocation. I now set to 0, which means don't timeout. On September 9 Tridge submitted a fix to CVS for that problem. See revision 1.25 at http://pserver.samba.org/cgi-bin/cvsweb/rsync/generator.c - Dave Dykstra
Re: Include / Exclude question
On Tue, Sep 25, 2001 at 12:18:31PM +0300, Coroiu Cosmin Marius wrote: Hello all ! I want to save some of the directories and files from my home dir.For that I create a new directory where I moved the file for sync and I made some symlinks to the dirs which I want to include in sync.The hidden directoris (like .netscape) are not included in sync. How can I say to rsync to include these directories too and how can I exclude a list of directories (I have .mozilla too and I don't want do synchronize the cache dir from .netscape and .mozilla ) ? Have you checked the man page for --include and --exclude? If you want rsync to copy files that are on the other end of a symlink, you'll need to use -L. - Dave Dykstra
Re: Does RSYNC work over NFS?
Yes, but remember that the rsync protocol is optimized for low bandwidth networks and high bandwidth disks. When you go over NFS you are increasing the access time for the disks, and especially if you are copying between two NFS mounts on the same machine you've got very high bandwidth for the network between the two sides. When you have a high bandwidth network, it is more efficient to disable the rsync rolling checksum algorithm with --whole-file, which increases the network traffic but decreases the disk traffic. That's the default in the rsync release that's currently being developed, when source and destination are on the same machine. Also, some people have reported some hangs when copying on NFS. It's unclear to me whether or not the no-hang patches that have been posted to this mailing list have helped. - Dave Dykstra On Tue, Sep 25, 2001 at 03:46:48PM -0500, Jacob Martinson wrote: Beautifully. That's how I have been backing up our NetApp filer. Jacob On Tue, Sep 25, 2001 at 01:41:16PM -0700, Karl Kopper wrote: Does rsync work over an NFS mount? --Karl _ Get your FREE download of MSN Explorer at http://explorer.msn.com/intl.asp
Re: Include / Exclude question
On Wed, Sep 26, 2001 at 09:17:03AM +0300, Coroiu Cosmin Marius wrote: Dave Dykstra wrote: On Tue, Sep 25, 2001 at 12:18:31PM +0300, Coroiu Cosmin Marius wrote: Hello all ! I want to save some of the directories and files from my home dir.For that I create a new directory where I moved the file for sync and I made some symlinks to the dirs which I want to include in sync.The hidden directoris (like .netscape) are not included in sync. How can I say to rsync to include these directories too and how can I exclude a list of directories (I have .mozilla too and I don't want do synchronize the cache dir from .netscape and .mozilla ) ? Have you checked the man page for --include and --exclude? If you want rsync to copy files that are on the other end of a symlink, you'll need to use -L. - Dave Dykstra Hello Dave , I used already the -L option and it works but only for symlinks with other name than .name ..All this symlinks (.netscape , .mozilla ) are not included in synchronization.If I rename them it works ...That it's my question : how can include hidden dirs and symlinks ? Normally rsync will not skip . files. Send me your complete rsync command line and the ls -l listings of your symlinks. If your whole reason for the symlinks is to control what to include and exclude, chances are you can skip the symlinks completely and just use --exclude. - Dave Dykstra
Re: problems with large transfers form solaris 2.6
On Wed, Sep 26, 2001 at 10:46:51AM +1000, Tomasz Ciolek wrote: Hi all. I have a large data set on solaris 2.6 (approx 16gb and about 900K files and rectory structures). Rsync is misbehaving in the most obnoxious way: it is launched as such: rsync -e ssh -aLv --bwlimit=512 --timeout=300 source dest A transfer starts, a list of files and directories is built, and data starts flowing. then for no apparent reason data stops flowing form the rsync on the solaris end the the connection times out after 5 minutes of no IO activity. When the timeout parameter is not there, the rsync will just stop and hang around for hours untill killed. The only other this worth mentioning is that the solaris box is hidden behind a firewall and we use a ssh proxy (TIS plug proxy) to connect to the box. Addibd many v's on the commandline produces a lot of debug but no real information as to whats going on. The data flow just stops and there is no error messages on either end in rsync output or logs... I am stumped... I wander is there is no known issues with globbing functions (if rsync uses them) on solaris or some such... Any isnights or pointers would be appreceated. What version of rsync is running on both sides, and what version of ssh are you using? There have been versions of both that have hanging problems, and there are even patches available for the current version of rsync although I'm not sure whether they're relevant over ssh. People have reported that the -v option makes things worse. Using netstat on both sides (and probably on your proxy machine too) may help to detect where packets are piling up in queues. - Dave Dykstra
Re: optimization for rsync
On Wed, Oct 03, 2001 at 11:50:07AM -0700, Jason Helfman wrote: What flags or options could be used to optimize rsync? I have seen a huge amount of cpu and memory be used in this process we have. I think using the --delete function is a lot of overhead. The options we normally use are -avrpg I would like to reduce all unnecessary processing and overhead. If you're using too much memory, you probably need to break up the transfer into smaller pieces. Rsync currently keeps some memory for every file it visits in a run. If you have a high speed network and are willing to trade decreased cpu usage and disk access for higher network usage, use -W. That disables the rsync algorithm of rolling checksums to send only pieces of files that have changed. - Dave Dykstra
Re: group ownership
On Wed, Oct 17, 2001 at 03:15:00PM -0500, Ronald D. Walkup wrote: willem wrote: Ronald D. Walkup wrote: I am attempting to rsync data from a rsync server and set the permissions to a different gid on the client: my servers name is rserver01 my clients name is rclient01 here is the rysync.conf contained on rserver01: # log file log file = /var/adm/rsync_log # global options for all modules dont compress = *.gz *.tgz *.zip *.z *.rpm *.deb *.iso *.bz2 *.tbz uid = nobody gid = nobody transfer logging=true list=true [test] path=/var/tmp/rsync_test comment = rsync_test read only = true hosts allow=rclient gid = cdburners on the server, ls -al of the source directory indicates that group ownership is visitor: rserver01ls -al total 12 drwxr-xr-x 3 rwalkup visitor 512 Oct 17 13:43 . drwxrwxrwt 6 sys sys 1536 Oct 17 13:31 .. -rw-r--r-- 1 rwalkup visitor 79 Mar 13 2001 hosts drwxr-xr-x 3 rwalkup visitor 512 Mar 13 2001 inet -rw-r--r-- 1 rwalkup visitor 330 Mar 21 2001 vfstab I then execute the following syntax from the client: rclien01 rsync -avz rserver01::test /var/tmp/rsync_test after the copy has completed, I then execute: rclient01 ls -al /var/tmp/rsync_test total 12 drwxr-xr-x 3 rwalkup visitor 512 Oct 17 13:49 . drwxrwxrwt 6 sys sys 1536 Oct 17 13:49 .. -rw-r--r-- 1 rwalkup visitor 79 Mar 13 2001 hosts drwxr-xr-x 3 rwalkup visitor 512 Mar 13 2001 inet -rw-r--r-- 1 rwalkup visitor 330 Mar 21 2001 vfstab Is it not possible that the group ownerhip be changed to cdburners as the rsyncd.conf file indicates? cdburners exist on both client and server. any ideas would be greatly appreciated. Thanks, Ron you need to execute the comman as root on the client I tried that earlier to no avail: rclient01id uid=0(root) gid=1(other) rclient01rsync -avpz yam02::test /var/tmp/rsync_test rclient01ls -al /var/tmp/rsync_test total 12 drwxr-xr-x 3 rwalkup visitor 512 Oct 17 15:01 . drwxrwxrwt 6 sys sys 1536 Oct 17 15:00 .. -rw-r--r-- 1 rwalkup visitor 79 Mar 13 2001 hosts drwxr-xr-x 3 rwalkup visitor 512 Mar 13 2001 inet -rw-r--r-- 1 rwalkup visitor 330 Mar 21 2001 vfstab No, sorry, the gid of files *read* is not set by setting the gid in rsyncd.conf. All that does is effectively change the group id of the running process to that id on the server (assuming the server process has permission to do so), it doesn't change the group id of the files. The gid in rsyncd.conf really only has an effect on creating files. - Dave Dykstra
Re: rsync logging and permission problems
On Wed, Oct 17, 2001 at 08:36:22PM -0400, Faheem Mitha wrote:
Dear rsync people,
I have just started using rsync for backups. I have had a couple of
issues. Note I'm trying to use rsync as user using ssh between two
machines both running Debian GNU/Linux potato (2.2r3). The local machine
is currently running 2.4.6-1 and the remote 2.3.2-1.2.
1) When I run rsync with the vv option, stuff scrolls of my screen faster
than I can read it. I was wondering if there is a logging option I could
use. I didn't see it in the documentation. I can use something like rsync
-avvz -e ssh localdir/ remotemachine:remotedir/ 21 | less to look at
the output, piping both standard output and standard error to less.
However, I'm puzzled why lines like Mail/190 is uptodate are sent to
standard error. Surely they should be sent to standard output?
Hmm, I can't reproduce that. When I use vv and /dev/null, nothing prints.
I'm using the latest version from CVS, but cvsweb tells me the same logic
has been in there at least 3 years:
from log.c:
if (code == FERROR) {
log_got_error = 1;
f = stderr;
}
if (code == FINFO) {
if (am_server)
f = stderr;
else
f = stdout;
}
from rsync.c:
if (verbose 1 ...
rprintf(FINFO,%s is uptodate\n,fname);
The variable am_server is set on the remote side, but in that case usually
code higher up in log.c will already have sent the message out another way
before it gets to the above code, so the message will get printed on the
client side (the side that initiated the rsync command).
2) I am trying to copy the file /var/spool/mail/faheem on the local
machine to /var/spool/mail/faheem on the remote machine. The problem is
that only the file on the remote machine (but not the directory) is owned
by me. rsync first tries to write some other file to /var/spool/mail but
can't. I then used --temp-dir=/tmp and rsync stopped complaining about not
being able to write to /var/spool/mail. However, it still does not seem
able to write the file correctly to the remote machine (I get error
messages like :rename /tmp/.faheem.Tb4wnj - /var/spool/mail/faheem :
Permission denied). I don't understand why this is happening, though.
Rsync always replaces files and doesn't just overwrite them. That's because
it always has to build the new file in a temporary location, and after that
it makes the most sense to just move the temporary file into place.
Can anyone make suggestions about what to do? I would prefer not to run
rsync as root on the remote machine. In any case, I tried to do so and
could not, perhaps because Debian does not allow remote logins as root by
default.
Sorry, rsync always need write permission to the directory.
Please cc any reply to me; I'm not on the mailing list. Thanks in advance.
Sincerely, Faheem Mitha.
- Dave Dykstra
Re: Exclude
On Fri, Oct 19, 2001 at 07:00:14AM -0600, [EMAIL PROTECTED] wrote: --daemon doesn't take the ---exclude* options. Put exclude from = /etc/rsync.exclude in your rsyncd.conf. Yes, and I noticed a couple other mistakes too: Chao,Eisen R. [EMAIL PROTECTED]@lists.samba.org on 10/18/2001 05:52:29 PM Sent by: [EMAIL PROTECTED] To: '[EMAIL PROTECTED]' [EMAIL PROTECTED] cc: (bcc: Tim Conway/LMT/SC/PHILIPS) Subject: Re: Exclude Classification: Hi Folks, I'm running rsync between an NT client and a Solaris 7 Server (Ver 2.4.6). I'd like to be able to exclude the following directory: /opt/IBMHTTPDLOG/archivelogs but retrieve /opt/IBMHTTPDLOG My setup is: rsync -av -p -e [EMAIL PROTECTED]:/opt/IBMHTTPDLOG/ F:/opt/IBMHTTDLOG (client) -e takes a parameter of a command like rsh which I don't see above. Since you got as far as you did, I'm assuming you must have not copied your command line accurately into your email. rsync --daemon --exclude-from=/etc/rsync.exclude (server) Oh, if you intended to connect to a daemon you should have used the double colon syntax in the command line above. If you're connecting to a daemon, a -e command_name is ignored. I suspect you don't want to use the daemon mode at all, and just let your rsync client launch the rsync on the remote side via rsh. Then a --exclude /archivelogs/ on the client side command line should do what you want. - Dave Dykstra What I get is *everything* including the contents of the dir ./archivelogs. What is the proper syntax ? I've tried setting up my entry in rsync.exclude as: '/opt/IBMHTTPDLOG/archivelogs' archivelogs/ 'archivelogs/' ./archivelogs ./archivelogs/ I've also tried setting the server-side rsync daemon as --exclude blah-blah to no avail. Thanks in advance, Eisen Chao - Dave Dykstra CNA Insurance
Re: rsync on cygwin: Connection reset by peer
On Mon, Oct 22, 2001 at 08:42:17AM -0600, [EMAIL PROTECTED] wrote: Dave: I remember a thread where it was decided to default to use chroot = no on systems that don't support it. Is it a pretty recent build? Am I remembering wrong? No, I think we decided to just improve the error message (although it looks like Martin hasn't gotten around to doing that). The relevant messages are at the bottom of the page http://lists.samba.org/pipermail/rsync/2001-August/thread.html and near the top of http://lists.samba.org/pipermail/rsync/2001-September/thread.html - Dave Dykstra
Re: rsync on cygwin: Connection reset by peer
Getting back to the original complaint that started this thread: On Fri, Oct 19, 2001 at 09:04:20PM -0700, Jake Repp wrote: I have been trying to get rsync running correctly on cygwin for the past couple of days. I found a post on the cygwin list that said there was a bug in cygwin when using socketpair() but when I compiled the sample code: (http://jrepp.com/rsync/socketpair.c) both ways it works fine. Here's the article for reference: http://sources.redhat.com/ml/cygwin/2001-08/msg00357.html I am running the latest update from cvs Here's what happens when I try to get a module list from the cgywin system from a freebsd4.4 system: http://jrepp.com/rsync/packetlog.txt fisher Fisher Replicated Directory rsync: read error: Connection reset by peer rsync error: error in rsync protocol data stream (code 12) at io.c(151) I do see the same error when listing modules on an rsync --daemon running under cygwin, connecting to it from Solaris. I haven't tried to debug it though, and transfers work OK. - Dave Dykstra
Re: NT rsync
On Mon, Oct 22, 2001 at 09:46:43AM -0700, Mike Singleton wrote: ** Reply Requested by 10/23/2001 (Tuesday) ** Here is the syntax I have been using, yet it is unable to connect... rsync --progress ftp.funet.fi::pub/languages/CPAN/ \\c/ftp failed to connect to ftp.funet.fi - Attempt to connect timed out establishing a connection. I'm able to connect from cygwin under W2K. You've got the path wrong though. It's ftp.funet.fi::ftp/pub/languages/perl/CPAN/ - Dave Dykstra
Re: rsync recursion question
On Tue, Oct 23, 2001 at 09:08:25PM -0500, Justin Banks wrote: Tim == tim conway [EMAIL PROTECTED] writes: Tim That's the way it is. If it's really a one-off change, a huge change Tim in your structure, telnet ssh, rsh, and so forth, work really well for Tim dropping in and deleting stuff (unless you're supplying the master, and Tim other systems out of your control copy from you). Rsync is opTimized Tim for taking a filesystem in an unknown state, and making it identical to Tim another filesystem in an unknown state, using network bandwidth as Tim efficiently as possible. Well, that doesn't cut it here. It seemed like all the guts were there - I mean, the functionality already exists on the receiving side, mostly, right? Anyway, I took a look, and I added an option (-d). This means that you can do rsync --delete -d /some/removed/directory foo@wherever::module:/some/removed and the remote side will remove /some/removed/directory. It will work whether it's a file or directory. Let me know if anyone wants a patch, if this is a feature that would help other folks. I can't imagine people are in my particular circumstances, where the source filesystem is on the order of a terabyte, but you never know ;) This rang a bell. Back in December 1998 while I was official maintainer of rsync, Phil Dietz submitted a patch to remotely make directories on rsync --daemons and discussed one to remotely delete directories. I was opposed to the former, and came up with a way to do the latter without a patch. Unfortunately the messages were not on the mailing list so I can't refer them to you. The way to recursively delete a single directory while ignoring the rest is: mkdir /tmp/empty rsync -a --delete --include /directorytodelete** --exclude * \ /tmp/empty/ foo@wherever::module - Dave Dykstra
Re: Generating cpio list.
On Wed, Oct 24, 2001 at 07:30:59AM +, Thaddeus L. Olczyk wrote:
One of the things I want to do besides mirror a hard drive is
maintain an incremental backup on tape.
The way for me to do this ( of course ) is to take the
output of rsync and use it to generate a list of files for cpio.
The basic ideal is something like this
declare -i head tail
# gets the first line with total: at the begining
head=$(grep -n ^total: $log_rsync|awk -F: '{print $1}')
head=$head-1
# chop off first two lines
tail=$head-2
head -n $head $log_rsync|tail -n $tail -|grep -v uptodate|cpio -o
/tape_device
Is this a way of accurately generating the list of modified files?
[I assume your $rsync_log is made with -vv; you can probably get rid of
your grep -v uptodate if you used just -v]. I think that should work,
but note that rsync will print a file name any time it considers updating a
file, which is by default whenever the modification time has changed. I'm
not sure if that's any better than just touching a special file whenever
you do an incremental update and using a find -cnewer backup_time_file
[available in GNU find] to find the list of files whose inodes have
changed.
Also does it make sense to set the -a option of cpio ( it doesn't
if rsync modifies the atime of the source files ).
Please, no. As a user I have been burned several times by backup schemes
that have used cpio -a. The -a option resets the access time of the input
file back to what it was, at the price of changing the inode time. That
will mess up find -cnewer.
- Dave Dykstra
Re: Feature req: multiple source trees
How is that different from the behavior now when multiple source directories are given? The rsync Usage says: Usage: rsync [OPTION]... SRC [SRC]... [USER@]HOST:DEST or rsync [OPTION]... [USER@]HOST:SRC DEST or rsync [OPTION]... SRC [SRC]... DEST or rsync [OPTION]... [USER@]HOST::SRC [DEST] or rsync [OPTION]... SRC [SRC]... [USER@]HOST::DEST or rsync [OPTION]... rsync://[USER@]HOST[:PORT]/SRC [DEST] SRC on single-colon remote HOST will be expanded by remote shell SRC on server remote HOST may contain shell wildcards or multiple sources separated by space as long as they have same top-level Multiple sources can be specified in all cases. - Dave Dykstra On Wed, Oct 24, 2001 at 11:19:01AM +0200, Ph. Marek wrote: Hi everybody, I want to share a feature request I wrote here some time ago. I'd like to give rsync more than 1 source-tree, where the sum (as described below) is rsync'd to the destination. suppose I have some machines, which have almost the same configuration, different only in small details. Then I want to rsync all machines from a master, with the details as specific to every machine. I propose the following mechanism: - the commandline is parsed as now, but instead of a single source name an array of source names is saved. - on reading the source directories the first (master) directory is read (as now). - then the rest of the source directories is read, and every file there which is already in memory is discarded. So only additional files are added to memory. and -basedir now points to the correct source directory. so given this example BASE/ a b c 1/ d e 2/ d e if I run rsync BASE --source 1 machine1:destination the trees BASE and 1 are taken and rsync'd to machine1. Another simple extension to this is to give a list of filenames (or a file with a list of filenames) as parameter, and every file in this list is not substituted but APPENDED. So I could have a base /etc/passwd and another with the user accounts in the different version directories and have them pushed together to the other machine! Here a bitfield saying in which trees the file is present would be good. If no-one else starts working on that I'll do my worst - but it can take some time. Comments, please??? Regards, Phil
Re: rsync recursion question
On Wed, Oct 24, 2001 at 02:26:29PM -0500, Justin Banks wrote: Dave == Dave Dykstra [EMAIL PROTECTED] writes: Dave The filename paths that the exclude algorithm sees are relative to the Dave destination path, they do not include it. It should work if you use Dave --include /bob**. The slash at the beginning does not indicate the Dave root of a filesystem, it indicates match only the beginning of a Dave path. It doesn't, unless I'm misunderstanding what you're saying. rsync -a --ignore-errors --delete --include /bob** -exclude * /tmp/empty/ [EMAIL PROTECTED]::module/tmp/foo/bob results in everything in /tmp/foo being removed. I don't understand how it can affect anything above the destination directory you give; you should be giving a destination directory of /tmp/foo. Is that what you meant? I tested it and it worked for me. - Dave Dykstra
Re: rsync under NT
On Thu, Oct 25, 2001 at 05:47:00AM -0700, Mike Singleton wrote: Here is the syntax I am using... rsync ftp.orst.edu::pub/CPAN \\c/ftp Unable to establish a connection is the error... DaVita Inc. They're not running an rsync server on that host: $ telnet ftp.orst.edu 873 telnet: Unable to connect to remote host: Connection refused - Dave Dykstra
Re: one log for each module
On Fri, Oct 26, 2001 at 06:08:55PM +0200, Ivan Renedo wrote: There are some way to have a log for each module? If there aren't, it should be a a great functionality for maintaining the server. Thanks for the help and continue with this great application ! Currently the log file option is a global but I don't see why it can't be made into a local. Do you want to work up a patch? At one point the log file was opened early and never closed reopened on a long-running rsync --daemon, but I changed that a while ago to re-open the log file on every new connection, and that may have taken away the reason for making it a global. - Dave Dykstra
Re: what is this messages mean?
On Mon, Oct 29, 2001 at 04:43:03PM +0900, shakey001 wrote: Hi, friends.. I saw dmesg messages, but I don't know this messages. === cut === TCP: Treason uncloaked! Peer 111.222.333.444:873/36106 shrinks window 1566013416:1566014864. Repaired. TCP: Treason uncloaked! Peer 111.222.333.444:873/36106 shrinks window 1566013416:1566014864. Repaired. TCP: Treason uncloaked! Peer 111.222.333.444:873/36106 shrinks window 1566013416:1566014864. Repaired. === cut === I am use rsync tool for backup-system. what is this mean..? I have no idea, but I can assure you that it didn't come from rsync. - Dave Dykstra
Re: 2.4.7p1 protocol differences?
On Fri, Oct 26, 2001 at 02:19:55PM -0400, Dave Wreski wrote: Hi all, rsync-2.4.6 has been running for quite a while with no problems, until about a week ago when for some reason it blocked on the same host ever time. I use it to backup about thirty hosts to my backup server, but for some reason it's illiciting that blocking bug that some people experience with 2.4.6. So, I upgraded the client side with 2.4.7, but it complained of protocol differences. I certainly can't upgrade all 30 hosts to 2.4.7p1, especially when it's not production-ready yet. I'm not aware of any protocol differences. Can you be more specific? Try to give us some simple steps to reproduce it if you can. The host that it's blocking on isn't even transferring all that much data. Maybe fourty or so meg per day. What could be happening? I'm not sure how much data it takes to trigger the bug. What kinds of hosts, and what transfer method are you using? Perhaps you'd like to try appling Wayne Davison's no-hang patch on 2.4.6 rathern than using 2.4.7p1, from http://www.clari.net/~wayne/rsync-nohang2.patch - Dave Dykstra
Re: compilation error
Sounds like your gcc is not installed properly.
- Dave Dykstra
On Mon, Oct 29, 2001 at 06:17:20PM -0500, SANJEEV wrote:
I have two machine on which I am getting this error. Both machine is Solaris
7 and rsync version I am trying is 2.4.6.
This is the error
checking whether the C compiler
(gcc -O -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 ) works... no
configure: error: installation or configuration problem: C compiler cannot
create executables.
when I saw the config.log file. Both have almost same error.
Machine1/config.log==
configure:698: checking for gcc
configure:811: checking whether the C compiler
(gcc -O -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 ) works
configure:827: gcc -o conftest -O -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
conftest.c 15
/usr/local/lib/gcc-lib/sparc-sun-solaris2.7/3.0/../../../../sparc-sun-solari
s2.7/bin/ld: cannot open values-Xa.
o: No such file or directory
collect2: ld returned 1 exit status
configure: failed program was:
#line 822 configure
#include confdefs.h
main(){return(0);}
Machine2/config.log ==
configure:698: checking for gcc
configure:811: checking whether the C compiler
(gcc -O -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 ) works
configure:827: gcc -o conftest -O -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
conftest.c 15
/usr/local/sparc-sun-solaris2.7/bin/ld: cannot open values-Xa.o: No such
file or directory
collect2: ld returned 1 exit status
configure: failed program was:
#line 822 configure
#include confdefs.h
main(){return(0);}
Thanks for any help here.
-S
- Original Message -
From: Dave Dykstra [EMAIL PROTECTED]
To: SANJEEV [EMAIL PROTECTED]
Sent: Wednesday, October 10, 2001 12:03 PM
Subject: Re: compilation error
On Fri, Oct 05, 2001 at 03:50:46PM -0400, SANJEEV wrote:
While running ./configure, I got an error as below -
compiler cannot create executables
before this error , it declares that OFFSET is 64.
I also found on rsync.samba.org, which explains my problem.
Note that on some systems you will have to force configure not to use
gcc because gcc may not support some features (such as 64 bit file
offsets) that your system may support. Set the environment variable CC
to the name of your native compiler before running configure in this
case.
Now my question is how to set OFFSET -- 32 ?
Thanks.
You may get a better response if you post again listing your operating
system and rsync version number.
- Dave Dykstra
rsync hang w/ linux, openssh, latest rsync
On Mon, Oct 29, 2001 at 06:22:07PM -0500, Dave Wreski wrote: Subject: Re: 2.4.7p1 protocol differences? I'm not aware of any protocol differences. Can you be more specific? Try to give us some simple steps to reproduce it if you can. Actually, I screwed up. The remote side had quite an old version of rsync. Upgrading to 2.4.6 fixed the protocol problems but hasn't fixed the hang problem. It's really strange because it hangs in the same place every time, and it's the only instance of rsync from the 30 or so in my shell script that cause a problem, with other instances transferring much more data. Other instances are also on the same server that work fine. We have had situations where -v appeared to cause a hang, although I thought those had been fixed. Try without that. I was almost thinking it was getting stuck on a socket or symlink or something. While there aren't any fifos in that directory, there are about 50 symlinks.. Try with trial error to eliminate particular files. I once found a single 80 byte file that could hang rcp every time, we assumed because of some buggy network element. I'm not sure how much data it takes to trigger the bug. What kinds of hosts, and what transfer method are you using? Perhaps you'd like to try appling Wayne Davison's no-hang patch on 2.4.6 rathern than using 2.4.7p1, from They're all Linux boxes with OpenSSH_2.3.0p1 as the transport on both sides, using ssh1. I just tried the patch and it hangs in the same place as 2.4.6 stock and 2.4.7p1. OpenSSH 2.3 is rather old, although I'm not aware of any hanging problems in it. Which Linux kernel? There have been TCP bugs in Linux fixed in more recent releases. It often helps on hangs to run netstat on both sides and look at the queues for the connection to see if there are things in the send queue on one side but nothing in the receive queue on the other. If that is true, it's an operating system bug. - Dave Dykstra
Re: Widespread implementation of rsync
On Sat, Oct 27, 2001 at 08:34:14PM +0500, Java Dukes wrote: I came to know about rsync algorithm for transferring files few days ago. The algorithm provides an efficient way to transfer files from one computer to another over a slow communication link. But I have been wondering why don't we see a widespread implementation of rsync in most popular file transfer tools. I know that the algorithm was conceived by Andrew Tridgell and Paul Mackerras from Department of Computer Science, Australian National University. I was thinking about implementing this algorithm in Java and make it available as a freeware/shareware. I guess I would need some kind of permission/license to develop a program that uses rsync. I know there's no patent on it, and other people have done implementations. The resources page on rsync.samba.org points to a REXX implementation. Josh Macdonald wrote an implementation in C http://sourceforge.net/projects/xdelta/ and Martin Pool has turned it into a library in C (I don't see where the home page is -- Martin?). I believe somebody else had talked about a java implementation but never actually did it. - Dave Dykstra
Re: rsync not copying owner/group on some files
On Mon, Oct 29, 2001 at 02:02:17PM -0800, Ian Kettleborough wrote: I am trying to use rsync 2.4.6 to try creating a backup using another locally connected hard disk. Thing copy, but sometimes the group is eith numeric (valid). Often the owner is not even shown. This is the command I use for rsync: rsync-2.4.6/rsync -arxlHpogDt /usr/ /backup/usr/ --delete I login as root. Any help would be appreciated. I cannot reproduce it. Please tell us your operating system and give us a simple set of steps to reproduce it, from creating logins and groups with specific ids and creating specific files owned by those logins and groups. - Dave Dykstra
Where to get 2.4.7pre1
On Tue, Oct 30, 2001 at 02:14:53AM -0800, Ian Kettleborough wrote: Subject: Re: 2.4.7p1 protocol differences? Can you please tell me where I can doenload a copy of rsync 2.4.7??? The pre release tarball is at http://rsync.samba.org/ftp/rsync/preview/rsync-2.4.7pre1.tar.bz2 You'll need to have bzcat/bunzip to uncompress it. Martin or Tridge: please put a pointer to preview releases and/or this specific release on the download web page. You can also get the absolute latest out of CVS by following the instructions at http://www.samba.org/samba/cvs.html but using the package name rsync instead of samba. - Dave Dykstra
Re: Where to get 2.4.7pre1
On Wed, Oct 31, 2001 at 08:05:09AM -0600, Dave Dykstra wrote: On Tue, Oct 30, 2001 at 02:14:53AM -0800, Ian Kettleborough wrote: Subject: Re: 2.4.7p1 protocol differences? Can you please tell me where I can doenload a copy of rsync 2.4.7??? The pre release tarball is at http://rsync.samba.org/ftp/rsync/preview/rsync-2.4.7pre1.tar.bz2 You'll need to have bzcat/bunzip to uncompress it. Martin or Tridge: please put a pointer to preview releases and/or this specific release on the download web page. You can also get the absolute latest out of CVS by following the instructions at http://www.samba.org/samba/cvs.html but using the package name rsync instead of samba. And for those of you without CVS access or brain-dead firewalls you can also get the latest development version at http://rsync.samba.org/ftp/unpacked/rsync/ or ftp://rsync.samba.org/pub/unpacked/rsync/ Those should be mentioned on download.html too; it currently gives the impression that they're only available through rsync. - Dave Dykstra
Re: No action from rsync 2.4.6 on HP-UX 11.0
On Wed, Oct 31, 2001 at 09:23:57AM -0700, [EMAIL PROTECTED] wrote: Greetings: I have two systems, HP model 9000/800/R380. They are identical systems running 64-bit HP-UX 11.0 with 1GB RAM. I have downloaded and installed the HP rsync binaries, version 2.4.6 on both the source and target systems. The directory tree on the source system contains over 58GB of data in 1.3 million objects (including directories, links and files), 1.2 million normal files. The target directory is empty with 110GB free. With sample data sets, with several thousand files and hundreds of MB of data, rsync works. When I go to my production data, rsync chugs away for about an hour and a half and then I get a prompt back. Nothing is transferred between systems. I run rsync in command line mode. Here's what it looks like: Mainsys # cd /export/data/stuff /usr/local/bin/rsync -vca --delete --rsync-path=/usr/local/bin/rsync . Altsys:/export/data building file list ... Mainsys # Memory usage is not exceeding 50%. Any ideas? How can I get rsync to work in this situation? I can't say why rsync is dying, but definitely do not use -c. That runs a complete checksum on every file on both sides. Without it, rsync will only look at files whose timestamps don't match. Since the target directory is empty in this case, it's computing checksums on all the source files for nothing. - Dave Dykstra
Re: rsync hang w/ linux, openssh, latest rsync
On Thu, Nov 01, 2001 at 11:47:15AM -0500, Dave Wreski wrote: We have had situations where -v appeared to cause a hang, although I thought those had been fixed. Try without that. No difference. Try with trial error to eliminate particular files. I once found a single 80 byte file that could hang rcp every time, we assumed because of some buggy network element. I'm not sure how to do that since there are so many files in the directory. Make a copy of your data on the both sides, and try cutting out half the data on the send side at a time to find the smallest set that causes it to fail. strace shows me both sides are stuck in a select() and the sendq and recvq are seemingly empty according to netstat. Ok, that eliminates one kind of operating system failure. I did find this when using --dry-run: # rsync --dry-run -avve 'ssh ... -i ..' remote:/path/ /mnt/backup ... bits/eng/html/cells/cell_220_article.txt Invalid file index 1541696587 in recv_files (count=2062) unexpected EOF in read_timeout What is an invalid file index? I looked at the the that appeared to logically come next in the list, even added it to an exclude list, and no change. A file index is the number of a file should never be larger than the total number of files being transferred. It sounds like some data is getting lost or garbled somewhere. OpenSSH 2.3 is rather old, although I'm not aware of any hanging problems in it. Which Linux kernel? There have been TCP bugs in Linux fixed in more recent releases. It's 2.2.19-ac7 or so and has worked for months without incident, until now. That sounds recent enough, but I do recommend upgrading your OpenSSH. - Dave Dykstra
Re: timeout and process cleanup bugs 2.4.7pre1+
On Fri, Nov 09, 2001 at 10:29:14AM -0700, [EMAIL PROTECTED] wrote: I have written an application in perl, to run multiple rsyncs, retrying those which fail. I have started getting this error on very large modules since i moved to the latest CVS version. ++ ToolSyncProgress Succeeded Getting big/tools/vlsi_local/etc from willy at 1004767211:2001:11:3:6:0:11. ToolSyncProgress Getting big1/cadappl1/hpux from willy at 1004767211:2001:11:3:6:0:11. rsync: connection unexpectedly closed (50466734 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(139) ++ The script checks exit status and retries, ordinarily. However, when i get THIS error, the rsync process seems to be killing the perl process. I think it's a bug in the part of the code that cleans up the spawned processes. Here's another example, which shows that the timeout as total transfer time but is still active. It also ends by killing its calling process. The error looks different, though, but it's on a local nfs-to-nfs, instead of rsyncd-to-nfs. ++ ToolSyncProgress Replicating big1/cadappl1/hpux from chi-tools1 to chi-tools2 at 1004913115:2001:11:4:22:31:55. io timeout after 3600 seconds - exiting rsync error: timeout in data send/receive (code 30) at io.c(75) ToolSyncProgress chinetsvr failed Replicating big1/cadappl1/hpux from /wan/chi-tools1/chi-tools1 to /wan/chi-tools2/chi-tools2 at 1004925444:2001:11:5:1:57:24 with 7680, retrying. ToolSyncProgress Replicating big1/cadappl1/hpux from chi-tools1 to chi-tools2 at 1004925444:2001:11:5:1:57:24. io timeout after 3600 seconds - exiting rsync error: timeout in data send/receive (code 30) at io.c(75) ToolSyncProgress chinetsvr failed Replicating big1/cadappl1/hpux from /wan/chi-tools1/chi-tools1 to /wan/chi-tools2/chi-tools2 at 1004937647:2001:11:5:5:20:47 with 7680, retrying. ToolSyncProgress Replicating big1/cadappl1/hpux from chi-tools1 to chi-tools2 at 1004937647:2001:11:5:5:20:47. io timeout after 3600 seconds - exiting rsync error: timeout in data send/receive (code 30) at io.c(75) ToolSyncProgress ToolSyncProgress Replicating io timeout after 3600 seconds - exiting rsync error: timeout in data send/receive (code 30) at io.c(75) ToolSyncProgress ToolSyncProgress Replicating rsync error: received SIGUSR1 or SIGINT (code 20) at main.c(741) ++ Any ideas? I can try to track the killing and timeout logic, if there's not already a fix out there that was just accidentally dropped. I don't recall anybody else reporting a similar problem, so please try to track and debug it further. - Dave Dykstra
Re: direct write patch
Oh boy, I think you're getting into quite a can of worms there.
At a minimum this option should imply the --partial option because if the
operation is aborted the file will be left partially transferred. Note
that if you're trying to use the rsync rolling checksum algorithm to
minimize bandwidth that if a transfer is interrupted all the previous
data will be lost so there will be less data to compare against when the
transfer is retried.
Next, keep in mind that the receiving side of an rsync transfer uses two
independent processes, one for generating checksums and one for creating
files. I'm not knowledgable enough to know whether or not the creating files
operation is guaranteed to begin only after the checksum generation is
completed on each file, but if it isn't then overwriting a file could be
a big problem. Have you tried transferring any very large files?
I'm surprised you didn't need to do anything to finish_transfer to keep
the robust_rename from returning an error.
The last thing that comes to mind is that overwriting files that are open
by another process, such as a running executable, can be a problem. An
unlink/rename works much better for that.
I'm sure there are more issues too.
- Dave Dykstra
On Mon, Nov 12, 2001 at 10:12:54AM -0800, Don Mahurin wrote:
I have attached a patch that supports a new --direct-write option.
The result of using this option is to write directly to the destination
files, instead of a temporary file first.
The reason this patch is needed is for rsyncing to a device where the
device is full or nearly full.
Say that I am writing to a device that has 1 Meg free, and a 2 meg file
on that device is out of date.
Rsync will first attempt to write a new temp file, and fail, SIGUSR1'ing
itself, and outputting Error 20.
Specifically, I am writing a linux root fs to a 32Meg compact flash, and
libc needs to be updated, and rsync fails.
This patch simply sets fnametmp to fname.
Two issues with the patch, that hopefully developers can answer:
- In direct-write mode, I open without O_EXCL, as the file likely does
exist.
Should the destination file be deleted instead? (I do not know what
exactly the race condition is)
- There is a section after the assignment of fnametmp, and before the
open that does do_mktemp, then
receive_data. What is the purpose of this part? I skip it for
direct-write, and it works, but what do I know?
-don
Only in rsync-2.4.6-direct-write/lib: dummy
diff -ru rsync-2.4.6/options.c rsync-2.4.6-direct-write/options.c
--- rsync-2.4.6/options.c Tue Sep 5 19:46:43 2000
+++ rsync-2.4.6-direct-write/options.cSun Nov 11 10:40:01 2001
@@ -22,6 +22,7 @@
int make_backups = 0;
+int direct_write = 0;
int whole_file = 0;
int copy_links = 0;
int preserve_links = 0;
@@ -147,6 +148,7 @@
rprintf(F, --ignore-errors delete even if there are IO errors\n);
rprintf(F, --max-delete=NUMdon't delete more than NUM files\n);
rprintf(F, --partial keep partially transferred files\n);
+ rprintf(F, --direct-write write directly to the destination
files\n);
rprintf(F, --force force deletion of directories even if not
empty\n);
rprintf(F, --numeric-ids don't map uid/gid values by user/group
name\n);
rprintf(F, --timeout=TIME set IO timeout in seconds\n);
@@ -188,7 +190,7 @@
OPT_LOG_FORMAT, OPT_PASSWORD_FILE, OPT_SIZE_ONLY, OPT_ADDRESS,
OPT_DELETE_AFTER, OPT_EXISTING, OPT_MAX_DELETE, OPT_BACKUP_DIR,
OPT_IGNORE_ERRORS, OPT_BWLIMIT, OPT_BLOCKING_IO,
- OPT_MODIFY_WINDOW};
+ OPT_MODIFY_WINDOW, OPT_DIRECT_WRITE};
static char *short_options = oblLWHpguDCtcahvqrRIxnSe:B:T:zP;
@@ -227,6 +229,7 @@
{perms, 0, 0,'p'},
{links, 0, 0,'l'},
{copy-links, 0, 0,'L'},
+ {direct-write, 0, 0,OPT_DIRECT_WRITE},
{copy-unsafe-links, 0, 0, OPT_COPY_UNSAFE_LINKS},
{safe-links, 0, 0,OPT_SAFE_LINKS},
{whole-file, 0, 0,'W'},
@@ -400,6 +403,10 @@
safe_symlinks=1;
break;
+ case OPT_DIRECT_WRITE:
+ direct_write = 1;
+ break;
+
case 'h':
usage(FINFO);
exit_cleanup(0);
@@ -554,6 +561,8 @@
keep_partial = 1;
break;
+
+
case OPT_IGNORE_ERRORS:
ignore_errors = 1;
break;
@@ -691,6 +700,9 @@
slprintf(mdelete,sizeof(mdelete),--max-delete=%d,max_delete);
args[ac++] = mdelete;
}
+
+ if (direct_write)
+ args[ac++] = --direct-write;
if (io_timeout) {
slprintf(iotime,sizeof(iotime),--timeout=%d,io_timeout);
diff -ru
Re: direct write patch
On Mon, Nov 12, 2001 at 11:50:01AM -0800, Don Mahurin wrote: Perhaps, all that I need is a --delete-before-update option that just unlinks the file before it starts to write the temp file. Then we avoid the possible issues that you raised. I can still see a case where --direct-write may be useful (read-write file in a read-only dir), but this is probably not a common situation, and I don't want to tackle those issues yet. Wait, I forgot something more fundamental about the way the rsync implementation works. The one process on the receiver side generates checksums, but the other one puts together pieces of the old file as well as the pieces from the new file that get sent across the network. If it happens to want pieces that are earlier in the file than what is being written, your file will be corrupted if you're overwriting it. For example, if it needs to move the data at the point 50K into the file forward to the point 100K in the file, it will already have been overwritten by the new data. If on the other hand you use your proposed --delete-before-update option, it won't do you any good because the operating system will not actually delete the file until rsync closes it after it is completed, because rsync will hold the file descriptor open the whole time. So it won't save you any space. Sorry, but I don't think it's possible to do what you want and still use the rsync algorithm. You could probably do it in conjunction with the --whole-file option which turns off the rsync algorithm, but then you give up a lot. - Dave Dykstra
Re: rsync exclude/include [FIDUCIA virengeprüft - ohne Gewähr, daß alle bekannten Viren und deren Varianten erkannt wurden.]
On Tue, Nov 13, 2001 at 01:44:45AM +0100, Thomas Schweikle wrote: Hi again! Tomasz Ciolek [EMAIL PROTECTED] on 13.11.2001 01:21:27 An: Thomas Schweikle [EMAIL PROTECTED] Kopie: Thema:Re: rsync exclude/include [FIDUCIA virengeprüft - ohne Gewähr, daß alle bekannten Viren und deren Varianten erkannt wurden.] Try dumoping the whole file list into a text file, prefix each one with minus for exclude and a + for include. Then run rsync like this: rsync -avz --exclude-from=exclude.list ftp3.sourceforge.net::/netbsd/iso/ ./iso/ have a look at the rsync man page for more information on the exact syntax of the unified include/exclude stuff Regards. Done that. Result: rsync include/exclude matching is definitively wrong. If ... matching is done using shell filename matching ... this file should work: - /* + /iso/1.5.*/i386* but doesn't. Looks like * is matched, but does not lead to a true matched against /iso/1.5.1/i386cd.iso. Here's what you neeed: + /iso + /iso/1.5.* + /iso/1.5.*/i386* - /iso/1.5.*/* - /iso/* - /* In order to understand why you need to realize that the rsync algorithm recurses down into each directory and applies each pattern in order and stops when it finds a match, so if you don't explicitly include the parent directories they get excluded by the exclude wildcard and rsync never descends down into the directory. Note that /* only excludes things at the top level, that's why I had to explicitly exclude things at the next two levels too. A simplification is this: + /iso + /iso/1.5.* + /iso/1.5.*/i386* - * because if you don't have the slash preceding the exclude * it applies at all levels. Any way to debug this, making sure how matches are really done? The most recent snapshots of the development version of rsync will give you more helpful information if you use -vv, but it's not yet in any released version. The patch is at http://rsync.samba.org/cgi-bin/cvsweb/rsync/exclude.c.diff?r1=1.32r2=1.33 - Dave Dykstra
Re: rsync exclude/include
On Tue, Nov 13, 2001 at 10:00:59AM -0600, Dave Dykstra wrote: ... A simplification is this: + /iso + /iso/1.5.* + /iso/1.5.*/i386* - * because if you don't have the slash preceding the exclude * it applies at all levels. Wait, if i386* matches directories and not just files you'll need + /iso + /iso/1.5.* + /iso/1.5.*/i386** - * so that it will include all files below the i386* directories. Any double asterisk in a pattern tells the matching routine to go beyond slashes. - Dave
Re: Ptoblem for update 2.4.1 - 2.4.6
On Tue, Nov 13, 2001 at 12:41:10PM +0300, Eugene Frolenkov wrote: After installing nevest version of sync 2.4.6 from 2.4.1, sync --daemon wrote to logfile: 2001/11/13 12:27:46 [12261] rsyncd version 2.4.6 starting 2001/11/13 12:27:46 [12261] bind failed on port 873 2001/11/13 12:27:46 [12261] transfer interrupted (code 10) at socket.c(233) sync 2.4.1 worked normally... Chances are another process is still holding port 873 open, or you're not running as root. My question: Where i can find WATHSNEW or CHANGES documentations any versions of rsync? Unfortunately there's no long-running file like that for rsync. I see for the next release the new maintainer has added a file called NEWS so maybe it will be coming. Required rsync-2.4.6 newest version of bind? Sys: RedHat Knl: 2.2.14 bind: 8.2.3 It has nothing do with the program called bind, it is the system call bind(2). - Dave Dykstra
Re: transfer interrupted (code20)
On Wed, Nov 14, 2001 at 11:05:55AM +0800, Michael P. Carel wrote: ... Hi, Im trying now to implement my mail server mirror but as i look at the logs of the mail server i used to mirror i see this type of log transfer interrupted (code20) at rsync.c(229). and it the remote who's mirrorring the mail server does not transferring file and got stocked in recieving file list. I've tried transffering other path and it works fine but in the users home directory im recieving this error. I have alot of users in my mail server. What seems to be the problem? I'm sorry, but that kind of problem is rather difficult to make a guess at remotely if we can't repeat the problem. Perhaps it might help if you told us your rsync version and operating system on both sides, but that probably won't be enough. You're going to have to do some further debugging yourself. Try narrowing it down to the smallest repeatable operation you can do to get the error message, and then use a system call tracer, a debugger, or added print statements in the code to try to focus in on the problem. In general transfer interrupted just means that a connection dropped prematurely for some unknown reason. - Dave Dykstra
Re: using rsync to backup windows workstations
On Wed, Nov 14, 2001 at 10:12:27AM -0700, [EMAIL PROTECTED] wrote: OK, I might be actually getting this. Here is part of my sshd_config file ... # To disable tunneled clear text passwords, change to no here! PasswordAuthentication yes PermitEmptyPasswords no this REQUIRES that a password be used, No it does not. It only makes password authentication one of the options and when it is used, the password can't be empty. # Comment to enable s/key passwords or PAM interactive authentication # NB. Neither of these are compiled in by default. Please read the # notes in the sshd(8) manpage before enabling this on a PAM system. ChallengeResponseAuthentication no Do I just need to change the last line to yes (ChallengeResponseAuthentication yes) and then when I generate my key on the NT box, put in a passphrase. No, ChallengeResponse is a totally different kind of authentication method. +++ If PasswordAuthentication is yes, then you'll need to ensure that PermitEmptyPasswords is also yes, or you won't be allowed passphraseless authentication. WRONG. Passwords have nothing to do with passphrases. To: [EMAIL PROTECTED] cc: (bcc: Tim Conway/LMT/SC/PHILIPS) Subject:Re: using rsync to backup windows workstations Classification: I have about 12 NT servers that I want to backup 1 directory to my Linux Server. The problem I am having is trying to get it not to ask for the user's password. I am trying to use ssh. I just installed the latest cygwin on my NT server (1.3.4). Rsync is 2.4.6. What I have tried was on the NT machine, I ran ssh-keygen (no passphrase) and then added the identity.pub to the Linux box in the /home/thomas/.ssh/authorized_keys. When I run: rsync -uv -e ssh *.* thomas@linuxbox:backup/ it asks for a password. Besides asking for the password, it works fine. Only the files changed are updated and it is FAST! I saw the ssh-agent. Do I need to use it? and how? Is there something I need to do on my Linux server? I am pretty new to Linux, and have a long way to go to be an expert, but I can get around. The problem is probably that it's defaulting to use ssh protocol 2, and and ssh-keygen defaults to makeing a key for protocol 1. If you either run 'ssh -1' or start over with an 'ssh-keygen -t rsa' you should be ok. Use ssh -v verbose mode to find out for sure what's going on. You may also have permissions problems on the server; it requires that all files and parent directories not be world readable. - Dave Dykstra
Re: rsync: Re: using rsync to backup windows workstations
On Wed, Nov 14, 2001 at 02:47:42PM -0600, Ben Ricker wrote: On Wed, 2001-11-14 at 14:35, Thomas Lambert wrote: I thought just a straight rsync daemon was unsecure. I am sending data from remote dial-up sites, through the internet to my server. That is why I am using ssh (well trying to use it). If this was just for my internal network, then YES, I would probably just use the daemon. One other problem is that they are dial-up, so each time they connect to the internet, they get a new IP. You are right: Rsync as a daemon in your environment would be less secure than ssh. Agreed. I tried just doing ssh linuxbox username -a and I was asked for a password. So I changed PasswordAuthentication to no in my sshd_config file on the server. Restarted sshd and now I am getting permission denied. I did copy the contents of my identity.pub, rsa_id.pub, dsa_id.pub to authorized_keys on the server. I'm going to read some more on ssh, but if anyone knows a quick fix, let me know. There are two: make sure the dsa_id.pub is in authorized_keys2 if you just use '-e ssh' as an option for rsync, this automagically uses ssh version 2, so you need the '2' you keys file. 'authorized_keys' is for ssh 1. You can change the -e option to -e 'ssh 1' to use auhtorized_keys (I believe that is the command line; I got ssh 1 doing authentication but I switched to 2 because it is much more secure). Not anymore in OpenSSH 3.0p1: the regular known_hosts and authorized_keys files hold both kinds of keys, and the *2 files are deprecated. He's using cygwin 1.3.4 which includes OpenSSH 3.0p1. - Dave Dykstra
Re: rsync: Re: using rsync to backup windows workstations
On Wed, Nov 14, 2001 at 04:23:32PM -0500, Thomas Lambert wrote: THAT WAS IT!! I just had authorized_keys and not authorized_keys2. I knew I had to be close. THANK YOU! THANK YOU! THANK YOU! Then your server just be older than openssh 3.0. Right? - Dave Dykstra
Re: transfer interrupted (code20)
On Thu, Nov 15, 2001 at 09:42:53AM +0800, Michael P. Carel wrote: I'm using RedHat 6.1 and the latest rsync rpm package. I've adding or incresing its verbosity by having a triple v : rsync -acvvv --progress --stats --timeout=0 host::shared path It does start downloading the file and succcessfully completed. Any idea why should i need to put a trple v instead of just a single v? Assuming that truly did make the difference and it wasn't just luck, it sounds like a timing problem. - Dave
Re: include/exclude directory question
On Thu, Nov 15, 2001 at 04:38:26PM -0800, Dave Madole wrote: I want to send a subset of directories sepcified in include arguments to a client, but I am creating all the peer and parent directories as well, although they are empty - here is basically what I'm doing. assuming I have /staging/upgrade/dir1, /staging/upgrade/dir2 and /staging/upgrade/dir3 on the source tree. --include */ -include upgrade/dir1/** -include upgrade/dir3/** -exclude * /staging/* / what I get on the client is /ugrade/dir1/* /upgrade/dir2 (an empty directory) /upgrade/dir3/* How can I get rsync to filter out and stop creating the empty dir2 directory without specifucally excluding it? Because of your --include */ you'll always get all the directories, empty or not. If you want to skip them, you'll need to individually include the parent directories that you want to keep, in your example upgrade/, upgrade/dir1/, and upgrade/dir3/. Sorry, but there's no other way around it. - Dave Dykstra
Re: rsync hangs or exists without copying anything
On Fri, Nov 16, 2001 at 04:15:29PM -, Daniel J. Conlon wrote: I am trying to mirror a file system using rsync. The command I am using is of the form: rsync -a /fs/home/6/ /usr/fs/home/6 /fs/home/6/ is an NFS file system, /usr/fs/home/6 is a local disk. With versions 2.4.6 and 2.4.7pre1, rsync hangs at random places during the building file list phase. I tried with and without -v option(s) and tried breaking the file system down into smaller chunks with no improvement. I have read of many similar problems with later versions of rsync so I tried with version 2.3.2. This time rsync did not produce any errors, but did not copy anything. /usr/fs/home/6 is at this point empty so it should have copied the entire file system. By using the v options(s) and by breaking the file system down into smaller and smaller sections I found that rsync was existing at the same point during the building file list phase, specifically, when it hit's this directory: make_file(4,a/d/adrenalinuk/public_html/cgi-bin/misc/ib3/Database/Databa se/Database/active_sessions/Database/Database/address_books/Database/Dat abase/Database/authorisation/Database/Database/board_statistics/Database /Database/categories/Database/Database/config/Database/config/Database/c onfig/Database/config/Database/config/Database/config/Database/config/Da tabase/config/Database/config/Database/config/Database/config/Database/c onfig/Database/config/Database/config/Database/config/Database/config/Da tabase/config/Database/config/Database/config/Database/config/Database/c onfig/Database/config/Database/config/Database/config/Database/config/Da tabase/config/Database/config/Database/Database/email_templates/Database /Database/forum_info/Database/Database/forum_moderators/Database/Databas e/forum_polls/Database/Database/forum_posts/Database/Database/forum_rule s/Database/Database/forum_subscriptions/Database/Database/forum_topics/D atabase/Database/help/Database/Database/Database/mem_groups/Database/mem _groups) Is there some kind limit on the depth of file system rsync can cope with? Or the length of the filename? If so, how can I increase it or simply have rsync ignore paths or depths greater than the maximum, rather than just exiting? If there is no limit, how can I overcome this problem? Many thanks for your help in advance. Yes, MAXPATHLEN defaults to 1024 in rsync.h. You didn't mention your operating system, but on Solaris, it is also set the same in param.h so I would think you would have problems with other tools as well. I don't know if forcing MAXPATHLEN to a larger value will help or not. - Dave Dykstra
Re: patch to enable faster mirroring of large filesystems
Before I look at this closely, I have a couple questions. First, what options do you use to copy? I once saw somebody who went through a lot of work to cache things and it turned out to be just because he was using the --checksum option when he shouldn't have. Next, have you taken a look at the 2.4.7 pre-release? It includes a couple options --write-batch and --read-batch which may do want you want, I'm not sure. I think the intention of that, though, was more to be able to efficiently do copies to multiple targets, not to speed up a single copy. You can get the latest development version of rsync by doing rsync -a rsync://rsync.samba.org/ftp/unpacked/rsync . - Dave Dykstra On Mon, Nov 19, 2001 at 04:06:45PM -0500, Andrew J. Schorr wrote: I have attached a patch that adds 4 options to rsync that have helped me to speed up my mirroring. I hope this is useful to someone else, but I fear that my relative inexperience with rsync has caused me to miss a way to do what I want without having to patch the code. So please let me know if I'm all wet. Here's my story: I have a large filesystem (around 20 gigabytes of data) that I'm mirroring over a T1 link to a backup site. Each night, about 600 megabytes of data needs to be transferred to the backup site. Much of this data has been appended to the end of various existing files, so a tool like rsync that sends partial updates instead of the whole file is appropriate. Normally, one could just use rsync with the --recursive and --delete features to do this. However, this takes a lot more time than necessary, basically because rsync spends a lot of time walking through the directory tree (which contains over 300,000 files). One can speed this up by caching a listing of the directory tree. I maintain an additional state file at the backup site that contains a listing of the state of the tree after the last backup operation. This is essentially equivalent to saving the output of find . -ls in a file. Then, the next night, one generates the updated directory tree for the source file system and does a diff with the directory listing on the backup file system to find out what has changed. This seems to be much faster than using rsync's recursive and delete features. I have my own script and programs to delete any files that have been removed, and then I just need to update the files that have been added or changed. One could use cpio for this, but it's too slow when only partial files have changed. So I added the following options to rsync: --source-list SRC arg will be a (local) file name containing a list of files, or - to read file names from stdin --null used with --source-list to indicate that the file names will be separated by null (zero) bytes instead of linefeed characters; useful with gfind -print0 --send-dirs send directory entries even though not in recursive mode --no-implicit-dirs do not send implicit directories (parents of the file being sent) The --source-list option allows me to supply an explicit list of filenames to transport without using the --recursive feature and without playing around with include and exclude files. I'm not really clear on whether the include and exclude files could have gotten me the same place, but it seems to me that they work hand-in-hand with the --recursive feature that I don't want to use. The --null flag allows me to handle files with embedded linefeeds. This is in the style of gnu find's -print0 operator. The --send-dirs overcomes a problem where rsync refuses to send directories unless it's in recursive mode. One needs this to make sure that even empty directories get mirrored. And the --no-implicit-dirs option turns off the default behavior in which all the parent directories of a file are transmitted before sending the file. That default behavior is very inefficient in my scenario where I am taking the responsibility for sending those directories myself. So, the patch is attached. If you think it's an abomination, please let me know what the better solution is. If you would like some elaboration on how this stuff really works, please let me know. Cheers, Andy
Re: --no-detach option?
On Wed, Nov 21, 2001 at 12:23:29PM -, Max Bowsher wrote: I did the same, for cygwin, but I called the option --debug-daemon. --no-detach is probably a better name. Heres my patch. - Original Message - From: Jos Backus [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Tuesday, November 20, 2001 11:05 PM Subject: --no-detach option? How about adding a --no-detach option (to be used in combination with --daemon) to rsync so it can be run under Dan Bernstein's daemontools' supervise? If there's interest I'll provide a patch. Rsync will already not detach if stdin is not a socket. Is that sufficient? - Dave Dykstra
Re: --no-detach option?
Excuse me, I meant to say if stdin in IS a socket. - Dave Dykstra On Wed, Nov 21, 2001 at 07:46:29AM -0700, [EMAIL PROTECTED] wrote: Not in daemon mode. = # ps -ef |grep rsync # rsync --daemon # ps -ef |grep rsync root 23716 1 0 07:42:58 ?0:00 rsync --daemon # rsync localhost:: tconway # Like most daemons, it forks by default. I've wanted this feature for some time, for debugging, but instead, just started it with truss, with the fork option. Tim Conway [EMAIL PROTECTED] 303.682.4917 Philips Semiconductor - Longmont TC 1880 Industrial Circle, Suite D Longmont, CO 80501 Available via SameTime Connect within Philips, n9hmg on AIM perl -e 'print pack(, 19061,29556,8289,28271,29800,25970,8304,25970,27680,26721,25451,25970), .\n ' There are some who call me Tim? Dave Dykstra [EMAIL PROTECTED] Sent by: [EMAIL PROTECTED] 11/21/2001 07:15 AM To: [EMAIL PROTECTED] cc: (bcc: Tim Conway/LMT/SC/PHILIPS) Subject:Re: --no-detach option? Classification: On Wed, Nov 21, 2001 at 12:23:29PM -, Max Bowsher wrote: I did the same, for cygwin, but I called the option --debug-daemon. --no-detach is probably a better name. Heres my patch. - Original Message - From: Jos Backus [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Tuesday, November 20, 2001 11:05 PM Subject: --no-detach option? How about adding a --no-detach option (to be used in combination with --daemon) to rsync so it can be run under Dan Bernstein's daemontools' supervise? If there's interest I'll provide a patch. Rsync will already not detach if stdin is not a socket. Is that sufficient? - Dave Dykstra
Re: --no-detach option?
On Wed, Nov 21, 2001 at 11:07:21AM -0800, Jos Backus wrote: On Wed, Nov 21, 2001 at 08:54:18AM -0600, Dave Dykstra wrote: Excuse me, I meant to say if stdin in IS a socket. That mode of operation is indeed useful when running rsync --daemon from inetd or tcpserver. I am talking about a mode in which rsync still listen()s, etc. but can be managed by its parent (supervise in the daemontools case). I'm not familiar with daemontools; I checked the web site and it wasn't obvious to me. What's the advantage of using daemontools supervise? Why does it need to run the daemon as a subprocess? Assuming it's a command to make sure a daemon keeps running, it seems to me that it ought to be able to work if given a process id and not require any deaemon it could supervise to be modified to not fork. I have a script for my own use which does that. - Dave Dykstra
Re: Password-less daemon or rsh setup
On Thu, Nov 22, 2001 at 04:52:02PM -, [EMAIL PROTECTED] wrote: I'm new to rsync and trying to configure it to run through either the daemon or rsh (ssh is not avalilable). Doing an 'rlogin production' results in a password prompt - which I want rid of, as we're basically going to have a robot user running a cron job that utilises rsync. I'm at a loss as to why I can't configure rlogin to user hosts.equiv or similar - any help on this would be greatly appreciated as it's definitely the preferred method. That depends on your system operating system, but it's usually a very simple matter. The hardest part is usually finding out what the server thinks the name of the client is. Logging in and saying who am i will tell you on some operating systems. I've had some limited success with the daemon approach - but again I'm prompted for a password. Here's the command I'm trying to run: backup(nemxb01)$ rsync -nr production::test_module /export/home/nemxb01 If I supply nemxb01's password I get the correct results. FYI: nemxb01 doesn't exist on the production box. That depends on production's rsyncd.conf; if you post that we could tell you what needs to be changed. It's surprising that it would prompt for a password because it needs to be explicitly set up to do so. If someone could give me a step-by-step approach as to how to configure rlogin, and subsequently rsync to run through rsh I'd appreciate it. - Dave Dykstra
Re: rsync-ing compressed archives
On Sun, Nov 25, 2001 at 08:39:23PM +0100, Mauro Condarelli wrote: Hi there! I'm quite happily using rsync. There is only one case where I couldn't figure out how to use it efficently: I sometimes have large compressed files (either .tar.gz or .zip) that I need to keep synchronized. The exploded files are usually not available on the machines i use for rsync (to keep everything unpacked would mean wasting a lot of space on the server). The problem is that even if I change slightly the content of the archive (just add a file) the compressed form will be almost completely different making rsync algorythm practically useless. Is there any easy way to tell the daemon to unpack (or just uncompress) a compressed archive before doing the matching? The problem is not severe with .zip archives, but .gz and .bz2 are really inefficient. Thanks in advance for any hint Mauro No, sorry rsync can't do it. I answered a similar question just last week. Many people have asked but nobody has made a patch. It's probably doable but not trivial. - Dave Dykstra
Re: Rsync: Re: patch to enable faster mirroring of large filesystems
On Tue, Nov 27, 2001 at 02:34:22PM -0500, Lenny Foner wrote: ... I know you're trying to get reliable statistics so it's clear what sort of performance we're talking about here. But may I respectfully suggest that -having- to be so careful about whether optimization actually got turned on is a clue that there is still a big problem here? No, the difficulty of turning on the optimization is irrelevant because the optimization is no longer in the current version of rsync. It is only needed to do the performance test which is a one-time thing. You seem to be missing my point. I agree that --files-from is useful even if it has no impact or even negative impact on performance. Nevertheless, I want to know what the impact on performance will be compared to using an explicit include-from list, and I am bartering my volunteer effort of developing the code for someone else's volunteer effort of doing performance tests of the old optimized case which I expect to be practically identical to the performance of --files-from. I personally don't need --files-from because the --include-from list is working fine for me, so I need extra motivation to put some time into it. I think it has to be done much like that optimization was done and since I wrote the optimization in the first place I expect it will probably be more efficient for me to do it than it would be for somebody else to do it; otherwise I'd probably just say forget it and wait for somebody else to write the code. ... [ranting deleted] ... P.S. Would --files-from= reduce rsync's large memory consumption as well, or does it still imply rsync caching some info about every file it sees during its entire run, and never flushing this info until the end? I'm pretty sure that rsync won't use up memory for excluded files so it would make no difference. - Dave Dykstra
Re: Not all files synched - hard link problems???
Unfortunately there is no way to search the archive. That would be very useful. I haven't heard of any similar problems reported with hard links before, and I've been following this list closely for several years. I notice your command line looks pretty complicated, so I suggest that you try to narrow it down to the smallest reproducible case, preferably one that you can completely describe to someone else how to reproduce starting from scratch. Often such an exercise alone will reveal a solution, but if not at least it allows somebody else to debug it. - Dave Dykstra On Tue, Nov 27, 2001 at 11:29:54AM -0800, Dave Madole wrote: I am sorry if this has been covered before: I have done a couple of futile searches in the bug reporting system and is there any way to search the archive? I am having a strange symptom: I am synching directories (that have very long file names, by the time the full path is specified) and a lot of hard links. It seems that the directory is being copied piecemeal - that is, if I run rsync enough times, the entire contents ultimately get copied. It seems like I am running into some hard limit in the size of the filelist or something. I am running 2.4.6 on linux - the source directory is remote mounted on solaris, the destination is linux. For instance, I have a directory that is in the tree that contains 173 files at the source - most of which are hard links - here is the effect of an ls | wc on the destination after five succesive identical runs of rsync on the source (this is in a subdirectory of one of the directories in the command below). The directory did not exist before running the sync. [root@ks-s0-107-1- SC]# ls | wc 49 491286 [root@ks-s0-107-1- SC]# ls | wc 85 852234 [root@ks-s0-107-1- SC]# ls | wc 120 1203243 [root@ks-s0-107-1- SC]# ls | wc 152 1524112 [root@ks-s0-107-1- SC]# ls | wc 173 1734739 So that it seems to have synched 49, then 36, then 35, then 32, the finally the last 21 files in the directory. (The increment seems to vary if I try it again, i.e. from 49 to 90). I get no error mesages (that I can see). In fact, the first time I run the program, it seems to notice all the files and produces 163 messages that file blah-de-blah is a hard link, but then doesn't seem to make the link for file blah-de-blah - this behavior remains constant with each succesive run. Here is the rsync command (generated by a perl script and broken into little pieces by my mailer) rsync -e 'ssh -l root -p 8989' --rsync-path /usr/bin/rsync --stats --progress -tpgoCHr -v -v --include install/ --include install/kickstart/ --include install/kickstart/zantaz_upgrade/ --include install/kickstart/zantaz_upgrade/20011121/ --include install/redhat-7.1/ --include install/redhat-7.1/zantaz_rpm_upgrade/ --include install/redhat-7.1/zantaz_rpm_upgrade/DS05_00_00-SAM-SUN-20011121/ --include install/redhat-7.1/zantaz_rpm_upgrade/DS05_00_00-SAM-SUN-20011121-devel/ --include install/kickstart/zantaz_upgrade/20011121/** --include install/redhat-7.1/zantaz_rpm_upgrade/DS05_00_00-SAM-SUN-20011121/** --include install/redhat-7.1/zantaz_rpm_upgrade/DS05_00_00-SAM-SUN-20011121-devel/** --exclude * /net/swdev/staging/* [EMAIL PROTECTED]:/ Thanks, Dave
Re: Rsync: Re: patch to enable faster mirroring of large filesystems
On Tue, Nov 27, 2001 at 05:00:14PM -0500, Lenny Foner wrote: ... [ . . . ] I'm pretty sure that rsync won't use up memory for excluded files so it would make no difference. ...though this also implies (since you say it'd probably use basically the same mechanism internally) that it -would- nonetheless keep info around for the entire run about each file that -was- going to be/had been transferred, yes? Yes, that is a limitation in the current implementation. Long ago the original author Andrew Tridgell proposed a rewrite to avoid that but he never got to it. This is a separate problem from how the files are selected, but I've lost track of what the right solution here should be, except for dropping each directory's info after you leave it--- which would presumably not necessarily be easy if you're getting the file list in arbitrary order via --files-from, but might be easier if they were being generated via rsync's current traversal algorithm. I'm not clear on the details of why rsync keeps some information in memory for every file, but I don't think it has anything to do with order or per-directory limitations. In any event, I -hope- that the memory issue is cleanly separable from the issue of how files get selected; this might be a good time to at least ponder the issue, if --files-from might soon exist. I believe the two are cleanly separable. - Dave
Re: How to avoid copying empty directories?
On Tue, Nov 27, 2001 at 04:03:58PM -0800, [EMAIL PROTECTED] wrote: rsync -avu --include 'tmp1/*/*.c' --include */ --exclude * tmp1 tmp2 The above command copies all the empty directories under tmp1/ . Is there any way to avoid it? Currently the only way is to explicitly include the parent directories you want instead of --include */. - Dave Dykstra
Re: cannot create .hosts.b0WX1x : File exists
You probably need to set read only = no in rsyncd.conf. - Dave On Wed, Nov 28, 2001 at 04:29:35PM +0100, Rok Krulec wrote: Hello, when I do: /opt/rsync/bin/rsync /etc/hosts targethost::bkp/ I get: cannot create .hosts.b0WX1x : File exists I check the targethost and I get empty file .hosts.b0WX1x When trying with other targethost-s it works, but on this one it doesn't. On the other targethosts I have exactly the same LinuX distribution, permissions and users than on the problem targethost. I have exactly the same configuration than on the other machines, except on probelm targethost I have SCSI hard disk. When i strace the rsync --deamon i get: 4697 open(.hosts.b0WX1x, O_RDWR|O_CREAT|O_EXCL|0x8000, 0600) = 4 4697 open(.hosts.b0WX1x, O_WRONLY|O_CREAT|O_EXCL|0x8000, 0600) = -1 EEXIST (File exists) It seems like it opens the file ReadWrite and then tries to open it readonly. I don't know why. It bangs then. Thanks for any hint, Rok Krulec Portoroz/Slovenia, 28.Nov 2001 @ 15:34 CET Public GnuPG key @ http://rok.fpp.edu/pgp.key Portoroz, 28.Nov 2001 @ 15:39 CET Public GnuPG key @ http://rok.fpp.edu/pgp.key
Re: rsync server over SSH [includes code patches]
On Thu, Nov 29, 2001 at 09:11:58AM +1100, Martin Pool wrote: ... 1. remove AC_FUNC_MEMCMP from configure.in because it causes Sunos 4.1.4 to die. memcmp() there fails the 8-bit clean test, but it doesn't matter because memcmp() in rsync is only used to test if something is equal or not equal, not to compare if it is less or greater. What breaks if we leave it in? Is memcmp a compiler intrinsic or something like that? No objection to taking it out, I'm just curious. It says don't know how to build memcmp.o. The configure rule only adds memcmp.o to $LIBOBJS, which was not used before. AC_FUNC_MEMCMP has always been in configure.in but before now it has been ignored because @LIBOBJS@ was not used in Makefile.in now and that was the only thing it affected. I think it is, isn't it? LIBOBJS = ... @LIBOBJS@, etc. It is now, but it wasn't a week ago. See http://pserver.samba.org/cgi-bin/cvsweb/rsync/Makefile.in.diff?r1=1.55r2=1.56f=h - Dave
Re: Rsync: Re: patch to enable faster mirroring of large filesyst ems
On Thu, Nov 29, 2001 at 12:59:00PM -0600, Keating, Tim wrote: I was at first, but then removed it. The results were still insufficiently fast. Were you using the -c option of rsync? It sounds like you were and it's extremely slow. I knew somebody who once went to extraordinary lengths to avoid the overhead of -c, making a big patch to rsync to cache checksums, when all he had to do was not use -c. 23 minutes to check 3200 files is definitely unexpected. What options did you end up using? Normally rsync will only check the modification timestamps and the sizes of the files on both sides (that is, only a stat()) and if they match it will not do anything else. - Dave Dykstra
Re: cannot create .hosts.b0WX1x : File exists
On Thu, Nov 29, 2001 at 11:46:30PM +0100, Rok Krulec wrote: Hello Dave, On Thu, 29 Nov 2001, Dave Dykstra wrote: What version of rsync are you using? The cannot create message is coming from receiver.c. Assuming you're using a released version of rsync and not a development version, it appears that the mktemp library call is creating the file and rsync isn't expecting that. Are you sure the Linuxes are exactly the same; do you perhaps have a different version of libc on that one machine (I'm guessing that maybe mktemp has been changed to create the file instead of just returning its name)? Andrew Tridgell changed the relevant section of code in the development version of rsync 6 months ago to use mkstemp (when configure determines it to be secure) instead of mktemp but he didn't say why. This is it. All working machines have 2.2 lnx kernel and libc-2.1.3.so nonworking one has 2.4 lnx kernel and libc-2.2.1.so. I changed line 121 of syscall.c to use: mktemp(template); instead of mkstemp(template); I got his linker warning, but rsync was linked OK and is working fine now. syscall.o: In function `do_mktemp': syscall.o(.text+0x32f): the use of `mktemp' is dangerous, better use `mkstemp' Dave, thank You for Your help. You guys are really great. What version of sources is that which had mkstemp at line 121 of syscall.c? It's surprising that you could just replace one with the other, as mkstemp is supposed to open the file and mktemp is not supposed to. It sounds like you have some inconsistent version of the sources. - Dave
Re: Rsync: Re: patch to enable faster mirroring of large filesystems
On Thu, Nov 29, 2001 at 11:02:07AM -0500, Alberto Accomazzi wrote: ... These numbers show that reading the filenames this way rather than using the code in place to deal with the include/exclude list cuts the startup time down to 0 (from 1hr). The actual sending of the filenames is down from 2h 15m to 1h 40m. The reason this isn't better is due to the fact that turning buffering on only helps the client, while the server still has to do unbuffered reads because of the way the list is sent across. Are you sure about that? I don't see any unbuffered reads. 2.3.2 did have the read_check() hack which was there to prevent SSH pipes from getting stuck, maybe that's what you're seeing. That was taken out in 2.4.0 so maybe that would greatly speed it up. As far as I can tell there is no way to get around the buffering without a protocol change or a different approach to sending this list. Given the data above, I think implementing --files-from this way would be the wrong way to go, for a number of reasons: I've been starting to think along those lines too. It should be a protocol change to just send the files and not treat it like excludes. In fact, the file list is normally sent from the sender to the receiver, but if the client is the receiver maybe we could figure out a way to have --files-from only send the list in the other direction. - Dave Dykstra
Re: @ERROR: chroot failed
On Wed, Oct 10, 2001 at 11:47:11AM -0700, Jason Helfman wrote: Why would I be getting this on a box? I have the same permissions on many other directories. Using verbose mode shows nothing more then the error in the Subject line. That error is coming from an rsync daemon server. Are you administering that side? If so, what does the rsyncd.conf look like? Most likely it is either a) it's run as root but there's no uid = root b) it's not run as root and there's no use chroot = no - Dave Dykstra
Re: Rsync sync then chown/chgrp feature
On Wed, Oct 10, 2001 at 07:51:58PM -0700, Doug wrote: the attached patch adds the synchronize these files and then chown/chgrp them to a specified user/group, no matter who owns the originals feature. the relevant usage lines are now: -o, --owner preserve owner (root only) -o, --owner=OWNER specify owner (root only) -g, --group preserve group -g, --group=GROUP specify group with this patch, the feature works locally but not remotely. in options.c:server_options(), i've added code to pass args on to the server, but i don't think i'm doing it correctly. if anyone would be willing to take a look and see what they think is happening, i'd be grateful. other comments/nitpicks also welcome. My first reaction to this is that it isn't necessary because root can just su to another user before installing, but then I remembered that cpio has a -R id option to assign a different user id and owner for convenience. Maybe going that route would be better, and it is handy sometimes for root, especially when copying locally. Oh, I see that GNU cpio has a long name equivalent of --owner for -R, and allows setting the group separately with the syntax of OWNER:GROUP or OWNER.GROUP. So maybe Doug's suggestion is fine. I would suggest that if the owner is specified but the group isn't then it should default to the owner's group in the password file rather than preserving the original group. - Dave Dykstra
Re: Confused by modules
On Fri, Oct 12, 2001 at 09:37:18AM +0100, Paul Donovan wrote: Hi, I'm confused as to the purpose of rsync modules. I know how to set them up in rsyncd.conf, and can list the available modules on a remote linux machine using 'rsync camlinux01::' from a Win32 client, but I can't find any information on what they are actually for. Basically they're a convenience to be able specify different top level directories and give different characteristics to each. Normally people do not want to expose their whole filesystem to people without logins, which would be the other alternative. The man page for rsyncd.conf talks about them as if they are vitaly important, but the man page for rsync doesn't mention them anywhere other than the part about how to list them on a remote machine. I'd like to be able to set up a module such as: [tlibs] path = /rsynctest/tlibs comment = Latest SCEI libraries and tools and then be able to get the full hierarchy of folders and files inside the /rsynctest/tlibs folder. I can get them by doing 'rsync -acvzP -e ssh root@camlinux01:/rsynctest/tlibs .' , but that means I have to know the specific directory. How do I say that I want the [tlibs] module without knowing the path? Just refer to it as tlibs: rsync -acvzP camlinox01::tlibs . My apologies if I've missed anything obvious. Thanks, Paul Donovan Senior Programmer SCEE Cambridge - Dave Dykstra
Re: rsync binary for ncr mp ras svr4
On Fri, Oct 12, 2001 at 03:56:27PM +0100, Feroz F. Basir wrote: Hi again, can you upload the rsync binary for ncr mp ras svr4 unix system, please? can you upload at : http://rsync.samba.org/ftp/rsync/binaries/ Uh oh, I haven't tried it since the machine samba.org was migrated to new hardware, and I see the files have the wrong owner and group so I have no permission to update them. Tridge or Martin, can you please fix those files to all be in the rsync group (and have a sane owner)? usually, do you compile rsync every time they release the new stable version? Yes, although I only maintain some of the binaries on the rsync.samba.org site and other people maintain others (and some of them aren't maintained at all anymore). The rsync binaries at http://www.bell-labs.com/wwexptools are often better than the stable release because I include important fixes that come along in between. - Dave Dykstra
Re: open issue regarding local copies and -W
On 4 Sep 2001 21:24:33 +1000, Martin Poole [EMAIL PROTECTED] wrote: On 14 Aug 2001, Dave Dykstra [EMAIL PROTECTED] wrote: Martin, I want to remind you that there's still an open issue regarding copies when both source and destination are on the local machine (including over NFS): Thanks for the reminder. I hope the regression suite will help chase these things out and keep them out. I haven't had time yet to merge and test the nohang patch. I put a patch into the CVS to automatically imply the '-W' option in that case, and somebody said he was concerned about that because he found that with -W on local copies he had cases where rsync would lock up, and it wouldn't without -W (I presume because -W artificially slows it down). Should there be an option to turn -W back off, overriding that new default (or should the new default be taken back out)? There are a few other options that would be nice to be able to negate (--blocking-io comes to mind), maybe there should be a consistent way of specifying them. Should the bug that causes lockup with -W be investigated (or has it already been fixed with the Wayne Davison patches)? I suppose we should have the --no-* option syntax. That's pretty simple. This didn't make it into 2.5.0. Can you take care of it Martin? Maybe you thought I was going to do it. - Dave
Re: cannot create .hosts.b0WX1x : File exists
On Fri, Nov 30, 2001 at 12:08:19AM +0100, Rok Krulec wrote: Hello Dave, What version of sources is that which had mkstemp at line 121 of syscall.c? It's surprising that you could just replace one with the other, as mkstemp is supposed to open the file and mktemp is not supposed to. It sounds like you have some inconsistent version of the sources. rsync --version returns: rsync version 2.4.6 protocol version 24 Rsync 2.4.6 did not use mkstemp at all. A comment in receiver.c says mktemp is deliberately used here instead of mkstemp. and line 120 of syscall.c calls mktemp. I suspect you had a locally hacked version of rsync source in which someone changed the mktemp call to mkstemp, and you just changed it back. I don't have any 2.4 kernel systems with libc-2.2.1, but I tried a 2.4 kernel with a later libc and mktemp and mkstemp worked as expected. Ah, I bet I know what happened. I see that gcc warns when you use mktemp: the use of `mktemp' is dangerous, better use `mkstemp' so somebody person must have thought they could just replace one with the other without looking at the man page. - Dave Dykstra
Re: include/exclude ?
On Fri, Nov 30, 2001 at 03:18:16PM -0700, [EMAIL PROTECTED] wrote: You're right. the order dependency actually lets you create very complex include/exclude rules. for each item, each --include and --exclude is evaluated in commandline order, until the item has either passed all the tests, and is thus included, or has been excluded, at which point evaluation restarts with the next item. if you --exclude=* first, every file gets excluded right away, and never gets checked to see if it matches --include=an* . Group: That's a correct synopsis of the behaviour, isn't it? If not, don't let my error stand. Correct except that I would say that it has gone past all the tests rather than has passed. Saying has passed makes it sound like it matched all the patterns when actually you mean that it didn't match any of the patterns. - Dave
Re: rsyncing devices
On Sun, Dec 02, 2001 at 04:14:15PM +1100, Martin Pool wrote: On 27 Nov 2001, Dave Dykstra [EMAIL PROTECTED] wrote: I can reproduce it, and I think you've stumbled on a significant bug. The problem is that rsync always assumes stat variable st_rdev, which contains both the major and minor number, is always 4 bytes and is always the same format on both sides. On the rsync I tried on Linux, it is 8 bytes, whereas it was 4 bytes on Solaris. I assume it depends on whether or not 64 bit mode gets enabled on the operating system version you're on. Are the top bits actually used? Perhaps truncation would be marginally better, or perhaps we should have a ./configure warning that this will be broken if sizeof(st_rdev)4? I don't think we can assume that they're not used, although yes truncation would be marginally better and we could probably easily have a patch that would scrunch the major minor device numbers down to the 32 bit format but I think it would be better to fix it properly with a protocol change as long as something's being changed. I don't think it's worth a configure warning, just fix it. This will take a protocol change to fix. I think probably rsync should split out the major and minor numbers as two separate 4 byte quantities. I wonder if there are other 64 bit stat values that are being misinterpreted. I agree, and added it to TODO. We will need this on Linux as well, though it's anybody's guess whether devfs and dynamic device numbers will make it mostly irrelevant there. - Dave
Re: bug in permissions on symlinks
On Sun, Dec 02, 2001 at 11:52:08PM +1100, Martin Pool wrote: I see rsync has this in rsync.h #ifndef HAVE_LCHOWN #define lchown chown #endif So on Linux lchown changes the ownership on a symlink, whereas chown on a symlink will change the ownership of its target. man lchown says In versions of Linux prior to 2.1.81 (and distinct from 2.1.46), chown did not follow symbolic links. Since Linux 2.1.81, chown does follow symbolic links, and there is a new system call lchown that does not follow symbolic links. Since Linux 2.1.86, this new call (that has the same semantics as the old chown) has got the same syscall number, and chown got the newly introduced number. I'm not at all sure the way we're calling it is correct. There could be systems where there is no lchown, but chown on a symlink still follows the link. I seem to remember Solaris behaving this way, but I'm not sure if that is the system call or just the chown command. In this case we might be accidentally setting the ownership of the destination. On such a machine we should just not set the ownership of the symlink at all. (To be really tricky I guess we could seteuid() before creating the link.) David, do you have a machine that behaves like this? No, I don't. Even my oldest systems have symbolic links and also have lchown. I think any system that has symbolic links would have to implement the lchown/chown distinction, so I don't think it's something to worry about. - Dave Dykstra
-v prints directories twice
Does it bother anybody else that -v prints directories twice? It first prints all affected directories and files in recursive order and then prints new directories again. I can't recall noticing that rsync always did that, but I don't think it's a very recent change. The directory is being printed both times by recv_generator(), and I checked CVS and it's been there since the beginning of generator.c. Investigation told me that the second time around it is fixing up the modification time directories that had some file in them change during the run, done by a special call from recv_files() in receiver.c. I note that on that special call, the f_out parameter is set to -1 so the following little patch fixes it. I think this is simple enough that I'll just submit it to the rsync CVS as one of the small changes I put in. - Dave Dykstra --- generator.c.O Mon Dec 3 12:29:31 2001 +++ generator.c Mon Dec 3 12:27:04 2001 @@ -247,7 +247,9 @@ fname,strerror(errno)); } } - if (set_perms(fname,file,NULL,0) verbose) + /* f_out is set to -1 when doing final directory + permission and modification time repair */ + if (set_perms(fname,file,NULL,0) verbose (f_out != -1)) rprintf(FINFO,%s/\n,fname); return; }
--help output too long?
On Mon, Dec 03, 2001 at 12:15:41AM +1100, Martin Pool wrote: Subject: Re: --no-detach option? I'm starting to think we need to not show all the options in the default --help output. I think perhaps the default should be to show the commonly-used options (-avz, --include, : vs ::) and then have --help-options and so on for more details. It is getting quite ridiculous. There's one or two packages that do this, but I can't remember which ones right at the moment. I think the current way is OK. Other commands have long lists too, for example gpg and wget. They do both put them in labeled groups, however. That's a possibility, although I can't think of any nice logical grouping off the top of my head. - Dave Dykstra
Re: question on permissions
On Mon, Dec 03, 2001 at 01:21:29PM -0800, Sri Ramkrishna wrote: I have a question regarding how rsync changes ownership when syncing two areas. Currently, I have this situation: I have two areas over a WAN, we are trying to mirror from one site to another. One site is not controlled by us and has different unix groups. When we copy one to the other, we are running rsync on an account that exists at both places (different UIDs though) In one area we have no problems the groups/owners exist at both places. However in another area, the owner exists but the group name does not. When we run rsync between the two we get a lot of chown errors. Now this makes sense as some OSs (we are using HPUX 11) do not allow a person to change the ownership of a file. However, we have another area where both the group and the ownership exist on both ends but we have no chown errors. Can anybody explain this behavior? In the end, the area still gets owned by the uid and gid of the rsync running on our end. Thanks, sri Are you using rsync --daemon mode on one side? There are several tricky issues related to that, alluded to under the --owner option in the rsync man page. What user id are you doing the transfers under? Also, rsync enforces bsd-style ownership and group semantics, regardless of whether or not the underlying operating system permits more freedom with chown and chgrp. That means it won't allow doing a chown to another user if you're not root, and the only groups it will allow a non-privileged user to chgrp to are those which groups which the user belongs to. - Dave Dykstra
Re: Netware modify bit changed
On Tue, Dec 04, 2001 at 12:51:07AM -0300, Juan wrote: Hi, I run rsync, with a Netware-directory mounted on a linux system (using ncpfs) as DESTINATION of the copy. This configuration works fine, but rsync incorrectly sets the archive bit (modify) for ALL the files included in the transfered directory (both modified and not modified). In consecuence, the incremental backups on my netware server don't work. Do you mean the access time? If so, tell me what rsync command line you use because rsync normally will not change the access time of unmodified files. If you really mean archive bit, it must be something in Netware that I've never heard of, and please explain what it is. - Dave Dykstra
Re: bug in permissions on symlinks
On Fri, Dec 07, 2001 at 12:58:31AM +1100, Cameron Simpson wrote: ... Not so. The sunos4 boxen don't have lchown() You're right. However, the chown man page says it doesn't follow symlinks: If the final component of path is a symbolic link, the own- ership and group of the symbolic link is changed, not the ownership and group of the file or directory to which it points. and I'd expect the orignal BSD stuff where symlinks came in didn't have it. (I certainly had never tripped over it). I don't have any other BSD systems, but I would think they'd be like Sunos4. Another counter example - Apollo symlinks were special directory entries, not objects with inodes, and chowning them was meaningless. SInce the target permissions were always enough anyway (permissions on a symlinks can be trivially bypassed by opening the full path, and symlinks are not writable themself - only creatable), so inodeless implementations are both feasible and sensible. Does anybody run rsync on Apollo? Please - if there's no lchown DO NOT chown symlinks. It is silently destructive. I say let's don't bother to change it unless somebody reports a problem. RANT It has long offended me that the default chown/chgrp/chmod calls follow links - to my mind the _only_ system calls that should follow links are open() and chdir(). It would have made for a much saner and safer world. /RANT Looks like the BSD inventors of symlinks agreed with you for chown chgrp (but not chmod). Only SystemV messed up when they added symlinks, and as a result had to come up with lchown. - Dave Dykstra
