Re: does --whole-file always work?
--whole-file definitely does something: it turns off the rsync rolling checksum algorithm, which avoids reading the destination file before writing it. It just doesn't avoid passing the data between two processes and the other things that rsync normally does. - Dave On Sat, May 16, 2009 at 07:49:05AM +0300, Amir Rapson wrote: > Thanks Matt and Dave, > But now I'm a bit confused, it doesn't work because it has not been done > yet, but it's the default? > Are there any plans of improving the performance with local files? > Do you mind if I give it a go? > > Thanks, > Amir > > ------ > From: "Dave Dykstra" > Sent: Friday, May 15, 2009 10:43 PM > To: "Amir Rapson" > Cc: > Subject: Re: does --whole-file always work? > >> On Thu, May 14, 2009 at 09:59:16AM -0400, Matt McCutchen wrote: >>> On Thu, 2009-05-14 at 15:59 +0300, Amir Rapson wrote: >>> > Running rsync with --whole-file yields poorer performance results than >>> > a simple "cp". >>> > Looking at the code - it looks like "copy_file" isn't really called >>> > when I add the --whole-file flag. The regular "receive_data" is doing >>> > the copy. >>> >>> That's right. In the current design, file transfers are always done >>> over the connection between the sender and receiver and copy_file is >>> used only for copies within the destination (which happen with a few >>> options such as --copy-dest). It would be possible to make a special >>> case for local runs and have the receiver open the source file directly >>> (or something like that), but this has not been done yet. >> >> Also note that --whole-file is the default if both source and destination >> are local. Compare it to --no-whole-file and you'll see worse >> performance. >> >> - Dave > > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: does --whole-file always work?
On Thu, May 14, 2009 at 09:59:16AM -0400, Matt McCutchen wrote: > On Thu, 2009-05-14 at 15:59 +0300, Amir Rapson wrote: > > Running rsync with --whole-file yields poorer performance results than > > a simple "cp". > > Looking at the code - it looks like "copy_file" isn't really called > > when I add the --whole-file flag. The regular "receive_data" is doing > > the copy. > > That's right. In the current design, file transfers are always done > over the connection between the sender and receiver and copy_file is > used only for copies within the destination (which happen with a few > options such as --copy-dest). It would be possible to make a special > case for local runs and have the receiver open the source file directly > (or something like that), but this has not been done yet. Also note that --whole-file is the default if both source and destination are local. Compare it to --no-whole-file and you'll see worse performance. - Dave -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --link-dest option with the destination directory containing old files.
On Tue, Dec 23, 2008 at 01:52:21PM -0500, Matt McCutchen wrote: > On Tue, 2008-12-23 at 23:45 +1100, David Overton wrote: > > I would also very much like to see this feature. Indeed, this seems > > far more logical than the current --link-dest behaviour and it's what > > I assumed --link-dest would do until I read the man page thoroughly > > (you have to follow the references back from --link-dest to > > --copy-dest and then --compare-dest and even then the only mention of > > the actual behaviour is a parenthesised comment "(if the files are > > missing in the destination directory)"). I be interested to know what > > use cases the current behaviour was designed for, because I can't > > see any advantage to not making use of the --link-dest file if it's > > available. Providing the proposed alternative behaviour as an extra > > option, if not the default for --link-dest, would be very useful. > > I too find the current semantics of basis dirs illogical and would > support an option for better semantics (though I think we should avoid > changing defaults some users may be relying on). That's why I entered a > series of enhancement requests, the first of which is Robert's proposal: > > https://bugzilla.samba.org/show_bug.cgi?id=5644 > https://bugzilla.samba.org/show_bug.cgi?id=5645 > https://bugzilla.samba.org/show_bug.cgi?id=5646 > > I would consider it worth forking rsync to have these features, if I > ever got around to doing so. It (that is, the proposal in your bug first report) sounds like not a very difficult or intrusive addition to me, and useful for rotating backups. Wayne said he wouldn't write it, but maybe he'd accept it if someone else wrote it. That would be better than forking unless it is more intrusive than it seems. (I'm taking an interest because I contributed the code for the original --*-dest option --compare-dest and that is primarily what pulled me in to maintaining rsync for a year or so. I also integrated JW Schultz's --link-dest. --*dest options have evolved quite a bit since then I see.) - Dave -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --link-dest option with the destination directory containing old files.
On Tue, Dec 23, 2008 at 11:45:36PM +1100, David Overton wrote: > 2008/12/23 Robert Bell : > > It seems to us that there is a good case for the enhanced functionality. > > > > "When a file is found in the destination which should be replaced by one > > in the source, look in the --link-dest directory first for a candidate, > > and hard-link that in preference to doing a copy from source to > > destination." > > I would also very much like to see this feature. Indeed, this seems > far more logical than the current --link-dest behaviour and it's what > I assumed --link-dest would do until I read the man page thoroughly > (you have to follow the references back from --link-dest to > --copy-dest and then --compare-dest and even then the only mention of > the actual behaviour is a parenthesised comment "(if the files are > missing in the destination directory)"). I be interested to know what > use cases the current behaviour was designed for, because I can't > see any advantage to not making use of the --link-dest file if it's > available. Providing the proposed alternative behaviour as an extra > option, if not the default for --link-dest, would be very useful. That's a parenthetical comment in --compare-dest because that's normal rsync behavior. If a file already exists at the destination, rsync assumes as normal that it has to compare against it when copying the file from the origin. The *-dest options only come into effect when the destination file is missing. You're asking to have an option that will always ignore the destination file, then behave as --link-dest does now, and then overwrite the destination? I don't understand what's the point of doing link-dest if you're not going to keep around a linked copy of the entire directory tree, so I don't see why you would right away rm -r an old tree. Describe more fully how you want to the disk to be left after the operation and what you're trying to accomplish. Depending on what you're trying to do, there may be another combination you can use. Rob gave the example of rsync -a --link-dest=../backup.1 source/ backup.0/ where backup.0 already had files in it. Is it that you're trying to alternate between backup.0 and backup.1 so you have two complete backups with as much hardlinked between them as you can? - Dave -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync feature needed: preserve atime
On Tue, May 08, 2007 at 07:09:06PM -0400, Matt McCutchen wrote: > On 5/8/07, Brent Thompson <[EMAIL PROTECTED]> wrote: > >Often we need to preserve the information atime conveys, but I have found > >no way to get rsync to preserve this, nor any hint it is being worked on. > >It would be great if 'rsync -t' also set atime not just mtime -- or maybe a > >new option is desired for atime. > > The standard version of rsync does not support preserving atimes, but > the rsync source distribution includes a patch "patches/atimes.diff" > that adds an option --atimes to preserve atimes. I recommend you > compile your own copy of rsync including this patch (reply if you need > help/instructions for this) and then use it with its --atimes option. What does it really mean to preserve access times? When rsync reads a file to copy it, it will change the access time just because it is reading it, so the backup should then have the correct access time, the time the backup file was created. Some tools have an option to reset the access time of the file they copy, but in so doing the tools update the inode change time of the input file which is generally more important to users than access time. It is not possible to preserve both as a filesystem user like rsync. - Dave -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
dwd signing off rsync list
I've had the good fortune of getting a short-term job, but it is going to be very intense and I won't have time for the rsync mailing list. I'm sure the rest of the rsync team will be able to cover most of the questions and future updates, and I've told them they can contact me if they need my help. I've currently got two good prospects for a long term job, and with one of them I may be able to have some involvement with rsync but with the other I definitely won't. We'll see what happens. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: [PATCH] open O_TEXT and O_BINARY for cygwin/windows
On Tue, Jan 28, 2003 at 01:26:32PM -, Max Bowsher wrote: > Dave Dykstra wrote: > > I give up. The msleep(100) consistently causes hangs of the > > unsafe-links test on my friend's cygwin Windows 2000 machine. I > > suspect it's also the reason why the build.samba.org cygwin machine > > hasn't reported a result in the last 9 hours. I'm taking it back out > > and washing my hands of the cygwin rsync port, I'm fed up. > > You've been most considerate in actively seeking out and merging Cygwin > patches for this release. Alas, it backfired in merging the msleep thing. I > definitely agree with it not being in the upstream sources. Making a race > condition harder to hit, with a simple delay, is obviously not a fix. You > may be pleased to know that the SHUTDOWN_ALL_SOCKETS change has eliminated > "Connection reset by peer" messages when running the testsuite. Well it's good to know that one of the cygwin changes did some good. > It may be that the hang has just vanished. If it hasn't, it's probably a bug > in Cygwin, not rsync. Hopefully we'll get a nice bugreport, and it can be > traced and squashed. Hopefully. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Announcing rsync release 2.5.6
Rsync release 2.5.6 is now available at http://ftp.samba.org/ftp/rsync/rsync-2.5.6.tar.gz ftp://ftp.samba.org/pub/rsync/rsync-2.5.6.tar.gz rsync://ftp.samba.org/ftp/rsync/rsync-2.5.6.tar.gz There is a '.sig' file at corresponding URLs with a gpg signature; the public key is available on the pgp keyservers. NEWS for rsync version 2.5.6, aka the dwd-between-jobs release Changes since version 2.5.5: ENHANCEMENTS: * The --delete-after option now implies --delete. (Wayne Davison) * The --suffix option can now be used with --backup-dir. (Michael Zimmerman) * Combining "::" syntax with the -rsh/-e option now uses the specified remote-shell as a transport to talk to a (newly-spawned) server-daemon. This allows someone to use daemon features, such as modules, over a secure protocol, such as ssh. (JD Paul) * The rsync:// syntax for daemon connections is now accepted in the destination field. * If the file name given to --include-from or --exclude-from is "-", rsync will read from standard input. (J.W. Schultz) * New option --link-dest which is like --compare-dest except that unchanged files are hard-linked in to the destination directory. (J.W. Schultz) * Don't report an error if an excluded file disappears during an rsync run. (Eugene Chupriyanov and Bo Kersey) * Added .svn to --cvs-exclude list to support subversion. (Jon Middleton) * Properly support IPv6 addresses in the rsyncd.conf "hosts allow" and "hosts deny" fields. (Hideaki Yoshifuji) * Changed exclude file handling to permit DOS or MAC style line terminations. (J.W. Schultz) * Ignore errors from chmod when -p/-a/--preserve-perms is not set. (Dave Dykstra) BUG FIXES: * Fix "forward name lookup failed" errors on AIX 4.3.3. (John L. Allen, Martin Pool) * Generate each file's rolling-checksum data as we send it, not in a separate (memory-eating) pass before hand. This prevents timeout errors on really large files. (Stefan Nehlsen) * Fix compilation on Tru64. (Albert Chin, Zoong Pham) * Better handling of some client-server errors. (Martin Pool) * Fixed a crash that would occur when sending a list of files that contains a duplicate name (if it sorts to the end of the file list) and using --delete. (Wayne Davison) * Fixed the file-name duplicate-removal code when dealing with multiple dups in a row. (Wayne Davison) * Fixed a bug that caused rsync to lose the exit status of its child processes and sometimes return an exit code of 0 instead of showing an error. (David R. Staples, Dave Dykstra) * Fixed bug in --copy-unsafe-links that caused it to be completely broken. (Dave Dykstra) * Prevent infinite recursion in cleanup code under certain circumstances. (Sviatoslav Sviridov and Marc Espie) * Fixed a bug that prevented rsync from creating intervening directories when --relative-paths/-R is set. (Craig Barratt) * Prevent "Connection reset by peer" messages from Cygwin. (Randy O'Meara) INTERNAL: * Many code cleanups and improved internal documentation. (Martin Pool, Nelson Beebe) * Portability fixes. (Dave Dykstra and Wayne Davison) * More test cases. (Martin Pool) * Some test-case fixes. (Brian Poole, Wayne Davison) * Updated included popt to the latest vendor drop, version 1.6.4. (Jos Backus) * Updated config.guess and config.sub to latest versions; this means rsync should build on more platforms. (Paul Green) - Dave Dykstra msg06318/pgp0.pgp Description: PGP signature -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: [PATCH] open O_TEXT and O_BINARY for cygwin/windows
On Mon, Jan 27, 2003 at 04:01:46PM -0600, Dave Dykstra wrote: > On Mon, Jan 27, 2003 at 04:45:25PM -, Max Bowsher wrote: ... > > Cygwin has gone through many minor versions since then. I suggest releasing > > an rsync without the hack, but with a command line option to turn it on. > > That way, we get to see for certain whether the problem still exists, and > > there is an easy workaround available if it does. > > So many people have complained about Cygwin hangs that I'm inclined to > leave the msleep(100) in on this release by default, but only on Cygwin. > When the problem is shown to be fixed on Cygwin, it can be taken back > out. I give up. The msleep(100) consistently causes hangs of the unsafe-links test on my friend's cygwin Windows 2000 machine. I suspect it's also the reason why the build.samba.org cygwin machine hasn't reported a result in the last 9 hours. I'm taking it back out and washing my hands of the cygwin rsync port, I'm fed up. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Cygwin issues: modify-window and hangs
Thanks, you saved me some trouble. I put it in. - Dave On Mon, Jan 27, 2003 at 06:52:30PM -0800, jw schultz wrote: > On Mon, Jan 27, 2003 at 03:57:16PM -0600, Dave Dykstra wrote: > > On Mon, Jan 27, 2003 at 04:38:09PM -, Max Bowsher wrote: > > > Dave Dykstra wrote: > > > > This modify-window default of 1 has been causing some trouble on the > > > > rsync test suite on the Cygwin test machine on build.samba.org. The > > > > problem is that some files get created and immediately copied within > > > > one second, and then the rsync code that implements '-p' checks to > > > > see if the copied file's time is within one second before deciding > > > > whether or not to change it. The test machine is presumably using an > > > > NTFS filesystem so it has one second granuarlity. > > > > > > Yes, noticed this running "make check" last night. However, I also got > > > random (i.e. re-run the testsuite and get different results) failures of > > > some other tests. Is this showing up in your test machine? > > > > I did notice strange failure of the "hands" test on the build.samba.org > > Cygwin test machine. > > > > > > Last night I considered 4 possible solutions: > > > > 1. Change the test suite to always wait 2 seconds before copying, > > > > at > > > > the beginning of the "checkit" function. > > > > 2. Change the test suite to always pass --modify-window=0 to > > > > rsync by including it in the $RSYNC variable in testsuite/rsync.fns. > > > > 3. Change the set_perms() function in rsync.c to check for exact > > > > time rather than calling cmp_modtime. > > > > 4. Back out the default of --modify-window=1 on cygwin and go > > > > back to > > > > a default of 0. > > > > > > > > I implemented solution 1, but I'm not very comfortable with it because > > > > it slows the tests on all platforms. I'm now leaning toward solution > > > > 3. Discussion? > > > > > > How about 4? It seems wrong to penalize NTFS for a FAT problem, especially > > > as NTFS is now the preferred Windows filesystem. > > > > > > Unless someone feels like making a FAT-detection patch, the previous status > > > quo looks to me like the best option. > > > > > > You've talked me into it. > > If you are going to revert cygwin to a --modify-window=0 > default we should have a notice in the bugs section of the > manpage. > > I've attached a patch that adds this, improves (in my > opinion) the --modify-window section slightly, removes > reference to a bug tracking system and updates the version > the manpage reports current with. > > -- > > J.W. SchultzPegasystems Technologies > email address: [EMAIL PROTECTED] > > Remember Cernan and Schmitt > Index: rsync.yo > === > RCS file: /cvsroot/rsync/rsync.yo,v > retrieving revision 1.113 > diff -u -b -r1.113 rsync.yo > --- rsync.yo 27 Jan 2003 03:13:46 - 1.113 > +++ rsync.yo 28 Jan 2003 01:56:31 - > @@ -396,8 +396,8 @@ > the timestamps as being equal if they are within the value of > modify_window. This is normally zero, but you may find it useful to > set this to a larger value in some situations. In particular, when > -transferring to/from FAT filesystems which cannot represent times with > -a 1 second resolution this option is useful. > +transferring to FAT filesystems which cannot represent times with > +a 1 second resolution --modify-window=1 is useful. > > dit(bf(-c, --checksum)) This forces the sender to checksum all files using > a 128-bit MD4 checksum before transfer. The checksum is then > @@ -1110,16 +1110,20 @@ > > times are transferred as unix time_t values > > +When transferring to FAT filesystmes rsync may resync > +unmodified files. > +See the comments on the --modify-window option. > + > file permissions, devices etc are transferred as native numerical > values > > see also the comments on the --delete option > > -Please report bugs! The rsync bug tracking system is online at > -url(http://rsync.samba.org/rsync/)(http://rsync.samba.org/rsync/) > +Please report bugs! See the website at > +url(http://rsync.samba.org/)(http://rsync.samba.org/) > > manpagesection(VERSION) > -This man page is current for version 2.0 of rsync > +This man page is current for version 2.5.6 of rsync > > manpagesection(CREDITS) > -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: [PATCH] open O_TEXT and O_BINARY for cygwin/windows
On Mon, Jan 27, 2003 at 04:45:25PM -, Max Bowsher wrote: > Dave Dykstra wrote: > > On Mon, Jan 27, 2003 at 10:46:16AM +0100, Lapo Luchini wrote: > >> I can test that before releasing cygwin's 2.5.6-1 package, > >> unfortunately I have almost *no* free time until after friday > >> (university test on thursday ^_^) > >> Anyway Max is also right, local issues can be resolved locally with > >> much speed, even if "in the long run" it's of course better to put > >> them in rsync cvs too. > > > > I'm reluctant to rely on cygwin-stream-specific patches because I know > > that other people compile it from source too. Probably the msleep > > should be put in for Cygwin only with an ifdef though, or the waiting > > of the child removed altogether on Cygwin only. > > Does it not make sense that anyone wishing to compile their own rsync for > Cygwin should start by looking at the Cygwin package? (Which, btw, includes > the clean rsync source, and patchfiles, so it is easy to see what is being > done to rsync specially for Cygwin). In my case I used to use the same source code to compile on many different platforms and no, I wouldn't look at the Cygwin package first to compile on Cygwin. > The waiting for the child is presumably there for a reason. If is removed > for Cygwin, it should probably be removed for all platforms. If there is an > argument against that, then it applies equally to Cygwin as to any other > platforms. Perhaps the waiting in the child was put in for specific problems on some platforms. > >> BTW: I wonder if the sleepms(30) is still needed, it's quite a bit of > >> time that my "build.samba.org" hourly cron doesn't produce a "hanged > >> rsync"... (must check that too) > > > > I know that some people still experience it, and I don't think > > anything has changed in that area since the time that the > > person who experienced the hang discovered the hack. > > Cygwin has gone through many minor versions since then. I suggest releasing > an rsync without the hack, but with a command line option to turn it on. > That way, we get to see for certain whether the problem still exists, and > there is an easy workaround available if it does. So many people have complained about Cygwin hangs that I'm inclined to leave the msleep(100) in on this release by default, but only on Cygwin. When the problem is shown to be fixed on Cygwin, it can be taken back out. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Cygwin issues: modify-window and hangs
On Mon, Jan 27, 2003 at 04:38:09PM -, Max Bowsher wrote: > Dave Dykstra wrote: > > This modify-window default of 1 has been causing some trouble on the > > rsync test suite on the Cygwin test machine on build.samba.org. The > > problem is that some files get created and immediately copied within > > one second, and then the rsync code that implements '-p' checks to > > see if the copied file's time is within one second before deciding > > whether or not to change it. The test machine is presumably using an > > NTFS filesystem so it has one second granuarlity. > > Yes, noticed this running "make check" last night. However, I also got > random (i.e. re-run the testsuite and get different results) failures of > some other tests. Is this showing up in your test machine? I did notice strange failure of the "hands" test on the build.samba.org Cygwin test machine. > > Last night I considered 4 possible solutions: > > 1. Change the test suite to always wait 2 seconds before copying, > > at > > the beginning of the "checkit" function. > > 2. Change the test suite to always pass --modify-window=0 to > > rsync by including it in the $RSYNC variable in testsuite/rsync.fns. > > 3. Change the set_perms() function in rsync.c to check for exact > > time rather than calling cmp_modtime. > > 4. Back out the default of --modify-window=1 on cygwin and go > > back to > > a default of 0. > > > > I implemented solution 1, but I'm not very comfortable with it because > > it slows the tests on all platforms. I'm now leaning toward solution > > 3. Discussion? > > How about 4? It seems wrong to penalize NTFS for a FAT problem, especially > as NTFS is now the preferred Windows filesystem. > > Unless someone feels like making a FAT-detection patch, the previous status > quo looks to me like the best option. You've talked me into it. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Cygwin issues: modify-window and hangs
On Fri, Jan 24, 2003 at 05:18:07PM -0600, Dave Dykstra wrote: > I had a friend run some Cygwin tests and we found that --modify-window=1 > works just as well as --modify-window=2 on FAT filesystems to copy files > from Unix and detect the difference in granularity. FAT filesystems always > have timestamps that have an even number of seconds. On the other hand, > NTFS filesystems can store the modification time down to the second, > whereas previously people on this mailing list thought it was just like > FAT for modification time. Nevertheless, I decided to change the default > value for --modify-window to 1 on Cygwin rather than 0, under the thinking > that even on NTFS filesystems it wouldn't do any harm because it's highly > unlikely that any individual file will be modified again within one second > after it is copied by rsync. This modify-window default of 1 has been causing some trouble on the rsync test suite on the Cygwin test machine on build.samba.org. The problem is that some files get created and immediately copied within one second, and then the rsync code that implements '-p' checks to see if the copied file's time is within one second before deciding whether or not to change it. The test machine is presumably using an NTFS filesystem so it has one second granuarlity. Last night I considered 4 possible solutions: 1. Change the test suite to always wait 2 seconds before copying, at the beginning of the "checkit" function. 2. Change the test suite to always pass --modify-window=0 to rsync by including it in the $RSYNC variable in testsuite/rsync.fns. 3. Change the set_perms() function in rsync.c to check for exact time rather than calling cmp_modtime. 4. Back out the default of --modify-window=1 on cygwin and go back to a default of 0. I implemented solution 1, but I'm not very comfortable with it because it slows the tests on all platforms. I'm now leaning toward solution 3. Discussion? - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: [PATCH] open O_TEXT and O_BINARY for cygwin/windows
On Mon, Jan 27, 2003 at 10:46:16AM +0100, Lapo Luchini wrote: > Dave Dykstra wrote: > > >>What, in particular? I'm not a very good testcase, because I use binary > >>mounts and unix line endings everywhere. > >> > >>It compiles and does syncs with remote rsync daemons, which is my normal > >>usage. > >> > >>Max. > >> > >> > >See if exclude files with DOS line endings work ok for you, and also the > >other config files if you can. > > > >- Dave > > > > > I can test that before releasing cygwin's 2.5.6-1 package, unfortunately > I have almost *no* free time until after friday (university test on > thursday ^_^) > Anyway Max is also right, local issues can be resolved locally with much > speed, even if "in the long run" it's of course better to put them in > rsync cvs too. I'm reluctant to rely on cygwin-stream-specific patches because I know that other people compile it from source too. Probably the msleep should be put in for Cygwin only with an ifdef though, or the waiting of the child removed altogether on Cygwin only. > BTW: I wonder if the sleepms(30) is still needed, it's quite a bit of > time that my "build.samba.org" hourly cron doesn't produce a "hanged > rsync"... (must check that too) I know that some people still experience it, and I don't think anything has changed in that area since the time that the person who experienced the hang discovered the hack. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
one more day of delay for release of 2.5.6
Well I was hoping to get the 2.5.6 out today, but I think I made too many Cygwin changes this evening for comfort and I'd like to allow one more day of testing. Cygwin users, please test as much as you can and post any problems. - Dave Dykstra -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Cygwin issues: modify-window and hangs
On Sat, Jan 25, 2003 at 06:32:08PM +0100, Greger Cronquist wrote: > --- Max Bowsher <[EMAIL PROTECTED]> skrev: > Dave Dykstra > wrote: > > > I'm using the current Cygwin release > > (rsync-2.5.5-2). That is rsync-2.5.5, > > with an added msleep(30) which is intended to deal > > with a possible problem > > with signals in Cygwin. > > Making that msleep(100) works even better for me. Ah, I found the messages that were discussing this and I see this was supposed to prevent hangs at the ends of runs. I went ahead and put an msleep(100) in before the final kill...USR2 in main.c do_recv(). I also finally remembered that the calls to shutdown() were for "Connection reset by peer" errors and not hangs. I went ahead and put in a define SHUTDOWN_ALL_SOCKETS which is only set by configure on Cygwin and call shutdown when it is set. Hopefully that will get rid of messages. - Dave Dykstra -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Storage compression patch for Rsync (unfinished)
On Sun, Jan 26, 2003 at 02:46:43PM -0800, Craig Barratt wrote: > > Is there any reason why caching programs would need to set the > > value, rather than it just being a fixed value? > > I think it is hard to describe what this is for and what it should be > > set to. Maybe a --fixed-checksum-seed option would make some sense, > > or for a caching mechanism to be built in to rsync if it is shown to > > be very valuable. > > A fixed value would be perfectly ok; the same magic value that batch > mode uses (32761) would make sense. > > > I know people have proposed some caching mechanisms in the past and > > they've been rejected for one reason or another. > > One difficulty is that additional files, or new file formats, are needed > for storing the checksums, and that moves rsync further away from its > core purpose. > > > I don't think I'll include the option in 2.5.6. > > If I submitted a new patch with --fixed-checksum-seed, would you be > willing to at least add it to the patches directory for 2.5.6? > > I will be adding block and file checksum caching to BackupPC, and > that needs --fixed-checksum-seed. This will save me from providing > a customized rsync (or rsync patches) as part of BackupPC; I would > much rather tell people to get a vanilla 2.5.6 rsync release and > apply the specific patch that comes with the release. Sorry, but I don't think it would be good to do even that until we've all had a chance to look at what's involved in the caching and whether or not it would make better sense to have it be a modification to rsync rather than mostly external. I'm concerned that people might misuse the option without understanding the consequences. You could always keep the patch on the BackupPC web site in the meantime. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Storage compression patch for Rsync (unfinished)
Is there any reason why caching programs would need to set the
value, rather than it just being a fixed value? I think it is hard
to describe what this is for and what it should be set to. Maybe a
--fixed-checksum-seed option would make some sense, or for a caching
mechanism to be built in to rsync if it is shown to be very valuable.
I don't think I'll include the option in 2.5.6. I know people have
proposed some caching mechanisms in the past and they've been rejected
for one reason or another.
- Dave
On Fri, Jan 17, 2003 at 11:19:35PM -0800, Craig Barratt wrote:
> > While the idea of rsyncing with compression is mildly
> > attractive i can't say i care for the new compression
> > format. It would be better just to use the standard gzip or
> > other format. If you are going to create a new file type
> > you could at least discuss storing the blocksums in it so
> > that the receiver wouldn't have to generate them.
>
> Yes! Caching the block checksums and file checksums could yield a large
> improvement for the receiver. However, an integer checksum seed is used
> in each block and file MD4 checksum. The default value is unix time() on
> the server, sent to the client at startup.
>
> So currently you can't cache block and file checksums (technically it is
> possible for block checksums since the checksum seed is appended at the
> end of each block, so you could cache the MD4 state prior to the checksum
> seed being added; for files you can't since the checksum seed is at the
> start).
>
> Enter a new option, --checksum-seed=NUM, that allows the checksum seed to
> be fixed. I've attached a patch below against 2.5.6pre1.
>
> The motivation for this is that BackupPC (http://backuppc.sourceforge.net)
> will shortly release rsync support, and I plan to support caching
> block and file checksums (in addition to the existing compression,
> hardlinking among any identical files etc). So it would be really
> great if this patch, or something similar, could make it into 2.5.6
> or at a minimum the contributed patch area in 2.5.6.
>
> [Also, this option is convenient for debugging because it makes the
> rsync traffic identical between runs, assuming the file states at
> each end are the same too.]
>
> Thanks,
> Craig
>
> ###
> diff -bur rsync-2.5.6pre1/checksum.c rsync-2.5.6pre1-csum/checksum.c
> --- rsync-2.5.6pre1/checksum.cMon Apr 8 01:31:57 2002
> +++ rsync-2.5.6pre1-csum/checksum.c Thu Jan 16 23:38:47 2003
> @@ -23,7 +23,7 @@
>
> #define CSUM_CHUNK 64
>
> -int checksum_seed = 0;
> +extern int checksum_seed;
> extern int remote_version;
>
> /*
> diff -bur rsync-2.5.6pre1/compat.c rsync-2.5.6pre1-csum/compat.c
> --- rsync-2.5.6pre1/compat.c Sun Apr 7 20:50:13 2002
> +++ rsync-2.5.6pre1-csum/compat.c Fri Jan 17 21:18:35 2003
> @@ -35,7 +35,7 @@
> extern int preserve_times;
> extern int always_checksum;
> extern int checksum_seed;
> -
> +extern int checksum_seed_set;
>
> extern int remote_version;
> extern int verbose;
> @@ -64,11 +64,14 @@
>
> if (remote_version >= 12) {
> if (am_server) {
> - if (read_batch || write_batch) /* dw */
> + if (read_batch || write_batch) { /* dw */
> + if ( !checksum_seed_set )
> checksum_seed = 32761;
> - else
> + } else {
> + if ( !checksum_seed_set )
> checksum_seed = time(NULL);
> write_int(f_out,checksum_seed);
> + }
> } else {
> checksum_seed = read_int(f_in);
> }
> diff -bur rsync-2.5.6pre1/options.c rsync-2.5.6pre1-csum/options.c
> --- rsync-2.5.6pre1/options.c Fri Jan 10 17:30:11 2003
> +++ rsync-2.5.6pre1-csum/options.cThu Jan 16 23:39:17 2003
> @@ -116,6 +116,8 @@
> char *backup_dir = NULL;
> int rsync_port = RSYNC_PORT;
> int link_dest = 0;
> +int checksum_seed = 0;
> +int checksum_seed_set;
>
> int verbose = 0;
> int quiet = 0;
> @@ -274,6 +276,7 @@
>rprintf(F," --bwlimit=KBPS limit I/O bandwidth, KBytes per
>second\n");
>rprintf(F," --write-batch=PREFIXwrite batch fileset starting with
>PREFIX\n");
>rprintf(F," --read-batch=PREFIX read batch fileset starting with
>PREFIX\n");
> + rprintf(F," --checksum-seed=NUM set MD4 checksum seed\n");
>rprintf(F," -h, --help show this help screen\n");
> #ifdef INET6
>rprintf(F," -4 prefer IPv4\n");
> @@ -293,7 +296,7 @@
>OPT_COPY_UNSAFE_LINKS, OPT_SAFE_LINKS, OPT_COMPARE_DEST, OPT_LINK_DEST,
>OPT_LOG_FORMAT, OPT_PASSWORD_FILE, OPT_SIZE_ONLY, OPT_ADDRESS,
>OPT_DELETE_AFTER, OPT_EXISTING, OPT_MAX_DELETE, OPT_BACKUP_DIR,
> - OPT_IGNORE_ERRORS, OPT_BWLIMIT, OPT_BLOCKING_IO,
> + OPT_IGNORE_ERRORS, OPT_BWLIMIT, OPT_B
Re: [PATCH] open O_TEXT and O_BINARY for cygwin/windows
On Sun, Jan 26, 2003 at 08:54:19PM -, Max Bowsher wrote: > Dave Dykstra wrote: > > Alright. Max, could you quickly verify if the latest CVS version > > works OK for you on Cygwin? > > What, in particular? I'm not a very good testcase, because I use binary > mounts and unix line endings everywhere. > > It compiles and does syncs with remote rsync daemons, which is my normal > usage. > > Max. See if exclude files with DOS line endings work ok for you, and also the other config files if you can. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: [PATCH] open O_TEXT and O_BINARY for cygwin/windows
Alright. Max, could you quickly verify if the latest CVS version works
OK for you on Cygwin?
It's better to always handle all three styles of line terminations anyway,
because there are other situations than systems that have O_TEXT defined
where it might be wanted, for example a Linux system that has a Windows
filesystem mounted.
- Dave
On Sun, Jan 26, 2003 at 12:07:07PM -0800, Wayne Davison wrote:
> On Sun, Jan 26, 2003 at 08:16:50AM -0800, jw schultz wrote:
> > authenticate.c: fd = open(fname,O_RDONLY | O_TEXT);
> > get_secret() already discards \r
> >
> > authenticate.c: if ( (fd=open(filename,O_RDONLY | O_TEXT)) == -1) {
> > getpassf() treats \r the same as \n
>
> Yeah, these already handle both Mac and PC line endings, so I removed
> the O_TEXT bit from these.
>
> > clientserver.c: FILE *f = fopen(motd,"r" O_TEXT_STR);
> > simply passes contents of motd file unchanged to client
> > does it matter if there are \r chars in the file?
>
> I'm trying to imagine what will happen for Mac systems as well. Since
> the default for files is to open them in text mode on a Mac (and they
> don't have the Cygwin weirdness of trying to override a binary-mode
> mount, right?), then I think you're right here -- the extra \r that
> might come from a PC system should hopefully cause no problems (but we
> should test this). In the meantime, I think it would be safe to revert
> this change.
>
> > exclude.c: f = fopen(fname,"r" O_TEXT_STR);
> > exclude.c: f = fdopen(0, "r" O_TEXT_STR);
> > the file is processed with
> > while (fgets(line,MAXPATHLEN,f)) {
> > int l = strlen(line);
> > if (l && line[l-1] == '\n') l--;
> > line[l] = 0;
> > if we add
> > if (l && line[l-1] == '\r') l--;
> > after the \n processing that will handle the \r\n case
> > or we can replace the \n line with
> > while (l && (line[l-1] == '\n'|| line[l-1] == '\r')) l--;
> > and this function will be covered.
>
> As long as Mac systems are reading the file in text mode without the
> O_TEXT_STR, we should be fine (and I believe that to be the case).
>
> > params.c: OpenedFile = fopen( FileName, "r" O_TEXT_STR );
> > since \r is a whitespace character it should be
> > ignored.
>
> Sounds right (though I haven't verified your analysis).
>
> Since it sounds like the "t" has a potential for problems, I'm taking
> these recent changes out and will just put in the one suggested change
> to ignore a '\r' at the end of the line. This should be safer for the
> current release.
>
> ...wayne..
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: [patch] Still a problem with cleanup.c
I couldn't reproduce the problem on Linux, but I believe you that it's
a problem. If you think about it, it's easy to see why code that called
_exit_cleanup() might behave strangely if the function returns so I
like the fix of not returning until the situation gets deeply nested.
I submitted the patch.
Thanks Marc & Brian.
- Dave Dykstra
On Thu, Jan 23, 2003 at 08:29:40PM -0500, Brian Poole wrote:
> Hello,
>
> There is still a problem with the _exit_cleanup() function in
> cleanup.c despite the patch that was put in last week that prevented
> recursion. It turns out that sometimes multiple calls in close
> sequence are done and this causes rsync to spin out of control
> instead of exiting.
>
> This bug was found by Marc Espie and the patch I'm including is
> his as well. It hasn't been commited to the OpenBSD tree yet and
> is still undergoing testing but it does fix the immediate easily
> detectable problem. His post with the bug fix was here:
>
> http://marc.theaimsgroup.com/?l=openbsd-tech&m=104238879828005&w=2
>
> I've changed the variable names to match what is in the rsync
> tree already.
>
> This bug is easily triggerable with a local rsync copy.
>
> rsync -av /some/reasonably/large/dir /tmp/somewhere
>
> Wait until after the copy has started (after it is done with
> building the file list) and then hit ctrl-c. Watch rsync fill
> the screen with messages like:
>
> rsync: connection unexpectedly closed (94819 bytes read so far)
> rsync: connection unexpectedly closed (94819 bytes read so far)
> rsync: connection unexpectedly closed (94819 bytes read so far)
>
> Curse.
> Switch to another window and kill -9 the errant rsync process.
> Apply patch and notice it no longer goes crazy.
>
>
> -b
>
> Index: cleanup.c
> ===
> RCS file: /cvsroot/rsync/cleanup.c,v
> retrieving revision 1.15
> diff -u -r1.15 cleanup.c
> --- cleanup.c 16 Jan 2003 20:09:31 - 1.15
> +++ cleanup.c 24 Jan 2003 01:24:09 -
> @@ -63,11 +63,11 @@
> extern int log_got_error;
> static int inside_cleanup = 0;
>
> - if (inside_cleanup != 0) {
> + if (inside_cleanup > 10) {
> /* prevent the occasional infinite recursion */
> return;
> }
> - inside_cleanup = 1;
> + inside_cleanup++;
>
> signal(SIGUSR1, SIG_IGN);
> signal(SIGUSR2, SIG_IGN);
> Index: cleanup.c
> ===
> RCS file: /cvsroot/rsync/cleanup.c,v
> retrieving revision 1.15
> diff -u -r1.15 cleanup.c
> --- cleanup.c 16 Jan 2003 20:09:31 - 1.15
> +++ cleanup.c 24 Jan 2003 01:24:09 -
> @@ -63,11 +63,11 @@
> extern int log_got_error;
> static int inside_cleanup = 0;
>
> - if (inside_cleanup != 0) {
> + if (inside_cleanup > 10) {
> /* prevent the occasional infinite recursion */
> return;
> }
> - inside_cleanup = 1;
> + inside_cleanup++;
>
> signal(SIGUSR1, SIG_IGN);
> signal(SIGUSR2, SIG_IGN);
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: opendir(somedir/somefile): Not enough space -- why?
On Fri, Jan 24, 2003 at 04:02:28PM -0800, jw schultz wrote: > On Fri, Jan 24, 2003 at 04:49:20PM -0700, Bill Geddes wrote: > > Am I running out of memory on the source server? Seems perhaps more > > plausible than on the client - it has nothing going on and 1Gb RAM. > > The server has rsync version 2.5.5. The client has 2.5.6. > > Yes, it looks like the sender is running out of memory. > Break the job up. The important factor is the number of files. Rsync uses up some memory for every file it looks at in a run. That's why JW said to break it up into smaller jobs. > > I am currently mounting the server directory via NFS, and using rsync to > > copy from the mounted directory to the target. Slow, but it is running! > > If you continue over NFS be sure you do checksums of > everything afterward. NFS has (or at least used to) a > measurable error rate. Rsync is optimized for a slow "network" and high speed "disk" access. If you're doing NFS over a slow network, that looks to rsync as a slow "disk". If you can, it's probably better to run rsync directly on the server. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Cygwin issues: modify-window and hangs
On Sat, Jan 25, 2003 at 12:10:39AM -, Max Bowsher wrote: > Dave Dykstra wrote: > > While doing the tests we too experienced hangs at the end of copies. > > We were going over openssh from a Solaris 9 box to Windows 2000 > > Cygwin. > > We tried the test from > > http://lists.samba.org/pipermail/rsync/2002-August/008130.html > > but it still experienced hangs. It wasn't clear if the patch reduced > > the frequency or not. > > Data point: I regularly rsync from sources.redhat.com (Linux, I presume. Not > Cygwin, anyway) to my local machine (Cygwin). I have never experienced a > hang. Since you posted the above message, does that mean you used to experience hangs and that the rsyncs you do now are with the patch in? > > Has *anybody* been able to figure out a fix for this that really > > works? > > It sure would be nice to get something into 2.5.6 but we're about out > > of time for that because I need to put it out this weekend before I > > start a new (temporary) job on Monday. I imagine that the issue I > > was experiencing could be a separate one and this patch really would > > help other hangs people are having; can anybody give me an argument > > for putting in the calls to "shutdown()" anyway? I would make it > > only happen on Cygwin, because it is unknown if they will cause harm > > on other platforms. > > Those platforms would have to be fairly broken. The behaviour of shutdown is > pretty clearly defined. It is a relatively rarely used function, however, so it probably doesn't get much testing. Cygwin/windows is what's really broken and I'd rather not risk all the other ports to suit it. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Cygwin issues: modify-window and hangs
I had a friend run some Cygwin tests and we found that --modify-window=1 works just as well as --modify-window=2 on FAT filesystems to copy files from Unix and detect the difference in granularity. FAT filesystems always have timestamps that have an even number of seconds. On the other hand, NTFS filesystems can store the modification time down to the second, whereas previously people on this mailing list thought it was just like FAT for modification time. Nevertheless, I decided to change the default value for --modify-window to 1 on Cygwin rather than 0, under the thinking that even on NTFS filesystems it wouldn't do any harm because it's highly unlikely that any individual file will be modified again within one second after it is copied by rsync. While doing the tests we too experienced hangs at the end of copies. We were going over openssh from a Solaris 9 box to Windows 2000 Cygwin. We tried the test from http://lists.samba.org/pipermail/rsync/2002-August/008130.html but it still experienced hangs. It wasn't clear if the patch reduced the frequency or not. Has *anybody* been able to figure out a fix for this that really works? It sure would be nice to get something into 2.5.6 but we're about out of time for that because I need to put it out this weekend before I start a new (temporary) job on Monday. I imagine that the issue I was experiencing could be a separate one and this patch really would help other hangs people are having; can anybody give me an argument for putting in the calls to "shutdown()" anyway? I would make it only happen on Cygwin, because it is unknown if they will cause harm on other platforms. - Dave Dykstra -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: [PATCH] open O_TEXT and O_BINARY for cygwin/windows
On Fri, Jan 24, 2003 at 08:57:02AM +0200, Ville Herva wrote: > On Thu, Jan 23, 2003 at 01:55:40PM -0600, you [Dave Dykstra] wrote: > > Why did you skip the fopen in log.c, which appends to the log file? > > I thought about that for a while. My reasoning was that the log file is > probably read with unix/cygwin tools anyway - if not, the administrator is > free to mount the filesystem binary or ascii. So I didn't want to enforce > CR/LF on anyone. > > In general, I think forcing ascii makes more sense for reading than writing, > because it in most imaginable cases will do no harm. > > > Because of the question about whether or not the "t" is ignored on all > > platforms, I'm a little nervous about putting this into 2.5.6. > > It could be ifdefed, would it be safe to assume the following to work? > > #if !defined(O_TEXT) > #define O_TEXT 0 > #define O_TEXT_STR "" > #else > #define O_TEXT_STR "t" > #endif > #if !defined(O_BINARY) > #define O_BINARY 0 > #define O_BINARY_STR "" > #else > #define O_BINARY_STR "b" > #endif > > and then instead of > > fopen(foo, "rt"); > > do this > > fopen(foo, "r" O_TEXT_STR); > > Comments? > > > I don't have any problem with it for 2.6.0. Maybe it should be just in > > the 'patches' directory this time. > > > > Did you check to see whether any of the code that's reading config files > > is ignoring carriage returns at the end of lines anyway? > > ISTR at least the password files were not recognized and I *think* > rsync.conf was not parsed correctly when I originally wrote the patch. > > Looking at the source, though > > - exclude.c uses fgets() to read the lines from exclude file. I don't think > it handles '\r' > - param.c Ignores all '\r' in values, so rsync.conf should be fine. I'm not > sure, whether reading the file as text would be more elegant (in theory > there _could_ be other issues than the \r\n thing...) > - authentivate.c seems to skip '\r' for passwords as well. > > So I guess you are right for the most crucial part of code (as of current > rsync). I think I'll go ahead and put in your patch with the modification of using O_TEXT_STR as you suggest. I think the risk is low. I had been concerned that perhaps older CPPs might not be able to handle that syntax, but I see other places in the rsync code that are depending on it so I think it's OK. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Error message
With file sizes over 2GB I'd suspect problems with >31-bit numbers being returned from stat() et. al. The rsync version you have on AIX is very old, and more recent versions handle large files better. The first thing to do is to upgrade rsync your AIX machine. - Dave On Thu, Jan 23, 2003 at 01:44:42PM +0100, Michael Salmon wrote: > On Thursday, January 23, 2003 12:36:52 PM +0100 Boris Gegenheimer > <[EMAIL PROTECTED]> wrote: > +-- > | The versions of the different machines I use are the initiating machine : > | Sun 450 > | Solaris 8 > | Rsync 2.5.5 > | > | The machine that I try to copy from : > | IBM rs6000 > | AIX 4.3 > | Rsync 2.3.1 > | > | The command I use is > | /usr/local/bin/rsync -au -q --stats --progress > | --rsync-path=/usr/local/bin/rsync maximus:/ptsfs2 > | /diskbackup/kanal1/maximus/ > | > | The error message comes when he tries to copy that file mentioned below > | which is 14GB big it is not the filesystem or the name of the file because > | it worked with ncftpget. > | But that program has other problems that I will not mention here. > [...] > | On Thu, Jan 23, 2003 at 10:55:32AM +0100, Boris Gegenheimer wrote: > | > Hello when i get this message what is wrong: > | > > | > /PTS.0.db2inst1.NODE.CATN.20030123062324.001: Value too large > | > to be stored in data type > | > > | > Regards Boris > [...] > +-X8 > > I have also seen that message with times before 1970. This is the standard > text for EOVERFLOW. > > /Michael > -- > This space intentionally left non-blank. > -- > To unsubscribe or change options: > http://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Rsyncing of block devices to a remote file
I'm not sure why it would really need to know the size, but given that it wants it, can you open the file and lseek to the end of it? - Dave Dykstra On Thu, Jan 23, 2003 at 09:17:04AM +0100, David Heremans wrote: > Hello, > > I'm looking for some assistance in modifying the rsync code > > Situation: I used to back-up some of my (unmounted NTFS) disk partitions > remotely using the following shells construct: > dd if=/dev/hda1 | gzip | ssh me@backupmachine 'cat > > /backup-dir/hda1-backup.gz' > > But now I want to use rsync for this, since transfering 10 gig takes a litle > to long if a file of a few kB has changed :-). I looked trough the archive > and found an old mail talking about the same problem, and a response > explaining the problem of writting to a block device (since rsync writes a > copy file and then moves it over the original). But the same mail also said > that reading from it should be trivial, so I set out trying to implement > this. > > So I created a '--read-block-file' option in options.c and avoided the > 'skipping non regular file' message in generator.c > By altering all this the receiver is ready to get the data, but in the flist > structure it still says that '/dev/hda1' is still a devicefile > and that the size is 0, and so the sender does send over zero bytes > > Now here is the problem : How do I get the size of partition in this > datastructure ? Or more generique, how would I handle this for /dev/fd0, > since you can insert 1.44MB disks or 720kB disks ? > > Anybody else tried to read blockdevices over rsync already ? > > After all, sending the data of a block device doesn't seem to be as trivial > as the archived mail sugested :-) > > David > > -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: [PATCH] open O_TEXT and O_BINARY for cygwin/windows
Why did you skip the fopen in log.c, which appends to the log file?
Because of the question about whether or not the "t" is ignored on all
platforms, I'm a little nervous about putting this into 2.5.6. I don't
have any problem with it for 2.6.0. Maybe it should be just in the
'patches' directory this time.
Did you check to see whether any of the code that's reading config files
is ignoring carriage returns at the end of lines anyway?
- Dave
On Thu, Jan 23, 2003 at 09:49:02AM +0200, Ville Herva wrote:
> > > perhaps you would want to consider the "force binary open for data files
> > > on CYGWIN" -patch I sent ages ago? I feel it's quite important, and to
> > > me it never makes sense to open data files in ascii (CR/LF translation
> > > mode) nor config files in BINARY mode (opening them in ascii makes it
> > > possible to edit them with both NOTEPAD etc and the unix line ending
> > > compliant cygwin editors.)
> > >
> > > See the thread starting at:
> > >
> > >
>http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=utf-8&threadm=a4vfnh%242ec%241%40FreeBSD.csie.NCTU.edu.tw&rnum=1&prev=/groups%3Fq%3Drsync%2Bon%2Bcygwin%2B-%2Btextmode%2Bconfig%2Bfiles%26num%3D50%26hl%3Den%26lr%3D%26ie%3DUTF-8%26oe%3Dutf-8%26sa%3DN%26tab%3Dwg
> > >
> > > I'm mostly talking about (4) - others are not that important.
>
> On Wed, Jan 22, 2003 at 03:53:27PM -0600, you [Dave Dykstra] wrote:
> > Could you please port the patch to 2.5.6pre2 and post it to the mailing
> > list?
>
> Here it is.
>
> In short: on Cygwin you can mount volumes as text or binary. For text mounts
> CR/LF->CR translation is done on the fly, binary mounts do not alter the
> data. For applications that deal with raw data (gzip, tar, md5sum, cp,
> rsync) it makes sense to access the data in binary always (and actually
> rsync does nowadays, the O_BINARY flag is set in do_open). On the other hand
> it makes sense to open config files in text so that the user can change
> them with windows editors - the CR/LF translation will do no harm even if
> the file has unix line endings.
>
> This batch adds a few O_TEXT and O_BINARY flags to *open() calls - files
> that are obviously text are opened as such; data files are opened as
> binary.
>
> - Files batch.c checksum.c generator.c receiver.c sender.c util.c use
> do_open which does O_BINARY already (which has chanced since I originally
> made the patch years ago), so no need to touch those. Other than that
> I went through all open, fdopen, and fopen calls and tried to
> add either O_TEXT or O_BINARY where it made sense.
> - O_BINARY and O_TEXT are defined in rsync.h unless they are available
> on the platform:
>
>+#if !defined(O_TEXT)
>+#define O_TEXT 0
>+#endif
>+#if !defined(O_BINARY)
>+#define O_BINARY 0
>+#endif
>
> - I also use fopen(filename, "rt"). I _think_ that all fopen implementations
> should ignore "t" unless they recognize it. At least linux does (it
> seems to ignore all chars it doesn't recognize. "rt" is one of the
> alternatives Chris Faylor recommends for CYGWIN:
>
> http://www.cygwin.com/ml/cygwin/2000-10/msg00213.html:
> > You do that one of three ways:
> >
> > open ("foo", O_RDONLY | O_TEXT);
> > fopen ("foo", "rt");
> > setmode (fd, O_TEXT);
>
> "rt" one is the only one for FILEs (and setmode doesn't even exists on
> all other platforms AFAICT.
>
> man fopen says:
>
>The mode string can also include the letter ``b'' either
>as a last character or as a character between the characÂ
>ters in any of the two-character strings described above.
>This is strictly for compatibility with ANSI X3.159-1989
>(``ANSI C'') and has no effect; the ``b'' is ignored on
>all POSIX conforming systems, including Linux. (Other
>systems may treat text files and binary files differently,
>and adding the ``b'' may be a good idea if you do I/O to a
>binary file and expect that your program may be ported to
>non-Unix environments.)
>
> So I would hope that "t" is handled the same way.
>
> Anyway, if someone knows better, please tell.
>
> Patch attached - compiled, briefly tested (the previous incarnations tested
> pretty extensively.) It doesn't alter the behaviour on platforms that don't
> do O_BINARY and O_TEXT differently so I thinks it should be quite safe to
> apply.
>
>
Re: Please test rsync-2.5.6pre2
On Tue, Jan 21, 2003 at 08:00:51PM -0801, Jos Backus wrote: > On Mon, Jan 20, 2003 at 10:19:42PM -0600, Dave Dykstra wrote: > > The second rsync-2.5.6 pre-release version is now available at: > > There's the popt-1.7 update (with Dave's sprintf() workaround) at > > http://www.catnook.com/patches/rsync-popt-1.7.diff > > Perhaps we can apply this after the release as it is a very low priority > update; I'd just like it not to rot :) Yes, it should go in after the 2.5.6 release. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Rsyncd - Help me
Look for errors in /var/log/rsyncd.log. For security reasons they are not all passed to the client. - Dave On Wed, Jan 22, 2003 at 06:52:33PM -0200, Breno Cardoso Perucchi wrote: > Hello, > > I'm trying to do my rsync to copy a mirror of my site in another webserver. > I am trying to use RSYNCD . > How I'll do for my rsync to copy my site website in the mirror ? > Which is a command to copy the mirror with the password ? > I've this websites about rsync > http://sunsite.dk/info/guides/rsync/rsync-mirroring.html > http://rsync.samba.org > > Somebody could help me? or Link ? Doc ? Howto ? > > > This is my rsyncd.conf > /usr/local/etc/rsyncd.conf > > log file = /var/log/rsyncd.log > pid file = /var/run/rsyncd.pid > lock file = /var/run/rsync.lock > > [rsyncd] >path = /usr/local/www/data-dist/ >comment = Rsync Server >use chroot = no >max connections = 4 >uid = nobody >gid = nobody >read only = yes >list = yes >auth users = rsyncd >secrets file = /usr/local/etc/rsyncd.scrt > > /usr/local/etc/rsyncd.scrt > rsyncd:rsyncd > > bash-2.05b# rsync -a 10.0.0.1:: > rsyncd Rsync Server > bash-2.05b# rsync -a 10.0.0.1::rsyncd > Password: > @ERROR: auth failed on module rsyncd > rsync: connection unexpectedly closed (90 bytes read so far) > rsync error: error in rsync protocol data stream (code 12) at io.c(150) > > Breno Cardoso Perucchi > [EMAIL PROTECTED] > -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: exclude hell !!!!!
On Wed, Jan 22, 2003 at 03:42:06AM +0100, wim delvaux wrote: ... > > > ALSO when you specify an exclude list on the server in the rsyncd.conf > > > file and you call rsync with the --delete option WHY DO THE FILES that > > > SHOULD BE EXCLUDED get deleted ?? > > > > > > Jeeez, what a bummer > > > W > > > > I assume you're comying from the server to the client? From "exclude" > > in rsyncd.conf.5: > > > > This is equivalent to the client specifying these patterns with the > > --exclude option except that the exclude list is not passed to the > > client and thus only apply on the server. > > > > which means that if you do a --delete they are not excluded on the client. > > It's as if the files don't exist at all on the server. > > Ah! so it means that if the file exist on the client and it is excluded on > the server,the client assumes the file should be removed (--delete). > > How can I EXCLUDE files from the sync on the server (I want the server > to define the rules so each client has as little config as possible) Currently that's not supported. You could make it two separate commands, with the first command only copying a file from the server which contains the exclude patterns and the second command using the file as an --exclude-from. Wayne Davison's new --files-from patch proposal has a syntax where the file name can be "remote:module/..."; I don't know if there's enough motivation to do something similar for --exclude-from. Probably not. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: exclude hell !!!!!
On Wed, Jan 22, 2003 at 02:49:03AM +0100, wim delvaux wrote: > WHY OH WHY does rsync SEND and CREATE a file called RepositoryPath ALTHOUGH > the rsync is called like > > rsync "--exclude=*.bup RepositoryPath" ... > > can ANYBODY explain. It doesn't happen for me. Is your "rsync" command actually a shell script which causes an extra round of substitution? > ALSO when you specify an exclude list on the server in the rsyncd.conf file > and you call rsync with the --delete option WHY DO THE FILES that SHOULD BE > EXCLUDED get deleted ?? > > Jeeez, what a bummer > W I assume you're comying from the server to the client? From "exclude" in rsyncd.conf.5: This is equivalent to the client specifying these patterns with the --exclude option except that the exclude list is not passed to the client and thus only apply on the server. which means that if you do a --delete they are not excluded on the client. It's as if the files don't exist at all on the server. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Please test rsync-2.5.6pre2
On Tue, Jan 21, 2003 at 11:37:12AM -, Max Bowsher wrote: > Dave Dykstra wrote: > > Changes since 2.5.5pre1: > > > > ENHANCEMENTS: > > > > * Set the default value of --modify-window to 2 on Cygwin. (Max > > Bowsher) > > No, not me. I only got involved to suggest an alternative to the deprecated > AC_CYGWIN autoconf macro. > I am, in fact, undecided on whether the --modify-window change is a good > idea or not. I knew that was all that you had contributed, but since you'd rather not have your name associated with it I changed the name to Lapo Luchini who had the original idea. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Please test rsync-2.5.6pre2
The second rsync-2.5.6 pre-release version is now available at: http://rsync.samba.org/ftp/rsync/preview/rsync-2.5.6pre2.tar.gz ftp://rsync.samba.org/pub/rsync/preview/rsync-2.5.6pre2.tar.gz rsync://rsync.samba.org/ftp/rsync/preview/rsync-2.5.6pre2.tar.gz There's also a corresponding '.sig' file that contains a gpg signature of the file; the public key is available on the pgp keyservers. This is a release candidate. If no-one reports an urgent fixes within the next few days, it will probably become version 2.5.6. Known issues that will probably not get resolved before the release unless a volunteer provides a solution: 1. The NetBSD i386 port is failing the chgrp test. See build output on build.samba.org. 2. I just noticed that the cygwin chgrp test has timestamps off by two seconds. Maybe --modify-window is supposed to be 1? Changes since 2.5.5pre1: ENHANCEMENTS: * Set the default value of --modify-window to 2 on Cygwin. (Max Bowsher) * Ignore errors from chmod when -p/-a/--preserve-perms is not set. (Dave Dykstra) BUG FIXES: * Fixed the file-name duplicate-removal code when dealing with multiple dups in a row. (Wayne Davison) * Fixed bug in --copy-unsafe-links that caused it to be completely broken. (Dave Dykstra) * Prevent infinite recursion in cleanup code under certain circumstances. (Sviatoslav Sviridov) * Fixed a bug that prevented rsync from creating intervening directories when --relative-paths/-R is set. (Craig Barratt) INTERNAL: * Portability fixes. (Dave Dykstra and Wayne Davison) - Dave Dykstra msg06197/pgp0.pgp Description: PGP signature
Re: am I missing something, or are permissions always preserved?
I submitted a change to ignore errors from chmod when -p is not set. GNU cp does the same thing. - Dave Dykstra On Mon, Jan 13, 2003 at 01:13:48PM -0800, jw schultz wrote: > On Mon, Jan 13, 2003 at 02:38:29PM -0600, Dave Dykstra wrote: > > On Mon, Jan 13, 2003 at 11:00:58AM -0800, Ben wrote: > > > Well that's annoying. I've tried changing the encoding of this > > > attempt... maybe it'll get through this time. > > > > Yes, it came through better this time. > > > > > As a rule of thumb, I think silent errors a very bad idea. It means > > > things might not be behaving like you expect, but you have no idea. In > > > general, rsync's current behavior is correct. In my case niche case, it > > > doesn't work. I think the proper solution is to change rsync's behavior, > > > not reduce the errors it produces. > > > > Maybe it wouldn't be so bad if it only looked for certain error codes from > > the failed chmod. The nice thing about ignoring errors is that it's one > > less thing for users to have to learn and understand. > > > > Any other opinions? > > First, identify the errno. Is it the same for cygwin and > samba mounts? Then do the treatment for that errno(s) only. > > As far as i can tell except where the filesystem doesn't > support chmod an error chmoding here is insane. We just > created the file. > > I'd rather kick out a warning on the first one and then > ignore. That warns the user that something isn't quite > right without flooding the log. > > Dave, see my post a litter earlier in the thread for a > possible code change. > > > > > > If it comes through, take a look at the patch. I think I did a decent > > > job of updating the docs so that purpose of the new flag is clear. > > > > At a minimum I think the option is too limited. I'd want a solution > > that would work for both the fixed permission samba case and the vfat > > filesystem under Linux case. In particular, I don't see it skipping > > the fchmod in do_mkstemp. I don't know, maybe that's the only addition > > that would be needed. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: possible typo/bug in receiver.c
On Sat, Jan 18, 2003 at 12:25:05AM -0800, Wayne Davison wrote: > On Fri, Jan 17, 2003 at 11:39:31PM -0800, Craig Barratt wrote: > > If mkstemp() fails (for various reasons, including the directory not > > existing) then fd == -1. So the first if () executes, which flushes > > the data and does a continue. So the next two if () statements will > > never execute. > > Good catch. The code got tweaked into this form in version 1.32 when it > was changed from a set of calls (using do_mktemp() and do_open()) into a > single call (using just do_mkstemp()). At that point the new (simpler) > error-check section got added, and it made it impossible to get down to > the older section. > > > Is rsync meant to create deep directories that don't exist? > > Not usually. In -R mode, it sends implied directories for the files, so > it should be the case that the directories are normally created already. > I find it interesting that this old code exists because there has been a > lot of talk about the sending of implied directories (and an option to > turn this off). If we put this code back the way it was back in 2001, > then the newly suggested --no-implied-dirs option would not cause rsync > to fail if a deep file didn't have a directory yet -- rsync would just > create the intervening directories. I'm not sure if I want it to do > that or not at the moment. > > ...wayne.. I removed the error check that skipped the creation of the directory, just a piece of the code that was added in version 1.32. It's pretty clear to me that it was just an oversight, because Tridge also modified the dead code leg. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: IPv6 hosts allow|deny
Thanks for the quick tutorial. I submitted your patch, changing eth0 to link1 as Hideaki suggested. - Dave On Mon, Jan 20, 2003 at 06:35:09PM +0100, Bert Vermeulen wrote: > On Mon, 20 Jan 2003, Dave Dykstra wrote: > > > On Fri, Jan 17, 2003 at 12:31:26AM +0100, Bert Vermeulen wrote: > > > On Mon, 13 Jan 2003, Dave Dykstra wrote: > > > > > > > I don't understand enough how that syntax works in order to be able > > > > to write an explanation for people. Could you please try to write > > > > something up based on the current version checked in to CVS? > > > > > > Here's a patch for the manpage: > > > > > > http://biot.com/patches/rsync-ipv6-acl-doc.patch > > > > I have never been introduced to IPv6 addresses before and there are some > > things about your documentation that baffle me. Perhaps I just need to > > be educated a little, or perhaps it would be useful to include a little > > more explnation in the documentation. > > > > Your patch includes this: > > > - it() a dotted decimal IP address of the form a.b.c.d for IPv4 and > > > - a.b.c.d.e.f for IPv6. In this case the incoming machine's IP address > > > + it() a dotted decimal IPv4 address of the form a.b.c.d, or an IPv6 address > > > + of the form a:b:c::d:e:f. In this case the incoming machine's IP address > > >must match exactly. > > > > Why is there a blank field between "c" and "d"?Do you think you > > should you say something about the fields being hexadecimal for IPv6, > > or is that obvious to anyone who has seen IPv6 addresses before? > > What Hideaki said, RFC 2373 describes the address format for IPv6. > > But yes, if you've ever dealt with IPv6, certainly this is obvious. I don't > think it really needs explaining in the rsync docs as such... If you have > IPv6 working on the machine, you pretty much already know the address > format. > > Anyway, briefly... :: denotes a number of 0 values, it's an abbreviation. > You can only do this once per IPv6 address. It's exactle the same as writing > this: > > 000a:000b:000c:::000d:000e:000f > > So you can see how a:b:c::d:e:f is easier to type. This is standardized > stuff, so you're guaranteed to be able to type that into anything that > parses IPv6 addresses. > > > Your patch also includes this: > > > + it() an address/mask in the form ipaddr/n where ipaddr is the IP address > > > + and n is the number of one bits in the netmask. All IP addresses which > > > + match the masked IP address will be allowed in. > > > + > > > + it() an address/mask in the form ipaddr/maskaddr where ipaddr is the > > > + IP address and maskaddr is the netmask in dotted decimal notation for IPv4, > > > + or similar for IPv6, e.g. ::::: instead of /64. All IP > > > + addresses which match the masked IP address will be allowed in. > > > > > > How come a mask includes 4 fields followed by 2 blanks? What do you > > mean by "/64"? > > The /64 is just CIDR notation as used in IPv4 as well, i.e. the number of 1 > bits in the network mask. In IPv4, they go up to 32, where a /32 means a > host. With IPv6, they go up to /128 for a host, however /64 is the > "standard" netmask for a subnet. > > > Your patch also has this addition: > > > +Note IPv6 link-local addresses can have a scope in the address specification: > > > + > > > +quote(fe80::1%eth0) > > > +quote(fe80::%eth0/64) > > > +quote(fe80::%eth0/:::::) > > > > What's a link-local address? > > An address that is only valid for a certain link, i.e. only reachable > through that interface. Thus the special prefix (fe80) -- and naturally, > those are only useful if you specify the interface as well, as you might > have the same link-local address on two different interfaces. > > > What's a scope? > > In the case of a link-local address, the scope of an interface would be the > subnet behind it. > > > I think I get that the parts after the '/' are masks, but what does it > > mean to have just two colons in the address, with one field blank and the > > third one either 1%eth0 or %eth0? > > The 1 is the last part of the address. It goes on until the "%", which > denotes the start of the interface specification. The "/64" after the > interface is indeed the netmask. > > > Bert Vermeulen > [EMAIL PROTECTED] -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Windows and default --modify-window=0
I submitted a patch similar to what Max suggested. Nobody came up with a definitive answer on whether the modify-window default should be 1 or 2 so I left it at 2. - Dave On Fri, Jan 10, 2003 at 06:27:31PM -, Max Bowsher wrote: > Dave Dykstra wrote: > > Ok, I agree --modify-window should default to 2 (or 1 if that's all > > that's really needed) on cygwin. However, I don't like os-specific > > defines. > > They have the advantage that the entire logic is visible in the .c file, > rather than split between the .c and configure.in. > > > That _WIN32 we've got there is the only one currently in the code, and > > there's no occurrances of __CYGWIN__. I'd rather have a configure.in > > rule for it. Any suggestions on a good one? At a minimum there's an > > AC_CYGWIN macro but that's not much different than just checking for > > __CYGWIN__. Probably it would be worth calling AC_CYGWIN and then > > setting > > a DEFAULT_MODIFY_WINDOW to 2 in configure if $CYGWIN=yes. > > AC_CYGWIN is obsolete - the autoconf manual recommends something like this: > AC_CANONICAL_HOST # at some point above > case $host_os in > *cygwin* ) DEFAULT_MODIFY_WINDOW=2;; > * ) DEFAULT_MODIFY_WINDOW=0;; > esac > AC_SUBST(DEFAULT_MODIFY_WINDOW) > > As far as I > > know there is no other port of rsync to Windows yet except under > > Cygwin > > so I don't think it is worth putting in extra code to support > > something > > that may never be used. Or, perhaps we should key off of the $EXEEXT > > variable which gets set to '.exe' on Windows; anybody have an opinion > > on > > that? > > Yuk! :-) > > > > Max. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: IPv6 hosts allow|deny
On Fri, Jan 17, 2003 at 12:31:26AM +0100, Bert Vermeulen wrote: > On Mon, 13 Jan 2003, Dave Dykstra wrote: > > > I don't understand enough how that syntax works in order to be able > > to write an explanation for people. Could you please try to write > > something up based on the current version checked in to CVS? > > Here's a patch for the manpage: > > http://biot.com/patches/rsync-ipv6-acl-doc.patch I have never been introduced to IPv6 addresses before and there are some things about your documentation that baffle me. Perhaps I just need to be educated a little, or perhaps it would be useful to include a little more explnation in the documentation. Your patch includes this: > - it() a dotted decimal IP address of the form a.b.c.d for IPv4 and > - a.b.c.d.e.f for IPv6. In this case the incoming machine's IP address > + it() a dotted decimal IPv4 address of the form a.b.c.d, or an IPv6 address > + of the form a:b:c::d:e:f. In this case the incoming machine's IP address >must match exactly. Why is there a blank field between "c" and "d"?Do you think you should you say something about the fields being hexadecimal for IPv6, or is that obvious to anyone who has seen IPv6 addresses before? Your patch also includes this: > + it() an address/mask in the form ipaddr/n where ipaddr is the IP address > + and n is the number of one bits in the netmask. All IP addresses which > + match the masked IP address will be allowed in. > + > + it() an address/mask in the form ipaddr/maskaddr where ipaddr is the > + IP address and maskaddr is the netmask in dotted decimal notation for IPv4, > + or similar for IPv6, e.g. ::::: instead of /64. All IP > + addresses which match the masked IP address will be allowed in. How come a mask includes 4 fields followed by 2 blanks? What do you mean by "/64"? Your patch also has this addition: > +Note IPv6 link-local addresses can have a scope in the address specification: > + > +quote(fe80::1%eth0) > +quote(fe80::%eth0/64) > +quote(fe80::%eth0/:::::) What's a link-local address? What's a scope? I think I get that the parts after the '/' are masks, but what does it mean to have just two colons in the address, with one field blank and the third one either 1%eth0 or %eth0? - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: [patch] Recent IPv6 changes in rsync/access.c
Thanks, I will apply your patch, especially since (for the record)
Hideaki posted in a reply:
> > The suggested fix Daniel Hartmeier (an OpenBSD developer) had when
> > I asked (for I'm not really a coder and I wanted to double check
> > that this was a valid problem) was making the mask[w] assignment
> > conditional like thus:
> >
> > if (w < addrlen)
> > mask[w] = 0xff & (0xff<<(8-b));
>
> Absolutely... Please apply the suggested fix.
> Thanks for finding / fixing this serious bug.
By the way, in the future I would appreciate it if you would post bugs to
the rsync mailing list instead of to me because other rsync team members
should see these things too and might be able to reply faster than me.
- Dave Dykstra
On Sat, Jan 18, 2003 at 06:21:28PM -0500, Brian Poole wrote (in private email):
> Hello,
>
> Here is a proposed patch to fix a couple problems with recent changes
> incorporated into access.c.
>
> The first is the biggest problem, there is a possible overflow
> in make_mask() if w is the same size as addrlen. Feel free to
> skip the explanation, its just there in case you don't see what
> I see.. patch is attached & at the bottom.
>
> I will walkthrough where I see the bug to have this make sense. I
> start in match_address() where I note char mask[16] at the top.
> Continuing on, we get to a switch statement for the address family;
> we imagine we take the PF_INET6 case. Two things of note here, 1st
> is addrlen = 16; 2nd is what appears to be a pasto, two copies of
>
> a = (char *)&sin6a->sin6_addr;
> t = (char *)&sin6t->sin6_addr;
>
> are found, surrounding an if statement that shouldn't affect the
> values. Thats the 2nd, minor, problem I wanted to note. I also
> moved the addrlen in my diff for cosmetic purposes. Doesn't
> matter which set of assignments is taken out.
>
> Continuing on for the bigger problem, we exit the switch statement
> and come to an if (p). Imagine p is not set and we take the else.
> This sets bits = 128.
>
> Next up is a make_mask() call with args char mask[16], 128, and 16.
> Going up to make_mask we set w = 128 >> 3, or w = 16. A few lines
> later you then have mask[w] = mask[16] is not in mask[] and
> so you have your overflow.
>
> The suggested fix Daniel Hartmeier (an OpenBSD developer) had when
> I asked (for I'm not really a coder and I wanted to double check
> that this was a valid problem) was making the mask[w] assignment
> conditional like thus:
>
> if (w < addrlen)
> mask[w] = 0xff & (0xff<<(8-b));
>
> That sounds like the correct fix to me but I shall leave it to you
> guys to finish this off. I've inlined the patch as well as attaching
> it in case you have a preference.
>
> This bug was noticed due to the propolice patches that are in
> OpenBSD-current which caused the rsync tests (which I do regularly
> for the build farm) to log messages like this:
>
> Jan 18 14:33:41 pandemonium rsync: stack overflow in function match_address
>
> I should have noticed this about 10 days ago but as this is a test
> machine I don't routinely watch the logs ;-) This gets the
> OpenBSD/sparc64 build moving along a bit further than it had been,
> though there is still one test failing (daemon-gzip-download now
> passes, daemon-gzip-upload still fails.)
>
> Will look into the other failure if I have time.
>
>
> -b
>
>
> Index: access.c
> ===
> RCS file: /cvsroot/rsync/access.c,v
> retrieving revision 1.7
> diff -u -r1.7 access.c
> --- access.c 9 Jan 2003 21:30:24 - 1.7
> +++ access.c 18 Jan 2003 23:12:57 -
> @@ -51,7 +51,8 @@
>
> if (w)
> memset(mask, 0xff, w);
> - mask[w] = 0xff & (0xff<<(8-b));
> + if (w < addrlen)
> + mask[w] = 0xff & (0xff<<(8-b));
> if (w+1 < addrlen)
> memset(mask+w+1, 0, addrlen-w-1);
>
> @@ -121,6 +122,8 @@
> a = (char *)&sin6a->sin6_addr;
> t = (char *)&sin6t->sin6_addr;
>
> + addrlen = 16;
> +
> #ifdef HAVE_SOCKADDR_IN6_SCOPE_ID
> if (sin6t->sin6_scope_id &&
> sin6a->sin6_scope_id != sin6t->sin6_scope_id) {
> @@ -128,10 +131,6 @@
> goto out;
> }
> #endif
> -
> - a = (char *)&sin6a->sin6_addr;
> - t = (char *)&sin6t->sin6_addr;
> - addrlen = 16;
>
> break;
> }
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: specifying a list of files to transfer
On Thu, Jan 16, 2003 at 11:14:50PM -0800, Wayne Davison wrote: > On Thu, Jan 16, 2003 at 07:06:05PM -0800, jw schultz wrote: > > [...] and that entries therein are not flattened like they would be on > > the command-line (sans -R). > > But they *are* flattened exactly like on the command-line, at least in > my current patch they are. That's what -R is for -- telling rsync not > to do that. So, without -R there are no implied directories to create > except for the destination dir (which is created if it doesn't exist). Oh, right, I hadn't thought of that implication of the way this is implemented. Definitely we want the -R functionality implied. That's the only way I can imagine people wanting to use this. > > The permissions and ownership should be derived from the source. > > so effectively it should be as though > > ./deltapics > > where in the file list. > > Right. In fact, that's exactly what happens with -R -- all intermediate > directories get added to the file list (if they aren't already in it) > without causing any extra recursion (even if -r was specified/implied). In my former hack implementation of the "exclude optimization" (when there were only includes with no wildcards and a final exclude '*') it was able to skip sending the parent directories completely. Come to think of it, I'm not sure what kind of permissions were used for the directories that were not explicitly included, maybe it just use the default. > If people want the "--files-from" to imply "-R" then I'd want to see a > "--no-relative" option to let people turn it off. That would be easy to implement so I guess it wouldn't hurt but I really can't see people wanting to do that. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: unsafe_symlink change (Re: CVS update: rsync)
On Thu, Jan 16, 2003 at 03:18:34PM -0600, Dave Dykstra wrote: > On Thu, Jan 16, 2003 at 04:08:01PM -0500, Scott Mcdermott wrote: > > Dave Dykstra on Thu 16/01 14:57 -0600: > > > > > The patch from 2-1/2 years ago for changing copy-unsafe-links to > > > > > follow unsafe links on the destination side also included > > > > > essentially this patch. > > > > > > > > just to be clear, without using copy-unsafe-links, rsync still > > > > copies absolute symlinks which point out of the source tree, > > > > verbatim to the destination (even if it ends up dangling on the > > > > destination), right? > > > > > > Right, with the -l or --links option (implied by -a). > > > > not to beat a dead horse, but copy-unsafe-links is certainly a misnomer. > > I liked your follow-unsafe-links much better. Copying the links doesn't > > really have much to do with following them, which is what the option > > actually does... > > Yeah, what it really means is copy-in the files pointed to by unsafe links, > but follow-unsafe-links is better except that the name applies equally > well to source and destination. I don't think it's worth changing the > name unless functionality is also changed. I just remembered that the name was based on the existing --copy-links option so there was a precedent. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: unsafe_symlink change (Re: CVS update: rsync)
On Thu, Jan 16, 2003 at 04:08:01PM -0500, Scott Mcdermott wrote: > Dave Dykstra on Thu 16/01 14:57 -0600: > > > > The patch from 2-1/2 years ago for changing copy-unsafe-links to > > > > follow unsafe links on the destination side also included > > > > essentially this patch. > > > > > > just to be clear, without using copy-unsafe-links, rsync still > > > copies absolute symlinks which point out of the source tree, > > > verbatim to the destination (even if it ends up dangling on the > > > destination), right? > > > > Right, with the -l or --links option (implied by -a). > > not to beat a dead horse, but copy-unsafe-links is certainly a misnomer. > I liked your follow-unsafe-links much better. Copying the links doesn't > really have much to do with following them, which is what the option > actually does... Yeah, what it really means is copy-in the files pointed to by unsafe links, but follow-unsafe-links is better except that the name applies equally well to source and destination. I don't think it's worth changing the name unless functionality is also changed. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: unsafe_symlink change (Re: CVS update: rsync)
Sorry, another mistake on my part -- I didn't realize the strtok side
effect. It doesn't look to me like your new code looks at the last
component of the path like the old one might have. It would probably
be a good idea to put tests ending in "/..", for example "foo/../.."
and "foo/..", into testsuite/unsafe-byname.test.
I don't expect that the strdups are of significant overhead, but as long
as you've gone through the trouble to rewrite it I suppose we might as
well use it (assuming you have handled the last component ok).
- Dave
On Thu, Jan 16, 2003 at 05:34:25PM -0800, Wayne Davison wrote:
> On Thu, Jan 16, 2003 at 02:33:26PM -0600, Dave Dykstra wrote:
> > I think you could get rid of the calls to strdup() and the new local
> > variables and possibly do a couple casts inside the function instead.
>
> You wouldn't want to carve up the const strings with strtok() (which
> adds nulls to the strings). The function could be rewritten to scan the
> string for '/'s without using strtok(), which would be a more efficient
> way to go (eliminating the need for the strdups).
>
> Here's a quick rewrite that I minimally tested. See if you like it.
>
> ...wayne..
>
> ---8<--8<--8<--8<---cut here--->8-->8-->8-->8---
> Index: util.c
> --- util.c15 Jan 2003 17:49:44 - 1.118
> +++ util.c17 Jan 2003 01:26:36 -
> @@ -793,49 +793,38 @@
> *
> * @sa t_unsafe.c
> **/
> -int unsafe_symlink(const char *dest_path, const char *src_path)
> +int unsafe_symlink(const char *dest, const char *src)
> {
> - char *tok, *src, *dest;
> + const char *name, *slash;
> int depth = 0;
>
> /* all absolute and null symlinks are unsafe */
> - if (!dest_path || !*dest_path || *dest_path == '/') return 1;
> -
> - src = strdup(src_path);
> - if (!src) out_of_memory("unsafe_symlink");
> + if (!dest || !*dest || *dest == '/') return 1;
>
> /* find out what our safety margin is */
> - for (tok=strtok(src,"/"); tok; tok=strtok(NULL,"/")) {
> - if (strcmp(tok,"..") == 0) {
> + for (name = src; (slash = strchr(name, '/')); name = slash+1) {
> + if (strncmp(name, "../", 3) == 0) {
> depth=0;
> - } else if (strcmp(tok,".") == 0) {
> + } else if (strncmp(name, "./", 2) == 0) {
> /* nothing */
> } else {
> depth++;
> }
> }
> - free(src);
> -
> - /* drop by one to account for the filename portion */
> - depth--;
> -
> - dest = strdup(dest_path);
> - if (!dest) out_of_memory("unsafe_symlink");
>
> - for (tok=strtok(dest,"/"); tok; tok=strtok(NULL,"/")) {
> - if (strcmp(tok,"..") == 0) {
> - depth--;
> - } else if (strcmp(tok,".") == 0) {
> + for (name = dest; (slash = strchr(name, '/')); name = slash+1) {
> + if (strncmp(name, "../", 3) == 0) {
> + /* if at any point we go outside the current directory
> +then stop - it is unsafe */
> + if (--depth < 0)
> + break;
> + } else if (strncmp(name, "./", 2) == 0) {
> /* nothing */
> } else {
> depth++;
> }
> - /* if at any point we go outside the current directory then
> -stop - it is unsafe */
> - if (depth < 0) break;
> }
>
> - free(dest);
> return (depth < 0);
> }
>
> ---8<--8<--8<--8<---cut here--->8-->8-->8-->8---
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: umask for rsync
In that case it looks like you want the proposed --chmod option discussed at http://lists.samba.org/pipermail/rsync/2002-March/006477.html and http://lists.samba.org/pipermail/rsync/2003-January/009298.html Once it is a command line option it could easily be made into a per-module option as well. - Dave On Thu, Jan 16, 2003 at 01:45:50PM -0500, Aaron Morris wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > I suppose if you wanted a hack, that would work. I was really looking > at something that could be used on a per module basis in rsyncd.conf or > on the command line. A commercial rsync competitor, TeamSite > OpenDeploy, has a file mode and a dir mode option. Even Samba has a > file and dir mode option on a per "module" (or share) basis. I was a > little suprised that rsync did not have a similar option when it has the > user/group option. > > Perhaps I should have said a file mask and directory mask instead of > UMASK. My primary use of rsync right now is the deployment of web > content from development machines to production machines. I do not > always have control of the content or when it is deployed, but the > file/dir mask option would be a nice security feature. Tt would keep > people from deploying files as 666 or directories as 777. > > Here is an example of how it could be useful. > A few months ago, I started having problems with file modes when > transferring with rsync (AIX 4.3.3, rsync 2.5.4, rsync --daemon from > inetd). All of the sudden, rsync would deploy files as 640 instead of > 644 like it was doing the day before. The default umask on the box had > been changed to 027 several months before this point, but rsync had been > deploying files as if the umask was still 022. Come to find out, the > box had been rebooted the night before; inetd was never restarted when > the umask was changed (and usually is never restarted) and was still > operating with a umask of 022. > > Dave Dykstra wrote: > | On Wed, Jan 15, 2003 at 02:01:30AM -0500, Aaron Morris wrote: > | > |>I have another small feature suggestion, it should not cause such a stir > |>as the whole "file list" issue (I kind of wish I never said anything > |>about it :) ). > |> > |>What about adding a "UMASK" setting to rsync (for use on the command > |>line and/or perhaps rsyncd.conf)? I realize that it may be a > |>problem/conflict with -p, but I think it could be a useful feature. > | > | > | In what way would it be different than making a 2-line script > | umask NNN > | rsync "$@" > | ? > | > | - Dave > | > > - -- > Aaron W Morris > decep > PGP Key ID: 259978D1 -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: unsafe_symlink change (Re: CVS update: rsync)
On Thu, Jan 16, 2003 at 03:44:44PM -0500, Scott Mcdermott wrote: > Dave Dykstra on Thu 16/01 14:33 -0600: > > The patch from 2-1/2 years ago for changing copy-unsafe-links to follow > > unsafe links on the destination side also included essentially this patch. > > just to be clear, without using copy-unsafe-links, rsync still copies > absolute symlinks which point out of the source tree, verbatim to the > destination (even if it ends up dangling on the destination), right? Right, with the -l or --links option (implied by -a). - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: specifying a list of files to transfer
On Wed, Jan 15, 2003 at 02:49:08PM -0800, jw schultz wrote: > On Wed, Jan 15, 2003 at 10:03:33AM -0800, Wayne Davison wrote: > > On Tue, Jan 14, 2003 at 07:57:48PM -0800, jw schultz wrote: > > > with the -r or -a options does this [recurse] on > > > directories in the --files-from list? > > > > Yes, it treats them like command-line args with the following two > > exceptions: if -r is not specified, we WILL create an explicitly named > > directory (but not send its contents), if -R is specified, we do NOT > > create implied directories (which was your next question). This latter > > exception means that we currently require the user to ensure that the > > destination directory tree is valid (which could be done once with a > > separate rsync --files-from run that didn't use -r and specified all > > the dirs that we needed to ensure exist). If this turns out to be too > > much of a hassle, perhaps a new option named --implied-dirs could be > > added to have rsync do its normal -R dir handling. > > Plus the third difference, that relative paths in the > files-from list are relative to the tree, not to the current > directory. > > We may well want the --implied-dirs option or some logic to > created it. If you don't have -r (or -a) you need to have > all the intermediate dirs listed. If you do have -r listing > intermediate dirs efectively disables the file list. > > I'm not sure i like that. I'm inclined to think the file-list > should disable recoursion. Perhaps [fighting resistance of > yuck] recoursion would have to be specified explicitly after > the --file-list. I hate sequence sensitive options but > but requiring users to remember -lptgoD instead of -a may be > worse. > > You seem to see the files-from as a way of replacing > command-line args where i see it as a way of replacing the > tree scan. However, if we can pin down the semantics i > think we can acheive both ends. I agree more with JW. I envisioned --files-from as replacing the tree scan. I would rather have the '-r' option ignored when --files-from is in effect. I think it should be a complete list of the files and/or directories that are to be sent. If people leave out the directories, missing parent directories should be automatically created. As it is now, if somebody just does "find . -print | rsync -a --files-from- ..." are they going to get repeated files because the directories are listed? Yuck. Also, if the transfer is being sent from the remote side, the file names are all getting sent over to the remote side first for --files-from and then sent back as part of the normal protocol, right? I had hoped we'd be able to avoid that round trip because the list could get long. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
unsafe_symlink change (Re: CVS update: rsync)
The patch from 2-1/2 years ago for changing copy-unsafe-links to follow unsafe links on the destination side also included essentially this patch. When I looked at it, however, I asked why in the world is unsafe_symlink() doing strdup() in the first place. I think you could get rid of the calls to strdup() and the new local variables and possibly do a couple casts inside the function instead. - Dave On Wed, Jan 15, 2003 at 12:49:44PM -0500, [EMAIL PROTECTED] wrote to rsync-cvs: > Re: CVS update: rsync > > Date: Wed Jan 15 17:49:44 2003 > Author: wayned > > Update of /data/cvs/rsync > In directory dp.samba.org:/tmp/cvs-serv3194 > > Modified Files: > proto.h util.c > Log Message: > Make unsafe_symlink() take const args so that we don't get any > compiler warnings when calling it with a const char *. > > > Revisions: > proto.h 1.150 => 1.151 > http://www.samba.org/cgi-bin/cvsweb/rsync/proto.h?r1=1.150&r2=1.151 > util.c1.117 => 1.118 > http://www.samba.org/cgi-bin/cvsweb/rsync/util.c?r1=1.117&r2=1.118 > ___ > rsync-cvs mailing list > [EMAIL PROTECTED] > http://lists.samba.org/mailman/listinfo/rsync-cvs -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: --copy-unsafe-links fix checked in
On Wed, Jan 15, 2003 at 02:24:10PM -0800, jw schultz wrote: > On Wed, Jan 15, 2003 at 10:27:31AM -0600, Dave Dykstra wrote: > > In my research into this I also found that 2-1/2 years ago somebody > > posted a patch that included a fix for this buried in it, and that I > > had promised to integrate his patch but I never did it, I think due to > > various other crises that were happening at the time. I also don't think > > I realized that the existing option was broken. His patch expanded the > > --copy-unsafe-links option so that unsafe symlinks on the destination > > side are also followed. I'm sure that could be a useful option, allowing > > things like moving some parts of a directory structure on the destination > > side to a separate filesystem. However, it really doesn't have anything > > to do with the name "copy-unsafe-links". I'm thinking that either I > > should rename the --copy-unsafe-links option to --follow-unsafe-links > > and add this functionality, or add a separate option by that name that > > only affects the destination side. Does anybody have an opinion on those? > > By follow-unsafe-links do you mean that if it finds a > link to a directory it will follow that link and descend > that directory tree located outside the tree being rsynced? > That is what i would infer following to do. While > potentially useful this is also potentially dangerous. > Following directory links should, if added, be distinguished > from I/O through file links. > > Even if it is just links to files that it would write > through it should be a separate option. It is one thing to > read through an unsafe link, quite another to write through it. > A distinction should be made between them. > > There is a reason why apache disables following links by default. > > I may be reading you incorrectly but the ideas presented > would, i think, need four options for "unsafe" links. > I have made up names for the other three for the sake of > enumeration. > > --copy-unsafe-links read through to files > --write-unsafe-linkswrite through to files > > --follow-src-links descend directory links on src > --follow-dest-links descend directory links on dest I certainly wouldn't want to have 4 options, it's confusing enough as it is. Rsync never overwrites files, it just replaces them, so at a minimum I don't think --write-unsafe-links would be of value. I don't think it's worth distinguishing between files and directories on the source side because I don't think that descending into a directory link on the source side has much risk associated with it. You have at least persuaded me to not have copy-unsafe-links affect the destination. I think I'll probably just drop the whole idea. I had briefly considered the problem of somebody malicious first copying in a symlink to an external directory and then copying again to recurse into it, but I dismissed it because I figure that pretty much the only time time rsync users need to worry about malicious use is when uploading to an rsync daemon module, and that's already protected by either chroot or sanitize_path. I had not considered the problem of accidental damage that could be done, however, and for that I think it's better to not have the option. - Dave Dykstra -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync-2.5.5 memory eater problem
Thanks for the reminder. I checked in the second version. - Dave On Wed, Oct 09, 2002 at 12:45:03PM +0300, Sviatoslav Sviridov wrote: > On Wed, 9 Oct 2002 10:03:33 +0200 (CEST) > Ruediger Oertel <[EMAIL PROTECTED]> wrote: > > > Hi, > > > > we ran into a little problem with rsync-2.5.5. > > > > Setup: you run rsync-2.5.5 as normal rsync over ssh > > (ie. not connecting to a rsync server). If you start > > such a rsync but interrupt the pulling process with Ctrl-C, > > the process on the other side may start to allocate all > > memory on the remote machine. > > > > As fa as we have analyzed the problem, the remote rsync > > process wants to issue a error message that it could not > > write 4096 bytes, receives a SIGPIPE and errno is set to EPIPE > > in the fwrite call (line 279 in log.c). Then exit_cleanup is called > > which in turn will call the same routine again and will of course > > get EPIPE again and the loop continues (as long as we have memory > > to recursivly call the same functions again and again). > > > > As a quick workaround we have the following patch, since there is > > no real point in trying to issue an error message if you have > > already received EPIPE on write. > > > > --- log.c > > +++ log.c 2002/10/08 11:55:52 > > @@ -276,7 +276,7 @@ > > > > if (!f) exit_cleanup(RERR_MESSAGEIO); > > > > - if (fwrite(buf, len, 1, f) != 1) exit_cleanup(RERR_MESSAGEIO); > > + if (fwrite(buf, len, 1, f) != 1 && errno != EPIPE) > > exit_cleanup(RERR_MESSAGEIO); > > > > if (buf[len-1] == '\r' || buf[len-1] == '\n') fflush(f); > > } > > > > I've had the same problem, but in different situation: when rsync was started > as piped child (I did it in "rsync" method for apt). My fist path was above > one, but later I've found that problem is in exit_cleanup(). I use this patch: > > --- rsync-2.5.5/cleanup.c 2002-03-27 02:03:13 +0100 > +++ rsync-2.5.5.cleanup/cleanup.c 2002-07-11 12:53:52 +0200 > @@ -43,6 +43,11 @@ > int ocode = code; > extern int keep_partial; > extern int log_got_error; > + static int inside_cleanup = 0; > + > + if (inside_cleanup != 0) > + return; > + inside_cleanup = 1; > > signal(SIGUSR1, SIG_IGN); > signal(SIGUSR2, SIG_IGN); > > > -- > Sviatoslav Sviridov > [mailto:[EMAIL PROTECTED]] [ICQ#10845380] [Lintec Project] [MLUG] -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Long-pending patch for Stratus VOS build
I think the below patch is the cause of the failure of the Irix 6.5 type on build.samba.org, because it is complaining about line 73 of the Makefile which at least on my Linux machine is rsync$(EXEEXT): $(OBJS) It might be easiest for somebody who has an Irix machine that they can use to debug this further. - Dave On Fri, Jan 10, 2003 at 01:49:37PM -0600, Dave Dykstra wrote: > On Fri, Jan 10, 2003 at 11:33:00AM -0800, Wayne Davison wrote: > > On Fri, Jan 10, 2003 at 01:23:45PM -0500, Green, Paul wrote: > > > The following patch still applies cleanly to the current cvs copy of rsync. > > > > Or did before the most recent Makefile.in changes. It's easy to merge > > this one problem, though. > > > > > Does anyone object to having these changes applied now, during the > > > pre-release phase? > > > > Here are my comments on the changes: > > > > + The Makefile.in changes look very safe and needed. > > > > + The install-sh change to the dsttmp value looks good. > > > > ? I have a question about the portability of the u_FOO -> uFOO_t > >changes. The former is the BSD syntax for the unsigned FOO typedefs, > >and the latter is what, POSIX? The changes work on Linux, at least. > >Perhaps we should just make these changes and try it out on the > >compile farm. > > > > + The inet_pton changes look right to make the code consistent. The > >only possible glitch might be a system that has a prototype for > >inet_pton() but not the library code -- if the prototype conflicts, > >the compile would fail (there are ways to work around this, but let's > >worry about that if we actually find some weird system with this > >problem). > > > > I'd be glad to check in the "+" changes now if Dave thinks now is a good > > time. > > > Ok, go ahead. I think it's ok to try out the ? patch on the compile > farm too. I know that Cygwin also uses $(EXEEXT), but apparently its > make has been more thoroughly modified to handle things automatically > so the Makefile doesn't have to use it as much. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: signing tarballs
Martin left off some context that might confuse some list readers. I had inquired about how to sign the pre-release tarball. I signed 2.5.6pre1 with my personal key, but Martin suggested there be a team key. On Thu, Jan 16, 2003 at 10:42:53AM +1100, Martin Pool wrote: > [replied to list] > > There was a discussion about this on the Samba list a while ago > > http://lists.samba.org/pipermail/samba-technical/2002-November/040931.html > > Briefly > > We should create a team signing key, with an lifetime of about a > year. It has to be relatively short to allow for turnover in the > people who have access to the key. > > The signing key must only be stored on secure machines, certainly > *not* on samba.org.(If it was on samba.org, somebody who > compromised that machine could also generate new signatures and it > would be pointless.) > > The key should be signed by team members and other relevant people; > we should also sign each others' keys. > > The key should be on the keyservers and on the web site. > > Unless you've already done so I'll create the key and send the private > half to you and the public half to the website, keyservers, and list. I have not done so, and if you're willing to set that up please go ahead. A web page describing how to use the signature, like what you were talking about on the samba list, would be great. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync hanging with openssh-2.9.9p2 as the transport
On Wed, Jan 15, 2003 at 05:12:21PM -0500, Andrew J. Schorr wrote: > On Wed, Jan 15, 2003 at 09:27:37AM -0600, Dave Dykstra wrote: > > Unfortunately I don't think anybody is going to be able to tell you. > > I've not heard of anybody lately posting a similar problem. In the > > past hanging problems have been traced to many different sources. > > Rsync stresses network (and filesystem) implementations greatly, and > > combining it with ssh stresses things that much more. I think it's > > worth a try to use openssh 3.1 or 3.2 (I've not been happy with versions > > after 3.2). What's the network between the local and remote machines? > > Does the name /nfs/mirror imply that the files are not directly mounted > > on "remote" but are instead on an NFS server? That has been the cause > > of many problems in the past. > > Actually, it turns out that my testing on this issue has been invalid > since I've been using the -vvv flag to rsync to get a grip on where > it's hanging. But I just discovered that the -vvv flag itself seems > to cause rsync to hang. So all the testing I've done so far is useless, > so I'm not sure whether this is related to openssh or the slow connection > or what. I know that the problem exists in 2.5.5, but I need to do more > testing without the -vvv flag to figure out what's causing it. Is it > well-known that the -vvv flag can cause rsync to hang? That has been shown to cause hangs in the past, but I thought it was resolved. Maybe it hasn't; people don't tend to care so much to solve problems like that once they discover the workaround of removing the verbosity. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
--copy-unsafe-links fix checked in
Below is the patch I just checked in to fix --copy-unsafe-links. As far
as I can tell it has been broken for years, ever since I first put it in.
It bothers me a bit that I can't figure out why I went through the
trouble and mess at the time to introduce a global variable to compare
to wrong information rather than using the parameter already available
in the function, but I've given up trying to figure out why.
In my research into this I also found that 2-1/2 years ago somebody
posted a patch that included a fix for this buried in it, and that I
had promised to integrate his patch but I never did it, I think due to
various other crises that were happening at the time. I also don't think
I realized that the existing option was broken. His patch expanded the
--copy-unsafe-links option so that unsafe symlinks on the destination
side are also followed. I'm sure that could be a useful option, allowing
things like moving some parts of a directory structure on the destination
side to a separate filesystem. However, it really doesn't have anything
to do with the name "copy-unsafe-links". I'm thinking that either I
should rename the --copy-unsafe-links option to --follow-unsafe-links
and add this functionality, or add a separate option by that name that
only affects the destination side. Does anybody have an opinion on those?
- Dave
diff -ur ../rsync.current/flist.c ./flist.c
--- ../rsync.current/flist.cSat Jan 11 08:18:02 2003
+++ ./flist.c Tue Jan 14 16:04:14 2003
@@ -61,8 +61,6 @@
extern int read_batch;
extern int write_batch;
-static char topsrcname[MAXPATHLEN];
-
static struct exclude_struct **local_exclude_list;
static struct file_struct null_file;
@@ -221,8 +219,11 @@
if (l == -1)
return -1;
linkbuf[l] = 0;
- if (copy_unsafe_links && (topsrcname[0] != '\0') &&
- unsafe_symlink(linkbuf, topsrcname)) {
+ if (copy_unsafe_links && unsafe_symlink(linkbuf, path)) {
+ if (verbose > 1) {
+ rprintf(FINFO,"copying unsafe symlink \"%s\" ->
+\"%s\"\n",
+ path, linkbuf);
+ }
return do_stat(path, buffer);
}
}
@@ -895,7 +896,8 @@
}
for (i = 0; i < argc; i++) {
- char *fname = topsrcname;
+ char fname2[MAXPATHLEN];
+ char *fname = fname2;
strlcpy(fname, argv[i], MAXPATHLEN);
@@ -1000,8 +1002,6 @@
}
}
- topsrcname[0] = '\0';
-
if (f != -1) {
send_file_entry(NULL, f, 0);
}
diff -ur ../rsync.current/testsuite/unsafe-byname.test ./testsuite/unsafe-byname.test
--- ../rsync.current/testsuite/unsafe-byname.test Mon Apr 8 23:49:42 2002
+++ ./testsuite/unsafe-byname.test Tue Jan 14 16:37:20 2003
@@ -44,12 +44,3 @@
test_unsafe ../../unsafe/unsafefile`pwd`/from/safe safe
test_unsafe ../files/file1`pwd`/from/safe safe
-
-# I think these are correct, but I don't really understand the code.
-# Waiting for an answer from dwd
-test_skipped "correct behaviour is unclear"
-
-# Test suspicious handling of '..' in srcdir
-test_unsafe ../../unsafe/unsafefilefrom/safe/../safe unsafe
-test_unsafe ../files/file1from/safe/../safesafe
-
diff -ur ../rsync.current/testsuite/unsafe-links.test ./testsuite/unsafe-links.test
--- ../rsync.current/testsuite/unsafe-links.testThu Jan 9 10:07:15 2003
+++ ./testsuite/unsafe-links.test Tue Jan 14 16:45:06 2003
@@ -46,20 +46,13 @@
test_regular to/links/file2
test_regular to/links/unsafefile
-#next rsync copy correctly
echo "rsync with relative path and --copy-unsafe-links";
$RSYNC -avv --copy-unsafe-links from/safe/ to
test_symlink to/links/file1
test_symlink to/links/file2
test_regular to/links/unsafefile
-# I think these are correct, but I don't really understand the code.
-# Waiting for an answer from dwd
-test_skipped "correct behaviour is unclear"
-
-
rm -rf to
-#next rsync copy incorrectly - links are copied as files not as links
echo "rsync with relative2 path";
(cd from; $RSYNC -avv --copy-unsafe-links safe/ ../to)
test_symlink to/links/file1
@@ -67,7 +60,6 @@
test_regular to/links/unsafefile
rm -rf to
-#next rsync copy uncorectly - all links are copied
echo "rsync with absolute path";
$RSYNC -avv --copy-unsafe-links `pwd`/from/safe/ to
test_symlink to/links/file1
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync hanging with openssh-2.9.9p2 as the transport
On Tue, Jan 14, 2003 at 05:29:06PM -0500, Andrew J. Schorr wrote: > Hi, > > This is perhaps a stupid question. I apologize in advance if it's already > been covered, but I'm stumped... > > I'm using rsync to backup some file systems to a remote host. > The transport is openssh-2.9.9p2. > > The problem does not occur when the transport is rsh. > > I'm invoking rsync as follows: > > rsync -RlHptgoD --numeric-ids -e "ssh" -rzx --stats . remotehost:/nfs/mirror > > But it hangs when there are large numbers of files in the "." filesystem. > If I run with -vvv, I see lots of make_file entries as send_file_list > starts sending the list of files to the remote host, but then it freezes > after around 4500 to 6500 make_file messages. > > I see the same problem with versions 2.4.6, 2.5.5, and 2.5.6pre1. > And I see it on linux 2.4.18 (red hat 8.0) and solaris 8/x86. > > I can get it to work by breaking it up into chunks of 1000 files or > so. It might work with more, I haven't tested exhaustively. > > I have tried using the --blocking-io and --no-blocking-io options, but > neither one solves the problem. > > I could provide more info about where it hangs, but I thought somebody > might know the answer, since this is clearly related to the > interaction with openssh (since there's no problem with rsh). > > Is there a trick to make rsync work nicely with openssh? I searched the > archives and haven't found anything... Does upgrading to openssh 3 solve > the problem? Unfortunately I don't think anybody is going to be able to tell you. I've not heard of anybody lately posting a similar problem. In the past hanging problems have been traced to many different sources. Rsync stresses network (and filesystem) implementations greatly, and combining it with ssh stresses things that much more. I think it's worth a try to use openssh 3.1 or 3.2 (I've not been happy with versions after 3.2). What's the network between the local and remote machines? Does the name /nfs/mirror imply that the files are not directly mounted on "remote" but are instead on an NFS server? That has been the cause of many problems in the past. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: specifying a list of files to transfer
On Tue, Jan 14, 2003 at 01:21:29PM -0800, Wayne Davison wrote: > On Tue, Jan 14, 2003 at 11:02:44AM -0500, Andrew J. Schorr wrote: > > I am attaching an updated version of my patch to allow you to specify > > a list of files to transfer. > > Cool. I'm looking into making this work when fetching files. Towards > that end, I'd like to suggest an alternate command-line syntax to make > the --source-file option take a filename. This will allow it to accept > "-" as stdin, and will make it easy to parse for the pull syntax. This > means that we need to omit the SRC spec on a push or specify it as > empty. E.g. these will all work: > > rsync --source-list=file remote:/path > rsync --source-list file : remote:/path > rsync --source-list=- "" remote:/path > A pull looks like this: > > rsync --source-list=file remote: /path > rsync --source-list - remote::module /path > What do people think? Of course this is not for the rsync release we're > currently working on, but could be included as a patch, if desired. I haven't looked at the implementation, but comments on the user interface: 1. Yes it should take a filename or - as a parameter. 2. I don't like the idea of skipping the SRC spec. Paths should be relative to the SRC. If somebody wants to use full paths they can always have a SRC of "/". 3. It should be called --files-from. 4. --send-dirs and --no-implicit-dirs shouldn't be separate options, they should be automatically turned on with the --files-from option. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: configure issue (ac_cv_lib_inet_connect) on DYNIX/ptx
On Mon, Jan 13, 2003 at 09:03:23PM -0500, Michael Sterrett -Mr. Bones.- wrote: > Greetings - > > In trying to get rsync-2.5.6pre1 working on DYNIX/ptx, I found I > needed to pass ac_cv_lib_inet_connect=no to configure in order for > it actually create a properly configured config.h file. > > I think this is because configure looks for connect in libinet, > (and finds one there somehow). But connect() is resolved by libsocket > (but configure doesn't find it when it looks there). I've never taken > the time to delve into the guts of autoconf, so I'm not sure if this > is a general autoconf problem or a problem that is specific to rsync. > > At any rate, hopefully, this will be useful to someone. I can send > a config.log file to anyone who's interested. > > After passing ac_cv_lib_inet_connect=no to configure, all tests either > PASS or SKIP so it looks good. Ok, we'll assume that you'll continue to do it that way unless you post a patch to configure.in. Really there should be a modification to autoconf, probably an AC_FUNC_CONNECT macro, but I'm not sure anybody wants to do the work. I do see a couple mentions of DYNIX in the autoconf 2.53 macros. > I also get this warning, which is trivial, but it would be nice to > clean it up: > > "cleanup.c", line 36: portability warning: trigraph sequence replaced > > This is from the () in cleanup.c. To get rid of the warning, > just remove the () from line 36. I don't want to remove it completely because apparently it means something to the person who put it in. Please suggest another form that conveys the same meaning and doesn't cause a warning. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: 2.5.6pre1 bombs on Sunos4 in popthelp.c on use of sprintf
On Tue, Jan 14, 2003 at 10:24:04AM -0600, Dave Dykstra wrote:
> 2.5.6pre1 bombs on Sunos4 gcc with these errors
> popt/popthelp.c: In function `singleOptionDefaultValue':
> popt/popthelp.c:137: invalid operands to binary +
> popt/popthelp.c:141: invalid operands to binary +
> popt/popthelp.c:145: invalid operands to binary +
> popt/popthelp.c:149: invalid operands to binary +
> because it's depending on sprintf to return the number of bytes written
> and that doesn't happen on Sunos4.
If nobody objects to the following patch, I'll put it in. It's a little
slower but this is definitely not time-critical code.
- Dave
--- popt/popthelp.c.O Tue Jan 14 13:02:47 2003
+++ popt/popthelp.c Tue Jan 14 13:04:45 2003
@@ -134,19 +134,23 @@
case POPT_ARG_VAL:
case POPT_ARG_INT:
{ long aLong = *((int *)opt->arg);
- le += sprintf(le, "%ld", aLong);
+ sprintf(le, "%ld", aLong);
+ le += strlen(le);
} break;
case POPT_ARG_LONG:
{ long aLong = *((long *)opt->arg);
- le += sprintf(le, "%ld", aLong);
+ sprintf(le, "%ld", aLong);
+ le += strlen(le);
} break;
case POPT_ARG_FLOAT:
{ double aDouble = *((float *)opt->arg);
- le += sprintf(le, "%g", aDouble);
+ sprintf(le, "%g", aDouble);
+ le += strlen(le);
} break;
case POPT_ARG_DOUBLE:
{ double aDouble = *((double *)opt->arg);
- le += sprintf(le, "%g", aDouble);
+ sprintf(le, "%g", aDouble);
+ le += strlen(le);
} break;
case POPT_ARG_STRING:
{ const char * s = *(const char **)opt->arg;
@@ -271,7 +275,8 @@
*le++ = '=';
if (negate) *le++ = '~';
/*@-formatconst@*/
- le += sprintf(le, (ops ? "0x%lx" : "%ld"), aLong);
+ sprintf(le, (ops ? "0x%lx" : "%ld"), aLong);
+ le += strlen(le);
/*@=formatconst@*/
*le++ = ']';
} break;
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
2.5.6pre1 bombs on Sunos4 in popthelp.c on use of sprintf
2.5.6pre1 bombs on Sunos4 gcc with these errors popt/popthelp.c: In function `singleOptionDefaultValue': popt/popthelp.c:137: invalid operands to binary + popt/popthelp.c:141: invalid operands to binary + popt/popthelp.c:145: invalid operands to binary + popt/popthelp.c:149: invalid operands to binary + because it's depending on sprintf to return the number of bytes written and that doesn't happen on Sunos4. Wayne or anybody, do you have a suggested solution? The popt maintainers should also be told about that. This foible of Sunos4 is mentioned in the autoconf documentation of things to watch out for but I don't think they suggest a solution. We need to get one of these Sunos4 machines into the build farm. I know where to get one, but I had been hesitant to add machines to the build farm because of security concerns. I think the new rsync module over ssh feature will eliminate those concerns. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Rsync performance increase through buffering
Craig, I'd like to get your patch into the 2.5.6 patches directory. Could you please make sure it applies cleanly onto version 2.5.6pre1 (see news on http://rsync.samba.org home page if you haven't been following the mailing list) and repost it? Thanks, - Dave Dykstra On Wed, Dec 11, 2002 at 03:20:57PM -0600, Dave Dykstra wrote: > On Sun, Dec 08, 2002 at 11:48:57PM -0800, Craig Barratt wrote: > > I've been studying the read and write buffering in rsync and it turns > > out most I/O is done just a couple of bytes at a time. This means there > > are lots of system calls, and also most network traffic comprises lots > > of small packets. The behavior is most extreme when sending/receiving > > file deltas of identical files. > ... > > Below is a patch to a few files that adds read and write buffering in > > the places where the I/O was unbuffered, adds buffering to write_file() > > and removes the unneeded gettimeofday() system call in show_progress(). > ... > > However, on a network test doing a send from cygwin/WinXP to rsyncd > > on rh-linux the running time improves from about 700 seconds to 215 > > seconds (with a cpu load of around 17% versus 58%, if you believe > > cygwin's cpu stats). This is probably an extreme case since the system > > call penalty in cygwin is high. But I would suspect a significant > > improvement is possible with a slow network connection, since a lot > > less data is being sent. > > Looks like a good candidate for the "patches" directory now and for > integration in 2.6.0. > > - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: am I missing something, or are permissions always preserved?
On Mon, Jan 13, 2003 at 11:00:58AM -0800, Ben wrote: > Well that's annoying. I've tried changing the encoding of this > attempt... maybe it'll get through this time. Yes, it came through better this time. > As a rule of thumb, I think silent errors a very bad idea. It means > things might not be behaving like you expect, but you have no idea. In > general, rsync's current behavior is correct. In my case niche case, it > doesn't work. I think the proper solution is to change rsync's behavior, > not reduce the errors it produces. Maybe it wouldn't be so bad if it only looked for certain error codes from the failed chmod. The nice thing about ignoring errors is that it's one less thing for users to have to learn and understand. Any other opinions? > If it comes through, take a look at the patch. I think I did a decent > job of updating the docs so that purpose of the new flag is clear. At a minimum I think the option is too limited. I'd want a solution that would work for both the fixed permission samba case and the vfat filesystem under Linux case. In particular, I don't see it skipping the fchmod in do_mkstemp. I don't know, maybe that's the only addition that would be needed. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: [rsync] Please test rsync-2.5.6pre1
On Mon, Jan 13, 2003 at 01:59:47PM -0500, Scott Evans wrote: > > Known issues that will probably need to be resolved before the final > > release: > > [...] > > Is there any interest in trying to hunt down the "linux -> cygwin > rsync hanging" problem I'm consistently seeing before this release? > I'm more than happy to help track it down but I don't know enough about > rsync to do it myself. Well, at a minimum it would be nice to know if it is still happening with version 2.5.6pre1. If it's no worse than previous versions, we won't hold up a release of 2.5.6. If a cygwin user tracks it down and provides a fix that doesn't break anybody else, we'll probably put it in. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Please test rsync-2.5.6pre1
The first rsync-2.5.6 pre-release version is now available at: http://rsync.samba.org/ftp/rsync/preview/rsync-2.5.6pre1.tar.gz ftp://rsync.samba.org/pub/rsync/preview/rsync-2.5.6pre1.tar.gz rsync://rsync.samba.org/ftp/rsync/preview/rsync-2.5.6pre1.tar.gz There's also a corresponding '.sig' file that contains a gpg signature of the file; the public key is available on the pgp keyservers. Please test it out thoroughly and report problems and bug fix patches to this mailing list. Known issues that will probably need to be resolved before the final release: 1. The Redhat Linux ia64 and Alpha and the OpenBSD Sparc64 ports are all failing the daemon-gzip-upload and daemon-gzip-download tests even though the basic daemon test succeeds. The OpenBSD test reports a core dump, and I suspect they're all having the same problem related to 64 bit. See build output on build.samba.org. Somebody else is going to have to fix this, I don't have the resources. 2. The NetBSD i386 port is failing the chgrp test. NetBSD users, see build.samba.org. 3. --copy-unsafe-links is badly broken. I'm planning on trying to fix this myself. 4. The --modify-window option should default to 2 seconds or 1 second on Cygwin. I'll put this in once somebody determines which it should be. 5. The documentation hosts allow & hosts deny needs to be updated for IPv6. Changes since the 2.5.5 release: ENHANCEMENTS: * The --delete-after option now implies --delete. (Wayne Davison) * The --suffix option can now be used with --backup-dir. (Michael Zimmerman) * Combining "::" syntax with the -rsh/-e option now uses the specified remote-shell as a transport to talk to a (newly-spawned) server-daemon. This allows someone to use daemon features, such as modules, over a secure protocol, such as ssh. (JD Paul) * The rsync:// syntax for daemon connections is now accepted in the destination field. * If the file name given to --include-from or --exclude-from is "-", rsync will read from standard input. (J.W. Schultz) * New option --link-dest which is like --compare-dest except that unchanged files are hard-linked in to the destination directory. (J.W. Schultz) * Don't report an error if an excluded file disappears during an rsync run. (Eugene Chupriyanov and Bo Kersey) * Added .svn to --cvs-exclude list to support subversion. (Jon Middleton) * Properly support IPv6 addresses in the rsyncd.conf "hosts allow" and "hosts deny" fields. (Hideaki Yoshifuji) BUG FIXES: * Fix "forward name lookup failed" errors on AIX 4.3.3. (John L. Allen, Martin Pool) * Generate each file's rolling-checksum data as we send it, not in a separate (memory-eating) pass before hand. This prevents timeout errors on really large files. (Stefan Nehlsen) * Fix compilation on Tru64. (Albert Chin, Zoong Pham) * Better handling of some client-server errors. (Martin Pool) * Fixed a crash that would occur when sending a list of files that contains a duplicate name (if it sorts to the end of the file list) and using --delete. (Wayne Davison) * Fixed a bug that caused rsync to lose the exit status of its child processes and sometimes return an exit code of 0 instead of showing an error. (David R. Staples, Dave Dykstra) INTERNAL: * Many code cleanups and improved internal documentation. (Martin Pool, Nelson Beebe) * More test cases. (Martin Pool) * Updated included popt to the latest vendor drop, version 1.6.4. (Jos Backus) - Dave Dykstra msg06044/pgp0.pgp Description: PGP signature
Re: am I missing something, or are permissions always preserved?
On Sat, Jan 11, 2003 at 11:19:33AM -0800, Ben wrote [off list]: > On Tue, 2003-01-07 at 14:49, Dave Dykstra wrote: > > That sounds reasonable to me that rsync shouldn't try to preserve those > > extra bits without -p. Try making a patch and seeing if that works. > > > > I don't have Samba set up, but on a vfat filesystem on Linux rsync is > > doing even worse because after the initial file creation it does a > > "fchmod(fd, 0600)" which fails and it doesn't even copy files at all. > > > > > Okay, attached is a patch I just whipped up that seems to work well. It > adds the --skip-perms flag, which has the effect of making rsync not try > to set permissions, not even to sane values. I was not able to read your patch because it seems to have lots of characters that I can't view on my email reader. A header on the attachment says X-MIME-Autoconverted: from 8bit to base64 by mx6.mx.voyager.net so my ISP must have had trouble with it. Anyway, I wasn't thinking of an option, I was thinking of it doing something that was the default whenever there was no -p. I'd rather avoid adding an option if we can get away with it. I'm now having second thoughts about stripping off the setuid/setgid bits by default without '-p', because that's not what GNU cp does. I think it would be better if it just silently ignored errors from chmod and fchmod when -p is not set. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: IPv6 hosts allow|deny
On Sun, Jan 12, 2003 at 11:51:07PM +0100, Bert Vermeulen wrote: > On Fri, 10 Jan 2003, Bert Vermeulen wrote: > > > On Thu, 9 Jan 2003, Dave Dykstra wrote: > > > > > I went ahead and submitted Hideaki's patch pretty much as is. I took off > > > the ifdef around the memset at the beginning of client_sockaddr() because > > > I figured it wouldn't hurt to do anywhere. Besides, it was checking > > > for a specific operating system (__linux__) which we like to avoid and > > > because there was a typo in the check for HAVE_SOCKADDR_IN6_SCOPE_ID > > > (it was HAVE_SOCKADDR_IN6_SOCPE_ID). I noticed that there was code in > > > access.c that was accessing the same sockaddr_in6 sin6_scope_id field that > > > was protected by ifdef in clientname.c so I put the same ifdef around > > > the code in access.c. I also updated the rsyncd.conf.yo documentation > > > to allow for IP address of the IPv6 form. > > > > > > I have not tested this, I hope it works. > > > > Dave, > > > > Sorry for the late reply, been busy. I'm going to test with the committed > > patch, and will let you know. I can only test on linux, however. > > Well, it works great for me. It might be a good idea to include Hideaki's > comments on this syntax in the docs: > > > Note that scope-id is also supported: > > - fe80::1%eth0 > > - fe80::%eth0/64 > > - fe80::%eth0/::::: > > The example is significant -- this restricts access to the local subnet > only. I don't understand enough how that syntax works in order to be able to write an explanation for people. Could you please try to write something up based on the current version checked in to CVS? I am totally mystified by Hideaki's examples. I didn't think colons were allowed, and I don't see any code in access.c for recognizing them. As far as I can tell the code only allows decimal notation, not hexadecimal. And do the blank fields mean anything? For the record, here are the rest of Hideaki's examples: > > - 3ffe::dead:beef::1 > > - 3ffe::dead:beef::/64 > > - 3ffe::dead:beef::/::::: > > or even > > - ::0123:4567:89ab:cdef/::::: - Dave Dykstra -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Long-pending patch for Stratus VOS build
On Fri, Jan 10, 2003 at 11:33:00AM -0800, Wayne Davison wrote: > On Fri, Jan 10, 2003 at 01:23:45PM -0500, Green, Paul wrote: > > The following patch still applies cleanly to the current cvs copy of rsync. > > Or did before the most recent Makefile.in changes. It's easy to merge > this one problem, though. > > > Does anyone object to having these changes applied now, during the > > pre-release phase? > > Here are my comments on the changes: > > + The Makefile.in changes look very safe and needed. > > + The install-sh change to the dsttmp value looks good. > > ? I have a question about the portability of the u_FOO -> uFOO_t >changes. The former is the BSD syntax for the unsigned FOO typedefs, >and the latter is what, POSIX? The changes work on Linux, at least. >Perhaps we should just make these changes and try it out on the >compile farm. > > + The inet_pton changes look right to make the code consistent. The >only possible glitch might be a system that has a prototype for >inet_pton() but not the library code -- if the prototype conflicts, >the compile would fail (there are ways to work around this, but let's >worry about that if we actually find some weird system with this >problem). > > I'd be glad to check in the "+" changes now if Dave thinks now is a good > time. Ok, go ahead. I think it's ok to try out the ? patch on the compile farm too. I know that Cygwin also uses $(EXEEXT), but apparently its make has been more thoroughly modified to handle things automatically so the Makefile doesn't have to use it as much. I'm looking at the --copy-unsafe-links problem and it looks very badly broken. I was the one who put it in the first place based on a contributed patch, and now I'm not sure it ever worked. It's going to take a while to trace through the history to figure out if it worked once and if so when it broke. The fix is not likely to impact much, though, so I don't think 2.5.6pre1 will need to be held up because of it because it can go in later. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Windows and default --modify-window=0
Ok, I agree --modify-window should default to 2 (or 1 if that's all that's really needed) on cygwin. However, I don't like os-specific defines. That _WIN32 we've got there is the only one currently in the code, and there's no occurrances of __CYGWIN__. I'd rather have a configure.in rule for it. Any suggestions on a good one? At a minimum there's an AC_CYGWIN macro but that's not much different than just checking for __CYGWIN__. Probably it would be worth calling AC_CYGWIN and then setting a DEFAULT_MODIFY_WINDOW to 2 in configure if $CYGWIN=yes. As far as I know there is no other port of rsync to Windows yet except under Cygwin so I don't think it is worth putting in extra code to support something that may never be used. Or, perhaps we should key off of the $EXEEXT variable which gets set to '.exe' on Windows; anybody have an opinion on that? - Dave On Thu, Jan 09, 2003 at 02:41:21PM -0800, jw schultz wrote: > On Thu, Jan 09, 2003 at 11:21:36PM +0100, Lapo Luchini wrote: > > jw schultz wrote: > > > > >The 2 second timestamp resolution only applies to some > > >windows filesystems. I think NTFS has timestamp resolution > > >in the milliseconds (unlrelated to precision). It should > > >only become an issue when the windows filesystem is the > > >destination. > > > > > This message says it is 2 seconds aso on NTFSD actually: > > http://lists.samba.org/pipermail/rsync/2000-July/002491.html > > I don't use windows myself. I just recall that one of the > windows filesystem has timestamps in the milliseconds. > > > >Further examination shows that it is > > >already hacked in options.c for win32: > > >| #ifdef _WIN32 > > >| int modify_window=2; > > >| #else > > >| int modify_window=0; > > >| #endif > > >perhaps something just needs to be added to the #ifdef. > > > > > Ohhh... I'd add CygWin to the #ifdef too, then. > > Grr, I *never* remember what does cygwin define and what not ^_^ > > > > Let me check... this message > > http://sources.redhat.com/ml/gdb-patches/2001-04/msg00039.html > > suggests that to include djgpp, mingw and cygwin the full ifdef should be: > > > > #if defined(__GO32__) || defined(_WIN32) || defined (__CYGWIN__) > > > > Should I add that to the current 2.5.5 cygwin port (creating a -3 release) > > or 2.5.6 will be short in time? > > Depends on how urgent this is. Given you have a workaround > i'd get it in cvs and wait. Assuming the 2.5.6 really does > happen soon. > > Please try a modify-window of 1. Maybe my math is off > somehow but i don't think it needs to be greater than 1. > Take a look at the code in util.c. If you truncate to 2 > second resolution and compare against 1 second resolution > the difference should never be greater than 1 second. > It won't make a difference very often but i'd still feel > better making the window as small as possible. > > -- > > J.W. SchultzPegasystems Technologies > email address: [EMAIL PROTECTED] > > Remember Cernan and Schmitt > -- > To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: make clean
I checked in this patch. - Dave On Thu, Jan 09, 2003 at 06:41:19PM -0800, jw schultz wrote: > I just did a make clean to tidy up a bit and found four > objects were missed. The seem to belong to the CHECK_PROGS > set. Not sure where best to put them so created a > CHECK_OBJS variable and added it to the clean rule. > > > -- > > J.W. SchultzPegasystems Technologies > email address: [EMAIL PROTECTED] > > Remember Cernan and Schmitt > Index: Makefile.in > === > RCS file: /cvsroot/rsync/Makefile.in,v > retrieving revision 1.84 > diff -u -r1.84 Makefile.in > --- Makefile.in 8 Apr 2002 06:23:34 - 1.84 > +++ Makefile.in 10 Jan 2003 02:40:34 - > @@ -43,6 +43,9 @@ > # Programs we must have to run the test cases > CHECK_PROGS = rsync tls getgroups trimslash t_unsafe > > +# Objects for CHECK_PROGS to clean > +CHECK_OBJS=getgroups.o t_stub.o t_unsafe.o trimslash.o > + > # note that the -I. is needed to handle config.h when using VPATH > .c.o: > @OBJ_SAVE@ > @@ -106,7 +109,7 @@ > cat $(srcdir)/*.c $(srcdir)/lib/compat.c | awk -f $(srcdir)/mkproto.awk > >$(srcdir)/proto.h > > clean: cleantests > - rm -f *~ $(OBJS) rsync $(TLS_OBJ) $(CHECK_PROGS) > + rm -f *~ $(OBJS) rsync $(TLS_OBJ) $(CHECK_PROGS) $(CHECK_OBJS) > > cleantests: > rm -rf ./testtmp* -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: working on a 2.5.6pre1 release
On Fri, Jan 10, 2003 at 12:46:13AM -0800, Wayne Davison wrote: > On Thu, Jan 09, 2003 at 05:09:07PM -0600, Dave Dykstra wrote: > > I'm stuck on a problem that some machines on build.samba.org are > > showing on the 'chgrp' test. > > I've checked in a fix for this bug. Thanks, Wayne! Great work. I think the remaining build on build.samba.org are not enough to delay putting out pre1, but it would be good if those of you who care about the following ports would try to come up with some fixes before we do the final release: 1. The CRAY SV1 UNICOS machine is not compiling, complaining that it can't take the address of the bit field in sin.sin_addr.s_addr. I think the solution may be to assign that value to a local variable and then copy it into the structure element, but it isn't clear to me what the type of the local variable should be. 2. The Linux ia64 and Alpha and the OpenBSD Sparc64 ports are all failing the daemon-gzip-upload and daemon-gzip-download tests even though the basic daemon test succeeds. The OpenBSD test reports a core dump, and I suspect they're all having the same problem related to 64 bit. 3. The Stratus VOS port is failing all 3 daemon tests in code that is used just for testing, saying it can't create the test socket. I don't know if there's a corresponding problem in the corresponding non-test code. 4. The NetBSD i386 port is failing the chgrp test. I think if I can find the time today I want to work a little yet on the --copy-unsafe-links problem and on the unsafe-* tests and then I'll make the pre1 version. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
working on a 2.5.6pre1 release
I'm working on trying to get rsync 2.5.6pre1 available for people to test more widely. I'm out of time for today, and I'm stuck on a problem that some machines on build.samba.org are showing on the 'chgrp' test. I can reproduce this on my home redhat 7.3 system too. It appears to be a timing problem because when I do strace -F -f on it the problem goes away. Everything seems to go through normally but then it exits with an exit code of 12, I think because the child receiver process is terminated with a SIGUSR2 which is signal 12 and because the bug that was preventing exit codes from being properly reported from children has now been fixed. It's very hard to debug because it is a timing problem and because it happens after rprintf handling is already shut down in the child process. I suspected that maybe the catching of the SIGUSR2 signal was not getting inherited from its parent, but it doesn't help to re-set it in the child. Nevertheless, I'm not sure whether or not the sigusr2_handler function is getting called in the child. I'd appreciate some help with this if anybody else thinks they can figure it out. - Dave Dykstra -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: IPv6 hosts allow|deny
I went ahead and submitted Hideaki's patch pretty much as is. I took off the ifdef around the memset at the beginning of client_sockaddr() because I figured it wouldn't hurt to do anywhere. Besides, it was checking for a specific operating system (__linux__) which we like to avoid and because there was a typo in the check for HAVE_SOCKADDR_IN6_SCOPE_ID (it was HAVE_SOCKADDR_IN6_SOCPE_ID). I noticed that there was code in access.c that was accessing the same sockaddr_in6 sin6_scope_id field that was protected by ifdef in clientname.c so I put the same ifdef around the code in access.c. I also updated the rsyncd.conf.yo documentation to allow for IP address of the IPv6 form. I have not tested this, I hope it works. - Dave On Wed, Jan 08, 2003 at 08:24:13AM -0600, Dave Dykstra wrote: > On Wed, Jan 08, 2003 at 12:25:15AM +0100, Bert Vermeulen wrote: > > On Tue, 7 Jan 2003, Dave Dykstra wrote: > > > > > Even though rsync maintenance isn't as bad as wget's, the maintainers > > > are all VERY part time so that is a big part of the problem. Most of us > > > don't have ipv6 systems to test things on. Can you vouch for the quality > > > of the patch? I was able to get it with > > > > > > wget --passive >ftp://ftp.linux-ipv6.org/pub/usagi/misc/rsync-2_5_5-v6auth-20021016.patch.gz > > > > > > and it looks quite extensive. > > > > Yes, it's very extensive, and there's some code in there that's pretty > > advanced. > > > > The fact of the matter is, I made my own patch for this functionality first, > > and only then noticed Hideaki's patch -- and his, while not as readable to > > the casual observer, seems like better code. He is rewriting the linux > > kernel IPv6 stack, so I'd expect his rsync patch to be decent :-) > > > > Nevertheless, feel free to take a look at my patch: > > http://biot.com/patches/rsync-ipv6-acl.patch > > > > So I haven't used Hideaki's patch, but I use mine, and it's good. Please do > > apply one of them however. > > > Since you're more familiar with that area of the code than any of the > rsync maintainers and you think his patch is better, please test out his > patch and look it over to see if you think there should be any changes. > In particular, I see that yours includes documentation changes and his > doesn't. > > - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: --exclude and --delete-exclude problem
On Thu, Jan 09, 2003 at 01:20:08AM -0400, Carlos Molina wrote: > Hi. > > I'm trying to accomplish the following ; > > 1) First, I did a rsync for a file, like this > /usr/bin/rsync -lptgoD --delete --rsh=/usr/bin/ssh -R --delete-excluded > -vv [EMAIL PROTECTED]:/usr/local/src/* /tmp/localhost/daily.0 Remove the '*'. In order for rsync to delete a file, it needs to deal with the directory that the files are in and the '*' tells it to only deal with the files in the directory. - Dave Dykstra -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
child exit code rsync bug patch finally submitted
I am preparing for making 2.5.6pre1 today and decided to submit David
Staples patch with a little rework. Here's the version I submitted.
- Dave Dykstra
--- main.c~ Thu Aug 1 15:46:59 2002
+++ main.c Thu Jan 9 12:43:55 2003
@@ -26,6 +26,16 @@
extern struct stats stats;
extern int verbose;
+/* there's probably never more than at most 2 outstanding child processes,
+ * but set it higher just in case.
+ */
+#define MAXCHILDPROCS 5
+
+struct pid_status {
+ pid_t pid;
+ int status;
+} pid_stat_table[MAXCHILDPROCS];
+
static void show_malloc_stats(void);
/
@@ -33,11 +43,27 @@
/
void wait_process(pid_t pid, int *status)
{
- while (waitpid(pid, status, WNOHANG) == 0) {
+ pid_t waited_pid;
+ int cnt;
+
+ while ((waited_pid = waitpid(pid, status, WNOHANG)) == 0) {
msleep(20);
io_flush();
}
+ if ((waited_pid == -1) && (errno == ECHILD)) {
+ /* status of requested child no longer available.
+* check to see if it was processed by the sigchld_handler.
+*/
+ for (cnt = 0; cnt < MAXCHILDPROCS; cnt++) {
+ if (pid == pid_stat_table[cnt].pid) {
+ *status = pid_stat_table[cnt].status;
+ pid_stat_table[cnt].pid = 0;
+ break;
+ }
+ }
+ }
+
/* TODO: If the child exited on a signal, then log an
* appropriate error message. Perhaps we should also accept a
* message describing the purpose of the child. Also indicate
@@ -848,7 +874,24 @@
static RETSIGTYPE sigchld_handler(int UNUSED(val)) {
#ifdef WNOHANG
- while (waitpid(-1, NULL, WNOHANG) > 0) ;
+ int cnt, status;
+ pid_t pid;
+ /* An empty waitpid() loop was put here by Tridge and we could never
+* get him to explain why he put it in, so rather than taking it
+* out we're instead saving the child exit statuses for later use.
+* The waitpid() loop presumably eliminates all possibility of leaving
+* zombie children, maybe that's why he did it.
+*/
+ while ((pid = waitpid(-1, &status, WNOHANG)) > 0) {
+/* save the child's exit status */
+for (cnt = 0; cnt < MAXCHILDPROCS; cnt++) {
+ if (pid_stat_table[cnt].pid == 0) {
+ pid_stat_table[cnt].pid = pid;
+ pid_stat_table[cnt].status = status;
+ break;
+ }
+}
+ }
#endif
}
On Tue, Sep 03, 2002 at 06:29:03PM -0700, jw schultz wrote:
> Subject: Re: [patch] for rsync
> On Tue, Sep 03, 2002 at 05:00:02PM -0700, [EMAIL PROTECTED] wrote:
> >
> > JW,
> >
> > I pushed everything to a LINUX box and did the diff again this time with the
> > -bur option.
> > It does look significantly different. I think I've got below what you are
> > looking for, so I am
> > resending it to [EMAIL PROTECTED]
> >
> > I'm including two additional people as they have asked for the patch information
> > as well.
> > The diff was performed on main.c (2.5.5 version from rsync.samba.org). I hope
> > everyone
> > knows that I'm my purpose in providing this information, is so that everyone can
> > critique
> > my code (to make it better) and we can eliminate this problem for everyone.
> >
> > Dave
>
> Thank you Dave. I've been abusing/educating him a bit
> offline so be nice everybody. He has not only diagnosed the
> bug independantly but has come up with a direction to go
> toward a permanent solution.
>
> Aside from my dislike of a fixed size array
> my comments are below.
>
>
> > -
> >
> > --- main.c.orig Tue Sep 3 16:38:23 2002
> > +++ main.c Tue Sep 3 16:41:08 2002
> > @@ -27,6 +27,11 @@
> >
> > extern int verbose;
> >
> > +struct pid_status {
> > +pid_t pid;
> > +int status;
> > + } pid_stat_table[10];
>
> The use of a literal here is bad news.
> Should be a #define CHILD_TABLE_SIZE or whatever.
> Then macro can be used in the loops.
>
> Question. Is 10 a good size? Should it be more?
> I've not dug into the child spawning of rsync.
>
> > +
> > static void show_malloc_stats(void);
> >
> > /*
Re: Windows and default --modify-window=0
On Thu, Jan 09, 2003 at 05:52:43PM +0100, Lapo Luchini wrote: > Dave Dykstra wrote: > > >it would fail if it were > >talking to a release of rsync that is too old to recognize the > >option[...]. The surprise factor on that may be too much to make it > >worthwhile. > > > > > I agree. I didn't think about that. > But, given the resilience to "upgrade" many people have, you sure are right. It could perhaps key off the remote protocol version number. The version number is not as a rule incremented when new options are added, but it could check for whatever version the protocol was incremented to after the option was added, and only add the option then. No, wait, that won't work because the protocol version number isn't exchanged until after the option parameters are passed. The only way to do it would be to modify the protocol to include the info instead of passing a parameter. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Windows and default --modify-window=0
On Thu, Jan 09, 2003 at 01:02:29PM +0100, Lapo Luchini wrote: > Uhm... rsync developers: what do you think about changing the default > modify-window in the Cygwin platform? > It seems to me that many problems people have are just because of the > granularity of timestamp. > Maybe changing the default could be the best choice. That would be > documented in the "man" and in my release announcement on > cygwni-annouce, of course. > > Just an idea, maybe it's bad 0=) As I understand it, --modify-window=2 is needed whenever you're copying between a Windows filesystem (maybe only specific types, does anybody know??) and a Unix filesystem. Just because you're running on Cygwin, that doesn't necessarily mean you're doing that although the risk of a file modification time legitimately being off by a few seconds is pretty low in the cases where it isn't really needed. (Conversely, on Linux you may be accessing a Windows filesystem directly but I realize you're not talking about that situation). The main problem is that if Cygwin always added the --modify-window=2 option to parameters it passes to the remote side it would fail if it were talking to a release of rsync that is too old to recognize the option. If it could be done on the cygwin side only, without passing the parameter to the remote side, it might not be so bad. I guess that would mean it would work only when receiving files from a Unix system and not when sending to them. The surprise factor on that may be too much to make it worthwhile. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: long directory name problem
I've never heard of a name length limit. One operating system(s) and rsync version(s) are you using? - Dave On Wed, Jan 08, 2003 at 07:32:19AM -0800, elijah reyen wrote: > Does rsync has directory name length limit? I was trying to rsync a directory that >has the name length of 34 characters and getting permission denied error. > > $ rsync -avz --delete --force -e ssh /usr/local/apache/htdocs >192.168.0.2:/usr/local/apache > building file list ... > readlink htdocs/CorporateLegalTimesEDITORIAL_files/cBug_blueXs.gif: Permission denied > readlink httpd/html/CorporateLegalTimesEDITORIAL_files/title_a.gif: Permission denied > > When I rename the directory to about half of its original length, the error goes >away. Any advise are appreciated. > -- > ER -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: directories that change into symlinks
Yes, I knew --force made a difference when not using --delete (that's very much like the example in the message I referred to), and I think it's good that way and --force should not be the default without --delete. I thought you were using --delete. If --force makes a difference in any case that --delete is used (and so far I have been unable to find any such case), I think it would make sense for --force to be automatically turned on by --delete. - Dave On Wed, Jan 08, 2003 at 07:39:22AM -0700, David Garamond wrote: > Dave Dykstra wrote: > >Could you please post your full command line and say where in the directory > >structure the directories were replaced by symlinks? I have not yet been > >able to come up with an example where --force makes a difference when > >using --delete and -r (or -a). Refer to > >http://lists.samba.org/pipermail/rsync/2002-January/005971.html > > this is the 'old' directory (the one that will become a mirror of the > 'new' dir). note that dir3 is still a directory and dir3 is not empty: > === > [dave@builder old]$ ls -laR > ...: > total 0 > drwxrwxr-x4 dave dave120 Jan 5 07:12 . > drwxrwxr-x5 dave dave120 Jan 5 07:13 .. > lrwxrwxrwx1 dave dave 4 Jan 5 07:12 dir1 -> dir2 > drwxrwxr-x4 dave dave 96 Jan 5 07:11 dir2 > drwxrwxr-x2 dave dave 72 Jan 5 07:12 dir3 > > .../dir2: > total 0 > drwxrwxr-x4 dave dave 96 Jan 5 07:11 . > drwxrwxr-x4 dave dave120 Jan 5 07:12 .. > drwxrwxr-x2 dave dave 72 Jan 5 07:11 dir11 > drwxrwxr-x2 dave dave 72 Jan 5 07:11 dir12 > > .../dir2/dir11: > total 0 > drwxrwxr-x2 dave dave 72 Jan 5 07:11 . > drwxrwxr-x4 dave dave 96 Jan 5 07:11 .. > -rw-rw-r--1 dave dave 0 Jan 5 07:11 file1 > > .../dir2/dir12: > total 0 > drwxrwxr-x2 dave dave 72 Jan 5 07:11 . > drwxrwxr-x4 dave dave 96 Jan 5 07:11 .. > -rw-rw-r--1 dave dave 0 Jan 5 07:11 file2 > > .../dir3: > total 0 > drwxrwxr-x2 dave dave 72 Jan 5 07:12 . > drwxrwxr-x4 dave dave120 Jan 5 07:12 .. > -rw-rw-r--1 dave dave 0 Jan 5 07:12 file3 > === > > this is the 'new' directory (the one that will be mirrored by the > 'old'). note that 'dir3' is now a symlink instead of a dir. > > === > [dave@builder new]$ ls -laR > ...: > total 0 > drwxrwxr-x4 dave dave120 Jan 5 07:13 . > drwxrwxr-x5 dave dave120 Jan 5 07:13 .. > drwxrwxr-x4 dave dave 96 Jan 5 07:11 dir1 > drwxrwxr-x2 dave dave 48 Jan 5 07:11 dir2 > lrwxrwxrwx1 dave dave 4 Jan 5 07:11 dir3 -> dir1 > > .../dir1: > total 0 > drwxrwxr-x4 dave dave 96 Jan 5 07:11 . > drwxrwxr-x4 dave dave120 Jan 5 07:13 .. > drwxrwxr-x2 dave dave 72 Jan 5 07:11 dir11 > drwxrwxr-x2 dave dave 72 Jan 5 07:11 dir12 > > .../dir1/dir11: > total 0 > drwxrwxr-x2 dave dave 72 Jan 5 07:11 . > drwxrwxr-x4 dave dave 96 Jan 5 07:11 .. > -rw-rw-r--1 dave dave 0 Jan 5 07:11 file1 > > .../dir1/dir12: > total 0 > drwxrwxr-x2 dave dave 72 Jan 5 07:11 . > drwxrwxr-x4 dave dave 96 Jan 5 07:11 .. > -rw-rw-r--1 dave dave 0 Jan 5 07:11 file2 > > .../dir2: > total 0 > drwxrwxr-x2 dave dave 48 Jan 5 07:11 . > drwxrwxr-x4 dave dave120 Jan 5 07:13 .. > === > > now compare these two commands. i find that the second one works. > > $ rsync -avz new/* old/ > $ rsync -avz --force new/* old/ > > -- > dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Not preserving permissions really preserves some
On Wed, Jan 08, 2003 at 03:08:38PM +1100, [EMAIL PROTECTED] wrote: > > > Thanks for the reply. > > I did see this on the to-do list, and thought that it might > be a preferable solution (just a bit more work). > It would be messy to do the xargs stuff as I am running rsync > over ssh with restricted commands based upon keys :-) > > Questions: >1. Is the proposed solution in the todo (--chmod) agreed? Yes. >2. Is anyone actively working on it? No. >3. If I wrote the patch to support the --chmod as outlined > in the todo file, would someone apply it (assuming my > code was okay ;-) ? Yes, although probably not in the very next release, probably in 2.6.0, unless the changes ended up being very isolated and we could clearly see there was very little risk in breaking something else. >4. Should the patch be applied to the "head", or should it > be against the latest release version? To the CVS "head". - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: IPv6 hosts allow|deny
On Wed, Jan 08, 2003 at 12:25:15AM +0100, Bert Vermeulen wrote: > On Tue, 7 Jan 2003, Dave Dykstra wrote: > > > Even though rsync maintenance isn't as bad as wget's, the maintainers > > are all VERY part time so that is a big part of the problem. Most of us > > don't have ipv6 systems to test things on. Can you vouch for the quality > > of the patch? I was able to get it with > > > > wget --passive >ftp://ftp.linux-ipv6.org/pub/usagi/misc/rsync-2_5_5-v6auth-20021016.patch.gz > > > > and it looks quite extensive. > > Yes, it's very extensive, and there's some code in there that's pretty > advanced. > > The fact of the matter is, I made my own patch for this functionality first, > and only then noticed Hideaki's patch -- and his, while not as readable to > the casual observer, seems like better code. He is rewriting the linux > kernel IPv6 stack, so I'd expect his rsync patch to be decent :-) > > Nevertheless, feel free to take a look at my patch: > http://biot.com/patches/rsync-ipv6-acl.patch > > So I haven't used Hideaki's patch, but I use mine, and it's good. Please do > apply one of them however. Since you're more familiar with that area of the code than any of the rsync maintainers and you think his patch is better, please test out his patch and look it over to see if you think there should be any changes. In particular, I see that yours includes documentation changes and his doesn't. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: am I missing something, or are permissions always preserved?
That sounds reasonable to me that rsync shouldn't try to preserve those extra bits without -p. Try making a patch and seeing if that works. I don't have Samba set up, but on a vfat filesystem on Linux rsync is doing even worse because after the initial file creation it does a "fchmod(fd, 0600)" which fails and it doesn't even copy files at all. - Dave On Wed, Jan 01, 2003 at 02:24:36PM -0800, Ben wrote: > I'm not sure what POSIX calls for, but on both Linux and AIX (the two > unix OSes I have access to) I can leave off the mode arg when calling > open(2). Granted, doing so gives me a mode that normally isn't so good, > but in my case this would be fine, as I have samba forcing modes to be > what I want them to be. > > However, after looking into this more, it turns out that samba > apparently causes chmod(2) to fail _only_ when trying to set modes > S_ISUID, S_ISGID, or S_ISVTX. Trying to set these modes in open(2), or > trying to set any other modes in open(2) or chmod(2), silently fails > without raising an error. > > So while this might well be a samba problem, it could be avoided if > rsync behaved as I would expect it to. The original files do indeed have > S_ISGUID set on them, but I don't see why rsync should try to preserve > this if I didn't tell it to preserve permissions. > > > On Wed, 2003-01-01 at 07:13, Green, Paul wrote: > > Ben [mailto:[EMAIL PROTECTED]] > > > > > I'm relatively new to rsync, but it seems to me that if there's an > > > option to perserve permissions and you don't set it, then rsync > > > shouldn't do anything with permissions. > > > > Sounds good, but recall how POSIX / Unix system calls work. To create a > > file, you have to specify what mode (permission) it should have. There is > > no way to tell creat() or open() to "do the default thing." > > > > I think a case could be made that this is a defect / incompatibility in the > > underlying file system layer that issues complaints about permissions. > > Before I attempted to code any kind of a fix, I'd sure want to read the > > specs of this layer and understand how the designers thought this problem > > should be approached. > > > > Thanks > > PG -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: filelist calculation algoritm
We've been calling this option --files-from rather than --file-list, to be like the GNU tar option. On Sun, Jan 05, 2003 at 09:55:50AM -0800, Wayne Davison wrote: > On Sat, Jan 04, 2003 at 05:03:02PM -0800, jw schultz wrote: > > that would produce destloc/srcdir/ > > when you might want a copy of srcdir at destloc instead of > > in destloc. > > Ah yes, I _was_ missing something. However, I still don't think we need > to clutter rsync with two types of --file-list options. This is already > something that people have to deal with when using the --relative option: > how to generate a file list that contains just the path information that > we need to be significant. I think that the removal of the undesired > prefixes should happen before the list gets to rsync rather than having > rsync do it (in your example the user would just chdir into "srcdir" and > do the "find" relative to '.'). I agree, there should only be one option. ... > FYI, the old rsync release that had a type of file-list functionality > was using a specialized include/exclude list. I believe that rsync > still walked the entire directory tree on both sides, and applied the > includes using a slightly different algorithm than the default (one that > did not require parent directories to be mentioned to get down to all > the specified files). I think that it would be nice to avoid the > directory-tree traversal, so I don't think we want to go this route. > However, this is another potential implementation method (and one that > would result in a syntax that is like what you suggested: one that uses > a single source dir on the command-line and doesn't require the use of > the --relative option). No, the include/exclude optimization completely bypassed the recursive traversal. It kicked in whenever the list ended with --exclude '*' and there were otherwise only includes with no wildcards. On Sun, Jan 05, 2003 at 01:18:06PM -0800, jw schultz wrote: > On Sun, Jan 05, 2003 at 12:44:32PM -0800, Wayne Davison wrote: > > On Sun, Jan 05, 2003 at 11:55:22AM -0800, jw schultz wrote: > > > The first problem is this would flatten things unless you used > > > relative and forced the user's CWD. That would cause considerable > > > confusion. > > > > Really? This is exactly how rsync works now with multiple file names on > > the command-line, so I don't see this as being any more confusing than > > what we already have. The rule would be you can specify the files on > > the command-line or on stdin (if you use '-' as the only source file). > > Since all names are treated in the same way regardless of where they > > were specified, everything works the same as it did before, only more > > names are now supported per invocation. I'm thinking that this way is > > more flexible since it allows someone to flatten things if that's what > > they really want to do. > > And the effect of that is that users wind looping to produce > scores of rsync sessions to transfer a single list. > > > > Secondly, how would you do it when the source location is remote? > > > Many of the users asking for this are doing pulls. > > > > I mentioned a protocol change that would send the extra file names to > > the other side after rsync starts up. Currently the send_files() > > routine always sends names from the sending side to the receiving side. > > The new protocol would change that to always send names from the user > > side to the server side when this option was specified. The user's > > command would look like this: > > > > rsync -avR remote:- /foo/bar > > > > The file list would be read from the local (user) side, of course. The > > remote command being run by rsync would look like this: > > > > ssh remote rsync --server --sender -vlogDtprR . - > > > > The presence of the '-' as the source would tell us to slurp names > > instead of send them. > > > > Since the file list is exchanged in total before we do any real work, I > > think this change would actually be really easy to implement. > > How many levels down should we allow - to mean "use this > directory as cwd for list"? > > rsync --relative remote::module/- dest/dir > > If it can only be "remote:-" then everything would have to > be relative to the user's home directory. > "../../usr/lib/somedir/somefile" anyone? And/or we > allow absolute paths in the list. So much for safe-links. > > In terms of implimentation i don't think we are that far > apart. As it stands now we walk the source list. For each > file/directory we check it against the pattern list prior to > insertion. At the time of insertion if recursion is turned on > each directory gets a readdir and the contents get the same > test and insert treatment. > > You are replacing the source list with stdin. I'm basically > saying that the list from stdin or from a file would be used > instead of readdir. > > Both cases require a protocol bump to support sending the > list for a pull. > > The discussion
Re: restricting rsync over ssh on the server side.
In the development version of rsync now in CVS, ssh and daemon mode can be used together by using '-e ssh' along with '::'. That is probably just what Rob needs, please check it out/test it. The documentation has been updated to describe putting a ssh wrapper key to to restrict rsync operations to only those defined by rsyncd.conf. - Dave Dykstra On Sun, Jan 05, 2003 at 01:07:30PM -0500, Aaron Morris wrote: > I do not think you can use it with ssh, but if you use rsync in rsync > mode (::) instead of just an interface to rsh (:), you can limit the > directories where you can transfer files (using modules). This involves > setting up the rsync daemon on the server side. The rsync daemon has > the ability to limit connections, chroot itself, prevent the use of > options (such as delete), use it's own authentication, setup > includes/excludes on the server side, and use IP based ACLs (outside of > tcpwrappers). See: `man rsyncd.conf` > > I only mention this because I do not believe most people even realize > there is this other mode to rsync. I tried describing it to a co-worker > who uses rsync regularly, but he kind of just stared at me blankly. > > Rob Browning wrote: > >I was wondering if it's possible to restrict rsync in various ways on > >the server side when it is invoked via ssh. Two restrictions I had in > >mind are disallowing deletes and/or restricting all actions to a > >particular subdirectory. I was hoping to be able to do this without > >having to be root (for a chroot) or having to set up special sshd > >server instances/chroots. > > > >If there's not already a way to do this, one possibility I had thought > >of is a ssh key command= wrapper, so that you could generate an ssh > >key like this: > > > > command="rsync-ssh-wrapper --root=/home/foo/bar --disable-delete",... > > > >and then when invoked rsync-ssh-wrapper would then look at > >SSH_ORIGINAL_COMMAND to see the actual incoming request (presuming > >there were any relevant options there; are rsync --server invocations > >documented anywhere?), and combine that with the wrapper options to > >decide how to invoke rsync --server. Of course this approach presumes > >that rsync --server would support suitable arguments. > > > >Is there interest in such a facility? It seems like something similar > >might be useful for sftp and scp as well, but I haven't managed to > >think of a way to implement a common solution. Also, I could imagine > >that this solution for rsync might be somewhat difficult to implement > >(perhaps complicated by symlinks, etc.), but it's the best thing I've > >thought of so far. > > > >Thanks > > > > -- > Aaron W Morris > decep > PGP Key ID: 259978D1 > > > -- > To unsubscribe or change options: > http://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: IPv6 hosts allow|deny
Even though rsync maintenance isn't as bad as wget's, the maintainers are all VERY part time so that is a big part of the problem. Most of us don't have ipv6 systems to test things on. Can you vouch for the quality of the patch? I was able to get it with wget --passive ftp://ftp.linux-ipv6.org/pub/usagi/misc/rsync-2_5_5-v6auth-20021016.patch.gz and it looks quite extensive. - Dave On Sun, Jan 05, 2003 at 03:53:13AM +0100, Bert Vermeulen wrote: > Hi all, > > Does anyone know what the status is of Hideaki Yoshifuji's IPv6 ACL patch? > He's submitted this twice now, and that was 3 months ago. Judging by the > comments made when this was submitted, it's apparently not understood how > important this is. > > - if you IPv6 on your rsync server, you can not secure it with "hosts > allow". > > - if you enable IPv6 on an rsync server that's secured with "hosts deny" to > keep certain hosts out, well, they can get in again -- if they also have > IPv6. > > Please, can this patch be merged into the next release? This is very > important! It is NOT new functionality, it's a security fix. > > > Bert Vermeulen > [EMAIL PROTECTED] -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: directories that change into symlinks
On Mon, Jan 06, 2003 at 08:36:35AM -0700, David Garamond wrote: > John Van Essen wrote: > >Dave, > > > >What you need is the --force option. It's not obvious from the > >all-too-generic name, but that will do the trick. > > > >To the rsync maintainers - this is somewhat of an FAQ. Perhaps > >the error message could also say '(see the --force option)' to > >help users discover the solution more easily. > > thanks john, that solved the problem. i should've perhaps looked at the > manpage twice. the manpage (2.5.5) already states this quite clearly: > > --force force deletion of directories even if not empty > > however, i personally think --force should be the default as it is more > common, or shouldn't it? Could you please post your full command line and say where in the directory structure the directories were replaced by symlinks? I have not yet been able to come up with an example where --force makes a difference when using --delete and -r (or -a). Refer to http://lists.samba.org/pipermail/rsync/2002-January/005971.html - Dave Dykstra -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Not preserving permissions really preserves some
I would object to that because it would change existing functionality
(emulating GNU cp) and isn't general enough. We've discussed a possible
general manipulation of permissions; see
http://lists.samba.org/pipermail/rsync/2002-March/006477.html
Alternatively, you could process the output of the -v option and run
xargs chmod on all files that have been touched.
- Dave Dykstra
On Thu, Jan 02, 2003 at 12:47:18PM +1100, [EMAIL PROTECTED] wrote:
>
> This appears to be a feature :-)
>
> I'm running rsync 2.5.5 on Solaris 8, and testing transferring
> files between two boxes with permission preservation turned *off*.
>
> I have three files:
>
> 700 with permissions 700
> 770 with permissions 770
> 777 with permissions 770
>
> The umask for the target system is set to: 007
> [I have checked this via prints from the rsync code]
>
> When the files get transferred, the permissions are:
>
> 700 -> permission 700
> 770 -> permission 770
> 777 -> permission 770
>
> This retains the permissions from the group part of the file 700.
> That is, even though my umask wants the group set, it doesn't
> get set.
>
> Why this is a problem for me:
>
> I don't trust the source system to get the permissions correct
> (it's a windows box running cygwin), and I want to force a set
> of permissions so the process using the synched directory will
> be able to access it (through group permissions).
>
> Suggested solution:
>
> I don't think changing the way permissions are created would be
> that backward-compatible, so maybe a new option?
>
> Create an option: --ignore-permissions
>
> in receive_file_entry(), in filelist.c
>
>
> == code block
> if (!preserve_perms) {
> extern int orig_umask;
> /* set an appropriate set of permissions based on original
>permissions and umask. This emulates what GNU cp does */
>
>file->mode &= ~orig_umask; <-- change this to line below
>
>file->mode = ((ignore_perms?0:0666) | file->mode) & ~orig_umask;
> }
>
> I used 0666 for the mask to pick up the execute bit (otherwise might
> screw up creating directories we can go down).
>
>
> I would be interested in any feedback on this problem from the maintainers.
> If there is any agreement on how it might be addressed and included in a
> new version, then I am happy to create all of the patch files and test it
> for submission to a maintainer.
>
> If any replies to the list could also cc my reply address, that would
> be appreciated.
>
> Regards,
> - Tim
>
>
>
>
>
> The information contained in this e-mail communication may be confidential.
> You should only read, disclose, re-transmit,copy, distribute, act in
> reliance on or commercialise the information if you are authorised to do
> so. If you are not the intended recipient of this e-mail communication,
> please immediately notify us by e-mail to [EMAIL PROTECTED], or reply
> by e-mail direct to the sender and then destroy any electronic and paper
> copy of this message.
>
> Any views expressed in this e-mail communication are those of the
> individual sender, except where the sender specifically states them to be
> the views of a member of the National Australia Bank Group of companies.
>
> The National Australia Bank Group of companies does not represent, warrant
> or guarantee that the integrity of this communication has been maintained
> nor that the communication is free of errors, virus or interference.
>
> --
> To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync windows -> unix still hanging :(
I've never seen anybody suggest any other value for --modify-window than "2". Does 90 really work better than 2? My understanding is that PC-style filesystems do not have 1 second timestamp granularity and so round the times up or down slightly. - Dave Dykstra On Fri, Jan 03, 2003 at 02:35:42PM -0800, Timothy Burt wrote: > > Glory Halelujah! > > I think I got it. > > It must have something to do with the file modification times and perhaps > the number of places of accuracy, perhaps in fractions of a second. > > When I add --modify-window=90 as a parameter. Then it works as it should. > > At least it does, for me now. You might try a smaller number (of seconds) > but for me. 90 seconds is close enough! > > On Fri, 3 Jan 2003, Mike Rubel wrote: > > > > > On Fri, 3 Jan 2003, Timothy Burt wrote: > > > I am seeing similar symptoms. > > > > Hi Timothy, > > > > I'm trying to find out more about this. May I ask, are the files you see > > sent even when they haven't changed regular files, or directories, or > > both? > > > > Mike > > > > > > -- > > Timothy Burt > General Manager > Arbor Group LLC > Los Angeles, Calif. USA > > -- > To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync feature suggestion
What Hadmut wants is the oft-requested and discussed "files-from" option that I once offered to write but haven't been able to get to. Andy Schor in http://lists.samba.org/pipermail/rsync/2001-November/005272.html posted a patch for something similar but it only worked when the sender was on the local machine and not when it was remote (among other issues). I don't believe you've posted your patch, Justin; does your "files-from" directly contain the list of files to send and skip the recursive traversal? If so, I don't see the point of having rsync have the extra regex options you mention because those could all be done by external greps that pre-process the file list. - Dave On Fri, Jan 03, 2003 at 11:51:05AM -0600, Justin Banks wrote: > Max Bowsher wrote > > Hadmut Danisch wrote: > > > I'd like to suggest a new feature to rsync. > > > > > I am mirroring a debian archive, but unfortunately, > > > debian mixes all files of several distributions in a > > > subtree /pool. There is no way to select only the files > > > of a certain distribution through a simple exclude/include > > > expression. > > > > > > There is a tool called debmirror, which first downloads > > > the distribution index files, extracts all the filenames/paths > > > of the files needed and then calls rsync for every single file. > > > Thats certainly not useful, especially since rsync shows the > > > servers motd for every single file. > > > > I was about to suggest: > > $ rsync --include-from=list-file --exclude=\* > > but of course that will exclude the parent directories of files you want, > > causing them to be ignored. > > > > This might work: > > $ rsync --include-from=list-file --include=\*\*/ --exclude=\* > > > > although it will mirror the entire directory structure (but not unspecified > > files). > > > > Probably, rsync should be taught that: "If I explicitly include a file, look > > for it explicitly, even if I've excluded a parent directory." > > Not too long ago, I modified/mangled rsync to do > > rsync --files-from /some/file --include-regexes /some/regular/expressions \ > --exclude-regexes /some/regular/expressions > > > such that all the files in /some/file would be sent iff they matched the > posix regexes in --include-regexes and didn't match the ones in > --exclude-regexes (if present). > > I don't have a wide variety of platforms to test it on, but it worked okay > on linux, solaris, and irix. > > -justinb -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: --copy-unsafe-links, links preserved in source tree or local directory?
That indeed is not right, and it appears that other people have noticed the
problem before but nobody has come up with a fix. The problem must be related
to the unsafe_symlink() function in util.c. Martin noticed a strange thing in
the code (see http://lists.samba.org/pipermail/rsync/2002-April/006759.html)
but I just tried his suggestion and it didn't help with your example. I'm
sure I could figure it out if I spent more time on it but I've got other
priorities right now. Try fiddling with it to see if you can come up with
a better solution. It might help for you to build the t_unsafe test program
that's in the Makefile but not built by default. I see Martin has also has
a test for the option in testsuite/unsafe-links.test, and it would be good if
that could be expanded to test for this case.
- Dave
On Thu, Jan 02, 2003 at 03:02:46PM -0800, Seth Olsen wrote:
>
> Hi,
>
> I'm a bit confused about what the '--copy-unsafe-links' option
> considers the 'source tree'. In the man page it says that for links
> pointing outside the 'source tree' the file will be copied, but when I
> try this, for all links pointing outside of the local directory the
> files are copied even though the links point to files that are in the
> directory tree that is being rsynced. I couldn't find anything definitive
> on the web either. See the example below.
>
> Is this the expected behaviour? Is the 'source tree' just the
> current directory? If so, it would be nice to have an option where
> only links pointing outside of the 'parent source tree'
> ('Test' in the example below) would be copied as files.
>
> rsyc-2.5.4 on RedHat-7.3 i386, XFS filesystem
>
> Thanks,
>
> Seth
> [EMAIL PROTECTED]
>
> example:
>
> chump0:~> ll Test/*
> Test/Q1:
> total 0
> -rw-rw-r--1 olsenolsen 0 Jan 2 14:05 q1_1
> -rw-rw-r--1 olsenolsen 0 Jan 2 14:05 q1_2
> lrwxrwxrwx1 olsenolsen 4 Jan 2 14:06 q1_3 -> q1_2
> lrwxrwxrwx1 olsenolsen 24 Jan 2 14:09 q1_4 ->
>/home/olsen/Test/Q2/q2_1
> lrwxrwxrwx1 olsenolsen 15 Jan 2 14:42 q1_5 -> ../../test/qw.F
>
> Test/Q2:
> total 0
> -rw-rw-r--1 olsenolsen 0 Jan 2 14:06 q2_1
> -rw-rw-r--1 olsenolsen 0 Jan 2 14:06 q2_2
> lrwxrwxrwx1 olsenolsen 4 Jan 2 14:06 q2_3 -> q2_1
> lrwxrwxrwx1 olsenolsen 10 Jan 2 14:06 q2_4 -> ../Q1/q1_1
> lrwxrwxrwx1 olsenolsen 10 Jan 2 14:06 q2_5 -> ../Q1/q1_3
>
> chump0:~> rsync -rl --copy-unsafe-links Test Test2
>
> chump0:~> ll Test2/Test/*
> Test2/Test/Q1:
> total 4.0k
> -rw-rw-r--1 olsenolsen 0 Jan 2 14:48 q1_1
> -rw-rw-r--1 olsenolsen 0 Jan 2 14:48 q1_2
> lrwxrwxrwx1 olsenolsen 4 Jan 2 14:48 q1_3 -> q1_2
> -rw-rw-r--1 olsenolsen 0 Jan 2 14:48 q1_4
> -rw-rw-r--1 olsenolsen 275 Jan 2 14:48 q1_5
>
> Test2/Test/Q2:
> total 0
> -rw-rw-r--1 olsenolsen 0 Jan 2 14:48 q2_1
> -rw-rw-r--1 olsenolsen 0 Jan 2 14:48 q2_2
> lrwxrwxrwx1 olsenolsen 4 Jan 2 14:48 q2_3 -> q2_1
> -rw-rw-r--1 olsenolsen 0 Jan 2 14:48 q2_4
> -rw-rw-r--1 olsenolsen 0 Jan 2 14:48 q2_5
> --
> To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Simulating rdist?
See the BATCH MODE section in the rsync 2.5.5 man page. It's stil pretty new and experimental, so it may not do what you want, but currently it's the best that rsync can do. - Dave Dykstra On Wed, Jan 01, 2003 at 04:39:06PM -, va_public <[EMAIL PROTECTED]> wrote: > rsync is great for syncing 2 directory trees, but I want to maintain > a master source tree on one machine and copy that to multiple > machines. i.e. basically what rdist does > > The only way I can see of doing this with rsync is to have multiple > cron jobs > > 0 * * * * rsync ... machine1:... > 0 * * * * rsync ... machine2:... > and so on. > > Is there a more elegant/compact way of doing this? > > Thanks > > -- > To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: am I missing something, or are permissions always preserved?
OK, now I understand. I have seen those errors also when writing onto a PC filesystem mounted by Linux. Other programs like tar and cpio have the same problem, so I've just ignored the messages. Maybe everybody else does the same. Maybe a well thought out and documented patch for a new flag would get put in. It would be even better if it could be detected automatically, but I'm not sure how to do it. How do people feel about just silently ignoring the error if -p wasn't specified? - Dave On Tue, Dec 31, 2002 at 01:31:07PM -0800, Ben wrote: > Yeah, my problem is that the chmod fails, so then rsync complains not > everything worked as expected. Because I'm writing files to a network > mount with forced permissions, I would like a way for rsync to simply > create the file and never try to chmod what it creates. Normally that > would probably be a bad idea, but when writing to a share with forced > permissions > > I'm actually a bit surprised nobody else has run into this. > > On Tue, 2002-12-31 at 13:20, Dave Dykstra wrote: > > Then what would you expect it to do? I'm guessing your only problem is > > that the chmod is failing, and you would rather have it create files with > > the final permissions in the first place; is that it? I believe it is > > done this way because of worries of potential security problems, where > > temporary files might be accessible by more people than the original > > file was. Or is the issue that you rather have it always create file > > permissions based only on umask? In that case I don't know how it would > > know whether or not to turn on the executable permission, among other > > problems. What exactly would your proposed flag do? > > > > - Dave > > > > > > On Tue, Dec 31, 2002 at 08:06:52AM -0800, Ben wrote: > > > No, these are for new files. Existing files work perfectly, but, like > > > you said before, for new files rsync creates the file then attempts to > > > alter the permissions based on the origional permissions and umask. > > > > > > On Tue, 2002-12-31 at 07:58, Dave Dykstra wrote: > > > > What do you mean, "altered"? Do the destination files already exist? > > > > It is supposed to preserve existing permissions on destination files > > > > when you don't use -p. > > > > > > > > - Dave > > > > > > > > On Mon, Dec 30, 2002 at 06:44:24PM -0800, Ben wrote: > > > > > Hmmm... while that makes sense, that doesn't really help me in my > > > > > situation, where permissions cannot be altered because of the network > > > > > mount they are being written to. > > > > > > > > > > Does it make sense to impliment a "don't touch permissions" flag? -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: am I missing something, or are permissions always preserved?
Then what would you expect it to do? I'm guessing your only problem is that the chmod is failing, and you would rather have it create files with the final permissions in the first place; is that it? I believe it is done this way because of worries of potential security problems, where temporary files might be accessible by more people than the original file was. Or is the issue that you rather have it always create file permissions based only on umask? In that case I don't know how it would know whether or not to turn on the executable permission, among other problems. What exactly would your proposed flag do? - Dave On Tue, Dec 31, 2002 at 08:06:52AM -0800, Ben wrote: > No, these are for new files. Existing files work perfectly, but, like > you said before, for new files rsync creates the file then attempts to > alter the permissions based on the origional permissions and umask. > > On Tue, 2002-12-31 at 07:58, Dave Dykstra wrote: > > What do you mean, "altered"? Do the destination files already exist? > > It is supposed to preserve existing permissions on destination files > > when you don't use -p. > > > > - Dave > > > > On Mon, Dec 30, 2002 at 06:44:24PM -0800, Ben wrote: > > > Hmmm... while that makes sense, that doesn't really help me in my > > > situation, where permissions cannot be altered because of the network > > > mount they are being written to. > > > > > > Does it make sense to impliment a "don't touch permissions" flag? -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: am I missing something, or are permissions always preserved?
What do you mean, "altered"? Do the destination files already exist?
It is supposed to preserve existing permissions on destination files
when you don't use -p.
- Dave
On Mon, Dec 30, 2002 at 06:44:24PM -0800, Ben wrote:
> Hmmm... while that makes sense, that doesn't really help me in my
> situation, where permissions cannot be altered because of the network
> mount they are being written to.
>
> Does it make sense to impliment a "don't touch permissions" flag?
>
> On Tue, 2002-12-24 at 05:08, Dave Dykstra wrote:
> > When preserve_perms is not set, rsync sets a default permission based on
> > the original permissions and the umask. A comment in flist.c says that is
> > what GNU cp does, so that's why rsync does it. Comments in generator.c
> > and receiver.c indicate that if a file already exists and preserve_perms
> > isn't set, the original permissions should be preserved.
> >
> > - Dave
> >
> > On Sat, Dec 21, 2002 at 12:30:55PM -0800, Ben wrote:
> > > They seem to be for me, even when I don't pass in the --perms flag. This
> > > is a problem because I'm rsyncing to a samba mount with fixed
> > > permissions.
> > >
> > > Looking at rsync.c (for version 2.5.5) starting at line 204, I see this
> > > code snippet:
> > >
> > > #ifdef HAVE_CHMOD
> > > if (!S_ISLNK(st->st_mode)) {
> > > if (st->st_mode != file->mode) {
> > > updated = 1;
> > > if (do_chmod(fname,file->mode) != 0) {
> > > rprintf(FERROR,"failed to set permissions on %s : %s\n",
> > > fname,strerror(errno));
> > > return 0;
> > > }
> > > }
> > > }
> > > #endif
> > >
> > > I would have expected to see a test for the setting of perserve_perms,
> > > but it's not there. Is this a bug or is there there a reason
> > > perserve_perms isn't checked?
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync 2.5.5 SCO Unixware patch
On Mon, Dec 23, 2002 at 11:42:07PM -0800, Jos Backus wrote:
> On Mon, Dec 23, 2002 at 02:13:51PM -0800, Stephen Friedl wrote:
> > The change in popt/popt.c is to work around a bug in the SCO UNIXWare 8
> > compiler: it doesn't properly deal with alloca() being called from the
> > middle of another function call. The code dumps badly on this platform
> > with all modes (debug, regular, optimized), and the workaround shown
> > gets around the bug and likely obviates similar bugs on other platforms.
...
> > --- popt/popt.c.origMon Dec 23 21:51:59 2002
> > +++ popt/popt.c Mon Dec 23 21:59:21 2002
> > @@ -443,9 +443,10 @@
> > }
> >
> > /* Make a copy we can hack at */
> > - localOptString = optString =
> > - strcpy(ALLOCA(strlen(origOptString) + 1),
> > - origOptString);
> > + { int n = strlen(origOptString) + 1;
> > + localOptString = optString = ALLOCA(n);
> > + strcpy(optString, origOptString);
> > + }
> >
> > if (!optString[0])
> > return POPT_ERROR_BADOPT;
...
> I don't feel brave enough to commit the popt/popt.c change without input from
> others.
The patch seems harmless, although I'm not sure it's the best style.
How about making strcpy a macro instead? popt/popt.c is borrowed code,
and if we make modifications to it it might get forgotten the next time
it is upgraded.
- Dave Dykstra
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: deflate on token returned 0 (16384 bytes left)
First, you should know that rsyncing a .gz file is not very worthwhile because the rsync algorithm does not do a good job of efficiently transferring them, unless you've got a modified gzip that adds a --rsyncable option and you used the option. The patch is in the 2.5.5 source patches/gzip-rsyncable.diff. According to my email record, I once found a workaround by deleting the "|| tx_strm.avail_in != 0" on line 285 of token.c. That change has not been put into 2.5.5 or into a later development version because it was unknown what the side effects were. - Dave Dykstra On Mon, Dec 23, 2002 at 09:53:21AM -0500, John Murtari wrote: > Hello All, > > I have searched for this error and found similiar errors, but > everything seems to indicate that this should be fixed in the 2.5.5 > version. > > rsync version 2.5.5 protocol version 26 > > I am running rsync with -axz, trying to sync up a large gz file : > > Source machine: > > -rw-r--r--1 root other175214792 Dec 22 00:17 >/export/tools2/local/etc/httpd/logs/access/access_log.12-22-02.gz > > Destination: > -rw-r--r--1 root other103372258 Dec 19 00:18 access_log.12-22-02.gz > > > Rsync is running with rsh to a remote machine running in daemon mode. > > > The full command line is : > /bin/nice rsync -R -l -H --quiet --stats -axz --delete --blocking-io /export/tools2 >drill::anvil > > welcome to drill.thebook.com > > building file list ... done > export/tools2/local/etc/httpd/logs/access/ > /export/tools2/local/etc/httpd/logs/access/access_log.12-22-02.gz > deflate on token returned 0 (16384 bytes left) > rsync error: error in rsync protocol data stream (code 12) at token.c(288) > > > > If the remote copy of the file is removed, rsync will run to > completion, so it seems to be a problem perhaps with the -z option > trying to update a .gz file??? > > TIA! > > -- > John > ___ > John Murtari Software Workshop Inc. > [EMAIL PROTECTED] 315.695.1301(x-211)"TheBook.Com" (TM) > http://www.thebook.com/ > -- > To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: am I missing something, or are permissions always preserved?
When preserve_perms is not set, rsync sets a default permission based on
the original permissions and the umask. A comment in flist.c says that is
what GNU cp does, so that's why rsync does it. Comments in generator.c
and receiver.c indicate that if a file already exists and preserve_perms
isn't set, the original permissions should be preserved.
- Dave
On Sat, Dec 21, 2002 at 12:30:55PM -0800, Ben wrote:
> They seem to be for me, even when I don't pass in the --perms flag. This
> is a problem because I'm rsyncing to a samba mount with fixed
> permissions.
>
> Looking at rsync.c (for version 2.5.5) starting at line 204, I see this
> code snippet:
>
> #ifdef HAVE_CHMOD
> if (!S_ISLNK(st->st_mode)) {
> if (st->st_mode != file->mode) {
> updated = 1;
> if (do_chmod(fname,file->mode) != 0) {
> rprintf(FERROR,"failed to set permissions on %s : %s\n",
> fname,strerror(errno));
> return 0;
> }
> }
> }
> #endif
>
> I would have expected to see a test for the setting of perserve_perms,
> but it's not there. Is this a bug or is there there a reason
> perserve_perms isn't checked?
>
> --
> To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Status of sigchld_handler bug?
On Wed, Dec 18, 2002 at 05:00:00PM -0800, [EMAIL PROTECTED] wrote: > Will this patch ever be integrated into the main code? > > http://lists.samba.org/pipermail/rsync/2002-February/006371.html That bug has got to get fixed before the next release, although I think the patch at http://lists.samba.org/pipermail/rsync/2002-September/008227.html is safer. As far as I know we've still never heard from Tridge why he introduced the code that caused the bug. - Dave Dykstra -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Problem with absolute symbolic links
First question: must you use an rsync daemon on the Unix side? Rsh or Ssh is a lot easier. If you do need to use it, you need to know that the original intention of rsync modules was to only permit files under that module to be accessible. With the default "use chroot = yes" option, rsync chroot's to the directory so no files outside of the top path can be reached. Long ago I added the "use chroot = no" option so an rsync daemon could be run under a non-root user id (because only root can run chroot()), but even then it goes through pains to filter out paths that attempt to go outside of the top module. However, I believe, but am not sure, that if --copy-links is used with "use chroot = no" it should pull in files pointed to by symlinks from outside the top. Try it out. - Dave Dykstra On Fri, Dec 13, 2002 at 01:44:30PM -0500, Kevin Minder wrote: > I'm trying to synchronize a particularly troublesome directory structure on a SunOS >box to a PC. I really want all symbolic links resolved to real files on the PC when >everything is done. I can't seem to find any combination of parameters that will >accomplish this. > > It boils down to two problems. > 1. It seems like that even if I exclude a symbolic link that is a directory it is >not excluded. > 2. It seems like copy-links and copy-unsafe-links don't work for symbolic links to >directories outside the source tree. > > Is this the correct behavior or am I doing something wrong. > Any input is greatly appreciated. > > Here is my rsyncd.conf file... > --- > motd file = /etc/rsyncd.motd > log file = /var/log/rsyncd.log > pid file = /var/run/rsyncd.pid > uid = kminder > gid = svrtech > > [test] > path = /home/kminder/rsync/test > comment = test > read only = yes > list = yes > --- > > Below is a simplified source structure on the unix side (source). > Note the .rel symlink and the .abs symlink. > > --- > /home/kminder/rsync> ls -al outside test > outside: > total 16 > drwxr-xr-x 2 kminder g9044096 Dec 13 11:37 . > drwxr-xr-x 5 kminder g9044096 Dec 13 11:29 .. > -rw-r--r-- 1 kminder g904 15 Dec 13 11:37 file > > test: > total 24 > drwxr-xr-x 3 kminder g9044096 Dec 13 11:36 . > drwxr-xr-x 5 kminder g9044096 Dec 13 11:29 .. > lrwxrwxrwx 1 kminder g904 27 Dec 13 11:36 .abs -> >/home/kminder/rsync/outside > lrwxrwxrwx 1 kminder g904 3 Dec 13 11:34 .rel -> dir > lrwxrwxrwx 1 kminder g904 9 Dec 13 11:36 absfile -> .abs/file > drwxr-xr-x 2 kminder g9044096 Dec 13 11:34 dir > -rw-r--r-- 1 kminder g904 10 Dec 13 11:32 file > lrwxrwxrwx 1 kminder g904 4 Dec 13 11:33 link -> file > lrwxrwxrwx 1 kminder g904 4 Dec 13 11:33 linklink -> link > lrwxrwxrwx 1 kminder g904 9 Dec 13 11:35 relfile -> .rel/file > --- > > I'm trying for the following structure on the pc (cygwin ls -al). > Note that all links should be resolved. > I want the duplicate files. > > --- > drwxr-xr-x+ 4 kminder None 4096 Dec 13 11:36 . > drwxrwxrwx+ 14 Administ Administ 8192 Dec 13 11:20 .. > drwxr-xr-x+ 2 kminder None0 Dec 13 11:34 dir > -rw-r--r--1 kminder None 10 Dec 13 11:32 file > -rw-r--r--1 kminder None 10 Dec 13 11:32 link > -rw-r--r--1 kminder None 10 Dec 13 11:32 linklink > -rw-r--r--1 kminder None 14 Dec 13 11:34 relfile > -rw-r--r--1 kminder None 14 Dec 13 11:34 absfile > --- > > When I execute the command that I think should accomplish this... > D:\sync>rsync -rptgDvP --port=34343 --copy-links --copy-unsafe-links --exclude .rel >--exclude .abs kminder@usunnbd25::test d:\sync > The resulting output follows... > > --- > receiving file list ... > readlink absfile: No such file or directory > 7 files to consider > ../ > dir/ > dir/file > 14 100%0.00kB/s0:00:00 > file > 10 100%9.77kB/s0:00:00 > link > 10 100%9.77kB/s0:00:00 > linklink > 10 100%9.77kB/s0:00:00 > relfile > 14 100% 13.67kB/s0:00:00 > wrote 190 bytes read 465 bytes 187.14 bytes/sec > total size is 58 speedup is 0.09 > rsync error: partial transfer (code D:\sync>dir23) at /tmp/rsync-2.5.5/main.c(926) > --- > > My disk ends up looking as
Re: Rsync performance increase through buffering
On Sun, Dec 08, 2002 at 11:48:57PM -0800, Craig Barratt wrote: > I've been studying the read and write buffering in rsync and it turns > out most I/O is done just a couple of bytes at a time. This means there > are lots of system calls, and also most network traffic comprises lots > of small packets. The behavior is most extreme when sending/receiving > file deltas of identical files. ... > Below is a patch to a few files that adds read and write buffering in > the places where the I/O was unbuffered, adds buffering to write_file() > and removes the unneeded gettimeofday() system call in show_progress(). ... > However, on a network test doing a send from cygwin/WinXP to rsyncd > on rh-linux the running time improves from about 700 seconds to 215 > seconds (with a cpu load of around 17% versus 58%, if you believe > cygwin's cpu stats). This is probably an extreme case since the system > call penalty in cygwin is high. But I would suspect a significant > improvement is possible with a slow network connection, since a lot > less data is being sent. Looks like a good candidate for the "patches" directory now and for integration in 2.6.0. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync stoped syncing
Is /home now a symlink? The fact that the "is uptodate" message is only
appearing on a single file ("home") makes me think it is not recursing
into the directory. If so, just change the end of your command line to
... /home/ slave:/home/
- Dave
On Mon, Dec 09, 2002 at 04:36:41PM +0100, Markus Lamers wrote:
> Hi,
>
> I want to keep the home-Directorys of 2 machines in sync.
> Therefor I start rsync every hour with cron.
> In the Beginning all worked fine, but now ...
> I searched, but can't see the problem.
>
> The home-directorys on the "master" change very often, but the
> changes are no longer replicated to the "slave". Both run Linux and
> rsync 2.5.5.
> The master calls (the long line may be wrapped here)
>
> rsync -auvxz --delete --exclude-from /root/.rsync/home-daily.exc /home
> slave:/
>
> once a hour and after building the filelist it says "home is uptodate".
> But when I make a "ls -rtl" on both machines .. its definitly not.
> master:
> (just the last few lines)
> drwxrwxrwx5 root root 143 Dec 9 14:51 DRKURSCH4722
> drwxrwxrwx5 root root 250 Dec 9 15:16 TESTBMZ2
> drwxrwxrwx5 root root 143 Dec 9 15:51 DRSTIEH1443
> drwxrwxrwx5 root root 2647 Dec 9 16:15 KRHBRUCHSAL3140
> drwxrwxrwx4 root root 124 Dec 9 16:17 NERESHEIM7059
> drwxrwxrwx5 root root 143 Dec 9 16:18 KURPFALZKHD4560
> drwxrwxrwx5 root root 143 Dec 9 16:18 DRREIFER1740
> drwxrwxrwx5 root root 143 Dec 9 16:25 DRFETTIG0690
> master:/home #
>
> slave:
>
> drwxrwxrwx5 root root 118 Oct 19 00:15 DRHAEBER7009
> drwxrwxrwx5 root root 118 Oct 19 00:15 DRELSER1847
> drwxrwxrwx5 root root 118 Oct 19 00:15 DRDIEDRP2878
> drwxr-xr-x 299 root root 8146 Oct 21 11:56 .
> drwxr-xr-x3 welteusers 948 Oct 22 00:15 welte
> drwxr-xr-x 13 oracle oinstall 1444 Nov 11 17:11 oracle
> drwxrwxrwx5 root root 193 Nov 29 15:37
> RECHSHGVOELKLINGEN4300
> drwxrwxrwx 26 horras users2981 Nov 29 15:45 horras
> drwxrwxrwx 11 szyman users1500 Dec 9 11:08 szyman
> drwxrwxrwx 13 labtimer users 13030 Dec 9 13:10 labtimer
> slave:/home #
>
> The date on both is the same.
>
> Any hints wants going wrong here ?
>
> Thanks and kind regards,
> Markus Lamers
> --
> Markus Lamers <[EMAIL PROTECTED]>
> Labor Prof. Seelig & Kollegen
>
> --
> To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Patch to ignore exluded files.
I remember that I put in that code, not so long ago, because I was getting
errors with symlinks pointing nowhere even though they were excluded.
It is revisions 1.96 and 1.95 at
http://cvs.samba.org/cgi-bin/cvsweb/rsync/flist.c
As far as I could see, the only time that the ENOENT error could occur
is if copy_links is on; readlink_stat() won't try to follow the symlink
otherwise. Ah, I see there might also be a case when copy_unsafe_links is
on; are you using --copy-unsafe-links? If not, can you please completely
specify a small set of steps, beginning from an empty directory, that
someone could do to demonstrate the problem for which you believe the
below patch will help?
Oh, I see the original poster was talking about files being dynamically
created and deleted during an rsync run; that was something I had not
thought of. In general, rsync does not even try to support such situations
because they are so difficult to handle robustly. However, I suppose
it makes sense to ignore files that are excluded. My first thought
was that it might impact performance for the normal case, but really it
won't because the excludes are only checked when a ENOENT failure occurs.
I'll go ahead and take out the check for copy_links.
- Dave Dykstra
On Thu, Dec 05, 2002 at 05:04:38PM -0600, Bo Kersey wrote:
> I came up with a patch to fix the problem of IO Errors caused by
> excluded files as did Eugene V. Chupriyanov below.
>
> Is there a chance that this change will show up in a future version of
> rsync?
>
> Is there a reason that we should not ignore IO errors when copy_links is
> off? Just want to make sure that I'm not missing something here that
> may corrupt my syncs
>
> Here's the version that I came up with before finding Eugene's patch:
>
> --- rsync-2.5.5/flist.c.orig 2002-03-14 15:20:20.0 -0600
> +++ rsync-2.5.5/flist.c 2002-12-02 19:27:02.0 -0600
> @@ -644,8 +644,8 @@
>
> if (readlink_stat(fname, &st, linkbuf) != 0) {
> int save_errno = errno;
> - if ((errno == ENOENT) && copy_links && !noexcludes) {
> - /* symlink pointing nowhere, see if excluded */
> + if ((errno == ENOENT) && !noexcludes) {
> + /* File or directory not found, see if excluded */
> memset((char *) &st, 0, sizeof(st));
> if (check_exclude_file(f, fname, &st)) {
> /* file is excluded anyway, ignore silently */
>
>
>
>
> > Dear friends!
> >
> > I really appriciate the great job you did on rsync package and hope for its
> > further improvement and development. It works great in our pretty complex
> > environment and helps us a lot in our daily duties.
> >
> > But we've run into couple of small inconveniences.
> >
> > First one:
> > We use rsyncd to propagate a tree of html files among several web-servers..
> > That files are generated on the fly and updated every few minutes.
> >
> > Here is relevant part of rsyncd.conf:
> > [sync-include]
> > path=/usr/local/www/rbc/win/include
> > exclude=*tmp
> > use chroot=false
> > max connections=2
> > read only=true
> > list=false
> > uid=rbcmaint
> > gid=www-all
> >
> > (Note the exlude line)
> >
> > Nevertheless I was getting errors like this:
> >
> > readlink .../include/cnews.html.tmp: No such file or directory
> > rsync error: partial transfer (code 23) at main.c(578)
> >
> > I'm not sure I fixed it clean, but I get rid of that errors by patching
> > flist.c. Here is the diff:
> >
> > --- flist.c.origMon Jul 8 17:48:12 2002
> > +++ flist.c Mon Jul 8 17:48:39 2002
> > @@ -644,7 +644,7 @@
> >
> > if (readlink_stat(fname, &st, linkbuf) != 0) {
> > int save_errno = errno;
> > - if ((errno == ENOENT) && copy_links && !noexcludes) {
> > + if ((errno == ENOENT) && !noexcludes) {
> > /* symlink pointing nowhere, see if excluded */
> > memset((char *) &st, 0, sizeof(st));
> > if (check_exclude_file(f, fname, &st)) {
> >
> > The second problem appears when we need that target files always have
> > correct permissions - we do some testing on them and we receive errors, when
> > file already copied and renamed from tempfile, but permission are not yet
> > set. So I fixed it with following diff:
> >
> > --- rsync.c.origFr
