Re: does --whole-file always work?

2009-05-16 Thread Dave Dykstra
--whole-file definitely does something: it turns off the rsync rolling
checksum algorithm, which avoids reading the destination file before
writing it.  It just doesn't avoid passing the data between two
processes and the other things that rsync normally does. 

- Dave

On Sat, May 16, 2009 at 07:49:05AM +0300, Amir Rapson wrote:
 Thanks Matt and Dave,
 But now I'm a bit confused, it doesn't work because it has not been done  
 yet, but it's the default?
 Are there any plans of improving the performance with local files?
 Do you mind if I give it a go?

 Thanks,
 Amir

 --
 From: Dave Dykstra dwdrs...@drdykstra.us
 Sent: Friday, May 15, 2009 10:43 PM
 To: Amir Rapson amir.rap...@gmail.com
 Cc: rsync@lists.samba.org
 Subject: Re: does --whole-file always work?

 On Thu, May 14, 2009 at 09:59:16AM -0400, Matt McCutchen wrote:
 On Thu, 2009-05-14 at 15:59 +0300, Amir Rapson wrote:
  Running rsync with --whole-file yields poorer performance results than
  a simple cp.
  Looking at the code - it looks like copy_file isn't really called
  when I add the --whole-file flag. The regular receive_data is doing
  the copy.

 That's right.  In the current design, file transfers are always done
 over the connection between the sender and receiver and copy_file is
 used only for copies within the destination (which happen with a few
 options such as --copy-dest).  It would be possible to make a special
 case for local runs and have the receiver open the source file directly
 (or something like that), but this has not been done yet.

 Also note that --whole-file is the default if both source and destination
 are local.  Compare it to --no-whole-file and you'll see worse  
 performance.

 - Dave 

 -- 
 Please use reply-all for most replies to avoid omitting the mailing list.
 To unsubscribe or change options: 
 https://lists.samba.org/mailman/listinfo/rsync
 Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: does --whole-file always work?

2009-05-15 Thread Dave Dykstra
On Thu, May 14, 2009 at 09:59:16AM -0400, Matt McCutchen wrote:
 On Thu, 2009-05-14 at 15:59 +0300, Amir Rapson wrote:
  Running rsync with --whole-file yields poorer performance results than
  a simple cp. 
  Looking at the code - it looks like copy_file isn't really called
  when I add the --whole-file flag. The regular receive_data is doing
  the copy.
 
 That's right.  In the current design, file transfers are always done
 over the connection between the sender and receiver and copy_file is
 used only for copies within the destination (which happen with a few
 options such as --copy-dest).  It would be possible to make a special
 case for local runs and have the receiver open the source file directly
 (or something like that), but this has not been done yet.

Also note that --whole-file is the default if both source and destination
are local.  Compare it to --no-whole-file and you'll see worse performance.

- Dave
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: rsync --link-dest option with the destination directory containing old files.

2008-12-23 Thread Dave Dykstra
On Tue, Dec 23, 2008 at 11:45:36PM +1100, David Overton wrote:
 2008/12/23 Robert Bell robert.b...@csiro.au:
  It seems to us that there is a good case for the enhanced functionality.
 
  When a file is found in the destination which should be replaced by one
  in the source, look in the --link-dest directory first for a candidate,
  and hard-link that in preference to doing a copy from source to
  destination.
 
 I would also very much like to see this feature.  Indeed, this seems
 far more logical than the current --link-dest behaviour and it's what
 I assumed --link-dest would do until I read the man page thoroughly
 (you have to follow the references back from --link-dest to
 --copy-dest and then --compare-dest and even then the only mention of
 the actual behaviour is a parenthesised comment (if the files are
 missing in the destination directory)).  I be interested to know what
 use cases the current  behaviour was designed for,  because I can't
 see any advantage to not making use of the --link-dest file if it's
 available.  Providing the proposed alternative behaviour as an extra
 option, if not the default for --link-dest, would be very useful.

That's a parenthetical comment in --compare-dest because that's normal
rsync behavior.  If a file already exists at the destination, rsync
assumes as normal that it has to compare against it when copying the
file from the origin.  The *-dest options only come into effect when
the destination file is missing.  You're asking to have an option that
will always ignore the destination file, then behave as --link-dest
does now, and then overwrite the destination?

I don't understand what's the point of doing link-dest if you're not
going to keep around a linked copy of the entire directory tree, so I
don't see why you would right away rm -r an old tree.  Describe more
fully how you want to the disk to be left after the operation and what
you're trying to accomplish.  Depending on what you're trying to do,
there may be another combination you can use.  Rob gave the example of
rsync -a --link-dest=../backup.1 source/ backup.0/
where backup.0 already had files in it.  Is it that you're trying
to alternate between backup.0 and backup.1 so you have two complete
backups with as much hardlinked between them as you can?

- Dave
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: rsync --link-dest option with the destination directory containing old files.

2008-12-23 Thread Dave Dykstra
On Tue, Dec 23, 2008 at 01:52:21PM -0500, Matt McCutchen wrote:
 On Tue, 2008-12-23 at 23:45 +1100, David Overton wrote:
  I would also very much like to see this feature. Indeed, this seems
  far more logical than the current --link-dest behaviour and it's what
  I assumed --link-dest would do until I read the man page thoroughly
  (you have to follow the references back from --link-dest to
  --copy-dest and then --compare-dest and even then the only mention of
  the actual behaviour is a parenthesised comment (if the files are
  missing in the destination directory)). I be interested to know what
  use cases the current behaviour was designed for, because I can't
  see any advantage to not making use of the --link-dest file if it's
  available. Providing the proposed alternative behaviour as an extra
  option, if not the default for --link-dest, would be very useful.
 
 I too find the current semantics of basis dirs illogical and would
 support an option for better semantics (though I think we should avoid
 changing defaults some users may be relying on).  That's why I entered a
 series of enhancement requests, the first of which is Robert's proposal:
 
 https://bugzilla.samba.org/show_bug.cgi?id=5644
 https://bugzilla.samba.org/show_bug.cgi?id=5645
 https://bugzilla.samba.org/show_bug.cgi?id=5646
 
 I would consider it worth forking rsync to have these features, if I
 ever got around to doing so.

It (that is, the proposal in your bug first report) sounds like not
a very difficult or intrusive addition to me, and useful for rotating
backups.  Wayne said he wouldn't write it, but maybe he'd accept it
if someone else wrote it.  That would be better than forking unless
it is more intrusive than it seems.

(I'm taking an interest because I contributed the code for the
original --*-dest option --compare-dest and that is primarily what
pulled me in to maintaining rsync for a year or so.  I also integrated
JW Schultz's --link-dest.  --*dest options have evolved quite a bit
since then I see.)

- Dave
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: rsync feature needed: preserve atime

2007-05-09 Thread Dave Dykstra
On Tue, May 08, 2007 at 07:09:06PM -0400, Matt McCutchen wrote:
 On 5/8/07, Brent Thompson [EMAIL PROTECTED] wrote:
 Often we need to preserve the information atime conveys, but I have found
 no way to get rsync to preserve this, nor any hint it is being worked on.
 It would be great if 'rsync -t' also set atime not just mtime -- or maybe a
 new option is desired for atime.
 
 The standard version of rsync does not support preserving atimes, but
 the rsync source distribution includes a patch patches/atimes.diff
 that adds an option --atimes to preserve atimes.  I recommend you
 compile your own copy of rsync including this patch (reply if you need
 help/instructions for this) and then use it with its --atimes option.


What does it really mean to preserve access times?  When rsync reads
a file to copy it, it will change the access time just because it is
reading it, so the backup should then have the correct access time,
the time the backup file was created.  Some tools have an option to
reset the access time of the file they copy, but in so doing the tools
update the inode change time of the input file which is generally more
important to users than access time.  It is not possible to preserve
both as a filesystem user like rsync.

- Dave
-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: [PATCH] open O_TEXT and O_BINARY for cygwin/windows

2003-01-28 Thread Dave Dykstra
On Tue, Jan 28, 2003 at 01:26:32PM -, Max Bowsher wrote:
 Dave Dykstra wrote:
  I give up.  The msleep(100) consistently causes hangs of the
  unsafe-links test on my friend's cygwin Windows 2000 machine.  I
  suspect it's also the reason why the build.samba.org cygwin machine
  hasn't reported a result in the last 9 hours.  I'm taking it back out
  and washing my hands of the cygwin rsync port, I'm fed up.
 
 You've been most considerate in actively seeking out and merging Cygwin
 patches for this release. Alas, it backfired in merging the msleep thing. I
 definitely agree with it not being in the upstream sources. Making a race
 condition harder to hit, with a simple delay, is obviously not a fix. You
 may be pleased to know that the SHUTDOWN_ALL_SOCKETS change has eliminated
 Connection reset by peer messages when running the testsuite.

Well it's good to know that one of the cygwin changes did some good.

 It may be that the hang has just vanished. If it hasn't, it's probably a bug
 in Cygwin, not rsync. Hopefully we'll get a nice bugreport, and it can be
 traced and squashed.

Hopefully.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



dwd signing off rsync list

2003-01-28 Thread Dave Dykstra
I've had the good fortune of getting a short-term job, but it is going
to be very intense and I won't have time for the rsync mailing list.
I'm sure the rest of the rsync team will be able to cover most of the
questions and future updates, and I've told them they can contact me if
they need my help.

I've currently got two good prospects for a long term job, and with one
of them I may be able to have some involvement with rsync but with the
other I definitely won't.  We'll see what happens.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: Cygwin issues: modify-window and hangs

2003-01-27 Thread Dave Dykstra
On Fri, Jan 24, 2003 at 05:18:07PM -0600, Dave Dykstra wrote:
 I had a friend run some Cygwin tests and we found that --modify-window=1
 works just as well as --modify-window=2 on FAT filesystems to copy files
 from Unix and detect the difference in granularity.  FAT filesystems always
 have timestamps that have an even number of seconds.  On the other hand,
 NTFS filesystems can store the modification time down to the second,
 whereas previously people on this mailing list thought it was just like
 FAT for modification time.  Nevertheless, I decided to change the default
 value for --modify-window to 1 on Cygwin rather than 0, under the thinking
 that even on NTFS filesystems it wouldn't do any harm because it's highly
 unlikely that any individual file will be modified again within one second
 after it is copied by rsync.

This modify-window default of 1 has been causing some trouble on the rsync
test suite on the Cygwin test machine on build.samba.org.  The problem
is that some files get created and immediately copied within one second,
and then the rsync code that implements '-p' checks to see if the copied
file's time is within one second before deciding whether or not to change it.
The test machine is presumably using an NTFS filesystem so it has one
second granuarlity.   Last night I considered 4 possible solutions:
1. Change the test suite to always wait 2 seconds before copying, at
the beginning of the checkit function.
2. Change the test suite to always pass --modify-window=0 to rsync by
including it in the $RSYNC variable in testsuite/rsync.fns.
3. Change the set_perms() function in rsync.c to check for exact time
rather than calling cmp_modtime.
4. Back out the default of --modify-window=1 on cygwin and go back to
a default of 0.

I implemented solution 1, but I'm not very comfortable with it because
it slows the tests on all platforms. I'm now leaning toward solution 3.
Discussion?

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: Cygwin issues: modify-window and hangs

2003-01-27 Thread Dave Dykstra
On Mon, Jan 27, 2003 at 04:38:09PM -, Max Bowsher wrote:
 Dave Dykstra wrote:
  This modify-window default of 1 has been causing some trouble on the
  rsync test suite on the Cygwin test machine on build.samba.org.  The
  problem is that some files get created and immediately copied within
  one second, and then the rsync code that implements '-p' checks to
  see if the copied file's time is within one second before deciding
  whether or not to change it. The test machine is presumably using an
  NTFS filesystem so it has one second granuarlity.
 
 Yes, noticed this running make check last night. However, I also got
 random (i.e. re-run the testsuite and get different results) failures of
 some other tests. Is this showing up in your test machine?

I did notice strange failure of the hands test on the build.samba.org
Cygwin test machine.

  Last night I considered 4 possible solutions:
  1. Change the test suite to always wait 2 seconds before copying,
  at
  the beginning of the checkit function.
  2. Change the test suite to always pass --modify-window=0 to
  rsync by including it in the $RSYNC variable in testsuite/rsync.fns.
  3. Change the set_perms() function in rsync.c to check for exact
  time rather than calling cmp_modtime.
  4. Back out the default of --modify-window=1 on cygwin and go
  back to
  a default of 0.
 
  I implemented solution 1, but I'm not very comfortable with it because
  it slows the tests on all platforms. I'm now leaning toward solution
  3. Discussion?
 
 How about 4? It seems wrong to penalize NTFS for a FAT problem, especially
 as NTFS is now the preferred Windows filesystem.
 
 Unless someone feels like making a FAT-detection patch, the previous status
 quo looks to me like the best option.


You've talked me into it.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: [PATCH] open O_TEXT and O_BINARY for cygwin/windows

2003-01-27 Thread Dave Dykstra
On Mon, Jan 27, 2003 at 04:45:25PM -, Max Bowsher wrote:
 Dave Dykstra wrote:
  On Mon, Jan 27, 2003 at 10:46:16AM +0100, Lapo Luchini wrote:
  I can test that before releasing cygwin's 2.5.6-1 package,
  unfortunately I have almost *no* free time until after friday
  (university test on thursday ^_^)
  Anyway Max is also right, local issues can be resolved locally with
  much speed, even if in the long run it's of course better to put
  them in rsync cvs too.
 
  I'm reluctant to rely on cygwin-stream-specific patches because I know
  that other people compile it from source too.  Probably the msleep
  should be put in for Cygwin only with an ifdef though, or the waiting
  of the child removed altogether on Cygwin only.
 
 Does it not make sense that anyone wishing to compile their own rsync for
 Cygwin should start by looking at the Cygwin package? (Which, btw, includes
 the clean rsync source, and patchfiles, so it is easy to see what is being
 done to rsync specially for Cygwin).

In my case I used to use the same source code to compile on many different
platforms and no, I wouldn't look at the Cygwin package first to compile
on Cygwin.


 The waiting for the child is presumably there for a reason. If is removed
 for Cygwin, it should probably be removed for all platforms. If there is an
 argument against that, then it applies equally to Cygwin as to any other
 platforms.

Perhaps the waiting in the child was put in for specific problems on
some platforms.


  BTW: I wonder if the sleepms(30) is still needed, it's quite a bit of
  time that my build.samba.org hourly cron doesn't produce a hanged
  rsync... (must check that too)
 
  I know that some people still experience it, and I don't think
  anything has changed in that area since the time that the
  person who experienced the hang discovered the hack.
 
 Cygwin has gone through many minor versions since then. I suggest releasing
 an rsync without the hack, but with a command line option to turn it on.
 That way, we get to see for certain whether the problem still exists, and
 there is an easy workaround available if it does.

So many people have complained about Cygwin hangs that I'm inclined to
leave the msleep(100) in on this release by default, but only on Cygwin.
When the problem is shown to be fixed on Cygwin, it can be taken back
out.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: Cygwin issues: modify-window and hangs

2003-01-27 Thread Dave Dykstra
Thanks, you saved me some trouble.  I put it in.

- Dave

On Mon, Jan 27, 2003 at 06:52:30PM -0800, jw schultz wrote:
 On Mon, Jan 27, 2003 at 03:57:16PM -0600, Dave Dykstra wrote:
  On Mon, Jan 27, 2003 at 04:38:09PM -, Max Bowsher wrote:
   Dave Dykstra wrote:
This modify-window default of 1 has been causing some trouble on the
rsync test suite on the Cygwin test machine on build.samba.org.  The
problem is that some files get created and immediately copied within
one second, and then the rsync code that implements '-p' checks to
see if the copied file's time is within one second before deciding
whether or not to change it. The test machine is presumably using an
NTFS filesystem so it has one second granuarlity.
   
   Yes, noticed this running make check last night. However, I also got
   random (i.e. re-run the testsuite and get different results) failures of
   some other tests. Is this showing up in your test machine?
  
  I did notice strange failure of the hands test on the build.samba.org
  Cygwin test machine.
  
Last night I considered 4 possible solutions:
1. Change the test suite to always wait 2 seconds before copying,
at
the beginning of the checkit function.
2. Change the test suite to always pass --modify-window=0 to
rsync by including it in the $RSYNC variable in testsuite/rsync.fns.
3. Change the set_perms() function in rsync.c to check for exact
time rather than calling cmp_modtime.
4. Back out the default of --modify-window=1 on cygwin and go
back to
a default of 0.
   
I implemented solution 1, but I'm not very comfortable with it because
it slows the tests on all platforms. I'm now leaning toward solution
3. Discussion?
   
   How about 4? It seems wrong to penalize NTFS for a FAT problem, especially
   as NTFS is now the preferred Windows filesystem.
   
   Unless someone feels like making a FAT-detection patch, the previous status
   quo looks to me like the best option.
  
  
  You've talked me into it.
 
 If you are going to revert cygwin to a --modify-window=0
 default we should have a notice in the bugs section of the
 manpage.
 
 I've attached a patch that adds this, improves (in my
 opinion) the --modify-window section slightly, removes
 reference to a bug tracking system and updates the version
 the manpage reports current with.
 
 -- 
 
   J.W. SchultzPegasystems Technologies
   email address:  [EMAIL PROTECTED]
 
   Remember Cernan and Schmitt

 Index: rsync.yo
 ===
 RCS file: /cvsroot/rsync/rsync.yo,v
 retrieving revision 1.113
 diff -u -b -r1.113 rsync.yo
 --- rsync.yo  27 Jan 2003 03:13:46 -  1.113
 +++ rsync.yo  28 Jan 2003 01:56:31 -
 @@ -396,8 +396,8 @@
  the timestamps as being equal if they are within the value of
  modify_window. This is normally zero, but you may find it useful to
  set this to a larger value in some situations. In particular, when
 -transferring to/from FAT filesystems which cannot represent times with
 -a 1 second resolution this option is useful.
 +transferring to FAT filesystems which cannot represent times with
 +a 1 second resolution --modify-window=1 is useful.
  
  dit(bf(-c, --checksum)) This forces the sender to checksum all files using
  a 128-bit MD4 checksum before transfer. The checksum is then
 @@ -1110,16 +1110,20 @@
  
  times are transferred as unix time_t values
  
 +When transferring to FAT filesystmes rsync may resync
 +unmodified files.
 +See the comments on the --modify-window option.
 +
  file permissions, devices etc are transferred as native numerical
  values
  
  see also the comments on the --delete option
  
 -Please report bugs! The rsync bug tracking system is online at
 -url(http://rsync.samba.org/rsync/)(http://rsync.samba.org/rsync/)
 +Please report bugs! See the website at
 +url(http://rsync.samba.org/)(http://rsync.samba.org/)
  
  manpagesection(VERSION)
 -This man page is current for version 2.0 of rsync
 +This man page is current for version 2.5.6 of rsync
  
  manpagesection(CREDITS)
  

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: [PATCH] open O_TEXT and O_BINARY for cygwin/windows

2003-01-27 Thread Dave Dykstra
On Mon, Jan 27, 2003 at 04:01:46PM -0600, Dave Dykstra wrote:
 On Mon, Jan 27, 2003 at 04:45:25PM -, Max Bowsher wrote:
...
  Cygwin has gone through many minor versions since then. I suggest releasing
  an rsync without the hack, but with a command line option to turn it on.
  That way, we get to see for certain whether the problem still exists, and
  there is an easy workaround available if it does.
 
 So many people have complained about Cygwin hangs that I'm inclined to
 leave the msleep(100) in on this release by default, but only on Cygwin.
 When the problem is shown to be fixed on Cygwin, it can be taken back
 out.

I give up.  The msleep(100) consistently causes hangs of the unsafe-links
test on my friend's cygwin Windows 2000 machine.  I suspect it's also the
reason why the build.samba.org cygwin machine hasn't reported a result
in the last 9 hours.  I'm taking it back out and washing my hands of the
cygwin rsync port, I'm fed up.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: [PATCH] open O_TEXT and O_BINARY for cygwin/windows

2003-01-26 Thread Dave Dykstra
On Sun, Jan 26, 2003 at 08:54:19PM -, Max Bowsher wrote:
 Dave Dykstra wrote:
  Alright.  Max, could you quickly verify if the latest CVS version
  works OK for you on Cygwin?
 
 What, in particular? I'm not a very good testcase, because I use binary
 mounts and unix line endings everywhere.
 
 It compiles and does syncs with remote rsync daemons, which is my normal
 usage.
 
 Max.


See if exclude files with DOS line endings work ok for you, and also the
other config files if you can.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: Storage compression patch for Rsync (unfinished)

2003-01-26 Thread Dave Dykstra
Is there any reason why caching programs would need to set the
value, rather than it just being a fixed value?  I think it is hard
to describe what this is for and what it should be set to.  Maybe a
--fixed-checksum-seed option would make some sense, or for a caching
mechanism to be built in to rsync if it is shown to be very valuable.
I don't think I'll include the option in 2.5.6.  I know people have
proposed some caching mechanisms in the past and they've been rejected
for one reason or another.

- Dave

On Fri, Jan 17, 2003 at 11:19:35PM -0800, Craig Barratt wrote:
  While the idea of rsyncing with compression is mildly
  attractive i can't say i care for the new compression
  format.  It would be better just to use the standard gzip or
  other format.  If you are going to create a new file type
  you could at least discuss storing the blocksums in it so
  that the receiver wouldn't have to generate them.
 
 Yes!  Caching the block checksums and file checksums could yield a large
 improvement for the receiver.  However, an integer checksum seed is used
 in each block and file MD4 checksum. The default value is unix time() on
 the server, sent to the client at startup.
 
 So currently you can't cache block and file checksums (technically it is
 possible for block checksums since the checksum seed is appended at the
 end of each block, so you could cache the MD4 state prior to the checksum
 seed being added; for files you can't since the checksum seed is at the
 start).
 
 Enter a new option, --checksum-seed=NUM, that allows the checksum seed to
 be fixed.  I've attached a patch below against 2.5.6pre1.
 
 The motivation for this is that BackupPC (http://backuppc.sourceforge.net)
 will shortly release rsync support, and I plan to support caching
 block and file checksums (in addition to the existing compression,
 hardlinking among any identical files etc).  So it would be really
 great if this patch, or something similar, could make it into 2.5.6
 or at a minimum the contributed patch area in 2.5.6.
 
 [Also, this option is convenient for debugging because it makes the
 rsync traffic identical between runs, assuming the file states at
 each end are the same too.]
 
 Thanks,
 Craig
 
 ###
 diff -bur rsync-2.5.6pre1/checksum.c rsync-2.5.6pre1-csum/checksum.c
 --- rsync-2.5.6pre1/checksum.cMon Apr  8 01:31:57 2002
 +++ rsync-2.5.6pre1-csum/checksum.c   Thu Jan 16 23:38:47 2003
 @@ -23,7 +23,7 @@
  
  #define CSUM_CHUNK 64
  
 -int checksum_seed = 0;
 +extern int checksum_seed;
  extern int remote_version;
  
  /*
 diff -bur rsync-2.5.6pre1/compat.c rsync-2.5.6pre1-csum/compat.c
 --- rsync-2.5.6pre1/compat.c  Sun Apr  7 20:50:13 2002
 +++ rsync-2.5.6pre1-csum/compat.c Fri Jan 17 21:18:35 2003
 @@ -35,7 +35,7 @@
  extern int preserve_times;
  extern int always_checksum;
  extern int checksum_seed;
 -
 +extern int checksum_seed_set;
  
  extern int remote_version;
  extern int verbose;
 @@ -64,11 +64,14 @@
   
   if (remote_version = 12) {
   if (am_server) {
 - if (read_batch || write_batch) /* dw */
 + if (read_batch || write_batch) { /* dw */
 + if ( !checksum_seed_set )
   checksum_seed = 32761;
 - else
 + } else {
 + if ( !checksum_seed_set )
   checksum_seed = time(NULL);
   write_int(f_out,checksum_seed);
 + }
   } else {
   checksum_seed = read_int(f_in);
   }
 diff -bur rsync-2.5.6pre1/options.c rsync-2.5.6pre1-csum/options.c
 --- rsync-2.5.6pre1/options.c Fri Jan 10 17:30:11 2003
 +++ rsync-2.5.6pre1-csum/options.cThu Jan 16 23:39:17 2003
 @@ -116,6 +116,8 @@
  char *backup_dir = NULL;
  int rsync_port = RSYNC_PORT;
  int link_dest = 0;
 +int checksum_seed = 0;
 +int checksum_seed_set;
  
  int verbose = 0;
  int quiet = 0;
 @@ -274,6 +276,7 @@
rprintf(F, --bwlimit=KBPS  limit I/O bandwidth, KBytes per 
second\n);
rprintf(F, --write-batch=PREFIXwrite batch fileset starting with 
PREFIX\n);
rprintf(F, --read-batch=PREFIX read batch fileset starting with 
PREFIX\n);
 +  rprintf(F, --checksum-seed=NUM set MD4 checksum seed\n);
rprintf(F, -h, --help  show this help screen\n);
  #ifdef INET6
rprintf(F, -4  prefer IPv4\n);
 @@ -293,7 +296,7 @@
OPT_COPY_UNSAFE_LINKS, OPT_SAFE_LINKS, OPT_COMPARE_DEST, OPT_LINK_DEST,
OPT_LOG_FORMAT, OPT_PASSWORD_FILE, OPT_SIZE_ONLY, OPT_ADDRESS,
OPT_DELETE_AFTER, OPT_EXISTING, OPT_MAX_DELETE, OPT_BACKUP_DIR, 
 -  OPT_IGNORE_ERRORS, OPT_BWLIMIT, OPT_BLOCKING_IO,
 +  OPT_IGNORE_ERRORS, OPT_BWLIMIT, OPT_BLOCKING_IO, OPT_CHECKSUM_SEED,
OPT_NO_BLOCKING_IO, OPT_WHOLE_FILE, OPT_NO_WHOLE_FILE,
OPT_MODIFY_WINDOW, 

Re: Storage compression patch for Rsync (unfinished)

2003-01-26 Thread Dave Dykstra
On Sun, Jan 26, 2003 at 02:46:43PM -0800, Craig Barratt wrote:
  Is there any reason why caching programs would need to set the
  value, rather than it just being a fixed value?
  I think it is hard to describe what this is for and what it should be
  set to.  Maybe a --fixed-checksum-seed option would make some sense,
  or for a caching mechanism to be built in to rsync if it is shown to
  be very valuable.
 
 A fixed value would be perfectly ok; the same magic value that batch
 mode uses (32761) would make sense.
 
  I know people have proposed some caching mechanisms in the past and
  they've been rejected for one reason or another.
 
 One difficulty is that additional files, or new file formats, are needed
 for storing the checksums, and that moves rsync further away from its
 core purpose.
 
  I don't think I'll include the option in 2.5.6.
 
 If I submitted a new patch with --fixed-checksum-seed, would you be
 willing to at least add it to the patches directory for 2.5.6?
 
 I will be adding block and file checksum caching to BackupPC, and
 that needs --fixed-checksum-seed.  This will save me from providing
 a customized rsync (or rsync patches) as part of BackupPC; I would
 much rather tell people to get a vanilla 2.5.6 rsync release and
 apply the specific patch that comes with the release.


Sorry, but I don't think it would be good to do even that until we've all
had a chance to look at what's involved in the caching and whether or not
it would make better sense to have it be a modification to rsync rather
than mostly external.  I'm concerned that people might misuse the option
without understanding the consequences.  You could always keep the patch
on the BackupPC web site in the meantime.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: Cygwin issues: modify-window and hangs

2003-01-26 Thread Dave Dykstra
On Sat, Jan 25, 2003 at 06:32:08PM +0100, Greger Cronquist wrote:
  --- Max Bowsher [EMAIL PROTECTED] skrev:  Dave Dykstra
 wrote:
 
  I'm using the current Cygwin release
  (rsync-2.5.5-2). That is rsync-2.5.5,
  with an added msleep(30) which is intended to deal
  with a possible problem
  with signals in Cygwin.
 
 Making that msleep(100) works even better for me.

Ah, I found the messages that were discussing this and I see this was
supposed to prevent hangs at the ends of runs.  I went ahead and put an
msleep(100) in before the final kill...USR2 in main.c do_recv().

I also finally remembered that the calls to shutdown() were for Connection
reset by peer errors and not hangs.  I went ahead and put in a define
SHUTDOWN_ALL_SOCKETS which is only set by configure on Cygwin and call
shutdown when it is set.  Hopefully that will get rid of messages.

- Dave Dykstra
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



one more day of delay for release of 2.5.6

2003-01-26 Thread Dave Dykstra
Well I was hoping to get the 2.5.6 out today, but I think I made too many
Cygwin changes this evening for comfort and I'd like to allow one more
day of testing.  Cygwin users, please test as much as you can and post
any problems.

- Dave Dykstra
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: [patch] Still a problem with cleanup.c

2003-01-25 Thread Dave Dykstra
I couldn't reproduce the problem on Linux, but I believe you that it's
a problem.  If you think about it, it's easy to see why code that called
_exit_cleanup() might behave strangely if the function returns so I
like the fix of not returning until the situation gets deeply nested.
I submitted the patch.

Thanks Marc  Brian.

- Dave Dykstra

On Thu, Jan 23, 2003 at 08:29:40PM -0500, Brian Poole wrote:
 Hello,
 
 There is still a problem with the _exit_cleanup() function in 
 cleanup.c despite the patch that was put in last week that prevented
 recursion. It turns out that sometimes multiple calls in close 
 sequence are done and this causes rsync to spin out of control 
 instead of exiting.
 
 This bug was found by Marc Espie and the patch I'm including is 
 his as well. It hasn't been commited to the OpenBSD tree yet and
 is still undergoing testing but it does fix the immediate easily 
 detectable problem. His post with the bug fix was here:
 
 http://marc.theaimsgroup.com/?l=openbsd-techm=104238879828005w=2
 
 I've changed the variable names to match what is in the rsync 
 tree already. 
 
 This bug is easily triggerable with a local rsync copy. 
 
 rsync -av /some/reasonably/large/dir /tmp/somewhere
 
 Wait until after the copy has started (after it is done with
 building the file list) and then hit ctrl-c. Watch rsync fill
 the screen with messages like:
 
 rsync: connection unexpectedly closed (94819 bytes read so far)
 rsync: connection unexpectedly closed (94819 bytes read so far)
 rsync: connection unexpectedly closed (94819 bytes read so far)
 
 Curse.
 Switch to another window and kill -9 the errant rsync process.
 Apply patch and notice it no longer goes crazy.
 
 
 -b
 
 Index: cleanup.c
 ===
 RCS file: /cvsroot/rsync/cleanup.c,v
 retrieving revision 1.15
 diff -u -r1.15 cleanup.c
 --- cleanup.c 16 Jan 2003 20:09:31 -  1.15
 +++ cleanup.c 24 Jan 2003 01:24:09 -
 @@ -63,11 +63,11 @@
   extern int log_got_error;
   static int inside_cleanup = 0;
  
 - if (inside_cleanup != 0) {
 + if (inside_cleanup  10) {
   /* prevent the occasional infinite recursion */
   return;
   }
 - inside_cleanup = 1;
 + inside_cleanup++;
  
   signal(SIGUSR1, SIG_IGN);
   signal(SIGUSR2, SIG_IGN);

 Index: cleanup.c
 ===
 RCS file: /cvsroot/rsync/cleanup.c,v
 retrieving revision 1.15
 diff -u -r1.15 cleanup.c
 --- cleanup.c 16 Jan 2003 20:09:31 -  1.15
 +++ cleanup.c 24 Jan 2003 01:24:09 -
 @@ -63,11 +63,11 @@
   extern int log_got_error;
   static int inside_cleanup = 0;
  
 - if (inside_cleanup != 0) {
 + if (inside_cleanup  10) {
   /* prevent the occasional infinite recursion */
   return;
   }
 - inside_cleanup = 1;
 + inside_cleanup++;
  
   signal(SIGUSR1, SIG_IGN);
   signal(SIGUSR2, SIG_IGN);

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: [PATCH] open O_TEXT and O_BINARY for cygwin/windows

2003-01-24 Thread Dave Dykstra
On Fri, Jan 24, 2003 at 08:57:02AM +0200, Ville Herva wrote:
 On Thu, Jan 23, 2003 at 01:55:40PM -0600, you [Dave Dykstra] wrote:
  Why did you skip the fopen in log.c, which appends to the log file?
 
 I thought about that for a while. My reasoning was that the log file is
 probably read with unix/cygwin tools anyway - if not, the administrator is
 free to mount the filesystem binary or ascii. So I didn't want to enforce
 CR/LF on anyone.
 
 In general, I think forcing ascii makes more sense for reading than writing,
 because it in most imaginable cases will do no harm.
  
  Because of the question about whether or not the t is ignored on all
  platforms, I'm a little nervous about putting this into 2.5.6.  
 
 It could be ifdefed, would it be safe to assume the following to work?
 
 #if !defined(O_TEXT)
 #define O_TEXT 0
 #define O_TEXT_STR 
 #else 
 #define O_TEXT_STR t
 #endif
 #if !defined(O_BINARY)
 #define O_BINARY 0
 #define O_BINARY_STR 
 #else
 #define O_BINARY_STR b
 #endif
 
 and then instead of
 
   fopen(foo, rt);
 
 do this
 
   fopen(foo, r O_TEXT_STR);
 
 Comments?
 
  I don't have any problem with it for 2.6.0.  Maybe it should be just in
  the 'patches' directory this time.
  
  Did you check to see whether any of the code that's reading config files
  is ignoring carriage returns at the end of lines anyway?
 
 ISTR at least the password files were not recognized and I *think*
 rsync.conf was not parsed correctly when I originally wrote the patch.
 
 Looking at the source, though
 
 - exclude.c uses fgets() to read the lines from exclude file. I don't think 
   it handles '\r'
 - param.c Ignores all '\r' in values, so rsync.conf should be fine. I'm not
   sure, whether reading the file as text would be more elegant (in theory
   there _could_ be other issues than the \r\n thing...)
 - authentivate.c seems to skip '\r' for passwords as well.
 
 So I guess you are right for the most crucial part of code (as of current
 rsync).


I think I'll go ahead and put in your patch with the modification of using
O_TEXT_STR as you suggest.  I think the risk is low.  I had been concerned
that perhaps older CPPs might not be able to handle that syntax, but I see
other places in the rsync code that are depending on it so I think it's OK.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Cygwin issues: modify-window and hangs

2003-01-24 Thread Dave Dykstra
I had a friend run some Cygwin tests and we found that --modify-window=1
works just as well as --modify-window=2 on FAT filesystems to copy files
from Unix and detect the difference in granularity.  FAT filesystems always
have timestamps that have an even number of seconds.  On the other hand,
NTFS filesystems can store the modification time down to the second,
whereas previously people on this mailing list thought it was just like
FAT for modification time.  Nevertheless, I decided to change the default
value for --modify-window to 1 on Cygwin rather than 0, under the thinking
that even on NTFS filesystems it wouldn't do any harm because it's highly
unlikely that any individual file will be modified again within one second
after it is copied by rsync.

While doing the tests we too experienced hangs at the end of copies.
We were going over openssh from a Solaris 9 box to Windows 2000 Cygwin.
We tried the test from 
http://lists.samba.org/pipermail/rsync/2002-August/008130.html
but it still experienced hangs.  It wasn't clear if the patch reduced
the frequency or not.

Has *anybody* been able to figure out a fix for this that really works?
It sure would be nice to get something into 2.5.6 but we're about out of
time for that because I need to put it out this weekend before I start a
new (temporary) job on Monday.  I imagine that the issue I was experiencing
could be a separate one and this patch really would help other hangs
people are having; can anybody give me an argument for putting in the
calls to shutdown() anyway?  I would make it only happen on Cygwin,
because it is unknown if they will cause harm on other platforms.

- Dave Dykstra
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: opendir(somedir/somefile): Not enough space -- why?

2003-01-24 Thread Dave Dykstra
On Fri, Jan 24, 2003 at 04:02:28PM -0800, jw schultz wrote:
 On Fri, Jan 24, 2003 at 04:49:20PM -0700, Bill Geddes wrote:
  Am I running out of memory on the source server?  Seems perhaps more
  plausible than on the client - it has nothing going on and 1Gb RAM.
  The server has rsync version 2.5.5.  The client has 2.5.6.
 
 Yes, it looks like the sender is running out of memory.
 Break the job up.

The important factor is the number of files.  Rsync uses up some memory
for every file it looks at in a run.  That's why JW said to break it
up into smaller jobs.


  I am currently mounting the server directory via NFS, and using rsync to
  copy from the mounted directory to the target.  Slow, but it is running!
 
 If you continue over NFS be sure you do checksums of
 everything afterward.  NFS has (or at least used to) a
 measurable error rate.

Rsync is optimized for a slow network and high speed disk access.
If you're doing NFS over a slow network, that looks to rsync as a slow
disk.  If you can, it's probably better to run rsync directly on
the server.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: Please test rsync-2.5.6pre2

2003-01-23 Thread Dave Dykstra
On Tue, Jan 21, 2003 at 08:00:51PM -0801, Jos Backus wrote:
 On Mon, Jan 20, 2003 at 10:19:42PM -0600, Dave Dykstra wrote:
  The second rsync-2.5.6 pre-release version is now available at:
 
 There's the popt-1.7 update (with Dave's sprintf() workaround) at
 
 http://www.catnook.com/patches/rsync-popt-1.7.diff
 
 Perhaps we can apply this after the release as it is a very low priority
 update; I'd just like it not to rot :)

Yes, it should go in after the 2.5.6 release.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: Rsyncing of block devices to a remote file

2003-01-23 Thread Dave Dykstra
I'm not sure why it would really need to know the size, but given that
it wants it, can you open the file and lseek to the end of it?

- Dave Dykstra

On Thu, Jan 23, 2003 at 09:17:04AM +0100, David Heremans wrote:
 Hello,
 
 I'm looking for some assistance in modifying the rsync code
 
 Situation: I used to back-up some of my (unmounted NTFS) disk partitions
 remotely using the following shells construct:
 dd if=/dev/hda1 | gzip | ssh me@backupmachine 'cat 
 /backup-dir/hda1-backup.gz'
 
 But now I want to use rsync for this, since transfering 10 gig takes a litle
 to long if a file of a few kB has changed :-). I looked trough the archive
 and found an old mail talking about the same problem, and a response
 explaining the problem of writting to a block device (since rsync writes a
 copy file and then moves it over the original). But the same mail also said
 that reading from it should be trivial, so I set out trying to implement
 this.
 
 So I created a '--read-block-file' option in options.c and avoided the
 'skipping non regular file' message in generator.c
 By altering all this the receiver is ready to get the data, but in the flist
 structure it still says that '/dev/hda1' is still a devicefile 
 and that the size is 0, and so the sender does send over zero bytes
 
 Now here is the problem : How do I get the size of partition in this
 datastructure ? Or more generique, how would I handle this for /dev/fd0,
 since you can insert 1.44MB disks or 720kB disks ?
 
 Anybody else tried to read blockdevices over rsync already ?
 
 After all, sending the data of a block device doesn't seem to be as trivial
 as the archived mail sugested :-)
 
 David 
 
 
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: Error message

2003-01-23 Thread Dave Dykstra
With file sizes over 2GB I'd suspect problems with 31-bit numbers being
returned from stat() et. al.  The rsync version you have on AIX is very
old, and more recent versions handle large files better.  The first thing
to do is to upgrade rsync your AIX machine.

- Dave

On Thu, Jan 23, 2003 at 01:44:42PM +0100, Michael Salmon wrote:
 On Thursday, January 23, 2003 12:36:52 PM +0100 Boris Gegenheimer 
 [EMAIL PROTECTED] wrote:
 +--
 | The versions of the different machines I use are the initiating machine :
 | Sun 450
 | Solaris 8
 | Rsync 2.5.5
 |
 | The machine that I try to copy from :
 | IBM rs6000
 | AIX 4.3
 | Rsync 2.3.1
 |
 | The command I use is
 | /usr/local/bin/rsync -au -q --stats --progress
 | --rsync-path=/usr/local/bin/rsync maximus:/ptsfs2
 | /diskbackup/kanal1/maximus/
 |
 | The error message comes when he tries to copy that file mentioned below
 | which is 14GB big it is not the filesystem or the name of the file because
 | it worked with ncftpget.
 | But that program has other problems that I will not mention here.
 [...]
 | On Thu, Jan 23, 2003 at 10:55:32AM +0100, Boris Gegenheimer wrote:
 |  Hello when i get this message what is wrong:
 | 
 |  /PTS.0.db2inst1.NODE.CATN.20030123062324.001: Value too large
 |  to be stored in data type
 | 
 |  Regards Boris
 [...]
 +-X8
 
 I have also seen that message with times before 1970. This is the standard 
 text for EOVERFLOW.
 
 /Michael
 --
 This space intentionally left non-blank.
 -- 
 To unsubscribe or change options: 
 http://lists.samba.org/mailman/listinfo/rsync
 Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: Rsyncd - Help me

2003-01-22 Thread Dave Dykstra
Look for errors in /var/log/rsyncd.log.  For security reasons they are
not all passed to the client.

- Dave

On Wed, Jan 22, 2003 at 06:52:33PM -0200, Breno Cardoso Perucchi wrote:
 Hello,
 
 I'm trying to do my rsync to copy a mirror of my site in another webserver.
 I am trying to use RSYNCD .
 How I'll do for my rsync to copy my site website in the mirror ?  
 Which is a command to copy the mirror with the password ?
 I've this websites about rsync 
 http://sunsite.dk/info/guides/rsync/rsync-mirroring.html
 http://rsync.samba.org
 
 Somebody could help me? or Link ? Doc ? Howto ?
 
 
 This is my rsyncd.conf
 /usr/local/etc/rsyncd.conf
 
 log file = /var/log/rsyncd.log
 pid file = /var/run/rsyncd.pid
 lock file = /var/run/rsync.lock
 
 [rsyncd]
path = /usr/local/www/data-dist/
comment = Rsync Server
use chroot = no
max connections = 4
uid = nobody
gid = nobody
read only = yes
list = yes
auth users = rsyncd
secrets file = /usr/local/etc/rsyncd.scrt
 
 /usr/local/etc/rsyncd.scrt
 rsyncd:rsyncd
 
 bash-2.05b# rsync -a 10.0.0.1::
 rsyncd  Rsync Server
 bash-2.05b# rsync -a 10.0.0.1::rsyncd
 Password: 
 @ERROR: auth failed on module rsyncd
 rsync: connection unexpectedly closed (90 bytes read so far)
 rsync error: error in rsync protocol data stream (code 12) at io.c(150)
 
 Breno Cardoso Perucchi
 [EMAIL PROTECTED]
 
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: Please test rsync-2.5.6pre2

2003-01-21 Thread Dave Dykstra
On Tue, Jan 21, 2003 at 11:37:12AM -, Max Bowsher wrote:
 Dave Dykstra wrote:
  Changes since 2.5.5pre1:
 
ENHANCEMENTS:
 
  * Set the default value of --modify-window to 2 on Cygwin.  (Max
  Bowsher)
 
 No, not me. I only got involved to suggest an alternative to the deprecated
 AC_CYGWIN autoconf macro.
 I am, in fact, undecided on whether the --modify-window change is a good
 idea or not.

I knew that was all that you had contributed, but since you'd rather not
have your name associated with it I changed the name to Lapo Luchini who
had the original idea.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: exclude hell !!!!!

2003-01-21 Thread Dave Dykstra
On Wed, Jan 22, 2003 at 03:42:06AM +0100, wim delvaux wrote:
...
   ALSO when you specify an exclude list on the server in the rsyncd.conf
   file and you call rsync with the --delete option WHY DO THE FILES that
   SHOULD BE EXCLUDED get deleted ??
  
   Jeeez, what a bummer
   W
 
  I assume you're comying from the server to the client?  From exclude
  in rsyncd.conf.5:
 
  This is equivalent to the client specifying these patterns with the
  --exclude option except that the exclude list is not passed to the
  client and thus only apply on the server.
 
  which means that if you do a --delete they are not excluded on the client.
  It's as if the files don't exist at all on the server.
 
 Ah! so it means that if the file exist on the client and it is excluded on
 the server,the client assumes the file should be removed (--delete).
 
 How can I EXCLUDE files from the sync on the server (I want the server
 to define the rules so each client has as little config as possible)

Currently that's not supported.  You could make it two separate commands,
with the first command only copying a file from the server which
contains the exclude patterns and the second command using the file as
an --exclude-from.

Wayne Davison's new --files-from patch proposal has a syntax where the
file name can be remote:module/...; I don't know if there's enough
motivation to do something similar for --exclude-from.  Probably not.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: [patch] Recent IPv6 changes in rsync/access.c

2003-01-20 Thread Dave Dykstra
Thanks, I will apply your patch, especially since (for the record)
Hideaki posted in a reply:

  The suggested fix Daniel Hartmeier (an OpenBSD developer) had when
  I asked (for I'm not really a coder and I wanted to double check
  that this was a valid problem) was making the mask[w] assignment
  conditional like thus:
 
  if (w  addrlen)
  mask[w] = 0xff  (0xff(8-b));

 Absolutely... Please apply the suggested fix.
 Thanks for finding / fixing this serious bug.


By the way, in the future I would appreciate it if you would post bugs to
the rsync mailing list instead of to me because other rsync team members
should see these things too and might be able to reply faster than me.

- Dave Dykstra


On Sat, Jan 18, 2003 at 06:21:28PM -0500, Brian Poole wrote (in private email):
 Hello,
 
 Here is a proposed patch to fix a couple problems with recent changes
 incorporated into access.c. 
 
 The first is the biggest problem, there is a possible overflow
 in make_mask() if w is the same size as addrlen. Feel free to 
 skip the explanation, its just there in case you don't see what 
 I see.. patch is attached  at the bottom.
 
 I will walkthrough where I see the bug to have this make sense. I 
 start in match_address() where I note char mask[16] at the top. 
 Continuing on, we get to a switch statement for the address family;
 we imagine we take the PF_INET6 case. Two things of note here, 1st
 is addrlen = 16; 2nd is what appears to be a pasto, two copies of 
 
 a = (char *)sin6a-sin6_addr;
 t = (char *)sin6t-sin6_addr;
 
 are found, surrounding an if statement that shouldn't affect the
 values. Thats the 2nd, minor, problem I wanted to note. I also 
 moved the addrlen in my diff for cosmetic purposes. Doesn't 
 matter which set of assignments is taken out.
 
 Continuing on for the bigger problem, we exit the switch statement
 and come to an if (p). Imagine p is not set and we take the else. 
 This sets bits = 128.
 
 Next up is a make_mask() call with args char mask[16], 128, and 16.
 Going up to make_mask we set w = 128  3, or w = 16. A few lines 
 later you then have mask[w] =  mask[16] is not in mask[] and 
 so you have your overflow.
 
 The suggested fix Daniel Hartmeier (an OpenBSD developer) had when
 I asked (for I'm not really a coder and I wanted to double check 
 that this was a valid problem) was making the mask[w] assignment
 conditional like thus:
 
 if (w  addrlen)
 mask[w] = 0xff  (0xff(8-b));
 
 That sounds like the correct fix to me but I shall leave it to you
 guys to finish this off. I've inlined the patch as well as attaching
 it in case you have a preference.
 
 This bug was noticed due to the propolice patches that are in 
 OpenBSD-current which caused the rsync tests (which I do regularly
 for the build farm) to log messages like this:
 
 Jan 18 14:33:41 pandemonium rsync: stack overflow in function match_address
 
 I should have noticed this about 10 days ago but as this is a test
 machine I don't routinely watch the logs ;-) This gets the 
 OpenBSD/sparc64 build moving along a bit further than it had been,
 though there is still one test failing (daemon-gzip-download now 
 passes, daemon-gzip-upload still fails.)
 
 Will look into the other failure if I have time.
 
 
 -b
 
 
 Index: access.c
 ===
 RCS file: /cvsroot/rsync/access.c,v
 retrieving revision 1.7
 diff -u -r1.7 access.c
 --- access.c  9 Jan 2003 21:30:24 -   1.7
 +++ access.c  18 Jan 2003 23:12:57 -
 @@ -51,7 +51,8 @@
  
   if (w)
   memset(mask, 0xff, w);
 - mask[w] = 0xff  (0xff(8-b));
 + if (w  addrlen)
 + mask[w] = 0xff  (0xff(8-b));
   if (w+1  addrlen)
   memset(mask+w+1, 0, addrlen-w-1);
  
 @@ -121,6 +122,8 @@
   a = (char *)sin6a-sin6_addr;
   t = (char *)sin6t-sin6_addr;
  
 + addrlen = 16;
 +
  #ifdef HAVE_SOCKADDR_IN6_SCOPE_ID
   if (sin6t-sin6_scope_id 
   sin6a-sin6_scope_id != sin6t-sin6_scope_id) {
 @@ -128,10 +131,6 @@
   goto out;
   }
  #endif
 -
 - a = (char *)sin6a-sin6_addr;
 - t = (char *)sin6t-sin6_addr;
 - addrlen = 16;
  
   break;
   }

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: IPv6 hosts allow|deny

2003-01-20 Thread Dave Dykstra
On Fri, Jan 17, 2003 at 12:31:26AM +0100, Bert Vermeulen wrote:
 On Mon, 13 Jan 2003, Dave Dykstra wrote:
 
  I don't understand enough how that syntax works in order to be able
  to write an explanation for people.  Could you please try to write
  something up based on the current version checked in to CVS?
 
 Here's a patch for the manpage:
 
   http://biot.com/patches/rsync-ipv6-acl-doc.patch

I have never been introduced to IPv6 addresses before and there are some
things about your documentation that baffle me.  Perhaps I just need to
be educated a little, or perhaps it would be useful to include a little
more explnation in the documentation.

Your patch includes this:
 -  it() a dotted decimal IP address of the form a.b.c.d for IPv4 and 
 -  a.b.c.d.e.f for IPv6. In this case the incoming machine's IP address
 +  it() a dotted decimal IPv4 address of the form a.b.c.d, or an IPv6 address
 +  of the form a:b:c::d:e:f. In this case the incoming machine's IP address
must match exactly.

Why is there a blank field between c and d?Do you think you
should you say something about the fields being hexadecimal for IPv6,
or is that obvious to anyone who has seen IPv6 addresses before?


Your patch also includes this:
 +  it() an address/mask in the form ipaddr/n where ipaddr is the IP address
 +  and n is the number of one bits in the netmask.  All IP addresses which
 +  match the masked IP address will be allowed in.
 +
 +  it() an address/mask in the form ipaddr/maskaddr where ipaddr is the
 +  IP address and maskaddr is the netmask in dotted decimal notation for IPv4,
 +  or similar for IPv6, e.g. ::::: instead of /64. All IP
 +  addresses which match the masked IP address will be allowed in.


How come a mask includes 4 fields followed by 2 blanks?   What do you
mean by /64?

Your patch also has this addition:
 +Note IPv6 link-local addresses can have a scope in the address specification:
 +
 +quote(fe80::1%eth0)
 +quote(fe80::%eth0/64)
 +quote(fe80::%eth0/:::::)
 
What's a link-local address?  What's a scope?  I think I get that the parts
after the '/' are masks, but what does it mean to have just two colons
in the address, with one field blank and the third one either 1%eth0 or
%eth0?

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: Windows and default --modify-window=0

2003-01-20 Thread Dave Dykstra
I submitted a patch similar to what Max suggested.  Nobody came up with
a definitive answer on whether the modify-window default should be 1 or
2 so I left it at 2.

- Dave

On Fri, Jan 10, 2003 at 06:27:31PM -, Max Bowsher wrote:
 Dave Dykstra wrote:
  Ok, I agree --modify-window should default to 2 (or 1 if that's all
  that's really needed) on cygwin.  However, I don't like os-specific
  defines.
 
 They have the advantage that the entire logic is visible in the .c file,
 rather than split between the .c and configure.in.
 
  That _WIN32 we've got there is the only one currently in the code, and
  there's no occurrances of __CYGWIN__.   I'd rather have a configure.in
  rule for it.  Any suggestions on a good one?  At a minimum there's an
  AC_CYGWIN macro but that's not much different than just checking for
  __CYGWIN__.  Probably it would be worth calling AC_CYGWIN and then
  setting
  a DEFAULT_MODIFY_WINDOW to 2 in configure if $CYGWIN=yes.
 
 AC_CYGWIN is obsolete - the autoconf manual recommends something like this:
 AC_CANONICAL_HOST  # at some point above
 case $host_os in
   *cygwin* ) DEFAULT_MODIFY_WINDOW=2;;
  * ) DEFAULT_MODIFY_WINDOW=0;;
 esac
 AC_SUBST(DEFAULT_MODIFY_WINDOW)
  As far as I
  know there is no other port of rsync to Windows yet except under
  Cygwin
  so I don't think it is worth putting in extra code to support
  something
  that may never be used.  Or, perhaps we should key off of the $EXEEXT
  variable which gets set to '.exe' on Windows; anybody have an opinion
  on
  that?
 
 Yuk! :-)
 
 
 
 Max.
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: IPv6 hosts allow|deny

2003-01-20 Thread Dave Dykstra
Thanks for the quick tutorial.  I submitted your patch, changing eth0
to link1 as Hideaki suggested.

- Dave

On Mon, Jan 20, 2003 at 06:35:09PM +0100, Bert Vermeulen wrote:
 On Mon, 20 Jan 2003, Dave Dykstra wrote:
 
  On Fri, Jan 17, 2003 at 12:31:26AM +0100, Bert Vermeulen wrote:
   On Mon, 13 Jan 2003, Dave Dykstra wrote:
  
I don't understand enough how that syntax works in order to be able
to write an explanation for people.  Could you please try to write
something up based on the current version checked in to CVS?
  
   Here's a patch for the manpage:
  
 http://biot.com/patches/rsync-ipv6-acl-doc.patch
 
  I have never been introduced to IPv6 addresses before and there are some
  things about your documentation that baffle me.  Perhaps I just need to
  be educated a little, or perhaps it would be useful to include a little
  more explnation in the documentation.
 
  Your patch includes this:
   -  it() a dotted decimal IP address of the form a.b.c.d for IPv4 and
   -  a.b.c.d.e.f for IPv6. In this case the incoming machine's IP address
   +  it() a dotted decimal IPv4 address of the form a.b.c.d, or an IPv6 address
   +  of the form a:b:c::d:e:f. In this case the incoming machine's IP address
  must match exactly.
 
  Why is there a blank field between c and d?Do you think you
  should you say something about the fields being hexadecimal for IPv6,
  or is that obvious to anyone who has seen IPv6 addresses before?
 
 What Hideaki said, RFC 2373 describes the address format for IPv6.
 
 But yes, if you've ever dealt with IPv6, certainly this is obvious. I don't
 think it really needs explaining in the rsync docs as such...  If you have
 IPv6 working on the machine, you pretty much already know the address
 format.
 
 Anyway, briefly... :: denotes a number of 0 values, it's an abbreviation.
 You can only do this once per IPv6 address. It's exactle the same as writing
 this:
 
   000a:000b:000c:::000d:000e:000f
 
 So you can see how a:b:c::d:e:f is easier to type. This is standardized
 stuff, so you're guaranteed to be able to type that into anything that
 parses IPv6 addresses.
 
  Your patch also includes this:
   +  it() an address/mask in the form ipaddr/n where ipaddr is the IP address
   +  and n is the number of one bits in the netmask.  All IP addresses which
   +  match the masked IP address will be allowed in.
   +
   +  it() an address/mask in the form ipaddr/maskaddr where ipaddr is the
   +  IP address and maskaddr is the netmask in dotted decimal notation for IPv4,
   +  or similar for IPv6, e.g. ::::: instead of /64. All IP
   +  addresses which match the masked IP address will be allowed in.
 
 
  How come a mask includes 4 fields followed by 2 blanks?   What do you
  mean by /64?
 
 The /64 is just CIDR notation as used in IPv4 as well, i.e. the number of 1
 bits in the network mask. In IPv4, they go up to 32, where a /32 means a
 host. With IPv6, they go up to /128 for a host, however /64 is the
 standard netmask for a subnet.
 
  Your patch also has this addition:
   +Note IPv6 link-local addresses can have a scope in the address specification:
   +
   +quote(fe80::1%eth0)
   +quote(fe80::%eth0/64)
   +quote(fe80::%eth0/:::::)
 
  What's a link-local address?
 
 An address that is only valid for a certain link, i.e. only reachable
 through that interface. Thus the special prefix (fe80) -- and naturally,
 those are only useful if you specify the interface as well, as you might
 have the same link-local address on two different interfaces.
 
  What's a scope?
 
 In the case of a link-local address, the scope of an interface would be the
 subnet behind it.
 
  I think I get that the parts after the '/' are masks, but what does it
  mean to have just two colons in the address, with one field blank and the
  third one either 1%eth0 or %eth0?
 
 The 1 is the last part of the address. It goes on until the %, which
 denotes the start of the interface specification. The /64 after the
 interface is indeed the netmask.
 
 
 Bert Vermeulen
 [EMAIL PROTECTED]
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: possible typo/bug in receiver.c

2003-01-20 Thread Dave Dykstra
On Sat, Jan 18, 2003 at 12:25:05AM -0800, Wayne Davison wrote:
 On Fri, Jan 17, 2003 at 11:39:31PM -0800, Craig Barratt wrote:
  If mkstemp() fails (for various reasons, including the directory not
  existing) then fd == -1.  So the first if () executes, which flushes
  the data and does a continue.  So the next two if () statements will
  never execute.
 
 Good catch.  The code got tweaked into this form in version 1.32 when it
 was changed from a set of calls (using do_mktemp() and do_open()) into a
 single call (using just do_mkstemp()).  At that point the new (simpler)
 error-check section got added, and it made it impossible to get down to
 the older section.
 
  Is rsync meant to create deep directories that don't exist?
 
 Not usually.  In -R mode, it sends implied directories for the files, so
 it should be the case that the directories are normally created already.
 I find it interesting that this old code exists because there has been a
 lot of talk about the sending of implied directories (and an option to
 turn this off).  If we put this code back the way it was back in 2001,
 then the newly suggested --no-implied-dirs option would not cause rsync
 to fail if a deep file didn't have a directory yet -- rsync would just
 create the intervening directories.  I'm not sure if I want it to do
 that or not at the moment.
 
 ...wayne..


I removed the error check that skipped the creation of the directory,
just a piece of the code that was added in version 1.32.  It's pretty
clear to me that it was just an oversight, because Tridge also modified
the dead code leg.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Please test rsync-2.5.6pre2

2003-01-20 Thread Dave Dykstra
The second rsync-2.5.6 pre-release version is now available at:

http://rsync.samba.org/ftp/rsync/preview/rsync-2.5.6pre2.tar.gz
ftp://rsync.samba.org/pub/rsync/preview/rsync-2.5.6pre2.tar.gz
rsync://rsync.samba.org/ftp/rsync/preview/rsync-2.5.6pre2.tar.gz

There's also a corresponding '.sig' file that contains a gpg signature
of the file; the public key is available on the pgp keyservers.

This is a release candidate.  If no-one reports an urgent fixes within
the next few days, it will probably become version 2.5.6.

Known issues that will probably not get resolved before the release
unless a volunteer provides a solution:
1.  The NetBSD i386 port is failing the chgrp test.   See build output
on build.samba.org.
2.  I just noticed that the cygwin chgrp test has timestamps off by
two seconds.  Maybe --modify-window is supposed to be 1?


Changes since 2.5.5pre1:

  ENHANCEMENTS:

* Set the default value of --modify-window to 2 on Cygwin.  (Max Bowsher)

* Ignore errors from chmod when -p/-a/--preserve-perms is not set.
  (Dave Dykstra)

  BUG FIXES:
  
* Fixed the file-name duplicate-removal code when dealing with multiple
  dups in a row. (Wayne Davison)

* Fixed bug in --copy-unsafe-links that caused it to be completely
  broken.  (Dave Dykstra)

* Prevent infinite recursion in cleanup code under certain circumstances.
  (Sviatoslav Sviridov)

* Fixed a bug that prevented rsync from creating intervening directories
  when --relative-paths/-R is set.  (Craig Barratt)


  INTERNAL:

* Portability fixes. (Dave Dykstra and Wayne Davison)

- Dave Dykstra



msg06197/pgp0.pgp
Description: PGP signature


Re: unsafe_symlink change (Re: CVS update: rsync)

2003-01-17 Thread Dave Dykstra
On Thu, Jan 16, 2003 at 03:18:34PM -0600, Dave Dykstra wrote:
 On Thu, Jan 16, 2003 at 04:08:01PM -0500, Scott Mcdermott wrote:
  Dave Dykstra on Thu 16/01 14:57 -0600:
 The patch from 2-1/2 years ago for changing copy-unsafe-links to
 follow unsafe links on the destination side also included
 essentially this patch.

just to be clear, without using copy-unsafe-links, rsync still
copies absolute symlinks which point out of the source tree,
verbatim to the destination (even if it ends up dangling on the
destination), right?
   
   Right, with the -l or --links option (implied by -a).
  
  not to beat a dead horse, but copy-unsafe-links is certainly a misnomer.
  I liked your follow-unsafe-links much better.  Copying the links doesn't
  really have much to do with following them, which is what the option
  actually does...
 
 Yeah, what it really means is copy-in the files pointed to by unsafe links,
 but follow-unsafe-links is better except that the name applies equally
 well to source and destination.  I don't think it's worth changing the
 name unless functionality is also changed.

I just remembered that the name was based on the existing --copy-links
option so there was a precedent. 

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: specifying a list of files to transfer

2003-01-17 Thread Dave Dykstra
On Thu, Jan 16, 2003 at 11:14:50PM -0800, Wayne Davison wrote:
 On Thu, Jan 16, 2003 at 07:06:05PM -0800, jw schultz wrote:
  [...] and that entries therein are not flattened like they would be on
  the command-line (sans -R).
 
 But they *are* flattened exactly like on the command-line, at least in
 my current patch they are.  That's what -R is for -- telling rsync not
 to do that.  So, without -R there are no implied directories to create
 except for the destination dir (which is created if it doesn't exist).

Oh, right, I hadn't thought of that implication of the way this is
implemented.  Definitely we want the -R functionality implied.  That's
the only way I can imagine people wanting to use this.



  The permissions and ownership should be derived from the source.
  so effectively it should be as though
  ./deltapics
  where in the file list.
 
 Right.  In fact, that's exactly what happens with -R -- all intermediate
 directories get added to the file list (if they aren't already in it)
 without causing any extra recursion (even if -r was specified/implied).


In my former hack implementation of the exclude optimization (when 
there were only includes with no wildcards and a final exclude '*') it
was able to skip sending the parent directories completely.  Come to
think of it, I'm not sure what kind of permissions were used for the
directories that were not explicitly included, maybe it just use the
default.


 If people want the --files-from to imply -R then I'd want to see a
 --no-relative option to let people turn it off.

That would be easy to implement so I guess it wouldn't hurt but I really
can't see people wanting to do that.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: signing tarballs

2003-01-16 Thread Dave Dykstra
Martin left off some context that might confuse some list readers.  I
had inquired about how to sign the pre-release tarball.  I signed 
2.5.6pre1 with my personal key, but Martin suggested there be a team key.

On Thu, Jan 16, 2003 at 10:42:53AM +1100, Martin Pool wrote:
 [replied to list]
 
 There was a discussion about this on the Samba list a while ago
 
   http://lists.samba.org/pipermail/samba-technical/2002-November/040931.html
 
 Briefly
 
   We should create a team signing key, with an lifetime of about a
   year.  It has to be relatively short to allow for turnover in the
   people who have access to the key.
 
   The signing key must only be stored on secure machines, certainly
   *not* on samba.org.(If it was on samba.org, somebody who
   compromised that machine could also generate new signatures and it
   would be pointless.)
 
   The key should be signed by team members and other relevant people;
   we should also sign each others' keys.
 
   The key should be on the keyservers and on the web site.
 
 Unless you've already done so I'll create the key and send the private
 half to you and the public half to the website, keyservers, and list.

I have not done so, and if you're willing to set that up please go ahead.
A web page describing how to use the signature, like what you were talking
about on the samba list, would be great.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: Long-pending patch for Stratus VOS build

2003-01-16 Thread Dave Dykstra
I think the below patch is the cause of the failure of the Irix 6.5 type
on build.samba.org, because it is complaining about line 73 of the
Makefile which at least on my Linux machine is
rsync$(EXEEXT): $(OBJS)

It might be easiest for somebody who has an Irix machine that they can
use to debug this further.

- Dave


On Fri, Jan 10, 2003 at 01:49:37PM -0600, Dave Dykstra wrote:
 On Fri, Jan 10, 2003 at 11:33:00AM -0800, Wayne Davison wrote:
  On Fri, Jan 10, 2003 at 01:23:45PM -0500, Green, Paul wrote:
   The following patch still applies cleanly to the current cvs copy of rsync.
  
  Or did before the most recent Makefile.in changes.  It's easy to merge
  this one problem, though.
  
   Does anyone object to having these changes applied now, during the
   pre-release phase?
  
  Here are my comments on the changes:
  
   + The Makefile.in changes look very safe and needed.
  
   + The install-sh change to the dsttmp value looks good.
  
   ? I have a question about the portability of the u_FOO - uFOO_t
 changes.  The former is the BSD syntax for the unsigned FOO typedefs,
 and the latter is what, POSIX?  The changes work on Linux, at least.
 Perhaps we should just make these changes and try it out on the
 compile farm.
  
   + The inet_pton changes look right to make the code consistent.  The
 only possible glitch might be a system that has a prototype for
 inet_pton() but not the library code -- if the prototype conflicts,
 the compile would fail (there are ways to work around this, but let's
 worry about that if we actually find some weird system with this
 problem).
  
  I'd be glad to check in the + changes now if Dave thinks now is a good
  time.
 
 
 Ok, go ahead.  I think it's ok to try out the ? patch on the compile
 farm too.  I know that Cygwin also uses $(EXEEXT), but apparently its
 make has been more thoroughly modified to handle things automatically
 so the Makefile doesn't have to use it as much.
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: rsync-2.5.5 memory eater problem

2003-01-16 Thread Dave Dykstra
Thanks for the reminder.  I checked in the second version.

- Dave

On Wed, Oct 09, 2002 at 12:45:03PM +0300, Sviatoslav Sviridov wrote:
 On Wed, 9 Oct 2002 10:03:33 +0200 (CEST)
 Ruediger Oertel [EMAIL PROTECTED] wrote:
 
  Hi,
  
  we ran into a little problem with rsync-2.5.5.
  
  Setup: you run rsync-2.5.5 as normal rsync over ssh
  (ie. not connecting to a rsync server). If you start
  such a rsync but interrupt the pulling process with Ctrl-C,
  the process on the other side may start to allocate all
  memory on the remote machine.
  
  As fa as we have analyzed the problem, the remote rsync
  process wants to issue a error message that it could not
  write 4096 bytes, receives a SIGPIPE and errno is set to EPIPE
  in the fwrite call (line 279 in log.c). Then exit_cleanup is called
  which in turn will call the same routine again and will of course
  get EPIPE again and the loop continues (as long as we have memory
  to recursivly call the same functions again and again).
  
  As a quick workaround we have the following patch, since there is
  no real point in trying to issue an error message if you have
  already received EPIPE on write.
  
  --- log.c
  +++ log.c   2002/10/08 11:55:52
  @@ -276,7 +276,7 @@
  
  if (!f) exit_cleanup(RERR_MESSAGEIO);
  
  -   if (fwrite(buf, len, 1, f) != 1) exit_cleanup(RERR_MESSAGEIO);
  +   if (fwrite(buf, len, 1, f) != 1  errno != EPIPE)
  exit_cleanup(RERR_MESSAGEIO);
  
  if (buf[len-1] == '\r' || buf[len-1] == '\n') fflush(f);
   }
  
 
 I've had the same problem, but in different situation: when rsync was started
 as piped child (I did it in rsync method for apt). My fist path was above 
 one, but later I've found that problem is in exit_cleanup(). I use this patch:
 
 --- rsync-2.5.5/cleanup.c 2002-03-27 02:03:13 +0100
 +++ rsync-2.5.5.cleanup/cleanup.c 2002-07-11 12:53:52 +0200
 @@ -43,6 +43,11 @@
   int ocode = code;
   extern int keep_partial;
   extern int log_got_error;
 + static int inside_cleanup = 0;
 +
 + if (inside_cleanup != 0)
 + return;
 + inside_cleanup = 1;
  
   signal(SIGUSR1, SIG_IGN);
   signal(SIGUSR2, SIG_IGN);
 
 
 -- 
 Sviatoslav Sviridov
 [mailto:[EMAIL PROTECTED]] [ICQ#10845380] [Lintec Project] [MLUG]
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: --copy-unsafe-links fix checked in

2003-01-16 Thread Dave Dykstra
On Wed, Jan 15, 2003 at 02:24:10PM -0800, jw schultz wrote:
 On Wed, Jan 15, 2003 at 10:27:31AM -0600, Dave Dykstra wrote:
  In my research into this I also found that 2-1/2 years ago somebody
  posted a patch that included a fix for this buried in it, and that I
  had promised to integrate his patch but I never did it, I think due to
  various other crises that were happening at the time.  I also don't think
  I realized that the existing option was broken.  His patch expanded the
  --copy-unsafe-links option so that unsafe symlinks on the destination
  side are also followed.  I'm sure that could be a useful option, allowing
  things like moving some parts of a directory structure on the destination
  side to a separate filesystem.  However, it really doesn't have anything
  to do with the name copy-unsafe-links.  I'm thinking that either I
  should rename the --copy-unsafe-links option to --follow-unsafe-links
  and add this functionality, or add a separate option by that name that
  only affects the destination side.  Does anybody have an opinion on those?
 
 By follow-unsafe-links do you mean that if it finds a
 link to a directory it will follow that link and descend
 that directory tree located outside the tree being rsynced?
 That is what i would infer following to do.  While
 potentially useful this is also potentially dangerous.
 Following directory links should, if added, be distinguished
 from I/O through file links.
 
 Even if it is just links to files that it would write
 through it should be a separate option.  It is one thing to
 read through an unsafe link, quite another to write through it.
 A distinction should be made between them.
 
 There is a reason why apache disables following links by default.
 
 I may be reading you incorrectly but the ideas presented
 would, i think, need four options for unsafe links.
 I have made up names for the other three for the sake of
 enumeration.
 
   --copy-unsafe-links read through to files
   --write-unsafe-linkswrite through to files
 
   --follow-src-links  descend directory links on src
   --follow-dest-links descend directory links on dest


I certainly wouldn't want to have 4 options, it's confusing enough as
it is.  Rsync never overwrites files, it just replaces them, so at a
minimum I don't think --write-unsafe-links would be of value.  I don't
think it's worth distinguishing between files and directories on the
source side because I don't think that descending into a directory link
on the source side has much risk associated with it.

You have at least persuaded me to not have copy-unsafe-links affect
the destination.  I think I'll probably just drop the whole idea.  I had
briefly considered the problem of somebody malicious first copying in a
symlink to an external directory and then copying again to recurse into
it, but I dismissed it because I figure that pretty much the only time
time rsync users need to worry about malicious use is when uploading to
an rsync daemon module, and that's already protected by either chroot
or sanitize_path.  I had not considered the problem of accidental damage
that could be done, however, and for that I think it's better to not have
the option.

- Dave Dykstra
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: specifying a list of files to transfer

2003-01-16 Thread Dave Dykstra
On Wed, Jan 15, 2003 at 02:49:08PM -0800, jw schultz wrote:
 On Wed, Jan 15, 2003 at 10:03:33AM -0800, Wayne Davison wrote:
  On Tue, Jan 14, 2003 at 07:57:48PM -0800, jw schultz wrote:
   with the -r or -a options does this [recurse] on
   directories in the --files-from list?
  
  Yes, it treats them like command-line args with the following two
  exceptions:  if -r is not specified, we WILL create an explicitly named
  directory (but not send its contents), if -R is specified, we do NOT
  create implied directories (which was your next question).  This latter
  exception means that we currently require the user to ensure that the
  destination directory tree is valid (which could be done once with a
  separate rsync --files-from run that didn't use -r and specified all
  the dirs that we needed to ensure exist).  If this turns out to be too
  much of a hassle, perhaps a new option named --implied-dirs could be
  added to have rsync do its normal -R dir handling.
 
 Plus the third difference, that relative paths in the
 files-from list are relative to the tree, not to the current
 directory.
 
 We may well want the --implied-dirs option or some logic to
 created it.  If you don't have -r (or -a) you need to have
 all the intermediate dirs listed.  If you do have -r listing
 intermediate dirs efectively disables the file list.
 
 I'm not sure i like that.  I'm inclined to think the file-list
 should disable recoursion.  Perhaps [fighting resistance of
 yuck] recoursion would have to be specified explicitly after
 the --file-list.  I hate sequence sensitive options but
 but requiring users to remember -lptgoD instead of -a may be
 worse.
 
 You seem to see the files-from as a way of replacing
 command-line args where i see it as a way of replacing the
 tree scan.  However, if we can pin down the semantics i
 think we can acheive both ends.


I agree more with JW.  I envisioned --files-from as replacing the tree
scan.  I would rather have the '-r' option ignored when --files-from
is in effect.  I think it should be a complete list of the files and/or
directories that are to be sent.  If people leave out the directories,
missing parent directories should be automatically created.  As it is
now, if somebody just does find . -print | rsync -a --files-from- ...
are they going to get repeated files because the directories are listed?
Yuck.

Also, if the transfer is being sent from the remote side, the file names
are all getting sent over to the remote side first for --files-from and
then sent back as part of the normal protocol, right?  I had hoped we'd
be able to avoid that round trip because the list could get long.

- Dave

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: umask for rsync

2003-01-16 Thread Dave Dykstra
In that case it looks like you want the proposed --chmod option discussed at
http://lists.samba.org/pipermail/rsync/2002-March/006477.html
and
http://lists.samba.org/pipermail/rsync/2003-January/009298.html

Once it is a command line option it could easily be made into a 
per-module option as well.

- Dave

On Thu, Jan 16, 2003 at 01:45:50PM -0500, Aaron Morris wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1
 
 I suppose if you wanted a hack, that would work.  I was really looking
 at something that could be used on a per module basis in rsyncd.conf or
 on the command line.  A commercial rsync competitor, TeamSite
 OpenDeploy, has a file mode and a dir mode option.  Even Samba has a
 file and dir mode option on a per module (or share) basis.  I was a
 little suprised that rsync did not have a similar option when it has the
 user/group option.
 
 Perhaps I should have said a file mask and directory mask instead of
 UMASK.  My primary use of rsync right now is the deployment of web
 content from development machines to production machines.  I do not
 always have control of the content or when it is deployed, but the
 file/dir mask option would be a nice security feature.  Tt would keep
 people from deploying files as 666 or directories as 777.
 
 Here is an example of how it could be useful.
 A few months ago, I started having problems with file modes when
 transferring with rsync (AIX 4.3.3, rsync 2.5.4, rsync --daemon from
 inetd).  All of the sudden, rsync would deploy files as 640 instead of
 644 like it was doing the day before.  The default umask on the box had
 been changed to 027 several months before this point, but rsync had been
 deploying files as if the umask was still 022.  Come to find out, the
 box had been rebooted the night before; inetd was never restarted when
 the umask was changed (and usually is never restarted) and was still
 operating with a umask of 022.
 
 Dave Dykstra wrote:
 | On Wed, Jan 15, 2003 at 02:01:30AM -0500, Aaron Morris wrote:
 |
 |I have another small feature suggestion, it should not cause such a stir
 |as the whole file list issue (I kind of wish I never said anything
 |about it :) ).
 |
 |What about adding a UMASK setting to rsync (for use on the command
 |line and/or perhaps rsyncd.conf)?  I realize that it may be a
 |problem/conflict with -p, but I think it could be a useful feature.
 |
 |
 | In what way would it be different than making a 2-line script
 | umask NNN
 | rsync $@
 | ?
 |
 | - Dave
 |
 
 - --
 Aaron W Morris
 decep
 PGP Key ID:  259978D1
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



unsafe_symlink change (Re: CVS update: rsync)

2003-01-16 Thread Dave Dykstra
The patch from 2-1/2 years ago for changing copy-unsafe-links to follow
unsafe links on the destination side also included essentially this patch.
When I looked at it, however, I asked why in the world is unsafe_symlink()
doing strdup() in the first place.  I think you could get rid of the
calls to strdup() and the new local variables and possibly do a couple
casts inside the function instead.

- Dave


On Wed, Jan 15, 2003 at 12:49:44PM -0500, [EMAIL PROTECTED] wrote to rsync-cvs:
 Re: CVS update: rsync
 
 Date: Wed Jan 15 17:49:44 2003
 Author:   wayned
 
 Update of /data/cvs/rsync
 In directory dp.samba.org:/tmp/cvs-serv3194
 
 Modified Files:
   proto.h util.c 
 Log Message:
 Make unsafe_symlink() take const args so that we don't get any
 compiler warnings when calling it with a const char *.
 
 
 Revisions:
 proto.h   1.150 = 1.151
   http://www.samba.org/cgi-bin/cvsweb/rsync/proto.h?r1=1.150r2=1.151
 util.c1.117 = 1.118
   http://www.samba.org/cgi-bin/cvsweb/rsync/util.c?r1=1.117r2=1.118
 ___
 rsync-cvs mailing list
 [EMAIL PROTECTED]
 http://lists.samba.org/mailman/listinfo/rsync-cvs
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: unsafe_symlink change (Re: CVS update: rsync)

2003-01-16 Thread Dave Dykstra
On Thu, Jan 16, 2003 at 03:44:44PM -0500, Scott Mcdermott wrote:
 Dave Dykstra on Thu 16/01 14:33 -0600:
  The patch from 2-1/2 years ago for changing copy-unsafe-links to follow
  unsafe links on the destination side also included essentially this patch.
 
 just to be clear, without using copy-unsafe-links, rsync still copies
 absolute symlinks which point out of the source tree, verbatim to the
 destination (even if it ends up dangling on the destination), right?

Right, with the -l or --links option (implied by -a).

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: unsafe_symlink change (Re: CVS update: rsync)

2003-01-16 Thread Dave Dykstra
Sorry, another mistake on my part -- I didn't realize the strtok side
effect.  It doesn't look to me like your new code looks at the last
component of the path like the old one might have.  It would probably
be a good idea to put tests ending in /.., for example foo/../..
and foo/.., into testsuite/unsafe-byname.test.

I don't expect that the strdups are of significant overhead, but as long
as you've gone through the trouble to rewrite it I suppose we might as
well use it (assuming you have handled the last component ok).

- Dave

On Thu, Jan 16, 2003 at 05:34:25PM -0800, Wayne Davison wrote:
 On Thu, Jan 16, 2003 at 02:33:26PM -0600, Dave Dykstra wrote:
  I think you could get rid of the calls to strdup() and the new local
  variables and possibly do a couple casts inside the function instead.
 
 You wouldn't want to carve up the const strings with strtok() (which
 adds nulls to the strings).  The function could be rewritten to scan the
 string for '/'s without using strtok(), which would be a more efficient
 way to go (eliminating the need for the strdups).
 
 Here's a quick rewrite that I minimally tested.  See if you like it.
 
 ...wayne..
 
 ---8--8--8--8---cut here---8--8--8--8---
 Index: util.c
 --- util.c15 Jan 2003 17:49:44 -  1.118
 +++ util.c17 Jan 2003 01:26:36 -
 @@ -793,49 +793,38 @@
   *
   * @sa t_unsafe.c
   **/
 -int unsafe_symlink(const char *dest_path, const char *src_path)
 +int unsafe_symlink(const char *dest, const char *src)
  {
 - char *tok, *src, *dest;
 + const char *name, *slash;
   int depth = 0;
  
   /* all absolute and null symlinks are unsafe */
 - if (!dest_path || !*dest_path || *dest_path == '/') return 1;
 -
 - src = strdup(src_path);
 - if (!src) out_of_memory(unsafe_symlink);
 + if (!dest || !*dest || *dest == '/') return 1;
  
   /* find out what our safety margin is */
 - for (tok=strtok(src,/); tok; tok=strtok(NULL,/)) {
 - if (strcmp(tok,..) == 0) {
 + for (name = src; (slash = strchr(name, '/')); name = slash+1) {
 + if (strncmp(name, ../, 3) == 0) {
   depth=0;
 - } else if (strcmp(tok,.) == 0) {
 + } else if (strncmp(name, ./, 2) == 0) {
   /* nothing */
   } else {
   depth++;
   }
   }
 - free(src);
 -
 - /* drop by one to account for the filename portion */
 - depth--;
 -
 - dest = strdup(dest_path);
 - if (!dest) out_of_memory(unsafe_symlink);
  
 - for (tok=strtok(dest,/); tok; tok=strtok(NULL,/)) {
 - if (strcmp(tok,..) == 0) {
 - depth--;
 - } else if (strcmp(tok,.) == 0) {
 + for (name = dest; (slash = strchr(name, '/')); name = slash+1) {
 + if (strncmp(name, ../, 3) == 0) {
 + /* if at any point we go outside the current directory
 +then stop - it is unsafe */
 + if (--depth  0)
 + break;
 + } else if (strncmp(name, ./, 2) == 0) {
   /* nothing */
   } else {
   depth++;
   }
 - /* if at any point we go outside the current directory then
 -stop - it is unsafe */
 - if (depth  0) break;
   }
  
 - free(dest);
   return (depth  0);
  }
  
 ---8--8--8--8---cut here---8--8--8--8---
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: unsafe_symlink change (Re: CVS update: rsync)

2003-01-16 Thread Dave Dykstra
On Thu, Jan 16, 2003 at 04:08:01PM -0500, Scott Mcdermott wrote:
 Dave Dykstra on Thu 16/01 14:57 -0600:
The patch from 2-1/2 years ago for changing copy-unsafe-links to
follow unsafe links on the destination side also included
essentially this patch.
   
   just to be clear, without using copy-unsafe-links, rsync still
   copies absolute symlinks which point out of the source tree,
   verbatim to the destination (even if it ends up dangling on the
   destination), right?
  
  Right, with the -l or --links option (implied by -a).
 
 not to beat a dead horse, but copy-unsafe-links is certainly a misnomer.
 I liked your follow-unsafe-links much better.  Copying the links doesn't
 really have much to do with following them, which is what the option
 actually does...

Yeah, what it really means is copy-in the files pointed to by unsafe links,
but follow-unsafe-links is better except that the name applies equally
well to source and destination.  I don't think it's worth changing the
name unless functionality is also changed.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: rsync hanging with openssh-2.9.9p2 as the transport

2003-01-15 Thread Dave Dykstra
On Tue, Jan 14, 2003 at 05:29:06PM -0500, Andrew J. Schorr wrote:
 Hi,
 
 This is perhaps a stupid question.  I apologize in advance if it's already
 been covered, but I'm stumped...
 
 I'm using rsync to backup some file systems to a remote host.
 The transport is openssh-2.9.9p2.
 
 The problem does not occur when the transport is rsh.
 
 I'm invoking rsync as follows:
 
   rsync -RlHptgoD --numeric-ids -e ssh -rzx --stats . remotehost:/nfs/mirror
 
 But it hangs when there are large numbers of files in the . filesystem.
 If I run with -vvv, I see lots of make_file entries as send_file_list
 starts sending the list of files to the remote host, but then it freezes
 after around 4500 to 6500 make_file messages.
 
 I see the same problem with versions 2.4.6, 2.5.5, and 2.5.6pre1.
 And I see it on linux 2.4.18 (red hat 8.0) and solaris 8/x86.
 
 I can get it to work by breaking it up into chunks of 1000 files or
 so.  It might work with more, I haven't tested exhaustively.
 
 I have tried using the --blocking-io and --no-blocking-io options, but
 neither one solves the problem.
 
 I could provide more info about where it hangs, but I thought somebody
 might know the answer, since this is clearly related to the
 interaction with openssh (since there's no problem with rsh).
 
 Is there a trick to make rsync work nicely with openssh?  I searched the
 archives and haven't found anything...  Does upgrading to openssh 3 solve
 the problem?

Unfortunately I don't think anybody is going to be able to tell you.
I've not heard of anybody lately posting a similar problem.  In the
past hanging problems have been traced to many different sources.
Rsync stresses network (and filesystem) implementations greatly, and
combining it with ssh stresses things that much more.  I think it's
worth a try to use openssh 3.1 or 3.2 (I've not been happy with versions
after 3.2).  What's the network between the local and remote machines?
Does the name /nfs/mirror imply that the files are not directly mounted
on remote but are instead on an NFS server?  That has been the cause
of many problems in the past.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



--copy-unsafe-links fix checked in

2003-01-15 Thread Dave Dykstra
Below is the patch I just checked in to fix --copy-unsafe-links.  As far
as I can tell it has been broken for years, ever since I first put it in.
It bothers me a bit that I can't figure out why I went through the
trouble and mess at the time to introduce a global variable to compare
to wrong information rather than using the parameter already available
in the function, but I've given up trying to figure out why.

In my research into this I also found that 2-1/2 years ago somebody
posted a patch that included a fix for this buried in it, and that I
had promised to integrate his patch but I never did it, I think due to
various other crises that were happening at the time.  I also don't think
I realized that the existing option was broken.  His patch expanded the
--copy-unsafe-links option so that unsafe symlinks on the destination
side are also followed.  I'm sure that could be a useful option, allowing
things like moving some parts of a directory structure on the destination
side to a separate filesystem.  However, it really doesn't have anything
to do with the name copy-unsafe-links.  I'm thinking that either I
should rename the --copy-unsafe-links option to --follow-unsafe-links
and add this functionality, or add a separate option by that name that
only affects the destination side.  Does anybody have an opinion on those?

- Dave


diff -ur ../rsync.current/flist.c ./flist.c
--- ../rsync.current/flist.cSat Jan 11 08:18:02 2003
+++ ./flist.c   Tue Jan 14 16:04:14 2003
@@ -61,8 +61,6 @@
 extern int read_batch;
 extern int write_batch;
 
-static char topsrcname[MAXPATHLEN];
-
 static struct exclude_struct **local_exclude_list;
 
 static struct file_struct null_file;
@@ -221,8 +219,11 @@
if (l == -1) 
return -1;
linkbuf[l] = 0;
-   if (copy_unsafe_links  (topsrcname[0] != '\0') 
-   unsafe_symlink(linkbuf, topsrcname)) {
+   if (copy_unsafe_links  unsafe_symlink(linkbuf, path)) {
+   if (verbose  1) {
+   rprintf(FINFO,copying unsafe symlink \%s\ - 
+\%s\\n, 
+   path, linkbuf);
+   }
return do_stat(path, buffer);
}
}
@@ -895,7 +896,8 @@
}
 
for (i = 0; i  argc; i++) {
-   char *fname = topsrcname;
+   char fname2[MAXPATHLEN];
+   char *fname = fname2;
 
strlcpy(fname, argv[i], MAXPATHLEN);
 
@@ -1000,8 +1002,6 @@
}
}
 
-   topsrcname[0] = '\0';
-
if (f != -1) {
send_file_entry(NULL, f, 0);
}
diff -ur ../rsync.current/testsuite/unsafe-byname.test ./testsuite/unsafe-byname.test
--- ../rsync.current/testsuite/unsafe-byname.test   Mon Apr  8 23:49:42 2002
+++ ./testsuite/unsafe-byname.test  Tue Jan 14 16:37:20 2003
@@ -44,12 +44,3 @@
 
 test_unsafe ../../unsafe/unsafefile`pwd`/from/safe safe
 test_unsafe ../files/file1`pwd`/from/safe  safe
-
-# I think these are correct, but I don't really understand the code.
-# Waiting for an answer from dwd
-test_skipped correct behaviour is unclear
-
-# Test suspicious handling of '..' in srcdir
-test_unsafe ../../unsafe/unsafefilefrom/safe/../safe   unsafe
-test_unsafe ../files/file1from/safe/../safesafe
-
diff -ur ../rsync.current/testsuite/unsafe-links.test ./testsuite/unsafe-links.test
--- ../rsync.current/testsuite/unsafe-links.testThu Jan  9 10:07:15 2003
+++ ./testsuite/unsafe-links.test   Tue Jan 14 16:45:06 2003
@@ -46,20 +46,13 @@
 test_regular to/links/file2
 test_regular to/links/unsafefile
 
-#next rsync copy correctly
 echo rsync with relative path and --copy-unsafe-links;
 $RSYNC -avv --copy-unsafe-links from/safe/ to
 test_symlink to/links/file1
 test_symlink to/links/file2
 test_regular to/links/unsafefile
 
-# I think these are correct, but I don't really understand the code.
-# Waiting for an answer from dwd
-test_skipped correct behaviour is unclear
-
-
 rm -rf to
-#next rsync copy incorrectly - links are copied as files not as links
 echo rsync with relative2 path;
 (cd from; $RSYNC -avv --copy-unsafe-links safe/ ../to)
 test_symlink to/links/file1
@@ -67,7 +60,6 @@
 test_regular to/links/unsafefile
 
 rm -rf to
-#next rsync copy uncorectly - all links are copied
 echo rsync with absolute path;
 $RSYNC -avv --copy-unsafe-links `pwd`/from/safe/ to
 test_symlink to/links/file1
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: rsync hanging with openssh-2.9.9p2 as the transport

2003-01-15 Thread Dave Dykstra
On Wed, Jan 15, 2003 at 05:12:21PM -0500, Andrew J. Schorr wrote:
 On Wed, Jan 15, 2003 at 09:27:37AM -0600, Dave Dykstra wrote:
  Unfortunately I don't think anybody is going to be able to tell you.
  I've not heard of anybody lately posting a similar problem.  In the
  past hanging problems have been traced to many different sources.
  Rsync stresses network (and filesystem) implementations greatly, and
  combining it with ssh stresses things that much more.  I think it's
  worth a try to use openssh 3.1 or 3.2 (I've not been happy with versions
  after 3.2).  What's the network between the local and remote machines?
  Does the name /nfs/mirror imply that the files are not directly mounted
  on remote but are instead on an NFS server?  That has been the cause
  of many problems in the past.
 
 Actually, it turns out that my testing on this issue has been invalid
 since I've been using the -vvv flag to rsync to get a grip on where
 it's hanging.  But I just discovered that the -vvv flag itself seems
 to cause rsync to hang.  So all the testing I've done so far is useless,
 so I'm not sure whether this is related to openssh or the slow connection
 or what.  I know that the problem exists in 2.5.5, but I need to do more
 testing without the -vvv flag to figure out what's causing it.  Is it
 well-known that the -vvv flag can cause rsync to hang?


That has been shown to cause hangs in the past, but I thought it was
resolved.  Maybe it hasn't; people don't tend to care so much to solve
problems like that once they discover the workaround of removing the
verbosity.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: Rsync performance increase through buffering

2003-01-14 Thread Dave Dykstra
Craig, I'd like to get your patch into the 2.5.6 patches directory.
Could you please make sure it applies cleanly onto version 2.5.6pre1 (see
news on http://rsync.samba.org home page if you haven't been following
the mailing list) and repost it?

Thanks,

- Dave Dykstra

On Wed, Dec 11, 2002 at 03:20:57PM -0600, Dave Dykstra wrote:
 On Sun, Dec 08, 2002 at 11:48:57PM -0800, Craig Barratt wrote:
  I've been studying the read and write buffering in rsync and it turns
  out most I/O is done just a couple of bytes at a time.  This means there
  are lots of system calls, and also most network traffic comprises lots
  of small packets.  The behavior is most extreme when sending/receiving
  file deltas of identical files.
 ...
  Below is a patch to a few files that adds read and write buffering in
  the places where the I/O was unbuffered, adds buffering to write_file()
  and removes the unneeded gettimeofday() system call in show_progress().
 ...
  However, on a network test doing a send from cygwin/WinXP to rsyncd
  on rh-linux the running time improves from about 700 seconds to 215
  seconds (with a cpu load of around 17% versus 58%, if you believe
  cygwin's cpu stats).  This is probably an extreme case since the system
  call penalty in cygwin is high.  But I would suspect a significant
  improvement is possible with a slow network connection, since a lot
  less data is being sent.
 
 Looks like a good candidate for the patches directory now and for
 integration in 2.6.0.
 
 - Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



2.5.6pre1 bombs on Sunos4 in popthelp.c on use of sprintf

2003-01-14 Thread Dave Dykstra
2.5.6pre1 bombs on Sunos4 gcc with these errors
popt/popthelp.c: In function `singleOptionDefaultValue':
popt/popthelp.c:137: invalid operands to binary +
popt/popthelp.c:141: invalid operands to binary +
popt/popthelp.c:145: invalid operands to binary +
popt/popthelp.c:149: invalid operands to binary +
because it's depending on sprintf to return the number of bytes written
and that doesn't happen on Sunos4.

Wayne or anybody, do you have a suggested solution?  The popt maintainers
should also be told about that.  This foible of Sunos4 is mentioned in
the autoconf documentation of things to watch out for but I don't think
they suggest a solution.

We need to get one of these Sunos4 machines into the build farm.  I know
where to get one, but I had been hesitant to add machines to the build
farm because of security concerns.  I think the new rsync module over
ssh feature will eliminate those concerns.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: configure issue (ac_cv_lib_inet_connect) on DYNIX/ptx

2003-01-14 Thread Dave Dykstra
On Mon, Jan 13, 2003 at 09:03:23PM -0500, Michael Sterrett -Mr. Bones.- wrote:
 Greetings -
 
 In trying to get rsync-2.5.6pre1 working on DYNIX/ptx, I found I
 needed to pass ac_cv_lib_inet_connect=no to configure in order for
 it actually create a properly configured config.h file.
 
 I think this is because configure looks for connect in libinet,
 (and finds one there somehow).  But connect() is resolved by libsocket
 (but configure doesn't find it when it looks there).  I've never taken
 the time to delve into the guts of autoconf, so I'm not sure if this
 is a general autoconf problem or a problem that is specific to rsync.
 
 At any rate, hopefully, this will be useful to someone.  I can send
 a config.log file to anyone who's interested.
 
 After passing ac_cv_lib_inet_connect=no to configure, all tests either
 PASS or SKIP so it looks good.

Ok, we'll assume that you'll continue to do it that way unless you post
a patch to configure.in.  Really there should be a modification to 
autoconf, probably an AC_FUNC_CONNECT macro, but I'm not sure anybody
wants to do the work.  I do see a couple mentions of DYNIX in the autoconf
2.53 macros.



 I also get this warning, which is trivial, but it would be nice to
 clean it up:
 
   cleanup.c, line 36: portability warning: trigraph sequence replaced
 
 This is from the () in cleanup.c.  To get rid of the warning,
 just remove the () from line 36.

I don't want to remove it completely because apparently it means something
to the person who put it in.  Please suggest another form that conveys the
same meaning and doesn't cause a warning.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: 2.5.6pre1 bombs on Sunos4 in popthelp.c on use of sprintf

2003-01-14 Thread Dave Dykstra
On Tue, Jan 14, 2003 at 10:24:04AM -0600, Dave Dykstra wrote:
 2.5.6pre1 bombs on Sunos4 gcc with these errors
 popt/popthelp.c: In function `singleOptionDefaultValue':
 popt/popthelp.c:137: invalid operands to binary +
 popt/popthelp.c:141: invalid operands to binary +
 popt/popthelp.c:145: invalid operands to binary +
 popt/popthelp.c:149: invalid operands to binary +
 because it's depending on sprintf to return the number of bytes written
 and that doesn't happen on Sunos4.

If nobody objects to the following patch, I'll put it in.  It's a little
slower but this is definitely not time-critical code.

- Dave


--- popt/popthelp.c.O   Tue Jan 14 13:02:47 2003
+++ popt/popthelp.c Tue Jan 14 13:04:45 2003
@@ -134,19 +134,23 @@
 case POPT_ARG_VAL:
 case POPT_ARG_INT:
 {  long aLong = *((int *)opt-arg);
-   le += sprintf(le, %ld, aLong);
+   sprintf(le, %ld, aLong);
+   le += strlen(le);
 }  break;
 case POPT_ARG_LONG:
 {  long aLong = *((long *)opt-arg);
-   le += sprintf(le, %ld, aLong);
+   sprintf(le, %ld, aLong);
+   le += strlen(le);
 }  break;
 case POPT_ARG_FLOAT:
 {  double aDouble = *((float *)opt-arg);
-   le += sprintf(le, %g, aDouble);
+   sprintf(le, %g, aDouble);
+   le += strlen(le);
 }  break;
 case POPT_ARG_DOUBLE:
 {  double aDouble = *((double *)opt-arg);
-   le += sprintf(le, %g, aDouble);
+   sprintf(le, %g, aDouble);
+   le += strlen(le);
 }  break;
 case POPT_ARG_STRING:
 {  const char * s = *(const char **)opt-arg;
@@ -271,7 +275,8 @@
*le++ = '=';
if (negate) *le++ = '~';
/*@-formatconst@*/
-   le += sprintf(le, (ops ? 0x%lx : %ld), aLong);
+   sprintf(le, (ops ? 0x%lx : %ld), aLong);
+   le += strlen(le);
/*@=formatconst@*/
*le++ = ']';
}   break;
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: specifying a list of files to transfer

2003-01-14 Thread Dave Dykstra
On Tue, Jan 14, 2003 at 01:21:29PM -0800, Wayne Davison wrote:
 On Tue, Jan 14, 2003 at 11:02:44AM -0500, Andrew J. Schorr wrote:
  I am attaching an updated version of my patch to allow you to specify
  a list of files to transfer.
 
 Cool.  I'm looking into making this work when fetching files.  Towards
 that end, I'd like to suggest an alternate command-line syntax to make
 the --source-file option take a filename.  This will allow it to accept
 - as stdin, and will make it easy to parse for the pull syntax.  This
 means that we need to omit the SRC spec on a push or specify it as
 empty.  E.g. these will all work:
 
   rsync --source-list=file remote:/path
   rsync --source-list file : remote:/path
   rsync --source-list=-  remote:/path file
 
 A pull looks like this:
 
   rsync --source-list=file remote: /path
   rsync --source-list - remote::module /path file
 
 What do people think?  Of course this is not for the rsync release we're
 currently working on, but could be included as a patch, if desired.

I haven't looked at the implementation, but comments on the user
interface:
1. Yes it should take a filename or - as a parameter.
2. I don't like the idea of skipping the SRC spec.  Paths should be
relative to the SRC.  If somebody wants to use full paths they
can always have a SRC of /.
3. It should be called --files-from.
4. --send-dirs and --no-implicit-dirs shouldn't be separate options,
they should be automatically turned on with the --files-from option.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: IPv6 hosts allow|deny

2003-01-13 Thread Dave Dykstra
On Sun, Jan 12, 2003 at 11:51:07PM +0100, Bert Vermeulen wrote:
 On Fri, 10 Jan 2003, Bert Vermeulen wrote:
 
  On Thu, 9 Jan 2003, Dave Dykstra wrote:
 
   I went ahead and submitted Hideaki's patch pretty much as is.  I took off
   the ifdef around the memset at the beginning of client_sockaddr() because
   I figured it wouldn't hurt to do anywhere.  Besides, it was checking
   for a specific operating system (__linux__) which we like to avoid and
   because there was a typo in the check for HAVE_SOCKADDR_IN6_SCOPE_ID
   (it was HAVE_SOCKADDR_IN6_SOCPE_ID).  I noticed that there was code in
   access.c that was accessing the same sockaddr_in6 sin6_scope_id field that
   was protected by ifdef in clientname.c so I put the same ifdef around
   the code in access.c.  I also updated the rsyncd.conf.yo documentation
   to allow for IP address of the IPv6 form.
  
   I have not tested this, I hope it works.
 
  Dave,
 
  Sorry for the late reply, been busy. I'm going to test with the committed
  patch, and will let you know. I can only test on linux, however.
 
 Well, it works great for me. It might be a good idea to include Hideaki's
 comments on this syntax in the docs:
 
  Note that scope-id is also supported:
   - fe80::1%eth0
   - fe80::%eth0/64
   - fe80::%eth0/:::::
 
 The example is significant -- this restricts access to the local subnet
 only.


I don't understand enough how that syntax works in order to be able
to write an explanation for people.  Could you please try to write
something up based on the current version checked in to CVS?

I am totally mystified by Hideaki's examples.  I didn't think colons were
allowed, and I don't see any code in access.c for recognizing them.  As far
as I can tell the code only allows decimal notation, not hexadecimal.  And
do the blank fields mean anything?

For the record, here are the rest of Hideaki's examples:
   - 3ffe::dead:beef::1
   - 3ffe::dead:beef::/64
   - 3ffe::dead:beef::/:::::
  or even
   - ::0123:4567:89ab:cdef/:::::

- Dave Dykstra
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: am I missing something, or are permissions always preserved?

2003-01-13 Thread Dave Dykstra
On Sat, Jan 11, 2003 at 11:19:33AM -0800, Ben wrote [off list]:
 On Tue, 2003-01-07 at 14:49, Dave Dykstra wrote:
  That sounds reasonable to me that rsync shouldn't try to preserve those
  extra bits without -p.  Try making a patch and seeing if that works.
  
  I don't have Samba set up, but on a vfat filesystem on Linux rsync is
  doing even worse because after the initial file creation it does a
  fchmod(fd, 0600) which fails and it doesn't even copy files at all.
  
 
 
 Okay, attached is a patch I just whipped up that seems to work well. It
 adds the --skip-perms flag, which has the effect of making rsync not try
 to set permissions, not even to sane values.

I was not able to read your patch because it seems to have lots of
characters that I can't view on my email reader.  A header on the
attachment says
X-MIME-Autoconverted: from 8bit to base64 by mx6.mx.voyager.net
so my ISP must have had trouble with it.

Anyway, I wasn't thinking of an option, I was thinking of it doing
something that was the default whenever there was no -p.  I'd rather
avoid adding an option if we can get away with it.  I'm now having second
thoughts about stripping off the setuid/setgid bits by default without
'-p', because that's not what GNU cp does.  I think it would be better if
it just silently ignored errors from chmod and fchmod when -p is not set.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Please test rsync-2.5.6pre1

2003-01-13 Thread Dave Dykstra
The first rsync-2.5.6 pre-release version is now available at:

http://rsync.samba.org/ftp/rsync/preview/rsync-2.5.6pre1.tar.gz
ftp://rsync.samba.org/pub/rsync/preview/rsync-2.5.6pre1.tar.gz
rsync://rsync.samba.org/ftp/rsync/preview/rsync-2.5.6pre1.tar.gz

There's also a corresponding '.sig' file that contains a gpg signature
of the file; the public key is available on the pgp keyservers.

Please test it out thoroughly and report problems and bug fix patches to
this mailing list.

Known issues that will probably need to be resolved before the final
release:
1.  The Redhat Linux ia64 and Alpha and the OpenBSD Sparc64 ports are
all failing the daemon-gzip-upload and daemon-gzip-download tests
even though the basic daemon test succeeds.  The OpenBSD test
reports a core dump, and I suspect they're all having the same
problem related to 64 bit.  See build output on build.samba.org.
Somebody else is going to have to fix this, I don't have the
resources.
2.  The NetBSD i386 port is failing the chgrp test.  NetBSD users, see
build.samba.org.
3.  --copy-unsafe-links is badly broken.  I'm planning on trying to
fix this myself.
4.  The --modify-window option should default to 2 seconds or 1 second
on Cygwin.  I'll put this in once somebody determines which it
should be.
5.  The documentation hosts allow  hosts deny needs to be updated for
IPv6.


Changes since the 2.5.5 release:

  ENHANCEMENTS:

* The --delete-after option now implies --delete.  (Wayne Davison)

* The --suffix option can now be used with --backup-dir.  (Michael
  Zimmerman)

* Combining :: syntax with the -rsh/-e option now uses the
  specified remote-shell as a transport to talk to a (newly-spawned)
  server-daemon.  This allows someone to use daemon features, such
  as modules, over a secure protocol, such as ssh.  (JD Paul)

* The rsync:// syntax for daemon connections is now accepted in the
  destination field.

* If the file name given to --include-from or --exclude-from is -,
  rsync will read from standard input.  (J.W. Schultz)

* New option --link-dest which is like --compare-dest except that
  unchanged files are hard-linked in to the destination directory.
  (J.W. Schultz)

* Don't report an error if an excluded file disappears during an
  rsync run.  (Eugene Chupriyanov and Bo Kersey)

* Added .svn to --cvs-exclude list to support subversion.  (Jon
  Middleton)

* Properly support IPv6 addresses in the rsyncd.conf hosts allow
  and hosts deny fields.  (Hideaki Yoshifuji)

  BUG FIXES:
  
* Fix forward name lookup failed errors on AIX 4.3.3.  (John
  L. Allen, Martin Pool)

* Generate each file's rolling-checksum data as we send it, not
  in a separate (memory-eating) pass before hand.  This prevents
  timeout errors on really large files. (Stefan Nehlsen)

* Fix compilation on Tru64.  (Albert Chin, Zoong Pham)

* Better handling of some client-server errors.  (Martin Pool)

* Fixed a crash that would occur when sending a list of files that
  contains a duplicate name (if it sorts to the end of the file
  list) and using --delete.  (Wayne Davison)

* Fixed a bug that caused rsync to lose the exit status of its child
  processes and sometimes return an exit code of 0 instead of showing
  an error.  (David R. Staples, Dave Dykstra)

  INTERNAL:

* Many code cleanups and improved internal documentation.  (Martin 
  Pool, Nelson Beebe)

* More test cases.  (Martin Pool)

* Updated included popt to the latest vendor drop, version 1.6.4.
  (Jos Backus)

- Dave Dykstra



msg06044/pgp0.pgp
Description: PGP signature


Re: [rsync] Please test rsync-2.5.6pre1

2003-01-13 Thread Dave Dykstra
On Mon, Jan 13, 2003 at 01:59:47PM -0500, Scott Evans wrote:
  Known issues that will probably need to be resolved before the final
  release:
  [...]
 
 Is there any interest in trying to hunt down the linux - cygwin
 rsync hanging problem I'm consistently seeing before this release?
 I'm more than happy to help track it down but I don't know enough about
 rsync to do it myself.

Well, at a minimum it would be nice to know if it is still happening
with version 2.5.6pre1.  If it's no worse than previous versions, we
won't hold up a release of 2.5.6.  If a cygwin user tracks it down and
provides a fix that doesn't break anybody else, we'll probably put it in.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: am I missing something, or are permissions always preserved?

2003-01-13 Thread Dave Dykstra
On Mon, Jan 13, 2003 at 11:00:58AM -0800, Ben wrote:
 Well that's annoying. I've tried changing the encoding of this
 attempt... maybe it'll get through this time.

Yes, it came through better this time.

 As a rule of thumb, I think silent errors a very bad idea. It means
 things might not be behaving like you expect, but you have no idea. In
 general, rsync's current behavior is correct. In my case niche case, it
 doesn't work. I think the proper solution is to change rsync's behavior,
 not reduce the errors it produces.

Maybe it wouldn't be so bad if it only looked for certain error codes from
the failed chmod.  The nice thing about ignoring errors is that it's one
less thing for users to have to learn and understand.

Any other opinions?

 If it comes through, take a look at the patch. I think I did a decent
 job of updating the docs so that purpose of the new flag is clear.

At a minimum I think the option is too limited.  I'd want a solution
that would work for both the fixed permission samba case and the vfat
filesystem under Linux case.  In particular, I don't see it skipping
the fchmod in do_mkstemp.  I don't know, maybe that's the only addition
that would be needed.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: working on a 2.5.6pre1 release

2003-01-10 Thread Dave Dykstra
On Fri, Jan 10, 2003 at 12:46:13AM -0800, Wayne Davison wrote:
 On Thu, Jan 09, 2003 at 05:09:07PM -0600, Dave Dykstra wrote:
  I'm stuck on a problem that some machines on build.samba.org are
  showing on the 'chgrp' test.
 
 I've checked in a fix for this bug.

Thanks, Wayne!  Great work.

I think the remaining build on build.samba.org are not enough to delay
putting out pre1, but it would be good if those of you who care about
the following ports would try to come up with some fixes before we do
the final release:

1. The CRAY SV1 UNICOS machine is not compiling, complaining that it
can't take the address of the bit field in sin.sin_addr.s_addr.
I think the solution may be to assign that value to a local 
variable and then copy it into the structure element, but it
isn't clear to me what the type of the local variable should be.
2. The Linux ia64 and Alpha and the OpenBSD Sparc64 ports are all
failing the daemon-gzip-upload and daemon-gzip-download tests
even though the basic daemon test succeeds.  The OpenBSD test
reports a core dump, and I suspect they're all having the same
problem related to 64 bit.
3. The Stratus VOS port is failing all 3 daemon tests in code
that is used just for testing, saying it can't create the test
socket.  I don't know if there's a corresponding problem in the
corresponding non-test code.
4. The NetBSD i386 port is failing the chgrp test.

I think if I can find the time today I want to work a little yet on the
--copy-unsafe-links problem and on the unsafe-* tests and then I'll make
the pre1 version.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: make clean

2003-01-10 Thread Dave Dykstra
I checked in this patch.

- Dave

On Thu, Jan 09, 2003 at 06:41:19PM -0800, jw schultz wrote:
 I just did a make clean to tidy up a bit and found four
 objects were missed.  The seem to belong to the CHECK_PROGS
 set.  Not sure where best to put them so created a
 CHECK_OBJS variable and added it to the clean rule.
 
 
 -- 
 
   J.W. SchultzPegasystems Technologies
   email address:  [EMAIL PROTECTED]
 
   Remember Cernan and Schmitt

 Index: Makefile.in
 ===
 RCS file: /cvsroot/rsync/Makefile.in,v
 retrieving revision 1.84
 diff -u -r1.84 Makefile.in
 --- Makefile.in   8 Apr 2002 06:23:34 -   1.84
 +++ Makefile.in   10 Jan 2003 02:40:34 -
 @@ -43,6 +43,9 @@
  # Programs we must have to run the test cases
  CHECK_PROGS = rsync tls getgroups trimslash t_unsafe
  
 +# Objects for CHECK_PROGS to clean
 +CHECK_OBJS=getgroups.o t_stub.o t_unsafe.o trimslash.o
 +
  # note that the -I. is needed to handle config.h when using VPATH
  .c.o:
  @OBJ_SAVE@
 @@ -106,7 +109,7 @@
   cat $(srcdir)/*.c $(srcdir)/lib/compat.c | awk -f $(srcdir)/mkproto.awk  
$(srcdir)/proto.h
  
  clean: cleantests
 - rm -f *~ $(OBJS) rsync $(TLS_OBJ) $(CHECK_PROGS)
 + rm -f *~ $(OBJS) rsync $(TLS_OBJ) $(CHECK_PROGS) $(CHECK_OBJS)
  
  cleantests:
   rm -rf ./testtmp*

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: Windows and default --modify-window=0

2003-01-10 Thread Dave Dykstra
Ok, I agree --modify-window should default to 2 (or 1 if that's all that's
really needed) on cygwin.  However, I don't like os-specific defines.
That _WIN32 we've got there is the only one currently in the code, and
there's no occurrances of __CYGWIN__.   I'd rather have a configure.in
rule for it.  Any suggestions on a good one?  At a minimum there's an
AC_CYGWIN macro but that's not much different than just checking for
__CYGWIN__.  Probably it would be worth calling AC_CYGWIN and then setting
a DEFAULT_MODIFY_WINDOW to 2 in configure if $CYGWIN=yes.  As far as I
know there is no other port of rsync to Windows yet except under Cygwin
so I don't think it is worth putting in extra code to support something
that may never be used.  Or, perhaps we should key off of the $EXEEXT
variable which gets set to '.exe' on Windows; anybody have an opinion on
that?

- Dave

On Thu, Jan 09, 2003 at 02:41:21PM -0800, jw schultz wrote:
 On Thu, Jan 09, 2003 at 11:21:36PM +0100, Lapo Luchini wrote:
  jw schultz wrote:
  
  The 2 second timestamp resolution only applies to some
  windows filesystems.  I think NTFS has timestamp resolution
  in the milliseconds (unlrelated to precision).  It should
  only become an issue when the windows filesystem is the
  destination.
  
  This message says it is 2 seconds aso on NTFSD actually:
  http://lists.samba.org/pipermail/rsync/2000-July/002491.html
 
 I don't use windows myself.  I just recall that one of the
 windows filesystem has timestamps in the milliseconds.
 
  Further examination shows that it is
  already hacked in options.c for win32:  
  | #ifdef _WIN32
  | int modify_window=2;
  | #else
  | int modify_window=0;
  | #endif
  perhaps something just needs to be added to the #ifdef.
  
  Ohhh... I'd add CygWin to the #ifdef too, then.
  Grr, I *never* remember what does cygwin define and what not ^_^
  
  Let me check... this message
  http://sources.redhat.com/ml/gdb-patches/2001-04/msg00039.html
  suggests that to include djgpp, mingw and cygwin the full ifdef should be:
  
  #if defined(__GO32__) || defined(_WIN32) || defined (__CYGWIN__)
  
  Should I add that to the current 2.5.5 cygwin port (creating a -3 release) 
  or 2.5.6 will be short in time?
 
 Depends on how urgent this is.  Given you have a workaround
 i'd get it in cvs and wait.  Assuming the 2.5.6 really does
 happen soon.
 
 Please try a modify-window of 1.  Maybe my math is off
 somehow but i don't think it needs to be greater than 1.
 Take a look at the code in util.c.  If you truncate to 2
 second resolution and compare against 1 second resolution
 the difference should never be greater than 1 second.
 It won't make a difference very often but i'd still feel
 better making the window as small as possible.
 
 -- 
 
   J.W. SchultzPegasystems Technologies
   email address:  [EMAIL PROTECTED]
 
   Remember Cernan and Schmitt
 -- 
 To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
 Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: Long-pending patch for Stratus VOS build

2003-01-10 Thread Dave Dykstra
On Fri, Jan 10, 2003 at 11:33:00AM -0800, Wayne Davison wrote:
 On Fri, Jan 10, 2003 at 01:23:45PM -0500, Green, Paul wrote:
  The following patch still applies cleanly to the current cvs copy of rsync.
 
 Or did before the most recent Makefile.in changes.  It's easy to merge
 this one problem, though.
 
  Does anyone object to having these changes applied now, during the
  pre-release phase?
 
 Here are my comments on the changes:
 
  + The Makefile.in changes look very safe and needed.
 
  + The install-sh change to the dsttmp value looks good.
 
  ? I have a question about the portability of the u_FOO - uFOO_t
changes.  The former is the BSD syntax for the unsigned FOO typedefs,
and the latter is what, POSIX?  The changes work on Linux, at least.
Perhaps we should just make these changes and try it out on the
compile farm.
 
  + The inet_pton changes look right to make the code consistent.  The
only possible glitch might be a system that has a prototype for
inet_pton() but not the library code -- if the prototype conflicts,
the compile would fail (there are ways to work around this, but let's
worry about that if we actually find some weird system with this
problem).
 
 I'd be glad to check in the + changes now if Dave thinks now is a good
 time.


Ok, go ahead.  I think it's ok to try out the ? patch on the compile
farm too.  I know that Cygwin also uses $(EXEEXT), but apparently its
make has been more thoroughly modified to handle things automatically
so the Makefile doesn't have to use it as much.

I'm looking at the --copy-unsafe-links problem and it looks very
badly broken.  I was the one who put it in the first place based on a
contributed patch, and now I'm not sure it ever worked.  It's going to
take a while to trace through the history to figure out if it worked once
and if so when it broke.  The fix is not likely to impact much, though,
so I don't think 2.5.6pre1 will need to be held up because of it because
it can go in later.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: Windows and default --modify-window=0

2003-01-09 Thread Dave Dykstra
On Thu, Jan 09, 2003 at 01:02:29PM +0100, Lapo Luchini wrote:
 Uhm... rsync developers: what do you think about changing the default 
 modify-window in the Cygwin platform?
 It seems to me that many problems people have are just because of the 
 granularity of timestamp.
 Maybe changing the default could be the best choice. That would be 
 documented in the man and in my release announcement on 
 cygwni-annouce, of course.
 
 Just an idea, maybe it's bad 0=)


As I understand it, --modify-window=2 is needed whenever you're copying
between a Windows filesystem (maybe only specific types, does anybody
know??) and a Unix filesystem.  Just because you're running on Cygwin,
that doesn't necessarily mean you're doing that although the risk of a
file modification time legitimately being off by a few seconds is pretty
low in the cases where it isn't really needed.  (Conversely, on Linux
you may be accessing a Windows filesystem directly but I realize you're
not talking about that situation).

The main problem is that if Cygwin always added the --modify-window=2
option to parameters it passes to the remote side it would fail if it were
talking to a release of rsync that is too old to recognize the option.
If it could be done on the cygwin side only, without passing the parameter
to the remote side, it might not be so bad.  I guess that would mean it
would work only when receiving files from a Unix system and not when
sending to them.  The surprise factor on that may be too much to make
it worthwhile.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: Windows and default --modify-window=0

2003-01-09 Thread Dave Dykstra
On Thu, Jan 09, 2003 at 05:52:43PM +0100, Lapo Luchini wrote:
 Dave Dykstra wrote:
 
 it would fail if it were
 talking to a release of rsync that is too old to recognize the 
 option[...].  The surprise factor on that may be too much to make it 
 worthwhile.
  
 
 I agree. I didn't think about that.
 But, given the resilience to upgrade many people have, you sure are right.


It could perhaps key off the remote protocol version number.  The version
number is not as a rule incremented when new options are added, but it
could check for whatever version the protocol was incremented to after
the option was added, and only add the option then.  No, wait, that won't
work because the protocol version number isn't exchanged until after the
option parameters are passed.  The only way to do it would be to modify
the protocol to include the info instead of passing a parameter.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



child exit code rsync bug patch finally submitted

2003-01-09 Thread Dave Dykstra
I am preparing for making 2.5.6pre1 today and decided to submit David
Staples patch with a little rework.  Here's the version I submitted.

- Dave Dykstra


--- main.c~ Thu Aug  1 15:46:59 2002
+++ main.c  Thu Jan  9 12:43:55 2003
@@ -26,6 +26,16 @@
 extern struct stats stats;
 extern int verbose;
 
+/* there's probably never more than at most 2 outstanding child processes,
+ * but set it higher just in case.
+ */
+#define MAXCHILDPROCS 5
+
+struct pid_status {
+   pid_t pid;
+   int   status;
+} pid_stat_table[MAXCHILDPROCS];
+
 static void show_malloc_stats(void);
 
 /
@@ -33,11 +43,27 @@
 /
 void wait_process(pid_t pid, int *status)
 {
-   while (waitpid(pid, status, WNOHANG) == 0) {
+   pid_t waited_pid;
+   int cnt;
+
+   while ((waited_pid = waitpid(pid, status, WNOHANG)) == 0) {
msleep(20);
io_flush();
}
 
+   if ((waited_pid == -1)  (errno == ECHILD)) {
+   /* status of requested child no longer available.
+* check to see if it was processed by the sigchld_handler.
+*/
+   for (cnt = 0;  cnt  MAXCHILDPROCS; cnt++) {
+   if (pid == pid_stat_table[cnt].pid) {
+   *status = pid_stat_table[cnt].status;
+   pid_stat_table[cnt].pid = 0;
+   break;
+   }
+   }
+   }
+
 /* TODO: If the child exited on a signal, then log an
  * appropriate error message.  Perhaps we should also accept a
  * message describing the purpose of the child.  Also indicate
@@ -848,7 +874,24 @@
 
 static RETSIGTYPE sigchld_handler(int UNUSED(val)) {
 #ifdef WNOHANG
-   while (waitpid(-1, NULL, WNOHANG)  0) ;
+   int cnt, status;
+   pid_t pid;
+   /* An empty waitpid() loop was put here by Tridge and we could never
+* get him to explain why he put it in, so rather than taking it 
+* out we're instead saving the child exit statuses for later use.
+* The waitpid() loop presumably eliminates all possibility of leaving
+* zombie children, maybe that's why he did it.
+*/
+   while ((pid = waitpid(-1, status, WNOHANG))  0) {
+/* save the child's exit status */
+for (cnt = 0; cnt  MAXCHILDPROCS; cnt++) {
+ if (pid_stat_table[cnt].pid == 0) {
+  pid_stat_table[cnt].pid = pid;
+  pid_stat_table[cnt].status = status;
+  break;
+ }
+}
+   }
 #endif
 }
 



On Tue, Sep 03, 2002 at 06:29:03PM -0700, jw schultz wrote:
 Subject: Re: [patch] for rsync
 On Tue, Sep 03, 2002 at 05:00:02PM -0700, [EMAIL PROTECTED] wrote:
  
  JW,
  
  I pushed everything to a LINUX box and did the diff again this time with the
  -bur option.
  It does look significantly different.  I think I've got below what you are
  looking for, so I am
  resending it to [EMAIL PROTECTED]
  
  I'm including two additional people as they have asked for the patch information
  as well.
  The diff was performed on main.c (2.5.5 version from rsync.samba.org).  I hope
  everyone
  knows that I'm my purpose in providing this information, is so that everyone can
  critique
  my code (to make it better) and we can eliminate this problem for everyone.
  
  Dave
 
 Thank you Dave.  I've been abusing/educating him a bit
 offline so be nice everybody.  He has not only diagnosed the
 bug independantly but has come up with a direction to go
 toward a permanent solution.
 
 Aside from my dislike of a fixed size array
 my comments are below.
 
 
  -
  
  --- main.c.orig Tue Sep  3 16:38:23 2002
  +++ main.c Tue Sep  3 16:41:08 2002
  @@ -27,6 +27,11 @@
  
   extern int verbose;
  
  +struct pid_status {
  +pid_t pid;
  +int   status;
  + } pid_stat_table[10];
 
 The use of a literal here is bad news.
 Should be a #define CHILD_TABLE_SIZE or whatever.
 Then macro can be used in the loops.
 
 Question.  Is 10 a good size?  Should it be more?
 I've not dug into the child spawning of rsync.
 
  +
   static void show_malloc_stats(void);
  
   /
  @@ -34,9 +39,27 @@
   /
   void wait_process(pid_t pid, int *status)
   {
  -while (waitpid(pid, status, WNOHANG) == 0) {
  +pid_t waited_pid;
  +int   cnt;
  +
  +while ( 1 ) {
  + waited_pid = waitpid(pid, status, WNOHANG);
  + if ( waited_pid == 0 ) {
msleep(20);
io_flush

Re: --exclude and --delete-exclude problem

2003-01-09 Thread Dave Dykstra
On Thu, Jan 09, 2003 at 01:20:08AM -0400, Carlos Molina wrote:
 Hi.
 
 I'm trying to accomplish the following ;
 
 1) First, I did a rsync for a file, like this
 /usr/bin/rsync -lptgoD --delete --rsh=/usr/bin/ssh -R --delete-excluded 
 -vv [EMAIL PROTECTED]:/usr/local/src/* /tmp/localhost/daily.0

Remove the '*'.  In order for rsync to delete a file, it needs to deal
with the directory that the files are in and the '*' tells it to only
deal with the files in the directory.

- Dave Dykstra
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: IPv6 hosts allow|deny

2003-01-09 Thread Dave Dykstra
I went ahead and submitted Hideaki's patch pretty much as is.  I took off
the ifdef around the memset at the beginning of client_sockaddr() because
I figured it wouldn't hurt to do anywhere.  Besides, it was checking
for a specific operating system (__linux__) which we like to avoid and
because there was a typo in the check for HAVE_SOCKADDR_IN6_SCOPE_ID
(it was HAVE_SOCKADDR_IN6_SOCPE_ID).  I noticed that there was code in
access.c that was accessing the same sockaddr_in6 sin6_scope_id field that
was protected by ifdef in clientname.c so I put the same ifdef around
the code in access.c.  I also updated the rsyncd.conf.yo documentation
to allow for IP address of the IPv6 form.

I have not tested this, I hope it works.

- Dave


On Wed, Jan 08, 2003 at 08:24:13AM -0600, Dave Dykstra wrote:
 On Wed, Jan 08, 2003 at 12:25:15AM +0100, Bert Vermeulen wrote:
  On Tue, 7 Jan 2003, Dave Dykstra wrote:
  
   Even though rsync maintenance isn't as bad as wget's, the maintainers
   are all VERY part time so that is a big part of the problem.  Most of us
   don't have ipv6 systems to test things on.  Can you vouch for the quality
   of the patch?  I was able to get it with
  
   wget --passive 
ftp://ftp.linux-ipv6.org/pub/usagi/misc/rsync-2_5_5-v6auth-20021016.patch.gz
  
   and it looks quite extensive.
  
  Yes, it's very extensive, and there's some code in there that's pretty
  advanced.
  
  The fact of the matter is, I made my own patch for this functionality first,
  and only then noticed Hideaki's patch -- and his, while not as readable to
  the casual observer, seems like better code. He is rewriting the linux
  kernel IPv6 stack, so I'd expect his rsync patch to be decent :-)
  
  Nevertheless, feel free to take a look at my patch:
  http://biot.com/patches/rsync-ipv6-acl.patch
  
  So I haven't used Hideaki's patch, but I use mine, and it's good. Please do
  apply one of them however.
 
 
 Since you're more familiar with that area of the code than any of the
 rsync maintainers and you think his patch is better, please test out his
 patch and look it over to see if you think there should be any changes.
 In particular, I see that yours includes documentation changes and his
 doesn't.
 
 - Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



working on a 2.5.6pre1 release

2003-01-09 Thread Dave Dykstra
I'm working on trying to get rsync 2.5.6pre1 available for people to
test more widely.  I'm out of time for today, and I'm stuck on a problem
that some machines on build.samba.org are showing on the 'chgrp' test.
I can reproduce this on my home redhat 7.3 system too.  It appears to be a
timing problem because when I do strace -F -f on it the problem goes away.
Everything seems to go through normally but then it exits with an exit
code of 12, I think because the child receiver process is terminated with
a SIGUSR2 which is signal 12 and because the bug that was preventing exit
codes from being properly reported from children has now been fixed.
It's very hard to debug because it is a timing problem and because it
happens after rprintf handling is already shut down in the child process.
I suspected that maybe the catching of the SIGUSR2 signal was not getting
inherited from its parent, but it doesn't help to re-set it in the child.
Nevertheless, I'm not sure whether or not the sigusr2_handler function
is getting called in the child.  I'd appreciate some help with this if
anybody else thinks they can figure it out.

- Dave Dykstra
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: IPv6 hosts allow|deny

2003-01-08 Thread Dave Dykstra
On Wed, Jan 08, 2003 at 12:25:15AM +0100, Bert Vermeulen wrote:
 On Tue, 7 Jan 2003, Dave Dykstra wrote:
 
  Even though rsync maintenance isn't as bad as wget's, the maintainers
  are all VERY part time so that is a big part of the problem.  Most of us
  don't have ipv6 systems to test things on.  Can you vouch for the quality
  of the patch?  I was able to get it with
 
  wget --passive 
ftp://ftp.linux-ipv6.org/pub/usagi/misc/rsync-2_5_5-v6auth-20021016.patch.gz
 
  and it looks quite extensive.
 
 Yes, it's very extensive, and there's some code in there that's pretty
 advanced.
 
 The fact of the matter is, I made my own patch for this functionality first,
 and only then noticed Hideaki's patch -- and his, while not as readable to
 the casual observer, seems like better code. He is rewriting the linux
 kernel IPv6 stack, so I'd expect his rsync patch to be decent :-)
 
 Nevertheless, feel free to take a look at my patch:
   http://biot.com/patches/rsync-ipv6-acl.patch
 
 So I haven't used Hideaki's patch, but I use mine, and it's good. Please do
 apply one of them however.


Since you're more familiar with that area of the code than any of the
rsync maintainers and you think his patch is better, please test out his
patch and look it over to see if you think there should be any changes.
In particular, I see that yours includes documentation changes and his
doesn't.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: Not preserving permissions really preserves some

2003-01-08 Thread Dave Dykstra
On Wed, Jan 08, 2003 at 03:08:38PM +1100, [EMAIL PROTECTED] wrote:
 
 
 Thanks for the reply.
 
 I did see this on the to-do list, and thought that it might
 be a preferable solution (just a bit more work).
 It would be messy to do the xargs stuff as I am running rsync
 over ssh with restricted commands based upon keys :-)
 
 Questions:
1. Is the proposed solution in the todo (--chmod) agreed?

Yes.


2. Is anyone actively working on it?

No.


3. If I wrote the patch to support the --chmod as outlined
   in the todo file, would someone apply it (assuming my
   code was okay ;-) ?

Yes, although probably not in the very next release, probably in 2.6.0,
unless the changes ended up being very isolated and we could clearly see
there was very little risk in breaking something else.


4. Should the patch be applied to the head, or should it
   be against the latest release version?

To the CVS head.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: directories that change into symlinks

2003-01-08 Thread Dave Dykstra
Yes, I knew --force made a difference when not using --delete (that's
very much like the example in the message I referred to), and I think
it's good that way and --force should not be the default without --delete.
I thought you were using --delete.  If --force makes a difference in any
case that --delete is used (and so far I have been unable to find any
such case), I think it would make sense for --force to be automatically
turned on by --delete.

- Dave

On Wed, Jan 08, 2003 at 07:39:22AM -0700, David Garamond wrote:
 Dave Dykstra wrote:
 Could you please post your full command line and say where in the directory
 structure the directories were replaced by symlinks?  I have not yet been
 able to come up with an example where --force makes a difference when
 using --delete and -r (or -a).  Refer to
 http://lists.samba.org/pipermail/rsync/2002-January/005971.html
 
 this is the 'old' directory (the one that will become a mirror of the 
 'new' dir). note that dir3 is still a directory and dir3 is not empty:
 ===
 [dave@builder old]$ ls -laR
 ...:
 total 0
 drwxrwxr-x4 dave   dave120 Jan  5 07:12 .
 drwxrwxr-x5 dave   dave120 Jan  5 07:13 ..
 lrwxrwxrwx1 dave   dave  4 Jan  5 07:12 dir1 - dir2
 drwxrwxr-x4 dave   dave 96 Jan  5 07:11 dir2
 drwxrwxr-x2 dave   dave 72 Jan  5 07:12 dir3
 
 .../dir2:
 total 0
 drwxrwxr-x4 dave   dave 96 Jan  5 07:11 .
 drwxrwxr-x4 dave   dave120 Jan  5 07:12 ..
 drwxrwxr-x2 dave   dave 72 Jan  5 07:11 dir11
 drwxrwxr-x2 dave   dave 72 Jan  5 07:11 dir12
 
 .../dir2/dir11:
 total 0
 drwxrwxr-x2 dave   dave 72 Jan  5 07:11 .
 drwxrwxr-x4 dave   dave 96 Jan  5 07:11 ..
 -rw-rw-r--1 dave   dave  0 Jan  5 07:11 file1
 
 .../dir2/dir12:
 total 0
 drwxrwxr-x2 dave   dave 72 Jan  5 07:11 .
 drwxrwxr-x4 dave   dave 96 Jan  5 07:11 ..
 -rw-rw-r--1 dave   dave  0 Jan  5 07:11 file2
 
 .../dir3:
 total 0
 drwxrwxr-x2 dave   dave 72 Jan  5 07:12 .
 drwxrwxr-x4 dave   dave120 Jan  5 07:12 ..
 -rw-rw-r--1 dave   dave  0 Jan  5 07:12 file3
 ===
 
 this is the 'new' directory (the one that will be mirrored by the 
 'old'). note that 'dir3' is now a symlink instead of a dir.
 
 ===
 [dave@builder new]$ ls -laR
 ...:
 total 0
 drwxrwxr-x4 dave   dave120 Jan  5 07:13 .
 drwxrwxr-x5 dave   dave120 Jan  5 07:13 ..
 drwxrwxr-x4 dave   dave 96 Jan  5 07:11 dir1
 drwxrwxr-x2 dave   dave 48 Jan  5 07:11 dir2
 lrwxrwxrwx1 dave   dave  4 Jan  5 07:11 dir3 - dir1
 
 .../dir1:
 total 0
 drwxrwxr-x4 dave   dave 96 Jan  5 07:11 .
 drwxrwxr-x4 dave   dave120 Jan  5 07:13 ..
 drwxrwxr-x2 dave   dave 72 Jan  5 07:11 dir11
 drwxrwxr-x2 dave   dave 72 Jan  5 07:11 dir12
 
 .../dir1/dir11:
 total 0
 drwxrwxr-x2 dave   dave 72 Jan  5 07:11 .
 drwxrwxr-x4 dave   dave 96 Jan  5 07:11 ..
 -rw-rw-r--1 dave   dave  0 Jan  5 07:11 file1
 
 .../dir1/dir12:
 total 0
 drwxrwxr-x2 dave   dave 72 Jan  5 07:11 .
 drwxrwxr-x4 dave   dave 96 Jan  5 07:11 ..
 -rw-rw-r--1 dave   dave  0 Jan  5 07:11 file2
 
 .../dir2:
 total 0
 drwxrwxr-x2 dave   dave 48 Jan  5 07:11 .
 drwxrwxr-x4 dave   dave120 Jan  5 07:13 ..
 ===
 
 now compare these two commands. i find that the second one works.
 
 $ rsync -avz new/* old/
 $ rsync -avz --force new/* old/
 
 -- 
 dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: long directory name problem

2003-01-08 Thread Dave Dykstra
I've never heard of a name length limit.  One operating system(s) and rsync
version(s) are you using?

- Dave

On Wed, Jan 08, 2003 at 07:32:19AM -0800, elijah reyen wrote:
 Does rsync has directory name length limit?  I was trying to rsync a directory that 
has the name length of 34 characters and getting permission denied error.
 
 $ rsync -avz --delete --force -e ssh /usr/local/apache/htdocs 
192.168.0.2:/usr/local/apache
 building file list ... 
 readlink htdocs/CorporateLegalTimesEDITORIAL_files/cBug_blueXs.gif: Permission denied
 readlink httpd/html/CorporateLegalTimesEDITORIAL_files/title_a.gif: Permission denied
 
 When I rename the directory to about half of its original length, the error goes 
away.  Any advise are appreciated.
 --
 ER

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: --copy-unsafe-links, links preserved in source tree or local directory?

2003-01-07 Thread Dave Dykstra
That indeed is not right, and it appears that other people have noticed the
problem before but nobody has come up with a fix.  The problem must be related
to the unsafe_symlink() function in util.c.  Martin noticed a strange thing in
the code (see http://lists.samba.org/pipermail/rsync/2002-April/006759.html)
but I just tried his suggestion and it didn't help with your example.  I'm
sure I could figure it out if I spent more time on it but I've got other
priorities right now.  Try fiddling with it to see if you can come up with
a better solution.  It might help for you to build the t_unsafe test program
that's in the Makefile but not built by default.  I see Martin has also has
a test for the option in testsuite/unsafe-links.test, and it would be good if
that could be expanded to test for this case.

- Dave


On Thu, Jan 02, 2003 at 03:02:46PM -0800, Seth Olsen wrote:
 
 Hi,
 
 I'm a bit confused about what the '--copy-unsafe-links' option 
 considers the 'source tree'.  In the man page it says that for links
 pointing outside the 'source tree' the file will be copied, but when I
 try this, for all links pointing outside of the local directory the
 files are copied even though the links point to files that are in the
 directory tree that is being rsynced.  I couldn't find anything definitive
 on the web either. See the example below. 
 
 Is this the expected behaviour? Is the 'source tree' just the 
 current directory? If so, it would be nice to have an option where 
 only links pointing outside of the 'parent source tree' 
 ('Test' in the example below) would be copied as files.
 
 rsyc-2.5.4 on RedHat-7.3 i386, XFS filesystem
 
 Thanks,
 
 Seth
 [EMAIL PROTECTED]
 
 example:
 
 chump0:~ ll Test/*
 Test/Q1:
 total 0
 -rw-rw-r--1 olsenolsen   0 Jan  2 14:05 q1_1
 -rw-rw-r--1 olsenolsen   0 Jan  2 14:05 q1_2
 lrwxrwxrwx1 olsenolsen   4 Jan  2 14:06 q1_3 - q1_2
 lrwxrwxrwx1 olsenolsen  24 Jan  2 14:09 q1_4 - 
/home/olsen/Test/Q2/q2_1 
 lrwxrwxrwx1 olsenolsen  15 Jan  2 14:42 q1_5 - ../../test/qw.F
 
 Test/Q2:
 total 0
 -rw-rw-r--1 olsenolsen   0 Jan  2 14:06 q2_1
 -rw-rw-r--1 olsenolsen   0 Jan  2 14:06 q2_2
 lrwxrwxrwx1 olsenolsen   4 Jan  2 14:06 q2_3 - q2_1
 lrwxrwxrwx1 olsenolsen  10 Jan  2 14:06 q2_4 - ../Q1/q1_1 
 lrwxrwxrwx1 olsenolsen  10 Jan  2 14:06 q2_5 - ../Q1/q1_3
 
 chump0:~ rsync -rl --copy-unsafe-links Test Test2
 
 chump0:~ ll Test2/Test/*
 Test2/Test/Q1:
 total 4.0k
 -rw-rw-r--1 olsenolsen   0 Jan  2 14:48 q1_1
 -rw-rw-r--1 olsenolsen   0 Jan  2 14:48 q1_2
 lrwxrwxrwx1 olsenolsen   4 Jan  2 14:48 q1_3 - q1_2
 -rw-rw-r--1 olsenolsen   0 Jan  2 14:48 q1_4
 -rw-rw-r--1 olsenolsen 275 Jan  2 14:48 q1_5
 
 Test2/Test/Q2:
 total 0
 -rw-rw-r--1 olsenolsen   0 Jan  2 14:48 q2_1
 -rw-rw-r--1 olsenolsen   0 Jan  2 14:48 q2_2
 lrwxrwxrwx1 olsenolsen   4 Jan  2 14:48 q2_3 - q2_1
 -rw-rw-r--1 olsenolsen   0 Jan  2 14:48 q2_4
 -rw-rw-r--1 olsenolsen   0 Jan  2 14:48 q2_5
 -- 
 To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
 Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: rsync feature suggestion

2003-01-07 Thread Dave Dykstra
What Hadmut wants is the oft-requested and discussed files-from option
that I once offered to write but haven't been able to get to.  Andy Schor
in http://lists.samba.org/pipermail/rsync/2001-November/005272.html posted
a patch for something similar but it only worked when the sender was on the
local machine and not when it was remote (among other issues).  I don't
believe you've posted your patch, Justin; does your files-from directly
contain the list of files to send and skip the recursive traversal?  If so,
I don't see the point of having rsync have the extra regex options you
mention because those could all be done by external greps that pre-process
the file list.

- Dave


On Fri, Jan 03, 2003 at 11:51:05AM -0600, Justin Banks wrote:
 Max Bowsher wrote
  Hadmut Danisch wrote:
   I'd like to suggest a new feature to rsync.
  
   I am mirroring a debian archive, but unfortunately,
   debian mixes all files of several distributions in a
   subtree /pool. There is no way to select only the files
   of a certain distribution through a simple exclude/include
   expression.
  
   There is a tool called debmirror, which first downloads
   the distribution index files, extracts all the filenames/paths
   of the files needed and then calls rsync for every single file.
   Thats certainly not useful, especially since rsync shows the
   servers motd for every single file.
  
  I was about to suggest:
  $ rsync --include-from=list-file --exclude=\*
  but of course that will exclude the parent directories of files you want,
  causing them to be ignored.
  
  This might work:
  $ rsync --include-from=list-file --include=\*\*/ --exclude=\*
  
  although it will mirror the entire directory structure (but not unspecified
  files).
  
  Probably, rsync should be taught that: If I explicitly include a file, look
  for it explicitly, even if I've excluded a parent directory.
 
 Not too long ago, I modified/mangled rsync to do
 
 rsync --files-from /some/file --include-regexes /some/regular/expressions \
   --exclude-regexes /some/regular/expressions
 
 
 such that all the files in /some/file would be sent iff they matched the
 posix regexes in --include-regexes and didn't match the ones in 
 --exclude-regexes (if present).
 
 I don't have a wide variety of platforms to test it on, but it worked okay
 on linux, solaris, and irix.
 
 -justinb 
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: rsync windows - unix still hanging :(

2003-01-07 Thread Dave Dykstra
I've never seen anybody suggest any other value for --modify-window than 2.
Does 90 really work better than 2?  My understanding is that PC-style
filesystems do not have 1 second timestamp granularity and so round the
times up or down slightly.

- Dave Dykstra

On Fri, Jan 03, 2003 at 02:35:42PM -0800, Timothy Burt wrote:
 
 Glory Halelujah!
 
 I think I got it.
 
 It must have something to do with the file modification times and perhaps 
 the number of places of accuracy, perhaps in fractions of a second.
 
 When I add --modify-window=90 as a parameter.  Then it works as it should.
 
 At least it does, for me now.  You might try a smaller number (of seconds) 
 but for me.  90 seconds is close enough!
 
 On Fri, 3 Jan 2003, Mike Rubel wrote:
 
  
  On Fri, 3 Jan 2003, Timothy Burt wrote:
   I am seeing similar symptoms.
  
  Hi Timothy,
  
  I'm trying to find out more about this.  May I ask, are the files you see
  sent even when they haven't changed regular files, or directories, or
  both?
  
  Mike
  
  
 
 -- 
 
 Timothy Burt
 General Manager
 Arbor Group LLC
 Los Angeles, Calif. USA
 
 -- 
 To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
 Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: Not preserving permissions really preserves some

2003-01-07 Thread Dave Dykstra
I would object to that because it would change existing functionality
(emulating GNU cp) and isn't general enough.  We've discussed a possible
general manipulation of permissions; see
http://lists.samba.org/pipermail/rsync/2002-March/006477.html

Alternatively, you could process the output of the -v option and run
xargs chmod on all files that have been touched.

- Dave Dykstra

On Thu, Jan 02, 2003 at 12:47:18PM +1100, [EMAIL PROTECTED] wrote:
 
 This appears to be a feature :-)
 
 I'm running rsync 2.5.5 on Solaris 8, and testing transferring
 files between two boxes with permission preservation turned *off*.
 
 I have three files:
 
 700 with permissions 700
 770 with permissions 770
 777 with permissions 770
 
 The umask for the target system is set to: 007
 [I have checked this via prints from the rsync code]
 
 When the files get transferred, the permissions are:
 
 700 - permission 700
 770 - permission 770
 777 - permission 770
 
 This retains the permissions from the group part of the file 700.
 That is, even though my umask wants the group set, it doesn't
 get set.
 
 Why this is a problem for me:
 
 I don't trust the source system to get the permissions correct
 (it's a windows box running cygwin), and I want to force a set
 of permissions so the process using the synched directory will
 be able to access it (through group permissions).
 
 Suggested solution:
 
 I don't think changing the way permissions are created would be
 that backward-compatible, so maybe a new option?
 
 Create an option: --ignore-permissions
 
 in receive_file_entry(), in filelist.c
 
 
 == code block 
 if (!preserve_perms) {
 extern int orig_umask;
 /* set an appropriate set of permissions based on original
permissions and umask. This emulates what GNU cp does */
 
file-mode = ~orig_umask;  -- change this to line below
 
file-mode = ((ignore_perms?0:0666) | file-mode)  ~orig_umask;
 }
 
 I used 0666 for the mask to pick up the execute bit (otherwise might
 screw up creating directories we can go down).
 
 
 I would be interested in any feedback on this problem from the maintainers.
 If there is any agreement on how it might be addressed and included in a
 new version, then I am happy to create all of the patch files and test it
 for submission to a maintainer.
 
 If any replies to the list could also cc my reply address, that would
 be appreciated.
 
 Regards,
 - Tim
 
 
 
 
 
 The information contained in this e-mail communication may be confidential.
 You should only read, disclose, re-transmit,copy, distribute, act in
 reliance on or commercialise the information if you are authorised to do
 so. If you are not the intended recipient of this e-mail communication,
 please immediately notify us by e-mail to [EMAIL PROTECTED], or reply
 by e-mail direct to the sender and then destroy any electronic and paper
 copy of this message.
 
 Any views expressed in this e-mail communication are those of the
 individual sender, except where the sender specifically states them to be
 the views of a member of the National Australia Bank Group of companies.
 
 The National Australia Bank Group of companies does not represent, warrant
 or guarantee that the integrity of this communication has been maintained
 nor that the communication is free of errors, virus or interference.
 
 -- 
 To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
 Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: directories that change into symlinks

2003-01-07 Thread Dave Dykstra
On Mon, Jan 06, 2003 at 08:36:35AM -0700, David Garamond wrote:
 John Van Essen wrote:
 Dave,
 
 What you need is the --force option.  It's not obvious from the
 all-too-generic name, but that will do the trick.
 
 To the rsync maintainers - this is somewhat of an FAQ.  Perhaps
 the error message could also say '(see the --force option)' to
 help users discover the solution more easily.
 
 thanks john, that solved the problem. i should've perhaps looked at the 
 manpage twice. the manpage (2.5.5) already states this quite clearly:
 
  --force   force deletion of directories even if not empty
 
 however, i personally think --force should be the default as it is more 
 common, or shouldn't it?


Could you please post your full command line and say where in the directory
structure the directories were replaced by symlinks?  I have not yet been
able to come up with an example where --force makes a difference when
using --delete and -r (or -a).  Refer to
http://lists.samba.org/pipermail/rsync/2002-January/005971.html

- Dave Dykstra
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: IPv6 hosts allow|deny

2003-01-07 Thread Dave Dykstra
Even though rsync maintenance isn't as bad as wget's, the maintainers
are all VERY part time so that is a big part of the problem.  Most of us
don't have ipv6 systems to test things on.  Can you vouch for the quality
of the patch?  I was able to get it with

wget --passive 
ftp://ftp.linux-ipv6.org/pub/usagi/misc/rsync-2_5_5-v6auth-20021016.patch.gz

and it looks quite extensive.

- Dave

On Sun, Jan 05, 2003 at 03:53:13AM +0100, Bert Vermeulen wrote:
 Hi all,
 
 Does anyone know what the status is of Hideaki Yoshifuji's IPv6 ACL patch?
 He's submitted this twice now, and that was 3 months ago. Judging by the
 comments made when this was submitted, it's apparently not understood how
 important this is.
 
 - if you IPv6 on your rsync server, you can not secure it with hosts
   allow.
 
 - if you enable IPv6 on an rsync server that's secured with hosts deny to
   keep certain hosts out, well, they can get in again -- if they also have
   IPv6.
 
 Please, can this patch be merged into the next release? This is very
 important! It is NOT new functionality, it's a security fix.
 
 
 Bert Vermeulen
 [EMAIL PROTECTED]
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: restricting rsync over ssh on the server side.

2003-01-07 Thread Dave Dykstra
In the development version of rsync now in CVS, ssh and daemon mode can
be used together by using '-e ssh' along with '::'.  That is probably
just what Rob needs, please check it out/test it.  The documentation has
been updated to describe putting a ssh wrapper key to to restrict rsync
operations to only those defined by rsyncd.conf.

- Dave Dykstra


On Sun, Jan 05, 2003 at 01:07:30PM -0500, Aaron Morris wrote:
 I do not think you can use it with ssh, but if you use rsync in rsync 
 mode (::) instead of just an interface to rsh (:), you can limit the 
 directories where you can transfer files (using modules).  This involves 
 setting up the rsync daemon on the server side.  The rsync daemon has 
 the ability to limit connections, chroot itself, prevent the use of 
 options (such as delete), use it's own authentication, setup 
 includes/excludes on the server side, and use IP based ACLs (outside of 
 tcpwrappers).  See:  `man rsyncd.conf`
 
 I only mention this because I do not believe most people even realize 
 there is this other mode to rsync.  I tried describing it to a co-worker 
 who uses rsync regularly, but he kind of just stared at me blankly.
 
 Rob Browning wrote:
 I was wondering if it's possible to restrict rsync in various ways on
 the server side when it is invoked via ssh.  Two restrictions I had in
 mind are disallowing deletes and/or restricting all actions to a
 particular subdirectory.  I was hoping to be able to do this without
 having to be root (for a chroot) or having to set up special sshd
 server instances/chroots.
 
 If there's not already a way to do this, one possibility I had thought
 of is a ssh key command= wrapper, so that you could generate an ssh
 key like this:
 
   command=rsync-ssh-wrapper --root=/home/foo/bar --disable-delete,...
 
 and then when invoked rsync-ssh-wrapper would then look at
 SSH_ORIGINAL_COMMAND to see the actual incoming request (presuming
 there were any relevant options there; are rsync --server invocations
 documented anywhere?), and combine that with the wrapper options to
 decide how to invoke rsync --server.  Of course this approach presumes
 that rsync --server would support suitable arguments.
 
 Is there interest in such a facility?  It seems like something similar
 might be useful for sftp and scp as well, but I haven't managed to
 think of a way to implement a common solution.  Also, I could imagine
 that this solution for rsync might be somewhat difficult to implement
 (perhaps complicated by symlinks, etc.), but it's the best thing I've
 thought of so far.
 
 Thanks
 
 
 -- 
 Aaron W Morris
 decep
 PGP Key ID:  259978D1
 
 
 -- 
 To unsubscribe or change options: 
 http://lists.samba.org/mailman/listinfo/rsync
 Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: filelist calculation algoritm

2003-01-07 Thread Dave Dykstra
We've been calling this option --files-from rather than --file-list,
to be like the GNU tar option.

On Sun, Jan 05, 2003 at 09:55:50AM -0800, Wayne Davison wrote:
 On Sat, Jan 04, 2003 at 05:03:02PM -0800, jw schultz wrote:
  that would produce destloc/srcdir/
  when you might want a copy of srcdir at destloc instead of
  in destloc.
 
 Ah yes, I _was_ missing something.  However, I still don't think we need
 to clutter rsync with two types of --file-list options.  This is already
 something that people have to deal with when using the --relative option:
 how to generate a file list that contains just the path information that
 we need to be significant.  I think that the removal of the undesired
 prefixes should happen before the list gets to rsync rather than having
 rsync do it (in your example the user would just chdir into srcdir and
 do the find relative to '.').

I agree, there should only be one option.


...
 FYI, the old rsync release that had a type of file-list functionality
 was using a specialized include/exclude list.  I believe that rsync
 still walked the entire directory tree on both sides, and applied the
 includes using a slightly different algorithm than the default (one that
 did not require parent directories to be mentioned to get down to all
 the specified files).  I think that it would be nice to avoid the
 directory-tree traversal, so I don't think we want to go this route.
 However, this is another potential implementation method (and one that
 would result in a syntax that is like what you suggested:  one that uses
 a single source dir on the command-line and doesn't require the use of
 the --relative option).

No, the include/exclude optimization completely bypassed the recursive
traversal.  It kicked in whenever the list ended with --exclude '*'
and there were otherwise only includes with no wildcards.



On Sun, Jan 05, 2003 at 01:18:06PM -0800, jw schultz wrote:
 On Sun, Jan 05, 2003 at 12:44:32PM -0800, Wayne Davison wrote:
  On Sun, Jan 05, 2003 at 11:55:22AM -0800, jw schultz wrote:
   The first problem is this would flatten things unless you used
   relative and forced the user's CWD.  That would cause considerable
   confusion.
  
  Really?  This is exactly how rsync works now with multiple file names on
  the command-line, so I don't see this as being any more confusing than
  what we already have.  The rule would be you can specify the files on
  the command-line or on stdin (if you use '-' as the only source file).
  Since all names are treated in the same way regardless of where they
  were specified, everything works the same as it did before, only more
  names are now supported per invocation.  I'm thinking that this way is
  more flexible since it allows someone to flatten things if that's what
  they really want to do.
 
 And the effect of that is that users wind looping to produce
 scores of rsync sessions to transfer a single list.
 
   Secondly, how would you do it when the source location is remote?
   Many of the users asking for this are doing pulls.
  
  I mentioned a protocol change that would send the extra file names to
  the other side after rsync starts up.  Currently the send_files()
  routine always sends names from the sending side to the receiving side.
  The new protocol would change that to always send names from the user
  side to the server side when this option was specified.  The user's
  command would look like this:
  
  rsync -avR remote:- /foo/bar
  
  The file list would be read from the local (user) side, of course.  The
  remote command being run by rsync would look like this:
  
  ssh remote rsync --server --sender -vlogDtprR . -
  
  The presence of the '-' as the source would tell us to slurp names
  instead of send them.
  
  Since the file list is exchanged in total before we do any real work, I
  think this change would actually be really easy to implement.
 
 How many levels down should we allow - to mean use this
 directory as cwd for list?
 
   rsync --relative remote::module/- dest/dir
 
 If it can only be remote:- then everything would have to
 be relative to the user's home directory.
 ../../usr/lib/somedir/somefile anyone?  And/or we
 allow absolute paths in the list.  So much for safe-links.
 
 In terms of implimentation i don't think we are that far
 apart.  As it stands now we walk the source list.  For each
 file/directory we check it against the pattern list prior to
 insertion.  At the time of insertion if recursion is turned on
 each directory gets a readdir and the contents get the same
 test and insert treatment.
 
 You are replacing the source list with stdin.  I'm basically
 saying that the list from stdin or from a file would be used
 instead of readdir.
 
 Both cases require a protocol bump to support sending the
 list for a pull.
 
 The discussion seems to be fruitful but i'd like to see more
 participants with other perspectives before i'd bookmark it
 as a TODO.


I think the use of - 

Re: am I missing something, or are permissions always preserved?

2003-01-07 Thread Dave Dykstra
That sounds reasonable to me that rsync shouldn't try to preserve those
extra bits without -p.  Try making a patch and seeing if that works.

I don't have Samba set up, but on a vfat filesystem on Linux rsync is
doing even worse because after the initial file creation it does a
fchmod(fd, 0600) which fails and it doesn't even copy files at all.

- Dave

On Wed, Jan 01, 2003 at 02:24:36PM -0800, Ben wrote:
 I'm not sure what POSIX calls for, but on both Linux and AIX (the two
 unix OSes I have access to) I can leave off the mode arg when calling
 open(2). Granted, doing so gives me a mode that normally isn't so good,
 but in my case this would be fine, as I have samba forcing modes to be
 what I want them to be.
 
 However, after looking into this more, it turns out that samba
 apparently causes chmod(2) to fail _only_ when trying to set modes
 S_ISUID, S_ISGID, or S_ISVTX. Trying to set these modes in open(2), or
 trying to set any other modes in open(2) or chmod(2), silently fails
 without raising an error.
 
 So while this might well be a samba problem, it could be avoided if
 rsync behaved as I would expect it to. The original files do indeed have
 S_ISGUID set on them, but I don't see why rsync should try to preserve
 this if I didn't tell it to preserve permissions.
 
 
 On Wed, 2003-01-01 at 07:13, Green, Paul wrote:
  Ben [mailto:[EMAIL PROTECTED]]
  
   I'm relatively new to rsync, but it seems to me that if there's an
   option to perserve permissions and you don't set it, then rsync
   shouldn't do anything with permissions.
  
  Sounds good, but recall how POSIX / Unix system calls work. To create a
  file, you have to specify what mode (permission) it should have.  There is
  no way to tell creat() or open() to do the default thing.
  
  I think a case could be made that this is a defect / incompatibility in the
  underlying file system layer that issues complaints about permissions.
  Before I attempted to code any kind of a fix, I'd sure want to read the
  specs of this layer and understand how the designers thought this problem
  should be approached.
  
  Thanks
  PG
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: Simulating rdist?

2003-01-01 Thread Dave Dykstra
See the BATCH MODE section in the rsync 2.5.5 man page.  It's stil pretty
new and experimental, so it may not do what you want, but currently it's the
best that rsync can do.

- Dave Dykstra

On Wed, Jan 01, 2003 at 04:39:06PM -, va_public [EMAIL PROTECTED] 
wrote:
 rsync is great for syncing 2 directory trees, but I want to maintain 
 a master source tree on one machine and copy that to multiple 
 machines. i.e. basically what rdist does
 
 The only way I can see of doing this with rsync is to have multiple 
 cron jobs
 
 0 * * * * rsync ... machine1:...
 0 * * * * rsync ... machine2:...
 and so on.
 
 Is there a more elegant/compact way of doing this?
 
 Thanks
 
 -- 
 To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
 Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: am I missing something, or are permissions always preserved?

2002-12-31 Thread Dave Dykstra
What do you mean, altered?  Do the destination files already exist?
It is supposed to preserve existing permissions on destination files 
when you don't use -p.

- Dave

On Mon, Dec 30, 2002 at 06:44:24PM -0800, Ben wrote:
 Hmmm... while that makes sense, that doesn't really help me in my
 situation, where permissions cannot be altered because of the network
 mount they are being written to.
 
 Does it make sense to impliment a don't touch permissions flag?
 
 On Tue, 2002-12-24 at 05:08, Dave Dykstra wrote:
  When preserve_perms is not set, rsync sets a default permission based on
  the original permissions and the umask.  A comment in flist.c says that is
  what GNU cp does, so that's why rsync does it.   Comments in generator.c
  and receiver.c indicate that if a file already exists and preserve_perms
  isn't set, the original permissions should be preserved.
  
  - Dave
  
  On Sat, Dec 21, 2002 at 12:30:55PM -0800, Ben wrote:
   They seem to be for me, even when I don't pass in the --perms flag. This
   is a problem because I'm rsyncing to a samba mount with fixed
   permissions.
   
   Looking at rsync.c (for version 2.5.5) starting at line 204, I see this
   code snippet:
   
   #ifdef HAVE_CHMOD
   if (!S_ISLNK(st-st_mode)) {
   if (st-st_mode != file-mode) {
   updated = 1;
   if (do_chmod(fname,file-mode) != 0) {
   rprintf(FERROR,failed to set permissions on %s : %s\n,
   fname,strerror(errno));
   return 0;
   }
   }
   }
   #endif
   
   I would have expected to see a test for the setting of perserve_perms,
   but it's not there. Is this a bug or is there there a reason
   perserve_perms isn't checked?
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: am I missing something, or are permissions always preserved?

2002-12-31 Thread Dave Dykstra
Then what would you expect it to do?  I'm guessing your only problem is
that the chmod is failing, and you would rather have it create files with
the final permissions in the first place; is that it?  I believe it is
done this way because of worries of potential security problems, where
temporary files might be accessible by more people than the original
file was.  Or is the issue that you rather have it always create file
permissions based only on umask?  In that case I don't know how it would
know whether or not to turn on the executable permission, among other
problems.  What exactly would your proposed flag do?

- Dave


On Tue, Dec 31, 2002 at 08:06:52AM -0800, Ben wrote:
 No, these are for new files. Existing files work perfectly, but, like
 you said before, for new files rsync creates the file then attempts to
 alter the permissions based on the origional permissions and umask. 
 
 On Tue, 2002-12-31 at 07:58, Dave Dykstra wrote:
  What do you mean, altered?  Do the destination files already exist?
  It is supposed to preserve existing permissions on destination files 
  when you don't use -p.
  
  - Dave
  
  On Mon, Dec 30, 2002 at 06:44:24PM -0800, Ben wrote:
   Hmmm... while that makes sense, that doesn't really help me in my
   situation, where permissions cannot be altered because of the network
   mount they are being written to.
   
   Does it make sense to impliment a don't touch permissions flag?
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: am I missing something, or are permissions always preserved?

2002-12-31 Thread Dave Dykstra
OK, now I understand.  I have seen those errors also when writing onto a
PC filesystem mounted by Linux.  Other programs like tar and cpio have
the same problem, so I've just ignored the messages.  Maybe everybody
else does the same.  Maybe a well thought out and documented patch for
a new flag would get put in.  It would be even better if it could be
detected automatically, but I'm not sure how to do it.  How do people
feel about just silently ignoring the error if -p wasn't specified?

- Dave

On Tue, Dec 31, 2002 at 01:31:07PM -0800, Ben wrote:
 Yeah, my problem is that the chmod fails, so then rsync complains not
 everything worked as expected. Because I'm writing files to a network
 mount with forced permissions, I would like a way for rsync to simply
 create the file and never try to chmod what it creates. Normally that
 would probably be a bad idea, but when writing to a share with forced
 permissions
 
 I'm actually a bit surprised nobody else has run into this.
 
 On Tue, 2002-12-31 at 13:20, Dave Dykstra wrote:
  Then what would you expect it to do?  I'm guessing your only problem is
  that the chmod is failing, and you would rather have it create files with
  the final permissions in the first place; is that it?  I believe it is
  done this way because of worries of potential security problems, where
  temporary files might be accessible by more people than the original
  file was.  Or is the issue that you rather have it always create file
  permissions based only on umask?  In that case I don't know how it would
  know whether or not to turn on the executable permission, among other
  problems.  What exactly would your proposed flag do?
  
  - Dave
  
  
  On Tue, Dec 31, 2002 at 08:06:52AM -0800, Ben wrote:
   No, these are for new files. Existing files work perfectly, but, like
   you said before, for new files rsync creates the file then attempts to
   alter the permissions based on the origional permissions and umask. 
   
   On Tue, 2002-12-31 at 07:58, Dave Dykstra wrote:
What do you mean, altered?  Do the destination files already exist?
It is supposed to preserve existing permissions on destination files 
when you don't use -p.

- Dave

On Mon, Dec 30, 2002 at 06:44:24PM -0800, Ben wrote:
 Hmmm... while that makes sense, that doesn't really help me in my
 situation, where permissions cannot be altered because of the network
 mount they are being written to.
 
 Does it make sense to impliment a don't touch permissions flag?
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: am I missing something, or are permissions always preserved?

2002-12-24 Thread Dave Dykstra
When preserve_perms is not set, rsync sets a default permission based on
the original permissions and the umask.  A comment in flist.c says that is
what GNU cp does, so that's why rsync does it.   Comments in generator.c
and receiver.c indicate that if a file already exists and preserve_perms
isn't set, the original permissions should be preserved.

- Dave

On Sat, Dec 21, 2002 at 12:30:55PM -0800, Ben wrote:
 They seem to be for me, even when I don't pass in the --perms flag. This
 is a problem because I'm rsyncing to a samba mount with fixed
 permissions.
 
 Looking at rsync.c (for version 2.5.5) starting at line 204, I see this
 code snippet:
 
 #ifdef HAVE_CHMOD
 if (!S_ISLNK(st-st_mode)) {
 if (st-st_mode != file-mode) {
 updated = 1;
 if (do_chmod(fname,file-mode) != 0) {
 rprintf(FERROR,failed to set permissions on %s : %s\n,
 fname,strerror(errno));
 return 0;
 }
 }
 }
 #endif
 
 I would have expected to see a test for the setting of perserve_perms,
 but it's not there. Is this a bug or is there there a reason
 perserve_perms isn't checked?
 
 -- 
 To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
 Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: deflate on token returned 0 (16384 bytes left)

2002-12-24 Thread Dave Dykstra
First, you should know that rsyncing a .gz file is not very worthwhile
because the rsync algorithm does not do a good job of efficiently
transferring them, unless you've got a modified gzip that adds a
--rsyncable option and you used the option.  The patch is in the 2.5.5
source patches/gzip-rsyncable.diff.

According to my email record, I once found a workaround by deleting the
|| tx_strm.avail_in != 0 on line 285 of token.c.  That change has not
been put into 2.5.5 or into a later development version because it was
unknown what the side effects were.  

- Dave Dykstra

On Mon, Dec 23, 2002 at 09:53:21AM -0500, John Murtari wrote:
 Hello All,
 
 I have searched for this error and found similiar errors, but
 everything seems to indicate that this should be fixed in the 2.5.5
 version. 
 
 rsync  version 2.5.5  protocol version 26
 
 I am running rsync with -axz, trying to sync up a large gz file :
 
 Source machine:
 
 -rw-r--r--1 root other175214792 Dec 22 00:17 
/export/tools2/local/etc/httpd/logs/access/access_log.12-22-02.gz
 
 Destination:
 -rw-r--r--1 root other103372258 Dec 19 00:18 access_log.12-22-02.gz
 
 
 Rsync is running with rsh to a remote machine running in daemon mode.
 
 
 The full command line is :
 /bin/nice rsync -R -l -H --quiet --stats -axz --delete --blocking-io /export/tools2 
drill::anvil 
 
 welcome to drill.thebook.com
 
 building file list ... done
 export/tools2/local/etc/httpd/logs/access/
 /export/tools2/local/etc/httpd/logs/access/access_log.12-22-02.gz
 deflate on token returned 0 (16384 bytes left)
 rsync error: error in rsync protocol data stream (code 12) at token.c(288)
 
 
 
 If the remote copy of the file is removed, rsync will run to
 completion, so it seems to be a problem perhaps with the -z option
 trying to update a .gz file???
 
 TIA!
 
 -- 
   John
 ___
 John Murtari  Software Workshop Inc.
 [EMAIL PROTECTED] 315.695.1301(x-211)TheBook.Com (TM)
 http://www.thebook.com/
 -- 
 To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
 Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: rsync 2.5.5 SCO Unixware patch

2002-12-24 Thread Dave Dykstra
On Mon, Dec 23, 2002 at 11:42:07PM -0800, Jos Backus wrote:
 On Mon, Dec 23, 2002 at 02:13:51PM -0800, Stephen Friedl wrote:
  The change in popt/popt.c is to work around a bug in the SCO UNIXWare 8
  compiler: it doesn't properly deal with alloca() being called from the
  middle of another function call. The code dumps badly on this platform
  with all modes (debug, regular, optimized), and the workaround shown
  gets around the bug and likely obviates similar bugs on other platforms.
...
  --- popt/popt.c.origMon Dec 23 21:51:59 2002
  +++ popt/popt.c Mon Dec 23 21:59:21 2002
  @@ -443,9 +443,10 @@
  }
   
  /* Make a copy we can hack at */
  -   localOptString = optString =
  -   strcpy(ALLOCA(strlen(origOptString) + 1),
  -   origOptString);
  +   { int n = strlen(origOptString) + 1;
  + localOptString = optString = ALLOCA(n);
  + strcpy(optString, origOptString);
  +   }
   
  if (!optString[0])
  return POPT_ERROR_BADOPT;
...
 I don't feel brave enough to commit the popt/popt.c change without input from
 others.


The patch seems harmless, although I'm not sure it's the best style.
How about making strcpy a macro instead?  popt/popt.c is borrowed code,
and if we make modifications to it it might get forgotten the next time
it is upgraded.

- Dave Dykstra
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: Status of sigchld_handler bug?

2002-12-20 Thread Dave Dykstra
On Wed, Dec 18, 2002 at 05:00:00PM -0800, [EMAIL PROTECTED] 
wrote:
 Will this patch ever be integrated into the main code?
 
 http://lists.samba.org/pipermail/rsync/2002-February/006371.html


That bug has got to get fixed before the next release, although I think the
patch at
http://lists.samba.org/pipermail/rsync/2002-September/008227.html
is safer.  As far as I know we've still never heard from Tridge why he
introduced the code that caused the bug.

- Dave Dykstra
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: Problem with absolute symbolic links

2002-12-16 Thread Dave Dykstra
First question: must you use an rsync daemon on the Unix side?  Rsh or
Ssh is a lot easier.  If you do need to use it, you need to know that the
original intention of rsync modules was to only permit files under that
module to be accessible.  With the default use chroot = yes option,
rsync chroot's to the directory so no files outside of the top path can
be reached.  Long ago I added the use chroot = no option so an rsync
daemon could be run under a non-root user id (because only root can run
chroot()), but even then it goes through pains to filter out paths that
attempt to go outside of the top module.  However, I believe, but am not
sure, that if --copy-links is used with use chroot = no it should pull
in files pointed to by symlinks from outside the top.  Try it out.

- Dave Dykstra


On Fri, Dec 13, 2002 at 01:44:30PM -0500, Kevin Minder wrote:
 I'm trying to synchronize a particularly troublesome directory structure on a SunOS 
box to a PC.  I really want all symbolic links resolved to real files on the PC when 
everything is done.  I can't seem to find any combination of parameters that will 
accomplish this.
 
 It boils down to two problems.
 1.  It seems like that even if I exclude a symbolic link that is a directory it is 
not excluded.
 2.  It seems like copy-links and copy-unsafe-links don't work for symbolic links to 
directories outside the source tree.
 
 Is this the correct behavior or am I doing something wrong.
 Any input is greatly appreciated.
 
 Here is my rsyncd.conf file...
 ---
 motd file = /etc/rsyncd.motd
 log file = /var/log/rsyncd.log
 pid file = /var/run/rsyncd.pid
 uid = kminder
 gid = svrtech
 
 [test]
 path = /home/kminder/rsync/test
 comment = test
 read only = yes
 list = yes
 ---
 
 Below is a simplified source structure on the unix side (source).  
 Note the .rel symlink and the .abs symlink.
 
 ---
 /home/kminder/rsync ls -al outside test
 outside:
 total 16
 drwxr-xr-x   2 kminder  g9044096 Dec 13 11:37 .
 drwxr-xr-x   5 kminder  g9044096 Dec 13 11:29 ..
 -rw-r--r--   1 kminder  g904  15 Dec 13 11:37 file
 
 test:
 total 24
 drwxr-xr-x   3 kminder  g9044096 Dec 13 11:36 .
 drwxr-xr-x   5 kminder  g9044096 Dec 13 11:29 ..
 lrwxrwxrwx   1 kminder  g904  27 Dec 13 11:36 .abs - 
/home/kminder/rsync/outside
 lrwxrwxrwx   1 kminder  g904   3 Dec 13 11:34 .rel - dir
 lrwxrwxrwx   1 kminder  g904   9 Dec 13 11:36 absfile - .abs/file
 drwxr-xr-x   2 kminder  g9044096 Dec 13 11:34 dir
 -rw-r--r--   1 kminder  g904  10 Dec 13 11:32 file
 lrwxrwxrwx   1 kminder  g904   4 Dec 13 11:33 link - file
 lrwxrwxrwx   1 kminder  g904   4 Dec 13 11:33 linklink - link
 lrwxrwxrwx   1 kminder  g904   9 Dec 13 11:35 relfile - .rel/file
 ---
 
 I'm trying for the following structure on the pc (cygwin ls -al).  
 Note that all links should be resolved.  
 I want the duplicate files.
 
 ---
 drwxr-xr-x+   4 kminder  None 4096 Dec 13 11:36 .
 drwxrwxrwx+  14 Administ Administ 8192 Dec 13 11:20 ..
 drwxr-xr-x+   2 kminder  None0 Dec 13 11:34 dir
 -rw-r--r--1 kminder  None   10 Dec 13 11:32 file
 -rw-r--r--1 kminder  None   10 Dec 13 11:32 link
 -rw-r--r--1 kminder  None   10 Dec 13 11:32 linklink
 -rw-r--r--1 kminder  None   14 Dec 13 11:34 relfile
 -rw-r--r--1 kminder  None   14 Dec 13 11:34 absfile
 ---
 
 When I execute the command that I think should accomplish this...
 D:\syncrsync -rptgDvP --port=34343 --copy-links --copy-unsafe-links --exclude .rel 
--exclude .abs kminder@usunnbd25::test d:\sync
 The resulting output follows...
 
 ---
 receiving file list ...
 readlink absfile: No such file or directory
 7 files to consider
 ../
 dir/
 dir/file
   14 100%0.00kB/s0:00:00
 file
   10 100%9.77kB/s0:00:00
 link
   10 100%9.77kB/s0:00:00
 linklink
   10 100%9.77kB/s0:00:00
 relfile
   14 100%   13.67kB/s0:00:00
 wrote 190 bytes  read 465 bytes  187.14 bytes/sec
 total size is 58  speedup is 0.09
 rsync error: partial transfer (code D:\syncdir23) at /tmp/rsync-2.5.5/main.c(926)
 ---
 
 My disk ends up looking as follows.  Again using cygwin ls -al.
 
 ---
 drwxr-xr-x+   4 kminder  None 4096 Dec 13 11:36 .
 drwxrwxrwx+  14 Administ Administ 8192 Dec 13 11:20 ..
 drwxr-xr-x+   2 kminder  None0 Dec 13 11:34 .rel
 drwxr-xr-x+   2 kminder  None0 Dec 13 11:34 dir
 -rw-r--r--1 kminder  None   10 Dec 13 11:32 file
 -rw-r--r--1 kminder  None   10 Dec 13 11:32 link
 -rw-r--r--1 kminder  None

Re: bug reporting.. bugzilla

2002-12-11 Thread Dave Dykstra
I disagree that the number of rsync bugs is too low to bother tracking
in a bug tracking system.  I think that a lot of things have been posted
that are genuine bugs but have been forgotten about because none of the
developers have had the time to track them.  If bug reports were all
dealt with quickly I agree there'd be no need for a bug tracking system,
but that's not the case for rsync.  I think it would be a good idea to
have a bugzilla system set up for rsync, especially if we can get some
volunteers to watch the system and make determinations about all the
reports that are put in.  I think it would also help to forward all
messages put into the system to the rsync developers' mailing list.
Having a well-maintained FAQ would also be valuable, to keep the noise
put into the bug tracking system down.

- Dave


On Mon, Dec 09, 2002 at 03:42:03PM +1100, Martin Pool wrote:
 On  5 Dec 2002, Sriram Ramkrishna [EMAIL PROTECTED] wrote:
  What we do in the GNOME project is to find volunteers to run triage
  and catalog the bugs.  If you have a bugmaster position who could
  coordinate something like this.  
 
 The situation is rather different to GNOME, as jw notes: the code is
 not growing very quickly, and there are far fewer developers available
 to work on it.
 
   There used to be another bug reporting system but it was being ignored
   so Martin turned it off.
 
 Yes, I did.  I think the system was broken by some kind of
 infrastructure migration, and since nobody seemed to use it I put up
 this page rather than fixing it:
 
   http://rsync.samba.org/nobugs.html
 
 One problem was that it used tridge's Jitterbug system, which is a
 nice program but a bit harder to learn than Bugzilla, or at least less
 familiar to most people.  Also, because it runs mostly over email, it
 quickly fills up with spam.
 
 But the main thing that discouraged me from maintaining it was just
 that most of the entries were not valid bugs.  We had large numbers of 
 
  - misunderstandings of how to use rsync (operator error)
 
  - massively incomplete reports (e.g. just it fails, without any
error message.)
 
  - architectural limitations (e.g. upfront scan)
 
  - other junk entries
 
 and in addition many of them were redundantly reported.  I think
 probably 90% of entries were like this.
 
 You can see this to a lesser extent in the FAQ-O-Matic:
 
   http://rsync.samba.org/fom-serve/cache/223.html
   http://rsync.samba.org/fom-serve/cache/39.html
   http://rsync.samba.org/fom-serve/cache/233.html
 
 Too many people fail to realize that filing a useful bug is actuallly
 a lot of work and requires that the reporter actually put a bit of
 thought into the problem. /rant
 
 So the database was full of things that were not really bugs, which
 made it pretty useless either for people who wanted to find out about
 a bug they might be experiencing, or developers wanting to know how
 many bugs there are.
 
 I'm sure GNOME has had this too, but if I understand correctly they
 reduced their junk bug count in the first place by throwing out the
 whole database, and then by putting a lot of work into triage and
 cleaning.
 
 I think a better way go forward would be for volunteers to help
 maintain an FAQ.  This might be a good way to address common problems,
 whether they result from misunderstandings or from program errors.  It
 could be in FAQ-O-Matic or something else.  
 
 Since new bug reports are relatively rare, but problems and
 misunderstandings seem to occur repeatedly I think this would be the
 most useful way to get all the information in one place.
 
 So who's interested in working on that? 
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: Patch to ignore exluded files.

2002-12-11 Thread Dave Dykstra
I remember that I put in that code, not so long ago, because I was getting
errors with symlinks pointing nowhere even though they were excluded.
It is revisions 1.96 and 1.95 at
http://cvs.samba.org/cgi-bin/cvsweb/rsync/flist.c

As far as I could see, the only time that the ENOENT error could occur
is if copy_links is on; readlink_stat() won't try to follow the symlink
otherwise.  Ah, I see there might also be a case when copy_unsafe_links is
on; are you using --copy-unsafe-links?  If not, can you please completely
specify a small set of steps, beginning from an empty directory, that
someone could do to demonstrate the problem for which you believe the
below patch will help?

Oh, I see the original poster was talking about files being dynamically
created and deleted during an rsync run; that was something I had not
thought of.  In general, rsync does not even try to support such situations
because they are so difficult to handle robustly.  However, I suppose
it makes sense to ignore files that are excluded.  My first thought
was that it might impact performance for the normal case, but really it
won't because the excludes are only checked when a ENOENT failure occurs.
I'll go ahead and take out the check for copy_links.

- Dave Dykstra

On Thu, Dec 05, 2002 at 05:04:38PM -0600, Bo Kersey wrote:
 I came up with a patch to fix the problem of IO Errors caused by
 excluded files as did Eugene V. Chupriyanov below.  
 
 Is there a chance that this change will show up in a future version of
 rsync?  
 
 Is there a reason that we should not ignore IO errors when copy_links is
 off?  Just want to make sure that I'm not missing something here that
 may corrupt my syncs
 
 Here's the version that I came up with before finding Eugene's patch:
 
 --- rsync-2.5.5/flist.c.orig  2002-03-14 15:20:20.0 -0600
 +++ rsync-2.5.5/flist.c   2002-12-02 19:27:02.0 -0600
 @@ -644,8 +644,8 @@
  
   if (readlink_stat(fname, st, linkbuf) != 0) {
   int save_errno = errno;
 - if ((errno == ENOENT)  copy_links  !noexcludes) {
 - /* symlink pointing nowhere, see if excluded */
 + if ((errno == ENOENT)  !noexcludes) {
 + /* File or directory not found, see if excluded */
   memset((char *) st, 0, sizeof(st));
   if (check_exclude_file(f, fname, st)) {
   /* file is excluded anyway, ignore silently */
 
 
 
 
  Dear friends!
  
  I really appriciate the great job you did on rsync package and hope for its
  further improvement and development. It works great in our pretty complex
  environment and helps us a lot in our daily duties.
  
  But we've run into couple of small inconveniences.
  
  First one:
  We use rsyncd to propagate a tree of html files among several web-servers..
  That files are generated on the fly and updated every few minutes.
  
  Here is relevant part of rsyncd.conf:
  [sync-include]
  path=/usr/local/www/rbc/win/include
  exclude=*tmp
  use chroot=false
  max connections=2
  read only=true
  list=false
  uid=rbcmaint
  gid=www-all
  
  (Note the exlude line)
  
  Nevertheless I was getting errors like this:
  
  readlink .../include/cnews.html.tmp: No such file or directory
  rsync error: partial transfer (code 23) at main.c(578)
  
  I'm not sure I fixed it clean, but I get rid of that errors by patching
  flist.c. Here is the diff:
  
  --- flist.c.origMon Jul  8 17:48:12 2002
  +++ flist.c Mon Jul  8 17:48:39 2002
  @@ -644,7 +644,7 @@
  
  if (readlink_stat(fname, st, linkbuf) != 0) {
  int save_errno = errno;
  -   if ((errno == ENOENT)  copy_links  !noexcludes) {
  +   if ((errno == ENOENT)  !noexcludes) {
  /* symlink pointing nowhere, see if excluded */
  memset((char *) st, 0, sizeof(st));
  if (check_exclude_file(f, fname, st)) {
  
  The second problem appears when we need that target files always have
  correct permissions - we do some testing on them and we receive errors, when
  file already copied and renamed from tempfile, but permission are not yet
  set. So I fixed it with following diff:
  
  --- rsync.c.origFri Jul 19 13:37:25 2002
  +++ rsync.c Fri Jul 19 13:36:42 2002
  @@ -236,13 +236,15 @@
   {
  if (make_backups  !make_backup(fname))
  return;
  +   /* eric: set permisions before moving file */
  +   set_perms(fnametmp, file, NULL, 0);
  
  /* move tmp file over real file */
  if (robust_rename(fnametmp,fname) != 0) {
  if (errno == EXDEV) {
  /* rename failed on cross-filesystem link.
 Copy the file instead. */
  -   if (copy_file(fnametmp,fname, file-mode 
  INITACCESSPERMS

Re: rsync stoped syncing

2002-12-11 Thread Dave Dykstra
Is /home now a symlink?  The fact that the is uptodate message is only
appearing on a single file (home) makes me think it is not recursing
into the directory.  If so, just change the end of your command line to
... /home/ slave:/home/

- Dave

On Mon, Dec 09, 2002 at 04:36:41PM +0100, Markus Lamers wrote:
 Hi,
 
 I want to keep the home-Directorys of 2 machines in sync.
 Therefor I start rsync every hour with cron.
 In the Beginning all worked fine, but now ...
 I searched, but can't see the problem.
 
 The home-directorys on the master change very often, but the
 changes are no longer replicated to the slave. Both run Linux and
 rsync 2.5.5.
 The master calls (the long line may be wrapped here)
 
 rsync -auvxz --delete --exclude-from /root/.rsync/home-daily.exc /home
 slave:/
 
 once a hour and after building the filelist it says home is uptodate.
 But when I make a ls -rtl on both machines .. its definitly not.
 master:
  (just the last few lines)
 drwxrwxrwx5 root root  143 Dec  9 14:51 DRKURSCH4722
 drwxrwxrwx5 root root  250 Dec  9 15:16 TESTBMZ2
 drwxrwxrwx5 root root  143 Dec  9 15:51 DRSTIEH1443
 drwxrwxrwx5 root root 2647 Dec  9 16:15 KRHBRUCHSAL3140
 drwxrwxrwx4 root root  124 Dec  9 16:17 NERESHEIM7059
 drwxrwxrwx5 root root  143 Dec  9 16:18 KURPFALZKHD4560
 drwxrwxrwx5 root root  143 Dec  9 16:18 DRREIFER1740
 drwxrwxrwx5 root root  143 Dec  9 16:25 DRFETTIG0690
 master:/home #
 
 slave:
 
 drwxrwxrwx5 root root  118 Oct 19 00:15 DRHAEBER7009
 drwxrwxrwx5 root root  118 Oct 19 00:15 DRELSER1847
 drwxrwxrwx5 root root  118 Oct 19 00:15 DRDIEDRP2878
 drwxr-xr-x  299 root root 8146 Oct 21 11:56 .
 drwxr-xr-x3 welteusers 948 Oct 22 00:15 welte
 drwxr-xr-x   13 oracle   oinstall 1444 Nov 11 17:11 oracle
 drwxrwxrwx5 root root  193 Nov 29 15:37
 RECHSHGVOELKLINGEN4300
 drwxrwxrwx   26 horras   users2981 Nov 29 15:45 horras
 drwxrwxrwx   11 szyman   users1500 Dec  9 11:08 szyman
 drwxrwxrwx   13 labtimer users   13030 Dec  9 13:10 labtimer
 slave:/home #
 
 The date on both is the same. 
 
 Any hints wants going wrong here ?
 
 Thanks and kind regards,
   Markus Lamers
 -- 
 Markus Lamers [EMAIL PROTECTED]
 Labor Prof. Seelig  Kollegen
 
 -- 
 To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
 Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: Rsync performance increase through buffering

2002-12-11 Thread Dave Dykstra
On Sun, Dec 08, 2002 at 11:48:57PM -0800, Craig Barratt wrote:
 I've been studying the read and write buffering in rsync and it turns
 out most I/O is done just a couple of bytes at a time.  This means there
 are lots of system calls, and also most network traffic comprises lots
 of small packets.  The behavior is most extreme when sending/receiving
 file deltas of identical files.
...
 Below is a patch to a few files that adds read and write buffering in
 the places where the I/O was unbuffered, adds buffering to write_file()
 and removes the unneeded gettimeofday() system call in show_progress().
...
 However, on a network test doing a send from cygwin/WinXP to rsyncd
 on rh-linux the running time improves from about 700 seconds to 215
 seconds (with a cpu load of around 17% versus 58%, if you believe
 cygwin's cpu stats).  This is probably an extreme case since the system
 call penalty in cygwin is high.  But I would suspect a significant
 improvement is possible with a slow network connection, since a lot
 less data is being sent.

Looks like a good candidate for the patches directory now and for
integration in 2.6.0.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: symbolic links

2002-12-05 Thread Dave Dykstra
Personally I don't see why see why replacing a file with a symbolic
link is any more dangerous than replacing it with a different file.
Doesn't it back up the file if you use the --backup option?  If you use
the --update option rsync shouldn't replace any file or symlink that is
newer than the either that is older, will that help?

- Dave 

On Wed, Dec 04, 2002 at 09:19:44PM -0800, Peter Sturdza wrote:
 
 I don't see any option that works.  My problem is not
 how it copies the links; it is that rsync will delete
 a regular file and replace it with the symlink when
 the file is newer than the symlink.  The file is
 always erased with no backup.  This seems wrong.
 
 Basically I use rysnc to keep files on two different
 computers up to date.  Say I replace a symlink on one
 computer with a file by the same name (perhaps because
 I want to make a small change to this file but not
 change the file that is being referred to by the
 symlink), then rsync the two computers.  In this case,
 rsync will always erase the updated file and replace
 it with the older symlink.  
 
 In general, I still do want to preserve symlinks and
 use option -l, however it seems dangerous for a
 symlink to overwrite a regular file without backups or
 regard to timestamps.
 
 This happens on Linux with rsync version 2.5.5.  
 
 Peter
 
 --- Dave Dykstra [EMAIL PROTECTED] wrote:
  That's not a bug, it's a feature.  Note that the -a
  option is equivalent
  to -rlptgoD; replace it with all but the l and
  then look through the
  different options with the word link in it in the
  man page and see if
  one of them does what you want.
  
  - Dave Dykstra
 
 __
 Do you Yahoo!?
 Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
 http://mailplus.yahoo.com
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: rsync 2.5.5, unexpected tag failures, Solaris 2.6 vs. 2.8, --b locking-io workaround

2002-12-05 Thread Dave Dykstra
The problem is in your use of -e rsh; --blocking-io is assumed if
the -e value is equal to the RSYNC_RSH define which is usually remsh
on solaris but maybe it isn't on solaris 2.8.  The --blocking-io option
is required for most versions of rsh, but it's sometimes difficult for
rsync to know when rsh is being used.  It's very confusing, I know.

- Dave

On Tue, Dec 03, 2002 at 10:42:19AM -0800, Madole, Dave BGI SF wrote:
 
 Hi,
 
 I have been having a problem that seems related to previously reported
 problems
 but persists.
 
 I am syncing from a 2.6 Solaris box to a 2.8 solaris box.  All are running
 rsync 2.5.5.
 
 When I sync (commands simplified for example, always using rsh):
 
 2.8box% rsync -e rsh 2.6box:/path/path/stuff/ /path/path/newstuff/
 
 i.e., the 2.6 box is the server, I get spurious, persistent but inconsistent
 unexpected tag some number failures.  This is, unfortunately, on the 2.8
 box, the way I have to do it.
 
 When I sync:
 
 2.6box% rsync -e rsh /path/path/stuff/ 2.8box:/path/path/newstuff/
 
 (the 2.8box is the server), it works.  But the 2.6 box isn't where the
 script runs.
 
 The first command ALSO works if I use the --blocking-io option, 
 
 2.8box%rsync -e rsh 2.6box:/path/path/stuff/ /path/path/newstuff/
 --blocking-io
 
 but I'm not sure what the possible side-effects of that might be.  (I picked
 up from the CVS archive that there was some kind of relationship between a
 socketpair Solaris bug and the --blocking-io option, so tried it, but
 that seemed many versions ago).
 
 I am not doing the builds myself - sysadmins are doing it.  Is there
 anything special they
 need to do in the 2.6 build to avoid my having to use the --blocking-io
 flag?
 
 Thanks,
 
 Dave Madole
 -- 
 To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
 Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: Unexpected rsync behavior with --relative and symlink destinations

2002-12-05 Thread Dave Dykstra
--relative has caused confusion to many people.  If you could come up with
a patch and/or a change to the man page that made it easier to understand
and/or more useful I think it would be welcome.

- Dave Dykstra

On Wed, Dec 04, 2002 at 03:00:42PM -0500, Christopher Schanzle wrote:
 Hello, all.  Sorry if this topic has been hashed out before -- I didn't 
 see it searching the archives of this list.
 
 I want to mirror a local system's /apps and /usr/local directories (both 
 are real directories, not symlinks) to a remote system where /apps and 
 /usr/local are symlinks to /local/apps and /local/usrlocal, 
 respectively.  The target directories exist on the remote system.  I 
 would have thought the following would do it:
 
 rsync -n -e ssh -av --relative /apps /usr/local /home dest:
 
 shows me the expected files to update.  Removing the -n results in all 
 the files being copied, with the first file being listed as apps/ 
 followed by filenames that shouldn't need to be copied and were not 
 listed with the above rsync -n ...  On the destination side, /apps is 
 no longer a symlink and is replaced with a real directory.  All files 
 are resent/copied to this new directory.  I tried these too:
 
 rsync -av -e ssh --relative /apps/. /usr/local/.dest:
 rsync -av -e ssh --relative /apps/* /usr/local/*dest:
 rsync -av -e ssh --relative /apps/*/ /usr/local/*/  dest:
 
 In each case, the symlink on dest:/apps was replaced with a directory.
 
 I realize I can work around this via separate commands, which works fine:
 
 rsync -av -e ssh /apps/   dest:/apps
 rsync -av -e ssh /usr/local/  dest:/usr/local
 
 Is this the expected behavior of --relative?  If so, could the 
 documentation be clarified?
 
 More specifically, if dest:/apps/apache-1.3.26/ exists and is up to date 
 (again, where dest:/apps is a symlink):
 
 # rsync -n -av -e ssh --relative /apps/apache-1.3.26/   dest:
 building file list ... done
 wrote 177990 bytes  read 20 bytes  39557.78 bytes/sec
 total size is 256768589  speedup is 1442.44
 # rsync -av -e ssh --relative /apps/apache-1.3.26/   dest:
 building file list ... done
 apps/
 apps/apache-1.3.26/
 /apps/apache-1.3.26/Makefile
 /apps/apache-1.3.26/Makefile.tmpl
 /apps/apache-1.3.26/apache_1.3.26.tar
 ^Crsync error: received SIGUSR1 or SIGINT (code 20) at rsync.c(229)
 
 I wouldn't expect (1) rsync would change touch dest:/apps at all, (2) 
 differ so drastically in the output of -n and not, and (3) to replace 
 dest:/apps symlink with a new directory.
 
 Both hosts are SPARC Solaris 2.8 systems using UFS filesystems.
 
 TIA,
 Chris
 
 
 -- 
 To unsubscribe or change options: 
 http://lists.samba.org/mailman/listinfo/rsync
 Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: Suggestion: rsync and direct IO

2002-12-05 Thread Dave Dykstra
On Wed, Dec 04, 2002 at 03:41:33PM -0800, David Sisson wrote:
   We have some cases when copying indexes that we'd like rsync to avoid
 the system cache when copying a chunk of data from some other machine. 
 I am probably going to modify our own copy of rsync to do this, but if
 we could write directly to the disk using direct-io (or from for that
 matter) we could avoid polluting the operating system's buffer cache
 before we're ready to use the new data.  Obviously this feature isn't
 tied to Direct-IO, so I'd call the new feature something like
 -avoid-system-cache or something.  I will probably make the feature an
 optional one because I can imagine having a situation where sometimes
 I'd want it to hit the system filecache and sometimes I wouldn't.

Yeah, that sounds like a very good idea.

   On another topic while I'm thinking about it -- does any one know of a
 reason that you can't set socket options in rsync like you can in rsyncd
 -- if you have a push going in the opposite direction to the server it'd
 be nice to have a larger outgoing buffer size (useful for cross country
 pushes).  We have a local version that has been modified to set the
 buffer size and it performs 3 times faster than the unmodified version.

Probably just an oversight, although I would have thought the buffer size
would be up to the transport program (rsh or ssh).



On Thu, Dec 05, 2002 at 10:01:08AM -0800, Jos Backus wrote:
 On Thu, Dec 05, 2002 at 07:39:53AM -0500, Green, Paul wrote:
  But is there an OS-independent way to set this attribute?  I haven't
  memorized the POSIX standard, but I don't recall seeing a way to set or get
  such an attribute.

 Fyi: Solaris has directio(3C), FreeBSD has open(..., O_DIRECT).

What about Linux?  There's an O_DIRECT but the comment in asm/fcntl.h says
it is currently ignored.

- Dave Dykstra
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: bug reporting.. bugzilla

2002-12-05 Thread Dave Dykstra
On Wed, Dec 04, 2002 at 04:42:35PM -0800, Sriram Ramkrishna wrote:
 Have you considered using bugzilla for reporting bugs?  Just curious.
 I was looking on the rsync website and didn't see much in bug 
 reporting.

There used to be another bug reporting system but it was being ignored
so Martin turned it off.  I think it would be a good idea to have a bug
tracking system, but then somebody would need to respond to everything.
I think Martin preferred that questions get sent to the mailing list
where there are more people who might respond and not just the maintainer.
Perhaps a compromise like on the openssh mailing list would be in order,
where every update to bugzilla gets sent to the main developers' mailing
list.  Perhaps it would also make sense to have a separate developer and
user mailing list.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: Rsync --non-existent-dir-only feature :) ..

2002-12-03 Thread Dave Dykstra
Does the --ignore-existing option do what you want?  It will probably recurse
into the existing directories, but ignore all regular files that already exist.

- Dave Dykstra

On Tue, Dec 03, 2002 at 01:01:03AM +0100, Robert Raszuk wrote:
 
 All,
 
 I can't find a way to achive something probably quite wired :). Let's
 say I have two unix folders (on different machines):
 
 /dst-top-1n/
|
dir-a
| files 
|
dir-b
| files 
|
dir-c
| files 
 
 /src-top-2/
|
dir-a
| files 
|
dir-b
| files 
|
dir-x
| files 
 
 
 Always src is top-2  dst is top-1. All I need to do is to be able to
 send dir-x files to top-1 without even touching dir-a  dir-b simply
 because the overall tool does not have permissions to write access them. 
 
 To make it more interesting there are 100s of those top-1 directories
 (each on different client machine), the rsync is fully automated, and
 the context of each of the top-1 dir differs. Sync is triggered by src.
 
 What I am really looking for is something like in the subject one liner
 knob which would compare dir-x directories on src/dst and perform no
 action under matching already dirs on both machines.
 
 Huge Thx - Rsync Rocks 
 R.
 -- 
 To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
 Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



  1   2   3   4   5   6   >