[Bug 10449] New: Allow testing of supported parameter(s)

2014-02-14 Thread samba-bugs
https://bugzilla.samba.org/show_bug.cgi?id=10449

   Summary: Allow testing of supported parameter(s)
   Product: rsync
   Version: 3.1.0
  Platform: All
OS/Version: All
Status: NEW
  Severity: enhancement
  Priority: P5
 Component: core
AssignedTo: way...@samba.org
ReportedBy: m...@haravikk.com
 QAContact: rsync...@samba.org


In newer versions of rsync there are many desirable features that simply aren't
available in older versions, but unfortunately some systems continue to be
bundled older versions of rsync.


I'd like to recommend a special rsync mode which will attempt to perform a
compatibility check for particular features. Of course this will only work for
newer versions, but at least it should then fail when used with an older
version and thus still provide useful feedback.

Quite simply I'd like to suggest either an extension of the --help or --version
flag, whereby any additional parameters are treated as features to be tested
for. If source and destination parameters are also provided, then these will be
used for determining which features are available between the source and target
specifically (i.e - it will check with a remote system if able to do-so).


For example:
rsync --version --delete-during --ignore-case ~/Foo m...@bar.com:/Foo

This would attempt to determine whether the delete-during and ignore-case
switches are available for a transfer between the local volume and m...@bar.com.
If all listed features are supported then it will return a particular status
code (for compatibility with current versions this shouldn't be 0), otherwise
it will return a different status and will print to stdout a list of
unsupported features, so the script knows not to use these when performing the
actual transfer operation.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
--- You are receiving this mail because: ---
You are the QA contact for the bug.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: Beta testing a native win32 rsync client with auto sync

2012-01-18 Thread Konstantinos Skarlatos

On Τετάρτη, 18 Ιανουάριος 2012 4:58:21 πμ, Gang Chen wrote:

Hi, there,
We implemented a native win32 rsync client with auto synchronizer: 
http://www.acrosync.net
We are currently looking for beta testers.  If anyone is 
interested, please post a request in the forum 
(http://forum.acrosync.net) to receive a download link.

Thanks,
Acrosync Team




Hi I am very interested in a good rsync client for win32, so I have 
posted in your forum in order to test your software. But I am more 
interested in a rsync daemon-server for windows, so i can pull backups. 
VSS integration would be a major plus

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Beta testing a native win32 rsync client with auto sync

2012-01-17 Thread Gang Chen
Hi, there,

We implemented a native win32 rsync client with auto synchronizer:
http://www.acrosync.net
We are currently looking for beta testers.  If anyone is interested,
please post a request in the forum (http://forum.acrosync.net) to receive a
download link.
Thanks,
Acrosync Team
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Passed all tests with flying colors on Mac OS X 10.4.11 - synopsis of installation and testing

2008-03-03 Thread Rob Rye
Awesome work Wayne. I have been following the various threads about  
running rsync 3.0.0 on MacOS X 10.4.11 and wanted to confirm that  
following all the various bits of advice yields a clean running rsync  
(as tested by backup bouncer).


I am running MacOS X 10.4.11 on PPC. The following is a synopsis of  
my installation procedure (as gathered from the various emails in the  
archive) - note that I had trouble with crtimes.diff and went back to  
osx-create-time.diff. Please let me know if I have included anything  
spurious or missed something critical (though I am pleased that  
everything worked)...


I downloaded the source (3.0.0 release) and the patches (as posted on  
the front page as of this morning). I then added the patch Wayne  
provided called fileflags-fixes.diff (not in the folder when I  
downloaded it anyway)


patch -p1  patches/osx-create-time.diff
   patch -p1  patches/fileflags.diff
   patch -p1  patches/fileflags-fixes.diff
   ./prepare-source (spurious? it did not seem to do anything...)
   patch -p1  patches/backup-dir-dels.diff
   ./configure
   make
   make test
   sudo make install

I then used backup bouncer to test the installation:

cd into backup bouncer folder
./bbouncer create-vol Src
./bbouncer create-vol Dst
./bbouncer create /Volumes/Src
sudo rsync -aHAX --force-change /Volumes/Dst/ /Volumes/Src/
Finally, I ran the tests

./bbouncer verify -d /Volumes/Src /Volumes/Dst
Verifying:basic-permissions ... ok
Verifying:   timestamps ...
   Sub-test:modification time ... ok
ok
Verifying: symlinks ... ok
Verifying:symlink-ownership ... ok
Verifying:hardlinks ... ok
Verifying:   resource-forks ... ok
Verifying: finder-flags ... ok
Verifying: finder-locks ... ok
Verifying:creation-date ... ok
Verifying:bsd-flags ... ok
Verifying:   extended-attrs ...
   Sub-test: on files ... ok
   Sub-test:   on directories ... ok
   Sub-test:  on symlinks ... ok
ok
Verifying: access-control-lists ...
   Sub-test: on files ... ok
   Sub-test:  on dirs ... ok
ok
Verifying: fifo ... ok
Verifying:  devices ... ok
Verifying:  combo-tests ...
   Sub-test:  xattrs + rsrc forks ... ok
   Sub-test: lots of metadata ... ok
ok
-

Everything looks great. Thanks to Wayne and all those who have tested  
and refined this. I confess I am one of the myriad freeloaders who  
have been sitting back waiting for all of you to do the heavy  
lifting. I am a frequent beta tester on many open source projects,  
but I just don't have the guts to participate in such testing on a  
backup utility/app. So, I am thrilled to have a fully functional,  
modern, backup app for my Mac without having gone through the  
heartaches inherent in participating in the testing.


Cheers,

Rob
-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Passed all tests with flying colors on Mac OS X 10.4.11 - synopsis of installation and testing

2008-03-03 Thread Anthony Morton
I am running MacOS X 10.4.11 on PPC. The following is a synopsis of  
my installation procedure (as gathered from the various emails in  
the archive) - note that I had trouble with crtimes.diff and went  
back to osx-create-time.diff. Please let me know if I have included  
anything spurious or missed something critical (though I am pleased  
that everything worked)...


I also get perfect results on 3.0.0 with both MacOS X 10.4.11 on PPC  
and MacOS X 10.5.2 on Intel, using fileflags.diff, crtimes.diff and  
Wayne's new fileflags-fixes.diff.


I find it interesting that you have had no luck with crtimes.diff.   
Could you post your results?  Are you applying the patches in the  
right order (that is, fileflags.diff followed by crtimes.diff, not  
vice versa)?


I suspect the backup-dir-dels.diff patch is not strictly necessary,  
unless you actually want the functionality (as many of us do).


./prepare-source may or may not do anything, depending which source  
version you're using.


Tony M.

--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: Passed all tests with flying colors on Mac OS X 10.4.11 - synopsis of installation and testing

2008-03-03 Thread Anthony Morton
I think you hit the nail on the head. I put the crtimes.diff first  
because I was simply swapping it in for osx-create-time.diff in  
Axel's email 3.0.0 test failure MacOS X 10.4.11.


Upon invoking make, this error in the ordering of the patches, on my  
part yielded:


Check the output of your 'patch' commands and make sure there are no  
FAILED messages.  If there are, then your 'make' will probably fail  
unless they're resolved.


If you apply fileflags.diff followed by crtimes.diff then you  
shouldn't see any failures - if you do then it's probably worth  
letting the list know.


Tony M.

--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: testing rsync 2.6.3pre1

2004-08-17 Thread Paul Slootman
On Mon 16 Aug 2004, Wayne Davison wrote:

 I announced the release of 2.6.3pre1 on the rsync-announce mailing list
 back on the 12th.  I was thinking at the time that this email would get
 seen by those on the regular rsync list, but I forgot that this is not
 how the rsync lists are setup.  For those that missed it, here is the

I also missed the security announcement on the same day... Has the rsync
list setup changed? I guess I'll have to subscribe to -announce as
well...

 I'd appreciate it if people would let me know about any testing they do,
 even if it's just to say that things are working fine.

I've built a Debian package(*) (uploaded to the 'experimental'
distribution), and tried it on a couple of systems. It seems to work
fine...

(*) Debian version is 2.6.2.pre3.1-1, to prevent problems with version
ordering when 2.6.3 final is released...


Paul Slootman
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


testing rsync 2.6.3pre1

2004-08-16 Thread Wayne Davison
I announced the release of 2.6.3pre1 on the rsync-announce mailing list
back on the 12th.  I was thinking at the time that this email would get
seen by those on the regular rsync list, but I forgot that this is not
how the rsync lists are setup.  For those that missed it, here is the
text of the announcement:

--

I have released rsync 2.6.3pre1 -- the first pre-release version of
2.6.3.  Please help out with the testing so that the code gets a good
workout before 2.6.3 is released.  Thanks!

To see what has changed, read this text file:

http://rsync.samba.org/ftp/rsync/preview/rsync-2.6.3pre1-NEWS

To download a copy, use one of these links:

http://rsync.samba.org/ftp/rsync/preview/rsync-2.6.3pre1.tar.gz
http://rsync.samba.org/ftp/rsync/preview/rsync-2.6.3pre1.tar.gz.asc

http://rsync.samba.org/ftp/rsync/preview/rsync-2.6.2-2.6.3pre1.diffs.gz
http://rsync.samba.org/ftp/rsync/preview/rsync-2.6.2-2.6.3pre1.diffs.gz.asc

The diffs no longer include the patches subdir (since including them
makes the diffs double in size).  If you want an updated set of patches
without grabbing the tar file, you can use rsync to update the patches
dir in your rsync source using this command:

rsync -av --delete rsync://rsync.samba.org/ftp/unpacked/rsync/patches ~/src/rsync/

--

I'd appreciate it if people would let me know about any testing they do,
even if it's just to say that things are working fine.

..wayne..
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


testing QA aliases

2004-06-30 Thread Gerald (Jerry) Carter
please ignore.



-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


2.6.2 on Cygwin - initial testing results

2004-05-03 Thread Jim Salter
I compiled and installed rsync 2.6.2 from source (the site's download 
page) today, under Cygwin 1.5.9-1 / Windows 2000 Professional SP4.

Initial testing shows that it operates properly and smoothly, and that 
Wayne's bugfix for the daemon mode / --backup problem I reported earlier 
did the trick... --backup is working fine, no extraneous tildes.

Also, and this may be of interest to any other people forced to backup 
Windows servers from time to time, cwRsync - which is VERY handy in that 
among other things, it automagically creates a Windows service to run 
rsync in daemon mode for you - operates just fine if you replace its 
cygwin1.dll and rsync.exe with the 1.5.9-1 version of cygwin1.dll and a 
copy of rsync compiled from source.

I stopped my existing cwrsync service, did the dll and rsync.exe 
replacements, and restarted the service - no problems, everything seems 
to be running smoothly.

Hope this helps somebody!

-J
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Testing new subscriber

2003-11-25 Thread Johan
Hi,
Testing if working
-- 
Johan
May this be a good day for learning
Registered Linux User #330034 - still learning

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: Oops more testing was required....

2003-06-27 Thread jw schultz
On Wed, Jun 18, 2003 at 01:28:32PM +0200, Rogier Wolff wrote:
 On Wed, Jun 18, 2003 at 09:09:59PM +1000, Martin Pool wrote:
  On 17 Jun 2003, Rogier Wolff [EMAIL PROTECTED] wrote:
   
   Oops. Missed one line in the last patch
  
  Thankyou.  That looks good.
  
  If we're going to make this more accurate it might be worthwhile to
  actually look at how long we really did sleep for, and use that to
  adjust time_to_sleep rather than resetting to zero.
  
  Also I'd prefer the variable be called micros_to_sleep or
  us_to_sleep.  Small point I know.
 
 OK. Agreed. 
 
 If I'm going to do gettimeofday calls, I might just as well keep a
 variable that keeps the time until we've used our quotum. 
 
 Whenever we're ready to sleep, we'll get the current time, and only
 sleep for the difference.
 
 The trouble is, that if we end up using 5 seconds worth of CPU time, 
 we should not suddenly try to catch up for those whole 5 seconds 
 
 I like to set bwlimit to about 90% of my link capacity. This causes
 rsync not to fill the link: filling the link causes my provider to
 drop my packets. Also rsync won't fill my queues leading to higher
 latencies on interactive traffic.
 
 But if we allow bursting of those 5 seconds that we didn't have any
 data, we'll fill the link to the brim for 50 seconds! (if there
 suddenly is a nice supply of fresh data) This is undesirable. So, I'm
 still thinking on how to make this more accurate, but prevent big
 bursts. I'm considering not allowing more than bwlimit of
 backlog. (c.f. leaky bucket bandwidth limiting in the packet filtering
 world)

I'd suggest experimentation with shaping your own traffic.
lartc.org looks like it has some good stuff on it
particularly wondershaper.

Long term, i think the bwlimit stuff needs a complete
reexamination.  In addition to the sleeping times and
spreading the load as in the smoother bandwidth limiting
but also its converse, Craig's buffering.  I suspect the
approach is to have a network write routine that deals with
buffering, write splitting and sleeping in a coordinated
way.
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: Oops more testing was required....

2003-06-27 Thread Ben Escoto
 jws == jw schultz [EMAIL PROTECTED]
 wrote the following on Fri, 27 Jun 2003 19:49:22 -0700

  jws Long term, i think the bwlimit stuff needs a complete
  jws reexamination.  In addition to the sleeping times and spreading
  jws the load as in the smoother bandwidth limiting but also its
  jws converse, Craig's buffering.  I suspect the approach is to have
  jws a network write routine that deals with buffering, write
  jws splitting and sleeping in a coordinated way.

Do you think it is necessary for rsync to have to have bandwidth
control at all?  It seems this should be someone else's
responsibility.  For instance, if rsync is run over a pipe, something
like cstream or throttle could be used (I don't know if these are
really suitable, but there's no reason a separate utility couldn't do
it).  Even for the daemon, maybe it could be run over some bandwidth
limiting proxy or something, if the OS couldn't limit the bandwidth
directly.


-- 
Ben Escoto


pgp0.pgp
Description: PGP signature
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Oops more testing was required....

2003-06-18 Thread Martin Pool
On 17 Jun 2003, Rogier Wolff [EMAIL PROTECTED] wrote:
 
 Oops. Missed one line in the last patch

Thankyou.  That looks good.

If we're going to make this more accurate it might be worthwhile to
actually look at how long we really did sleep for, and use that to
adjust time_to_sleep rather than resetting to zero.

Also I'd prefer the variable be called micros_to_sleep or
us_to_sleep.  Small point I know.

 diff -ur rsync-2.5.6.orig/io.c rsync-2.5.6/io.c
 +++ rsync-2.5.6/io.c  Tue Jun 17 23:43:49 2003
 @@ -416,10 +416,19 @@
   * use a bit less bandwidth than specified, because it doesn't make up
   * for slow periods.  But arguably this is a feature.  In addition, we
   * ought to take the time used to write the data into account.
 + *
 + * During some phases of big transfers (file XXX is uptodate) this is
 + * called with a small bytes_written every time. As the kernel has to
 + * round small waits up to guarantee that we actually wait at least
 + * the requested number of microseconds, this can become grossly
 + * inaccurate. We therefore keep a cumulating number of microseconds
 + * to wait, and only actually perform the sleep when the rouding
 + * becomes insignificant. (less than 10%) -- REW.
   **/
  static void sleep_for_bwlimit(int bytes_written)
  {
   struct timeval tv;
 + static int time_to_sleep = 0; 
  
   if (!bwlimit)
   return;
 @@ -427,9 +436,13 @@
   assert(bytes_written  0);
   assert(bwlimit  0);
   
 - tv.tv_usec = bytes_written * 1000 / bwlimit;
 - tv.tv_sec  = tv.tv_usec / 100;
 - tv.tv_usec = tv.tv_usec % 100;
 + time_to_sleep += bytes_written * 1000 / bwlimit; 
 +
 + if (time_to_sleep  10) return;
 +
 + tv.tv_sec  = time_to_sleep / 100;
 + tv.tv_usec = time_to_sleep % 100;
 + time_to_sleep = 0; 
  
   select(0, NULL, NULL, NULL, tv);
  }

-- 
Martin 
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: Oops more testing was required....

2003-06-18 Thread Rogier Wolff
On Wed, Jun 18, 2003 at 09:09:59PM +1000, Martin Pool wrote:
 On 17 Jun 2003, Rogier Wolff [EMAIL PROTECTED] wrote:
  
  Oops. Missed one line in the last patch
 
 Thankyou.  That looks good.
 
 If we're going to make this more accurate it might be worthwhile to
 actually look at how long we really did sleep for, and use that to
 adjust time_to_sleep rather than resetting to zero.
 
 Also I'd prefer the variable be called micros_to_sleep or
 us_to_sleep.  Small point I know.

OK. Agreed. 

If I'm going to do gettimeofday calls, I might just as well keep a
variable that keeps the time until we've used our quotum. 

Whenever we're ready to sleep, we'll get the current time, and only
sleep for the difference.

The trouble is, that if we end up using 5 seconds worth of CPU time, 
we should not suddenly try to catch up for those whole 5 seconds 

I like to set bwlimit to about 90% of my link capacity. This causes
rsync not to fill the link: filling the link causes my provider to
drop my packets. Also rsync won't fill my queues leading to higher
latencies on interactive traffic.

But if we allow bursting of those 5 seconds that we didn't have any
data, we'll fill the link to the brim for 50 seconds! (if there
suddenly is a nice supply of fresh data) This is undesirable. So, I'm
still thinking on how to make this more accurate, but prevent big
bursts. I'm considering not allowing more than bwlimit of
backlog. (c.f. leaky bucket bandwidth limiting in the packet filtering
world)

Roger. 

 
  diff -ur rsync-2.5.6.orig/io.c rsync-2.5.6/io.c
  +++ rsync-2.5.6/io.cTue Jun 17 23:43:49 2003
  @@ -416,10 +416,19 @@
* use a bit less bandwidth than specified, because it doesn't make up
* for slow periods.  But arguably this is a feature.  In addition, we
* ought to take the time used to write the data into account.
  + *
  + * During some phases of big transfers (file XXX is uptodate) this is
  + * called with a small bytes_written every time. As the kernel has to
  + * round small waits up to guarantee that we actually wait at least
  + * the requested number of microseconds, this can become grossly
  + * inaccurate. We therefore keep a cumulating number of microseconds
  + * to wait, and only actually perform the sleep when the rouding
  + * becomes insignificant. (less than 10%) -- REW.
**/
   static void sleep_for_bwlimit(int bytes_written)
   {
  struct timeval tv;
  +   static int time_to_sleep = 0; 
   
  if (!bwlimit)
  return;
  @@ -427,9 +436,13 @@
  assert(bytes_written  0);
  assert(bwlimit  0);
  
  -   tv.tv_usec = bytes_written * 1000 / bwlimit;
  -   tv.tv_sec  = tv.tv_usec / 100;
  -   tv.tv_usec = tv.tv_usec % 100;
  +   time_to_sleep += bytes_written * 1000 / bwlimit; 
  +
  +   if (time_to_sleep  10) return;
  +
  +   tv.tv_sec  = time_to_sleep / 100;
  +   tv.tv_usec = time_to_sleep % 100;
  +   time_to_sleep = 0; 
   
  select(0, NULL, NULL, NULL, tv);
   }
 
 -- 
 Martin 

-- 
** [EMAIL PROTECTED] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
* The Worlds Ecosystem is a stable system. Stable systems may experience *
* excursions from the stable situation. We are currently in such an  * 
* excursion: The stable situation does not include humans. ***
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: Oops more testing was required....

2003-06-18 Thread jw schultz
On Wed, Jun 18, 2003 at 09:09:59PM +1000, Martin Pool wrote:
 On 17 Jun 2003, Rogier Wolff [EMAIL PROTECTED] wrote:
  
  Oops. Missed one line in the last patch
 
 Thankyou.  That looks good.
 
 If we're going to make this more accurate it might be worthwhile to
 actually look at how long we really did sleep for, and use that to
 adjust time_to_sleep rather than resetting to zero.

That would have to be a platform specific thing since not
all systems modify the timeout value to reflect the amount
of time not slept.  Nevertheless that is a nice idea.

 Also I'd prefer the variable be called micros_to_sleep or
 us_to_sleep.  Small point I know.

I'm not too keen on forcing the minimum sleep to be 100ms.
That is just too coarse for my taste and tends produce
fits-and-starts stalling on very slow links.

This should be balanced with the Smoother bandwidth limiting
thread.  Perhaps setting a minimum sleep to 10ms or
a period based on the value of bwlimit.

  diff -ur rsync-2.5.6.orig/io.c rsync-2.5.6/io.c
  +++ rsync-2.5.6/io.cTue Jun 17 23:43:49 2003
  @@ -416,10 +416,19 @@
* use a bit less bandwidth than specified, because it doesn't make up
* for slow periods.  But arguably this is a feature.  In addition, we
* ought to take the time used to write the data into account.
  + *
  + * During some phases of big transfers (file XXX is uptodate) this is
  + * called with a small bytes_written every time. As the kernel has to
  + * round small waits up to guarantee that we actually wait at least
  + * the requested number of microseconds, this can become grossly
  + * inaccurate. We therefore keep a cumulating number of microseconds
  + * to wait, and only actually perform the sleep when the rouding
  + * becomes insignificant. (less than 10%) -- REW.
**/
   static void sleep_for_bwlimit(int bytes_written)
   {
  struct timeval tv;
  +   static int time_to_sleep = 0; 
   
  if (!bwlimit)
  return;
  @@ -427,9 +436,13 @@
  assert(bytes_written  0);
  assert(bwlimit  0);
  
  -   tv.tv_usec = bytes_written * 1000 / bwlimit;
  -   tv.tv_sec  = tv.tv_usec / 100;
  -   tv.tv_usec = tv.tv_usec % 100;
  +   time_to_sleep += bytes_written * 1000 / bwlimit; 
  +
  +   if (time_to_sleep  10) return;
  +
  +   tv.tv_sec  = time_to_sleep / 100;
  +   tv.tv_usec = time_to_sleep % 100;
  +   time_to_sleep = 0; 
   
  select(0, NULL, NULL, NULL, tv);
   }
 
 -- 
 Martin 
 -- 
 To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
 Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
 

-- 

J.W. SchultzPegasystems Technologies
email address:  [EMAIL PROTECTED]

Remember Cernan and Schmitt
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: Oops more testing was required....

2003-06-18 Thread Martin Pool
On 18 Jun 2003, jw schultz [EMAIL PROTECTED] wrote:
 On Wed, Jun 18, 2003 at 09:09:59PM +1000, Martin Pool wrote:
  On 17 Jun 2003, Rogier Wolff [EMAIL PROTECTED] wrote:
   
   Oops. Missed one line in the last patch
  
  Thankyou.  That looks good.
  
  If we're going to make this more accurate it might be worthwhile to
  actually look at how long we really did sleep for, and use that to
  adjust time_to_sleep rather than resetting to zero.
 
 That would have to be a platform specific thing since not
 all systems modify the timeout value to reflect the amount
 of time not slept.  Nevertheless that is a nice idea.

Right, I know that is not portable but I forgot to say so.  As Rogier
say, you need to call gettimeofday() or some such.

-- 
Martin 
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: Oops more testing was required....

2003-06-18 Thread jw schultz
On Wed, Jun 18, 2003 at 01:28:32PM +0200, Rogier Wolff wrote:
 On Wed, Jun 18, 2003 at 09:09:59PM +1000, Martin Pool wrote:
  On 17 Jun 2003, Rogier Wolff [EMAIL PROTECTED] wrote:
   
   Oops. Missed one line in the last patch
  
  Thankyou.  That looks good.
  
  If we're going to make this more accurate it might be worthwhile to
  actually look at how long we really did sleep for, and use that to
  adjust time_to_sleep rather than resetting to zero.
  
  Also I'd prefer the variable be called micros_to_sleep or
  us_to_sleep.  Small point I know.
 
 OK. Agreed. 
 
 If I'm going to do gettimeofday calls, I might just as well keep a
 variable that keeps the time until we've used our quotum. 
 
 Whenever we're ready to sleep, we'll get the current time, and only
 sleep for the difference.
 
 The trouble is, that if we end up using 5 seconds worth of CPU time, 
 we should not suddenly try to catch up for those whole 5 seconds 

Something on the order of:
us_to_sleep = max(us_to_sleep - us_since_last_write, 0);
us_to_sleep += amount_to_sleep_for_this_write;

 I like to set bwlimit to about 90% of my link capacity. This causes
 rsync not to fill the link: filling the link causes my provider to
 drop my packets. Also rsync won't fill my queues leading to higher
 latencies on interactive traffic.

 But if we allow bursting of those 5 seconds that we didn't have any
 data, we'll fill the link to the brim for 50 seconds! (if there
 suddenly is a nice supply of fresh data) This is undesirable. So, I'm
 still thinking on how to make this more accurate, but prevent big
 bursts. I'm considering not allowing more than bwlimit of
 backlog. (c.f. leaky bucket bandwidth limiting in the packet filtering
 world)

It sounds like your problem is less that there are tiny
sleeps than that you have too many big transmissions.
Take a look at the Smoother bandwidth limiting thread.
Balancing these two issues for sleeps in the 10 to 100ms
range may be the best approach.

-- 

J.W. SchultzPegasystems Technologies
email address:  [EMAIL PROTECTED]

Remember Cernan and Schmitt
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: Oops more testing was required....

2003-06-18 Thread jw schultz
On Wed, Jun 18, 2003 at 01:52:10PM +0200, Rogier Wolff wrote:
 On Wed, Jun 18, 2003 at 04:26:48AM -0700, jw schultz wrote:
  On Wed, Jun 18, 2003 at 09:09:59PM +1000, Martin Pool wrote:
   On 17 Jun 2003, Rogier Wolff [EMAIL PROTECTED] wrote:

Oops. Missed one line in the last patch
   
   Thankyou.  That looks good.
   
   If we're going to make this more accurate it might be worthwhile to
   actually look at how long we really did sleep for, and use that to
   adjust time_to_sleep rather than resetting to zero.
  
  That would have to be a platform specific thing since not
  all systems modify the timeout value to reflect the amount
  of time not slept.  Nevertheless that is a nice idea.
 
 no, no.  
 
 Martin was suggesting to measure the elapsed time using gettimeofday.
 
   Also I'd prefer the variable be called micros_to_sleep or
   us_to_sleep.  Small point I know.
  
  I'm not too keen on forcing the minimum sleep to be 100ms.
  That is just too coarse for my taste and tends produce
  fits-and-starts stalling on very slow links.
 
 100ms is a tenth of a second on both slow links as well as on fast
 links 

Think about modem links that run at 1KB/sec or slower.  You
aren't just saying sleep at least 100ms, you are also saying
to pump enough data down the line to warrant a 100ms sleep
even if we are doing small writes.

 As Unix works with a system clock that increments only once per 10ms,
 the system has to round a requested time up to the next multiple of
 10ms. 

False.  System clocks vary a by orders of magnitude.  Some
are as slow as 50 or 60hz (power line frequency),
2.4 linux on ia32 is indeed 100hz as are some proprietary
UNIXes but most newer platforms are currently running at
tick rates of 1000 or 1024hz with 4Khz and higher on the
horizon thanks to faster CPUs.

 So if you request a sleep of 2 ms, we can round that to 10ms. Now if
 current time is 100.1 jiffies (1001ms), the wakeup could be scheduled
 at 101 jiffies (1010 ms). However, when the request to sleep for 2ms
 comes in at 100.9 jiffies (1009ms), the wakeup at 1010 ms (101
 jiffies) is too early. So we have to add one more to be sure to wait
 AT LEAST the requested amount of time.

If you were talking about adding a knowledge of timer
resolution to the code i'd say OK.  But you are talking
about hard-coding logic on the false assumption of a fixed
clock rate.

The round-down effect is true of select where the timeout is
an upper bound and a not sleeping is permissable.  Perhaps
what we should be doing instead is nanosleep which
guarantees a sleep at least as long as requested unless a
signal is received and per POSIX can provide the actual time
slept.

 My rsync was doing 70 byte writes, limited to 30k per second, leads to
 a sleep time of 2500 us. This is effectively rounded by the kernel to
 20ms. So we're off by a factor of 8. 
 
 I chose 100ms to make this rounding less than about 10%. 
 
 When limited to 30k per second, I saw my rsync write chunks of 16k.
 It then calculates a sleeptime of over 500ms, and sleeps that much 
 in one go. So there you have your coarse time-resolution..
 
 
  This should be balanced with the Smoother bandwidth limiting
  thread.  Perhaps setting a minimum sleep to 10ms or
  a period based on the value of bwlimit.
 
 No, the kernel rounding is time-based. Thus a time related heuristic
 works best.

Only if it actually knows the local clock rate.  Even then
you should be taking into account balancing so you don't
spend half seconds sleeping between overflowing the pipe.

-- 

J.W. SchultzPegasystems Technologies
email address:  [EMAIL PROTECTED]

Remember Cernan and Schmitt
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


2nd release of my new-protocol testing app

2002-06-13 Thread Wayne Davison

I've been having a lot of fun improving my new-protocol testing app.
It's seems to be in pretty good shape (for test code), so I figured I'd
announce another release for those brave souls that may want to help me
in my thinking about a (potential) new rsync protocol.  It's a tar.gz
file this time because I broke up the code into multiple files.  I named
it rzync just for fun (a very confusing name, no?):

http://www.clari.net/~wayne/rzync.tar.gz

The new stuff in this release is that it can get/put an entire directory
tree of files via getd/putd, and it has conditional get/put commands
that handle both files and directories (cget/cput).  (For those that
missed the first announcement, the program can be totally controlled by
an external application via a simple set of commands on stdin.)

I've included a perl script named rs that will take an rsync-like
command line (as long as the destination is a directory and not a file)
and drive rzync with it.  Keep in mind that rzync still has the -a
option hard-wired to on, so rs -v /path/foo remote:/path works like
rsync -av /path/foo remote:/path.

Things I've noticed so far:

- My single-proc generator/receiver seems to perform well when I send
  data over my DSL connection, but it goes much slower than rsync when
  sending data over a local pipe.  I'm guessing that this is because a
  multi-process setup can keep the generator pipeline filled to a
  greater degree.  If this is true, one solution would be to add a
  thread that would be responsible for handling all the generator tasks
  (and perhaps using the GNU portable thread library if we want to be
  compatible with systems that don't support process threads).

- The deltas produced by librsync are sometimes considerably larger than
  those produced by rsync, so the speedup of rzync sometimes suffers
  compared to rsync.  I believe that this is because (even without -z)
  rsync does some compression of the delta data that librsync does not
  do.

- The incremental directory scanning seems to work quite well.  I have
  not fleshed out all the areas that would need to grow dynamically for
  _really_ large jobs, so if someone wants to try to send some huge
  directory trees, we'll have to flesh out some more of the code first.

- My directory-scanning code does not attempt to handle symlinks,
  devices, or named sockets yet (it just skips them).

- Since the directory-scan data is shared between the two sides using
  the rsync algorithm, it has the potential to save a lot of transfer
  bytes when the directories on each side are similar.

Feel free to let me know what you think.

..wayne..


-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: 2nd release of my new-protocol testing app

2002-06-13 Thread Wayne Davison

On Thu, 13 Jun 2002, Wayne Davison wrote:
 http://www.clari.net/~wayne/rzync.tar.gz

I forgot to mention that I changed the order of the local/remote args
to the 2-arg version of the cd command to be cd LOCAL REMOTE (the
command cd DIR still changes both the local and remote sides).  This
only affects someone who had written a script or an input file to drive
my earlier rsynx_xfer release.  I hope that didn't trip someone up.

..wayne..


-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Testing a transfer-only rsync tool

2002-06-01 Thread Wayne Davison

I found some time in the past week to work on a simple test app that
would hopefully help to answer a few questions that came up recently:

1. Can a single-process generator+receiver work well?  (Looks good so far,
   but I haven't run any multi-processor timing tests yet.)

2. How easy is it to use librsync?  (Pretty easy.)

3. How small would a transfer-only tool be?  (It's currently around 1400
   lines of C code, not counting the librsync code.  It was around 900
   lines when I first considered releasing a simple working version, but
   it keeps growing as I flesh out the more advanced features.)

4. Should rsync be separated into a scanning tool and a transfer tool?
   Or should it contain both bits but also allow the user to override
   the scanner to fully control what gets transferred?  Or should we
   just try to optimize the current protocol? (No answers yet, but I'm
   leaving toward the 2nd option above.)

My test tool takes in commands on stdin and outputs messages on stdout.
It forks a second process as specified via the commandline (which can be
any command that runs another rsync_xfer, either locally or remotely).
It then allows you to send AND/OR receive any files you specify (as well
as delete files, mkdir directories, etc.).  Keep in mind that this tool
does not attempt to do any of the scan both systems, looking for files
that differ task.

The code is still fairly young and while some of it is pretty good,
other bits show signs of being written in haste.  I've tested it on a
small number of scenarios so far, but nothing exhaustive.

Commands accepted by the tool on stdin (* means not yet tested):

cd REMOTE_DIR [LOCAL_DIR]chdir both sides at once
tmpdir REMOTE_PATH [LOCAL_PATH]  where temp-files go
get REMOTE_FILE [LOCAL_FILE [BASIS_FILE]]rsync to the local system
put LOCAL_FILE [REMOTE_FILE [BASIS_FILE]]rsync to the remote system
mvget REMOTE_FILE [LOCAL_FILE [BASIS_FILE]]  get, then delete REMOTE_FILE
mvput LOCAL_FILE [REMOTE_FILE [BASIS_FILE]]  put, then delete LOCAL_FILE
del FILE delete a remote file
ldel FILEdelete a local file
md DIR   create a remote directory
lmd DIR  create a local directory
ln OLDNAME NEWNAME   create a remote hard link
lln OLDNAME NEWNAME  create a local hard link
sln OLDNAME NEWNAME  create a remote symlink
lsln OLDNAME NEWNAME create a local symlink
mkdev NAME NUMBERcreate a remote device*
lmkdev NAME NUMBER   create a local device*
quit quit

Spaces in filenames need to be backslash-quoted, as do backslash
characters.  E.g. get This\ File.txt That\ File.txt

You run the program like this:

rsync_xfer -vv ssh remote.com rsync_xfer -s

This starts up a local rsync_xfer process in double-verbose mode, and
tells it to run the ssh remote.com rsync_xfer -s command.  You can
make this latter command anything you like, as long as it starts up an
rsync_xfer with the slave (-s) option.

If you're feeling brave and you'd like to try it out, feel free, but
treat it like the pre-alpha code that it is.  Also keep in mind that
every time you tell the program to switch from get to put, all the
current outstanding get/put jobs must run to completion before any new
jobs start (which can slow down the transfer by reducing the pipe-lining
of data).

Some of the things yet to do:

- Need a way to override the per-file attributes on files we send (it
  currently preserves the attributes on each source file).
- Need a way to specify/set attributes for non-transferred files, new
  directories, and devices.
- It needs a way to output transfer statistics.
- There needs to be a timeout option.
- It needs some of the error-checking to be polished up.
- Some fatal errors might be better as warnings (if done right).
- There's no retry check if the file changes during the send.
- We need to catch SIGPIPE.
- There needs to be better handling of partially-transferred files.
- The code needs to be broken up into multiple files.
- There's no configuration support (it currently compiles on a modern
  Linux system).
- We might want an option that tells us to connect via socket to a
  particular hostname (instead of running a command).
- The code could use some more verbose-output messages.
- It needs more comments.

The code is here:

http://www.clari.net/~wayne/rsync_xfer.c

You need to have librsync installed or available.  You compile the code
as you would expect:

gcc -g -Wall -o rsync_xfer rsync_xfer.c -lrsync

..wayne..


-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: symmetric mirroring (was testing)

2002-04-30 Thread Michael Salmon

On Thursday, April 25, 2002 07:34:47 PM -0500 Rich Winkel 
[EMAIL PROTECTED] wrote:
+--
| Sorry for the junk mail, it seems my last post was lost in the ether,
| despite being successfully delivered to lists.samba.org (!)
| Very frustrating, since I didn't keep a copy of it.
|
| My question was regarding what might be called symmetric mirroring,
| where two sets of identical files, both being simultaneously updated,
| are periodically syncronized.
|
| Has anyone implemented this?  I posted a script which I think
| would work.  If my original post doesn't show up I guess I'll have
| to type it in again.  At any rate, implementing it in rsync doesn't
| seem like it would be too difficult.
+-X8

This is basically what unison does, its main drawback at present is that it 
doesn't understand hard links, that and the fact that it is written in 
Objective CAML ;^).

/Michael
--
This space intentionally left non-blank.

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



testing

2002-04-25 Thread Rich Winkel

123


-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



symmetric mirroring (was testing)

2002-04-25 Thread Rich Winkel

Sorry for the junk mail, it seems my last post was lost in the ether,
despite being successfully delivered to lists.samba.org (!)
Very frustrating, since I didn't keep a copy of it.

My question was regarding what might be called symmetric mirroring,
where two sets of identical files, both being simultaneously updated,
are periodically syncronized.

Has anyone implemented this?  I posted a script which I think
would work.  If my original post doesn't show up I guess I'll have
to type it in again.  At any rate, implementing it in rsync doesn't
seem like it would be too difficult.

Thanks,
Rich


-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: symmetric mirroring (was testing)

2002-04-25 Thread Martin Pool

On 25 Apr 2002, Rich Winkel [EMAIL PROTECTED] wrote:
 Sorry for the junk mail, it seems my last post was lost in the ether,
 despite being successfully delivered to lists.samba.org (!)
 Very frustrating, since I didn't keep a copy of it.
 
 My question was regarding what might be called symmetric mirroring,
 where two sets of identical files, both being simultaneously updated,
 are periodically syncronized.

If each side updates a non-overlapping set of files you can do this
using --update to push only the newer file to each side.

If single files are updated on both sides you need a content-dependent
way of merging them, which is out of the scope of rsync.  Have a look
at the Unison tool.

--
Martin

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: symmetric mirroring (was testing)

2002-04-25 Thread Rich Winkel

Hi Martin,

I guess I need to be more specific.  I have a unix user who has unix
machines at home and at work.  He wants local access to the same set of
files whether he's at home or at work.

 If each side updates a non-overlapping set of files you can do this
 using --update to push only the newer file to each side.

That much I figured out, but a problem arises when files are created
or deleted.  If a file on A has no counterpart on B, should it be
deleted or copied over to B?  My primitive idea is just to compare
the date stamp (or inode modification time) of the file with the
time of the last rsync, and either copy it over if it's newer, or
delete it.  Not very pretty, I know, but it would suffice for his
purposes, and it would be easy to implement with an option like
--symmetric=date_stamp_file
where date_stamp_file is touched every time rsync runs.
Ideally this would cause rsync to run in a duplex mode where
files could be copied in either direction depending on which is
newer.  But that part sounds like a pretty major project.

 If single files are updated on both sides you need a content-dependent
 way of merging them, which is out of the scope of rsync.  Have a look
 at the Unison tool.

Thanks for the tip, it sounds interesting.  I'll check it out!

Rich


-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html