Re: Oops more testing was required....
> "jws" == jw schultz <[EMAIL PROTECTED]> > wrote the following on Fri, 27 Jun 2003 19:49:22 -0700 jws> Long term, i think the bwlimit stuff needs a complete jws> reexamination. In addition to the sleeping times and spreading jws> the load as in the "smoother bandwidth limiting" but also its jws> converse, Craig's buffering. I suspect the approach is to have jws> a network write routine that deals with buffering, write jws> splitting and sleeping in a coordinated way. Do you think it is necessary for rsync to have to have bandwidth control at all? It seems this should be someone else's responsibility. For instance, if rsync is run over a pipe, something like cstream or throttle could be used (I don't know if these are really suitable, but there's no reason a separate utility couldn't do it). Even for the daemon, maybe it could be run over some bandwidth limiting proxy or something, if the OS couldn't limit the bandwidth directly. -- Ben Escoto pgp0.pgp Description: PGP signature -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Oops more testing was required....
On Wed, Jun 18, 2003 at 01:28:32PM +0200, Rogier Wolff wrote: > On Wed, Jun 18, 2003 at 09:09:59PM +1000, Martin Pool wrote: > > On 17 Jun 2003, Rogier Wolff <[EMAIL PROTECTED]> wrote: > > > > > > Oops. Missed one line in the last patch > > > > Thankyou. That looks good. > > > > If we're going to make this more accurate it might be worthwhile to > > actually look at how long we really did sleep for, and use that to > > adjust time_to_sleep rather than resetting to zero. > > > > Also I'd prefer the variable be called micros_to_sleep or > > us_to_sleep. Small point I know. > > OK. Agreed. > > If I'm going to do "gettimeofday" calls, I might just as well keep a > variable that keeps the "time until we've used our quotum". > > Whenever we're ready to "sleep", we'll get the current time, and only > sleep for the difference. > > The trouble is, that if we end up using 5 seconds worth of CPU time, > we should not suddenly try to catch up for those whole 5 seconds > > I like to set bwlimit to about 90% of my link capacity. This causes > rsync not to fill the link: filling the link causes my provider to > drop my packets. Also rsync won't fill my queues leading to higher > latencies on interactive traffic. > > But if we allow bursting of those 5 seconds that we didn't have any > data, we'll fill the link to the brim for 50 seconds! (if there > suddenly is a nice supply of fresh data) This is undesirable. So, I'm > still thinking on how to make this more accurate, but prevent big > bursts. I'm considering not allowing more than bwlimit of > backlog. (c.f. leaky bucket bandwidth limiting in the packet filtering > world) I'd suggest experimentation with shaping your own traffic. lartc.org looks like it has some good stuff on it particularly wondershaper. Long term, i think the bwlimit stuff needs a complete reexamination. In addition to the sleeping times and spreading the load as in the "smoother bandwidth limiting" but also its converse, Craig's buffering. I suspect the approach is to have a network write routine that deals with buffering, write splitting and sleeping in a coordinated way. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: windows port
On Fri, Jun 27, 2003 at 06:14:47PM -0400, lackluster lackluster wrote: > Looks like I'm going to have to port parts of or all of rsync to win2k. Has > this been done before? Any pointers? Why? That's a rhetorical question, you don't have to justify it to me. Given cygwin and other existing tools it shouldn't be a "have to" thing. Buy a case each of aspirin and antacid :) Read the list archives with attention to any non-unix systems. I recall someone did a partial DOS port a while back. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
windows port
Looks like I'm going to have to port parts of or all of rsync to win2k. Has this been done before? Any pointers? _ The new MSN 8: advanced junk mail protection and 2 months FREE* http://join.msn.com/?page=features/junkmail -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync does not sync files older than current year
On Fri, Jun 27, 2003 at 06:12:59PM +0200, Thomas Fuerle wrote: > Dear sirs, > > I installed rsync from www.sunfreeware.com on a Solaris 8 box, which > seems to be 2.5.6, but it does not sync files correctly, which elder > than the current year. see the following > > # ls -al httpd.conf > -rw-r--r-- 1 bin bin36935 Nov 9 2002 httpd.conf > # hostname > anui23 > # ls -al /net/anui24/usr/local/apache/conf/httpd.conf > -r--r--r-- 1 root other 41435 Jun 25 19:41 > /net/anui24/usr/local/apac > he/conf/httpd.conf > # rsync -axv --delete /net/anui24/usr/local/apache /usr/local/apache/ > > & /tmp/a > pache.txt > # tail -f /tmp/apache.txt > building file list ... done > wrote 11897 bytes read 20 bytes 7944.67 bytes/sec > total size is 9305753 speedup is 780.88 > ^C > # ls -al httpd.conf > -rw-r--r-- 1 bin bin36935 Nov 9 2002 httpd.conf > # uname -a > SunOS anui23 5.8 Generic_108528-16 sun4u sparc SUNW,Sun-Fire-280R > > any help is appreciated ... do a ls -l /usr/local/apache/apache/conf/httpd.conf -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync does not sync files older than current year
Dear sirs, I installed rsync from www.sunfreeware.com on a Solaris 8 box, which seems to be 2.5.6, but it does not sync files correctly, which elder than the current year. see the following # ls -al httpd.conf -rw-r--r-- 1 bin bin36935 Nov 9 2002 httpd.conf # hostname anui23 # ls -al /net/anui24/usr/local/apache/conf/httpd.conf -r--r--r-- 1 root other 41435 Jun 25 19:41 /net/anui24/usr/local/apac he/conf/httpd.conf # rsync -axv --delete /net/anui24/usr/local/apache /usr/local/apache/ > & /tmp/a pache.txt # tail -f /tmp/apache.txt building file list ... done wrote 11897 bytes read 20 bytes 7944.67 bytes/sec total size is 9305753 speedup is 780.88 ^C # ls -al httpd.conf -rw-r--r-- 1 bin bin36935 Nov 9 2002 httpd.conf # uname -a SunOS anui23 5.8 Generic_108528-16 sun4u sparc SUNW,Sun-Fire-280R any help is appreciated ... best regards, Thomas -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --exclude-from works but "exclude from" in rsyncd.conf doesn't ?
[The mailing-list software appears to have choked on my last reply, so I'll try again...] On Thu, Jun 26, 2003 at 05:05:04PM -0700, jw schultz wrote: > I know it isn't your patch adding it but these literals are really > more opaque than the code deserves. I'm thinking we/i should replace > them with #defines that indicate meaning. That would help make the code a bit more readable, so I went ahead and created some defines for these old literals. > We probably shouldn't be reporting to the client that we have matched > a server-side pattern. It apparently doesn't. I haven't tried to figure out exactly why, but I tested it with huge levels of 'v', and the server doesn't report the exclusions it's doing. > | + if (check_exclude_file(fname, S_ISDIR(st.st_mode) != 0, exclude_level)) > > Is != 0 necessary here? I think so -- for portability. This bit me once long ago when some system (I forget which) returned the literal bitfield value for these macros (unlike on Linux, which returns truth values). A high enough bit value can get truncated to 0 if cast to too-small a size. I guess I'm paranoid about that sort of thing these days -- passing the value as an int should _probably_ be safe, but I like to make sure that these macros get turned into real truth values if they are assigned to a variable or passed as an arg. > | + return 0 && list && name && name_is_dir; > > Maybe i'm having a brain-fart but how does this ever return > anything but 0? It doesn't -- it always returns 0. It's just stub code in the test wrapper. The routine doesn't actually get called, so it just references its args to avoid compiler warnings. Yeah, I should have added some comments to that effect -- fixed. > The patch looks like it contains some unrelated refactoring > work in util.c. There's a little bit of minor improvements that I made while working on the code for this patch. The majority of these changes are necessary to make sure that the server's exclusions get applied to the command-line args that the user specifies. Without this, the user could get past a "- /foo/" restriction by specifying host::module/foo/file. There are certainly a couple brace twiddles and some minor optimizations that I made (such as not calling strlen() over and over in a loop), but the rest of the changes are necessary (such as the fix to push_dir() that keeps it from adding "/." onto the end of the saved path -- that interferes with the new root-level exclusion matching). I've updated the patch with the result of your feedback. Thanks! http://www.blorf.net/rsync-daemon-exclude.patch ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Cygwin Rsync 2.5.6 over SSH hangs on Win2k->Win2k
> -Original Message-
> From: [EMAIL PROTECTED]
> [mailto:[EMAIL PROTECTED]
> Sent: Friday, June 27, 2003 1:48 AM
> To: Tillman, James
> Cc: '[EMAIL PROTECTED]'
> Subject: Re: Cygwin Rsync 2.5.6 over SSH hangs on Win2k->Win2k
>
>
[...]
> While I haven't found the root cause (and I don't know any cygwin
> internals), the following workaround solves the problem for me.
> Replace msleep() wit sleep():
>
> while ( 1 ) {
> sleep(1);
> }
>
> There's something about signals and msleep() that cygwin
> doesn't grok...
>
> Craig
>
The sleep(1) mod didn't work for me. I had a hung process within an hour
(processes run every 5 minutes). I'm going to try applying Anthony
Heading's patch, which was posted earlier this morning (Subject: PATCH/RFC:
Another stab at the Cygwin hang problem) to see if it works for me. I've
got nothing to lose at this point!
I had been questioning the wisdom of the whole kill/signal usage, myself,
but I'm certainly not experienced enough at this point to have a worthy
opinion in such matters... There's almost certainly a reason it was done
that way.
jpt
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
bug? -z option and large compressed data
Hi,
I think I found a bug in usage of zlib. rsync 2.5.6 with -z fails
like bellow.
% cp install-disk2.iso /var/tmp/install-disk2.iso
install-disk2.iso100% |*| 316 MB00:56
% rsync -vIz install-disk2.iso 127.0.0.1:/var/tmp/install-disk2.iso
install-disk2.iso
deflate on token returned 0 (16384 bytes left)
rsync error: error in rsync protocol data stream (code 12) at token.c(288)
"install-disk2.iso" is a CD-ROM image of "Turbo Linux Installer" and
this CD-ROM includes many gzip/bzip2 compressed datas(RPMs).
Let's looked at token.c.
280 tx_strm.next_in = (Bytef *) map_ptr(buf, offset, toklen);
281 tx_strm.avail_in = toklen;
282 tx_strm.next_out = (Bytef *) obuf;
283 tx_strm.avail_out = MAX_DATA_COUNT;
284 r = deflate(&tx_strm, Z_INSERT_ONLY);
285 if (r != Z_OK || tx_strm.avail_in != 0) {
286 rprintf(FERROR, "deflate on token returned %d (%d bytes left)\n",
287 r, tx_strm.avail_in);
288 exit_cleanup(RERR_STREAMIO);
289 }
MAX_DATA_COUNT is 16383 and toklen may be 16384. So, when deflate()'s
outputs data length is bigger than input, then above code will fail.
Normally output data length is smaller than input, but it is not true
to already compressed data.
--yasuoka
Yasuoka Masahiko([EMAIL PROTECTED])
***
"gdb output"
(gdb) run --version
Starting program: /home/yasuoka/datacube/src/bin/rsync/rsync --version
rsync version 2.5.6 protocol version 26
Copyright (C) 1996-2002 by Andrew Tridgell and others
Re: patch draft for extended attributes on linux
> "MP" == Martin Pool <[EMAIL PROTECTED]> > wrote the following on Wed, 25 Jun 2003 10:34:38 +1000 MP> This draft patch adds support for transferring extended MP> attributes with a new --xattr option. It ought to work on Linux MP> with XFS or ext2/ext3 filesystems with the SGI/bestbits MP> attribute system. What happens when ACLs are present on the source filesystem? It looks like listxattr will include the "system.xxx" attributes used by acls, and then the destination side will try to set them using setxattr, but this will fail because only "user.xxx" attributes can be set with setxattr. But I could be wrong in a number of places, because I'm not sure about the behavior of setxattr, and I just skimmed the patch quickly. -- Ben Escoto pgp0.pgp Description: PGP signature -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
PATCH/RFC: Another stab at the Cygwin hang problem
Hi,
In http://sources.redhat.com/ml/cygwin/2002-09/msg01155.html, I noted that
the often-observed hangs of rsync under Cygwin were assuaged by a call to
msleep().
After upgrading my Cygwin environment to rsync 2.5.6, I'm seeing these
hangs again, not surprisingly given a CVS entry for main.c notes that
this kludge was not harmless:
Revision 1.162 / (download) - annotate - [select for diffs] ,
Tue Jan 28 05:05:53 2003 UTC (4 months, 4 weeks ago) by dwd
Remove the Cygwin msleep(100) before the generator kills the receiver,
because it caused the testsuite/unsafe-links test to hang.
So it seems sensible to attempt something a bit more elegant.
And the first question is why kill/signals are being used
being used here at all.
The illustrative patch below I think effects an equivalent synchronization,
but does so by queuing a byte into a pipe rather than sending a signal.
Of course, since it's not currently done this way, I may be overlooking
something obvious. I can't quite see what though, since in the event
that an error occurs then exit_cleanup is available to send SIGUSR1
with extreme prejudice; but if the protocol in fact concludes cleanly
then there really should be no need for an asynchronous notification?
Comments sought, meanwhile I'll test the patch a bit...
Regards
Anthony
*** main.c.Orig Fri Jun 27 15:21:22 2003
--- main.c Fri Jun 27 15:30:09 2003
***
*** 390,395
--- 390,396
int status=0;
int recv_pipe[2];
int error_pipe[2];
+ int cleanup_pipe[2];
extern int preserve_hard_links;
extern int delete_after;
extern int recurse;
***
*** 416,426
--- 417,435
exit_cleanup(RERR_SOCKETIO);
}
+ if (pipe(cleanup_pipe) < 0) {
+ rprintf(FERROR,"cleanup pipe failed in do_recv\n");
+ exit_cleanup(RERR_SOCKETIO);
+ }
+
io_flush();
if ((pid=do_fork()) == 0) {
+ char tmp;
+
close(recv_pipe[0]);
close(error_pipe[0]);
+ close(cleanup_pipe[1]);
if (f_in != f_out) close(f_out);
/* we can't let two processes write to the socket at one time */
***
*** 436,450
write_int(recv_pipe[1],1);
close(recv_pipe[1]);
io_flush();
! /* finally we go to sleep until our parent kills us
! with a USR2 signal. We sleep for a short time as on
! some OSes a signal won't interrupt a sleep! */
! while (msleep(20))
! ;
}
close(recv_pipe[1]);
close(error_pipe[1]);
if (f_in != f_out) close(f_in);
io_start_buffering(f_out);
--- 445,465
write_int(recv_pipe[1],1);
close(recv_pipe[1]);
io_flush();
! do {
! status = read(cleanup_pipe[0], &tmp, 1);
! } while (status == -1 && errno == EINTR);
! if (status != 1) {
! rprintf(FERROR,"cleanup read returned %d in do_recv\n",
status);
! if (status == -1)
! rprintf(FERROR,"with errno %d (%s)\n", errno,
strerror(errno));
! _exit(RERR_PARTIAL);
! }
! _exit(0);
}
close(recv_pipe[1]);
close(error_pipe[1]);
+ close(cleanup_pipe[0]);
if (f_in != f_out) close(f_in);
io_start_buffering(f_out);
***
*** 462,469
io_flush();
io_set_error_fd(-1);
! kill(pid, SIGUSR2);
! wait_process(pid, &status);
return status;
}
--- 477,487
io_flush();
io_set_error_fd(-1);
! write(cleanup_pipe[1], ".", 1);
! if (waitpid(pid, &status, 0) != pid) {
! rprintf(FERROR,"cleanup in do_recv failed\n");
! exit_cleanup(RERR_SOCKETIO);
! }
return status;
}
***
*** 867,878
exit_cleanup(RERR_SIGNAL);
}
- static RETSIGTYPE sigusr2_handler(int UNUSED(val)) {
- extern int log_got_error;
- if (log_got_error) _exit(RERR_PARTIAL);
- _exit(0);
- }
-
static RETSIGTYPE sigchld_handler(int UNUSED(val)) {
#ifdef WNOHANG
int cnt, status;
--- 885,890
***
*** 964,970
orig_argv = argv;
signal(SIGUSR1, sigusr1_handler);
- signal(SIGUSR2, sigusr2_handler);
signal(SIGCHLD, sigchld_handler);
#ifdef MAINTAINER_MODE
signal(SIGSEGV, rsync_panic_handler);
--- 976,981
--
Anthony Heading
This communication is for informational purposes only. It is not intended as
an offer or solicitation for the purchase or sale of any financial instrument
or as an official confirmation of any transaction. All market prices,
