RE: Rsync: Re: patch to enable faster mirroring of large filesystems

2001-11-29 Thread Keating, Tim
PROTECTED]] Sent: Thursday, November 29, 2001 10:02 AM To: Dave Dykstra Cc: [EMAIL PROTECTED] Subject: Re: Rsync: Re: patch to enable faster mirroring of large filesystems

Re: Rsync: Re: patch to enable faster mirroring of large filesystems

2001-11-29 Thread Dave Dykstra
On Thu, Nov 29, 2001 at 11:02:07AM -0500, Alberto Accomazzi wrote: ... These numbers show that reading the filenames this way rather than using the code in place to deal with the include/exclude list cuts the startup time down to 0 (from 1hr). The actual sending of the filenames is down from

Re: Rsync: Re: patch to enable faster mirroring of large filesystems

2001-11-29 Thread Alberto Accomazzi
In message [EMAIL PROTECTED], Dave Dykstra writes: On Thu, Nov 29, 2001 at 11:02:07AM -0500, Alberto Accomazzi wrote: ... These numbers show that reading the filenames this way rather than using the code in place to deal with the include/exclude list cuts the startup time down to 0 (from

Re: Rsync: Re: patch to enable faster mirroring of large filesystems

2001-11-28 Thread Dave Dykstra
On Tue, Nov 27, 2001 at 05:00:14PM -0500, Lenny Foner wrote: ... [ . . . ] I'm pretty sure that rsync won't use up memory for excluded files so it would make no difference. ...though this also implies (since you say it'd probably use basically the same mechanism internally)

Re: Rsync: Re: patch to enable faster mirroring of large filesystems

2001-11-27 Thread Alberto Accomazzi
Dear all, here's my own (renewed) pitch to throw in a --files-from patch. As Dave has suggested in the past, transferring a list of files can be accomplished using --include and --exclude, and has called for people to test the performance gains of his old optimization when using these options

Rsync: Re: patch to enable faster mirroring of large filesystems

2001-11-27 Thread Lenny Foner
Date: Tue, 27 Nov 2001 10:49:11 -0600 From: Dave Dykstra [EMAIL PROTECTED] Thank you very much for doing the test Alberto. I didn't have any set of files that large on which I could do a test, and as I said when I tested the worse case I could think of with my application I

Re: Rsync: Re: patch to enable faster mirroring of large filesystems

2001-11-27 Thread Dave Dykstra
On Tue, Nov 27, 2001 at 02:34:22PM -0500, Lenny Foner wrote: ... I know you're trying to get reliable statistics so it's clear what sort of performance we're talking about here. But may I respectfully suggest that -having- to be so careful about whether optimization actually got turned on is

Re: patch to enable faster mirroring of large filesystems

2001-11-26 Thread Andrew J. Schorr
On Sun, Nov 25, 2001 at 03:21:51AM +1100, Martin Pool wrote: On 20 Nov 2001, Dave Dykstra [EMAIL PROTECTED] wrote: And, by the way, even if the batch stuff accomplishes the same performance gains, I would still argue that the --files-from type of behavior that I implemented is a nice

Re: patch to enable faster mirroring of large filesystems

2001-11-26 Thread Martin Pool
On 26 Nov 2001, Andrew J. Schorr [EMAIL PROTECTED] wrote: I understand your point of view, but I think it is a mistake to hold rsync's algorithm hostage to the directory tree traversal logic built into the program. IMHO, the basic file transfer algorithm of rsync is terrific, but the

patch to enable faster mirroring of large filesystems

2001-11-19 Thread Andrew J. Schorr
I have attached a patch that adds 4 options to rsync that have helped me to speed up my mirroring. I hope this is useful to someone else, but I fear that my relative inexperience with rsync has caused me to miss a way to do what I want without having to patch the code. So please let me know if

Re: patch to enable faster mirroring of large filesystems

2001-11-19 Thread Dave Dykstra
Before I look at this closely, I have a couple questions. First, what options do you use to copy? I once saw somebody who went through a lot of work to cache things and it turned out to be just because he was using the --checksum option when he shouldn't have. Next, have you taken a look at