Re: Dynamic hash table size (with static has load)

2006-02-27 Thread John Van Essen
On Sun, 26 Feb 2006, Wayne Davison [EMAIL PROTECTED] wrote: This would be such an improvement for really large files that I'm considering putting this into 2.6.7pre3. However, I want to be sure that it is in rock-solid shape first. If anyone wants to help with the testing and/or the

Re: Dynamic hash table size (with static has load)

2006-02-27 Thread Wayne Davison
On Sun, Feb 26, 2006 at 06:52:22PM -0800, Wayne Davison wrote: I want to be sure that it is in rock-solid shape first. FYI, there was one bug I found in the patch that I fixed: the gettag2() macro needed to mask the s1 variable to generate a proper tag value. I fixed this by just getting rid of

Re: Dynamic hash table size (with static has load)

2006-02-26 Thread Shachar Shemesh
Wayne Davison wrote: http://rsync.samba.org/ftp/unpacked/rsync/patches/dynamic_hash.diff A line of credit would have been nice :-) One thing this patch does is to (1) leave the array allocated to its largest size, (2) use realloc() if we need to make it bigger, (3) make the minimum

Re: Dynamic hash table size (with static has load)

2006-02-26 Thread Wayne Davison
On Sun, Feb 26, 2006 at 10:34:25AM +0200, Shachar Shemesh wrote: A line of credit would have been nice :-) Sure 'nuff. I think we can save the memory when handling smaller files. Yeah, I've been leaning the same way, so I like your change here. I also like your reason for realloc() being the

Re: Dynamic hash table size (with static has load)

2006-02-26 Thread Matt McCutchen
On Sun, 2006-02-26 at 18:52 -0800, Wayne Davison wrote: This would be such an improvement for really large files If the new hash table can handle really large numbers of blocks really well, it might actually be practical for rsync to load all files in the transfer into the hash table at the

Re: Dynamic hash table size (with static has load)

2006-02-25 Thread Wayne Davison
On Sat, Feb 25, 2006 at 01:25:52PM +0200, Shachar Shemesh wrote: Attached is a patch that uses a non-predetermined hash table size, so that the hash cell load (alpha) is never more than 80%. Thanks for the patch! Here's some comments: - You didn't change the size of the tag typedef (an

Re: Dynamic hash table size (with static has load)

2006-02-25 Thread Shachar Shemesh
Wayne Davison wrote: Thanks for the patch! Here's some comments: - You didn't change the size of the tag typedef (an unsigned short), and your patch makes the value potentially overflow. Gotcha. I'm sending an amended patch. - For smaller hash-table sizes, your algorithm does a lookup

Re: Dynamic hash table size (with static has load)

2006-02-25 Thread Wayne Davison
On Sat, Feb 25, 2006 at 08:42:37PM +0200, Shachar Shemesh wrote: I disagree. I think you may have meant to say was, Utterly, totally, and in all meaningful ways false. :-) In my hasty read-through of your patch I was obviously only thinking about powers of 2, so my second critique completely