Rsync friendly zlib/gzip compression - revisited

Hi, all - I've been digging into the question of using rsync to sync up compressed data, and read with interest the conversations regarding the rsync friendly patch available for gzip (I belive Rusty provided this patch). For anyone interested, the message thread is archived at

re[2]: Rsync friendly zlib/gzip compression - revisited

is nowhere near as flat. So optimization of the rsync block size for this one type of data may be a good idea. - Kevin Original Message Kevin Day wrote: At that point, I started to consider the question raised in the discussion threads by Kevin Easton - primarily

re[3]: Rsync friendly zlib/gzip compression - revisited

Per my comment yesterday: However, I just kicked off another parametric study varying the rsync block size and the block mask size, while holding the window size constant. If there is an optimization avaialble for rsync block size, then we should see a dependency in these results. The

Wooops - rsyncable patch had a default parameter mistake in it

RSync- Sorry about that. I accidentally reversed the default values I had specified for the window size and the reset block size. This only effects the outcome if you accept all default values, and don't initialize with your own. The correct default values in the deflate.c file (line

Patch for rsyncable zlib with new rolling checksum

Hi all- My test results so far indicate a pretty decent improvement in overall rsync performance when using a slightly more sophisticated checksum calculation. The attached patch has the required changes (in hindsight, I should have compressed this using zlib with the new algorithm :-) ).

re[2]: Rsyncing really large files

Shachar,It does use a hash table. rsync adds the two components of the rolling checksum together to come up with a 16 bit hash, and performs a table lookup. The table contains an offset into the sorted rsync checksum table (which contains both weak and strong checksums), and does a

re[2]: Rsyncing really large files

for big files that don't change much. It may be that the square root of file size for block size isn't appropriate for files as big as you are working with... - K Original Message Kevin Day wrote: I would *strongly* recommend that you dig into the thesis a bit

re[2]: Rsyncing really large files

Shachar- I think Wayne is mostly pointing you to the correct location here. If you look at the code where the checksum is computer, you'll find that the rolling checksum actually consists of two parts - one is the sum of all of the byte values in the window, the other is the offset

re[2]: Encryption

Gary- Shachar Shemesh has created an AES encryption module that generates RSync friendly encrypted files. Google his name and you should be able to find it. - K Original Message I am using a SSH tunnel for transit, so just the backup

re: RSync with File Compression

Drew- There is a build of GZip (under Debian I think) that has an rsync friendly option. I have posted a patch for zlib to this newsgroup with code that makes zlib rsync friendly (along with some extra optimizations over the GZip implementation). Take a look in the list archives and