debdelta , Re: effectiveness of rsync and apt

2006-05-20 Thread A Mennucc1
hi I recollected my earlier experiments on computing package diffs, and prepared a package called 'debdelta'. It is available at http://tonelli.sns.it/pub/mennucc1/debdelta and also uploaded into experimental. This package contains both a command 'debdelta', to compute deltas of Debian

Re: effectiveness of rsync and apt

2006-05-10 Thread A Mennucc
hi I had the same idea some time ago if you ever decide to work on that, I may help Goswin von Brederlow wrote: I actualy have a little hack how one could implement patch debs now to test this out: 1. Create an archive mirror with rsync batch files (or xdelta or whatever) between the

Re: effectiveness of rsync and apt

2006-05-04 Thread Marc Haber
On Mon, 01 May 2006 09:30:55 +0200, Florian Weimer [EMAIL PROTECTED] wrote: The downside is that anything that doesn't work on entire .debs is very likely to change them at the byte stream level (you only need to use slightly different zlib versions or parameters). This means that the chain of

Re: effectiveness of rsync and apt

2006-05-04 Thread Goswin von Brederlow
Marc Haber [EMAIL PROTECTED] writes: On Mon, 01 May 2006 09:30:55 +0200, Florian Weimer [EMAIL PROTECTED] wrote: The downside is that anything that doesn't work on entire .debs is very likely to change them at the byte stream level (you only need to use slightly different zlib versions or

Re: effectiveness of rsync and apt

2006-05-02 Thread A Mennucc
hi I did a similar thing some time ago; I used 'xdelta' on two versions of kernel and of tetex; the results were impressive; I could prepare a 'debdiff' that was 10% (AFAICR) of the size, and that would recreate an exact copy of the new version of the package, given the previous version of the

Re: effectiveness of rsync and apt

2006-05-02 Thread Goswin von Brederlow
Tyler MacDonald [EMAIL PROTECTED] writes: Goswin von Brederlow [EMAIL PROTECTED] wrote: Bittorrent has a per chunk hash so it can validate each chunk when it recieves it instead of waiting for the full file. It won't see if a chunk is present at some other position in the file, not even if

Re: effectiveness of rsync and apt

2006-05-02 Thread Goswin von Brederlow
Peter Samuelson [EMAIL PROTECTED] writes: * Goswin von Brederlow: Look at zsync and help develope it far enough so it can look into debs. Without that the gain is practicaly 0 or less. The other thing to do would be to lobby for dpkg-deb and dpkg-source to use 'gzip --rsyncable' when

Re: effectiveness of rsync and apt

2006-05-02 Thread Goswin von Brederlow
Pierre Habouzit [EMAIL PROTECTED] writes: Le Lun 1 Mai 2006 15:31, Brian Eaton a écrit : On 4/30/06, Goswin von Brederlow wrote: Look at zsync and help develope it far enough so it can look into debs. Without that the gain is practicaly 0 or less. It's entirely possible that the gain

Re: effectiveness of rsync and apt

2006-05-02 Thread Darren Salt
I demand that Pierre Habouzit may or may not have written... [snip; delta packages?] The real question is: do people clean their apt cache or not? I do, because after a full X.org/kde/openoffice upgrade, it takes quite a lot of disk in /var (that is small on my computers). And with that cache

Re: effectiveness of rsync and apt

2006-05-02 Thread Darren Salt
I demand that Andreas Barth may or may not have written... * Brian Eaton ([EMAIL PROTECTED]) [060501 19:21]: On 5/1/06, Andreas Barth [EMAIL PROTECTED] wrote: Or you could create the diffdebs before upload or on ftp-master, and include the diffdebs somehow in the Packages file (so they're

Re: effectiveness of rsync and apt

2006-05-02 Thread Andreas Barth
* Darren Salt ([EMAIL PROTECTED]) [060502 19:03]: I demand that Andreas Barth may or may not have written... * Brian Eaton ([EMAIL PROTECTED]) [060501 19:21]: On 5/1/06, Andreas Barth [EMAIL PROTECTED] wrote: Or you could create the diffdebs before upload or on ftp-master, and include the

Re: effectiveness of rsync and apt

2006-05-01 Thread Florian Weimer
* Goswin von Brederlow: Tyler MacDonald [EMAIL PROTECTED] writes: Brian Eaton [EMAIL PROTECTED] wrote: http://rsync.samba.org/rsync-and-debian/rsync-and-debian.html Has anyone ever done some log file analysis to figure out how much bandwidth would be saved by transferring package deltas

Re: effectiveness of rsync and apt

2006-05-01 Thread Peter Samuelson
* Goswin von Brederlow: Look at zsync and help develope it far enough so it can look into debs. Without that the gain is practicaly 0 or less. The other thing to do would be to lobby for dpkg-deb and dpkg-source to use 'gzip --rsyncable' when building stuff. (That, or sneak --rsyncable

Re: effectiveness of rsync and apt

2006-05-01 Thread Martijn van Oosterhout
On 5/1/06, Brian Eaton [EMAIL PROTECTED] wrote: Hello all - Regarding the ideas discussed here: http://rsync.samba.org/rsync-and-debian/rsync-and-debian.html A few comments: 3.2 rsync is too hard on servers That document claims it's an avoidable cost. It's not really, because the client

Re: effectiveness of rsync and apt

2006-05-01 Thread Brian Eaton
On 4/30/06, Goswin von Brederlow [EMAIL PROTECTED] wrote: Brian Eaton [EMAIL PROTECTED] wrote: http://rsync.samba.org/rsync-and-debian/rsync-and-debian.html Has anyone ever done some log file analysis to figure out how much bandwidth would be saved by transferring package deltas instead of

Re: effectiveness of rsync and apt

2006-05-01 Thread Andreas Barth
* Brian Eaton ([EMAIL PROTECTED]) [060501 15:51]: The only time delta packages will be a win is for upgrades where the client has the original package cached. If one does it right, it might be enough if the original package is *installed*. And that happens quite often, e.g. even for security

Re: effectiveness of rsync and apt

2006-05-01 Thread Pierre Habouzit
Le Lun 1 Mai 2006 15:31, Brian Eaton a écrit : On 4/30/06, Goswin von Brederlow wrote: Look at zsync and help develope it far enough so it can look into debs. Without that the gain is practicaly 0 or less. It's entirely possible that the gain will be nothing no matter what algorithm is

Re: effectiveness of rsync and apt

2006-05-01 Thread Brian Eaton
On 5/1/06, Andreas Barth [EMAIL PROTECTED] wrote: * Brian Eaton ([EMAIL PROTECTED]) [060501 15:51]: The only time delta packages will be a win is for upgrades where the client has the original package cached. If one does it right, it might be enough if the original package is *installed*. And

Re: effectiveness of rsync and apt

2006-05-01 Thread Pierre Habouzit
Le Lun 1 Mai 2006 16:35, Brian Eaton a écrit : On 5/1/06, Andreas Barth [EMAIL PROTECTED] wrote: * Brian Eaton ([EMAIL PROTECTED]) [060501 15:51]: The only time delta packages will be a win is for upgrades where the client has the original package cached. If one does it right, it

Re: effectiveness of rsync and apt

2006-05-01 Thread Andreas Barth
* Brian Eaton ([EMAIL PROTECTED]) [060501 16:42]: On 5/1/06, Andreas Barth [EMAIL PROTECTED] wrote: * Brian Eaton ([EMAIL PROTECTED]) [060501 15:51]: The only time delta packages will be a win is for upgrades where the client has the original package cached. If one does it right, it might

Re: effectiveness of rsync and apt

2006-05-01 Thread Andreas Barth
* Brian Eaton ([EMAIL PROTECTED]) [060501 17:49]: On 5/1/06, Andreas Barth [EMAIL PROTECTED] wrote: * Brian Eaton ([EMAIL PROTECTED]) [060501 16:42]: On 5/1/06, Andreas Barth [EMAIL PROTECTED] wrote: If one does it right, it might be enough if the original package is *installed*. And

Re: effectiveness of rsync and apt

2006-05-01 Thread Brian Eaton
On 5/1/06, Andreas Barth [EMAIL PROTECTED] wrote: * Brian Eaton ([EMAIL PROTECTED]) [060501 16:42]: On 5/1/06, Andreas Barth [EMAIL PROTECTED] wrote: If one does it right, it might be enough if the original package is *installed*. And that happens quite often, e.g. even for security

Re: effectiveness of rsync and apt

2006-05-01 Thread Brian Eaton
On 5/1/06, Andreas Barth [EMAIL PROTECTED] wrote: Or you could create the diffdebs before upload or on ftp-master, and include the diffdebs somehow in the Packages file (so they're signed as well by the usual mechanismn). My initial view is that any delta package system that doesn't reproduce

Re: effectiveness of rsync and apt

2006-05-01 Thread Andreas Barth
* Brian Eaton ([EMAIL PROTECTED]) [060501 19:21]: On 5/1/06, Andreas Barth [EMAIL PROTECTED] wrote: Or you could create the diffdebs before upload or on ftp-master, and include the diffdebs somehow in the Packages file (so they're signed as well by the usual mechanismn). My initial view is

effectiveness of rsync and apt

2006-04-30 Thread Brian Eaton
Hello all - Regarding the ideas discussed here: http://rsync.samba.org/rsync-and-debian/rsync-and-debian.html Has anyone ever done some log file analysis to figure out how much bandwidth would be saved by transferring package deltas instead of entire new packages? Assuming someone hasn't done

Re: effectiveness of rsync and apt

2006-04-30 Thread Tyler MacDonald
Brian Eaton [EMAIL PROTECTED] wrote: http://rsync.samba.org/rsync-and-debian/rsync-and-debian.html Has anyone ever done some log file analysis to figure out how much bandwidth would be saved by transferring package deltas instead of entire new packages? Slightly off-topic, but I

Re: effectiveness of rsync and apt

2006-04-30 Thread Goswin von Brederlow
Tyler MacDonald [EMAIL PROTECTED] writes: Brian Eaton [EMAIL PROTECTED] wrote: http://rsync.samba.org/rsync-and-debian/rsync-and-debian.html Has anyone ever done some log file analysis to figure out how much bandwidth would be saved by transferring package deltas instead of entire new

Re: effectiveness of rsync and apt

2006-04-30 Thread Tyler MacDonald
Goswin von Brederlow [EMAIL PROTECTED] wrote: Bittorrent has a per chunk hash so it can validate each chunk when it recieves it instead of waiting for the full file. It won't see if a chunk is present at some other position in the file, not even if that position is also on chunk boundaries.