Re: [Rpm-ecosystem] Proposed zchunk file format - V3

2018-03-12 Thread Jonathan Dieter
On Mon, 2018-03-12 at 15:42 +0100, Michal Domonkos wrote: > Hi Jonathan, > > To me, the zchunk idea looks good. > > Incidentally, for the last couple of months, I have been trying to > rethink the way we cache metadata on the clients, as part of the > libdnf (re)design efforts. My goal was to

Re: [Rpm-ecosystem] Proposed zchunk file format

2018-03-02 Thread Michael Schroeder
On Fri, Mar 02, 2018 at 02:33:09PM +0200, Jonathan Dieter wrote: > No, I didn't expect it to have much effect. Since openSUSE's xml file > are (presumably) ordered so new packages come last, do you have any old > primary.xml files lying around that I can test? > > If not, I'll grab them from the

Re: [Rpm-ecosystem] Proposed zchunk file format

2018-03-02 Thread Jonathan Dieter
On Thu, 2018-03-01 at 10:12 +, Michael Schroeder wrote: > On Wed, Feb 28, 2018 at 09:31:39AM +0200, Jonathan Dieter wrote: > > Ok, here are some numbers comparing zsync and zchunk. For testing, I > > have eight f27-updates primary.xml files dating from Dec 7 until Feb > > 12. 3-6 are on

Re: [Rpm-ecosystem] Proposed zchunk file format - V3

2018-02-28 Thread Jonathan Dieter
I've been working on a C implementation of this spec, and came up with a few other changes. I think it's important to have a checksum of the index as well as the data as we want to be able to verify that the index is as expected before trying to parse it. I've also added in the ability to use a

Re: [Rpm-ecosystem] Proposed zchunk file format

2018-02-27 Thread Jonathan Dieter
On Fri, 2018-02-23 at 14:14 +, Michael Schroeder wrote: > This may be an unfair question, but how does it compare to the > 'gzip --rsyncable' + zsync approach that we (openSUSE) are > using since almost eight years? I guess it's better, but how much? Ok, here are some numbers comparing zsync

Re: [Rpm-ecosystem] Proposed zchunk file format

2018-02-26 Thread Michael Schroeder
On Fri, Feb 23, 2018 at 11:15:40PM +0200, Jonathan Dieter wrote: > On Fri, 2018-02-23 at 14:14 +, Michael Schroeder wrote: > > Hi Jonathan! > > > > On Fri, Feb 16, 2018 at 08:52:23PM +0200, Jonathan Dieter wrote: > > > So here's my proposed file format for the zchunk file. Should I > > > add

Re: [Rpm-ecosystem] Proposed zchunk file format

2018-02-26 Thread Michael Schroeder
On Fri, Feb 23, 2018 at 03:23:00PM -0500, Colin Walters wrote: > > > On Fri, Feb 23, 2018, at 9:14 AM, Michael Schroeder wrote: > > > This may be an unfair question, but how does it compare to the > > 'gzip --rsyncable' + zsync approach that we (openSUSE) are > > using since almost eight years?

Re: [Rpm-ecosystem] Proposed zchunk file format

2018-02-23 Thread Jonathan Dieter
On Fri, 2018-02-23 at 14:14 +, Michael Schroeder wrote: > Hi Jonathan! > > On Fri, Feb 16, 2018 at 08:52:23PM +0200, Jonathan Dieter wrote: > > So here's my proposed file format for the zchunk file. Should I > > add > > some flags to facilitate possible different compression formats? > > >

Re: [Rpm-ecosystem] Proposed zchunk file format

2018-02-23 Thread Colin Walters
On Fri, Feb 23, 2018, at 3:54 PM, Jonathan Dieter wrote: > On Fri, 2018-02-23 at 15:23 -0500, Colin Walters wrote: > > > And I don't see any zsync files in e.g.: > > http://download.opensuse.org/distribution/leap/42.3/repo/oss/suse/ > > I found a copy of zsync at

Re: [Rpm-ecosystem] Proposed zchunk file format

2018-02-23 Thread Jonathan Dieter
On Fri, 2018-02-23 at 15:23 -0500, Colin Walters wrote: > And I don't see any zsync files in e.g.: > http://download.opensuse.org/distribution/leap/42.3/repo/oss/suse/ I found a copy of zsync at https://download.opensuse.org/repositories/n etwork/openSUSE_Tumbleweed/src/zsync-0.6.2-35.23.src.rpm

Re: [Rpm-ecosystem] Proposed zchunk file format

2018-02-23 Thread Colin Walters
On Fri, Feb 23, 2018, at 9:14 AM, Michael Schroeder wrote: > This may be an unfair question, but how does it compare to the > 'gzip --rsyncable' + zsync approach that we (openSUSE) are > using since almost eight years? I guess it's better, but how much? Where is that code? `git grep zsync` in