Re: [Rpm-ecosystem] Some points about zchunk

2018-09-27 Thread Jonathan Dieter
On Thu, 2018-09-27 at 14:55 -0400, Neal Gompa wrote: > On Thu, Sep 27, 2018 at 2:45 PM Jonathan Dieter wrote: > > On Thu, 2018-09-27 at 14:17 -0400, Neal Gompa wrote: > > > DNF is now using libdnf, so you shouldn't need to repeat it twice. > > > > > > But that said, the path you describe makes

Re: [Rpm-ecosystem] Some points about zchunk

2018-09-27 Thread Jonathan Dieter
On Thu, 2018-09-27 at 14:17 -0400, Neal Gompa wrote: > On Thu, Sep 27, 2018 at 1:49 PM Jonathan Dieter wrote: > > Apologies that it's taken so long for me to follow up on this. So, > > I've been working on getting librepo and libdnf up-to-date with this > > change and it's far more difficult

Re: [Rpm-ecosystem] Some points about zchunk

2018-09-27 Thread Jonathan Dieter
On Thu, 2018-08-09 at 10:49 +0200, Jonathan Dieter wrote: > On Sun, 2018-07-15 at 11:25 -0400, Neal Gompa wrote: > > On Thu, Jul 12, 2018 at 3:27 PM Jonathan Dieter > > wrote: > > > I'd go with _zck since is, by default, xml, but, other > > > than > > > that, I think what you (and Michael) are

Re: [Rpm-ecosystem] Some points about zchunk

2018-07-16 Thread Jonathan Dieter
On Sun, 2018-07-15 at 11:25 -0400, Neal Gompa wrote: > On Thu, Jul 12, 2018 at 3:27 PM Jonathan Dieter wrote: > > I'd go with _zck since is, by default, xml, but, other than > > that, I think what you (and Michael) are suggesting makes sense. > > > > Michael, I agree that there should be some

Re: [Rpm-ecosystem] Some points about zchunk

2018-07-15 Thread Neal Gompa
On Thu, Jul 12, 2018 at 3:27 PM Jonathan Dieter wrote: > > On Wed, 2018-07-11 at 07:55 -0400, Neal Gompa wrote: > > On Wed, Jul 11, 2018 at 7:30 AM Michael Schroeder wrote: > > > > > > On Wed, Jul 11, 2018 at 12:23:47PM +0100, Jonathan Dieter wrote: > > > > That's something I didn't think of,

Re: [Rpm-ecosystem] Some points about zchunk

2018-07-11 Thread Jonathan Dieter
On Wed, 2018-07-11 at 11:08 +, Michael Schroeder wrote: > On Wed, Jul 11, 2018 at 11:20:00AM +0100, Jonathan Dieter wrote: > > I must be missing something because I don't understand how that > > follows. As I understand it, dnf requests the primary metadata. > > Librepo then downloads either

Re: [Rpm-ecosystem] Some points about zchunk

2018-07-11 Thread Michael Schroeder
On Wed, Jul 11, 2018 at 11:20:00AM +0100, Jonathan Dieter wrote: > I must be missing something because I don't understand how that > follows. As I understand it, dnf requests the primary metadata. > Librepo then downloads either primary.xml.gz or primary.xml.zck. > Librepo then asks libsolv to

Re: [Rpm-ecosystem] Some points about zchunk

2018-07-11 Thread Jonathan Dieter
On Wed, 2018-07-11 at 08:28 +, Michael Schroeder wrote: > On Tue, Jul 10, 2018 at 02:05:26PM +0100, Jonathan Dieter wrote: > > The top-level tool only needs to deal with the uncompressed metadata. > > dnf/libdnf requests the primary metadata from librepo, which downloads > > the zchunk

Re: [Rpm-ecosystem] Some points about zchunk

2018-07-11 Thread Michael Schroeder
On Tue, Jul 10, 2018 at 02:05:26PM +0100, Jonathan Dieter wrote: > The top-level tool only needs to deal with the uncompressed metadata. > dnf/libdnf requests the primary metadata from librepo, which downloads > the zchunk version, passes it to libsolv which decompresses it and > creates the

Re: [Rpm-ecosystem] Some points about zchunk

2018-07-10 Thread Jonathan Dieter
On Tue, 2018-07-10 at 11:17 +, Michael Schroeder wrote: > On Mon, Jul 09, 2018 at 09:32:13PM +0100, Jonathan Dieter wrote: > > I had originally planned to do something along these lines (I think I > > used primary-zck rather than primary@zchunk), but realized that this > > pushed the "choose

Re: [Rpm-ecosystem] Some points about zchunk

2018-07-10 Thread Neal Gompa
On Tue, Jul 10, 2018 at 7:18 AM Michael Schroeder wrote: > > On Mon, Jul 09, 2018 at 09:32:13PM +0100, Jonathan Dieter wrote: > > I had originally planned to do something along these lines (I think I > > used primary-zck rather than primary@zchunk), but realized that this > > pushed the "choose

Re: [Rpm-ecosystem] Some points about zchunk

2018-07-10 Thread Michael Schroeder
On Mon, Jul 09, 2018 at 09:32:13PM +0100, Jonathan Dieter wrote: > I had originally planned to do something along these lines (I think I > used primary-zck rather than primary@zchunk), but realized that this > pushed the "choose best format" code into the top-level tools, rather > than leaving the

Re: [Rpm-ecosystem] Some points about zchunk

2018-07-09 Thread Jonathan Dieter
On Mon, 2018-07-09 at 08:59 +, Michael Schroeder wrote: > I tought about this a bit more over the weekend, and maybe we > should do this in a bit more general way. Basically zchunk is > just another compression format, like "xz" or "zstd". If we > want to support yet another compression

Re: [Rpm-ecosystem] Some points about zchunk

2018-07-09 Thread Michael Schroeder
On Sun, Jul 08, 2018 at 07:45:36PM +0100, Jonathan Dieter wrote: > On Fri, 2018-07-06 at 11:48 +, Michael Schroeder wrote: > > On Thu, Jul 05, 2018 at 08:07:58PM +0300, Jonathan Dieter wrote: > > > My proposal is here: > > > https://www.jdieter.net/downloads/zchunk/repomd.dtd > > > > > > In

Re: [Rpm-ecosystem] Some points about zchunk

2018-07-08 Thread Jonathan Dieter
On Sun, 2018-07-08 at 19:45 +0100, Jonathan Dieter wrote: > On Fri, 2018-07-06 at 11:48 +, Michael Schroeder wrote: > > Ah, no, I think you misunderstood. Do *not* add md5 support. In fact, > > I'd ask you to remove sha1 support as well to make your code smaller. > > > > My point is that you

Re: [Rpm-ecosystem] Some points about zchunk

2018-07-08 Thread Jonathan Dieter
On Sun, 2018-07-08 at 19:45 +0100, Jonathan Dieter wrote: > On Fri, 2018-07-06 at 11:48 +, Michael Schroeder wrote: > > On Thu, Jul 05, 2018 at 08:07:58PM +0300, Jonathan Dieter wrote: > > > librepo first downloads header-size of the file and then verifies that > > > the header checksum

Re: [Rpm-ecosystem] Some points about zchunk

2018-07-08 Thread Jonathan Dieter
On Fri, 2018-07-06 at 11:48 +, Michael Schroeder wrote: > On Thu, Jul 05, 2018 at 08:07:58PM +0300, Jonathan Dieter wrote: > > My plan was to just keep the same dictionaries (a different one for > > each metadata file) for at least a whole release, if not more. My > > dictionary generation

Re: [Rpm-ecosystem] Some points about zchunk

2018-07-06 Thread Michael Schroeder
On Thu, Jul 05, 2018 at 08:07:58PM +0300, Jonathan Dieter wrote: > My plan was to just keep the same dictionaries (a different one for > each metadata file) for at least a whole release, if not more. My > dictionary generation script > (https://www.jdieter.net/downloads/zchunk-dicts/split.py) >

Re: [Rpm-ecosystem] Some points about zchunk

2018-07-05 Thread Jonathan Dieter
Michael, thank you so much for your detailed review! I really appreciate the time you took to look at this in such detail! I'm currently waiting to board a flight, so I'll make this brief and I'll probably be unavailable until Monday. Comments inline On Thu, 2018-07-05 at 14:18 +, Michael