Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-09-01 Thread Florian Weimer
* Nico Kadel-Garcia: > On Sun, Sep 1, 2019 at 3:52 AM Florian Weimer wrote: >> >> * Neal Gompa: >> >> > Well, technically, the correct thing to do would be for Mock to >> > support downloading bootstrap images to speed up bootstrap. These >> > images could be regularly produced by Fedora

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-09-01 Thread Nico Kadel-Garcia
On Sun, Sep 1, 2019 at 3:52 AM Florian Weimer wrote: > > * Neal Gompa: > > > Well, technically, the correct thing to do would be for Mock to > > support downloading bootstrap images to speed up bootstrap. These > > images could be regularly produced by Fedora infrastructure and made > > available

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-09-01 Thread Neal Gompa
On Sun, Sep 1, 2019 at 3:51 AM Florian Weimer wrote: > > * Neal Gompa: > > > Well, technically, the correct thing to do would be for Mock to > > support downloading bootstrap images to speed up bootstrap. These > > images could be regularly produced by Fedora infrastructure and made > > available

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-09-01 Thread Florian Weimer
* Neal Gompa: > Well, technically, the correct thing to do would be for Mock to > support downloading bootstrap images to speed up bootstrap. These > images could be regularly produced by Fedora infrastructure and made > available on the mirror network for all targets we support as a > project.

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-09-01 Thread Neal Gompa
On Sun, Sep 1, 2019 at 3:31 AM Florian Weimer wrote: > > * Kevin Fenzi: > > > On 8/29/19 11:44 PM, Miroslav Suchý wrote: > >> Dne 29. 08. 19 v 18:58 Kamil Dudka napsal(a): > >>> What is the recommended way to build Fedora 31+ RPMs on a RHEL-7 host? > >> > >> I think that no one contemplate

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-09-01 Thread Florian Weimer
* Kevin Fenzi: > On 8/29/19 11:44 PM, Miroslav Suchý wrote: >> Dne 29. 08. 19 v 18:58 Kamil Dudka napsal(a): >>> What is the recommended way to build Fedora 31+ RPMs on a RHEL-7 host? >> >> I think that no one contemplate supporting RHEL 7 regarding zstd. The real >> thing is "how to build

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-08-30 Thread Kevin Fenzi
On 8/29/19 11:44 PM, Miroslav Suchý wrote: > Dne 29. 08. 19 v 18:58 Kamil Dudka napsal(a): >> What is the recommended way to build Fedora 31+ RPMs on a RHEL-7 host? > > I think that no one contemplate supporting RHEL 7 regarding zstd. The real > thing is "how to build Fedora 31+ RPMs on a >

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-08-30 Thread Kamil Dudka
On Friday, August 30, 2019 8:44:39 AM CEST Miroslav Suchý wrote: > I think that no one contemplate supporting RHEL 7 regarding zstd. Panu said that the backport would be almost trivial: https://lists.pagure.io/archives/list/devel@lists.fedoraproject.org/message/Z4EFFNJNVAP35KH74N3W5GC4E2AQXC7V/

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-08-30 Thread Kamil Dudka
On Thursday, August 29, 2019 11:17:18 PM CEST Nico Kadel-Garcia wrote: > Sent from my iPhone > > > > On Aug 29, 2019, at 12:58 PM, Kamil Dudka wrote: > > > > > >> On Tuesday, June 4, 2019 9:43:53 AM CEST Panu Matilainen wrote: > >> > >>> On 6/4/19 9:56 AM, Daniel Mach wrote: > >>> > >>> Dne

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-08-30 Thread Miroslav Suchý
Dne 29. 08. 19 v 18:58 Kamil Dudka napsal(a): > What is the recommended way to build Fedora 31+ RPMs on a RHEL-7 host? I think that no one contemplate supporting RHEL 7 regarding zstd. The real thing is "how to build Fedora 31+ RPMs on a RHEL-8 host"?

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-08-29 Thread Nico Kadel-Garcia
Sent from my iPhone > On Aug 29, 2019, at 12:58 PM, Kamil Dudka wrote: > >> On Tuesday, June 4, 2019 9:43:53 AM CEST Panu Matilainen wrote: >>> On 6/4/19 9:56 AM, Daniel Mach wrote: >>> >>> Dne 31. 05. 19 v 2:15 Pavel Raiskup napsal(a): >>> > On Thursday, May 30, 2019 10:38:25 AM CEST

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-08-29 Thread Kamil Dudka
On Tuesday, June 4, 2019 9:43:53 AM CEST Panu Matilainen wrote: > On 6/4/19 9:56 AM, Daniel Mach wrote: > > > Dne 31. 05. 19 v 2:15 Pavel Raiskup napsal(a): > > > >> On Thursday, May 30, 2019 10:38:25 AM CEST Miroslav Suchý wrote: > >> > >>> Dne 29. 05. 19 v 23:52 Josh Boyer napsal(a): > >>> >

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-06-25 Thread Tomas Mraz
On Tue, 2019-06-25 at 07:16 -0400, Nico Kadel-Garcia wrote: > On Wed, Jun 19, 2019 at 9:31 AM Panu Matilainen > wrote: > > On 6/19/19 1:51 PM, Aleš Matěj wrote: > > > > At this point, the drpm library is the only blocker for zstd > > > > payloads, > > > > since createrepo_c needs to be able to

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-06-25 Thread Nico Kadel-Garcia
On Wed, Jun 19, 2019 at 9:31 AM Panu Matilainen wrote: > > On 6/19/19 1:51 PM, Aleš Matěj wrote: > > > >> At this point, the drpm library is the only blocker for zstd payloads, > >> since createrepo_c needs to be able to handle zstd drpms. > > > > I looked into the drpm library and I should be

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-06-19 Thread Panu Matilainen
On 6/19/19 1:51 PM, Aleš Matěj wrote: At this point, the drpm library is the only blocker for zstd payloads, since createrepo_c needs to be able to handle zstd drpms. I looked into the drpm library and I should be able to add the zstd support (and make sure it works with createrepo_c)

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-06-19 Thread Martin Kolman
On Wed, 2019-06-19 at 10:51 +, Aleš Matěj wrote: > > At this point, the drpm library is the only blocker for zstd payloads, > > since createrepo_c needs to be able to handle zstd drpms. > > I looked into the drpm library and I should be able to add the zstd support > (and make sure it works

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-06-19 Thread Aleš Matěj
> At this point, the drpm library is the only blocker for zstd payloads, > since createrepo_c needs to be able to handle zstd drpms. I looked into the drpm library and I should be able to add the zstd support (and make sure it works with createrepo_c) Working on it now.

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-06-14 Thread Neal Gompa
On Thu, May 30, 2019 at 3:07 PM Kevin Fenzi wrote: > > So, most of my concerns have already been mentioned by other folks in > this thread: > > * No rhel7/8 support will annoy people, and also increase burden on > fedora infrastructure since we would have to move our koji hubs to > Fedora instead

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-06-05 Thread Chris Murphy
On Wed, Jun 5, 2019 at 1:38 PM Zbigniew Jędrzejewski-Szmek wrote: > > On Wed, Jun 05, 2019 at 10:01:22AM -0600, Chris Murphy wrote: > > I found this on HN today. While xz is not expressly being used within > > Fedora/Red Hat packaging in an archive context, it does seem to have > > quite a lot of

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-06-05 Thread Zbigniew Jędrzejewski-Szmek
On Wed, Jun 05, 2019 at 10:01:22AM -0600, Chris Murphy wrote: > On Wed, Jun 5, 2019 at 2:11 AM Panu Matilainen wrote: > > > > On 6/5/19 12:53 AM, Chris Murphy wrote: > > > At least for small files, and there are many in any distribution, > > > using a dictionary very well could improve

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-06-05 Thread Chris Murphy
On Wed, Jun 5, 2019 at 2:11 AM Panu Matilainen wrote: > > On 6/5/19 12:53 AM, Chris Murphy wrote: > > At least for small files, and there are many in any distribution, > > using a dictionary very well could improve compression/decompression > > time, compression ratio, more than threads. Adding

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-06-05 Thread Panu Matilainen
On 6/5/19 12:53 AM, Chris Murphy wrote: On Mon, Jun 3, 2019 at 7:01 PM Jason L Tibbitts III wrote: "PM" == Panu Matilainen writes: PM> Note that rpm doesn't support parallel zstd compression, and while PM> it does for xz, that's not even utilized in Fedora. Doing parallel xz compression

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-06-04 Thread Chris Murphy
On Mon, Jun 3, 2019 at 7:01 PM Jason L Tibbitts III wrote: > > > "PM" == Panu Matilainen writes: > > PM> Note that rpm doesn't support parallel zstd compression, and while > PM> it does for xz, that's not even utilized in Fedora. > > Doing parallel xz compression has a surprising cost in

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-06-04 Thread Kevin Kofler
Jason L Tibbitts III wrote: > One problem is that I don't think anyone wants to see any quantifiable > regression in overall package size. Spins still struggle to fit within > fixed media sizes as the package set grows ever larger. The RPM compression method is pretty irrelevant to Spin sizes

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-06-04 Thread Panu Matilainen
On 6/4/19 9:56 AM, Daniel Mach wrote: Dne 31. 05. 19 v 2:15 Pavel Raiskup napsal(a): On Thursday, May 30, 2019 10:38:25 AM CEST Miroslav Suchý wrote: Dne 29. 05. 19 v 23:52 Josh Boyer napsal(a): If we did this, wouldn't it make it very difficult to use tools like mock on RHEL / CentOS 7 to

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-06-04 Thread Daniel Mach
Dne 31. 05. 19 v 2:15 Pavel Raiskup napsal(a): On Thursday, May 30, 2019 10:38:25 AM CEST Miroslav Suchý wrote: Dne 29. 05. 19 v 23:52 Josh Boyer napsal(a): If we did this, wouldn't it make it very difficult to use tools like mock on RHEL / CentOS 7 to build for Fedora 3x? Speaking of Mock:

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-06-04 Thread Daniel Mach
Dne 31. 05. 19 v 2:15 Pavel Raiskup napsal(a): On Thursday, May 30, 2019 10:38:25 AM CEST Miroslav Suchý wrote: Dne 29. 05. 19 v 23:52 Josh Boyer napsal(a): If we did this, wouldn't it make it very difficult to use tools like mock on RHEL / CentOS 7 to build for Fedora 3x? Speaking of Mock:

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-06-04 Thread Panu Matilainen
On 6/4/19 8:31 AM, Panu Matilainen wrote: On 6/4/19 4:00 AM, Jason L Tibbitts III wrote: "PM" == Panu Matilainen writes: PM> Note that rpm doesn't support parallel zstd compression, and while PM> it does for xz, that's not even utilized in Fedora. Doing parallel xz compression has a

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-06-03 Thread Panu Matilainen
On 6/4/19 4:00 AM, Jason L Tibbitts III wrote: "PM" == Panu Matilainen writes: PM> Note that rpm doesn't support parallel zstd compression, and while PM> it does for xz, that's not even utilized in Fedora. Doing parallel xz compression has a surprising cost in compression ratio which gets

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-06-03 Thread Jason L Tibbitts III
> "PM" == Panu Matilainen writes: PM> Note that rpm doesn't support parallel zstd compression, and while PM> it does for xz, that's not even utilized in Fedora. Doing parallel xz compression has a surprising cost in compression ratio which gets worse as the thread count increases (because

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-06-03 Thread Daniel Mach
Dne 30. 05. 19 v 17:14 Peter Robinson napsal(a): What about constrained systems where there's limited CPU, that could be a container or a low end cloud instance without any guarantee of resources or a Raspberry Pi. Raspberry Pi is hard to measure. I'm getting quite random results due to IO, I

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-06-03 Thread Daniel Mach
Dne 30. 05. 19 v 17:14 Peter Robinson napsal(a): What about constrained systems where there's limited CPU, that could be a container or a low end cloud instance without any guarantee of resources or a Raspberry Pi. Raspberry Pi is hard to measure. I'm getting quite random results due to IO, I

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-06-03 Thread Panu Matilainen
On 5/29/19 11:19 PM, Ben Cotton wrote: * The macro for setting the compression is: %define _binary_payload w19.zstdio * The recommended compression level is 19. The builds will take longer, but the additional compression time is negligible in the total build time and it pays off in better

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-31 Thread Kevin Fenzi
On 5/30/19 1:18 PM, Neal Gompa wrote: > I'm actually okay with the thought of Koji hub moving to Fedora. I'd > rather see most of our infra running on Fedora so that we don't get > kneecapped by RHEL moving too slowly. Our transition to Python 3 was > made way more complicated by the fact our

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-31 Thread Samuel Sieb
On 5/31/19 12:53 PM, Chris Murphy wrote: On Fri, May 31, 2019 at 12:40 PM Samuel Sieb wrote: On 5/31/19 6:57 AM, Martin Kolman wrote: I guess we can't just switch what the signature refers to as there are other tools that do this kind of verification on the compressed data, not just

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-31 Thread Chris Murphy
On Fri, May 31, 2019 at 12:40 PM Samuel Sieb wrote: > > On 5/31/19 6:57 AM, Martin Kolman wrote: > > I guess we can't just switch what the signature refers to as there are > > other tools > > that do this kind of verification on the compressed data, not just > > delta-RPM, right ? > > > > So

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-31 Thread drago01
On Thursday, May 30, 2019, Samuel Sieb wrote: > On 5/30/19 1:56 PM, Chris Murphy wrote: > >> On Thu, May 30, 2019 at 8:40 AM Daniel Mach wrote: >> >>> >>> Dne 30. 05. 19 v 0:05 Neal Gompa napsal(a): >>> I'm pretty sure this would break DeltaRPMs, since none of the drpm software

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-31 Thread Samuel Sieb
On 5/31/19 6:57 AM, Martin Kolman wrote: I guess we can't just switch what the signature refers to as there are other tools that do this kind of verification on the compressed data, not just delta-RPM, right ? So maybe, could we attach a second signature computed on the uncompressed payload

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-31 Thread Chris Murphy
On Fri, May 31, 2019 at 7:58 AM Martin Kolman wrote: > I guess we can't just switch what the signature refers to as there are other > tools > that do this kind of verification on the compressed data, not just delta-RPM, > right ? > > So maybe, could we attach a second signature computed on the

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-31 Thread Martin Kolman
- Original Message - > From: "Samuel Sieb" > To: devel@lists.fedoraproject.org > Sent: Thursday, May 30, 2019 11:52:55 PM > Subject: Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd > compression > > On 5/30/19 2:38 PM, Chris Murphy wrote

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-31 Thread Florian Weimer
* Jonathan Dieter: > On Wed, 2019-05-29 at 20:15 -0600, Chris Murphy wrote: >> 'dnf info deltarpm' says >> URL : http://gitorious.org/deltarpm/deltarpm >> which has an expired certificate, but pushing passed that it says >> current version 3.6 is 5 years old. Is this really maintained or

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-31 Thread Tomas Mraz
On Thu, 2019-05-30 at 16:18 -0400, Neal Gompa wrote: > > That said, I'm less happy about the thought that inspecting Fedora > RPMs on RHEL 8 or openSUSE is going to be a royal pain. > Ecosystem-wise, no one really prepared for a distribution to switch > to > zstd so quickly. Thankfully, it's

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-31 Thread Zbigniew Jędrzejewski-Szmek
On Thu, May 30, 2019 at 11:44:23AM -0700, Samuel Sieb wrote: > On 5/30/19 7:54 AM, Daniel Mach wrote: > >>I think before approving such changes, owners need to do mass > >>rebuilds on their own and provide a graph of changes in size > >>between original compression format and new one(s). > >Doing

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-31 Thread Igor Gnatenko
On Fri, May 31, 2019 at 9:36 AM Nicolas Mailhot via devel < devel@lists.fedoraproject.org> wrote: > Le vendredi 31 mai 2019 à 02:15 +0200, Pavel Raiskup a écrit : > > On Thursday, May 30, 2019 10:38:25 AM CEST Miroslav Suchý wrote: > > > Dne 29. 05. 19 v 23:52 Josh Boyer napsal(a): > > > > If we

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-31 Thread Nicolas Mailhot via devel
Le jeudi 30 mai 2019 à 14:29 -0700, Samuel Sieb a écrit : > On 5/30/19 1:56 PM, Chris Murphy wrote: > > On Thu, May 30, 2019 at 8:40 AM Daniel Mach > > wrote: > > > Dne 30. 05. 19 v 0:05 Neal Gompa napsal(a): > > > > I'm pretty sure this would break DeltaRPMs, since none of the > > > > drpm > > >

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-31 Thread Nicolas Mailhot via devel
Le vendredi 31 mai 2019 à 02:15 +0200, Pavel Raiskup a écrit : > On Thursday, May 30, 2019 10:38:25 AM CEST Miroslav Suchý wrote: > > Dne 29. 05. 19 v 23:52 Josh Boyer napsal(a): > > > If we did this, wouldn't it make it very difficult to use tools > > > like > > > mock on RHEL / CentOS 7 to build

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-30 Thread Pavel Raiskup
On Thursday, May 30, 2019 10:38:25 AM CEST Miroslav Suchý wrote: > Dne 29. 05. 19 v 23:52 Josh Boyer napsal(a): > > If we did this, wouldn't it make it very difficult to use tools like > > mock on RHEL / CentOS 7 to build for Fedora 3x? > > Speaking of Mock: > Either the RPM on host need to

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-30 Thread Samuel Sieb
On 5/30/19 2:38 PM, Chris Murphy wrote: On Thu, May 30, 2019 at 3:31 PM Samuel Sieb wrote: On 5/30/19 1:56 PM, Chris Murphy wrote: I have no idea how deltarpm works, but if working on bit level difference on uncompressed data, I don't see why local rebuild needs to use the same compression

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-30 Thread Chris Murphy
On Thu, May 30, 2019 at 3:31 PM Samuel Sieb wrote: > > On 5/30/19 1:56 PM, Chris Murphy wrote: > > On Thu, May 30, 2019 at 8:40 AM Daniel Mach wrote: > >> > >> Dne 30. 05. 19 v 0:05 Neal Gompa napsal(a): > >>> > >>> I'm pretty sure this would break DeltaRPMs, since none of the drpm > >>>

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-30 Thread Samuel Sieb
On 5/30/19 1:56 PM, Chris Murphy wrote: On Thu, May 30, 2019 at 8:40 AM Daniel Mach wrote: Dne 30. 05. 19 v 0:05 Neal Gompa napsal(a): I'm pretty sure this would break DeltaRPMs, since none of the drpm software has been updated to handle zstd compression. Neither drpm nor deltarpm handle it

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-30 Thread Chris Murphy
On Thu, May 30, 2019 at 8:40 AM Daniel Mach wrote: > > Dne 30. 05. 19 v 0:05 Neal Gompa napsal(a): > > > > I'm pretty sure this would break DeltaRPMs, since none of the drpm > > software has been updated to handle zstd compression. Neither drpm nor > > deltarpm handle it today. > > > Thanks for

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-30 Thread Neal Gompa
On Thu, May 30, 2019 at 3:07 PM Kevin Fenzi wrote: > > So, most of my concerns have already been mentioned by other folks in > this thread: > > * No rhel7/8 support will annoy people, and also increase burden on > fedora infrastructure since we would have to move our koji hubs to > Fedora instead

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-30 Thread Kevin Fenzi
So, most of my concerns have already been mentioned by other folks in this thread: * No rhel7/8 support will annoy people, and also increase burden on fedora infrastructure since we would have to move our koji hubs to Fedora instead of RHEL to be able to read the rpms made on builders. (Or ship a

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-30 Thread Samuel Sieb
On 5/30/19 7:54 AM, Daniel Mach wrote: I think before approving such changes, owners need to do mass rebuilds on their own and provide a graph of changes in size between original compression format and new one(s). Doing that equals to the mass rebuild. I'd rather do an analysis and if the

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-30 Thread Adam Jackson
On Thu, 2019-05-30 at 12:20 -0400, Adam Jackson wrote: > - What's the mean and/or median size of an rpm in Fedora, and what > difference in {de,}compression time would that likely experience? Just to follow up on this since it was quick to math out. For Fedora 30's x86_64 repo, various

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-30 Thread Adam Jackson
On Wed, 2019-05-29 at 16:19 -0400, Ben Cotton wrote: > * Faster koji builds (installations in build roots) The numbers here seem to indicate that you'll have faster koji build _setup_. But getting comparable compression rates as xz means spending (apparently) significantly more time at

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-30 Thread Igor Gnatenko
On Thu, May 30, 2019 at 5:01 PM Daniel Mach wrote: > Dne 30. 05. 19 v 8:39 Igor Gnatenko napsal(a): > > Last time I was about to propose this in F29, I did mass-rebuild myself > > and while decompressing was faster in most of the cases, the size was > > definitely worse. So definitely "Lower

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-30 Thread Peter Robinson
On Thu, May 30, 2019 at 4:16 PM Ben Cotton wrote: > > Daniel added more contingency details to the proposal[1], but I want > to call out the contingency date. Right now it is listed as a week > before the final freeze, which is way too late. I would suggest that > the end of the mass rebuild is

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-30 Thread Ben Cotton
Daniel added more contingency details to the proposal[1], but I want to call out the contingency date. Right now it is listed as a week before the final freeze, which is way too late. I would suggest that the end of the mass rebuild is the appropriate contingency point. If a second mass rebuild is

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-30 Thread Peter Robinson
> Dne 30. 05. 19 v 14:57 Kevin Kofler napsal(a): > > Jason L Tibbitts III wrote: > > > >>> "BC" == Ben Cotton writes: > >> BC> * The recommended compression level is 19. The builds will take > >> BC> longer, but the additional compression time is negligible in the > >> BC> total build time

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-30 Thread Peter Robinson
> Dne 30. 05. 19 v 8:39 Igor Gnatenko napsal(a): > > Last time I was about to propose this in F29, I did mass-rebuild myself > > and while decompressing was faster in most of the cases, the size was > > definitely worse. So definitely "Lower bandwidth on mirrors if we choose > > the highest

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-30 Thread Daniel Mach
Dne 30. 05. 19 v 14:57 Kevin Kofler napsal(a): Jason L Tibbitts III wrote: "BC" == Ben Cotton writes: BC> * The recommended compression level is 19. The builds will take BC> longer, but the additional compression time is negligible in the BC> total build time and it pays off in better

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-30 Thread Daniel Mach
Dne 30. 05. 19 v 8:39 Igor Gnatenko napsal(a): Last time I was about to propose this in F29, I did mass-rebuild myself and while decompressing was faster in most of the cases, the size was definitely worse. So definitely "Lower bandwidth on mirrors if we choose the highest compression level"

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-30 Thread Daniel Mach
Dne 30. 05. 19 v 0:05 Neal Gompa napsal(a): I'm pretty sure this would break DeltaRPMs, since none of the drpm software has been updated to handle zstd compression. Neither drpm nor deltarpm handle it today. Thanks for heads-up. We'll look into it and provide a fix soon.

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-30 Thread Daniel Mach
Dne 29. 05. 19 v 23:52 Josh Boyer napsal(a): On Wed, May 29, 2019 at 4:20 PM Ben Cotton wrote: https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_compression = Switch RPMs to zstd compression = == Summary == Binary RPMs are currently compressed with xz level 2. Switching to zstd

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-30 Thread Vitaly Zaitsev via devel
Hello, Ben Cotton. Wed, 29 May 2019 16:19:48 -0400 you wrote: > Switching to zstd would increase decompression speed significantly. Good change, but it will significantly increase Delta RPM rebuild process especially on HDD, that's why drpm should be disabled by default to achieve maximum

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-30 Thread Kamil Paral
On Wed, May 29, 2019 at 10:20 PM Ben Cotton wrote: > https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_compression > > = Switch RPMs to zstd compression = > > == Summary == > Binary RPMs are currently compressed with xz level 2. > Switching to zstd would increase decompression speed

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-30 Thread Kevin Kofler
Jason L Tibbitts III wrote: >> "BC" == Ben Cotton writes: > BC> * The recommended compression level is 19. The builds will take > BC> longer, but the additional compression time is negligible in the > BC> total build time and it pays off in better compression ratio than xz > BC> lvl2 has. >

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-30 Thread Neal Gompa
On Wed, May 29, 2019 at 9:27 PM Josh Boyer wrote: > > On Wed, May 29, 2019 at 6:07 PM Neal Gompa wrote: > > > > On Wed, May 29, 2019 at 5:53 PM Josh Boyer > > wrote: > > > > > > On Wed, May 29, 2019 at 4:20 PM Ben Cotton wrote: > > > > > > > >

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-30 Thread Miroslav Suchý
Dne 29. 05. 19 v 23:52 Josh Boyer napsal(a): > If we did this, wouldn't it make it very difficult to use tools like > mock on RHEL / CentOS 7 to build for Fedora 3x? Speaking of Mock: Either the RPM on host need to understand the new format/compression **or** the packages in @buildsys group

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-30 Thread Jonathan Dieter
On Wed, 2019-05-29 at 18:05 -0400, Neal Gompa wrote: > On Wed, May 29, 2019 at 5:53 PM Josh Boyer wrote: > > > > If we did this, wouldn't it make it very difficult to use tools like > > mock on RHEL / CentOS 7 to build for Fedora 3x? Or does RHEL 7 RPM > > support zstd? > > > > We're pretty

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-30 Thread Igor Gnatenko
Last time I was about to propose this in F29, I did mass-rebuild myself and while decompressing was faster in most of the cases, the size was definitely worse. So definitely "Lower bandwidth on mirrors if we choose the highest compression level" is under the question. I think before approving

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-30 Thread Jonathan Dieter
On Wed, 2019-05-29 at 18:32 -0400, James Cassell wrote: > > Would this help with drpms similar to how it helps with faster yum > repo metadata downloads? My biggest problem with drpms is the slow > rebuild speed which is usually slower than my download bandwidth. It > would be a big win if zstd

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-30 Thread Jonathan Dieter
On Wed, 2019-05-29 at 20:15 -0600, Chris Murphy wrote: > 'dnf info deltarpm' says > URL : http://gitorious.org/deltarpm/deltarpm > which has an expired certificate, but pushing passed that it says > current version 3.6 is 5 years old. Is this really maintained or > updatabled? Upstream

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-29 Thread John Reiser
If we did this, wouldn't it make it very difficult to use tools like mock on RHEL / CentOS 7 to build for Fedora 3x? Or does RHEL 7 RPM support zstd? We're pretty much screwed here. Also, since RHEL 8's rpm package does not have zstd support compiled in, it too cannot handle the RPMs.

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-29 Thread Chris Murphy
On Wed, May 29, 2019 at 4:07 PM Neal Gompa wrote: > > This is news to me, as I've never heard of any "side mass rebuilds". > They're prohibitively expensive to do, which is why we do only one per > release anyway. Is it sane to test this in Rawhide now with just new builds? As things get rebuilt

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-29 Thread Chris Murphy
On Wed, May 29, 2019 at 2:20 PM Ben Cotton wrote: > > https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_compression > > = Switch RPMs to zstd compression = > > == Summary == > Binary RPMs are currently compressed with xz level 2. > Switching to zstd would increase decompression speed

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-29 Thread Josh Boyer
On Wed, May 29, 2019 at 6:07 PM Neal Gompa wrote: > > On Wed, May 29, 2019 at 5:53 PM Josh Boyer wrote: > > > > On Wed, May 29, 2019 at 4:20 PM Ben Cotton wrote: > > > > > > https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_compression > > > > > > = Switch RPMs to zstd compression = >

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-29 Thread James Cassell
On Wed, May 29, 2019, at 4:20 PM, Ben Cotton wrote: > https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_compression > > = Switch RPMs to zstd compression = > [...] > == Benefit to Fedora == > * Faster installations/upgrades of user systems > * Faster koji builds (installations in

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-29 Thread Neal Gompa
On Wed, May 29, 2019 at 5:53 PM Josh Boyer wrote: > > On Wed, May 29, 2019 at 4:20 PM Ben Cotton wrote: > > > > https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_compression > > > > = Switch RPMs to zstd compression = > > > > == Summary == > > Binary RPMs are currently compressed with

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-29 Thread Josh Boyer
On Wed, May 29, 2019 at 4:20 PM Ben Cotton wrote: > > https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_compression > > = Switch RPMs to zstd compression = > > == Summary == > Binary RPMs are currently compressed with xz level 2. > Switching to zstd would increase decompression speed

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-29 Thread Jason L Tibbitts III
> "BC" == Ben Cotton writes: BC> * The change requires setting a new compression algorithm in rpm BC> macros. Then a mass rebuild of all packages is required. Technically there is no harm if a mass rebuild is not done; there will simply be no benefit for packages which aren't rebuilt.

Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression

2019-05-29 Thread Stuart D. Gathman
On Wed, 29 May 2019, Ben Cotton wrote: https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_compression = Switch RPMs to zstd compression = == Summary == Binary RPMs are currently compressed with xz level 2. Switching to zstd would increase decompression speed significantly. ...