Re: Fedora 40 beta freeze now over

On Tue, 2024-04-02 at 16:55 -0700, Kevin Fenzi wrote: > On Tue, Apr 02, 2024 at 09:28:31PM +0100, Jonathan Dieter wrote: > >  * Alternatively, we could update whatever's calling createrepo_c > > to add the `f` prefix to all non-rawhide builds. > > I like this option. ;) &

Re: Fedora 40 beta freeze now over

On Sat, 2024-03-30 at 09:39 -0700, Kevin Fenzi wrote: > On Fri, Mar 29, 2024 at 11:32:10PM +0000, Jonathan Dieter wrote: > > On Wed, 2024-03-27 at 09:12 -0700, Kevin Fenzi wrote: > > > Our next freeze is for Fedora 40 Final, currently scheduled for > > > 2024-0

Re: Fedora 40 beta freeze now over

On Wed, 2024-03-27 at 09:12 -0700, Kevin Fenzi wrote: > Our next freeze is for Fedora 40 Final, currently scheduled for > 2024-04-02, which is NEXT TUESDAY! Could you please update fedora-repo-zdicts to 2403.1 on the server(s) used to generate the metadata? This will reduce the size of the

Re: SQLAlchemy integration in Flask

Sorry for taking so long to reply.  I'm afraid I don't check this mailing list as often as I should. :) On Tue, 2021-12-07 at 08:52 +0100, Aurelien Bompard wrote: > Thanks for your input! > > > 1. We're using a clustered database (CockroachDB, for those who > > care) > > that uses optimistic

Re: SQLAlchemy integration in Flask

On Mon, 2021-12-06 at 18:36 +0100, Aurelien Bompard wrote: > Anyway, this long email is about finding a common ground for > SQLAlchemy integration in Flask, while taking into account our > difficult experiences with webframewoks in the past, but not being > locked in them. Is there something that

Re: Freeze break request: Re: Can we update fedora-repo-zdicts on the branched and rawhide composers?

On Wed, 2021-09-08 at 09:14 -0700, Kevin Fenzi wrote: > > I've updated it. > > kevin I can confirm that the latest F35 repodata has the dictionaries now. Thanks so much! Jonathan signature.asc Description: This is a digitally signed message part

Can we update fedora-repo-zdicts on the branched and rawhide composers?

Since branching, I've put out a new version of fedora-repo-zdicts with dictionaries for F35 and updated dictionaries for Rawhide. This version (2108.1) is now available in all active Fedora/EPEL branches, I think. Can we update fedora-repo-zdicts on the branched and rawhide composers so they get

Re: Please don't update zchunk to 1.1.14 on servers where createrepo_c is run

On Wed, 2021-06-02 at 09:35 -0700, Kevin Fenzi wrote: > On Tue, Jun 01, 2021 at 09:13:30PM +0100, Jonathan Dieter wrote: > > A major bug in zchunk-1.1.14 was flagged up to me today.  If zchunk- > > 1.1.14 (on a system with zstd 1.5.0+) is used to create a zck file with > &

Please don't update zchunk to 1.1.14 on servers where createrepo_c is run

A major bug in zchunk-1.1.14 was flagged up to me today. If zchunk- 1.1.14 (on a system with zstd 1.5.0+) is used to create a zck file with a zdict, the file will be impossible to decompress. Embarrassingly, the tests weren't testing this combination. The good news is that this doesn't affect

Re: Can we please update fedora-repo-zdicts on the metadata generation servers for F34 zchunk dictionaries?

On Sat, 2021-04-03 at 21:46 +0100, Jonathan Dieter wrote: > On Sat, 2021-04-03 at 11:09 -0700, Kevin Fenzi wrote: > > ok. I've installed fedora-repo-zdicts on both branched and rawhide > > composers. > > > > Lets see if that works in tomorrow's compose. > >

Re: Can we please update fedora-repo-zdicts on the metadata generation servers for F34 zchunk dictionaries?

On Sat, 2021-04-03 at 11:09 -0700, Kevin Fenzi wrote: > ok. I've installed fedora-repo-zdicts on both branched and rawhide > composers. > > Lets see if that works in tomorrow's compose. Thanks so much! Fingers crossed. :) Jonathan signature.asc Description: This is a digitally signed

Can we please update fedora-repo-zdicts on the metadata generation servers for F34 zchunk dictionaries?

Right now, we're not using zdicts for the F34 zchunk metadata because they were only added in fedora-repo-zdicts-2103.1-2 (which should now be in the updates repo in all current Fedora releases). If we could update fedora-repo-zdicts to 2103.1-2 on whichever servers generate the metadata

Re: [PATCH] bodhi-backend: Make sure zchunk dicts are installed

On Thu, 2019-05-23 at 17:01 -0400, Randy Barlow wrote: > On Thu, 2019-05-23 at 10:33 -0700, Kevin Fenzi wrote: > > Applied. Thanks. > > One note: The patch to do zchunking is part of Bodhi 4.0.0, which is > not yet in production; we plan to deploy it on Tuesday. Unless I'm mistaken, that patch

Re: [PATCH] bodhi-backend: Make sure zchunk dicts are installed

On Sun, 2019-05-19 at 21:25 +0100, Jonathan Dieter wrote: > The zchunk dictionaries used to reduce the size of zchunk metadata seems to > not currently be installed on the bodhi server. This patch makes sure they > are installed. > > Signed-off-by: Jonathan Dieter > --- >

[PATCH] bodhi-backend: Make sure zchunk dicts are installed

The zchunk dictionaries used to reduce the size of zchunk metadata seems to not currently be installed on the bodhi server. This patch makes sure they are installed. Signed-off-by: Jonathan Dieter --- roles/bodhi2/backend/tasks/main.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/roles

Re: [PATCH] bodhi-backend: Add zchunk support to updates and updates-testing repositories

On Thu, 2019-04-11 at 18:08 -0700, Kevin Fenzi wrote: > On 4/9/19 11:20 AM, Jonathan Dieter wrote: > > On Tue, 2019-04-09 at 19:14 +0100, Jonathan Dieter wrote: > > > This re-adds zchunk support for the updates and updates-testing > > > repositories > &g

Re: [PATCH] bodhi-backend: Add zchunk support to updates and updates-testing repositories

On Tue, 2019-04-09 at 19:14 +0100, Jonathan Dieter wrote: > This re-adds zchunk support for the updates and updates-testing repositories > for both rpms and modularity. > > Zchunk metadata was turned off due to a broken version of librepo that made it > out to stable, but a fixed v

[PATCH] bodhi-backend: Add zchunk support to updates and updates-testing repositories

on. 1: https://pagure.io/fesco/issue/2116 Signed-off-by: Jonathan Dieter --- roles/bodhi2/backend/templates/pungi.module.conf.j2 | 3 +++ roles/bodhi2/backend/templates/pungi.rpm.conf.j2| 3 +++ 2 files changed, 6 insertions(+) diff --git a/roles/bodhi2/backend/templates/pungi.module.conf.j2

Re: [Freeze Break Request] Add zchunk support to updates and updates-testing repositories

On Sun, 2019-03-31 at 10:37 -0700, Kevin Fenzi wrote: > On 3/31/19 10:35 AM, Jonathan Dieter wrote: > > On Sun, 2019-03-31 at 10:28 -0700, Kevin Fenzi wrote: > > > On 3/31/19 1:56 AM, Jonathan Dieter wrote: > > > > On Sun, 2019-03-31 at 09:09 +0100, Jonath

Re: [Freeze Break Request] Add zchunk support to updates and updates-testing repositories

On Sun, 2019-03-31 at 10:28 -0700, Kevin Fenzi wrote: > On 3/31/19 1:56 AM, Jonathan Dieter wrote: > > On Sun, 2019-03-31 at 09:09 +0100, Jonathan Dieter wrote: > > > Due to an unrelated *major* bug in the latest librepo update ( > > > https://bugzilla.redhat.com/show_bu

Re: [Freeze Break Request] Add zchunk support to updates and updates-testing repositories

On Sun, 2019-03-31 at 09:09 +0100, Jonathan Dieter wrote: > Due to an unrelated *major* bug in the latest librepo update ( > https://bugzilla.redhat.com/show_bug.cgi?id=1694411), I'd like to > request that we disable zchunk metadata generation in updates and > updates-testing unti

Re: [Freeze Break Request] Add zchunk support to updates and updates-testing repositories

On Sun, 2019-03-31 at 05:13 +, Peter Robinson wrote: > On Sun, Mar 31, 2019 at 6:01 AM Kevin Fenzi wrote: > > On 3/30/19 9:50 PM, Peter Robinson wrote: > > > > Great, thanks! I'll be keeping an eye on the composes to see if there > > > > are any issues. > > > > > > Wasn't this disabled in

Re: [Freeze Break Request] Add zchunk support to updates and updates-testing repositories

On Sat, 2019-03-30 at 16:00 -0700, Kevin Fenzi wrote: > On 3/30/19 3:32 PM, Jonathan Dieter wrote: > > On Sat, 2019-03-30 at 15:13 -0700, Kevin Fenzi wrote: > > > On 3/30/19 11:35 AM, Jonathan Dieter wrote: > > > > > > > Stephen and Kevin, thanks so much!

Re: [Freeze Break Request] Add zchunk support to updates and updates-testing repositories

On Sat, 2019-03-30 at 15:13 -0700, Kevin Fenzi wrote: > On 3/30/19 11:35 AM, Jonathan Dieter wrote: > > > Stephen and Kevin, thanks so much! > > Can you rebase and attach the patch? > > It's not applying cleanly for me... if not I can try and manually poke > it l

[Freeze Break Request] Add zchunk support to updates and updates-testing repositories

Rebased patch against master ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List

[PATCH] Add zchunk support to updates and updates-testing repositories

This adds zchunk support for the updates and updates-testing repositories for both rpms and modularity Signed-off-by: Jonathan Dieter --- roles/bodhi2/backend/templates/pungi.module.conf.j2 | 3 +++ roles/bodhi2/backend/templates/pungi.rpm.conf.j2| 3 +++ 2 files changed, 6 insertions

Re: [Freeze Break Request] Add zchunk support to updates and updates-testing repositories

On Sat, 2019-03-30 at 14:05 -0400, Stephen John Smoogen wrote: > +1 > > On Sat, 30 Mar 2019 at 13:53, Kevin Fenzi wrote: > > On 3/29/19 1:33 PM, Jonathan Dieter wrote: > > > On Mon, 2019-03-11 at 20:23 +, Jonathan Dieter wrote: > > > > On Mon, 2019-03-

Re: [Freeze Break Request] Add zchunk support to updates and updates-testing repositories

On Mon, 2019-03-11 at 20:23 +, Jonathan Dieter wrote: > On Mon, 2019-03-11 at 11:24 -0700, Kevin Fenzi wrote: > > On 3/11/19 12:26 AM, Jonathan Dieter wrote: > > > This adds zchunk support for the updates and updates-testing > > > repositories for both rpms and m

Re: [Freeze Break Request] Add zchunk support to updates and updates-testing repositories

On Mon, 2019-03-11 at 11:24 -0700, Kevin Fenzi wrote: > On 3/11/19 12:26 AM, Jonathan Dieter wrote: > > This adds zchunk support for the updates and updates-testing > > repositories for both rpms and modularity. We already have zchunk > > metadata being generated for the fe

[Freeze Break Request] Add zchunk support to updates and updates-testing repositories

-by: Jonathan Dieter --- roles/bodhi2/backend/templates/pungi.module.conf.j2 | 3 +++ roles/bodhi2/backend/templates/pungi.rpm.conf.j2| 3 +++ 2 files changed, 6 insertions(+) diff --git a/roles/bodhi2/backend/templates/pungi.module.conf.j2 b/roles/bodhi2/backend/templates/pungi.module.conf.j2 index

Re: How do we turn zchunk on for updates-testing for F30?

On Sun, 2019-03-10 at 15:47 +, Peter Robinson wrote: > git send-email so it's inline on the list for easy review. Thanks for the tip! Just sent it using git send-email. Jonathan ___ infrastructure mailing list --

[PATCH] Add zchunk support to updates and updates-testing repositories

This adds zchunk support for the updates and updates-testing repositories for both rpms and modularity Signed-off-by: Jonathan Dieter --- roles/bodhi2/backend/templates/pungi.module.conf.j2 | 3 +++ roles/bodhi2/backend/templates/pungi.rpm.conf.j2| 3 +++ 2 files changed, 6 insertions

Re: How do we turn zchunk on for updates-testing for F30?

On Sat, 2019-03-09 at 21:29 +0100, Mikolaj Izdebski wrote: > On Sat, Mar 9, 2019 at 2:29 PM Jonathan Dieter > wrote: > > Hey, I just noticed that, while we have zchunked metadata for the > > F30 > > base repository, it's not enabled to for updates-testing. > > >

Re: How do we turn zchunk on for updates-testing for F30?

On Sat, 2019-03-09 at 09:43 -0500, Neal Gompa wrote: > On Sat, Mar 9, 2019 at 8:28 AM Jonathan Dieter > wrote: > > Hey, I just noticed that, while we have zchunked metadata for the > > F30 > > base repository, it's not enabled to for updates-testing. > > >

How do we turn zchunk on for updates-testing for F30?

Hey, I just noticed that, while we have zchunked metadata for the F30 base repository, it's not enabled to for updates-testing. I've looked in the ansible repo and in pungi, but I can't see where createrepo_c is actually called for updates-testing. Can someone please point me in the right

Re: Enabling zchunk metadata generation in F30

On Fri, 2018-12-14 at 15:15 -0500, Randy Barlow wrote: > On Fri, 2018-12-14 at 19:02 +0000, Jonathan Dieter wrote: > > Hey Randy, at the moment the --zck option *only* applies to > > primary.xml, filelists.xml and other.xml. It should be pretty > > straightforward to add it

Re: Enabling zchunk metadata generation in F30

On Fri, 2018-12-14 at 19:24 +, Jonathan Dieter wrote: > On Fri, 2018-12-14 at 11:13 -0800, Kevin Fenzi wrote: > > On 12/14/18 10:52 AM, Jonathan Dieter wrote: > > > I suspect that the maintainers would like to see this feature tested > > > more before pushing it

Re: Enabling zchunk metadata generation in F30

On Fri, 2018-12-14 at 11:13 -0800, Kevin Fenzi wrote: > On 12/14/18 10:52 AM, Jonathan Dieter wrote: > > On Thu, 2018-12-13 at 16:42 -0800, Kevin Fenzi wrote: > > > Cool. > > > > > > I see the new createrepo_c only has a rawhide build... any chance for a

Re: Enabling zchunk metadata generation in F30

On Fri, 2018-12-14 at 11:20 -0500, Randy Barlow wrote: > On Thu, 2018-12-13 at 22:56 +0000, Jonathan Dieter wrote: > > The call to createrepo_c or mergerepo_c > > (whichever is run last to generate the final metadata) would need to > > be > > run with the new zchunk argu

Re: Enabling zchunk metadata generation in F30

On Thu, 2018-12-13 at 16:42 -0800, Kevin Fenzi wrote: > On 12/13/18 3:34 PM, Jonathan Dieter wrote: > > On Thu, 2018-12-13 at 15:12 -0800, Kevin Fenzi wrote: > > > pungi calls createrepo_c for us (in both rawhide/branched and updates) > > > so we need a pungi patch (pr

Re: Enabling zchunk metadata generation in F30

On Thu, 2018-12-13 at 15:12 -0800, Kevin Fenzi wrote: > pungi calls createrepo_c for us (in both rawhide/branched and updates) > so we need a pungi patch (probibly with a config option?) to enable > this. If it's added as a optional thing we would need to add that > setting to our pungi-fedora

Enabling zchunk metadata generation in F30

Createrepo_c in F30 has finally grown zchunk support and I've packaged up some zdicts that we can use for F30/rawhide, so I'd love to see us start building zchunk metadata for F30. To enable zchunk metadata generation, whichever systems are running createrepo_c/mergerepo_c for Rawhide would need

Best path for Fedora zchunk dictionaries

I'm currently in the process of packaging up fedora-repo-zdicts[1], a package which will contain the zchunk dictionaries for all active Fedora releases. When running createrepo_c or mergerepo_c with zchunk support, the directory containing the zdicts is passed in and createrepo_c will choose the

Re: Patch review request: zchunk patches for dnf, libsolv and librepo

On Tue, 2018-06-12 at 12:21 +0300, Jonathan Dieter wrote: > I would love to get these changes into Fedora 29, and the code is > testable now, but with only three weeks until System-Wide change > proposals are due, I'm not sure if I'm being ambitious. FWIW, I have a COPR availabl

Re: [Rpm-ecosystem] Patch review request: zchunk patches for dnf, libsolv and librepo

On Tue, 2018-06-12 at 05:24 -0400, Neal Gompa wrote: > On Tue, Jun 12, 2018 at 5:21 AM Jonathan Dieter wrote: > > > > I've finally finished writing patches to integrate zchunk support into > > dnf/libsolv/librepo[1], and I'd greatly appreciate some code review. A > >

Re: [Rpm-ecosystem] Patch review request: zchunk patches for dnf, libsolv and librepo

On Tue, 2018-06-12 at 05:24 -0400, Neal Gompa wrote: > On Tue, Jun 12, 2018 at 5:21 AM Jonathan Dieter wrote: > > > > I've finally finished writing patches to integrate zchunk support into > > dnf/libsolv/librepo[1], and I'd greatly appreciate some code review. A > >

Patch review request: zchunk patches for dnf, libsolv and librepo

I've finally finished writing patches to integrate zchunk support into dnf/libsolv/librepo[1], and I'd greatly appreciate some code review. A vast majority of the code is in librepo, but libsolv has been expanded to support zchunk files and dnf has a tiny patch that passes the base cache

Old deltarpms are being thrown away on each compose (was Re: dnf and deltarpm)

On Thu, 2018-05-31 at 22:34 +0100, Tomasz Kłoczko wrote: > Just checked on few mirrors usual location of f28 updates > (/pub/linux/dist/fedora/linux/updates/28/Everything/x86_64/drpms) and > in this directory there are at the moment only 56 files from May 31 > and nothing older. So not two drpm

Librepo/dnf zchunk integration question

Zchunk works by comparing an old version of the file with the one you want to download, but when dnf refreshes a repository, it downloads the new file into a temporary directory with no information passed to the handle about where the old files are. I've been trying to keep my code changes in

Re: Zchunk update

ere: https://copr.fedorainfracloud.org/coprs/jdieter/zchunk My next step is to add zchunk support to librepo. A quick summary of the features I wanted to add: On Mon, 2018-04-16 at 15:47 +0300, Jonathan Dieter wrote: > * A python API Still needs to be done. > * GPG signatures in

Re: [Rpm-ecosystem] Zchunk update

On Mon, 2018-04-23 at 00:27 -0400, Neal Gompa wrote: > On Tue, Apr 17, 2018 at 3:05 PM, Jonathan Dieter <jdie...@gmail.com> wrote: > > I'm assuming that you're referring here to getting zchunk packaged into > > Fedora. I'd really like to finalize the file format (we're close,

Re: Zchunk update

On Tue, 2018-04-17 at 17:39 +0200, Michal Novotny wrote: > On Tue, Apr 17, 2018 at 4:20 PM, Jonathan Dieter <jdie...@gmail.com> wrote: > > On Tue, 2018-04-17 at 09:08 +0200, Michal Novotny wrote: > > > Hello Jonathan, > > > > > > Once it is in creater

Re: Zchunk update

On Tue, 2018-04-17 at 09:08 +0200, Michal Novotny wrote: > Hello Jonathan, > > On Mon, Apr 16, 2018 at 2:47 PM, Jonathan Dieter <jdie...@gmail.com> > wrote: > > It's been a number of weeks since my last update, so I thought I'd > > let > > everyone know wher

Re: [Rpm-ecosystem] Zchunk update

On Mon, 2018-04-16 at 09:00 -0400, Neal Gompa wrote: > On Mon, Apr 16, 2018 at 8:47 AM, Jonathan Dieter <jdie...@gmail.com> wrote: > > I've also added zchunk support to createrepo_c (see > > https://github.com/jdieter/createrepo_c), but I haven't yet created a > > pull r

Proposed zchunk file format - V4

Here's version four with a swap from fixed-length integers to variable- length compressed integers which allow us to skip compression of the index (since the non-integer data is all uncompressable checksums). I've also added the uncompressed size of each chunk to the index to make it easier to

Zchunk update

It's been a number of weeks since my last update, so I thought I'd let everyone know where things are at. I've spent most of these last few weeks reworking zchunk's API to make it easier to use and more in line with what other compression tools use, and I'm mostly happy with it now. Writing a

Re: Proposed zchunk file format - V3

CC'ing fedora-infrastructure, as I think they got lost somewhere along the way. On Tue, 2018-03-20 at 17:04 +0100, Michal Domonkos wrote: > Yeah, the level doesn't really matter much. My point was, as long as > we chunk, some of the data that we will be downloading we will already > have

Re: Initial pre-alpha version of zchunk available for testing and comments

On Thu, 2018-03-22 at 11:55 +0200, Jonathan Dieter wrote: > I've got a working zchunk library, complete with some utilities at > https://github.com/jdieter/zchunk, but I wanted to get some feedback > before I went much further. It's only dependencies are libcurl and > (optional

Initial pre-alpha version of zchunk available for testing and comments

I've got a working zchunk library, complete with some utilities at https://github.com/jdieter/zchunk, but I wanted to get some feedback before I went much further. It's only dependencies are libcurl and (optionally, but very heavily recommended) libzstd. There are test files in

Proposed zchunk file format - V2

Neal, thanks for the feedback. After taking your comments into consideration, here's version 2. +-+-+-+-+-+--+-+-+-+-+-+-+-+-+ |ID | Compression type | Index size | +-+-+-+-+-+--+-+-+-+-+-+-+-+-+ +==+=+ | Compressed

Proposed zchunk file format

So here's my proposed file format for the zchunk file. Should I add some flags to facilitate possible different compression formats? +-+-+-+-+-+-+-+-+-+-+-+-+==+=+ | ID | Index size | Compressed Index | Compressed Dict |

Re: A proof-of-concept for delta'ing repodata

On Tue, 2018-02-13 at 10:52 +0100, Igor Gnatenko wrote: > What about zstd? Also in latest version of lz4 there is support for > dictionaries too. So I've investigated zstd, and, here are my results: Latest F27 primary.gz - 3.1MB zlib zchunk (including custom dict) primary.zck - 4.2MB ~35%

Re: A proof-of-concept for delta'ing repodata

On Wed, 2018-02-14 at 09:56 -0800, Kevin Fenzi wrote: > ...snip... > > I think it sounds interesting, but you should get buyin from dnf folks > and/or PackageKit folks and see if they can agree to use this format. Do you know if there's a dedicated list for dnf or PackageKit development (a quick

Re: A proof-of-concept for delta'ing repodata

On Tue, 2018-02-13 at 10:52 +0100, Igor Gnatenko wrote: > On Mon, 2018-02-12 at 23:53 +0200, Jonathan Dieter wrote: > > * Many changes to the metadata can mean a large number of ranges > >requested. I ran a check on our mirrors, and three (out of around > >15

Re: A proof-of-concept for delta'ing repodata

On Mon, 2018-02-12 at 23:53 +0200, Jonathan Dieter wrote: > <tl;dr> > I've come up with a method of splitting repodata into chunks that can > be downloaded and combined with chunks that are already on the local > system to create a byte-for-byte copy of the compressed

A proof-of-concept for delta'ing repodata

I've come up with a method of splitting repodata into chunks that can be downloaded and combined with chunks that are already on the local system to create a byte-for-byte copy of the compressed repodata. Tools and scripts are at: https://www.jdieter.net/downloads/ Background: With DNF,