Re: koji buildsystem changes
On Thu, Dec 17, 2020 at 10:20:38PM +0100, Pavel Raiskup wrote: > On Thursday, December 17, 2020 8:55:48 PM CET Kevin Fenzi wrote: > > On Thu, Dec 17, 2020 at 10:40:24AM +0100, Pavel Raiskup wrote: > > > On Sunday, December 13, 2020 11:02:53 PM CET Kevin Fenzi wrote: > > > > * Finally releng is looking at establishing a sidetag cleanup policy. > > > > A reminder that sidetags should be short lived and only created when > > > > needed. koji must generate buildroot repos for every single sidetag. > > > > ( You can list all your sidetags with 'fedpkg list-side-tags --mine' ) > > > > > > I'm just curious what's the most expensive thing for Koji to implement > > > side-tags, I'd expect something like: > > > > > > - the repositories are cloned (recursively hardlinked/symlinked?) > > > - the override package is added > > > - then createrepo_c is run > > > > I haven't closely looked, but I am pretty sure it's the createrepo_c > > calls. Not that they take that long, but that there has to be one for > > every single buildroot change. ie, if I build foo in rawhide, as soon as > > it lands in the f34 tag, the ~90 f34 side tags all have to run newrepo > > tasks. Repeat many many many times a day. > > We'd have to check the createrepo arguments, but according to that ^^^ > most probably there's some other useful optimization used in > Koji+createrepo. Otherwise Koji wouldn't be able to run createrepo > hundreds of times per day (subsequent, serial runs?) on such a large repo > like rawhide is. Well, it does use --skip-stat --update as you note below. > > > In Copr we don't have to clone the repositories, but we have to run the > > > createrepo_c after each build. In the past the bottleneck used to be > > > the createrepo_c run (recursive walk through all the packages in larger > > > repositories). > > > > > > If Koji suffers from the same problem, perhaps you could take a look at > > > the new createrepo_c option `--recycle-pkglist` - the createrepo_c costs > > > almost nothing then (matter of just reading and later writing the xml > > > metadata). > > > > That would work for the simple case of an updated package, but it > > wouldn't work for new packages added would it? I suppose we could tell > > people if they need to use a new package in their sidetag to delete and > > recreate it? Not sure how much hassle that would be. > > It shouldn't matter (i think) if the package is updated, or newly added. > Createrepo needs still to be explicitly told what metadata for which concrete > RPMs must be added (--pkglist), and what removed (--exclude). > > Anyway, based on the numbers above it looks like e.g. `--skip-stat --update` > is > used in Koji, and the --pkglist is carefully maintained by Koji. I guess that > --recycle-pkglist wouldn't help here. Yeah, koji keeps track of that. > Side note, before I was really curious why parsing and then writing of > medium sized (~20MB) repodata [1] takes about ~20s [2] with '--workers 8' > [3] (mostly no I/O). But I forgot that the repo is compressed, so it's > about 155MB of XML metadata. Considering that only decompression and > compression takes together ~3 seconds on my laptop, and that we still > generate the sqlite database files in Copr (I don't know the overhead of > this), it's likely there's not much space for performance optimizations. > Certainly not some low hanging fruit.. I wonder if '--compression-type gz' would save some... would save making some bz2 files and for koji, size isn't a big deal since builders are close to the hub/packages for the most part. kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: koji buildsystem changes
On Thursday, December 17, 2020 9:04:42 PM CET Kevin Fenzi wrote: > On Thu, Dec 17, 2020 at 10:17:45AM +0100, Pavel Raiskup wrote: > > On Wednesday, December 16, 2020 8:25:50 PM CET Kevin Fenzi wrote: > > > On Tue, Dec 15, 2020 at 10:54:07AM +0100, Miroslav Suchý wrote: > > > > > > > > https://github.com/rpm-software-management/mock/wiki/Feature-container-for-bootstrap > > > > which is still fresh feature - just one year old :) And so far not > > > > enabled in Koji. > > > > > > > > When *this* will be enabled in Koji, we can finally boost the RPM > > > > development to levels previously unimaginable. :) > > > > > > Yeah, we should discuss how to manage and land this. Ideally there would > > > be a lot of CI/QA/Gating on the image, and we would have easy ways to > > > roll back to a known good one in case. > > > > Sounds good! Likely we'd need an image for each chroot and arch. Where > > can we start? :) > > Still a lot of things to ponder here... > > I think it would be important to maintain the ability for users to build > things locally with mock and have them 'close' to what it is in koji. > That would I guess involve publishing all the bootstrap images and > getting mock to be able to download the right ones for the branch/arch. Having it published would be nice. OTOH even now the local mock build is far from what is done in Koji (as far as --enablerepo=local isn't used). And bootstrap can "only" affect the target buildroot installation transactions. > CI/Gating/testing on the container(s). Dnf/RPM should be tested, since that's the only thing used there. > Way for releng to go back to older container(s). > > Way for releng to go to newer/force regeneration of container(s). Sure, as symlink to the latest build repo exists, there could be the 'latest' image tag? > koji adjustments: way to record whats in the container/what container it was. > What does it mean for koji's srpm groups? I guess not much? The groups should be installed to build chroot from bootstrap chroot. > What does this mean for reproducability? Do we keep all containers ever used > for bootstrap so we can duplucate builds? Do we just keep a way to recreate > them? This should normally bring a really low amount of non-determinism, because mock only uses the DNF and RPM (and it's deps) from the bootstrap. How long back could we get if we wanted to restore some random Koji builder state from the past, namely to get the installed host dnf/rpm stack that was used for a concrete build? > Could we just re-use the fedora minimal container? I guess not, but it > would be nice if we could. I'm not sure, is the 'fedora:33' the minimal one? That is used now. But it would be good if 'dnf-plugins-core' was present ... so we could build a thin layer on top of that. > Also, I am guessing this all works with nspawn, we are (still) using old > chroot, we would have to move to nspawn first right? (which we need to do > anyhow). No, fortunately this has nothing to do with nspawn. The --use-bootstrap-image is yet another way to prepopulate bootstrap chroot contents -- and mock can both do chroot() or call systemd-nspawn instead to execute 'dnf' command inside the bootstrap. > I'm sure I'll think of more. :) Yeah, not a trivial thing. Pavel > kevin > ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: koji buildsystem changes
On Thursday, December 17, 2020 8:55:48 PM CET Kevin Fenzi wrote: > On Thu, Dec 17, 2020 at 10:40:24AM +0100, Pavel Raiskup wrote: > > On Sunday, December 13, 2020 11:02:53 PM CET Kevin Fenzi wrote: > > > * Finally releng is looking at establishing a sidetag cleanup policy. > > > A reminder that sidetags should be short lived and only created when > > > needed. koji must generate buildroot repos for every single sidetag. > > > ( You can list all your sidetags with 'fedpkg list-side-tags --mine' ) > > > > I'm just curious what's the most expensive thing for Koji to implement > > side-tags, I'd expect something like: > > > > - the repositories are cloned (recursively hardlinked/symlinked?) > > - the override package is added > > - then createrepo_c is run > > I haven't closely looked, but I am pretty sure it's the createrepo_c > calls. Not that they take that long, but that there has to be one for > every single buildroot change. ie, if I build foo in rawhide, as soon as > it lands in the f34 tag, the ~90 f34 side tags all have to run newrepo > tasks. Repeat many many many times a day. We'd have to check the createrepo arguments, but according to that ^^^ most probably there's some other useful optimization used in Koji+createrepo. Otherwise Koji wouldn't be able to run createrepo hundreds of times per day (subsequent, serial runs?) on such a large repo like rawhide is. > > In Copr we don't have to clone the repositories, but we have to run the > > createrepo_c after each build. In the past the bottleneck used to be > > the createrepo_c run (recursive walk through all the packages in larger > > repositories). > > > > If Koji suffers from the same problem, perhaps you could take a look at > > the new createrepo_c option `--recycle-pkglist` - the createrepo_c costs > > almost nothing then (matter of just reading and later writing the xml > > metadata). > > That would work for the simple case of an updated package, but it > wouldn't work for new packages added would it? I suppose we could tell > people if they need to use a new package in their sidetag to delete and > recreate it? Not sure how much hassle that would be. It shouldn't matter (i think) if the package is updated, or newly added. Createrepo needs still to be explicitly told what metadata for which concrete RPMs must be added (--pkglist), and what removed (--exclude). Anyway, based on the numbers above it looks like e.g. `--skip-stat --update` is used in Koji, and the --pkglist is carefully maintained by Koji. I guess that --recycle-pkglist wouldn't help here. Side note, before I was really curious why parsing and then writing of medium sized (~20MB) repodata [1] takes about ~20s [2] with '--workers 8' [3] (mostly no I/O). But I forgot that the repo is compressed, so it's about 155MB of XML metadata. Considering that only decompression and compression takes together ~3 seconds on my laptop, and that we still generate the sqlite database files in Copr (I don't know the overhead of this), it's likely there's not much space for performance optimizations. Certainly not some low hanging fruit.. [1] https://copr-be.cloud.fedoraproject.org/results/iucar/cran/fedora-32-x86_64/repodata/ [2] https://download.copr.fedorainfracloud.org/results/iucar/cran/fedora-32-x86_64/01839153-R-CRAN-yotover/build.log.gz [3] https://pagure.io/copr/copr/blob/4e6881371075c37bf1eddc4b09bd7e29162c4d89/f/backend/run/copr-repo#_133-134 Pavel ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: koji buildsystem changes
On Thu, Dec 17, 2020 at 10:17:45AM +0100, Pavel Raiskup wrote: > On Wednesday, December 16, 2020 8:25:50 PM CET Kevin Fenzi wrote: > > On Tue, Dec 15, 2020 at 10:54:07AM +0100, Miroslav Suchý wrote: > > > > > > https://github.com/rpm-software-management/mock/wiki/Feature-container-for-bootstrap > > > which is still fresh feature - just one year old :) And so far not > > > enabled in Koji. > > > > > > When *this* will be enabled in Koji, we can finally boost the RPM > > > development to levels previously unimaginable. :) > > > > Yeah, we should discuss how to manage and land this. Ideally there would > > be a lot of CI/QA/Gating on the image, and we would have easy ways to > > roll back to a known good one in case. > > Sounds good! Likely we'd need an image for each chroot and arch. Where > can we start? :) Still a lot of things to ponder here... I think it would be important to maintain the ability for users to build things locally with mock and have them 'close' to what it is in koji. That would I guess involve publishing all the bootstrap images and getting mock to be able to download the right ones for the branch/arch. CI/Gating/testing on the container(s). Way for releng to go back to older container(s). Way for releng to go to newer/force regeneration of container(s). koji adjustments: way to record whats in the container/what container it was. What does it mean for koji's srpm groups? What does this mean for reproducability? Do we keep all containers ever used for bootstrap so we can duplucate builds? Do we just keep a way to recreate them? Could we just re-use the fedora minimal container? I guess not, but it would be nice if we could. Also, I am guessing this all works with nspawn, we are (still) using old chroot, we would have to move to nspawn first right? (which we need to do anyhow). I'm sure I'll think of more. :) kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: koji buildsystem changes
On Thu, Dec 17, 2020 at 10:40:24AM +0100, Pavel Raiskup wrote: > On Sunday, December 13, 2020 11:02:53 PM CET Kevin Fenzi wrote: > > * Finally releng is looking at establishing a sidetag cleanup policy. > > A reminder that sidetags should be short lived and only created when > > needed. koji must generate buildroot repos for every single sidetag. > > ( You can list all your sidetags with 'fedpkg list-side-tags --mine' ) > > I'm just curious what's the most expensive thing for Koji to implement > side-tags, I'd expect something like: > > - the repositories are cloned (recursively hardlinked/symlinked?) > - the override package is added > - then createrepo_c is run I haven't closely looked, but I am pretty sure it's the createrepo_c calls. Not that they take that long, but that there has to be one for every single buildroot change. ie, if I build foo in rawhide, as soon as it lands in the f34 tag, the ~90 f34 side tags all have to run newrepo tasks. Repeat many many many times a day. > In Copr we don't have to clone the repositories, but we have to run the > createrepo_c after each build. In the past the bottleneck used to be > the createrepo_c run (recursive walk through all the packages in larger > repositories). > > If Koji suffers from the same problem, perhaps you could take a look at > the new createrepo_c option `--recycle-pkglist` - the createrepo_c costs > almost nothing then (matter of just reading and later writing the xml > metadata). That would work for the simple case of an updated package, but it wouldn't work for new packages added would it? I suppose we could tell people if they need to use a new package in their sidetag to delete and recreate it? Not sure how much hassle that would be. kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: koji buildsystem changes
Dne 17. 12. 20 v 10:17 Pavel Raiskup napsal(a): >> I wonder how much slower it is? > IOW, bootstrap image is even now slightly faster than normal bootstrap, but we > could make it almost as fast as mock without bootstrap. 1) the difference is mostly in first run (till the cache is populated). 2) Definitely faster. Want hard data? $ mock --scrub=all $ time mock --config-opts=use_bootstrap_image=False --shell 'echo ahoy' ... real3m9,534s user2m34,950s sys 0m10,617s $ mock --scrub=all $ time mock --config-opts=use_bootstrap_image=True --shell 'echo ahoy' ... real2m28,183s user1m34,371s sys 0m9,316s In data-center this can be different, but I am sitting on fat 100Mbit line. So not much different -- Miroslav Suchy, RHCA Red Hat, Associate Manager ABRT/Copr, #brno, #fedora-buildsys ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: koji buildsystem changes
On Sunday, December 13, 2020 11:02:53 PM CET Kevin Fenzi wrote: > * Finally releng is looking at establishing a sidetag cleanup policy. > A reminder that sidetags should be short lived and only created when > needed. koji must generate buildroot repos for every single sidetag. > ( You can list all your sidetags with 'fedpkg list-side-tags --mine' ) I'm just curious what's the most expensive thing for Koji to implement side-tags, I'd expect something like: - the repositories are cloned (recursively hardlinked/symlinked?) - the override package is added - then createrepo_c is run In Copr we don't have to clone the repositories, but we have to run the createrepo_c after each build. In the past the bottleneck used to be the createrepo_c run (recursive walk through all the packages in larger repositories). If Koji suffers from the same problem, perhaps you could take a look at the new createrepo_c option `--recycle-pkglist` - the createrepo_c costs almost nothing then (matter of just reading and later writing the xml metadata). Pavel ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: koji buildsystem changes
On Wednesday, December 16, 2020 8:25:50 PM CET Kevin Fenzi wrote: > On Tue, Dec 15, 2020 at 10:54:07AM +0100, Miroslav Suchý wrote: > > > > https://github.com/rpm-software-management/mock/wiki/Feature-container-for-bootstrap > > which is still fresh feature - just one year old :) And so far not enabled > > in Koji. > > > > When *this* will be enabled in Koji, we can finally boost the RPM > > development to levels previously unimaginable. :) > > Yeah, we should discuss how to manage and land this. Ideally there would > be a lot of CI/QA/Gating on the image, and we would have easy ways to > roll back to a known good one in case. Sounds good! Likely we'd need an image for each chroot and arch. Where can we start? :) Anyone can be experiment with this in Copr now (see the chroot/project/build "bootstrap" configuration options). Any publicly available podman image can be used to initialize the bootstrap. > I wonder how much slower it is? With --use-bootstrap-image, the host's package manager isn't involved in preparing the bootstrap at all. E.g. for the fedora-33-x86_64 chroot 1. 'fedora:33' container image is downloaded (this can be cached on builder) 2. container is executed from the image to install 'dnf-plugins-core' (needed by mock, we could skip this step if we had a complete image). 3. the container content is exported to /var/lib/mock/-bootstrap/root and from now on it is normal "bootstrap" (pkg management tools from there are used to install normal chroot). So we could make the bootstrap image feature overhead almost instant (equivalent to extraction of several-tens-of-MB image/tarball extraction), if we maintained the special image you talked about. IOW, bootstrap image is even now slightly faster than normal bootstrap, but we could make it almost as fast as mock without bootstrap. Pavel ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: koji buildsystem changes
On 12/16/20 12:18 AM, Miroslav Suchý wrote: Dne 15. 12. 20 v 11:39 Panu Matilainen napsal(a): BTW, I'm not aware of the details how the images are built these days, but of course *something* will still need to build those images, That is normal podman image of Fedora. Strictly speaking you will still need to have compatibility with latest released Fedora (I guess we do not have rawhide podman images, right?). But once you have that image you can run the Mock on anything archaic. As long as the thing is capable of running podman. I'm actually talking about the other extreme, the "archaic" use-case isn't interesting to me personally. To allow rpm and its dependencies to use features only in rawhide would require a rolling rawhide image to be used for the build of the next image, and as things can and will go wrong, it'd need to be possible to back down to a known good image. Besides other issues and complexities, rpm utilizing this to maximum would mean that you can't distro-upgrade from an N-1 version anymore (you'd need to boot into an image of the new distro first), and I don't think Fedora's upgrade-tooling is ready for that. Would've worked with the anaconda upgrade method of the old, but what's gone is gone... So with these caveats in mind, I think the current situation is about as good as it's going to get. Which is by no means a small thing, like said it's HUGE for me! - Panu - ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: koji buildsystem changes
On Tue, Dec 15, 2020 at 10:54:07AM +0100, Miroslav Suchý wrote: > Dne 14. 12. 20 v 11:22 Panu Matilainen napsal(a): > > > > When I started working on rpm back in 2007, landing new features to rpm > > meant looking 5+ years in the horizon to have > > said feature running on a released RHEL running in the builder so people > > could start trying it out on rawhide, which > > meant that by then it was far, far too late to address any of the > > inevitable design flaws you get when doing something > ... > > For me this is a career-long dream come true. THANK YOU for everybody in > > the involved in making it happen, one step at a > > time, over the years! > > Nice to hear :) > > But it is important to know that as long as bootstrap is big step, there is > still hidden caveat: When any RPM or DNF > package or any package they requires use some nice fancy feature which host's > rpm does not recognize, then the bootstrap > will fail anyway. This is solved by bootstrap image feature Well, depends on the feature, but sure... > > https://github.com/rpm-software-management/mock/wiki/Feature-container-for-bootstrap > which is still fresh feature - just one year old :) And so far not enabled in > Koji. > > When *this* will be enabled in Koji, we can finally boost the RPM development > to levels previously unimaginable. :) Yeah, we should discuss how to manage and land this. Ideally there would be a lot of CI/QA/Gating on the image, and we would have easy ways to roll back to a known good one in case. I wonder how much slower it is? kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: koji buildsystem changes
Dne 15. 12. 20 v 11:39 Panu Matilainen napsal(a): > BTW, I'm not aware of the details how the images are built these days, but of > course *something* will still need to > build those images, That is normal podman image of Fedora. Strictly speaking you will still need to have compatibility with latest released Fedora (I guess we do not have rawhide podman images, right?). But once you have that image you can run the Mock on anything archaic. As long as the thing is capable of running podman. -- Miroslav Suchy, RHCA Red Hat, Associate Manager ABRT/Copr, #brno, #fedora-buildsys ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: koji buildsystem changes
On 12/15/20 12:25 PM, Panu Matilainen wrote: On 12/15/20 11:54 AM, Miroslav Suchý wrote: Dne 14. 12. 20 v 11:22 Panu Matilainen napsal(a): When I started working on rpm back in 2007, landing new features to rpm meant looking 5+ years in the horizon to have said feature running on a released RHEL running in the builder so people could start trying it out on rawhide, which meant that by then it was far, far too late to address any of the inevitable design flaws you get when doing something ... For me this is a career-long dream come true. THANK YOU for everybody in the involved in making it happen, one step at a time, over the years! Nice to hear :) But it is important to know that as long as bootstrap is big step, there is still hidden caveat: When any RPM or DNF package or any package they requires use some nice fancy feature which host's rpm does not recognize, then the bootstrap > will fail anyway. This is solved by bootstrap image feature https://github.com/rpm-software-management/mock/wiki/Feature-container-for-bootstrap which is still fresh feature - just one year old :) And so far not enabled in Koji. When *this* will be enabled in Koji, we can finally boost the RPM development to levels previously unimaginable. :) Ooh, nice! I thought there was that caveat but wasn't aware of this work. We came up with the basic idea about reusing existing pre-built images for the purpose on internal #rpm in early 2013 but I don't know if it ever got to mock developers ears back then. At any rate it's just awesome to see this all coming together now after all that time, what a nice x-mas present :) BTW, I'm not aware of the details how the images are built these days, but of course *something* will still need to build those images, and for all features to be usable in the "bootstrap package set", that something needs to be running latest rawhide, otherwise there are still limits to the feature set available to rpm itself and its dependencies. But these days, running a set of scripts in a container on rawhide is a very different goal from the olden days where it would've entailed booting an actual system running rawhide - people used to laugh hysterically for days at the mere idea... - Panu - ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: koji buildsystem changes
> On Mon, Dec 14, 2020 at 12:08:59PM +0100, Vitaly Zaitsev via devel wrote: > > Feel free to convince maintainers: > > https://pagure.io/rpmdevtools/issue/63 > > kevin It's easier to provide my own rpmdevtools than convince the 'closed minded' maintainer. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: koji buildsystem changes
On 12/15/20 11:54 AM, Miroslav Suchý wrote: Dne 14. 12. 20 v 11:22 Panu Matilainen napsal(a): When I started working on rpm back in 2007, landing new features to rpm meant looking 5+ years in the horizon to have said feature running on a released RHEL running in the builder so people could start trying it out on rawhide, which meant that by then it was far, far too late to address any of the inevitable design flaws you get when doing something ... For me this is a career-long dream come true. THANK YOU for everybody in the involved in making it happen, one step at a time, over the years! Nice to hear :) But it is important to know that as long as bootstrap is big step, there is still hidden caveat: When any RPM or DNF package or any package they requires use some nice fancy feature which host's rpm does not recognize, then the bootstrap > will fail anyway. This is solved by bootstrap image feature https://github.com/rpm-software-management/mock/wiki/Feature-container-for-bootstrap which is still fresh feature - just one year old :) And so far not enabled in Koji. When *this* will be enabled in Koji, we can finally boost the RPM development to levels previously unimaginable. :) Ooh, nice! I thought there was that caveat but wasn't aware of this work. We came up with the basic idea about reusing existing pre-built images for the purpose on internal #rpm in early 2013 but I don't know if it ever got to mock developers ears back then. At any rate it's just awesome to see this all coming together now after all that time, what a nice x-mas present :) - Panu - ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: koji buildsystem changes
Dne 14. 12. 20 v 11:22 Panu Matilainen napsal(a): > > When I started working on rpm back in 2007, landing new features to rpm meant > looking 5+ years in the horizon to have > said feature running on a released RHEL running in the builder so people > could start trying it out on rawhide, which > meant that by then it was far, far too late to address any of the inevitable > design flaws you get when doing something ... > For me this is a career-long dream come true. THANK YOU for everybody in the > involved in making it happen, one step at a > time, over the years! Nice to hear :) But it is important to know that as long as bootstrap is big step, there is still hidden caveat: When any RPM or DNF package or any package they requires use some nice fancy feature which host's rpm does not recognize, then the bootstrap will fail anyway. This is solved by bootstrap image feature https://github.com/rpm-software-management/mock/wiki/Feature-container-for-bootstrap which is still fresh feature - just one year old :) And so far not enabled in Koji. When *this* will be enabled in Koji, we can finally boost the RPM development to levels previously unimaginable. :) -- Miroslav Suchy, RHCA Red Hat, Associate Manager ABRT/Copr, #brno, #fedora-buildsys ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: koji buildsystem changes
On Mon, Dec 14, 2020 at 4:08 PM Kevin Fenzi wrote: > > On Mon, Dec 14, 2020 at 12:08:59PM +0100, Vitaly Zaitsev via devel wrote: > > On 13.12.2020 23:02, Kevin Fenzi wrote: > > > This has caused one common issue: rhel7 rpm does not understand the > > > newer changelog format with timezone, so spec files merged from fedora > > > will break. > > > > Maybe the changelog format change should be reverted? I don't think datetime > > and timezone are useful at all. It's been leading me to simply throw out %changelog when I backport to CentOS 7. > Feel free to convince maintainers: > > https://pagure.io/rpmdevtools/issue/63 > > kevin Thanks for the pointer. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: koji buildsystem changes
On Mon, Dec 14, 2020 at 12:08:59PM +0100, Vitaly Zaitsev via devel wrote: > On 13.12.2020 23:02, Kevin Fenzi wrote: > > This has caused one common issue: rhel7 rpm does not understand the > > newer changelog format with timezone, so spec files merged from fedora > > will break. > > Maybe the changelog format change should be reverted? I don't think datetime > and timezone are useful at all. Feel free to convince maintainers: https://pagure.io/rpmdevtools/issue/63 kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: koji buildsystem changes
On 13.12.2020 23:02, Kevin Fenzi wrote: This has caused one common issue: rhel7 rpm does not understand the newer changelog format with timezone, so spec files merged from fedora will break. Maybe the changelog format change should be reverted? I don't think datetime and timezone are useful at all. -- Sincerely, Vitaly Zaitsev (vit...@easycoding.org) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: koji buildsystem changes
On 12/14/20 12:02 AM, Kevin Fenzi wrote: Greetings everyone. I thought I would share some recent changes to koji (The Fedora buildsystem): * All s390x, x86_64, and ppc64le builders have been upgraded to Fedora33 * aarch64 and armv7 builder should be upgraded soon. * I have enabled bootstrap mode for all builds. In this mode koji uses the builder tools to construct a 'bootstrap' repo from the target os tools, which it then uses to create the src.rpm and do the build. For now this is needed for epel7/8 builds after builders moved to f33 due to rpm changing to default to sqlite rpm db. It will likely be needed for many other things moving forward and allows easier landing of new features while keeping other branches working as expected. \o/ I doubt many people realize the enormity of this landmark. When I started working on rpm back in 2007, landing new features to rpm meant looking 5+ years in the horizon to have said feature running on a released RHEL running in the builder so people could start trying it out on rawhide, which meant that by then it was far, far too late to address any of the inevitable design flaws you get when doing something for the first time. And with rpm the format longetivity means that you only really get one shot at anything, so testing and feedback in the early stages of development is absolutely critical. The impossibility of this all meant that for years time, rpm efforts were directed to things that have a shorter term benefit. The first huge leap towards sanity was moving Fedora builders to run on Fedora several years ago, which cut several years away from that cycle. That things like file triggers and rich dependencies started appearing around that time is no coincidence. And from the early swap until now, the gap has shrunk from multi-year to something like one, which is all awesome but still missing that critical early phase. Thanks to the recent advancements in mock (the bootstrap mode) and now last but not least koji utilizing it, we finally have a situation where we can actually develop rawhide on rawhide. And for rpm this means we can try out new features in rawhide during the development cycle, address unforeseen flaws in new features in the alpha-beta stage as deity always intended, and if all goes wrong, just back out and try again next release. Almost like with normal software. Rather than live with the ever increasing weight of past mistakes because its already out there in released versions and forever unchangeable. For me this is a career-long dream come true. THANK YOU for everybody in the involved in making it happen, one step at a time, over the years! - Panu - ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: koji buildsystem changes
Il 14/12/20 04:42, Jerry James ha scritto: > > On this page: > > https://docs.fedoraproject.org/en-US/rawhide-gating/multi-builds/ > > there is a box marked "Important" that says: > > "If you have created a side-tag and have no use for it (and did not > create an update for it), please remove it so it does consume > resources on the build infrastructure. You can simply remove side-tags > you have created using fedpkg remove-side-tag and you can list your > side tags using fedpkg list-side-tags --user=." > > The parenthetical remark in the first sentence says to me that if I > have created an update from a side tag, that I should not remove the > side tag. Is that correct? Indeed, if I run fedpkg list-side-tags, I > see 5, when in fact every side tag that I have created has been > merged. Should I delete all of those? I created updates for each one > of them. > -- The side-tag should be removed from the builds of the update when the update goes stable. Then Koji garbage collector will delete the empty side-tag. Most likely a Bodhi failure: https://github.com/fedora-infra/bodhi/pull/4173 Mattia ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: koji buildsystem changes
On Sun, Dec 13, 2020 at 08:42:53PM -0700, Jerry James wrote: > On Sun, Dec 13, 2020 at 3:22 PM Kevin Fenzi wrote: > > * Finally releng is looking at establishing a sidetag cleanup policy. > > A reminder that sidetags should be short lived and only created when > > needed. koji must generate buildroot repos for every single sidetag. > > ( You can list all your sidetags with 'fedpkg list-side-tags --mine' ) > > On this page: > > https://docs.fedoraproject.org/en-US/rawhide-gating/multi-builds/ > > there is a box marked "Important" that says: > > "If you have created a side-tag and have no use for it (and did not > create an update for it), please remove it so it does consume > resources on the build infrastructure. You can simply remove side-tags > you have created using fedpkg remove-side-tag and you can list your > side tags using fedpkg list-side-tags --user=." > > The parenthetical remark in the first sentence says to me that if I > have created an update from a side tag, that I should not remove the > side tag. Is that correct? Indeed, if I run fedpkg list-side-tags, I > see 5, when in fact every side tag that I have created has been > merged. Should I delete all of those? I created updates for each one > of them. Normally side-tags that went through bodhi should be cleaned up by bodhi, if they have not been you can clean them manually without problem. The issue this text was trying to prevent was the case where someone builds one or more packages in their side-tag and delete said side-tag before submitting the builds as an update, thus effectively untagging the builds from all tags and making them available to be picked up by koji's garbage collection. Pierre ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: koji buildsystem changes
On Sun, Dec 13, 2020 at 3:22 PM Kevin Fenzi wrote: > * Finally releng is looking at establishing a sidetag cleanup policy. > A reminder that sidetags should be short lived and only created when > needed. koji must generate buildroot repos for every single sidetag. > ( You can list all your sidetags with 'fedpkg list-side-tags --mine' ) On this page: https://docs.fedoraproject.org/en-US/rawhide-gating/multi-builds/ there is a box marked "Important" that says: "If you have created a side-tag and have no use for it (and did not create an update for it), please remove it so it does consume resources on the build infrastructure. You can simply remove side-tags you have created using fedpkg remove-side-tag and you can list your side tags using fedpkg list-side-tags --user=." The parenthetical remark in the first sentence says to me that if I have created an update from a side tag, that I should not remove the side tag. Is that correct? Indeed, if I run fedpkg list-side-tags, I see 5, when in fact every side tag that I have created has been merged. Should I delete all of those? I created updates for each one of them. -- Jerry James http://www.jamezone.org/ ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
