Re: Please use side tags for backwards-incompatible bumps of major packages, not buildroot overrides
On 22. 03. 22 19:48, Adam Williamson wrote: now we have convenient self-service side tags,*please use them*. Especially for something as major as a bump of perl that changes dependencies of packages built against it like this. Side tags avoid this mess entirely. Using the mechanism to produce an update from a side tag also makes it easier to make sure all the right packages are in the update and they don't get incorrectly submitted as separate updates. Thanks folks! BTW it seems our packaging documentation does not recognize side tags for anything else than rawhide an early branched: https://pagure.io/fedora-docs/package-maintainer-docs/issue/72 -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Please use side tags for backwards-incompatible bumps of major packages, not buildroot overrides
On Tue, Mar 29, 2022 at 5:05 PM Mattia Verga via devel wrote: > Maybe BR overrides usage should be restricted only to users with special > needs (users in provenpackager or releng groups), while "normal" users > should be forced to take the side-tag way? As always, there are special cases. I certainly would not like to see packagers giving up on packaging because the impediments placed in their way to accomplish what they want (and need) to do are too onerous. Before taking any next steps I would like to see if Adam's request to reconsider BR overrides (and typically move to side tags) will reduce usage. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Please use side tags for backwards-incompatible bumps of major packages, not buildroot overrides
V Tue, Mar 29, 2022 at 05:04:29PM +, Mattia Verga via devel napsal(a): > Il 24/03/22 09:12, Petr Pisar ha scritto: > > V Wed, Mar 23, 2022 at 05:40:28PM +, Mattia Verga via devel napsal(a): > >> So, now that we have side-tags to perform this kind of builds, does the > >> buildroot override existence still make sense? Is there any use case > >> that still requires BR overrides and cannot be done with side-tags? > >> > > I use overrides pretty extensively when populating new EPEL. If you have > > a deep dependency tree and the packages are scattered among many > > maintainers, > > it's much faster to use an override than to block depending packages for > > a week. Side tags also do not work there because every new package would > > reset > > the testing period. And if I'm not mistaken non-proven packagers cannot edit > > other's updates. And I don't count communication overhead. People would end > > up > > mixing various side tags and crosstagging builds. > > > Maybe BR overrides usage should be restricted only to users with special > needs (users in provenpackager or releng groups), while "normal" users > should be forced to take the side-tag way? > I'm not sure. There are more people in Perl stack who do it and are not proven packagers. Maybe EPEL in its initial phase should work more like Rawhide, i.e. without updates-testing repo. -- Petr signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Please use side tags for backwards-incompatible bumps of major packages, not buildroot overrides
Il 24/03/22 09:12, Petr Pisar ha scritto: > V Wed, Mar 23, 2022 at 05:40:28PM +, Mattia Verga via devel napsal(a): >> So, now that we have side-tags to perform this kind of builds, does the >> buildroot override existence still make sense? Is there any use case >> that still requires BR overrides and cannot be done with side-tags? >> > I use overrides pretty extensively when populating new EPEL. If you have > a deep dependency tree and the packages are scattered among many maintainers, > it's much faster to use an override than to block depending packages for > a week. Side tags also do not work there because every new package would reset > the testing period. And if I'm not mistaken non-proven packagers cannot edit > other's updates. And I don't count communication overhead. People would end up > mixing various side tags and crosstagging builds. > Maybe BR overrides usage should be restricted only to users with special needs (users in provenpackager or releng groups), while "normal" users should be forced to take the side-tag way? Mattia ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Please use side tags for backwards-incompatible bumps of major packages, not buildroot overrides
V Wed, Mar 23, 2022 at 05:40:28PM +, Mattia Verga via devel napsal(a): > So, now that we have side-tags to perform this kind of builds, does the > buildroot override existence still make sense? Is there any use case > that still requires BR overrides and cannot be done with side-tags? > I use overrides pretty extensively when populating new EPEL. If you have a deep dependency tree and the packages are scattered among many maintainers, it's much faster to use an override than to block depending packages for a week. Side tags also do not work there because every new package would reset the testing period. And if I'm not mistaken non-proven packagers cannot edit other's updates. And I don't count communication overhead. People would end up mixing various side tags and crosstagging builds. -- Petr signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Please use side tags for backwards-incompatible bumps of major packages, not buildroot overrides
On Wed, Mar 23, 2022 at 10:16 AM Adam Williamson wrote: > 2) Just to note what I wound up doing here - aside from the special > polymake case, I found (I hope) all the packages that got built against > 5.34.1, bumped and rebuilt them against 5.34.0, and edited the > standalone updates to have the new builds, which will work with both > 5.34.0 and 5.34.1, so whatever order they get pushed in things should > be OK. It's probably time for me to chime in about polymake again. The polymake package has relatively few Fedora users, and takes a long time to build. Perl, on the other hand, has a large number of users. I don't think we should hold up perl releases for a polymake build. It's okay with me to just go ahead and update perl and let polymake be broken. I'll notice it is broken and do the necessary builds. The few polymake users will be annoyed that they can't update to the latest perl and will send me nastygrams. I'll respond with a canned email I have sitting around because exactly this situation has happened many times in the past. Of course, I am also trying to give polymake away, so maybe I should not put my (purely hypothetical so far) successor on the hook like that. :-) The other possibility is that somebody who knows more about perl than me breaks polymake's hard dependency on the exact version it was built with. I think upstream is concerned about that because polymake inserts its tentacles into all corners of perl's C innards, so even a small change could require a rebuild. -- Jerry James http://www.jamezone.org/ ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Please use side tags for backwards-incompatible bumps of major packages, not buildroot overrides
On Wed, 2022-03-23 at 18:13 -0400, Elliott Sales de Andrade wrote: > > > > 1) Neat trick: I'm pretty sure the buildroot override only needs to be > > valid until all the build dependencies have been installed. For my > > polymake rebuild, I put the override back in place, fired the polymake > > build, waited till all the build tasks for the different arch had > > installed build dependencies, then expired the override again. It > > doesn't need to stay valid for the whole time the actual compilation > > stage is happening. > > > > Note, this override isn't strictly needed either. You can create a > side tag, and tag in _any_ build you need to fix things. Pretty sure > you can even tag in older versions of things if necessary. You just > have to remember to untag the extra builds before creating the update > in Bodhi, if you're creating it from the side tag. Yeah, I could've made a new side tag for this, but at this point the override existed and I had it open in a browser tab so I could basically turn it off and on like a light switch, so I just went with that. :D I was just noting this as a detail about how buildroot overrides work, if you're really determined to use one. -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Please use side tags for backwards-incompatible bumps of major packages, not buildroot overrides
On Wed, Mar 23, 2022 at 12:16 PM Adam Williamson wrote: > > On Wed, 2022-03-23 at 08:39 +, Paul Howarth wrote: > > > > OK, so this is largely my fault. Whilst I didn't do the initial perl > > 5.34.1 build and update, I did set up the buildroot override and the > > builds of the two packages (perl-PAR-Packer and polymake) that have > > hard dependencies on the specific perl version they were built against. > > Unfortunately the polymake build took over 24 hours on armv7hl but > > after that I could have and should have expired the buildroot override > > to prevent the follow-up issues affecting other perl-related builds. > > The issue was already known about and it was already planned to do the > > forthcoming update for f35 to perl 5.34.1 in a side tag > > (https://bugzilla.redhat.com/show_bug.cgi?id=2064808#c5). > > Oh sorry, forgot to mention a couple of other things: > > 1) Neat trick: I'm pretty sure the buildroot override only needs to be > valid until all the build dependencies have been installed. For my > polymake rebuild, I put the override back in place, fired the polymake > build, waited till all the build tasks for the different arch had > installed build dependencies, then expired the override again. It > doesn't need to stay valid for the whole time the actual compilation > stage is happening. > Note, this override isn't strictly needed either. You can create a side tag, and tag in _any_ build you need to fix things. Pretty sure you can even tag in older versions of things if necessary. You just have to remember to untag the extra builds before creating the update in Bodhi, if you're creating it from the side tag. > 2) Just to note what I wound up doing here - aside from the special > polymake case, I found (I hope) all the packages that got built against > 5.34.1, bumped and rebuilt them against 5.34.0, and edited the > standalone updates to have the new builds, which will work with both > 5.34.0 and 5.34.1, so whatever order they get pushed in things should > be OK. > -- > Adam Williamson > Fedora QA > IRC: adamw | Twitter: adamw_ha > https://www.happyassassin.net > -- Elliott ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Please use side tags for backwards-incompatible bumps of major packages, not buildroot overrides
On Wed, 23 Mar 2022 10:41:52 -0700 Kevin Fenzi wrote: > I wonder... should we stop allowing buildroot overrides? > > Or at the very least add a admon to adding a new one in bodhi, > explaining that you should probibly use a side tag, etc? They're still very useful when bringing up new EPEL releases. When you have things like the perl stack with lots of interdependencies I think side tags would be quite unwieldy. They got lots of use (not just by me!) when bringing up EPEL-8 and EPEL-9 in particular. Paul. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Please use side tags for backwards-incompatible bumps of major packages, not buildroot overrides
On 23. 03. 22 18:40, Mattia Verga via devel wrote: So, now that we have side-tags to perform this kind of builds, does the buildroot override existence still make sense? Is there any use case that still requires BR overrides and cannot be done with side-tags? As I've said elsewhere in the thread: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/4BCV5BOA5JWEQBNIEWHPCGLBQB4QMHRW/ Happy to elaborate and seek for alternate solutions. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Please use side tags for backwards-incompatible bumps of major packages, not buildroot overrides
On 23. 03. 22 18:41, Kevin Fenzi wrote: I wonder... should we stop allowing buildroot overrides? I wondered this for a long time. Unfortunately I still find usecases for buildroot overrides. E.g. when we ship new versions of some macro packages etc. and we want them available even before the update reaches stable (e.g. to see them not breaking other packages in Kochei). They should certainly be discouraged as a mean to update interdependent packages. Or at the very least add a admon to adding a new one in bodhi, explaining that you should probibly use a side tag, etc? +1 -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Please use side tags for backwards-incompatible bumps of major packages, not buildroot overrides
I wonder... should we stop allowing buildroot overrides? Or at the very least add a admon to adding a new one in bodhi, explaining that you should probibly use a side tag, etc? kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Please use side tags for backwards-incompatible bumps of major packages, not buildroot overrides
So, now that we have side-tags to perform this kind of builds, does the buildroot override existence still make sense? Is there any use case that still requires BR overrides and cannot be done with side-tags? Mattia ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Please use side tags for backwards-incompatible bumps of major packages, not buildroot overrides
On Wed, 2022-03-23 at 08:39 +, Paul Howarth wrote: > > OK, so this is largely my fault. Whilst I didn't do the initial perl > 5.34.1 build and update, I did set up the buildroot override and the > builds of the two packages (perl-PAR-Packer and polymake) that have > hard dependencies on the specific perl version they were built against. > Unfortunately the polymake build took over 24 hours on armv7hl but > after that I could have and should have expired the buildroot override > to prevent the follow-up issues affecting other perl-related builds. > The issue was already known about and it was already planned to do the > forthcoming update for f35 to perl 5.34.1 in a side tag > (https://bugzilla.redhat.com/show_bug.cgi?id=2064808#c5). Oh sorry, forgot to mention a couple of other things: 1) Neat trick: I'm pretty sure the buildroot override only needs to be valid until all the build dependencies have been installed. For my polymake rebuild, I put the override back in place, fired the polymake build, waited till all the build tasks for the different arch had installed build dependencies, then expired the override again. It doesn't need to stay valid for the whole time the actual compilation stage is happening. 2) Just to note what I wound up doing here - aside from the special polymake case, I found (I hope) all the packages that got built against 5.34.1, bumped and rebuilt them against 5.34.0, and edited the standalone updates to have the new builds, which will work with both 5.34.0 and 5.34.1, so whatever order they get pushed in things should be OK. -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Please use side tags for backwards-incompatible bumps of major packages, not buildroot overrides
On Wed, 2022-03-23 at 08:39 +, Paul Howarth wrote: > > In mitigation, my thinking was that since the f36 beta freeze is still > ongoing, the perl update and its hard dependencies would almost > certainly have been pushed to stable at the same time anyway. In > addition, since those updates were created prior to the ones for > the other modules that were incidentally built against 5.34.1, perl > itself would have been stable before the later updates arrived. So in > practice there probably wouldn't have been any mysterious dependency > issues in this particular case. And whilst I could have added the > delayed polymake build to the perl+perl-PAR-Packer update, I chose not > to so as not to reset the timer on the perl+perl-PAR-Packer update, > which would have set it back by 2 days and potentially resulted in > other modules being pushed to stable before the underlying perl update. > Thanks Paul! So yeah, it's true about the freeze and the ordering, but it's still possible for undesirable things to happen, thanks to autopush. All the updates seem to have default autopush settings - +3 karma, 14 days time. So if one of the dependent updates happened to get +3 karma before the perl update, and the perl update wasn't submitted manually, they could still have been submitted first. The freeze could potentially be lifted quite soon, since go/no-go is tomorrow, so it wouldn't necessarily save us. And finally, since polymake depends on *exactly* the version of perl (it's not a >= situation), I believe polymake and perl really have to be submitted *exactly together* in order not to break anything. AIUI, if perl gets pushed before polymake, the polymake in stable will no longer be installable with the perl in stable. This is why I think it'd still be best to edit polymake into the perl update, even though the build does take forever (my rebuild is still running :|) What might be safest overall is to edit my polymake rebuild into the perl update when it's done, then make sure the perl update gets +1 or +2 karma (whichever it needs) and is submitted for stable manually ASAP. Thanks again! -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Please use side tags for backwards-incompatible bumps of major packages, not buildroot overrides
On 22. 03. 22 19:48, Adam Williamson wrote: I found quite a big mess today, caused by an attempt to bump perl to 5.34.1 in Fedora 36: https://bodhi.fedoraproject.org/updates/FEDORA-2022-cea638ebd4 Because some packages depend on the exact perl interpreter version, the maintainer made a buildroot override for perl: https://bodhi.fedoraproject.org/overrides/perl-5.34.1-486.fc36 and rebuilt them. Okay. The problem with this is, we have lots of perl modules. People build them all the time. And buildroot overrides apply to *everything*. So, while the buildroot override has been valid, multiple *other* perl modules have been unwittingly built against it: perl-Scalar-List-Utils: https://koji.fedoraproject.org/koji/buildinfo?buildID=1936732 perl-Parallel-Pipes: https://koji.fedoraproject.org/koji/buildinfo?buildID=1937527 perl-PPIx-Regexp: https://koji.fedoraproject.org/koji/buildinfo?buildID=1937522 perl-Crypt-SSLeay: https://koji.fedoraproject.org/koji/buildinfo?buildID=1937521 perl-Crypt-OpenSSL-PKCS10: https://koji.fedoraproject.org/koji/buildinfo?buildID=1937471 ...and those are just the ones I found so far. Because they were built against 5.34.1, all of those builds require "perl(:MODULE_COMPAT_5.34.1)" , which means they require perl-5.34.1 from updates-testing. They cannot be installed with perl-5.34.0 from stable. But the maintainers likely had no idea about this - buildroot overrides are not at all discoverable, there is zero indication your package is building against one when you build it - and have created separate updates for those packages: https://bodhi.fedoraproject.org/updates/FEDORA-2022-cb1170abf2 https://bodhi.fedoraproject.org/updates/FEDORA-2022-d83d0ba901 https://bodhi.fedoraproject.org/updates/FEDORA-2022-fac98e635f https://bodhi.fedoraproject.org/updates/FEDORA-2022-c2203f1964 https://bodhi.fedoraproject.org/updates/FEDORA-2022-0990e3309e Users testing those updates will likely not notice the problem, because they'll have *all* of updates-testing enabled when they update, and perl-5.34.1 will be pulled in. But if any of those updates is pushed stable before the perl update is pushed stable, it will fail to install for anyone who does not have updates-testing enabled. This is quite a mess. I'm going to try and clean it up, but it'll be a bit ugly (there are a couple of ways I can do it, but both will involve doing bumps-and-rebuilds of the affected packages with no changes). I've been worried about this sort of thing happening for a while due to the undesirable properties of buildroot overrides. This is the biggest real-world case I've seen, but it could certainly happen again. It might even have happened without anyone noticing exactly what was going on (just mysterious dependency issues that magically went away after a bit). So: now we have convenient self-service side tags, *please use them*. Especially for something as major as a bump of perl that changes dependencies of packages built against it like this. Side tags avoid this mess entirely. Using the mechanism to produce an update from a side tag also makes it easier to make sure all the right packages are in the update and they don't get incorrectly submitted as separate updates. Thanks folks! Thanks for this, Adam. I will add one more reason: When the automatic "fails to install" reports run, they use the Koji buidlroot. Every now and then, a bugzilla is reported like this one: https://bugzilla.redhat.com/show_bug.cgi?id=2066728 It is not a false positive, the dependency problem exists, but it only exists because there is an override: https://bodhi.fedoraproject.org/overrides/R-4.1.3-1.fc36 Please, use side tags, not overrides, when updating inter-dependent packages, it avoid (even temporary) breakage: https://docs.fedoraproject.org/en-US/fesco/Updates_Policy/#updating-inter-dependent-packages -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Please use side tags for backwards-incompatible bumps of major packages, not buildroot overrides
On Tue, 22 Mar 2022 11:48:57 -0700 Adam Williamson wrote: > I found quite a big mess today, caused by an attempt to bump perl to > 5.34.1 in Fedora 36: > > https://bodhi.fedoraproject.org/updates/FEDORA-2022-cea638ebd4 > > Because some packages depend on the exact perl interpreter version, > the maintainer made a buildroot override for perl: > > https://bodhi.fedoraproject.org/overrides/perl-5.34.1-486.fc36 > > and rebuilt them. Okay. > > The problem with this is, we have lots of perl modules. People build > them all the time. And buildroot overrides apply to *everything*. > > So, while the buildroot override has been valid, multiple *other* perl > modules have been unwittingly built against it: > > perl-Scalar-List-Utils: > https://koji.fedoraproject.org/koji/buildinfo?buildID=1936732 > perl-Parallel-Pipes: > https://koji.fedoraproject.org/koji/buildinfo?buildID=1937527 > perl-PPIx-Regexp: > https://koji.fedoraproject.org/koji/buildinfo?buildID=1937522 > perl-Crypt-SSLeay: > https://koji.fedoraproject.org/koji/buildinfo?buildID=1937521 > perl-Crypt-OpenSSL-PKCS10: > https://koji.fedoraproject.org/koji/buildinfo?buildID=1937471 > > ...and those are just the ones I found so far. Because they were built > against 5.34.1, all of those builds require > "perl(:MODULE_COMPAT_5.34.1)" , which means they require perl-5.34.1 > from updates-testing. They cannot be installed with perl-5.34.0 from > stable. But the maintainers likely had no idea about this - buildroot > overrides are not at all discoverable, there is zero indication your > package is building against one when you build it - and have created > separate updates for those packages: > > https://bodhi.fedoraproject.org/updates/FEDORA-2022-cb1170abf2 > https://bodhi.fedoraproject.org/updates/FEDORA-2022-d83d0ba901 > https://bodhi.fedoraproject.org/updates/FEDORA-2022-fac98e635f > https://bodhi.fedoraproject.org/updates/FEDORA-2022-c2203f1964 > https://bodhi.fedoraproject.org/updates/FEDORA-2022-0990e3309e > > Users testing those updates will likely not notice the problem, > because they'll have *all* of updates-testing enabled when they > update, and perl-5.34.1 will be pulled in. But if any of those > updates is pushed stable before the perl update is pushed stable, it > will fail to install for anyone who does not have updates-testing > enabled. > > This is quite a mess. I'm going to try and clean it up, but it'll be a > bit ugly (there are a couple of ways I can do it, but both will > involve doing bumps-and-rebuilds of the affected packages with no > changes). > > I've been worried about this sort of thing happening for a while due > to the undesirable properties of buildroot overrides. This is the > biggest real-world case I've seen, but it could certainly happen > again. It might even have happened without anyone noticing exactly > what was going on (just mysterious dependency issues that magically > went away after a bit). > > So: now we have convenient self-service side tags, *please use them*. > Especially for something as major as a bump of perl that changes > dependencies of packages built against it like this. Side tags avoid > this mess entirely. Using the mechanism to produce an update from a > side tag also makes it easier to make sure all the right packages are > in the update and they don't get incorrectly submitted as separate > updates. > > Thanks folks! OK, so this is largely my fault. Whilst I didn't do the initial perl 5.34.1 build and update, I did set up the buildroot override and the builds of the two packages (perl-PAR-Packer and polymake) that have hard dependencies on the specific perl version they were built against. Unfortunately the polymake build took over 24 hours on armv7hl but after that I could have and should have expired the buildroot override to prevent the follow-up issues affecting other perl-related builds. The issue was already known about and it was already planned to do the forthcoming update for f35 to perl 5.34.1 in a side tag (https://bugzilla.redhat.com/show_bug.cgi?id=2064808#c5). In mitigation, my thinking was that since the f36 beta freeze is still ongoing, the perl update and its hard dependencies would almost certainly have been pushed to stable at the same time anyway. In addition, since those updates were created prior to the ones for the other modules that were incidentally built against 5.34.1, perl itself would have been stable before the later updates arrived. So in practice there probably wouldn't have been any mysterious dependency issues in this particular case. And whilst I could have added the delayed polymake build to the perl+perl-PAR-Packer update, I chose not to so as not to reset the timer on the perl+perl-PAR-Packer update, which would have set it back by 2 days and potentially resulted in other modules being pushed to stable before the underlying perl update. Anyway, won't happen again. Regards, Paul. ___ devel mailing
Please use side tags for backwards-incompatible bumps of major packages, not buildroot overrides
I found quite a big mess today, caused by an attempt to bump perl to 5.34.1 in Fedora 36: https://bodhi.fedoraproject.org/updates/FEDORA-2022-cea638ebd4 Because some packages depend on the exact perl interpreter version, the maintainer made a buildroot override for perl: https://bodhi.fedoraproject.org/overrides/perl-5.34.1-486.fc36 and rebuilt them. Okay. The problem with this is, we have lots of perl modules. People build them all the time. And buildroot overrides apply to *everything*. So, while the buildroot override has been valid, multiple *other* perl modules have been unwittingly built against it: perl-Scalar-List-Utils: https://koji.fedoraproject.org/koji/buildinfo?buildID=1936732 perl-Parallel-Pipes: https://koji.fedoraproject.org/koji/buildinfo?buildID=1937527 perl-PPIx-Regexp: https://koji.fedoraproject.org/koji/buildinfo?buildID=1937522 perl-Crypt-SSLeay: https://koji.fedoraproject.org/koji/buildinfo?buildID=1937521 perl-Crypt-OpenSSL-PKCS10: https://koji.fedoraproject.org/koji/buildinfo?buildID=1937471 ...and those are just the ones I found so far. Because they were built against 5.34.1, all of those builds require "perl(:MODULE_COMPAT_5.34.1)" , which means they require perl-5.34.1 from updates-testing. They cannot be installed with perl-5.34.0 from stable. But the maintainers likely had no idea about this - buildroot overrides are not at all discoverable, there is zero indication your package is building against one when you build it - and have created separate updates for those packages: https://bodhi.fedoraproject.org/updates/FEDORA-2022-cb1170abf2 https://bodhi.fedoraproject.org/updates/FEDORA-2022-d83d0ba901 https://bodhi.fedoraproject.org/updates/FEDORA-2022-fac98e635f https://bodhi.fedoraproject.org/updates/FEDORA-2022-c2203f1964 https://bodhi.fedoraproject.org/updates/FEDORA-2022-0990e3309e Users testing those updates will likely not notice the problem, because they'll have *all* of updates-testing enabled when they update, and perl-5.34.1 will be pulled in. But if any of those updates is pushed stable before the perl update is pushed stable, it will fail to install for anyone who does not have updates-testing enabled. This is quite a mess. I'm going to try and clean it up, but it'll be a bit ugly (there are a couple of ways I can do it, but both will involve doing bumps-and-rebuilds of the affected packages with no changes). I've been worried about this sort of thing happening for a while due to the undesirable properties of buildroot overrides. This is the biggest real-world case I've seen, but it could certainly happen again. It might even have happened without anyone noticing exactly what was going on (just mysterious dependency issues that magically went away after a bit). So: now we have convenient self-service side tags, *please use them*. Especially for something as major as a bump of perl that changes dependencies of packages built against it like this. Side tags avoid this mess entirely. Using the mechanism to produce an update from a side tag also makes it easier to make sure all the right packages are in the update and they don't get incorrectly submitted as separate updates. Thanks folks! -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
