On Thu, May 7, 2026 at 12:42 PM Dan Horák <[email protected]> wrote: > On Thu, 7 May 2026 17:23:29 +0100 > Daniel P. Berrangé <[email protected]> wrote: > > > On Thu, May 07, 2026 at 12:00:37PM -0400, Neal Gompa wrote: > > > On Thu, May 7, 2026 at 11:44 AM Michael Catanzaro < > [email protected]> wrote: > > > > > > > > Wow. :( > > > > > > > > This week's total outage on s390x is an unusual occurrence, and we > shouldn't need to change anything based on this. I'm more concerned with > *persistent* capacity issues. We have a ticket > https://forge.fedoraproject.org/infra/tickets/issues/13141 but I think it > understates the severity of the problem. It's normal for my heavybuilder > channel builds to wait up to two days before an s390x build even *starts*. > That significantly delays Fedora updates for all architectures. > > > > > > > > s390x has been a problem for too long, and it's time to do something > about it. I was planning to make almost the same proposal as Hans, except I > don't think the problems with ppc64le are nearly as serious as the problems > with s390x. Our ppc64le builders are too slow relative to other > architectures, and that's a problem, but they are not *severely* slow like > the s390x builders. I think doubling the ppc64le builders would be > adequate, whereas s390x needs to be quadrupled or more. Also, packages > rarely fail to build only on ppc64le -- if it fails there, it's very likely > going to fail on at least one other architecture as well -- whereas it's > common for builds to fail only on s390x due to endianness issues. So it's > really only s390x that is causing problems for Fedora. > > > > > > > > On the other hand, our Workstation usage stats > https://forge.fedoraproject.org/workstation/tickets/issues/500#issuecomment-361654 > indicate we have almost no users on ppc64le, so while ppc64le is not > causing problems, I'd argue it's not worth building anymore on that basis > alone. That should move to secondary infrastructure. > > > > > > > > Proposal: future stable Fedoras will not be built for s390x or > ppc64le beginning with Fedora 45, leaving these architectures enabled in > rawhide and ELN only. That should eventually reduce load on the builders > when Fedora 43 and Fedora 44 reach EOL, by reducing the number of total > builds. It also allows Red Hat 3 months to improve capacity before Fedora > 45 is branched if Red Hat doesn't like this plan. If that's still not > enough to resolve our capacity problems, then in February next year > (arbitrarily selected to coincide with F46 branching) we should turn off > s390x in rawhide as well, leaving it only in ELN, so it will no longer > inconvenience Fedora community members at all. > > > > > > > > > > I don't think it'd be worth it to keep s390x at all for Fedora if we > > > drop it from stable releases. It doesn't save packagers any time or > > > effort, and if it's effectively never shipped, it's just not worth it > > > to keep. We might as well drop it everywhere and have Red Hat only > > > deal with it when RHEL is made. Or maybe RHEL will just drop s390x > > > because everything will be too broken anyway. If there's a reliance on > > > community support for s390x and no institutional support for it, we > > > should just pull the plug entirely. Half-measures will just frustrate > > > people even more. > > > > If rawhide build targets did drop s390x, presumably ELN could keep > > s390x enabled, so Red Hat retains visibility of the state of s390x > > builds prior to future RHEL branching. > > AFAIK ELN buildroot inherits from Rawhide buildroot, because that's how > ELN deals with side tags ... > > The reality is slightly more complicated[1] than "inherits from Rawhide buildroot", but yes: we cannot build ELN on s390x without an s390x Rawhide.
[1] https://sgallagh.wordpress.com/2026/05/07/sausage-factory-fedora-eln-rebuild-strategy-2026-edition/
-- _______________________________________________ devel mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/[email protected] Do not reply to spam, report it: https://forge.fedoraproject.org/infra/tickets/issues/new
