On Thu, May 07, 2026 at 12:00:37PM -0400, Neal Gompa wrote: > On Thu, May 7, 2026 at 11:44 AM Michael Catanzaro <[email protected]> > wrote: > > > > Wow. :( > > > > This week's total outage on s390x is an unusual occurrence, and we > > shouldn't need to change anything based on this. I'm more concerned with > > *persistent* capacity issues. We have a ticket > > https://forge.fedoraproject.org/infra/tickets/issues/13141 but I think it > > understates the severity of the problem. It's normal for my heavybuilder > > channel builds to wait up to two days before an s390x build even *starts*. > > That significantly delays Fedora updates for all architectures. > > > > s390x has been a problem for too long, and it's time to do something about > > it. I was planning to make almost the same proposal as Hans, except I don't > > think the problems with ppc64le are nearly as serious as the problems with > > s390x. Our ppc64le builders are too slow relative to other architectures, > > and that's a problem, but they are not *severely* slow like the s390x > > builders. I think doubling the ppc64le builders would be adequate, whereas > > s390x needs to be quadrupled or more. Also, packages rarely fail to build > > only on ppc64le -- if it fails there, it's very likely going to fail on at > > least one other architecture as well -- whereas it's common for builds to > > fail only on s390x due to endianness issues. So it's really only s390x that > > is causing problems for Fedora. > > > > On the other hand, our Workstation usage stats > > https://forge.fedoraproject.org/workstation/tickets/issues/500#issuecomment-361654 > > indicate we have almost no users on ppc64le, so while ppc64le is not > > causing problems, I'd argue it's not worth building anymore on that basis > > alone. That should move to secondary infrastructure. > > > > Proposal: future stable Fedoras will not be built for s390x or ppc64le > > beginning with Fedora 45, leaving these architectures enabled in rawhide > > and ELN only. That should eventually reduce load on the builders when > > Fedora 43 and Fedora 44 reach EOL, by reducing the number of total builds. > > It also allows Red Hat 3 months to improve capacity before Fedora 45 is > > branched if Red Hat doesn't like this plan. If that's still not enough to > > resolve our capacity problems, then in February next year (arbitrarily > > selected to coincide with F46 branching) we should turn off s390x in > > rawhide as well, leaving it only in ELN, so it will no longer inconvenience > > Fedora community members at all. > > > > I don't think it'd be worth it to keep s390x at all for Fedora if we > drop it from stable releases. It doesn't save packagers any time or > effort, and if it's effectively never shipped, it's just not worth it > to keep. We might as well drop it everywhere and have Red Hat only > deal with it when RHEL is made. Or maybe RHEL will just drop s390x > because everything will be too broken anyway. If there's a reliance on > community support for s390x and no institutional support for it, we > should just pull the plug entirely. Half-measures will just frustrate > people even more.
If rawhide build targets did drop s390x, presumably ELN could keep s390x enabled, so Red Hat retains visibility of the state of s390x builds prior to future RHEL branching. Since ELN builds are async, it would no longer be a problem for the general Fedora package maintainers. Red Hat downstream maintainers who own the packeg in RHEL would have to own the problem of keeping ELN builds working on s390x if problems arose. With regards, Daniel -- |: https://berrange.com ~~ https://hachyderm.io/@berrange :| |: https://libvirt.org ~~ https://entangle-photo.org :| |: https://pixelfed.art/berrange ~~ https://fstop138.berrange.com :| -- _______________________________________________ devel mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/[email protected] Do not reply to spam, report it: https://forge.fedoraproject.org/infra/tickets/issues/new
