On Thu, May 7, 2026 at 11:44 AM Michael Catanzaro <[email protected]> wrote: > > Wow. :( > > This week's total outage on s390x is an unusual occurrence, and we shouldn't > need to change anything based on this. I'm more concerned with *persistent* > capacity issues. We have a ticket > https://forge.fedoraproject.org/infra/tickets/issues/13141 but I think it > understates the severity of the problem. It's normal for my heavybuilder > channel builds to wait up to two days before an s390x build even *starts*. > That significantly delays Fedora updates for all architectures. > > s390x has been a problem for too long, and it's time to do something about > it. I was planning to make almost the same proposal as Hans, except I don't > think the problems with ppc64le are nearly as serious as the problems with > s390x. Our ppc64le builders are too slow relative to other architectures, and > that's a problem, but they are not *severely* slow like the s390x builders. I > think doubling the ppc64le builders would be adequate, whereas s390x needs to > be quadrupled or more. Also, packages rarely fail to build only on ppc64le -- > if it fails there, it's very likely going to fail on at least one other > architecture as well -- whereas it's common for builds to fail only on s390x > due to endianness issues. So it's really only s390x that is causing problems > for Fedora. > > On the other hand, our Workstation usage stats > https://forge.fedoraproject.org/workstation/tickets/issues/500#issuecomment-361654 > indicate we have almost no users on ppc64le, so while ppc64le is not causing > problems, I'd argue it's not worth building anymore on that basis alone. That > should move to secondary infrastructure. > > Proposal: future stable Fedoras will not be built for s390x or ppc64le > beginning with Fedora 45, leaving these architectures enabled in rawhide and > ELN only. That should eventually reduce load on the builders when Fedora 43 > and Fedora 44 reach EOL, by reducing the number of total builds. It also > allows Red Hat 3 months to improve capacity before Fedora 45 is branched if > Red Hat doesn't like this plan. If that's still not enough to resolve our > capacity problems, then in February next year (arbitrarily selected to > coincide with F46 branching) we should turn off s390x in rawhide as well, > leaving it only in ELN, so it will no longer inconvenience Fedora community > members at all. >
I don't think it'd be worth it to keep s390x at all for Fedora if we drop it from stable releases. It doesn't save packagers any time or effort, and if it's effectively never shipped, it's just not worth it to keep. We might as well drop it everywhere and have Red Hat only deal with it when RHEL is made. Or maybe RHEL will just drop s390x because everything will be too broken anyway. If there's a reliance on community support for s390x and no institutional support for it, we should just pull the plug entirely. Half-measures will just frustrate people even more. That said, there is a higher level ticket about the general problem already: https://redhat.atlassian.net/browse/FRCL-36 The specific issues with s390x are currently tracked in these tickets: * https://forge.fedoraproject.org/infra/tickets/issues/13141 * https://forge.fedoraproject.org/infra/tickets/issues/13326 * https://forge.fedoraproject.org/releng/tickets/issues/13255 > Proposal 2: do the same for i686 beginning with Fedora 46, if nobody is > working on a practical plan for reducing i686 architecture support before > then. Fedora really needs to reduce i686 down to only those packages > necessary to keep Bazzite and possibly Wine working, but nobody has been > working on this: after the change proposal to turn off i686 was defeated, > everybody just stopped and moved on instead of actually trying to fix things. > That's the wrong default action: the default action should be to turn off > i686 if nobody is interested in working on saving it. Otherwise, the day will > eventually come when we can no longer link some big important package on i686 > -- my fear is WebKitGTK -- and then *all* architectures will be blocked from > receiving updates. The problem is that the set required to keep all that working is still fairly large right now. We can't seriously reevaluate anything until after the new 64-bit Linux Steam client arrives, which is currently under development and expected to be released soon. -- 真実はいつも一つ！/ Always, there's only one truth! -- _______________________________________________ devel mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/[email protected] Do not reply to spam, report it: https://forge.fedoraproject.org/infra/tickets/issues/new
