This week's total outage on s390x is an unusual occurrence, and we shouldn't need to change anything based on this. I'm more concerned with *persistent* capacity issues. We have a ticket https://forge.fedoraproject.org/infra/tickets/issues/13141 but I think it understates the severity of the problem. It's normal for my heavybuilder channel builds to wait up to two days before an s390x build even *starts*. That significantly delays Fedora updates for all architectures.
s390x has been a problem for too long, and it's time to do something about it. I was planning to make almost the same proposal as Hans, except I don't think the problems with ppc64le are nearly as serious as the problems with s390x. Our ppc64le builders are too slow relative to other architectures, and that's a problem, but they are not *severely* slow like the s390x builders. I think doubling the ppc64le builders would be adequate, whereas s390x needs to be quadrupled or more. Also, packages rarely fail to build only on ppc64le -- if it fails there, it's very likely going to fail on at least one other architecture as well -- whereas it's common for builds to fail only on s390x due to endianness issues. So it's really only s390x that is causing problems for Fedora.
On the other hand, our Workstation usage stats https://forge.fedoraproject.org/workstation/tickets/issues/500#issuecomment-361654 indicate we have almost no users on ppc64le, so while ppc64le is not causing problems, I'd argue it's not worth building anymore on that basis alone. That should move to secondary infrastructure.
Proposal: future stable Fedoras will not be built for s390x or ppc64le beginning with Fedora 45, leaving these architectures enabled in rawhide and ELN only. That should eventually reduce load on the builders when Fedora 43 and Fedora 44 reach EOL, by reducing the number of total builds. It also allows Red Hat 3 months to improve capacity before Fedora 45 is branched if Red Hat doesn't like this plan. If that's still not enough to resolve our capacity problems, then in February next year (arbitrarily selected to coincide with F46 branching) we should turn off s390x in rawhide as well, leaving it only in ELN, so it will no longer inconvenience Fedora community members at all.
Proposal 2: do the same for i686 beginning with Fedora 46, if nobody is working on a practical plan for reducing i686 architecture support before then. Fedora really needs to reduce i686 down to only those packages necessary to keep Bazzite and possibly Wine working, but nobody has been working on this: after the change proposal to turn off i686 was defeated, everybody just stopped and moved on instead of actually trying to fix things. That's the wrong default action: the default action should be to turn off i686 if nobody is interested in working on saving it. Otherwise, the day will eventually come when we can no longer link some big important package on i686 -- my fear is WebKitGTK -- and then *all* architectures will be blocked from receiving updates.
Proposal: future stable Fedoras will not be built for s390x or ppc64le beginning with Fedora 45, leaving these architectures enabled in rawhide and ELN only. That should eventually reduce load on the builders when Fedora 43 and Fedora 44 reach EOL, by reducing the number of total builds. It also allows Red Hat 3 months to improve capacity before Fedora 45 is branched if Red Hat doesn't like this plan. If that's still not enough to resolve our capacity problems, then in February next year (arbitrarily selected to coincide with F46 branching) we should turn off s390x in rawhide as well, leaving it only in ELN, so it will no longer inconvenience Fedora community members at all.
Proposal 2: do the same for i686 beginning with Fedora 46, if nobody is working on a practical plan for reducing i686 architecture support before then. Fedora really needs to reduce i686 down to only those packages necessary to keep Bazzite and possibly Wine working, but nobody has been working on this: after the change proposal to turn off i686 was defeated, everybody just stopped and moved on instead of actually trying to fix things. That's the wrong default action: the default action should be to turn off i686 if nobody is interested in working on saving it. Otherwise, the day will eventually come when we can no longer link some big important package on i686 -- my fear is WebKitGTK -- and then *all* architectures will be blocked from receiving updates.
-- _______________________________________________ devel mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/[email protected] Do not reply to spam, report it: https://forge.fedoraproject.org/infra/tickets/issues/new
