On Thu, 7 May 2026 17:23:29 +0100 Daniel P. Berrangé <[email protected]> wrote:
> On Thu, May 07, 2026 at 12:00:37PM -0400, Neal Gompa wrote: > > On Thu, May 7, 2026 at 11:44 AM Michael Catanzaro <[email protected]> > > wrote: > > > > > > Wow. :( > > > > > > This week's total outage on s390x is an unusual occurrence, and we > > > shouldn't need to change anything based on this. I'm more concerned with > > > *persistent* capacity issues. We have a ticket > > > https://forge.fedoraproject.org/infra/tickets/issues/13141 but I think it > > > understates the severity of the problem. It's normal for my heavybuilder > > > channel builds to wait up to two days before an s390x build even > > > *starts*. That significantly delays Fedora updates for all architectures. > > > > > > s390x has been a problem for too long, and it's time to do something > > > about it. I was planning to make almost the same proposal as Hans, except > > > I don't think the problems with ppc64le are nearly as serious as the > > > problems with s390x. Our ppc64le builders are too slow relative to other > > > architectures, and that's a problem, but they are not *severely* slow > > > like the s390x builders. I think doubling the ppc64le builders would be > > > adequate, whereas s390x needs to be quadrupled or more. Also, packages > > > rarely fail to build only on ppc64le -- if it fails there, it's very > > > likely going to fail on at least one other architecture as well -- > > > whereas it's common for builds to fail only on s390x due to endianness > > > issues. So it's really only s390x that is causing problems for Fedora. > > > > > > On the other hand, our Workstation usage stats > > > https://forge.fedoraproject.org/workstation/tickets/issues/500#issuecomment-361654 > > > indicate we have almost no users on ppc64le, so while ppc64le is not > > > causing problems, I'd argue it's not worth building anymore on that basis > > > alone. That should move to secondary infrastructure. > > > > > > Proposal: future stable Fedoras will not be built for s390x or ppc64le > > > beginning with Fedora 45, leaving these architectures enabled in rawhide > > > and ELN only. That should eventually reduce load on the builders when > > > Fedora 43 and Fedora 44 reach EOL, by reducing the number of total > > > builds. It also allows Red Hat 3 months to improve capacity before Fedora > > > 45 is branched if Red Hat doesn't like this plan. If that's still not > > > enough to resolve our capacity problems, then in February next year > > > (arbitrarily selected to coincide with F46 branching) we should turn off > > > s390x in rawhide as well, leaving it only in ELN, so it will no longer > > > inconvenience Fedora community members at all. > > > > > > > I don't think it'd be worth it to keep s390x at all for Fedora if we > > drop it from stable releases. It doesn't save packagers any time or > > effort, and if it's effectively never shipped, it's just not worth it > > to keep. We might as well drop it everywhere and have Red Hat only > > deal with it when RHEL is made. Or maybe RHEL will just drop s390x > > because everything will be too broken anyway. If there's a reliance on > > community support for s390x and no institutional support for it, we > > should just pull the plug entirely. Half-measures will just frustrate > > people even more. > > If rawhide build targets did drop s390x, presumably ELN could keep > s390x enabled, so Red Hat retains visibility of the state of s390x > builds prior to future RHEL branching. AFAIK ELN buildroot inherits from Rawhide buildroot, because that's how ELN deals with side tags ... Dan > Since ELN builds are async, it would no longer be a problem for the > general Fedora package maintainers. Red Hat downstream maintainers > who own the packeg in RHEL would have to own the problem of keeping > ELN builds working on s390x if problems arose. > > With regards, > Daniel > -- > |: https://berrange.com ~~ https://hachyderm.io/@berrange :| > |: https://libvirt.org ~~ https://entangle-photo.org :| > |: https://pixelfed.art/berrange ~~ https://fstop138.berrange.com :| > > -- > _______________________________________________ > devel mailing list -- [email protected] > To unsubscribe send an email to [email protected] > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/[email protected] > Do not reply to spam, report it: > https://forge.fedoraproject.org/infra/tickets/issues/new -- _______________________________________________ devel mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/[email protected] Do not reply to spam, report it: https://forge.fedoraproject.org/infra/tickets/issues/new
