On Thu, May 07, 2026 at 10:43:37AM -0500, Michael Catanzaro wrote: > Wow. :( > > This week's total outage on s390x is an unusual occurrence, and we shouldn't > need to change anything based on this. I'm more concerned with *persistent* > capacity issues. We have a ticket > https://forge.fedoraproject.org/infra/tickets > /issues/13141 but I think it understates the severity of the problem. It's > normal for my heavybuilder channel builds to wait up to two days before an > s390x build even *starts*. That significantly delays Fedora updates for all > architectures.
Sigh. In that ticket... I see no mention of delays in the heavybuilder channel. Can you pretty pretty please with a cherry on top note that when you see it? That ticket is about cases where a bunch of very large builds that take a long time are fired off at once (ie, for rawhide,f44,f43,f42,epel10,epel10.1,epel10.2,eln) and occupy builders. heavybuilder builds should be in another channel and only affected by other heavybuilder builds. ie, if you fire off 5 webkitgtks they will each wait for the previous. > > s390x has been a problem for too long, and it's time to do something about it. > I was planning to make almost the same proposal as Hans, except I don't think > the problems with ppc64le are nearly as serious as the problems with s390x. > Our > ppc64le builders are too slow relative to other architectures, and that's a > problem, but they are not *severely* slow like the s390x builders. I think > doubling the ppc64le builders would be adequate, whereas s390x needs to be as I noted in the FRCL call we already were planning on adding power10 capacity (but I have no idea if it's affected by the supply chain issues that most hardware has now). > quadrupled or more. Also, packages rarely fail to build only on ppc64le -- if > it fails there, it's very likely going to fail on at least one other > architecture as well -- whereas it's common for builds to fail only on s390x > due to endianness issues. So it's really only s390x that is causing problems > for Fedora. > > On the other hand, our Workstation usage stats > https://forge.fedoraproject.org/ > workstation/tickets/issues/500#issuecomment-361654 indicate we have almost no > users on ppc64le, so while ppc64le is not causing problems, I'd argue it's not > worth building anymore on that basis alone. That should move to secondary > infrastructure. > > Proposal: future stable Fedoras will not be built for s390x or ppc64le > beginning with Fedora 45, leaving these architectures enabled in rawhide and > ELN only. That should eventually reduce load on the builders when Fedora 43 > and > Fedora 44 reach EOL, by reducing the number of total builds. It also allows > Red > Hat 3 months to improve capacity before Fedora 45 is branched if Red Hat > doesn't like this plan. If that's still not enough to resolve our capacity > problems, then in February next year (arbitrarily selected to coincide with > F46 > branching) we should turn off s390x in rawhide as well, leaving it only in > ELN, > so it will no longer inconvenience Fedora community members at all. I don't think thats very tenable... if we are dropping it, we should do that. Partly dropping it just will lead to more pain. > Proposal 2: do the same for i686 beginning with Fedora 46, if nobody is > working > on a practical plan for reducing i686 architecture support before then. Fedora > really needs to reduce i686 down to only those packages necessary to keep > Bazzite and possibly Wine working, but nobody has been working on this: after > the change proposal to turn off i686 was defeated, everybody just stopped and > moved on instead of actually trying to fix things. That's the wrong default > action: the default action should be to turn off i686 if nobody is interested > in working on saving it. Otherwise, the day will eventually come when we can > no > longer link some big important package on i686 -- my fear is WebKitGTK -- and > then *all* architectures will be blocked from receiving updates. See the long discussion thread about this... sadly, tools people want to keep i686 around for their needs and all the wine/steam stuff, etc. ;( kevin -- _______________________________________________ devel mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/[email protected] Do not reply to spam, report it: https://forge.fedoraproject.org/infra/tickets/issues/new
