On Thu, May 7, 2026 at 7:50 AM Daniel P. Berrangé <[email protected]> wrote:
> On Thu, May 07, 2026 at 07:29:52AM -0400, Neal Gompa wrote: > > On Thu, May 7, 2026 at 6:24 AM Hans de Goede <[email protected]> wrote: > > > > > > Hi, > > > > > > On 7-May-26 03:01, Kevin Fenzi via devel-announce wrote: > > > > Just to notify a wider audience, our s390x builders have been offline > > > > today due to a failure of a storage array. > > > > > > > > Folks have been working today to bring it back up, but the fix > > > > will require parts that need to be overnight shipped. > > > > > > > > Hopefully the machines will be back online tomorrow after the > > > > replacement is installed. > > > > > > > > Arched Builds submitted now will wait for s390x builders to > > > > be back to complete. > > > > > > > > You can watch https://www.fedorastatus.org/ and/or > > > > https://forge.fedoraproject.org/infra/tickets/issues/13326 > > > > > > > > for further status. > > > > > > First of all, thank you Kevin and infrastructure team for all > > > your work on the Fedora infra, the below is not meant as > > > criticism of the Fedora infra team. > > > > > > It seems that lately there have been various capacity issues > > > wrt s390x and I think sometimes also powerpc builders. > > > > > > AFAICT Fedora mainly supports these 2 architectures because > > > IBM wants Fedora to supports these 2 architectures. > > > > > > Yet IBM seems to lately consistently fail to make adequate > > > resources (builders, test-vms for contributors) available for > > > these resources. > > > > > > For example I believe that copr has been forced to switch to very > > > slow builds for s390x and powerpc using emulation in qemu on > > > x86_64 builders? > > > > > > Looking at the COPR example I believe the under-resourcing problem > > > has become so big that we (the Fedora project) need to seriously > > > consider if we want to keep supporting s390x and powerpc going > > > forward or if the time has come to drop these ? > > > > > > > We've just had this conversation at the leadership level with Red Hat. > > According to Josh Miller (CC'd to this email so he can jump in if he > > wants to), getting more capacity isn't the issue, the problem is that > > they are currently unable to deploy them in a way that makes the IT > > staff happy to maintain them (the mainframes are managed by Red Hat IT > > these days apparently) and the IT staff they had for this effort left > > two years ago. So right now we're stuck between a rock and a hard > > place because the ability to deploy new given capacity doesn't exist > > due to Red Hat internal issues. > > Was there any indication that Red Hat was actively intending to solve > this IT staff / hardware problem in a reasonable timeframe ? I fear > this is the kind of situation that could be left on the back burner > indefinitely, unless someone has made commitments to address it > within Red Hat as a priority task. Hi Daniel, (and everyone) Over the past several weeks, this has been escalated to us (Red Hat Engineering leadership) and we're actively working on it internally. As with most problems, the situation is complex, but it's a high priority and we've made some progress. It will probably take several more weeks before we have an update, and to be fully open... longer before we have a full solution. We'll keep the Fedora Infrastructure team looped in throughout the process. -Evan
-- _______________________________________________ devel mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/[email protected] Do not reply to spam, report it: https://forge.fedoraproject.org/infra/tickets/issues/new
