On Thu, May 07, 2026 at 07:29:52AM -0400, Neal Gompa wrote: > On Thu, May 7, 2026 at 6:24 AM Hans de Goede <[email protected]> wrote: > > > > Hi, > > > > On 7-May-26 03:01, Kevin Fenzi via devel-announce wrote: > > > Just to notify a wider audience, our s390x builders have been offline > > > today due to a failure of a storage array. > > > > > > Folks have been working today to bring it back up, but the fix > > > will require parts that need to be overnight shipped. > > > > > > Hopefully the machines will be back online tomorrow after the > > > replacement is installed. > > > > > > Arched Builds submitted now will wait for s390x builders to > > > be back to complete. > > > > > > You can watch https://www.fedorastatus.org/ and/or > > > https://forge.fedoraproject.org/infra/tickets/issues/13326 > > > > > > for further status. > > > > First of all, thank you Kevin and infrastructure team for all > > your work on the Fedora infra, the below is not meant as > > criticism of the Fedora infra team. > > > > It seems that lately there have been various capacity issues > > wrt s390x and I think sometimes also powerpc builders. > > > > AFAICT Fedora mainly supports these 2 architectures because > > IBM wants Fedora to supports these 2 architectures. > > > > Yet IBM seems to lately consistently fail to make adequate > > resources (builders, test-vms for contributors) available for > > these resources. > > > > For example I believe that copr has been forced to switch to very > > slow builds for s390x and powerpc using emulation in qemu on > > x86_64 builders? > > > > Looking at the COPR example I believe the under-resourcing problem > > has become so big that we (the Fedora project) need to seriously > > consider if we want to keep supporting s390x and powerpc going > > forward or if the time has come to drop these ? > > > > We've just had this conversation at the leadership level with Red Hat. > According to Josh Miller (CC'd to this email so he can jump in if he > wants to), getting more capacity isn't the issue, the problem is that > they are currently unable to deploy them in a way that makes the IT > staff happy to maintain them (the mainframes are managed by Red Hat IT > these days apparently) and the IT staff they had for this effort left > two years ago. So right now we're stuck between a rock and a hard > place because the ability to deploy new given capacity doesn't exist > due to Red Hat internal issues.
Was there any indication that Red Hat was actively intending to solve this IT staff / hardware problem in a reasonable timeframe ? I fear this is the kind of situation that could be left on the back burner indefinitely, unless someone has made commitments to address it within Red Hat as a priority task. With regards, Daniel -- |: https://berrange.com ~~ https://hachyderm.io/@berrange :| |: https://libvirt.org ~~ https://entangle-photo.org :| |: https://pixelfed.art/berrange ~~ https://fstop138.berrange.com :| -- _______________________________________________ devel mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/[email protected] Do not reply to spam, report it: https://forge.fedoraproject.org/infra/tickets/issues/new
