Re: Planned Outage - s390x builders - 2023-02-10 08:00 UTC -> 22:00 UTC
On Mon, Feb 13, 2023 at 03:01:34PM +0100, Luna Jernberg wrote: > ssmoogen > Current s390x status: Networking and storage on systems is not acting > appropriately. Site admins are trying to figure out what is going on. Everything is back up now. Sorry for the extended outage. We will will talking with various folks to try and prevent this happening again. kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Planned Outage - s390x builders - 2023-02-10 08:00 UTC -> 22:00 UTC
ssmoogen Current s390x status: Networking and storage on systems is not acting appropriately. Site admins are trying to figure out what is going on. On 2/12/23, Kevin Fenzi wrote: > On Sun, Feb 12, 2023 at 09:30:53PM +, Sérgio Basto wrote: >> On Sat, 2023-02-11 at 10:54 +0100, Dan Horák wrote: >> > On Sat, 11 Feb 2023 09:37:08 + >> > "Richard W.M. Jones" wrote: >> > >> > > On Thu, Feb 09, 2023 at 11:35:14AM -0500, Stephen Smoogen wrote: >> > > > Affected Services: >> > > > >> > > > s390x koji builders. Jobs will queue up, but not be processed >> > > > until >> > > > the builders are back online. Maintainers are urged to just avoid >> > > > submitting builds just before and during this outage. >> > > >> > > Do we need to cancel & resubmit hanging jobs, eg: >> > > >> > > https://koji.fedoraproject.org/koji/taskinfo?taskID=97341784 >> > >> > those are OK, they are waiting. When s390x builders are back up, they >> > will pick them and finish the builds. >> >> >> I saw that s390x are back but seems to me that they are all stuck :( > > They are not back. At least I see no evidence that they are. ;) > > Sorry this outage has gone on. The power was restored pretty quickly, > but apparently there's some issue with the mainframe storage now. ;( > > I hope it will be sorted out tomorrow. > > kevin > ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Planned Outage - s390x builders - 2023-02-10 08:00 UTC -> 22:00 UTC
On Sun, Feb 12, 2023 at 09:30:53PM +, Sérgio Basto wrote: > On Sat, 2023-02-11 at 10:54 +0100, Dan Horák wrote: > > On Sat, 11 Feb 2023 09:37:08 + > > "Richard W.M. Jones" wrote: > > > > > On Thu, Feb 09, 2023 at 11:35:14AM -0500, Stephen Smoogen wrote: > > > > Affected Services: > > > > > > > > s390x koji builders. Jobs will queue up, but not be processed > > > > until > > > > the builders are back online. Maintainers are urged to just avoid > > > > submitting builds just before and during this outage. > > > > > > Do we need to cancel & resubmit hanging jobs, eg: > > > > > > https://koji.fedoraproject.org/koji/taskinfo?taskID=97341784 > > > > those are OK, they are waiting. When s390x builders are back up, they > > will pick them and finish the builds. > > > I saw that s390x are back but seems to me that they are all stuck :( They are not back. At least I see no evidence that they are. ;) Sorry this outage has gone on. The power was restored pretty quickly, but apparently there's some issue with the mainframe storage now. ;( I hope it will be sorted out tomorrow. kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Planned Outage - s390x builders - 2023-02-10 08:00 UTC -> 22:00 UTC
On Sat, 2023-02-11 at 10:54 +0100, Dan Horák wrote: > On Sat, 11 Feb 2023 09:37:08 + > "Richard W.M. Jones" wrote: > > > On Thu, Feb 09, 2023 at 11:35:14AM -0500, Stephen Smoogen wrote: > > > Affected Services: > > > > > > s390x koji builders. Jobs will queue up, but not be processed > > > until > > > the builders are back online. Maintainers are urged to just avoid > > > submitting builds just before and during this outage. > > > > Do we need to cancel & resubmit hanging jobs, eg: > > > > https://koji.fedoraproject.org/koji/taskinfo?taskID=97341784 > > those are OK, they are waiting. When s390x builders are back up, they > will pick them and finish the builds. I saw that s390x are back but seems to me that they are all stuck :( -- Sérgio M. B. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Planned Outage - s390x builders - 2023-02-10 08:00 UTC -> 22:00 UTC
On Sat, 11 Feb 2023 09:37:08 + "Richard W.M. Jones" wrote: > On Thu, Feb 09, 2023 at 11:35:14AM -0500, Stephen Smoogen wrote: > > Affected Services: > > > > s390x koji builders. Jobs will queue up, but not be processed until > > the builders are back online. Maintainers are urged to just avoid > > submitting builds just before and during this outage. > > Do we need to cancel & resubmit hanging jobs, eg: > > https://koji.fedoraproject.org/koji/taskinfo?taskID=97341784 those are OK, they are waiting. When s390x builders are back up, they will pick them and finish the builds. Dan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Planned Outage - s390x builders - 2023-02-10 08:00 UTC -> 22:00 UTC
On Thu, Feb 09, 2023 at 11:35:14AM -0500, Stephen Smoogen wrote: > Affected Services: > > s390x koji builders. Jobs will queue up, but not be processed until > the builders are back online. Maintainers are urged to just avoid > submitting builds just before and during this outage. Do we need to cancel & resubmit hanging jobs, eg: https://koji.fedoraproject.org/koji/taskinfo?taskID=97341784 Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Planned Outage - s390x builders - 2023-02-10 08:00 UTC -> 22:00 UTC
Planned Outage - s390x builders - 2023-02-10 08:00 UTC There will be an outage starting at 2023-02-11 08:00 UTC, which will last approximately 12 to 24 hours. Systems will be powered off and then power to the facility will be dropped. After electrical work is completed work on some network equipment may extend the outage due to a different planned outage. To convert UTC to your local time, take a look at http://fedoraproject.org/wiki/Infrastructure/UTCHowto or run: date -d '2023-02-10 08:00UTC' Reason for outage: The facility that houses the Fedora s390x builders will be taking down power and network in order to troubleshoot current state and problems of the power infrastructure there. This data will hopefully allow fixes/solutions to make the power distribution there most robust and reliable. After power is restored, there are additional network upgrades and replacements which are planned to occur but have been delayed due to the other work. Affected Services: s390x koji builders. Jobs will queue up, but not be processed until the builders are back online. Maintainers are urged to just avoid submitting builds just before and during this outage. Ticket Link: https://pagure.io/fedora-infrastructure/issue/11082 Please join #fedora-admin or #fedora-noc on irc.libera.chat or add comments to the ticket for this outage above. -- Stephen Smoogen, Red Hat Automotive Let us be kind to one another, for most of us are fighting a hard battle. -- Ian MacClaren ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
