Re: Fedora Copr Outage - Updating to a March 2024 Version
Dne 09. 03. 24 v 11:13 Pavel Raiskup napsal(a): Would it make sense to have the "rebuild with ssh access" call mock with "--no-clean-after"? Without that, there is not that much more to inspect than the chroot itself, is there? Or is the idea that we run mock again? The idea, is that you will be able to execute the command in the chroot. So instead of building packages in mode "try this" and "try that", you will be able to run e.g. gcc in that chroot. Maybe put there some print() for debugging. To find the culprit of failure faster. Especially on non-intel architectures. -- Miroslav Suchy, RHCA Red Hat, Manager, Packit and CPT, #brno, #fedora-buildsys -- ___ copr-devel mailing list -- copr-devel@lists.fedorahosted.org To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/copr-devel@lists.fedorahosted.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Fedora Copr Outage - Updating to a March 2024 Version
Hello Michael, On čtvrtek 7. března 2024 16:31:14 CET Michael J Gruber wrote: > Am Do., 7. März 2024 um 13:09 Uhr schrieb Pavel Raiskup : > > > > And the outage is over. Release notes here: > > > > https://docs.pagure.org/copr.copr/release-notes/2024-03-07.html > > > > Special Highlight: Jakub added an exciting new feature in this release, > > allowing users to SSH to Copr builders. This is exciting news! > > > > Happy building! > > Pavel > > Thanks, that is neat indeed :) > > Would it make sense to have the "rebuild with ssh access" call mock > with "--no-clean-after"? Without that, there is not that much more to > inspect than the chroot itself, is there? Or is the idea that we run > mock again? Yes, we originally meant that the build would be resubmitted. Though the idea with --no-clean-after sounds like a reasonable RFE ;) Feel free to open a ticket https://github.com/fedora-copr/copr/issues Pavel > Michael > -- > ___ > copr-devel mailing list -- copr-devel@lists.fedorahosted.org > To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedorahosted.org/archives/list/copr-devel@lists.fedorahosted.org > Do not reply to spam, report it: > https://pagure.io/fedora-infrastructure/new_issue > -- ___ copr-devel mailing list -- copr-devel@lists.fedorahosted.org To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/copr-devel@lists.fedorahosted.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Fedora Copr Outage - Updating to a March 2024 Version
Am Do., 7. März 2024 um 13:09 Uhr schrieb Pavel Raiskup : > > And the outage is over. Release notes here: > > https://docs.pagure.org/copr.copr/release-notes/2024-03-07.html > > Special Highlight: Jakub added an exciting new feature in this release, > allowing users to SSH to Copr builders. This is exciting news! > > Happy building! > Pavel Thanks, that is neat indeed :) Would it make sense to have the "rebuild with ssh access" call mock with "--no-clean-after"? Without that, there is not that much more to inspect than the chroot itself, is there? Or is the idea that we run mock again? Michael -- ___ copr-devel mailing list -- copr-devel@lists.fedorahosted.org To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/copr-devel@lists.fedorahosted.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Fedora Copr Outage - Updating to a March 2024 Version
And the outage is over. Release notes here: https://docs.pagure.org/copr.copr/release-notes/2024-03-07.html Special Highlight: Jakub added an exciting new feature in this release, allowing users to SSH to Copr builders. This is exciting news! Happy building! Pavel On pondělí 4. března 2024 17:53:13 CET Pavel Raiskup wrote: > There will be a Fedora Copr outage starting at > > $ date --date '2024-03-07 11:30 UTC' > > The outage will last approximately 2 hours. The build queue processing will > be stopped during the outage, and the Frontend/Web-UI will be down most of > the time (no new tasks accepted). > > The DNF packages and repositories (hosted on copr-backend) will be available > during this outage. > > Reason for outage: > We will update the infrastructure machines to the latest packages that are > currently being developed. > > Affected Services: > https://copr.fedorainfracloud.org/ > https://download.copr.fedorainfracloud.org/results/ > > Upstream ticket: > https://github.com/fedora-copr/copr/issues/3151 > > Infrastructure ticket: > https://pagure.io/fedora-infrastructure/issue/11808 > > Please join Fedora Build System Matrix channel: > https://matrix.to/#/#buildsys:fedoraproject.org > > Outage Contacts: > @praiskup (Pavel Raiskup) / @frostyx (Jakub Kadlčík) > signature.asc Description: This is a digitally signed message part. -- ___ copr-devel mailing list -- copr-devel@lists.fedorahosted.org To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/copr-devel@lists.fedorahosted.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Fedora Copr Outage - Updating to a March 2024 Version
There will be a Fedora Copr outage starting at $ date --date '2024-03-07 11:30 UTC' The outage will last approximately 2 hours. The build queue processing will be stopped during the outage, and the Frontend/Web-UI will be down most of the time (no new tasks accepted). The DNF packages and repositories (hosted on copr-backend) will be available during this outage. Reason for outage: We will update the infrastructure machines to the latest packages that are currently being developed. Affected Services: https://copr.fedorainfracloud.org/ https://download.copr.fedorainfracloud.org/results/ Upstream ticket: https://github.com/fedora-copr/copr/issues/3151 Infrastructure ticket: https://pagure.io/fedora-infrastructure/issue/11808 Please join Fedora Build System Matrix channel: https://matrix.to/#/#buildsys:fedoraproject.org Outage Contacts: @praiskup (Pavel Raiskup) / @frostyx (Jakub Kadlčík) -- ___ copr-devel mailing list -- copr-devel@lists.fedorahosted.org To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/copr-devel@lists.fedorahosted.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Unplanned Copr outage - 2022-07-08 and 2022-07-09
Copr failed to allocate VMs and assign VMs to the build task for several hours now, which lead to many build failures. Please restart the builds if you were affected. The issue is tracked here: https://pagure.io/copr/copr/issue/2066 Sorry for inconveniences. Pavel ___ copr-devel mailing list -- copr-devel@lists.fedorahosted.org To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/copr-devel@lists.fedorahosted.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Unplanned Copr outage - 2020-09-16 15:00-17:00 UTC
On Wednesday, September 16, 2020 6:03:31 PM CEST Pavel Raiskup wrote: > ... Should be ready in about one hour: > > $ date --date "2020-09-16 17:00:00 UTC" We should be fully back again. Sorry for inconvenience, and happy building. Pavel ___ copr-devel mailing list -- copr-devel@lists.fedorahosted.org To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/copr-devel@lists.fedorahosted.org
Unplanned Copr outage - 2020-09-16 15:00-17:00 UTC
Due to problems with one partition, copr-dist-git machine stopped responding and we had to reboot it. But it got a different IP so we had to change the DNSs records, and now we are waiting to get those propagated. Should be ready in about one hour: $ date --date "2020-09-16 17:00:00 UTC" We have to fix several things in our stack to not repeat this situation next time. Sorry for inconvenience, Pavel ___ copr-devel mailing list -- copr-devel@lists.fedorahosted.org To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/copr-devel@lists.fedorahosted.org
Copr outage - details
tl;dr: On Sunday 23rd February, there will be Copr outage. It will last the whole day. PPC64LE builder and chroots will be deactivated. The PPC64LE builders should be back in a matter of weeks. Hi. As previously announced, Fedora's infrastructure is moving to a different datacenter. For some servers, the move is trivial. Copr servers are different. Copr build system consists of four servers, plus four staging servers. Eight TB of repos, four TB of dist-git, and several small volumes. The original plan was to move to Washington D.C. to IAD2 datacenter by June. Copr is running in Fedora OpenStack, and this cloud has to be evacuated by the beginning of March to free an IP range. The plan was to move Copr to new hardware (thanks to Red Hat) and later move this HW to the new datacenter. That would mean two outages, where the second one lasted at least 15 days (!). We were looking for another option and we found it. We are going to move Copr to Amazon AWS, shutdown old VM on Fedora Cloud. Move the new HW to IAD2 datacenter and then move Copr from AWS to new HW in IAD2 - FYI, the final destination is still subject to change. This still means two outages, but they should be just a few hours. And web server with DNF repositories should be available all the time. The second outage, will happen in May or June. Here is a detailed schedule. We are going to update this table during migration. You can watch the progress during migration: https://docs.google.com/spreadsheets/d/1jrCgdhseZwi91CTRlo9Y5DNwfl9VHoZfjHPK_pocuf4/edit?usp=sharing Here is a short abstract: * we are doing constant rsync to the new location * we spin up staging and production instances in the new location * on Sunday morning we stop frontend and therefore accepting new jobs. The backend with DNF repos will still be operational. * we do final rsync (~6 hours) * around 13:00 UTC we switch DNS to the new location * we then enable all services * once we confirm that everything is operational, the outage will be over There are several caveats: * After we enable services on Sunday 13:00 UTC you may see some failures. Be asured that we will swiftly address them. * Once we get out of Fedora Cloud, we lost access to PPC64LE builders. We are going to deactivate those chroots just before the migration. After a few weeks, we should get it back. ETA is unknown. The worst-case scenario is in June 2020. We will be aiming to bring it back as soon as possible. * Any small issue can easily change the schedule by hours. E.g., just simple 'chown -R' on backend runs ~4 hours. There are going to be three Copr engineers and one fedora-infrastructure member available whole Sunday. If you experienced a problem, do not hesitate to contact us. We are on #fedora-buildsys on Freenode. The link to the outage ticket is: https://pagure.io/fedora-infrastructure/issue/8668 -- Miroslav Suchy, RHCARed Hat, Associate Manager ABRT/Copr, #brno, #fedora-buildsys ___ copr-devel mailing list -- copr-devel@lists.fedorahosted.org To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/copr-devel@lists.fedorahosted.org
Copr outage - moving to new datacenter
FYI, the Copr servers have to move to the new data center by the end of this month. That will include a physical move. In a truck. In the worst-case scenario, it will mean at least 15 days outage. Right now, the whole Copr team + some fedora-infrastructure guys are working on this task. We are trying several approaches on how to handle the migration. I would love to see as small outage as possible. But it seems that at least one-day outage is inevitable. To get you the picture, the migration includes copying 8TB of repositories and more than one TB of dist-git data - which is all in small files. Painfull to copy using a standard method. As soon as we have an exact date and a precise duration of the outage, I will let you know. -- Miroslav Suchy, RHCA Red Hat, Associate Manager ABRT/Copr, #brno, #fedora-buildsys ___ copr-devel mailing list -- copr-devel@lists.fedorahosted.org To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/copr-devel@lists.fedorahosted.org
Planned Copr outage 2018-12-06
Hello, there will be an outage tomorrow starting at 11am UTC and last approximatelly four hours. Copr servers will be upgraded from f27 to f28. More details are here: https://pagure.io/fedora-infrastructure/issue/7422 Best regards! Copr team ___ copr-devel mailing list -- copr-devel@lists.fedorahosted.org To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/copr-devel@lists.fedorahosted.org
Fedora Copr outage Fri 11pm - Sat 10am
Hello, there was a fedorainfracloud outage yesterday that caused all the data volumes that Copr uses to be mounted read-only. We actually caught this problem yesterday on copr-fe but that was only a part of it, which we didn't realize until today. All cloud instances were affected by the ro remount and they are all finally fixed now. We are sorry for rather long Copr unavailability at the time Copr team ___ copr-devel mailing list -- copr-devel@lists.fedorahosted.org To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/copr-devel@lists.fedorahosted.org
Re: Copr outage
On Thu, Oct 4, 2018 at 6:11 AM Michal Novotny wrote: > > Hello, > > there is Copr outage right now. Spawning new builders does not work. See > > https://pagure.io/fedora-infrastructure/issue/7279 Thanks to puiterwijk, this is now fixed! > > We hope it can be fixed very soon. > Copr team ___ copr-devel mailing list -- copr-devel@lists.fedorahosted.org To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/copr-devel@lists.fedorahosted.org
Copr outage
Hello, there is Copr outage right now. Spawning new builders does not work. See https://pagure.io/fedora-infrastructure/issue/7279 We hope it can be fixed very soon. Copr team ___ copr-devel mailing list -- copr-devel@lists.fedorahosted.org To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/copr-devel@lists.fedorahosted.org
Re: COPR outage
fsck checks on both machines have passed successfully. Copr should be back now. Copr team On Fri, May 18, 2018 at 9:16 AM, Michal Novotny wrote: > Hello, > > there were some filesystem errors reported on both copr-dist-git and > copr-backend machines. Right now, both machines are stopped and fsck is > running on them. The service will be unavailable for some time. > > Copr team > ___ copr-devel mailing list -- copr-devel@lists.fedorahosted.org To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/copr-devel@lists.fedorahosted.org/message/TD6SJFE3GXKMCZH2G7G2BRDDUOODQDWX/
COPR outage
Hello, there were some filesystem errors reported on both copr-dist-git and copr-backend machines. Right now, both machines are stopped and fsck is running on them. The service will be unavailable for some time. Copr team ___ copr-devel mailing list -- copr-devel@lists.fedorahosted.org To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/copr-devel@lists.fedorahosted.org/message/P2JEK5UMYSKMBSDB26VVVUZMMDAWSGY4/
Re: COPR outage at 1am UTC
While the outage was planned to last only 1 hour, it lasted three. Sorry for this but we encountered errors that we didn't exactly account for :(. There is much more room for improvement here in the preparation process. There were problems with builder images that were initially prepared on staging machine which caused staging repos being enabled on them. We needed to prepare x86_64 image from scratch. Also copr-keygen was not singing for a while due to change IP after we deployed it on f27 from scratch. The affected builds by this were the following: https://copr.fedorainfracloud.org/coprs/teknoraver/Netperf/build/688726/ https://copr.fedorainfracloud.org/coprs/teknoraver/Netperf/build/688714/ https://copr.fedorainfracloud.org/coprs/g/rubygems/rubygems/build/688728/ https://copr-be.cloud.fedoraproject.org/results/hobbes1069/EPEL/epel-6-i386/00688721-yaml-cpp/ https://copr-be.cloud.fedoraproject.org/results/monotykamary/RetroArch/fedora-27-i386/00688731-RetroArch/ I reran the first three builds from scratch. The second two builds were partially succeeded so I just deleted the unsigned rpms for the failed chroots. They will need to be rerun for the complete results Sorry for the problems With apology COPR Team P.S.: there seems to be a problem with epel-7-ppc64le builds but that is a separate problem and we will take a look at it. ___ copr-devel mailing list -- copr-devel@lists.fedorahosted.org To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org
Re: COPR outage at 1am UTC
I am very sorry. I meant 1pm UTC today. Best regrads On behalf of COPR team clime On Mon, Dec 18, 2017 at 10:03 AM, Michal Novotny wrote: > Hello, > > today there will be a COPR outage lasting approximately for 1 hour > starting at 1 a.m. UTC. We will be upgrading most of the machines to the > latest released Fedora version. > > COPR Team > ___ copr-devel mailing list -- copr-devel@lists.fedorahosted.org To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org
COPR outage at 1am UTC
Hello, today there will be a COPR outage lasting approximately for 1 hour starting at 1 a.m. UTC. We will be upgrading most of the machines to the latest released Fedora version. COPR Team ___ copr-devel mailing list -- copr-devel@lists.fedorahosted.org To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org