Re: Fedora Copr Outage - Updating to a March 2024 Version

[Miroslav Suchý]

Dne 09. 03. 24 v 11:13 Pavel Raiskup napsal(a):

Would it make sense to have the "rebuild with ssh access" call mock
with "--no-clean-after"? Without that, there is not that much more to
inspect than the chroot itself, is there? Or is the idea that we run
mock again?


The idea, is that you will be able to execute the command in the chroot.

So instead of building packages in mode "try this" and "try that", you will be able to run e.g. gcc in that chroot. 
Maybe put there some print() for debugging. To find the culprit of failure faster. Especially on non-intel architectures.



--
Miroslav Suchy, RHCA
Red Hat, Manager, Packit and CPT, #brno, #fedora-buildsys
--
___
copr-devel mailing list -- copr-devel@lists.fedorahosted.org
To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedorahosted.org/archives/list/copr-devel@lists.fedorahosted.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Fedora Copr Outage - Updating to a March 2024 Version

[Pavel Raiskup]

Hello Michael,

On čtvrtek 7. března 2024 16:31:14 CET Michael J Gruber wrote:
> Am Do., 7. März 2024 um 13:09 Uhr schrieb Pavel Raiskup :
> >
> > And the outage is over.  Release notes here:
> >
> > https://docs.pagure.org/copr.copr/release-notes/2024-03-07.html
> >
> > Special Highlight: Jakub added an exciting new feature in this release,
> > allowing users to SSH to Copr builders.  This is exciting news!
> >
> > Happy building!
> > Pavel
> 
> Thanks, that is neat indeed :)
> 
> Would it make sense to have the "rebuild with ssh access" call mock
> with "--no-clean-after"? Without that, there is not that much more to
> inspect than the chroot itself, is there? Or is the idea that we run
> mock again?

Yes, we originally meant that the build would be resubmitted.  Though
the idea with --no-clean-after sounds like a reasonable RFE ;) Feel free
to open a ticket https://github.com/fedora-copr/copr/issues

Pavel

> Michael
> --
> ___
> copr-devel mailing list -- copr-devel@lists.fedorahosted.org
> To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org
> Fedora Code of Conduct: 
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedorahosted.org/archives/list/copr-devel@lists.fedorahosted.org
> Do not reply to spam, report it: 
> https://pagure.io/fedora-infrastructure/new_issue
> 



--
___
copr-devel mailing list -- copr-devel@lists.fedorahosted.org
To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedorahosted.org/archives/list/copr-devel@lists.fedorahosted.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Fedora Copr Outage - Updating to a March 2024 Version

[Michael J Gruber]

Am Do., 7. März 2024 um 13:09 Uhr schrieb Pavel Raiskup :
>
> And the outage is over.  Release notes here:
>
> https://docs.pagure.org/copr.copr/release-notes/2024-03-07.html
>
> Special Highlight: Jakub added an exciting new feature in this release,
> allowing users to SSH to Copr builders.  This is exciting news!
>
> Happy building!
> Pavel

Thanks, that is neat indeed :)

Would it make sense to have the "rebuild with ssh access" call mock
with "--no-clean-after"? Without that, there is not that much more to
inspect than the chroot itself, is there? Or is the idea that we run
mock again?

Michael
--
___
copr-devel mailing list -- copr-devel@lists.fedorahosted.org
To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedorahosted.org/archives/list/copr-devel@lists.fedorahosted.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Fedora Copr Outage - Updating to a March 2024 Version

[Pavel Raiskup]

And the outage is over.  Release notes here:

https://docs.pagure.org/copr.copr/release-notes/2024-03-07.html

Special Highlight: Jakub added an exciting new feature in this release,
allowing users to SSH to Copr builders.  This is exciting news!

Happy building!
Pavel

On pondělí 4. března 2024 17:53:13 CET Pavel Raiskup wrote:
> There will be a Fedora Copr outage starting at
> 
> $ date --date '2024-03-07 11:30 UTC'
> 
> The outage will last approximately 2 hours. The build queue processing will
> be stopped during the outage, and the Frontend/Web-UI will be down most of
> the time (no new tasks accepted).
> 
> The DNF packages and repositories (hosted on copr-backend) will be available
> during this outage.
> 
> Reason for outage:
> We will update the infrastructure machines to the latest packages that are
> currently being developed.
> 
> Affected Services:
> https://copr.fedorainfracloud.org/
> https://download.copr.fedorainfracloud.org/results/
> 
> Upstream ticket:
> https://github.com/fedora-copr/copr/issues/3151
> 
> Infrastructure ticket:
> https://pagure.io/fedora-infrastructure/issue/11808
> 
> Please join Fedora Build System Matrix channel:
> https://matrix.to/#/#buildsys:fedoraproject.org
> 
> Outage Contacts:
> @praiskup (Pavel Raiskup) / @frostyx (Jakub Kadlčík)
> 



signature.asc
Description: This is a digitally signed message part.
--
___
copr-devel mailing list -- copr-devel@lists.fedorahosted.org
To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedorahosted.org/archives/list/copr-devel@lists.fedorahosted.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Fedora Copr Outage - Updating to a March 2024 Version

[Pavel Raiskup]

There will be a Fedora Copr outage starting at

$ date --date '2024-03-07 11:30 UTC'

The outage will last approximately 2 hours. The build queue processing will
be stopped during the outage, and the Frontend/Web-UI will be down most of
the time (no new tasks accepted).

The DNF packages and repositories (hosted on copr-backend) will be available
during this outage.

Reason for outage:
We will update the infrastructure machines to the latest packages that are
currently being developed.

Affected Services:
https://copr.fedorainfracloud.org/
https://download.copr.fedorainfracloud.org/results/

Upstream ticket:
https://github.com/fedora-copr/copr/issues/3151

Infrastructure ticket:
https://pagure.io/fedora-infrastructure/issue/11808

Please join Fedora Build System Matrix channel:
https://matrix.to/#/#buildsys:fedoraproject.org

Outage Contacts:
@praiskup (Pavel Raiskup) / @frostyx (Jakub Kadlčík)


--
___
copr-devel mailing list -- copr-devel@lists.fedorahosted.org
To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedorahosted.org/archives/list/copr-devel@lists.fedorahosted.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Unplanned Copr outage - 2022-07-08 and 2022-07-09

[Pavel Raiskup]

Copr failed to allocate VMs and assign VMs to the build task for several hours
now, which lead to many build failures.  Please restart the builds if you were
affected.  The issue is tracked here: https://pagure.io/copr/copr/issue/2066

Sorry for inconveniences.
Pavel


___
copr-devel mailing list -- copr-devel@lists.fedorahosted.org
To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedorahosted.org/archives/list/copr-devel@lists.fedorahosted.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Unplanned Copr outage - 2020-09-16 15:00-17:00 UTC

[Pavel Raiskup]

Due to problems with one partition, copr-dist-git machine stopped
responding and we had to reboot it.  But it got a different IP so we had
to change the DNSs records, and now we are waiting to get those
propagated.  Should be ready in about one hour:

$ date --date "2020-09-16 17:00:00 UTC"

We have to fix several things in our stack to not repeat this situation next
time.  Sorry for inconvenience,

Pavel


___
copr-devel mailing list -- copr-devel@lists.fedorahosted.org
To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedorahosted.org/archives/list/copr-devel@lists.fedorahosted.org

Copr outage - details

[Miroslav Suchý]

tl;dr: On Sunday 23rd February, there will be Copr outage. It will last the 
whole day.
PPC64LE builder and chroots will be deactivated. The PPC64LE builders should be 
back in a matter of weeks.

Hi.
As previously announced, Fedora's infrastructure is moving to a different 
datacenter. For some servers, the move is
trivial. Copr servers are different. Copr build system consists of four 
servers, plus four staging servers. Eight TB of
repos, four TB of dist-git, and several small volumes.

The original plan was to move to Washington D.C. to IAD2 datacenter by June.  
Copr is running in Fedora OpenStack, and
this cloud has to be evacuated by the beginning of March to free an IP range.
The plan was to move Copr to new hardware (thanks to Red Hat) and later move 
this HW to the new datacenter.  That would
mean two outages, where the second one lasted at least 15 days (!).

We were looking for another option and we found it. We are going to move Copr 
to Amazon AWS, shutdown old VM on Fedora
Cloud. Move the new HW to IAD2 datacenter and then move Copr from AWS to new HW 
in IAD2 - FYI, the final destination is
still subject to change. This still means two outages, but they should be just 
a few hours. And web server with DNF
repositories should be available all the time.
The second outage, will happen in May or June.

Here is a detailed schedule. We are going to update this table during 
migration. You can watch the progress during
migration:

https://docs.google.com/spreadsheets/d/1jrCgdhseZwi91CTRlo9Y5DNwfl9VHoZfjHPK_pocuf4/edit?usp=sharing

Here is a short abstract:

 * we are doing constant rsync to the new location
 * we spin up staging and production instances in the new location
 * on Sunday morning we stop frontend and therefore accepting new jobs. The 
backend with DNF repos will still be
operational.
 * we do final rsync (~6 hours)
 * around 13:00 UTC we switch DNS to the new location
 * we then enable all services
 * once we confirm that everything is operational, the outage will be over

There are several caveats:

 * After we enable services on Sunday 13:00 UTC you may see some failures. Be 
asured that we will swiftly address them.
 * Once we get out of Fedora Cloud, we lost access to PPC64LE builders. We are 
going to deactivate those chroots just
before the migration. After a few weeks, we should get it back. ETA is unknown. 
The worst-case scenario is in June 2020.
We will be aiming to bring it back as soon as possible.
 * Any small issue can easily change the schedule by hours. E.g., just simple 
'chown -R' on backend runs ~4 hours.

There are going to be three Copr engineers and one fedora-infrastructure member 
available whole Sunday. If you
experienced a problem, do not hesitate to contact us. We are on 
#fedora-buildsys on Freenode.

The link to the outage ticket is:
https://pagure.io/fedora-infrastructure/issue/8668

-- 
Miroslav Suchy, RHCARed Hat, Associate Manager ABRT/Copr, #brno, 
#fedora-buildsys
___
copr-devel mailing list -- copr-devel@lists.fedorahosted.org
To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedorahosted.org/archives/list/copr-devel@lists.fedorahosted.org

Planned Copr outage 2018-12-06

[Michal Novotny]

Hello,

there will be an outage tomorrow starting at 11am UTC and last
approximatelly four hours. Copr servers will be upgraded from f27 to f28.

More details are here: https://pagure.io/fedora-infrastructure/issue/7422

Best regards!
Copr team
___
copr-devel mailing list -- copr-devel@lists.fedorahosted.org
To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedorahosted.org/archives/list/copr-devel@lists.fedorahosted.org

Fedora Copr outage Fri 11pm - Sat 10am

[Michal Novotny]

Hello,

there was a fedorainfracloud outage yesterday that caused all the data
volumes that Copr uses to be mounted read-only. We actually caught this
problem yesterday on copr-fe but that was only a part of it, which we
didn't realize until today. All cloud instances were affected by the ro
remount and they are all finally fixed now.

We are sorry for rather long Copr unavailability at the time
Copr team
___
copr-devel mailing list -- copr-devel@lists.fedorahosted.org
To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedorahosted.org/archives/list/copr-devel@lists.fedorahosted.org

Re: Copr outage

[Michal Novotny]

On Thu, Oct 4, 2018 at 6:11 AM Michal Novotny  wrote:
>
> Hello,
>
> there is Copr outage right now. Spawning new builders does not work. See
>
> https://pagure.io/fedora-infrastructure/issue/7279

Thanks to puiterwijk, this is now fixed!

>
> We hope it can be fixed very soon.
> Copr team
___
copr-devel mailing list -- copr-devel@lists.fedorahosted.org
To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedorahosted.org/archives/list/copr-devel@lists.fedorahosted.org

Copr outage

[Michal Novotny]

Hello,

there is Copr outage right now. Spawning new builders does not work. See

https://pagure.io/fedora-infrastructure/issue/7279

We hope it can be fixed very soon.
Copr team
___
copr-devel mailing list -- copr-devel@lists.fedorahosted.org
To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedorahosted.org/archives/list/copr-devel@lists.fedorahosted.org

Re: COPR outage

[Michal Novotny]

fsck checks on both machines have passed successfully. Copr should be back
now.

Copr team

On Fri, May 18, 2018 at 9:16 AM, Michal Novotny  wrote:

> Hello,
>
> there were some filesystem errors reported on both copr-dist-git and
> copr-backend machines. Right now, both machines are stopped and fsck is
> running on them. The service will be unavailable for some time.
>
> Copr team
>
___
copr-devel mailing list -- copr-devel@lists.fedorahosted.org
To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/copr-devel@lists.fedorahosted.org/message/TD6SJFE3GXKMCZH2G7G2BRDDUOODQDWX/

Re: COPR outage at 1am UTC

[Michal Novotný]

While the outage was planned to last only 1 hour, it lasted three. Sorry for 
this but we encountered errors that we didn't exactly account for :(. There is 
much more room for improvement here in the preparation process.

There were problems with builder images that were initially prepared on staging 
machine which caused staging repos being enabled on them. We needed to prepare 
x86_64 image from scratch. Also copr-keygen was not singing for a while due to 
change IP after we deployed it on f27 from scratch. The affected builds by this 
were the following:

https://copr.fedorainfracloud.org/coprs/teknoraver/Netperf/build/688726/
https://copr.fedorainfracloud.org/coprs/teknoraver/Netperf/build/688714/
https://copr.fedorainfracloud.org/coprs/g/rubygems/rubygems/build/688728/
https://copr-be.cloud.fedoraproject.org/results/hobbes1069/EPEL/epel-6-i386/00688721-yaml-cpp/
https://copr-be.cloud.fedoraproject.org/results/monotykamary/RetroArch/fedora-27-i386/00688731-RetroArch/

I reran the first three builds from scratch. The second two builds were 
partially succeeded so I just deleted the unsigned rpms for the failed chroots. 
They will need to be rerun for the complete results

Sorry for the problems
With apology
COPR Team

P.S.: there seems to be a problem with epel-7-ppc64le builds but that is a 
separate problem and we will take a look at it.
___
copr-devel mailing list -- copr-devel@lists.fedorahosted.org
To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org

Re: COPR outage at 1am UTC

[Michal Novotny]

I am very sorry. I meant 1pm UTC today.

Best regrads
On behalf of COPR team
clime

On Mon, Dec 18, 2017 at 10:03 AM, Michal Novotny <cl...@redhat.com> wrote:

> Hello,
>
> today there will be a COPR outage lasting approximately for 1 hour
> starting at 1 a.m. UTC. We will be upgrading most of the machines to the
> latest released Fedora version.
>
> COPR Team
>
___
copr-devel mailing list -- copr-devel@lists.fedorahosted.org
To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org

COPR outage at 1am UTC

[Michal Novotny]

Hello,

today there will be a COPR outage lasting approximately for 1 hour starting
at 1 a.m. UTC. We will be upgrading most of the machines to the latest
released Fedora version.

COPR Team
___
copr-devel mailing list -- copr-devel@lists.fedorahosted.org
To unsubscribe send an email to copr-devel-le...@lists.fedorahosted.org