Re: GitLab Container Registry scheduled maintenance, Friday May 29, 10 UTC

[Michael Catanzaro]

On Sun, May 31, 2020 at 8:14 pm, Bartłomiej Piotrowski 
 wrote:
Previously uploaded cache was still being used. Jobs reliably pass 
now,

or fail for reasons unrelated to infrastructure.


The infrastructure seems to be fixed now, thanks!


___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: GitLab Container Registry scheduled maintenance, Friday May 29, 10 UTC

[Bartłomiej Piotrowski]

On 31/05/2020 20.14, Bartłomiej Piotrowski wrote:
> I have the impression that CI changes made by infrastructure team are
> whims
It should be "are considered whims".
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: GitLab Container Registry scheduled maintenance, Friday May 29, 10 UTC

[Bartłomiej Piotrowski]

On 31/05/2020 19.23, Matthias Clasen wrote:
> On Sun, May 31, 2020 at 12:46 PM Michael Catanzaro  > wrote:
> 
> In GTK, I see the majority of ci runs now fails with:  "The script
> exceeded the maximum execution time set for the job"

Two runners in London have very slow connectivity to GitLab. They are
paused now, I will reach out to their donator tomorrow.
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: GitLab Container Registry scheduled maintenance, Friday May 29, 10 UTC

[Bartłomiej Piotrowski]

On 31/05/2020 18.46, Michael Catanzaro wrote:
> On Sun, May 31, 2020 at 11:35 am, Michael Catanzaro
>  wrote:
>> Still failing, now on a different runner:
>>
>> https://gitlab.gnome.org/GNOME/epiphany/-/jobs/744761
>>
>> Can you please try triggering a new CI run to make sure it actually
>> passes...?
> 
> I just triggered seven pipelines and five of them failed:
> 
> https://gitlab.gnome.org/jbrummer/epiphany/-/jobs/744763
> https://gitlab.gnome.org/jbrummer/epiphany/-/jobs/744768
> https://gitlab.gnome.org/GNOME/epiphany/-/jobs/743538
> https://gitlab.gnome.org/GNOME/epiphany/-/jobs/744761
> https://gitlab.gnome.org/GNOME/epiphany/-/jobs/744762
> 
> 
> 

And another 7 worked just fine, so I don't think your claim I haven't
checked is fair at all.

Previously uploaded cache was still being used. Jobs reliably pass now,
or fail for reasons unrelated to infrastructure.

I have the impression that CI changes made by infrastructure team are
whims rather than requests made by other developers we try to implement
while dealing with multiple moving parts we haven't developed ourselves.
I assure you we also wish to spend our time otherwise but it is what it is.
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: GitLab Container Registry scheduled maintenance, Friday May 29, 10 UTC

[Matthias Clasen via desktop-devel-list]

On Sun, May 31, 2020 at 12:46 PM Michael Catanzaro 
wrote:

In GTK, I see the majority of ci runs now fails with:  "The script exceeded
the maximum execution time set for the job"
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: GitLab Container Registry scheduled maintenance, Friday May 29, 10 UTC

[Michael Catanzaro]

On Sun, May 31, 2020 at 11:35 am, Michael Catanzaro 
 wrote:

Still failing, now on a different runner:

https://gitlab.gnome.org/GNOME/epiphany/-/jobs/744761

Can you please try triggering a new CI run to make sure it actually 
passes...?


I just triggered seven pipelines and five of them failed:

https://gitlab.gnome.org/jbrummer/epiphany/-/jobs/744763
https://gitlab.gnome.org/jbrummer/epiphany/-/jobs/744768
https://gitlab.gnome.org/GNOME/epiphany/-/jobs/743538
https://gitlab.gnome.org/GNOME/epiphany/-/jobs/744761
https://gitlab.gnome.org/GNOME/epiphany/-/jobs/744762



___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: GitLab Container Registry scheduled maintenance, Friday May 29, 10 UTC

[Michael Catanzaro]

On Sun, May 31, 2020 at 5:43 pm, Bartłomiej Piotrowski 
 wrote:
I don't see a good explanation why this particular runner is unhappy 
at

this point. Distributed and local cache are disabled and some flatpak
jobs pass just fine. I paused it for now.


Still failing, now on a different runner:

https://gitlab.gnome.org/GNOME/epiphany/-/jobs/744761

Can you please try triggering a new CI run to make sure it actually 
passes...?



___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: GitLab Container Registry scheduled maintenance, Friday May 29, 10 UTC

[Bartłomiej Piotrowski]

On 31/05/2020 16.09, Michael Catanzaro wrote:
> It's still broken as of right now, see e.g.
> 
> https://gitlab.gnome.org/Vanadiae/epiphany/-/jobs/744498
> 
> Or:
> 
> https://gitlab.gnome.org/GNOME/epiphany/-/jobs/744499
> 
> Or:
> 
> https://gitlab.gnome.org/jbrummer/epiphany/-/jobs/744537
> 
> It seems like nearly 100% of jobs are still failing. :(

I don't see a good explanation why this particular runner is unhappy at
this point. Distributed and local cache are disabled and some flatpak
jobs pass just fine. I paused it for now.

___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: GitLab Container Registry scheduled maintenance, Friday May 29, 10 UTC

[Michael Catanzaro]

It's still broken as of right now, see e.g.

https://gitlab.gnome.org/Vanadiae/epiphany/-/jobs/744498

Or:

https://gitlab.gnome.org/GNOME/epiphany/-/jobs/744499

Or:

https://gitlab.gnome.org/jbrummer/epiphany/-/jobs/744537

It seems like nearly 100% of jobs are still failing. :(



___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: GitLab Container Registry scheduled maintenance, Friday May 29, 10 UTC

[Bartłomiej Piotrowski]

On 30/05/2020 18.19, Michael Catanzaro wrote:
> On Thu, May 28, 2020 at 7:46 pm, Zander Brown  wrote:
>> Good luck sysadmining, sure it'll all be fine
> 
> So I also thought this would be low-risk maintenance, but half our CI
> runs are still failing and that halts development on projects that
> require successful CI. Definitely not a good situation right ahead of a
> release deadline.
> 
> 
> ___
> desktop-devel-list mailing list
> desktop-devel-list@gnome.org
> https://mail.gnome.org/mailman/listinfo/desktop-devel-list

For the record, all issues were caused by distributed cache, and have
nothing to do with the cleanup done during maintenance window.
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: GitLab Container Registry scheduled maintenance, Friday May 29, 10 UTC

[Bartłomiej Piotrowski]

On 30/05/2020 17.02, Michael Catanzaro wrote:
> On Fri, May 29, 2020 at 5:08 pm, Bartłomiej Piotrowski
>  wrote:
>> gnome-robots failure is indeed caused by the cache and our glorious
>> Docker/flatpak/AppArmor/SELinux/seccomp combination. Disabled
>> distributed cache for Flatpak runners for the time being.
>>
>> Nautilus was unlucky and landed on runner with almost full disk. Freed
>> the space there; I'll think about more aggressive space reclaiming.
> 
> We are still seeing strange permissions errors after yesterday's
> maintenance. E.g. https://gitlab.gnome.org/GNOME/epiphany/-/jobs/742554
> 
> 

I missed this runner when disabling the cache. It's fixed now.
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: GitLab Container Registry scheduled maintenance, Friday May 29, 10 UTC

[Michael Catanzaro]

On Thu, May 28, 2020 at 7:46 pm, Zander Brown  wrote:

Good luck sysadmining, sure it'll all be fine


So I also thought this would be low-risk maintenance, but half our CI 
runs are still failing and that halts development on projects that 
require successful CI. Definitely not a good situation right ahead of a 
release deadline.



___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: GitLab Container Registry scheduled maintenance, Friday May 29, 10 UTC

[Bartłomiej Piotrowski]

On 29/05/2020 16.31, Ondrej Holy wrote:
> Hi,
> 
> I also see failures for Nautilus CI:
> https://gitlab.gnome.org/GNOME/nautilus/pipelines. Various permission
> denied, or space left errors. But I guess that it rather relates to
> the distributed cache enablement. Bartłomiej, can you please take a
> look?
> 
> O.
> 
> pá 29. 5. 2020 v 16:27 odesílatel Michael Catanzaro
>  napsal:
> 
>>
>>
>> Looks like CI jobs are now failing when trying to download from GitLab
>> registry:
>>
>> https://gitlab.gnome.org/GNOME/gnome-robots/-/jobs/741566
>>
>>
>> ___
>> desktop-devel-list mailing list
>> desktop-devel-list@gnome.org
>> https://mail.gnome.org/mailman/listinfo/desktop-devel-list
>>
> 

gnome-robots failure is indeed caused by the cache and our glorious
Docker/flatpak/AppArmor/SELinux/seccomp combination. Disabled
distributed cache for Flatpak runners for the time being.

Nautilus was unlucky and landed on runner with almost full disk. Freed
the space there; I'll think about more aggressive space reclaiming.

Bart
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: GitLab Container Registry scheduled maintenance, Friday May 29, 10 UTC

[Ondrej Holy via desktop-devel-list]

Hi,

I also see failures for Nautilus CI:
https://gitlab.gnome.org/GNOME/nautilus/pipelines. Various permission
denied, or space left errors. But I guess that it rather relates to
the distributed cache enablement. Bartłomiej, can you please take a
look?

O.

pá 29. 5. 2020 v 16:27 odesílatel Michael Catanzaro
 napsal:

>
>
> Looks like CI jobs are now failing when trying to download from GitLab
> registry:
>
> https://gitlab.gnome.org/GNOME/gnome-robots/-/jobs/741566
>
>
> ___
> desktop-devel-list mailing list
> desktop-devel-list@gnome.org
> https://mail.gnome.org/mailman/listinfo/desktop-devel-list
>

___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: GitLab Container Registry scheduled maintenance, Friday May 29, 10 UTC

[Michael Catanzaro]

Looks like CI jobs are now failing when trying to download from GitLab 
registry:


https://gitlab.gnome.org/GNOME/gnome-robots/-/jobs/741566


___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: GitLab Container Registry scheduled maintenance, Friday May 29, 10 UTC

[Bartłomiej Piotrowski]

Hello,

The pruning job has finished and container registry is available to
write again.

Bart
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: GitLab Container Registry scheduled maintenance, Friday May 29, 10 UTC

[Milan Crha via desktop-devel-list]

Hi,

On Thu, 2020-05-28 at 19:46 +0100, Zander Brown wrote:
> I think it's clear it was supposed to be humorous

Right, right, I didn't mean to offend anyone. If I did, I apologize for
that.


I guess Fridays are also not the best days for maintenance, unless
admins are ready to deal with issues over the weekend. It's only my
humble opinion, of course. I've no idea of usual admin workflow.

I definitely appreciate admin's work, that's not a toy to keep the
infrastructure running, with all its services. Thanks for it.
Bye,
Milan

___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: GitLab Container Registry scheduled maintenance, Friday May 29, 10 UTC

[Zander Brown]

On Thu, 2020-05-28 at 20:14 +0200, Andre Klapper wrote:
> On Thu, 2020-05-28 at 19:16 +0200, Milan Crha via desktop-devel-list
> wrote:
> > this is the second time you plan something bigger, some server
> > maintenance, at the days when the releases are supposed to happen.
> 
> I don't think that snarky comments are constructive. I'd rather kindly
> ask to make checking https://wiki.gnome.org/ThreePointThirtyseven part
> of the process when scheduling future infrastructure maintenance tasks.

I think it's clear it was supposed to be humorous

Does seem we need to better sync calendars though (or de-sync...) and the
release schedule wiki pages are something Bart & Co can easily check

Good luck sysadmining, sure it'll all be fine


signature.asc
Description: This is a digitally signed message part
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: GitLab Container Registry scheduled maintenance, Friday May 29, 10 UTC

[Andre Klapper]

On Thu, 2020-05-28 at 19:16 +0200, Milan Crha via desktop-devel-list
wrote:
> On Thu, 2020-05-28 at 17:13 +0200, Bartłomiej Piotrowski wrote:
> > We're sorry for this inconvenience and at your disposal in case of
> > your questions.
>
> this is the second time you plan something bigger, some server
> maintenance, at the days when the releases are supposed to happen.

I don't think that snarky comments are constructive. I'd rather kindly
ask to make checking https://wiki.gnome.org/ThreePointThirtyseven part
of the process when scheduling future infrastructure maintenance tasks.

Cheers,
andre
--
Andre Klapper  |  ak...@gmx.net
https://blogs.gnome.org/aklapper/


___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: GitLab Container Registry scheduled maintenance, Friday May 29, 10 UTC

[Bartłomiej Piotrowski]

On 28/05/2020 19.16, Milan Crha via desktop-devel-list wrote:
> On Thu, 2020-05-28 at 17:13 +0200, Bartłomiej Piotrowski wrote:
>> We're sorry for this inconvenience and at your disposal in case of
>> your questions.
> 
>   Hi,
> this is the second time you plan something bigger, some server
> maintenance, at the days when the releases are supposed to happen.
> 
> Is there a pattern? To make life more interesting to the maintainers?
> 
> I've no idea what this particular work can influence and how it'll work
> with the release process, but still...
>   Thanks and bye,
>   Milan
> 
> 
> 
> ___
> desktop-devel-list mailing list
> desktop-devel-list@gnome.org
> https://mail.gnome.org/mailman/listinfo/desktop-devel-list
> 

I assure you we are not doing it on purpose (clearly whoever sets the
schedule is!); it is very unlikely you need Docker registry to work to
release new version.

Bart
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: GitLab Container Registry scheduled maintenance, Friday May 29, 10 UTC

[Milan Crha via desktop-devel-list]

On Thu, 2020-05-28 at 17:13 +0200, Bartłomiej Piotrowski wrote:
> We're sorry for this inconvenience and at your disposal in case of
> your questions.

Hi,
this is the second time you plan something bigger, some server
maintenance, at the days when the releases are supposed to happen.

Is there a pattern? To make life more interesting to the maintainers?

I've no idea what this particular work can influence and how it'll work
with the release process, but still...
Thanks and bye,
Milan



___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

GitLab Container Registry scheduled maintenance, Friday May 29, 10 UTC

[Bartłomiej Piotrowski]

Hello,

Starting tomorrow at 10 UTC, container registry service provided by
GitLab will be set to read-only mode. As it has accumulated over 5TB of
images, we plan to perform pruning job to remove unused tags and
unreachable image layers. Given it's the first time it will be run since
the switch to GitLab, I can't reliably estimate how long it may take.

We're sorry for this inconvenience and at your disposal in case of your
questions.

Bartłomiej
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list