Re: Changes to GitLab runners configuration
On Thu, Jun 18, 2020 at 5:58 pm, Michael Catanzaro wrote: It seems all privileged runners have disappeared. All my glib-net CI pipelines are stalled. :( Looks like this was some sort of temporary infrastructure problem last night, because it's fixed again now. ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
It seems all privileged runners have disappeared. All my glib-net CI pipelines are stalled. :( ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
On Sun, Jun 14, 2020 at 1:42 pm, philip.chime...@gmail.com wrote: Is GJS really the only project that tests merge requests using ASan and LSan? If not, please speak up so we can have a better idea of how many projects are affected. No, glib-networking also requires asan for every merge request. We should set a goal to have asan CI for all core apps and libraries. ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
On Thu, Mar 26, 2020 at 5:26 AM Jordan Petridis via desktop-devel-list < desktop-devel-list@gnome.org> wrote: > On Sunday, March 22, 2020 12:56 AM, Michael Catanzaro < > mcatanz...@gnome.org> wrote: > > > On Sat, Mar 21, 2020 at 1:21 pm, Christian Hergert > > christ...@hergert.me wrote: > > > > > > Those words sound incompatible to me in the same way that if you have > > > access to Linux's perf, you can sniff pretty much any data you want on > > > the system. > > > > > We're talking about CI runners... we only need privileged access inside > > the container running our CI, not outside it. Yes? > > It doesn't take much effort to get access outside a privilledged contianer > sadly. But maybe we can have a shared 'privilledged' runner that's setup in > a VM and gets wiped daily or such for the jobs outside the GNOME group that > need it, such as forked repos. > Hi, Has anyone thought any more about allowing CAP_SYS_PTRACE for unprivileged runners or somehow setting up a disposable privileged runner? This has continued to be a large inconvenience for me during the last three months. Almost all contributions to GJS are from people without a GNOME developer account, and so happen on forks, or even from people with a developer account who prefer to use a fork (like myself until this change happened.) This means as the maintainer I cannot use the GitLab website to merge these contributions. I have to check out the branch on my machine, compile it myself with sanitizers enabled since GitLab won't do it, and then merge it myself and push it back up to the repo. Many times I also have to field confused questions from these contributors. It seems like not a lot of work, but it's getting old very fast. Is GJS really the only project that tests merge requests using ASan and LSan? If not, please speak up so we can have a better idea of how many projects are affected. (And if so, you should really consider it, it's a very low-friction way to detect memory leaks, trivial to set up with Meson, faster than Valgrind, and fewer false positives than Valgrind! On the downside, it sometimes pinpoints the source of the memory leak less accurately than Valgrind.) Cheers, -- Philip C ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
On Sunday, March 22, 2020 12:56 AM, Michael Catanzaro wrote: > On Sat, Mar 21, 2020 at 1:21 pm, Christian Hergert > christ...@hergert.me wrote: > > > Those words sound incompatible to me in the same way that if you have > > access to Linux's perf, you can sniff pretty much any data you want on > > the system. > > We're talking about CI runners... we only need privileged access inside > the container running our CI, not outside it. Yes? It doesn't take much effort to get access outside a privilledged contianer sadly. But maybe we can have a shared 'privilledged' runner that's setup in a VM and gets wiped daily or such for the jobs outside the GNOME group that need it, such as forked repos. Jordan publickey - jordan@alatiera.com - 0x0BDAD30B.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
On Sat, Mar 21, 2020 at 1:21 pm, Christian Hergert wrote: Those words sound incompatible to me in the same way that if you have access to Linux's perf, you can sniff pretty much any data you want on the system. We're talking about CI runners... we only need privileged access inside the container running our CI, not outside it. Yes? ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
On 3/21/20 12:09 PM, Philip Chimento via desktop-devel-list wrote: > > I'd really appreciate if we could find a way to have the unprivileged > runners have CAP_SYS_PTRACE added to them. Those words sound incompatible to me in the same way that if you have access to Linux's perf, you can sniff pretty much any data you want on the system. -- Christian ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
On Sat, Mar 21, 2020 at 6:47 AM Michael Catanzaro wrote: > On Fri, Mar 20, 2020 at 8:20 pm, philip.chime...@gmail.com wrote: > > Has anyone managed to get lsan/asan to work without CAP_SYS_PTRACE > > yet or otherwise have any suggestions on what would need to be done > > to support it in an unprivileged setup? > > I marked my CI as privileged: > That I've also done, but it's not a good solution since the job still won't work on contributors' forks. I guess it's a slight improvement that contributors will now have their merge requests timeout instead of fail outright: https://gitlab.gnome.org/ptomato/gjs/-/commit/5621258ec5f81a3c389b49148faafc7c9f5c454f/pipelines?ref=tag-lsan-jobs But it's still confusing to contributors and it still means that I have to merge everything manually. I'd really appreciate if we could find a way to have the unprivileged runners have CAP_SYS_PTRACE added to them. Cheers, -- Philip C ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
On Fri, Mar 20, 2020 at 8:20 pm, philip.chime...@gmail.com wrote: Has anyone managed to get lsan/asan to work without CAP_SYS_PTRACE yet or otherwise have any suggestions on what would need to be done to support it in an unprivileged setup? I marked my CI as privileged: https://gitlab.gnome.org/GNOME/glib-networking/-/commit/0a7b0d3b112e3d14f238c0179166a74b48e44dfc ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
On Tue, Mar 3, 2020 at 6:29 AM Michael Catanzaro wrote: > On Mon, Mar 2, 2020 at 9:41 pm, Philip Chimento via desktop-devel-list > wrote: > > Also, has anyone successfully gotten a CI job that uses lsan or asan > > to work in the unprivileged setup? (See my previous question about > > CAP_SYS_PTRACE.) > > Hm, looks like my glib-networking CI is broken due to this. I hadn't > noticed yet > Has anyone managed to get lsan/asan to work without CAP_SYS_PTRACE yet or otherwise have any suggestions on what would need to be done to support it in an unprivileged setup? Thanks, -- Philip C ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
Hi again, > this seems to be because of > https://gitlab.com/gitlab-org/gitlab/issues/198518 Thanks Ondrej for finding the root cause of the change! I just pushed [1] a change to the template which should workaround the issue for now. Also left a comment on the upstream issue [2] If your pipeline was working before this and now is broken due to this, please ping me. Otherwise, your pipelines should be starting correctly now for those that were affected by the gitlab upgrade. [1] https://gitlab.gnome.org/GNOME/citemplates/-/commit/5bffac6dfbd797291a90e705925a7a3973e4cc57 [2] https://gitlab.com/gitlab-org/gitlab/-/issues/209102#note_302271286 Cheers, Jordan ‐‐‐ Original Message ‐‐‐ On Tuesday, March 10, 2020 10:32 AM, Ondrej Holy wrote: > Hi all, > > this seems to be because of > https://gitlab.com/gitlab-org/gitlab/issues/198518 and it is enough to > use some custom stages instead of .pre/.post to fix this, which I did > recently for Nautilus: > https://gitlab.gnome.org/GNOME/nautilus/-/commit/e7c2a0182a4ba2d6eb05fe170cc9b4d018a70feb. > > Regards > > Ondrej > > po 9. 3. 2020 v 21:34 odesílatel Felix Riemann friem...@gnome.org napsal: > > > Am Sonntag, den 08.03.2020, 18:34 + schrieb Jordan Petridis: > > > > > Hi Felix, > > > > Hi! > > > > > I've noticed that a while ago as well, but I was certain it wasn't > > > caused by the recent changes cause we had extensively > > > test them. > > > > Yes, I found the templates in gitlab after writing the mail and the > > changes do not look intrusive enough to make pipelines stop working > > entirely. > > > > > Today I found some time to track down the issue and looks like its a > > > regression on gitlab itself, probably on the latest update. > > > The issue is that the template now uses the default .pre and .post > > > stages to avoid over witting/forcing projects on a specific stages > > > schema. This seems to have been working fine, till roughly 2 weeks > > > ago. [1] > > > Some projects that define more jobs and naturally their custom > > > stages, such as epiphany, had pipelines working the whole time. > > > Briefly looking at gitlab, I've found this [2] bug report, but > > > there's probably an older one I suspect. > > > > Indeed, the bug description seems to fit pretty well. I also got the > > mentioned "No stages/jobs for this pipeline." error when I tried to > > start a pipeline manually. > > > > > I think we can try to workaround this in our template, and I will try > > > rolling something out tomorrow. Will most likely move back the > > > current jobs back to [build, deploy] stages which hopefully every > > > project has already. This does have the potential to break some CI > > > configurations and if so please do ping me on irc (or rocket chat) or > > > reply to this email. > > > I am sorry for the unfortunate timing and for not looking at it > > > sooner. Hopefully this will be resolved soon™️ > > > > No problem. I hadn't noticed earlier myself until I noticed the Flatpak > > repo not updating after the 3.36 release. Thanks for having a look into > > it. > > Regards, > > Felix > > > > desktop-devel-list mailing list > > desktop-devel-list@gnome.org > > https://mail.gnome.org/mailman/listinfo/desktop-devel-list publickey - jordan@alatiera.com - 0x0BDAD30B.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
Hi all, this seems to be because of https://gitlab.com/gitlab-org/gitlab/issues/198518 and it is enough to use some custom stages instead of .pre/.post to fix this, which I did recently for Nautilus: https://gitlab.gnome.org/GNOME/nautilus/-/commit/e7c2a0182a4ba2d6eb05fe170cc9b4d018a70feb. Regards Ondrej po 9. 3. 2020 v 21:34 odesílatel Felix Riemann napsal: > > Am Sonntag, den 08.03.2020, 18:34 + schrieb Jordan Petridis: > > Hi Felix, > > Hi! > > > I've noticed that a while ago as well, but I was certain it wasn't > > caused by the recent changes cause we had extensively > > test them. > > Yes, I found the templates in gitlab after writing the mail and the > changes do not look intrusive enough to make pipelines stop working > entirely. > > > Today I found some time to track down the issue and looks like its a > > regression on gitlab itself, probably on the latest update. > > > > The issue is that the template now uses the default .pre and .post > > stages to avoid over witting/forcing projects on a specific stages > > schema. This seems to have been working fine, till roughly 2 weeks > > ago. [1] > > > > Some projects that define more jobs and naturally their custom > > stages, such as epiphany, had pipelines working the whole time. > > > > Briefly looking at gitlab, I've found this [2] bug report, but > > there's probably an older one I suspect. > > Indeed, the bug description seems to fit pretty well. I also got the > mentioned "No stages/jobs for this pipeline." error when I tried to > start a pipeline manually. > > > I think we can try to workaround this in our template, and I will try > > rolling something out tomorrow. Will most likely move back the > > current jobs back to [build, deploy] stages which hopefully every > > project has already. This does have the potential to break some CI > > configurations and if so please do ping me on irc (or rocket chat) or > > reply to this email. > > > > I am sorry for the unfortunate timing and for not looking at it > > sooner. Hopefully this will be resolved soon™️ > > No problem. I hadn't noticed earlier myself until I noticed the Flatpak > repo not updating after the 3.36 release. Thanks for having a look into > it. > > Regards, > > Felix > > > ___ > desktop-devel-list mailing list > desktop-devel-list@gnome.org > https://mail.gnome.org/mailman/listinfo/desktop-devel-list ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
Am Sonntag, den 08.03.2020, 18:34 + schrieb Jordan Petridis: > Hi Felix, Hi! > I've noticed that a while ago as well, but I was certain it wasn't > caused by the recent changes cause we had extensively > test them. Yes, I found the templates in gitlab after writing the mail and the changes do not look intrusive enough to make pipelines stop working entirely. > Today I found some time to track down the issue and looks like its a > regression on gitlab itself, probably on the latest update. > > The issue is that the template now uses the default .pre and .post > stages to avoid over witting/forcing projects on a specific stages > schema. This seems to have been working fine, till roughly 2 weeks > ago. [1] > > Some projects that define more jobs and naturally their custom > stages, such as epiphany, had pipelines working the whole time. > > Briefly looking at gitlab, I've found this [2] bug report, but > there's probably an older one I suspect. Indeed, the bug description seems to fit pretty well. I also got the mentioned "No stages/jobs for this pipeline." error when I tried to start a pipeline manually. > I think we can try to workaround this in our template, and I will try > rolling something out tomorrow. Will most likely move back the > current jobs back to [build, deploy] stages which hopefully every > project has already. This does have the potential to break some CI > configurations and if so please do ping me on irc (or rocket chat) or > reply to this email. > > I am sorry for the unfortunate timing and for not looking at it > sooner. Hopefully this will be resolved soon™️ No problem. I hadn't noticed earlier myself until I noticed the Flatpak repo not updating after the 3.36 release. Thanks for having a look into it. Regards, Felix ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
Hi Felix, I've noticed that a while ago as well, but I was certain it wasn't caused by the recent changes cause we had extensively test them. Today I found some time to track down the issue and looks like its a regression on gitlab itself, probably on the latest update. The issue is that the template now uses the default .pre and .post stages to avoid over witting/forcing projects on a specific stages schema. This seems to have been working fine, till roughly 2 weeks ago. [1] Some projects that define more jobs and naturally their custom stages, such as epiphany, had pipelines working the whole time. Briefly looking at gitlab, I've found this [2] bug report, but there's probably an older one I suspect. I think we can try to workaround this in our template, and I will try rolling something out tomorrow. Will most likely move back the current jobs back to [build, deploy] stages which hopefully every project has already. This does have the potential to break some CI configurations and if so please do ping me on irc (or rocket chat) or reply to this email. I am sorry for the unfortunate timing and for not looking at it sooner. Hopefully this will be resolved soon™️ Cheers, Jordan [1] https://gitlab.gnome.org/GNOME/gnome-builder/pipelines/154333 [2] https://gitlab.com/gitlab-org/gitlab/-/issues/209102 ‐‐‐ Original Message ‐‐‐ On Sunday, March 8, 2020 4:30 PM, Felix Riemann wrote: > Am Mittwoch, den 19.02.2020, 14:50 +0100 schrieb Bartłomiej Piotrowski: > > > Hello, > > Hi! > > > For past few days I've been working to ensure that Flatpak builds are > > still functional without additional privileges. If your project is > > using > > citemplates[1], the configuration change should be invisible to your > > pipelines and you can keep on doing awesome GNOME work. > > Is is there something necessary to change on the project level? The > reason I am asking is that I noticed my project (eog) no longer > triggering CI pipelines at all since roughly the time this change was > implemented. > > Updating the configuration to the latest template didn't change > anything. Trying to trigger the pipeline manually from Gitlab doesn't > work either. > > I also looked around and found the gedit and gnome-builder repositories > showing the same behaviour. > > Regards, > > Felix > > desktop-devel-list mailing list > desktop-devel-list@gnome.org > https://mail.gnome.org/mailman/listinfo/desktop-devel-list publickey - jordan@alatiera.com - 0x0BDAD30B.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
Am Mittwoch, den 19.02.2020, 14:50 +0100 schrieb Bartłomiej Piotrowski: > Hello, Hi! > For past few days I've been working to ensure that Flatpak builds are > still functional without additional privileges. If your project is > using > citemplates[1], the configuration change should be invisible to your > pipelines and you can keep on doing awesome GNOME work. Is is there something necessary to change on the project level? The reason I am asking is that I noticed my project (eog) no longer triggering CI pipelines at all since roughly the time this change was implemented. Updating the configuration to the latest template didn't change anything. Trying to trigger the pipeline manually from Gitlab doesn't work either. I also looked around and found the gedit and gnome-builder repositories showing the same behaviour. Regards, Felix ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
On Tue, Mar 3, 2020 at 1:36 PM Bartłomiej Piotrowski wrote: > I've poked around yesterday and it's apparently not as trivial to run > buildah unprivileged in a container as it was the last time I tried. I > don't see better way than tagging jobs privileged at the moment. Thanks, tagging the job as privileged has fixed the errors that I was seeing: https://gitlab.gnome.org/GNOME/tracker-oci-images/-/merge_requests/15 Sam ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
On Mon, Mar 2, 2020 at 9:41 pm, Philip Chimento via desktop-devel-list wrote: Also, has anyone successfully gotten a CI job that uses lsan or asan to work in the unprivileged setup? (See my previous question about CAP_SYS_PTRACE.) Hm, looks like my glib-networking CI is broken due to this. I hadn't noticed yet ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
On 28/02/2020 15.17, Michael Catanzaro wrote: > Please revert the runner changes until you have time to fix this. Our CI > has been basically unusable all week and that blocks flatpak pushes. All issues with user namespaces on gcc* runners should be resolved by now. It slipped my mind to set it persistently in sysctl.conf and runners were rebooted right after I left for vacation. On 03/03/2020 06.41, Philip Chimento via desktop-devel-list wrote: > I have little knowledge of this problem space so I don't even know whereto > start to debug this. Is this the same privileges problem as "bwrap: Creating > new namespace failed" described earlier in the thread, or is it something > different? (...) > Also, has anyone successfully gotten a CI job that uses lsan or asan to > work in the unprivileged setup? (See my previous question about > CAP_SYS_PTRACE.) I've poked around yesterday and it's apparently not as trivial to run buildah unprivileged in a container as it was the last time I tried. I don't see better way than tagging jobs privileged at the moment. I can apply a custom seccomp profile if that helps, but someone has to write it. Bart ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
On Tue, Mar 3, 2020 at 6:42 AM Philip Chimento via desktop-devel-list wrote: > Does anyone have any advice on what to do with this? I followed Sam's helpful > hint about Tracker's CI images working to [1], copied the things that looked > like they might be relevant from there into my own gitlab-ci file [2], at > first got some errors that I knew basically how to deal with, but now I am > stuck at a bunch of error messages that look to me like gibberish but end > with "operation not permitted" [3]. I have little knowledge of this problem > space so I don't even know where to start to debug this. Is this the same > privileges problem as "bwrap: Creating new namespace failed" described > earlier in the thread, or is it something different? ... > [3] https://gitlab.gnome.org/ptomato/gjs/-/jobs/616717 It looks as though this problem is also affecting the tracker-oci-images build: Error during unshare(CLONE_NEWUSER): Operation not permitted From: https://gitlab.gnome.org/GNOME/tracker-oci-images/-/jobs/616919 And librsvg-oci-images build (which I copied for the Tracker repo :) Error during unshare(CLONE_NEWUSER): Invalid argument User namespaces are not enabled in /proc/sys/user/max_user_namespaces. From: https://gitlab.gnome.org/GNOME/librsvg-oci-images/-/jobs/614821 Sam ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
On Fri, Feb 28, 2020 at 6:18 AM Michael Catanzaro wrote: > On Mon, Feb 24, 2020 at 6:46 pm, Jordan Petridis via desktop-devel-list > wrote: > > No there isn't, it was working properly when it was first rolled out. > > I've started seen this issue today and looks like it only affecting > > some runners, so I am guessing something got updated or new runners > > where added. Bart is on vacation till Monday so let's poke av I guess. > > Hi sysadmins, > > Please revert the runner changes until you have time to fix this. Our > CI has been basically unusable all week and that blocks flatpak pushes. > Does anyone have any advice on what to do with this? I followed Sam's helpful hint about Tracker's CI images working to [1], copied the things that looked like they might be relevant from there into my own gitlab-ci file [2], at first got some errors that I knew basically how to deal with, but now I am stuck at a bunch of error messages that look to me like gibberish but end with "operation not permitted" [3]. I have little knowledge of this problem space so I don't even know where to start to debug this. Is this the same privileges problem as "bwrap: Creating new namespace failed" described earlier in the thread, or is it something different? Also, has anyone successfully gotten a CI job that uses lsan or asan to work in the unprivileged setup? (See my previous question about CAP_SYS_PTRACE.) [1] https://gitlab.gnome.org/GNOME/tracker-oci-images/-/blob/master/.gitlab-ci.yml [2] https://gitlab.gnome.org/ptomato/gjs/-/blob/test-ci/.gitlab-ci.yml#L275-345 [3] https://gitlab.gnome.org/ptomato/gjs/-/jobs/616717 Thanks, -- Philip C. ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
On Mon, Feb 24, 2020 at 6:46 pm, Jordan Petridis via desktop-devel-list wrote: No there isn't, it was working properly when it was first rolled out. I've started seen this issue today and looks like it only affecting some runners, so I am guessing something got updated or new runners where added. Bart is on vacation till Monday so let's poke av I guess. Hi sysadmins, Please revert the runner changes until you have time to fix this. Our CI has been basically unusable all week and that blocks flatpak pushes. ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
Hi,
> Is there anything else that needs to be done?
No there isn't, it was working properly when it was first rolled out. I've
started seen this issue today and looks like it only affecting some runners, so
I am guessing something got updated or new runners where added. Bart is on
vacation till Monday so let's poke av I guess.
Cheers,
Jordan
‐‐‐ Original Message ‐‐‐
On Monday, February 24, 2020 11:11 AM, Bastien Nocera wrote:
> On Wed, 2020-02-19 at 14:50 +0100, Bartłomiej Piotrowski wrote:
>
> > Hello,
> > For historical reasons™ all GitLab runners were running with
> > privileged
> > mode enabled. The happy side effect of this fact is that nothing
> > special
> > was ever needed to run Docker or flatpak builds. It also means we
> > were
> > extremely lucky that no one abused CAP_SYS_ADMIN and other elevated
> > privileges for bad things.
> > For past few days I've been working to ensure that Flatpak builds are
> > still functional without additional privileges. If your project is
> > using
> > citemplates[1], the configuration change should be invisible to your
> > pipelines and you can keep on doing awesome GNOME work. However, if
> > you
> > have modified default steps via 'extends' keyword (or by defining
> > them
> > completely manually), please make sure that:
>
> It seems like this isn't quite working as it should. This MR is porting
> sound-juicer to meson:
> https://gitlab.gnome.org/GNOME/sound-juicer/-/merge_requests/6
>
> It uses the flatpak_ci_initiative.yml template and throws this error:
> bwrap: Creating new namespace failed, likely because the kernel does
> not support user namespaces. bwrap must be installed setuid on such
> systems.
>
> > 1. you are using the
> > registry.gitlab.gnome.org/gnome/gnome-runtime-images/gnome image or
> > your
> > image does not run as root,
> >
>
> From the template:
> .flatpak:
> image: 'registry.gitlab.gnome.org/gnome/gnome-runtime-images/gnome:master'
>
> > 2. jobs using flatpak/flatpak-builder have "flatpak" tag defined,
>
> From the template:
> tags:
> - flatpak
> And in the pipeline output:
> https://gitlab.gnome.org/GNOME/sound-juicer/-/jobs/606529
>
> > 3. flatpak-builder invocation includes --user -disable-rofiles-fuse
> > for
> > building; 'flatpak-builder --run' includes --disable-rofiles-fuse.
> >
>
> In the template:
> script:
> - flatpak-builder --user --disable-rofiles-fuse --stop-at=${FLATPAK_MODULE}
> flatpak_app ${MANIFEST_PATH}
> (also visible in the pipeline output).
>
> Is there anything else that needs to be done?
>
> > If your project's pipeline is using Docker to build an image from
> > Dockerfile, consider switching to podman or buildah as they should
> > work
> > unprivileged.
> > The only exception from these changes are runners assigned to
> > gnome-build-meta.
> > If you encounter any problems with running CI unprivileged, please
> > poke
> > me on #sysadmin on irc.gnome.org or via Rocket.chat.
>
> desktop-devel-list mailing list
> desktop-devel-list@gnome.org
> https://mail.gnome.org/mailman/listinfo/desktop-devel-list
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
On Mon, Feb 24, 2020 at 11:08 am, Michael Catanzaro wrote: I have two failed builds on flatpak-gcc176 (also broken) and two failed builds on flatpak-gcc150.fsffrance.org (also broken). My plan is to keep retrying until I get flatpak-progress again. I need a successful build urgently because yesterday's Tech Preview build is super crashy. I succeeded on my seventh retry, when I got flatpak-progress. afaik this is currently the only working runner. Michael ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
On Mon, Feb 24, 2020 at 4:48 pm, Bastien Nocera wrote: Ran it 3 separate times, on 3 different runners, to no avail. I'm struggling with this currently. From yesterday: https://gitlab.gnome.org/GNOME/epiphany/pipelines/10/builds I have a failed build on flatpak-gcc175 (broken), then a successful build on flatpak-progress (good). From today: https://gitlab.gnome.org/GNOME/epiphany/pipelines/155799/builds I have two failed builds on flatpak-gcc176 (also broken) and two failed builds on flatpak-gcc150.fsffrance.org (also broken). My plan is to keep retrying until I get flatpak-progress again. I need a successful build urgently because yesterday's Tech Preview build is super crashy. Michael ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
On Mon, 2020-02-24 at 08:19 -0600, Michael Catanzaro wrote: > On Mon, Feb 24, 2020 at 10:11 am, Bastien Nocera > wrote: > > It uses the flatpak_ci_initiative.yml template and throws this > > error: > > bwrap: Creating new namespace failed, likely because the kernel > > does > > not support user namespaces. bwrap must be installed setuid on > > such > > systems. > > I'm seeing this randomly on Epiphany CI. Solution in my case is to > rerun the CI. Maybe one particular runner is broken? Ran it 3 separate times, on 3 different runners, to no avail. ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
On Mon, Feb 24, 2020 at 10:11 am, Bastien Nocera wrote: It uses the flatpak_ci_initiative.yml template and throws this error: bwrap: Creating new namespace failed, likely because the kernel does not support user namespaces. bwrap must be installed setuid on such systems. I'm seeing this randomly on Epiphany CI. Solution in my case is to rerun the CI. Maybe one particular runner is broken? ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
On Wed, 2020-02-19 at 14:50 +0100, Bartłomiej Piotrowski wrote:
> Hello,
>
> For historical reasons™ all GitLab runners were running with
> privileged
> mode enabled. The happy side effect of this fact is that nothing
> special
> was ever needed to run Docker or flatpak builds. It also means we
> were
> extremely lucky that no one abused CAP_SYS_ADMIN and other elevated
> privileges for bad things.
>
> For past few days I've been working to ensure that Flatpak builds are
> still functional without additional privileges. If your project is
> using
> citemplates[1], the configuration change should be invisible to your
> pipelines and you can keep on doing awesome GNOME work. However, if
> you
> have modified default steps via 'extends' keyword (or by defining
> them
> completely manually), please make sure that:
It seems like this isn't quite working as it should. This MR is porting
sound-juicer to meson:
https://gitlab.gnome.org/GNOME/sound-juicer/-/merge_requests/6
It uses the flatpak_ci_initiative.yml template and throws this error:
bwrap: Creating new namespace failed, likely because the kernel does
not support user namespaces. bwrap must be installed setuid on such
systems.
> 1) you are using the
> registry.gitlab.gnome.org/gnome/gnome-runtime-images/gnome image or
> your
> image does not run as root,
From the template:
.flatpak:
image: 'registry.gitlab.gnome.org/gnome/gnome-runtime-images/gnome:master'
> 2) jobs using flatpak/flatpak-builder have "flatpak" tag defined,
From the template:
tags:
- flatpak
And in the pipeline output:
https://gitlab.gnome.org/GNOME/sound-juicer/-/jobs/606529
> 3) flatpak-builder invocation includes --user -disable-rofiles-fuse
> for
> building; 'flatpak-builder --run' includes --disable-rofiles-fuse.
In the template:
script:
- flatpak-builder --user --disable-rofiles-fuse --stop-at=${FLATPAK_MODULE}
flatpak_app ${MANIFEST_PATH}
(also visible in the pipeline output).
Is there anything else that needs to be done?
> If your project's pipeline is using Docker to build an image from
> Dockerfile, consider switching to podman or buildah as they should
> work
> unprivileged.
>
> The only exception from these changes are runners assigned to
> gnome-build-meta.
>
> If you encounter any problems with running CI unprivileged, please
> poke
> me on #sysadmin on irc.gnome.org or via Rocket.chat.
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
19.02.2020 23:18, Bartłomiej Piotrowski пишет: However, I did not test it after described changes. I recall it used to be possible to run buildah in an unprivileged container, but just in case, I have also configured additional privileged runner available only to projects in GNOME group. Add "privileged" to tags to use it. Could you please make it available for Infrastructure group? I need it for extensions-web CI [1]. [1] https://gitlab.gnome.org/Infrastructure/extensions-web/-/blob/master/.gitlab-ci.yml Bart ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list -- Best regards, Yuri Konotopov ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
On Wed, Feb 19, 2020 at 11:18 AM Bartłomiej Piotrowski < bpiotrow...@gnome.org> wrote: > On 19/02/2020 19.52, Michael Catanzaro wrote: > > Have you tested this? I've tried many times and afaik GitLab is simply > > incompatible with podman images. I don't remember the exact error > > message offhand, but GitLab fails to detect podman containers as valid > > containers, even when using 'podman build --format=docker'. > > I'm pretty sure --format=docker is enough to maintain compatibility with > older Docker running on some runners. > > However, I did not test it after described changes. I recall it used to > be possible to run buildah in an unprivileged container, but just in > case, I have also configured additional privileged runner available only > to projects in GNOME group. Add "privileged" to tags to use it. > Note that Leak Sanitizer (lsan) does not work in unprivileged docker containers, as it needs to use ptrace (docker --cap-add SYS_PTRACE). (Hopefully this saves someone else some googling.) Is it possible to run the CI with this particular privilege added? Regards, -- Philip ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
On Wed, 2020-02-19 at 13:14 -0600, Michael Catanzaro wrote: > On Wed, Feb 19, 2020 at 8:00 pm, Sam Thursfield > wrote: > > We've been using podman successfully to build the Tracker CI > > images. > > The exact build instructions are here: > > > > > > https://gitlab.gnome.org/GNOME/tracker-oci-images/blob/master/.gitlab-ci.yml > > > > These containers are working fine in GitLab CI, see the > > tracker.git > > and > > tracker-miners.git pipelines > > I'll look at that. Thanks! Bastien fixed podman support for GLib recently, too. Take a look at https://gitlab.gnome.org/GNOME/glib/commit/36aacb8ae36db6352a36ffe1bbaf15c79fe41532 . Philip ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
On 19/02/2020 19.52, Michael Catanzaro wrote: > Have you tested this? I've tried many times and afaik GitLab is simply > incompatible with podman images. I don't remember the exact error > message offhand, but GitLab fails to detect podman containers as valid > containers, even when using 'podman build --format=docker'. I'm pretty sure --format=docker is enough to maintain compatibility with older Docker running on some runners. However, I did not test it after described changes. I recall it used to be possible to run buildah in an unprivileged container, but just in case, I have also configured additional privileged runner available only to projects in GNOME group. Add "privileged" to tags to use it. Bart ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
On Wed, Feb 19, 2020 at 8:00 pm, Sam Thursfield wrote: We've been using podman successfully to build the Tracker CI images. The exact build instructions are here: https://gitlab.gnome.org/GNOME/tracker-oci-images/blob/master/.gitlab-ci.yml These containers are working fine in GitLab CI, see the tracker.git and tracker-miners.git pipelines I'll look at that. Thanks! ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
On Wed, Feb 19, 2020 at 7:53 PM Michael Catanzaro wrote: > > Have you tested this? I've tried many times and afaik GitLab is simply > incompatible with podman images. I don't remember the exact error > message offhand, but GitLab fails to detect podman containers as valid > containers, even when using 'podman build --format=docker'. We've been using podman successfully to build the Tracker CI images. The exact build instructions are here: https://gitlab.gnome.org/GNOME/tracker-oci-images/blob/master/.gitlab-ci.yml These containers are working fine in GitLab CI, see the tracker.git and tracker-miners.git pipelines Note that we're using Fedora 30 there, I haven't tested with Fedora 31. Sam ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: Changes to GitLab runners configuration
On Wed, Feb 19, 2020 at 2:50 pm, Bartłomiej Piotrowski wrote: If your project's pipeline is using Docker to build an image from Dockerfile, consider switching to podman or buildah as they should work unprivileged. Have you tested this? I've tried many times and afaik GitLab is simply incompatible with podman images. I don't remember the exact error message offhand, but GitLab fails to detect podman containers as valid containers, even when using 'podman build --format=docker'. I actually use a F30 VM to create my Docker images because Docker is broken in F31 due to unified cgroup hierarchy. Would be wonderful if it was possible to use podman. ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
