CI moved to Qt 6.7 for Linux builds

[Ben Cooksley]

Hi all,

I have just flipped the switch that has moved the CI system over to using
Qt 6.7 for Linux builds on our SUSE images.

Should you see any issues with builds failing as a result of packages being
missing in the registry then please submit a merge request to
sysadmin/ci-management to ensure that build dependency is added to our seed
jobs.

I'll leave the Qt 6.6 package registry and container images in place for
another week or so then will schedule them for removal.

As part of this I have also updated the list of projects with Qt 6 only
master branches. Any residual Qt 5 build artifacts the CI system was
holding for those projects have now been purged, which may impact
downstream projects that depend on those projects that are still on Qt 5.

On an adjacent note, i'd also like to schedule removing CI support for
release/23.08 and Plasma/5.27 builds (by purging all their binaries we
currently hold) for the Qt 5.15 series.

While checking however I note that several projects still have activity on
those branches. Can we please confirm whether any further releases are
expected, as i'd prefer to remove anything that isn't being properly
maintained anymore.

Thanks,
Ben

Please cleanup old forks

[Ben Cooksley]

Hi all,

As part of routine maintenance on Invent (Gitlab) i've been reviewing the
total amount of space currently being consumed by repositories.

At this time, the top 100 largest repositories on Invent (we have 14,000 or
so for context) are consuming around 21% of the total space. It would
therefore be good if we could please clean this up.

To add a bit of context, we currently have around 140GB of repositories,
and a further 30GB of pooled repository data (shared among the canonical
repository and it's forks). Those top 100 repositories currently use 30GB
of space - so as much as all of the pools.

Forks of KDevelop and docs-krita-org feature heavily on the list of large
repositories.

Of particular interest are old forks (created more than 2-3 years ago) as
these are often not participating in Gitlab's mechanism to reduce the disk
space taken by forks (known as pool repositories).

As an additional note: please do not fork a fork, as these forks-of-forks
cannot participate in a pool and must carry a full copy of the upstream
repository data. If you have a situation where a network of forks of forks
has developed please consider options to resolve this.

I know Krita at least has this situation presently with their fork of
various parts of Qt that is being maintained in people's personal
namespaces.

Thanks,
Ben

Re: KDIff3 Craft CI for MacOS broken

[Ben Cooksley]

On Tue, Feb 27, 2024 at 1:12 PM Michael Reeves  wrote:

> Does anyone have insight as to why this is happeing:
>
>
> /Resources/qml/org/kde/i18n/localeData/libki18nlocaledataqmlplugin.dylib'] 
> succeeded with exit code 0
> Library dependency 
> '/Users/gitlab/ws/builds/GZwHuM5x/0/sdk/kdiff3/macos-arm-clang/build/extragear/kdiff3/archive/Applications/KDE/kdiff3.app/Contents/Frameworks/libKF6I18nLocaleData.6.dylib'
>  does not exist
> /Users/gitlab/ws/builds/GZwHuM5x/0/sdk/kdiff3/macos-arm-clang/build/extragear/kdiff3/archive/Applications/KDE/kdiff3.app/Contents/Resources/qml/org/kde/i18n/localeData/libki18nlocaledataqmlplugin.dylib:
>  Failed to add library dependency 
> '/Users/gitlab/ws/builds/GZwHuM5x/0/sdk/kdiff3/macos-arm-clang/build/extragear/kdiff3/archive/Applications/KDE/kdiff3.app/Contents/Frameworks/libKF6I18nLocaleData.6.dylib'
>  into bundle
> Action: package for extragear/kdiff3:1.11 FAILED
>
>
Hi Michael,

This is extremely bad timing to be trying to do anything in the application
packaging space, as the constant version bumping within Frameworks and
elsewhere is causing carnage and breakage just about everywhere.
See Flatpak for instance.

I'd suggest waiting a good week or so before trying again, as we will need
to get Craft updated to the 6.0 release made public yesterday.

Have a sneaking suspicion that this will be symlink related though.

Regards,
Ben

Re: Retirement of Binary Factory

[Ben Cooksley]

On Mon, Feb 19, 2024 at 1:22 PM Harald Sitter  wrote:

> On Sun, Feb 18, 2024 at 10:23 AM Ben Cooksley  wrote:
> >
> > On Sun, Feb 18, 2024 at 2:58 PM Loren Burkholder <
> computersemiexp...@outlook.com> wrote:
> >>
> >> On Saturday, February 17, 2024 8:35:48 PM EST Ben Cooksley wrote:
> >> > On Sun, Feb 4, 2024 at 10:26 AM Ben Cooksley 
> wrote:
> >> >
> >> > The Binary Factory, alongside it's two build servers, and the Flatpak
> >> > repository it provided at https://distribute.kde.org/ has now been
> >> > decommissioned.
> >>
> >> Hi all,
> >>
> >> Just last evening, I was downloading Filelight on a Windows machine.
> Due to the machine being rather ancient and slow, I ended up going for the
> direct binary download instead of the Microsoft Store download. The
> apps.kde.org page had me download from a Binary Factory link. As of right
> now, that link is still on https://apps.kde.org/filelight/, but it
> obviously doesn't work. I haven't checked the apps.kde.org source, but it
> seems that perhaps those URLs are automatically generated for each app, so
> it should be trivial to change or remove them.
> >
> >
> > It would appear that Filelight has not yet enabled themselves for any
> form of continuous delivery builds aside from Flatpak.
> > See
> https://invent.kde.org/utilities/filelight/-/blob/master/.gitlab-ci.yml?ref_type=heads
> >
> > If Filelight contributors are still interested in supporting other
> platforms those builds will need to be added.
>
> I'm curious, why didn't you enable stuff for the things that were
> previously building on binary factory?
>

Because in the past what people have done is turn on builds for projects on
the Binary Factory that the underlying projects actually didn't care about
and had little interest in.
That was part of the reason why the Binary Factory was a bit of a
maintainability nightmare - as constant fixes were required for some
projects as they broke builds.

Hence why projects were asked to step up and enable builds themselves.

Cheers,
Ben


>
> On Sun, Feb 18, 2024 at 10:23 AM Ben Cooksley  wrote:
> >
> > On Sun, Feb 18, 2024 at 2:58 PM Loren Burkholder <
> computersemiexp...@outlook.com> wrote:
> >>
> >> On Saturday, February 17, 2024 8:35:48 PM EST Ben Cooksley wrote:
> >> > On Sun, Feb 4, 2024 at 10:26 AM Ben Cooksley 
> wrote:
> >> >
> >> > The Binary Factory, alongside it's two build servers, and the Flatpak
> >> > repository it provided at https://distribute.kde.org/ has now been
> >> > decommissioned.
> >>
> >> Hi all,
> >>
> >> Just last evening, I was downloading Filelight on a Windows machine.
> Due to the machine being rather ancient and slow, I ended up going for the
> direct binary download instead of the Microsoft Store download. The
> apps.kde.org page had me download from a Binary Factory link. As of right
> now, that link is still on https://apps.kde.org/filelight/, but it
> obviously doesn't work. I haven't checked the apps.kde.org source, but it
> seems that perhaps those URLs are automatically generated for each app, so
> it should be trivial to change or remove them.
> >
> >
> > It would appear that Filelight has not yet enabled themselves for any
> form of continuous delivery builds aside from Flatpak.
> > See
> https://invent.kde.org/utilities/filelight/-/blob/master/.gitlab-ci.yml?ref_type=heads
> >
> > If Filelight contributors are still interested in supporting other
> platforms those builds will need to be added.
> >
> >>
> >>
> >> Are there other places that need this URL replaced as well?
> https://lxr.kde.org/search?%21v=kf6-qt6&_filestring=&_string=binary-factory.kde.org
> shows that there are a number of READMEs that still link to Binary Factory,
> and Krita has some CI stuff that still references binary-factory.kde.org,
> but I have a sneaking suspicion that there are binary-factory.kde.org
> links in KDE webpages and other repositories that aren't indexed by
> lxr.kde.org.
> >
> >
> > I have a checkout of most website repositories on my local system and
> did a quick grep which showed a variety of hits. Most of them were on
> README files that were intended to show build status of the website itself.
> > Those links would have been broken for some time as websites were
> converted over a while ago.
> >
> > Affected sites content wise includes:
> > - develop.kde.org
> > - digikam.org
> > - haruna.kde.org
> > - kaidan.im
> > - kate-editor.org
> > - kdeconnect.kde.org
> > - kde.ru
> > - kdevelop.org
> > - kirogi.org
> > - kmymoney.org
> > - konversation.kde.org
> > - krita.org
> > - okular.kde.org
> > - plasma-mobile.org
> > - rkward.kde.org
> > - umbrello.kde.org
> >
> >
> >>
> >>
> >> - Loren
> >
> >
> > Cheers,
> > Ben
>

Re: Retirement of Binary Factory

[Ben Cooksley]

On Sun, Feb 18, 2024 at 2:58 PM Loren Burkholder <
computersemiexp...@outlook.com> wrote:

> On Saturday, February 17, 2024 8:35:48 PM EST Ben Cooksley wrote:
> > On Sun, Feb 4, 2024 at 10:26 AM Ben Cooksley  wrote:
> >
> > The Binary Factory, alongside it's two build servers, and the Flatpak
> > repository it provided at https://distribute.kde.org/ has now been
> > decommissioned.
>
> Hi all,
>
> Just last evening, I was downloading Filelight on a Windows machine. Due
> to the machine being rather ancient and slow, I ended up going for the
> direct binary download instead of the Microsoft Store download. The
> apps.kde.org page had me download from a Binary Factory link. As of right
> now, that link is still on https://apps.kde.org/filelight/, but it
> obviously doesn't work. I haven't checked the apps.kde.org source, but it
> seems that perhaps those URLs are automatically generated for each app, so
> it should be trivial to change or remove them.
>

It would appear that Filelight has not yet enabled themselves for any form
of continuous delivery builds aside from Flatpak.
See
https://invent.kde.org/utilities/filelight/-/blob/master/.gitlab-ci.yml?ref_type=heads

If Filelight contributors are still interested in supporting other
platforms those builds will need to be added.


>
> Are there other places that need this URL replaced as well?
> https://lxr.kde.org/search?%21v=kf6-qt6&_filestring=&_string=binary-factory.kde.org
> shows that there are a number of READMEs that still link to Binary Factory,
> and Krita has some CI stuff that still references binary-factory.kde.org,
> but I have a sneaking suspicion that there are binary-factory.kde.org
> links in KDE webpages and other repositories that aren't indexed by
> lxr.kde.org.
>

I have a checkout of most website repositories on my local system and did a
quick grep which showed a variety of hits. Most of them were on README
files that were intended to show build status of the website itself.
Those links would have been broken for some time as websites were converted
over a while ago.

Affected sites content wise includes:
- develop.kde.org
- digikam.org
- haruna.kde.org
- kaidan.im
- kate-editor.org
- kdeconnect.kde.org
- kde.ru
- kdevelop.org
- kirogi.org
- kmymoney.org
- konversation.kde.org
- krita.org
- okular.kde.org
- plasma-mobile.org
- rkward.kde.org
- umbrello.kde.org



>
> - Loren


Cheers,
Ben

Re: Retirement of Binary Factory

[Ben Cooksley]

On Sun, Feb 4, 2024 at 10:26 AM Ben Cooksley  wrote:

> Hi all,
>
> For some time now we have been steadily working on getting our ability to
> build everything that was previously built on the Binary Factory being
> built on Gitlab.
>
> I'm now very happy to advise that it is now possible to perform all of
> those builds on Gitlab (and depending on how far you configure it, go even
> further than what was being done before). This transition to Gitlab builds
> should also ensure we use resources more efficiently, as builds will only
> take place when code actually changes (rather than every day - which is
> what the Binary Factory did).
>
> As such, with Gitlab now able to replace it, I am scheduling the Binary
> Factory for decommissioning in 2 weeks time.
> Projects relying on signed binary builds of their project should therefore
> look into migrating their builds immediately and without delay.
>

The Binary Factory, alongside it's two build servers, and the Flatpak
repository it provided at https://distribute.kde.org/ has now been
decommissioned.


>
> The legacy Flatpak repository at http://distribute.kde.org/ will also be
> retired as part of this. Anyone still using this should migrate to the
> per-application Flatpak repositories at https://cdn.kde.org/flatpak/
> noting that if you are adding the repository directly you may need to add
> the KF6 runtime first.
>
> Redirects will not be provided from the older Binary Factory URLs as there
> is no successor URL for many of the Binary Factory endpoints.
>
> Details on how to setup your project with binary builds can be found at
> https://invent.kde.org/sysadmin/ci-utilities/-/tree/master/gitlab-templates
>
> For enabling signed builds of your project builds please see
> https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/signing/README.md.
> Please note that signing is only permitted on mainline repository branches
> marked as protected within Gitlab (which is only done for release branches
> such as master and release/24.04). If your project is missing from the
> configuration please submit a merge request to sysadmin/ci-utilities.
>
> Signing services include publishing Android builds to our F-Droid
> repositories (https://cdn.kde.org/android/), Flatpak repositories (
> https://cdn.kde.org/flatpak/), as well as the staging publishers that
> upload draft submissions to Google Play and the Microsoft Store (for those
> applications using those) - in addition to the actual signing of binaries
> (supported for Linux Flatpak, Android, Windows and Mac).
>
> With regards to macOS, pending fixes in the upstream tooling we use
> (rcodesign) the signing service will also be able to provide notarised
> builds.
> See https://invent.kde.org/sysadmin/ci-notary-service/-/merge_requests/38
> for more information on that.
>
> At this time signing of Appimages to allow use with AppimageUpdate is not
> supported, however to my knowledge this is not used anywhere in KDE outside
> of Krita.
>
> Similarly, work on changing our GItlab configuration to protect tags will
> be needed before we can perform signed builds on tags (see
> https://docs.gitlab.com/ee/user/project/protected_tags.html).
> Based on the documentation this should not be too difficult however so
> i'll add that to the list to look into sooner rather than later (as making
> this change also has implications for the CI system in general)
>
> Thanks,
> Ben
>
>
>
Cheers,
Ben

Re: Flatpak jobs on KDE CI vs. continuous integration on main/master/devel branches

[Ben Cooksley]

On Mon, Feb 5, 2024 at 4:28 AM Friedrich W. H. Kossebau 
wrote:

> Hi,
>
> ((cc:kde-frameworks-devel for heads-up, replies please only to
> kde-core-deve))
>
> I hit the problem that when working on a repo which would like to use
> latest
> KF development state to integrate some new KF API just added in
> cooperation
> with that very repo wanting to use it, I cannot do so when someone had
> added a
> flatpak job on CI to that repo.
>
> Because with such flatpak jobs it seems they are limiting the available KF
> version not to the current latest one, as expected for continuous
> integration,
> but some older (anywhere documented?) snapshot:
>
> "runtime-version": "6.6-kf6preview",
>
> What can be done here to reestablish the old immediate continuous
> integration
> workflow? Where new APIs (also from KF) are instantly available?
>
> Right now this is a new extra burden which makes working on new features
> with
> KF and apps more complicated. Thus less interesting, and one/I would
> rather
> duplicate code in apps to get things done.
>
> Blocking latest KF API from usage also means less testing of that before
> the
> initial release.
>
> Besides all the resource costs to create flatpaks on master builds by
> default
> every time, when those are usually not used by anyone anyway.
>
> So, how to solve those problems? Did I miss something?
> Could flatpak builds on master branches be made on-demand rather?
>

For the record, my rebuild of the 6.6-kf6preview Flatpak Runtime/SDK was
successful, and the failure that kicked this off in KUserFeedback has now
been fixed.
https://invent.kde.org/frameworks/kuserfeedback/-/jobs/1561435


> Cheers
> Friedrich
>
>
>
Cheers,
Ben

Re: Flatpak jobs on KDE CI vs. continuous integration on main/master/devel branches

[Ben Cooksley]

On Mon, Feb 5, 2024 at 4:28 AM Friedrich W. H. Kossebau 
wrote:

> Hi,
>
> ((cc:kde-frameworks-devel for heads-up, replies please only to
> kde-core-deve))
>
> I hit the problem that when working on a repo which would like to use
> latest
> KF development state to integrate some new KF API just added in
> cooperation
> with that very repo wanting to use it, I cannot do so when someone had
> added a
> flatpak job on CI to that repo.
>
> Because with such flatpak jobs it seems they are limiting the available KF
> version not to the current latest one, as expected for continuous
> integration,
> but some older (anywhere documented?) snapshot:
>
> "runtime-version": "6.6-kf6preview",
>

Please see https://invent.kde.org/packaging/flatpak-kde-runtime/-/tree/kf6
for what is in the KF6 preview.


>
> What can be done here to reestablish the old immediate continuous
> integration
> workflow? Where new APIs (also from KF) are instantly available?
>

With Flatpak new APIs were never instantly available - there has always
been a delay as the Flatpak Runtime uses the most recent released version
of our software.


>
> Right now this is a new extra burden which makes working on new features
> with
> KF and apps more complicated. Thus less interesting, and one/I would
> rather
> duplicate code in apps to get things done.
>
> Blocking latest KF API from usage also means less testing of that before
> the
> initial release.


> Besides all the resource costs to create flatpaks on master builds by
> default
> every time, when those are usually not used by anyone anyway.
>

Those applications that have a hard dependency on features being added to
Frameworks are not good candidates for making use of our Continuous
Delivery systems i'm afraid.
Both Flatpak and Craft based (Linux Appimages, Android APKs, Windows and
macOS) CD jobs are best optimised for those applications that rely on the
stable Frameworks releases.

There are ways (in .craft.ini) to make newer Frameworks available, but that
requires that the system recompiles that Framework each time you trigger a
build and is therefore not recommended.

Allowing those systems to use the "latest" artifacts of Frameworks would be
a non-trivial exercise.


> So, how to solve those problems? Did I miss something?
> Could flatpak builds on master branches be made on-demand rather?


> Cheers
> Friedrich
>

Cheers,
Ben

Retirement of Binary Factory

[Ben Cooksley]

Hi all,

For some time now we have been steadily working on getting our ability to
build everything that was previously built on the Binary Factory being
built on Gitlab.

I'm now very happy to advise that it is now possible to perform all of
those builds on Gitlab (and depending on how far you configure it, go even
further than what was being done before). This transition to Gitlab builds
should also ensure we use resources more efficiently, as builds will only
take place when code actually changes (rather than every day - which is
what the Binary Factory did).

As such, with Gitlab now able to replace it, I am scheduling the Binary
Factory for decommissioning in 2 weeks time.
Projects relying on signed binary builds of their project should therefore
look into migrating their builds immediately and without delay.

The legacy Flatpak repository at http://distribute.kde.org/ will also be
retired as part of this. Anyone still using this should migrate to the
per-application Flatpak repositories at https://cdn.kde.org/flatpak/ noting
that if you are adding the repository directly you may need to add the KF6
runtime first.

Redirects will not be provided from the older Binary Factory URLs as there
is no successor URL for many of the Binary Factory endpoints.

Details on how to setup your project with binary builds can be found at
https://invent.kde.org/sysadmin/ci-utilities/-/tree/master/gitlab-templates

For enabling signed builds of your project builds please see
https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/signing/README.md.
Please note that signing is only permitted on mainline repository branches
marked as protected within Gitlab (which is only done for release branches
such as master and release/24.04). If your project is missing from the
configuration please submit a merge request to sysadmin/ci-utilities.

Signing services include publishing Android builds to our F-Droid
repositories (https://cdn.kde.org/android/), Flatpak repositories (
https://cdn.kde.org/flatpak/), as well as the staging publishers that
upload draft submissions to Google Play and the Microsoft Store (for those
applications using those) - in addition to the actual signing of binaries
(supported for Linux Flatpak, Android, Windows and Mac).

With regards to macOS, pending fixes in the upstream tooling we use
(rcodesign) the signing service will also be able to provide notarised
builds.
See https://invent.kde.org/sysadmin/ci-notary-service/-/merge_requests/38
for more information on that.

At this time signing of Appimages to allow use with AppimageUpdate is not
supported, however to my knowledge this is not used anywhere in KDE outside
of Krita.

Similarly, work on changing our GItlab configuration to protect tags will
be needed before we can perform signed builds on tags (see
https://docs.gitlab.com/ee/user/project/protected_tags.html).
Based on the documentation this should not be too difficult however so i'll
add that to the list to look into sooner rather than later (as making this
change also has implications for the CI system in general)

Thanks,
Ben

Re: Ready-to-use Docker images for KF6-based development ?

[Ben Cooksley]

On Tue, Jan 30, 2024 at 10:17 AM Alexander Neundorf 
wrote:

> Hi,
>

Hi Alex,


>
> I'm still running a Qt5/KF5-based setup on my machine.
> Is there a ready-to-use docker image/Dockerfile which I could use to work
> on a
> KF6-based application, maybe something from KDE neon ?
>

I'd suggest taking a look at invent.kde.org/sysadmin/ci-images which has a
number of images that contain dependencies for just about all KDE software.
These images are used by the CI system and are published in the container
registry so you don't have to build them yourself - you can just "docker
pull" them and you should be good to go.

The Linux images there are based on OpenSUSE and are based on the
assumption you will be building all KDE software (including Frameworks and
Phonon) yourself but you could easily install those if you wanted.


>
> Sorry for the maybe stupid question
> Alex
>

Cheers,
Ben

Re: What can we expect from our developers

[Ben Cooksley]

On Mon, Jan 29, 2024 at 10:38 PM Sune Vuorela  wrote:

> On 2024-01-29, Jonathan Riddell  wrote:
> > This sort of comment  makes me really sad. The All About the Apps goal,
> > which in principle is still ongoing, was an attempt to get KDE developers
> > to realise it was important not just to write apps but to actually make
> > them available to users, I find it astonishing how we still don't have a
> > culture where making our apps available to users is part of our
> > responsibility.  There's teams in KDE to help with all these formats.
> > Sorry to complain here as the issue is larger than just this one app but
> > it's so sad that nobody within KDE wants to help get users using our
> > software directly.
>
> I think this is taking it too far. I think the goal is more about not
> getting in the way of people who wants to do this.
>
> We need to draw a line somewhere about what we expect from *all* of our
> developers.
>
> I don't think that we can expect all of our developers to be great at
>  * c++
>  * qtquick
>  * cmake
>  * windows weirdness
>  * linux weirdness
>  * freebsd weirdness
>  * osx weirdness
>  * android weirdness
>  * packaging flatpaks, appimages, snaps, debs, rpm, .. for linux
>  * making osx installers
>  * making apk's
>  * making windows installers
> Beside the domain of the application.
>
> Yet it is kind of what we are doing with having gating CI setups for
> many of these, and adding more.


> I'm quite a seasoned developer and I know I can't care for all of that.
> I also don't have the time to care for all of these.


> We also don't have extra manpower in the teams that knows about these to
> help everywhere.
>
> We might have a goal about this, but this is just far too many thing to
> be good at everything.
>
> I don't think we should shame individual developers for also realizing
> this.
>
> But where should we draw the line ?
>

For the various platform weirdnesses, it really comes down to whether you
as the team behind an application (even if that team is one person) are
looking to support said platform in some way or another.
If there is interest in supporting the application - then you'll need to
resolve those weirdnesses for that platform - and can then usually build on
that to get it packaged. If you are not, then it shouldn't be there.

This is part of the reason why the KDE on Windows project transitioned to
providing single installers for a specific application (using what is now
called Craft) rather than the package manager type approach taken in the
KDE 4 era.
You can't just compile everything and hope the user experience will be
good, as tuning does need to be done to deliver an optimised installer and
to ensure that everything works properly (such as the application being
able to find the various assets it needs such as icons, QML files, etc).

The only exception to the above would be libraries or other shared
components where other people depend on you (where even if you as a
specific library aren't too concerned about say Android support, there may
still be applications that use you that do care so you should continue to
have CI for those platforms).

>From my perspective it is unreasonable to require people to package for
every platform/format under the sun.

That comes with a caveat though - that if you are looking to get visibility
for your software on other platforms (say Android and Windows) then you
would need to look into the requisite packaging work for that platform (as
it is unreasonable to expect the team that maintains Craft, etc to do that
work). The same would apply for nightly Linux builds, where you'd be
expected to look into the Appimage or Flatpak packaging. That isn't to say
they wouldn't provide pointers or advice, but a degree of investment from
your side in making it work seems to be a reasonable expectation.

The good news here though is that Craft packaging is pretty good at being
shared between platforms - so you can often get things like Windows and
Linux Appimages for the price of one piece of work.

In terms of the original goal that Jonathan was mentioning - my
interpretation of it was to make this as pain free as possible to achieve.
Something that we have mostly done through Gitlab CD systems, which have
the ability to turn out Flatpak bundles, Linux Appimages, Android APKs and
Windows installers and portable archives.


>
> /Sune
>
>
Cheers,
Ben

Re: Major CI changes - FreeBSD and Linux

[Ben Cooksley]

On Mon, Jan 22, 2024 at 10:08 PM Ben Cooksley  wrote:

> Hi all,
>
> Over the past few weeks significant work has been undertaken to develop
> the ability to make use of containerised builds for FreeBSD.
>
> Over the weekend i'm happy to report that this has now been rolled out and
> is now in use across all 5 CI workers that support invent.kde.org. This
> means going forward that we should no longer experience issues with running
> out of disk space on our FreeBSD CI jobs anymore, and we will have the
> ability to ensure others can easily reproduce our setup on their local
> system.
>
> The FreeBSD images for Qt 5.15 and Qt 6.6 that are in use can be found at
> https://invent.kde.org/sysadmin/ci-images along with the other images we
> publish. For those curious about how to set up their own builder,
> instructions can be found in the gitlab-templates/ folder of
> sysadmin/ci-utilities (instructions are also present there for Linux and
> Windows).
>
> Alongside this, we've also transitioned from using Docker on the Linux
> side of the CI workers to using rootless (unprivileged) Podman containers.
> This change was necessitated by changes to Bubblewrap, which is the
> underlying container technology used by flatpak-builder, that made it
> incompatible with the workarounds we previously had in place to run it
> under Docker.
>
> For most projects, this should not pose any issues however due to a last
> minute issue that was discovered during the rollout the DRM virtual gem
> devices while present won't be accessible. To my knowledge this only
> impacts KWin.
> It is possible other projects that were doing actions in their tests that
> need some form of privilege (such as invoking a debugger) may also be
> affected, however in theory there should not be much difference between the
> two container implementations.
>
> This change has also come along with a switch to Debian Bookworm (and the
> 6.1.0 kernel that comes with it) which depending on your tests could also
> have an impact.
>
> The underlying operating system in use within our Linux CI images is not
> changed and continues to be OpenSUSE for desktop Linux, and Ubuntu 22.04
> for Android mobile builds.
>
> At this time the setup for building of Linux images has yet to be adapted,
> so that capability is temporarily unavailable. It is expected to be
> restored in the coming days.
> Until that is completed, we will be unable to rebuild any of our Linux
> images.
>

This has now been corrected and should be functional again.


>
> Thanks,
> Ben
>

Cheers,
Ben

Major CI changes - FreeBSD and Linux

[Ben Cooksley]

Hi all,

Over the past few weeks significant work has been undertaken to develop the
ability to make use of containerised builds for FreeBSD.

Over the weekend i'm happy to report that this has now been rolled out and
is now in use across all 5 CI workers that support invent.kde.org. This
means going forward that we should no longer experience issues with running
out of disk space on our FreeBSD CI jobs anymore, and we will have the
ability to ensure others can easily reproduce our setup on their local
system.

The FreeBSD images for Qt 5.15 and Qt 6.6 that are in use can be found at
https://invent.kde.org/sysadmin/ci-images along with the other images we
publish. For those curious about how to set up their own builder,
instructions can be found in the gitlab-templates/ folder of
sysadmin/ci-utilities (instructions are also present there for Linux and
Windows).

Alongside this, we've also transitioned from using Docker on the Linux side
of the CI workers to using rootless (unprivileged) Podman containers. This
change was necessitated by changes to Bubblewrap, which is the underlying
container technology used by flatpak-builder, that made it incompatible
with the workarounds we previously had in place to run it under Docker.

For most projects, this should not pose any issues however due to a last
minute issue that was discovered during the rollout the DRM virtual gem
devices while present won't be accessible. To my knowledge this only
impacts KWin.
It is possible other projects that were doing actions in their tests that
need some form of privilege (such as invoking a debugger) may also be
affected, however in theory there should not be much difference between the
two container implementations.

This change has also come along with a switch to Debian Bookworm (and the
6.1.0 kernel that comes with it) which depending on your tests could also
have an impact.

The underlying operating system in use within our Linux CI images is not
changed and continues to be OpenSUSE for desktop Linux, and Ubuntu 22.04
for Android mobile builds.

At this time the setup for building of Linux images has yet to be adapted,
so that capability is temporarily unavailable. It is expected to be
restored in the coming days.
Until that is completed, we will be unable to rebuild any of our Linux
images.

Thanks,
Ben

Gitlab update - CI future proofing required

[Ben Cooksley]

Hi all,

Over this weekend I completed a series of updates to invent.kde.org, moving
it to the latest supported version of Postgres (14) and Gitlab (16.6).

As part of that Gitlab update, additional security policies began to be
enforced by Gitlab which mean our existing method of including CI templates
is now beginning to be problematic.

To correct this, we need to port our .gitlab-ci.yml files over to the
include:project syntax (see
https://docs.gitlab.com/ee/ci/yaml/#includeproject)

As an example, this might look like for a Qt 6 only project with Linux,
FreeBSD and Windows builds:

include:
  - project: sysadmin/ci-utilities
file:
  - /gitlab-templates/linux-qt6.yml
  - /gitlab-templates/freebsd-qt6.yml
  - /gitlab-templates/windows-qt6.yml

While we've been able to permit the existing syntax to work for now, it is
recommended that projects please look into porting their CI configurations
now to avoid future issues.

Thanks,
Ben

Change to release flows for non-centrally released projects

[Ben Cooksley]

Hi all,

For some time now the workflow for independently released KDE software
(that is, projects outside of Frameworks, Plasma and Gear) has been to
upload it to ftp://upload.kde.org/incoming/ and then file a Sysadmin ticket
(with the file hashes and destination)

There has now been a small change to that workflow, where our tooling that
validates the hashes will now also be validating GPG signatures where they
are provided. For tarballs it is expected that you provide a GPG signature
(*.sig), but these won't be required for binary packages.

GPG signatures will be validated against a keyring built from the keys
located at https://invent.kde.org/sysadmin/release-keyring/ - so you will
now need to have your key added there in advance of filing a Sysadmin
ticket to have your release published.

Please send a merge request to that repository with your key(s) following
the format of $gitlabusern...@keyx.asc to have them added.

Many thanks,
Ben

Re: CI log verbosity

[Ben Cooksley]

On Sat, Nov 4, 2023 at 2:48 AM Ingo Klöcker  wrote:

> On Freitag, 3. November 2023 13:01:26 CET Harald Sitter wrote:
> > What are your thoughts on having the CI be less verbose by default and
> > instead have an env var or some other toggle to switch into verbose
> > mode?
>
> +1
>
> Ideally, the verbose logs would be written to an artifact. Otherwise, it
> will
> be painful to debug intermittent problems. I think the Craft jobs now do
> this
> (also in response to insanely verbose output of some builds which made it
> impossible to see the actual error).
>

Unfortunately the test log output is all governed by CMake/CTest so there
isn't too much we can do ourselves - not without doing quite a bit of work
to read XML files produced by CTest anyway.


>
> > Specifically I'm talking about the qtlogging rules that are currently
> > enabling everything and the kitchen sink. To my mind we should just
> > use the default rules by default.
> > I find that 99% of the time the output is entirely useless in finding
> > what is wrong, if anything it gets in the way because I first have to
> > find where the test failure is and then instead of reading walls of qt
> > plugin info I will just proceed to reproduce the problem locally
> > anyway.
>
> Full ACK. I'm almost always only interested in finding the error which
> caused
> the CI job to fail and sometimes in seeing compiler warnings.
>
> Regards,
> Ingo
>

Cheers,
Ben

Re: CI log verbosity

[Ben Cooksley]

On Sat, Nov 4, 2023 at 1:01 AM Harald Sitter  wrote:

> What are your thoughts on having the CI be less verbose by default and
> instead have an env var or some other toggle to switch into verbose
> mode?
>
> Specifically I'm talking about the qtlogging rules that are currently
> enabling everything and the kitchen sink. To my mind we should just
> use the default rules by default.
> I find that 99% of the time the output is entirely useless in finding
> what is wrong, if anything it gets in the way because I first have to
> find where the test failure is and then instead of reading walls of qt
> plugin info I will just proceed to reproduce the problem locally
> anyway.
>

The current approach to enabling all of the debugging information was
adopted a few years back when dfaure was trying to debug some tests.
We were back on Jenkins then though, which made it quite difficult to set
things yourself, unlike what you can do now with Gitlab CI and a work
branch.

Behaviour is changed from
https://invent.kde.org/sysadmin/ci-utilities/-/commit/67e9aaac0f540005834bd61c60abf01eed01cb12

If anyone would like to propose a set of more narrow, sane logging rules
then please send an MR (would suggest that it is wrapped behind a check for
whether the logging rules variable is set already though)

Cheers,
Ben


>
> Thoughts?
>
> HS
>

[sysadmin/ci-utilities] gitlab-templates: Move Linux CI for Qt 6 over to Qt 6.6.

[Ben Cooksley]

Git commit 55f8993e028b2597dea44077cd49eb91bb9d87e4 by Ben Cooksley.
Committed on 31/10/2023 at 10:23.
Pushed by bcooksley into branch 'master'.

Move Linux CI for Qt 6 over to Qt 6.6.

CCMAIL: kde-de...@kde.org
CCMAIL: kde-core-devel@kde.org
CCMAIL: kde-frameworks-de...@kde.org
CCMAIL: plasma-de...@kde.org

M  +5-5gitlab-templates/linux-qt6-static.yml
M  +5-5gitlab-templates/linux-qt6.yml

https://invent.kde.org/sysadmin/ci-utilities/-/commit/55f8993e028b2597dea44077cd49eb91bb9d87e4

diff --git a/gitlab-templates/linux-qt6-static.yml 
b/gitlab-templates/linux-qt6-static.yml
index 3e1f3fb..852f0b6 100644
--- a/gitlab-templates/linux-qt6-static.yml
+++ b/gitlab-templates/linux-qt6-static.yml
@@ -1,13 +1,13 @@
-suse_tumbleweed_qt65_static:
+suse_tumbleweed_qt66_static:
   stage: build
-  image: invent-registry.kde.org/sysadmin/ci-images/suse-qt65:latest
+  image: invent-registry.kde.org/sysadmin/ci-images/suse-qt66:latest
   tags:
 - Linux
   variables:
-KDECI_CC_CACHE: /mnt/caches/suse-qt6.5-static/
-KDECI_CACHE_PATH: /mnt/artifacts/suse-qt6.5-static/
+KDECI_CC_CACHE: /mnt/caches/suse-qt6.6-static/
+KDECI_CACHE_PATH: /mnt/artifacts/suse-qt6.6-static/
 KDECI_GITLAB_SERVER: https://invent.kde.org/
-KDECI_PACKAGE_PROJECT: teams/ci-artifacts/suse-qt6.5-static
+KDECI_PACKAGE_PROJECT: teams/ci-artifacts/suse-qt6.6-static
   interruptible: true
   before_script:
 - git clone https://invent.kde.org/sysadmin/ci-utilities.git --depth=1
diff --git a/gitlab-templates/linux-qt6.yml b/gitlab-templates/linux-qt6.yml
index 71e5c03..5f0ef50 100644
--- a/gitlab-templates/linux-qt6.yml
+++ b/gitlab-templates/linux-qt6.yml
@@ -1,13 +1,13 @@
-suse_tumbleweed_qt65:
+suse_tumbleweed_qt66:
   stage: build
-  image: invent-registry.kde.org/sysadmin/ci-images/suse-qt65:latest
+  image: invent-registry.kde.org/sysadmin/ci-images/suse-qt66:latest
   tags:
 - Linux
   variables:
-KDECI_CC_CACHE: /mnt/caches/suse-qt6.5/
-KDECI_CACHE_PATH: /mnt/artifacts/suse-qt6.5/
+KDECI_CC_CACHE: /mnt/caches/suse-qt6.6/
+KDECI_CACHE_PATH: /mnt/artifacts/suse-qt6.6/
 KDECI_GITLAB_SERVER: https://invent.kde.org/
-KDECI_PACKAGE_PROJECT: teams/ci-artifacts/suse-qt6.5
+KDECI_PACKAGE_PROJECT: teams/ci-artifacts/suse-qt6.6
   interruptible: true
   before_script:
 - git clone https://invent.kde.org/sysadmin/ci-utilities.git --depth=1

Re: KDE Review: Hash-o-Matic

[Ben Cooksley]

On Mon, Oct 2, 2023 at 10:16 AM Ingo Klöcker  wrote:

> On Sonntag, 1. Oktober 2023 21:49:36 CEST Carl Schwan wrote:
> > It would be great if someone could create a product on bugs.kde.org and
> > assign myself to the product.
>
> Are you sure you cannot create the product yourself?
>
> https://bugs.kde.org/editproducts.cgi


Only a small number of people have the 'editcomponents' permission on
Bugzilla (of which it appears you are one of those)

Carl - we'll need a bit more information to create a product, such as a
list of components and the descriptions that should be set on both the
product and the various components, then we can set this up.


>
> Regards,
> Ingo


Cheers,
Ben

General Availability - Updated Gitlab Runners

[Ben Cooksley]

Hi all,

Today we deployed replacements to node3, node4 and node5 - which were the
remaining old workers attached to Invent.

This means that all workers have now completed being updated to a more
modern host operating system (Ubuntu 22.04) as well as newer generation
hardware (with the CPU now being a Ryzen 7700 compared to the previous
Ryzen 3700X).

Windows builds have also been moved over, with the only change there being
a reduction from 24GB RAM to 16GB RAM being allocated to Windows on those
three nodes. Windows Server 2022 Datacenter Edition continues to be used as
the host OS for those builds.

The legacy builders are still currently performing FreeBSD builds, however
I will provision those FreeBSD VMs shortly so they should transition across
soon as well.

Please let us know if there are any issues.

Cheers,
Ben

Re: drkonqi's many debuggers

[Ben Cooksley]

On Tue, Aug 29, 2023 at 4:25 PM Thiago Macieira  wrote:

> On Monday, 28 August 2023 20:30:36 PDT Harald Sitter wrote:
> > I am not quite following. We can depend on any debugger we want,
> > surely? I mean, there are practical implications to consider for sure,
> > but gdb being the dominant debugger on Linux doesn't prevent us from
> > depending on lldb instead.
>
> It does because it might be missing in the system far more often than gdb.
> We'd get more backtraces and therefore more data if we focused on gdb
>
> Another point is that Linux distributions have been shipping gdb with
> debuginfod support for a year or two. lldb support for it is still pending:
> https://github.com/llvm/llvm-project/issues/52732
>
> Therefore, I repeat: if there's room for only one, it's gdb.
>
> If we do support both, then on Linux gdb must be used in preference.
>

As an additional note here, my preference would also be for gdb to be used,
if only because it is more reliable and dependable.

Currently we have special code in the CI tooling to ensure that stray lldb
processes left behind by unit tests on our FreeBSD CI workers are all
killed off.
Otherwise they sit there eating up an entire CPU core at 100% utilisation
until an admin logs in and kills them off.

Cheers,
Ben


> --
> Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org
>Software Architect - Intel DCAI Cloud Engineering
> .
>
>
>

Re: Review of Codevis (ie - Making Codevis a KDE Project)

[Ben Cooksley]

On Sat, Aug 19, 2023 at 3:37 AM Tomaz Canabrava  wrote:

> (some help / I need to set the default branch to master, from main,
> because the tooling doesn't accept the later, I don't think I have the
> permission to do that).
>

That has been done now.

Cheers,
Ben


>
> On Fri, Aug 18, 2023 at 5:10 PM Tomaz Canabrava 
> wrote:
>
>> Small update that the CI is now fully passing.
>>
>> On Fri, Aug 18, 2023 at 2:25 PM Tomaz Canabrava 
>> wrote:
>>
>>> Carl, Sysadmins:
>>>
>>> The current error on the KDE ci is this:
>>>
>>> Looking for clang tool headers at /usr/lib64/clang/16.0.6/include. You
>>> can change this by defining CT_CLANG_HEADERS_DIR
>>> CMake Error at CMakeLists.txt:87 (message):
>>> Cannot find clang tool headers at /usr/lib64/clang/16.0.6/include
>>> -- Configuring incomplete, errors occurred!
>>>
>>> (to which I understand that carl said there's an error with Clang6. This
>>> is not an error - it basically says that we are unable to find `stddef.h`
>>> on the  path `${LLVM_LIBRARY_DIR}/clang/${LLVM_PACKAGE_VERSION}/include`
>>>
>>> This is needed for the tool to run properly, but not compile, so I
>>> removed the FATAL from the message.
>>>
>>> On Thu, Aug 17, 2023 at 6:51 PM Tomaz Canabrava 
>>> wrote:
>>>


 On Thu, 17 Aug 2023 at 18:29 Carl Schwan  wrote:

> On Thursday, August 17, 2023 11:18:24 AM CEST Tomaz Canabrava wrote:
> > Hello Fellow KDE Devs,
> >
> > I'm here, formally asking for a review of the Codevis project, to
> move
> > forward and make it a part of kdesdk.
>
> Very cool project, I was amazed by the presentation of it from
> tarcisio at
> Akademy.
>
> > Currently we are using parts of KWdigetsAddons as a submodule
> > Most things that are related to buildsystems will be moved to craft /
> > kdesrc-build as soon as possible, right now we rely in conan for
> windows
> > and mac, plus a hand-written build script that downloads and builds
> llvm
> > for those platforms.
> >
> > Things that I know that are out of KDE Accordance:
> > - Translation System (uses Qt's tr() system)
>
> This isn't an issue and we have other KDE projects using the tr()
> system. But
> if you want to port to ki18n, it's best to do it now since you don't
> seems to
> have any translations yet.
>
> > - Settings System (it uses my own configuration parser that
> resembles QML)
>
> Yeah probably best to use kconfigxt or make your configuration parser
> part of
> kconfigxt next gen ;)
>
> > - Folder naming specification (follows the lakosian naming
> specification)
>
> I don't think we have any folder (and file) naming specification in
> kde, or at
> least if we have one, it varies a lot between projects.
>
> > - CI used is based on Gitlab, but fails on KDE
>
> When trying to build it on my laptop it failed, due to the requirement
> of
> clang 16. This might also be an issue with the kde ci on tumbleweed.


 Carl,

 There’s no requirement for clang16 (I build with 15, tarcisio builds
 with 14, the previous ci had 13, I believe)

 Mind if you share the build logs?

 Best




>
> > The current repository of Codevis is:
> > https://invent.kde.org/tcanabrava/codevis
> >
> > The KDE developers on this project are me, tarcisio fischer (that
> presented
> > Codevis on Akademy), and Richard Dale.
> >
> > Best regards,
> > Tomaz
>
>

Re: Review of Codevis (ie - Making Codevis a KDE Project)

[Ben Cooksley]

On Thu, Aug 17, 2023 at 9:20 PM Tomaz Canabrava  wrote:

> (Another thing to add, it is currently on Qt5 but we plan to move it to
> Qt6 as soon as possible, and as soon as we have a working CI on windows,
> mac and linux on KDE infrastructure)
>
>
> On Thu, Aug 17, 2023 at 11:18 AM Tomaz Canabrava 
> wrote:
>
>> Hello Fellow KDE Devs,
>>
>> I'm here, formally asking for a review of the Codevis project, to move
>> forward and make it a part of kdesdk.
>>
>> Currently we are using parts of KWdigetsAddons as a submodule
>> Most things that are related to buildsystems will be moved to craft /
>> kdesrc-build as soon as possible, right now we rely in conan for windows
>> and mac, plus a hand-written build script that downloads and builds llvm
>> for those platforms.
>>
>> Things that I know that are out of KDE Accordance:
>> - Translation System (uses Qt's tr() system)
>> - Settings System (it uses my own configuration parser that resembles QML)
>> - Folder naming specification (follows the lakosian naming specification)
>> - CI used is based on Gitlab, but fails on KDE
>>
>
For your CI items here:

On Linux, we don't provide non-Docker workers, and Docker-in-Docker is not
permitted for security reasons (as it essentially means you are root on the
runner).
You'll need to adjust the job to give Gitlab the name of the image you need
to run under - usage of the provided images at
invent.kde.org/sysadmin/ci-images is preferred.

Due to the ageing out of CentOS 7 we're currently in the process of
migrating towards SLES15 based images for our Appimage builds.

For Windows, we also don't provide non-Docker workers, and we only allow
the use of blessed images we provide (in invent.kde.org/sysadmin/ci-images).
It looks like you are doing quite a bit of stand up work in your custom
build script though, so please consider whether one of the images we
already provide is suitable for your purposes in terms of dependencies
provided (I believe LLVM may already be available in the -qt515 and -qt65
images)

Assuming your project is a regular CMake project though you might want to
consider starting by adding the regular KDE CI jobs to the project though
and seeing how they go (see the templates at
invent.kde.org/sysadmin/ci-utilities)


>> The current repository of Codevis is:
>> https://invent.kde.org/tcanabrava/codevis
>>
>> The KDE developers on this project are me, tarcisio fischer (that
>> presented Codevis on Akademy), and Richard Dale.
>>
>> Best regards,
>> Tomaz
>>
>
Cheers,
Ben

New CI workers - node1 and node2

[Ben Cooksley]

Hi all,

Over the last 2 days i've been busy connecting two new CI workers to
GitLab, which are the beginning of long overdue improvements to our CI
arrangements needed to support the final retirement of the Binary Factory.

While developers shouldn't notice much in the way of changes, this will
bring us a series of long term benefits which include:
- The builders host OS being significantly more up to date, which will
allow certain GPU related tests to be run again
- The ability to build Webengine on the CI system (for Craft caches
primarily) which will support our builds of Linux appimages
- Reduced time to wait for a build to start as more build resources will be
available to Gitlab (while we've had 5 workers for a long time, 2 were
connected to the Binary Factory and were unavailable to Gitlab)

On the Linux and Windows side, the two new builders - node1 and node2 - are
already available and should be carrying out builds already.
I'm still waiting on some details for FreeBSD, but once that has been
sorted we should be able to provision those as well.

For those curious, these two builders (node1 and node2) are equipped with
Ryzen 7 7700 CPUs, 128GB RAM and 1TB of Gen4 NVMe storage, although not all
of this is available to Linux builds as it is apportioned partly to Windows
and FreeBSD VMs.

Once these two machines are in full service, two of the current nodes
(node3 and node4) will be retired to allow them to be replaced with newer
machines as the Binary Factory shutdown approaches (watch this space).

Thanks,
Ben

Re: ACTION REQUIRED - Gitlab and Subversion server migration

[Ben Cooksley]

On Tue, Jul 25, 2023 at 1:35 AM Vít Pelčák  wrote:

>
> ne 23. 7. 2023 v 12:01 odesílatel Ben Cooksley  napsal:
>
>> Good morning KDE Developers,
>>
>> As many of you will be aware, today Gitlab and our Subversion repository
>> were both migrated to a new home - on a more modern and more powerful
>> server, which should better support future work.
>>
>> As a consequence the host key of the server has now changed, which means
>> you will need to take steps on your system otherwise you won't be allowed
>> to connect to the new server.
>>
>> Please ensure you run the following two commands to clear out any
>> existing host keys:
>> - ssh-keygen -R invent.kde.org -f ~/.ssh/known_hosts
>> - ssh-keygen -R svn.kde.org ~/.ssh/known_hosts
>>
>
> I suppose you meant
> ssh-keygen -R svn.kde.org -f ~/.ssh/known_hosts
>

That is correct.


>
> right?
>
>
Cheers,
Ben

ACTION REQUIRED - Gitlab and Subversion server migration

[Ben Cooksley]

Good morning KDE Developers,

As many of you will be aware, today Gitlab and our Subversion repository
were both migrated to a new home - on a more modern and more powerful
server, which should better support future work.

As a consequence the host key of the server has now changed, which means
you will need to take steps on your system otherwise you won't be allowed
to connect to the new server.

Please ensure you run the following two commands to clear out any existing
host keys:
- ssh-keygen -R invent.kde.org -f ~/.ssh/known_hosts
- ssh-keygen -R svn.kde.org ~/.ssh/known_hosts

Following these commands the next time you try to connect you will be
prompted to confirm the new host key and trust it for use. For those who
would like to confirm that host key, it is as follows:

256 SHA256:zHdK2R/S6s5Oj71N0s8LHWCXXsUt+DCztd+GjzW9KlU root@lerwini
(ED25519)
256 SHA256:ZNBg4AkRxbt/N6xzpt7GbmmS78A3WFy5lz0l/cPHbcE root@lerwini (ECDSA)
3072 SHA256:KxAoV6VsbKvAocFZCJlxtmPDScmUCRNiUiOCSXNSC/k root@lerwini (RSA)

Please let us know, via either sysad...@kde.org or kde-de...@kde.org if you
encounter any issues with the new system.

Many thanks,
Ben Cooksley
KDE Sysadmin

Downtime notification - Gitlab server migration

[Ben Cooksley]

Good morning all,

This evening (UTC time) I will be moving Gitlab from the current system it
is located on to a new server.

This move is being undertaken to ensure we have the most up to date
software stack, as well as take advantage of newer hardware that is both
more efficient and more powerful (the existing server is around 4 years old
or so).

Based on performance benchmarks, the new system should have a significant
performance uplift, which will help ensure we're able to support new
workloads - such as the incoming migration of binary build jobs from the
Binary Factory.

This migration will take several hours unfortunately, during which time
Gitlab will be completely unavailable. This includes read and write
operations to Git repositories, authentication services for KDE.org sites
that use Invent login, issues, wiki pages and everything else hosted by
Gitlab.

This migration will be undertaken from approximately 10pm UTC, Saturday
22nd July.

Please let me know if you have any questions on the above.

Cheers,
Ben

Re: XWayland Video Bridge in kdereview

[Ben Cooksley]

On Fri, Jun 30, 2023 at 10:01 AM Aleix Pol  wrote:

> On Mon, Jun 19, 2023 at 10:00 PM Jonathan Riddell  wrote:
> >
> > I got this compiling on neon unstable.  Our kpipewire is qt6 so I'm not
> sure how that affects it.  I don't have a way to test it quickly though.
> >
> https://build.neon.kde.org/job/jammy_unstable_neon-packaging_xwaylandvideobridge/
>
> The Qt 6 port is still ongoing:
> https://invent.kde.org/system/xwaylandvideobridge/-/merge_requests/11
>
> > How come it's in the system group on invent if you want it to be part of
> Plasma?
>
> To be honest, the invent groups are a mystery to me.
>

That is of some degree of concern that you don't understand the groupings.

For the most part the groups are intended to reflect to an extent how the
software relates to each other. You will find all of the Educational
software under Education for instance, and all Games in Games.
Likewise, applications for handling images and other similar content are in
Graphics, those that deal with video and audio are in Multimedia and
libraries for use by other developers are in Libraries.

All fairly straightforward I would have hoped.

The reason for the placement of the XWayland Video Bridge comes back to the
initial ticket that requested it be moved from a personal repository -
which you filed Aleix.
This was targeting Graphics, which is clearly inappropriate as it would
have put it alongside Krita and Digikam. For want of a better place as it
was a bit of infrastructure / behind the scenes tooling it was therefore
put in System.

There was never any mention in the ticket that it was going to be exclusive
to Plasma and not usable outside of a Plasma setting, hence why that was
not considered.


>
> Aleix
>

Cheers,
Ben

Re: KSvg in kdereview

[Ben Cooksley]

On Wed, Jun 21, 2023 at 10:12 PM Harald Sitter  wrote:

> LGTM now +2
>
> On Wed, Jun 21, 2023 at 10:04 AM Marco Martin  wrote:
> >
> > I fixed CI, passes now
>

Thanks for correcting that.

As Friedrich raised the initial concerns it would be nice to have him
confirm that the code quality issues he found have all been corrected.
Note that having something step into Frameworks is a big deal, so this is
something that should not be rushed and if there are issues with the code
we should fix them sooner vs. later.

Thanks,
Ben


> >
> > On Tue, Jun 20, 2023 at 8:44 PM Ben Cooksley  wrote:
> > >
> > > Hi all,
> > >
> > > Sysadmin has just received a request to move this repository to
> Frameworks, however after seeing some of the comments raised here regarding
> the repository I took a look myself to see if they had been corrected.
> > >
> > > At this time CI is still broken in KSvg, for both platforms -
> something that was mentioned in an earlier email here.
> > > This does not inspire confidence that the code is up to scratch.
> > >
> > > I've therefore declined to move it to Frameworks at this time.
> > >
> > > Would be appreciated, given this is looking to be promoted to
> Frameworks, if people could please have a further look at this repository
> and comment as appropriate.
> > >
> > > Thanks,
> > > Ben
> > >
> > > On Wed, Jun 14, 2023 at 9:12 PM Marco Martin 
> wrote:
> > >>
> > >> Hi all,
> > >> Some time has passes, and changes have been done in the repo to
> > >> address some of the points.
> > >> Now there are review requests in plasma-framework which depends on
> > >> this repo (and accompanying plasma-workspace, plasma-desktop etc)
> > >> It would probably be better to have this in frameworks to have the
> > >> rest depending from it?
> > >>
> > >> On Thu, Apr 20, 2023 at 10:25 AM Marco Martin 
> wrote:
> > >> >
> > >> > Hi all,
> > >> > A part of plasma-framewrok, which is the one to do SVG-based themes,
> > >> > has now been splitted in a standalone library which is intended to
> be
> > >> > a new framework in KF6 (all usages of the plasma-framework version
> > >> > will be ported once this officially lands, and then those classes
> will
> > >> > be removed)
> > >> > The repo for now lives in
> > >> > https://invent.kde.org/libraries/plasmasvg
> > >> >
> > >> > In the end it will be renamed in ksvg
> > >> >
> > >> > Comments? reviews?
> > >> >
> > >> >
> > >> > --
> > >> > Marco Martin
> > >>
> > >> --
> > >> Marco Martin
>

Re: KSvg in kdereview

[Ben Cooksley]

Hi all,

Sysadmin has just received a request to move this repository to Frameworks,
however after seeing some of the comments raised here regarding the
repository I took a look myself to see if they had been corrected.

At this time CI is still broken in KSvg, for both platforms - something
that was mentioned in an earlier email here.
This does not inspire confidence that the code is up to scratch.

I've therefore declined to move it to Frameworks at this time.

Would be appreciated, given this is looking to be promoted to Frameworks,
if people could please have a further look at this repository and comment
as appropriate.

Thanks,
Ben

On Wed, Jun 14, 2023 at 9:12 PM Marco Martin  wrote:

> Hi all,
> Some time has passes, and changes have been done in the repo to
> address some of the points.
> Now there are review requests in plasma-framework which depends on
> this repo (and accompanying plasma-workspace, plasma-desktop etc)
> It would probably be better to have this in frameworks to have the
> rest depending from it?
>
> On Thu, Apr 20, 2023 at 10:25 AM Marco Martin  wrote:
> >
> > Hi all,
> > A part of plasma-framewrok, which is the one to do SVG-based themes,
> > has now been splitted in a standalone library which is intended to be
> > a new framework in KF6 (all usages of the plasma-framework version
> > will be ported once this officially lands, and then those classes will
> > be removed)
> > The repo for now lives in
> > https://invent.kde.org/libraries/plasmasvg
> >
> > In the end it will be renamed in ksvg
> >
> > Comments? reviews?
> >
> >
> > --
> > Marco Martin
>
> --
> Marco Martin
>

Retirement of IRC Services and KDETalk.net (Jabber)

[Ben Cooksley]

Hi all,

For some time now, the level of use of our IRC services - notably being
Pursuivant and the Telegram Matterbridge, but also including sKreamer - has
been on the decline.

I'd therefore like to permanently retire all three of these services.

Depending on the level of community interest, we may opt to retain
pursuivant however i'd like for it to be rebuilt as a Matrix native service
rather than being a continuation of our existing irker based bot that
occasionally has issues and falls off.

Given that we are now fairly well migrated to Matrix, the need to maintain
our Telegram bridging is much reduced, and i'd therefore like to retire
that without replacement.

In terms of sKreamer, it's primary utility has been to provide
announcements of Forum posts and bugbot services. With Matrix providing
site previews, and the Forum in imminent replacement by Discourse, both of
these are no longer necessary - so i'd like to retire it without
replacement as well.

The only remaining service of contention here is the BNC, which has
significantly less use now than it did many years ago - with only 30 active
connections at the time of writing. It therefore appears to be of much less
need than it was in years past, and i'd also like to retire it as well.

Finally, many years ago (prior to my time in Sysadmin) we started providing
Jabber services for the domains KDETalk.net and KDE.org. Due to abuse
however, we have for a long time had to have registration on KDETalk.net
disabled (KDE.org was always a manual registration). Much like the BNC,
this appears to only have 19 active clients at the time of writing. As our
official channel for chat is essentially Matrix now, I would like to retire
this as well.

Together, all of these retirements will allow us to retire one of our
smaller DigitalOcean servers (the load all of these generate is
computationally small and thus cheap, however they do occupy mental
headspace that is better served focusing on other areas of our
infrastructure).

Comments on the above?

Thanks,
Ben

[sysadmin/ci-management] latest: Remove kdewebkit from the CI job seeds.

[Ben Cooksley]

Git commit 41002ce5f78d403f001b269153410c5910f529f2 by Ben Cooksley.
Committed on 22/04/2023 at 19:55.
Pushed by bcooksley into branch 'master'.

Remove kdewebkit from the CI job seeds.
Due to the unmaintained status (and removal) of QtWebKit with openSUSE we have 
dropped it from our images, meaning kdewebkit can no longer be built.

All projects are advised that they should also be dropping any dependency they 
have on kdewebkit / QtWebKit as it is no longer supported within a CI system 
context.

CCMAIL: kde-frameworks-de...@kde.org
CCMAIL: kde-core-devel@kde.org
CCMAIL: kde-de...@kde.org
CCMAIL: kde-...@kde.org
CCMAIL: kmymoney-de...@kde.org

M  +0-1latest/frameworks.yml

https://invent.kde.org/sysadmin/ci-management/commit/41002ce5f78d403f001b269153410c5910f529f2

diff --git a/latest/frameworks.yml b/latest/frameworks.yml
index aa0e3cc..5a917d7 100644
--- a/latest/frameworks.yml
+++ b/latest/frameworks.yml
@@ -55,7 +55,6 @@
 "frameworks/kdbusaddons": "kf5"
 "frameworks/kdeclarative": "kf5"
 "frameworks/kdesignerplugin": "kf5"
-"frameworks/kdewebkit": "kf5"
 "frameworks/kdnssd": "kf5"
 "frameworks/kdoctools": "kf5"
 "frameworks/kemoticons": "kf5"

Re: Gitlab Downtime

[Ben Cooksley]

On Tue, Apr 11, 2023 at 9:23 PM Ben Cooksley  wrote:

> Hi all,
>
> Tomorrow I will need to conduct some maintenance on our Gitlab instance
> which may take approximately 60 to 90 minutes in time, depending on how
> things go.
>
> This downtime is needed to facilitate the update of several components
> that Gitlab relies upon, including the underlying Ruby interpreter and will
> pave the way for us to migrate to a newer version of GitLab in the coming
> days - as well as a replacement GitLab server (which can now be considered
> as GitLab have finally moved to using Ruby 3).
>
> All going well, i'm hopeful the maintenance will take significantly less
> time than i'm allowing for here.
>
> Maintenance will start at approximately 1930 NZST ( 0730 UTC ) on 11
> April, during which time all services associated with Gitlab (including SSO
> login, Git repositories, etc) will be unavailable.
>

Small typo correction here: 12 April, ie. tomorrow.


>
> Apologies in advance for the disruption.
>
> Many thanks,
> Ben
>

Thanks,
Ben

Gitlab Downtime

[Ben Cooksley]

Hi all,

Tomorrow I will need to conduct some maintenance on our Gitlab instance
which may take approximately 60 to 90 minutes in time, depending on how
things go.

This downtime is needed to facilitate the update of several components that
Gitlab relies upon, including the underlying Ruby interpreter and will pave
the way for us to migrate to a newer version of GitLab in the coming days -
as well as a replacement GitLab server (which can now be considered as
GitLab have finally moved to using Ruby 3).

All going well, i'm hopeful the maintenance will take significantly less
time than i'm allowing for here.

Maintenance will start at approximately 1930 NZST ( 0730 UTC ) on 11 April,
during which time all services associated with Gitlab (including SSO login,
Git repositories, etc) will be unavailable.

Apologies in advance for the disruption.

Many thanks,
Ben

Forks cleanup

[Ben Cooksley]

Hi all,

While doing some routine maintenance and system assessments on our Gitlab
instance over the past week, it has become apparent that in certain
circumstances the de-duplication of objects between parent repositories and
their forks does not work correctly in some circumstances.

While this isn't a significant issue as the system is well oversized to
handle growth, it is causing backups to take longer than they should to run
and makes certain estimates of future resource needs difficult to determine.

It would therefore be appreciated if people could please remove any forks
they are no longer using.

If the fork in question is in the below list, please consider recreating
your fork (by deleting it and creating a new one) when the opportunity next
arises.

This especially applies if you have a fork of any of the following projects:
- websites/kde-org
- graphics/krita
- graphics/digikam
- education/kstars
- documentation/digikam-doc

Thanks,
Ben

Re: PSA: Mark SVG as binary

[Ben Cooksley]

On Sat, Mar 4, 2023 at 11:03 AM Ingo Klöcker  wrote:

> On Freitag, 3. März 2023 22:49:06 CET Ben Cooksley wrote:
> > In all cases i'm aware of, the files have been in either SVG, JSON or XML
> > in format.
> >
> > To fix this, and allow changes to be merged please add a ".gitattributes"
> > file at the top level of the Git repository you are seeing this behaviour
> > in on the master branch with the following contents:
> >
> > *.svg binary
>
> Wouldn't replacing the *.svg files with compressed SVGs also solve the
> problem?
> If git is anyway told to treat the SVGs as binary, then keeping them as
> uncompressed text files makes little sense to me. Of course, there may be
> valid
> exceptions, but I'm certain that in general replacing the uncompressed
> SVGs
> with compressed SVGs would be fine.
>

Yes it would, however that would defeat the ability of Git's data
compression capabilities to keep the size of our repositories down so we
should still keep them as text files.
The .gitattributes file as far as I am aware does not influence that part
of Git, it is used by the parts relating to generating diffs and converting
EOL formats between operating systems (among others).


>
> Regards,
> Ingo


Cheers,
Ben

PSA: Mark SVG as binary

[Ben Cooksley]

Hi all,

Over the past few months Sysadmin has periodically received reports of our
Git hooks failing when processing commits in certain circumstances.

Looking at all of these instances a clear trend has emerged, with files
that have lines in them that are extremely and excessively long being the
culprit of this issue. In all instances, the files in question have been
generated by software and not been written by humans.

While it is not entirely clear why this issue has suddenly appeared (as the
code within the Git hooks causing the failure has been in production for
10+ years) I am suspecting a change in behaviour in either Git or Python to
be the cause of this (with Git being most likely).

In all cases i'm aware of, the files have been in either SVG, JSON or XML
in format.

To fix this, and allow changes to be merged please add a ".gitattributes"
file at the top level of the Git repository you are seeing this behaviour
in on the master branch with the following contents:

*.svg binary

Amending as needed to cover other files that also contain content that has
a meaningless diff.

Note that the file *must* be placed at the root of the repository and on
the master branch, otherwise it will not be propagated by GitLab to our
hooks.

Thanks,
Ben

CI Outage

[Ben Cooksley]

Hi all,

As many of you will have noticed, the Linux side of our Gitlab CI setup,
including those runs for Android and other miscellaneous jobs (such as
cppcheck) were all KO yesterday due to a Docker error.

This has now been corrected, and was due to a defect in an update shipped
by the Docker upstream maintainers. They added a hard dependency on an
Apparmor CLI tool, however failed to add the corresponding dependency to
their packages (and to top matters off, part of that package is kernel side
and only does it's initialisation on system boot...)

The nodes have now had their setups corrected and have been rebooted, and
everything is back in service. I have also kicked off builds of just about
everything that is failing on the CI system.

Apologies for the disruption here.

While doing so I noticed a fairly sad number of actual
failures-to-build-from-source:
- Mandatory X11 dependency on Windows:
https://invent.kde.org/utilities/keditbookmarks
- Blind KF6 porting: https://invent.kde.org/pim/kontact/-/jobs/786385
- (and others yet to finish working their way through)

It also looks like we have some packages that are not being rebuilt by seed
jobs (see https://invent.kde.org/pim/kalendar/-/jobs/786366), so once the
system has finished playing catch up we may need to do some additional
infrastructure work there.

Please contact me if you'd like to assist with that.

Thanks,
Ben

Re: Downtime Notice: Bugzilla - bugs.kde.org

[Ben Cooksley]

On Mon, Feb 6, 2023 at 2:30 AM Thomas Baumgart  wrote:

> On Sonntag, 5. Februar 2023 12:06:16 CET Ben Cooksley wrote:
>
> > Hi all,
> >
> > This migration has now been completed this evening, and Bugzilla should
> be
> > up and running now in its new home.
> >
> > As part of this I did have to apply a patch to correct Bugzilla's use of
> > Perl's email libraries, so please report any issues you see in case other
> > parts of the codebase have also bitrotted and broken.
> > At some point in the not too distant future we may need to evaluate a
> > replacement platform to Bugzilla if upstream does not release a newer
> > version.
>
> I noticed, that a bug was not closed based on a commit though it should
> have
> been. Here's the console log of the push
>
> Enumerating objects: 9, done.
> Counting objects: 100% (9/9), done.
> Delta compression using up to 12 threads
> Compressing objects: 100% (5/5), done.
> Writing objects: 100% (5/5), 504 bytes | 504.00 KiB/s, done.
> Total 5 (delta 4), reused 0 (delta 0), pack-reused 0
> remote: The commits in this series can be viewed at:
> remote:
> https://invent.kde.org/office/kmymoney/commit/4d5599ca64795245a2f5fa4d4e5203364c74619c
> remote: Closing bug 464055
> remote:
> remote: To create a merge request for 5.1, visit:
> remote:
> https://invent.kde.org/office/kmymoney/-/merge_requests/new?merge_request%5Bsource_branch%5D=5.1
> remote:
> To ssh://invent.kde.org/office/kmymoney.git
>3f8e65e56..4d5599ca6  HEAD -> 5.1
>
> but on https://bugs.kde.org/show_bug.cgi?id=464055 the commit cannot be
> seen
> and the status did not change :(
>

It seems that in your particular instance your domain is protected by a
DMARC policy that flags any email which fails SPF validation for quarantine.
Which our mail infrastructure happily did - preventing the changes from
being made to the bug.

I've tweaked the Bugzilla hooks so that they'll evade SPF restrictions now
which should help with that hopefully.

Good news is because it was quarantined, your hook trigger email could be
released - which i've now done.

Thanks,
Ben


>
> --
>
> Regards
>
> Thomas Baumgart
>
> -
> There are two rules for success in life:
> Rule 1: Don't tell people everything you know.
> -
>

Re: Downtime Notice: Bugzilla - bugs.kde.org

[Ben Cooksley]

Hi all,

This migration has now been completed this evening, and Bugzilla should be
up and running now in its new home.

As part of this I did have to apply a patch to correct Bugzilla's use of
Perl's email libraries, so please report any issues you see in case other
parts of the codebase have also bitrotted and broken.
At some point in the not too distant future we may need to evaluate a
replacement platform to Bugzilla if upstream does not release a newer
version.

Thanks,
Ben

On Sun, Jan 29, 2023 at 11:22 PM Ben Cooksley  wrote:

> Hi Community,
>
> This email is advance notice that Bugzilla will be unavailable for several
> hours this coming weekend as part of a server migration.
>
> Due to the size of the Bugzilla database it is anticipated that it will
> take several hours to take a final backup copy from the old system and then
> import it on the new system.
>
> An import was done somewhat recently of a bugs.kde.org backup snapshot
> into bugstest.kde.org, which will allow for content to be accessed for
> older bugs during this downtime.
>
> Please let me know if there are any queries on this.
>
> Regards,
> Ben Cooksley
> KDE Sysadmin
>

Downtime Notice: Bugzilla - bugs.kde.org

[Ben Cooksley]

Hi Community,

This email is advance notice that Bugzilla will be unavailable for several
hours this coming weekend as part of a server migration.

Due to the size of the Bugzilla database it is anticipated that it will
take several hours to take a final backup copy from the old system and then
import it on the new system.

An import was done somewhat recently of a bugs.kde.org backup snapshot into
bugstest.kde.org, which will allow for content to be accessed for older
bugs during this downtime.

Please let me know if there are any queries on this.

Regards,
Ben Cooksley
KDE Sysadmin

Re: Request to relicense all CC0-1.0 code to MIT (or similar permissive license)

[Ben Cooksley]

On Mon, Jan 23, 2023 at 2:13 AM Neal Gompa  wrote:

> Hey folks,
>

Hi Neal,


>
> During a review for flatpak-kcm for inclusion in Fedora, I discovered
> that KDE currently licenses its CI scripts under the CC0 (SPDX:
> CC0-1.0) license. This is no longer generally permitted in Fedora for
> software/code due to the explicit exclusion of patent license
> grants[1].
>
> While I realize that the likelihood of practical issues around this is
> pretty low for the case that KDE has been licensing the CI scripts
> under, it does cause headaches for us. I've made a request for a
> exception[2] so that we can continue to ship KDE software without
> having to take extraordinary actions, but I would like to request that
> KDE disallow code to be licensed under CC0-1.0 and recommend all
> existing code under that license to be transitioned to an accepted
> permissive license. My suggestion would be to recommend transitioning
> to the MIT license[3].
>

Not sure where this is coming from, given that the vast bulk of our CI
files are just reference statements for Gitlab to sysadmin/ci-utilities,
which likely doesn't meet the threshold for being a copyrightable work.
As for the content of sysadmin/ci-utilities, it isn't under CC0.

At best this likely only covers those projects with custom CI.

Cheers,
Ben


>
> Thanks in advance and best regards,
> Neal
>
> P.S.: I originally sent this to kde-licens...@kde.org, which
> apparently doesn't exist. Sorry for those CC'd getting this twice.
> Oops!
>
> [1]:
> https://lists.fedoraproject.org/archives/list/le...@lists.fedoraproject.org/message/RRYM3CLYJYW64VSQIXY6IF3TCDZGS6LM/
> [2]:
> https://lists.fedoraproject.org/archives/list/le...@lists.fedoraproject.org/message/Q67KFP4NT3WEL2Z75BYCXMP2IOQHLBJT/
> [3]: https://opensource.org/licenses/MIT
>
> --
> 真実はいつも一つ！/ Always, there's only one truth!
>

Re: GItlab update

[Ben Cooksley]

On Mon, Jan 9, 2023 at 1:05 AM Thomas Baumgart  wrote:

> On Sonntag, 8. Januar 2023 11:03:09 CET Ben Cooksley wrote:
>
> > Hi all,
> >
> > This evening I updated our Gitlab instance at invent.kde.org to the
> latest
> > version - 15.7.1.
> > The release notes for this can be found at
> > https://about.gitlab.com/releases/2022/12/22/gitlab-15-7-released/
> >
> > Of particular note for folks are the following:
> > - Gitlab CLI Tool: https://docs.gitlab.com/ee/integration/glab/
> > - Using a SSH key to sign your commits:
> > https://docs.gitlab.com/ee/user/project/repository/ssh_signed_commits/
> > - New Web IDE: https://docs.gitlab.com/ee/user/project/web_ide_beta/
> (based
> > on VS Code)
> >
> > Please note that the new VS Code based Web IDE is currently in beta, if
> you
> > experience issues with it you can disable it in your preferences:
> > https://invent.kde.org/-/profile/preferences
> >
> > Should there be any issues, please let us know.
>
> As suggested by tsdgeos, not sure if it's related to/caused by the update
> though:
>
>  11:37:28  bcooksley[m]: hi. any chance that CI pipelines are no
> longer triggered after the gitlab update?
>  11:38:20  though push does not yet show up on
> https://invent.kde.org/dashboard/activity, so perhaps some processing
> still happening?
>  11:39:34  https://invent.kde.org/sdk/kdesvn/-/commits/master
> shows the commits (4 min old), but
> https://invent.kde.org/sdk/kdesvn/activity also does not yet show the push
>  12:32:24  we also seem to be hitting some gitlab problem in
> https://invent.kde.org/network/neochat/-/merge_requests/752
>  12:33:14  This looks similar to
> https://invent.kde.org/office/kmymoney/-/merge_requests/191
>  12:34:33  I also noticed, that windows jobs take a long time to
> setup and then fail. Now it doesn't even start after re-launch
>  12:42:05  Kind of late for bcooksley[m] so i'd say answer his
> email to the community mailing list?
>

This has now been corrected, apologies for the disruption.

For those curious, Gitlab accomplishes the large part of it's processing
using a background worker, Sidekiq.
This background worker has a supervisor process, sidekiq-cluster, which
launched successfully but was unable to spawn it's worker processes due to
an issue with Bundler.

That has been fixed now and it has caught up on all the missed processing.

CI has approximately 200 jobs to work through at this time, which the nodes
are busy working on at this very moment.

Thanks,
Ben


>
> --
>
> Regards
>
> Thomas Baumgart
>
> -
> An optimist laughs to forget.
> A pessimist forgets to laugh.
> -
>

GItlab update

[Ben Cooksley]

Hi all,

This evening I updated our Gitlab instance at invent.kde.org to the latest
version - 15.7.1.
The release notes for this can be found at
https://about.gitlab.com/releases/2022/12/22/gitlab-15-7-released/

Of particular note for folks are the following:
- Gitlab CLI Tool: https://docs.gitlab.com/ee/integration/glab/
- Using a SSH key to sign your commits:
https://docs.gitlab.com/ee/user/project/repository/ssh_signed_commits/
- New Web IDE: https://docs.gitlab.com/ee/user/project/web_ide_beta/ (based
on VS Code)

Please note that the new VS Code based Web IDE is currently in beta, if you
experience issues with it you can disable it in your preferences:
https://invent.kde.org/-/profile/preferences

Should there be any issues, please let us know.

Thanks,
Ben

Bugzilla server move

[Ben Cooksley]

Hi all,

As part of efforts to ensure our systems remain up to date and secure, we
will soon need to move Bugzilla from it's existing home (based on Ubuntu
18.04) to it's new home (based on Ubuntu 22.04).

To this end, this morning I moved over the Sandbox, bugstest.kde.org, to
the new server.

This is populated with the contents of our latest backup, taken
approximately 24 hours ago. While i've done a reasonable degree of testing
it would be appreciated if developers could please test their common
workflows to ensure that none of them are broken by this move.

DNS propagation is currently underway for bugstest.kde.org - you'll know if
you have got the new server as it will have the same theme as bugs.kde.org
(while the old server has an older theme).

I'm particularly concerned that there may be breakage as Bugzilla has not
been updated in many years now - which means there is a significant risk of
bitrot having taken place. Among this is that Bugzilla is incompatible with
MySQL 8 (necessitating use of MariaDB 10.6 instead).

Please note that as the Bugzilla database is quite large it may take
several hours to move it to the new system when we move bugs.kde.org itself
so this is also a heads up that next weekend we're likely to have a
downtime period of several hours as it is moved.

Thanks,
Ben

Re: Docker CI Image Change Freeze

[Ben Cooksley]

On Tue, Dec 13, 2022 at 11:48 AM Albert Astals Cid  wrote:

> El dilluns, 12 de desembre de 2022, a les 18:28:08 (CET), Ben Cooksley va
> escriure:
> > On Mon, Dec 12, 2022 at 12:29 PM Albert Astals Cid 
> wrote:
> > > El dimarts, 29 de novembre de 2022, a les 10:15:33 (CET), Ben Cooksley
> va
> > >
> > > escriure:
> > > > Hi all,
> > > >
> > > > As an update to this, sufficient changes have been made within Craft
> > > > that
> > > > it is now possible to build Qt 5 images so i'm releasing the freeze
> for
> > > > those images.
> > > > Qt 6 remains broken, and therefore remains frozen at this time (see
> > > > https://invent.kde.org/sysadmin/ci-images/-/jobs/619808)
> > > >
> > > > As mentioned previously, I believe this to be a CMake bug given the
> lack
> > >
> > > of
> > >
> > > > change in Ninja.
> > > >
> > > > At this stage I would suggest investigation be focused on either
> > >
> > > upgrading
> > >
> > > > to a newer version of Ninja that can handle the newer version of
> CMake,
> > >
> > > or
> > >
> > > > downgrading CMake back to an older version that is compatible with
> Ninja
> > > > being built without re2c being available.
> > > >
> > > > Qt 6 CI will be globally disabled in 2 weeks time if this remains
> > >
> > > unfixed,
> > >
> > > > as dependencies move quickly and I'm not in favour of retaining
> parts of
> > > > the CI system which cannot be rebuilt.
> > >
> > > This is now done, don't be like me and waste your time trying to figure
> > > out why
> > > it works when it's not supposed to work.
> >
> > Please note that the underlying reason for Ninja failing to compile
> within
> > the CI Image build environment was never found,
>
> Are you sure about that?
>
> Isn't
> https://invent.kde.org/packaging/craft/-/commit/49670cd2772e352df64749dd59d3a0437bf09d26
> the fix?
>

Looks like it could be.

The change that I was talking about resulted in an image build a week ago -
https://invent.kde.org/sysadmin/ci-images/-/jobs/632354 - which resolved
the issue.


>
> Cheers,
>   Albert
>

Cheers,
Ben


>
> > however it was worked
> > around by using a pre-built version from Craft's cache.
> > At some point in time in the not too distant future we will begin
> building
> > Craft caches within our Docker images as well, which may expose this
> > problem again.
> >
> > For now though the immediate issue is resolved though yes.
> >
> > > Cheers,
> > >
> > >   Albert
> >
> > Cheers,
> > Ben
> >
> > > > Regards,
> > > > Ben
> > > >
> > > > On Sat, Nov 19, 2022 at 7:55 AM Ben Cooksley 
> wrote:
> > > > > Hi all,
> > > > >
> > > > > Recently Sysadmin received a series of requests to rebuild the
> Docker
> > > > > images used to support KDE CI services on invent.kde.org.
> > > > >
> > > > > Unfortunately one of these rebuilds has exposed a bug of unknown
> > > > > origin
> > > > > (as it fails on our side but by all accounts works elsewhere) where
> > >
> > > Craft
> > >
> > > > > is unable to compile Ninja (with the compilation dying due to a
> > >
> > > Makefile
> > >
> > > > > syntax error that looks like a CMake bug).
> > > > >
> > > > > The failure log can be found at
> > > > > https://invent.kde.org/sysadmin/ci-images/-/jobs/601722
> > > > >
> > > > > Subsequent to this we have also received a request to rebuild our
> > > > > Linux
> > > > > images to allow for Grantlee 5.3 to be used.
> > > > >
> > > > > Given how development is conducted within some projects that make
> > > > > heavy
> > > > > use of Grantlee, and how some of that technology is used across
> > >
> > > multiple
> > >
> > > > > platforms it would be harmful to the wider CI system and KDE
> Community
> > >
> > > to
> > >
> > > > > allow for Grantlee 5.3 to become available on any of our platforms.
> > > > >
> > > > > I'm therefore imposing a change freeze on all KDE CI Docker images
> > >
> > > until
> > >
> > > > > the issue with Craft/Ninja/CMake is resolved.
> > > > >
> > > > > Should any project have prematurely adopted a mandatory dependency
> on
> > > > > Grantlee 5.3 then as they have failed to follow the correct change
> > >
> > > process
> > >
> > > > > as documented on our wikis that change is deemed to be outside
> policy
> > >
> > > and
> > >
> > > > > should be reverted immediately.
> > > > >
> > > > > Regards,
> > > > > Ben Cooksley
> > > > > KDE Sysadmin
>
>
>
>
>

Re: Docker CI Image Change Freeze

[Ben Cooksley]

On Mon, Dec 12, 2022 at 12:29 PM Albert Astals Cid  wrote:

> El dimarts, 29 de novembre de 2022, a les 10:15:33 (CET), Ben Cooksley va
> escriure:
> > Hi all,
> >
> > As an update to this, sufficient changes have been made within Craft that
> > it is now possible to build Qt 5 images so i'm releasing the freeze for
> > those images.
> > Qt 6 remains broken, and therefore remains frozen at this time (see
> > https://invent.kde.org/sysadmin/ci-images/-/jobs/619808)
> >
> > As mentioned previously, I believe this to be a CMake bug given the lack
> of
> > change in Ninja.
> >
> > At this stage I would suggest investigation be focused on either
> upgrading
> > to a newer version of Ninja that can handle the newer version of CMake,
> or
> > downgrading CMake back to an older version that is compatible with Ninja
> > being built without re2c being available.
> >
> > Qt 6 CI will be globally disabled in 2 weeks time if this remains
> unfixed,
> > as dependencies move quickly and I'm not in favour of retaining parts of
> > the CI system which cannot be rebuilt.
>
> This is now done, don't be like me and waste your time trying to figure
> out why
> it works when it's not supposed to work.
>

Please note that the underlying reason for Ninja failing to compile within
the CI Image build environment was never found, however it was worked
around by using a pre-built version from Craft's cache.
At some point in time in the not too distant future we will begin building
Craft caches within our Docker images as well, which may expose this
problem again.

For now though the immediate issue is resolved though yes.


>
> Cheers,
>   Albert
>

Cheers,
Ben


>
> >
> > Regards,
> > Ben
> >
> > On Sat, Nov 19, 2022 at 7:55 AM Ben Cooksley  wrote:
> > > Hi all,
> > >
> > > Recently Sysadmin received a series of requests to rebuild the Docker
> > > images used to support KDE CI services on invent.kde.org.
> > >
> > > Unfortunately one of these rebuilds has exposed a bug of unknown origin
> > > (as it fails on our side but by all accounts works elsewhere) where
> Craft
> > > is unable to compile Ninja (with the compilation dying due to a
> Makefile
> > > syntax error that looks like a CMake bug).
> > >
> > > The failure log can be found at
> > > https://invent.kde.org/sysadmin/ci-images/-/jobs/601722
> > >
> > > Subsequent to this we have also received a request to rebuild our Linux
> > > images to allow for Grantlee 5.3 to be used.
> > >
> > > Given how development is conducted within some projects that make heavy
> > > use of Grantlee, and how some of that technology is used across
> multiple
> > > platforms it would be harmful to the wider CI system and KDE Community
> to
> > > allow for Grantlee 5.3 to become available on any of our platforms.
> > >
> > > I'm therefore imposing a change freeze on all KDE CI Docker images
> until
> > > the issue with Craft/Ninja/CMake is resolved.
> > >
> > > Should any project have prematurely adopted a mandatory dependency on
> > > Grantlee 5.3 then as they have failed to follow the correct change
> process
> > > as documented on our wikis that change is deemed to be outside policy
> and
> > > should be reverted immediately.
> > >
> > > Regards,
> > > Ben Cooksley
> > > KDE Sysadmin
>
>
>
>
>

Re: Docker CI Image Change Freeze

[Ben Cooksley]

Hi all,

As an update to this, sufficient changes have been made within Craft that
it is now possible to build Qt 5 images so i'm releasing the freeze for
those images.
Qt 6 remains broken, and therefore remains frozen at this time (see
https://invent.kde.org/sysadmin/ci-images/-/jobs/619808)

As mentioned previously, I believe this to be a CMake bug given the lack of
change in Ninja.

At this stage I would suggest investigation be focused on either upgrading
to a newer version of Ninja that can handle the newer version of CMake, or
downgrading CMake back to an older version that is compatible with Ninja
being built without re2c being available.

Qt 6 CI will be globally disabled in 2 weeks time if this remains unfixed,
as dependencies move quickly and I'm not in favour of retaining parts of
the CI system which cannot be rebuilt.

Regards,
Ben

On Sat, Nov 19, 2022 at 7:55 AM Ben Cooksley  wrote:

> Hi all,
>
> Recently Sysadmin received a series of requests to rebuild the Docker
> images used to support KDE CI services on invent.kde.org.
>
> Unfortunately one of these rebuilds has exposed a bug of unknown origin
> (as it fails on our side but by all accounts works elsewhere) where Craft
> is unable to compile Ninja (with the compilation dying due to a Makefile
> syntax error that looks like a CMake bug).
>
> The failure log can be found at
> https://invent.kde.org/sysadmin/ci-images/-/jobs/601722
>
> Subsequent to this we have also received a request to rebuild our Linux
> images to allow for Grantlee 5.3 to be used.
>
> Given how development is conducted within some projects that make heavy
> use of Grantlee, and how some of that technology is used across multiple
> platforms it would be harmful to the wider CI system and KDE Community to
> allow for Grantlee 5.3 to become available on any of our platforms.
>
> I'm therefore imposing a change freeze on all KDE CI Docker images until
> the issue with Craft/Ninja/CMake is resolved.
>
> Should any project have prematurely adopted a mandatory dependency on
> Grantlee 5.3 then as they have failed to follow the correct change process
> as documented on our wikis that change is deemed to be outside policy and
> should be reverted immediately.
>
> Regards,
> Ben Cooksley
> KDE Sysadmin
>

Email server update - migration from Mailman 2

[Ben Cooksley]

Hi all,

This evening I have been investigating our options surrounding Mailman 2
upgrade paths as part of an evaluation to replace our current primary mail
server (Letterbox) which runs Ubuntu 20.04 with a newer system running
22.04.

One of the consequences of this upgrade will be that we will be forced to
migrate off Mailman 2 as it is no longer available and distributed in
Ubuntu 22.04.

The most likely candidate for this is naturally Mailman 3 (an instance of
which can be found at
https://mail.python.org/archives/list/python-...@python.org/)

It appears to be a substantial improvement in all regards over Mailman 2,
and therefore I intend to upgrade to that at this stage.

Important considerations to note as part of this:
- Existing list moderator and administrator passwords will cease to work,
as Mailman 3 uses individual accounts to manage this
- Mailman 3 uses a completely different URL format, so existing list
archive links will likely be broken. It may be possible to retain static
copies of the existing Pipermail archives to mitigate the impact of this
but they won't be updated any further following the upgrade.

Further information will be announced closer to the time.

Thanks,
Ben

Docker CI Image Change Freeze

[Ben Cooksley]

Hi all,

Recently Sysadmin received a series of requests to rebuild the Docker
images used to support KDE CI services on invent.kde.org.

Unfortunately one of these rebuilds has exposed a bug of unknown origin (as
it fails on our side but by all accounts works elsewhere) where Craft is
unable to compile Ninja (with the compilation dying due to a Makefile
syntax error that looks like a CMake bug).

The failure log can be found at
https://invent.kde.org/sysadmin/ci-images/-/jobs/601722

Subsequent to this we have also received a request to rebuild our Linux
images to allow for Grantlee 5.3 to be used.

Given how development is conducted within some projects that make heavy use
of Grantlee, and how some of that technology is used across multiple
platforms it would be harmful to the wider CI system and KDE Community to
allow for Grantlee 5.3 to become available on any of our platforms.

I'm therefore imposing a change freeze on all KDE CI Docker images until
the issue with Craft/Ninja/CMake is resolved.

Should any project have prematurely adopted a mandatory dependency on
Grantlee 5.3 then as they have failed to follow the correct change process
as documented on our wikis that change is deemed to be outside policy and
should be reverted immediately.

Regards,
Ben Cooksley
KDE Sysadmin

Re: Plasma Welcome Center on KDEReview

[Ben Cooksley]

On Tue, Oct 25, 2022 at 4:30 AM Nate Graham  wrote:

> Since it seems there are no more objections, I'd like to consider
> plasma-welcome to have graduated form kdereview.
>
> Sysadmins, can we get it moved out of Playground and into Plasma?
>

This was already done by Jonathan about 24 hours or so ago.
(The placement of projects in playground / KDE Review / etc is just a
single line in a YAML file which anyone can commit to)


>
> Nate
>

Cheers,
Ben


>
>
> On 10/21/22 12:19, Nate Graham wrote:
> > Any more concerns or objections?
> >
> > Nate
> >
> >
> >
> > On 10/19/22 11:56, Nate Graham wrote:
> >>
> >>
> >> On 10/19/22 10:03, Albert Astals Cid wrote:
> >>> El dimecres, 19 d’octubre de 2022, a les 5:22:09 (CEST), Nate Graham va
> >>> escriure:
>  Ping; if there are no objections I'd consider it approved.
> >>>
> >>> I can't still see the toolbar :/
> >>
> >> Thanks to Albert's
> >> https://invent.kde.org/plasma/plasma-welcome/-/merge_requests/19, this
> >> is working now.
> >>
> >> Nate
>

Re: Gitlab update, 2FA now mandatory

[Ben Cooksley]

On Mon, Oct 24, 2022 at 11:56 PM Raghavendra Kamath 
wrote:

> On Sunday, 23 October, 2022 12:02:23 PM IST Ben Cooksley wrote:
> > I
> > have also enabled Mandatory 2FA, which Gitlab will ask you to configure
> > next time you access it.
>
> Is the 2FA in KDE identity website same as this. The KDE identity shows a
> grid
> based system where you combine the grid and your password for 2FA.
>
> I have also already enabled 2FA for KDE identity with totp, does this
> supersede it?
>

Gitlab will be replacing KDE Identity for authentication, so this 2FA setup
supersedes that yes.

Cheers,
Ben


>
>
> --
> Raghavendra Kamath
> emblik.studio
>
>
>

Re: Gitlab update, 2FA now mandatory

[Ben Cooksley]

On Mon, Oct 24, 2022 at 12:37 PM Kevin Kofler 
wrote:

> Ben Cooksley wrote:
> > On Mon, Oct 24, 2022 at 3:36 AM Kevin Kofler 
> > wrote:
> >> IMHO, this is both an absolutely unacceptable barrier to entry and a
> >> constant annoyance each time one has to log in.
> >
> > You shouldn't have any issues with remaining logged in as long as your
> > browser remains open.
>
> I wrote "each time one has to log in", not "remaining logged in".
>
> I sure hope that I just have to jump through the 2FA hoops only once per
> log
> in and not several times. But that is still one time too many.
>
> And "as long as your browser remains open" is at most one day. I turn the
> computer off while I sleep. So if this change forces me to log in each
> time
> I restart the browser, and hence at least each time I restart the computer
> (which is currently *not* the case, I can remain logged in for days
> throughout hundreds of browser sessions), that would mean going through
> the
> 2FA procedure at least every day.
>

The 2FA prompt (for normal users) is only applied on login yes.
Note that I can't examine your experience exactly as admins get prompted to
reauthenticate more frequently, especially when undertaking sensitive
actions.

See https://gitlab.com/gitlab-org/gitlab/-/issues/16656 for more
surrounding 2FA on each login.

With respect to logins being remembered, I have just performed a test using
a vanilla version of Firefox as shipped by OpenSUSE.
Logging into invent.kde.org (with the "Remember me" box ticked), completing
2FA authentication, performing a few actions and then closing the browser
followed by reopening it a few moments later led to the result I expected -
that I was still logged into Gitlab.


> > I did not supply a list of applications that people should be using as
> > there is a diverse range of devices and appstore ecosystems in use by
> > different people, and I don't have access to hardware such as a PinePhone
> > to validate any of that.
>
> So you are single-handedly forcing a new requirement on everyone, but are
> not willing to help us in any way with it, even just by telling us how to
> fulfill it. That is very unhelpful.
>

I could have provided links to a few applications.
They wouldn't have suited everyone though, so I opted not to do so on the
basis that there are dozens of apps that support handling TOTP.


>
> And you conveniently evaded my main questions:
> * why such a change can be decided by one person suddenly on a Sunday
> morning, with no warning (well, the software "gracefully" gives us 2 days
> to
> comply… only two days!), let alone (transparent) discussion.
>

As mentioned in my initial email - securing us against suspicious activity
that has been detected.
This is also why there was no discussion in advance.

One of the responsibilities that Sysadmin is charged with is ensuring our
data is protected and kept safe.
That is exactly what I am doing - using industry standard best practices.


> * what the point of two-factor is at all considering that you have no way
> to
> prevent the developer from storing the password and the OTP generator on
> the
> same device.
>

** Caution - a strawman argument has been detected **

The point of 2FA is to prevent stolen credentials from being misused by an
attacker.
If your device is compromised, 2FA isn't going to stop anything because
they can just wait (or otherwise prompt) for you to login to the site and
steal your session to do whatever it is they want to do.


>
> In short, the 2FA requirement is unacceptable and needs to be disabled
> immediately.
>

On that we disagree fundamentally.

Regards,
Ben


>
> Kevin Kofler
>
> PS/OT:
>
> > For most people the set of addresses they will be logging in from won't
> > change much (given that the vast majority of people use always-on
> internet
> > connections now, which means IP addresses - even if theoretically dynamic
> > - are in practice fairly static).
>
> "fairly static" does not mean it never changes, as in my case. But we need
> not discuss this tangent any further. The mandatory 2FA nonsense is the
> real
> issue, let us please focus on that.
>

Re: Gitlab update, 2FA now mandatory

[Ben Cooksley]

On Mon, Oct 24, 2022 at 3:36 AM Kevin Kofler  wrote:

> Hi,
>

Hi Kevin,


>
> Ben Cooksley wrote:
> > As part of securing Invent against recently detected suspicious activity
>
> What kind of suspicious activity would that be? Yesterday, Invent even
> considered it "suspicious" enough to send a warning e-mail that my semi-
> static IP address (TV-cable broadband ISP) has changed after several
> months.
> Dynamic IP addresses are not exactly unusual.
>

It was likely just flagging that you were logging in from a different IP
address to your usual address.
For most people the set of addresses they will be logging in from won't
change much (given that the vast majority of people use always-on internet
connections now, which means IP addresses - even if theoretically dynamic -
are in practice fairly static).

The suspicious activity is not related to static/dynamic IP addresses, and
as it is an ongoing matter i'd prefer not to comment until it is
satisfactorily resolved.


>
> > I have also enabled Mandatory 2FA, which Gitlab will ask you to configure
> > next time you access it.
>
> IMHO, this is both an absolutely unacceptable barrier to entry and a
> constant annoyance each time one has to log in.
>

You shouldn't have any issues with remaining logged in as long as your
browser remains open.

If this is not the behaviour you are seeing then please check the browser
addons/extensions you are using as these can often break functionality in
unexpected ways.
This is especially when they claim to offer benefits relating to privacy or
security (the EFF's HTTPS Everywhere extension several years back broke
links for some KDE sites by completely changing the subdomain)


>
> > This can be done using either a Webauthn token (such as a Yubikey) or
> TOTP
> > (using the app of choice on your phone)
>
> What am I expected to use with my PinePhone? Does
> https://apps.kde.org/keysmith/ work?
>

Please see the other responses to this thread.

I did not supply a list of applications that people should be using as
there is a diverse range of devices and appstore ecosystems in use by
different people, and I don't have access to hardware such as a PinePhone
to validate any of that.


>
> And how do you intend to prevent users from running the TOTP app on the
> same
> device as the web browser (both on the smartphone or even both on the
> desktop/notebook)? You just cannot. (As far as I know, even Yubikeys can
> be
> emulated in software.) Two-factor is a farce.


> Kevin Kofler
>

Regards,
Ben

Gitlab update, 2FA now mandatory

[Ben Cooksley]

Hi all,

This afternoon I updated invent.kde.org to the latest version of Gitlab,
15.5.
Release notes for this can be found at
https://about.gitlab.com/releases/2022/10/22/gitlab-15-5-released/

There isn't much notable feature wise in this release, however there have
been some bug fixes surrounding the "Rebase without Pipeline"
functionality that was introduced in an earlier update.

As part of securing Invent against recently detected suspicious activity I
have also enabled Mandatory 2FA, which Gitlab will ask you to configure
next time you access it. This can be done using either a Webauthn token
(such as a Yubikey) or TOTP (using the app of choice on your phone)

Should you lose access to your 2FA device you can obtain a recovery token
to log back in via SSH, see
https://docs.gitlab.com/ee/user/profile/account/two_factor_authentication.html#generate-new-recovery-codes-using-ssh
for more details on this.

Please let us know if there are any queries on the above.

Thanks,
Ben

Re: Plasma Welcome Center on KDEReview

[Ben Cooksley]

On Tue, Sep 20, 2022 at 12:19 PM Nate Graham  wrote:

> On 9/18/22 11:03, Harald Sitter wrote:
> > Not all code is licensing policy compliant (e.g. appdata is missing
> > spdx tags). I'd suggest enabling the reuse linter pipeline.
>
> How do I do this? Is there an example I can copy from elsewhere (which
> in case people haven't noticed, is basically how I do everything)?
>

Please see
https://invent.kde.org/sysadmin/ci-utilities/-/tree/master/gitlab-templates
for a list of available templates for use on invent.kde.org.

Cheers,
Ben


>
> Nate
>

Re: Gitlab CI Dashboards and retirement of build.kde.org

[Ben Cooksley]

On Sun, Sep 4, 2022 at 8:51 PM Gilles Caulier 
wrote:

> Hi Ben,
>

HI Gilles,


>
> With build/binary-factory , it was possible to get an Embeddable Build
> Status Icon as this one :
>
>
> https://binary-factory.kde.org/view/AppImage/job/Digikam_Nightly_appimage-centos7/badge/
>
> Does this feature still exist with Gitlab infrastructure ?
>

Yes, quoting my earlier email:

[quote]
Gitlab provides a limited selection of badges - which can be found at:
- https://invent.kde.org/multimedia/kdenlive/badges/master/pipeline.svg
- https://invent.kde.org/multimedia/kdenlive/badges/master/coverage.svg
- https://invent.kde.org/multimedia/kdenlive/-/badges/release.svg
[/quote]

You'll need to swap multimedia/kdenlive to graphics/digikam but otherwise
that should work fine.

Please note that the Binary Factory is not impacted by this, so anything
relating to the Binary Factory is unchanged at this time.


>
> Thanks
>
> Gilles
>

Regards,
Ben


>
> Le sam. 27 août 2022 à 11:45, Ben Cooksley  a écrit :
> >
> > Hi all,
> >
> > This evening I completed the necessary setup required to complete our
> Gitlab CI dashboards, which can now be found at
> https://metrics.kde.org/dashboards/f/aNxvXJW4k/gitlab-ci (KDE Developer
> account login required)
> >
> > These allow any developer to get a view on the current CI status of
> projects and groups of projects on a branch and platform basis - and should
> hopefully provide useful insight into the overall status that can currently
> be obtained from Jenkins.
> >
> > As this was the last thing holding us back from shutting down
> build.kde.org, i'd like to proceed with retiring and shutting down
> build.kde.org as soon as possible so the capacity it occupies can be
> released and reallocated to Gitlab.
> >
> > If anyone would like to experiment with customised views for their own
> purposes (where the above provided ones are insufficient) please file a
> Sysadmin ticket.
> >
> > Please let me know if there are any questions on the above.
> >
> > Thanks,
> > Ben
>

Notice of impending change to Gitlab CI

[Ben Cooksley]

Hi all,

Currently our Gitlab CI jobs for Linux (SUSE) and Android (Ubuntu) run
their respective jobs as root within the Docker containers that Gitlab
spawns for them.

This is a restriction that was previously required by the simultaneous use
of these same images by Jenkins, which following the shutdown of
build.kde.org this weekend is no longer a problem.

I will look into making this adjustment to our CI image(s) in the coming
week for Linux (SUSE). Once implemented, tests which made use of root
privileges may begin to fail (and those that were incompatible with it will
begin to pass).

Android Qt 5 builds still share the image with the Binary Factory and
therefore will have to continue to run as root at this time until we are
able to discontinue the BInary Factory (or at least, Android's use of it).

Thanks,
Ben

Re: Gitlab CI Dashboards and retirement of build.kde.org

[Ben Cooksley]

On Sun, Sep 4, 2022 at 7:54 AM Johnny Jazeix  wrote:

>
>
> Le sam. 3 sept. 2022 à 21:28, Ben Cooksley  a écrit :
>
>> On Sat, Sep 3, 2022 at 9:29 PM Gleb Popov <6year...@gmail.com> wrote:
>>
>>> On Sat, Sep 3, 2022 at 7:46 AM Ben Cooksley  wrote:
>>> >
>>> > As previously indicated, I have now shutdown build.kde.org along with
>>> the domain that supported it's version of the CI tooling.
>>> > The repository containing that tooling has now also been archived, and
>>> the former build.kde.org domain has been redirected to metrics.kde.org.
>>> >
>>> > The server which was acting as a builder for build.kde.org will be
>>> rebuilt in the coming days and reallocated to support Gitlab CI workloads.
>>> >
>>> > Thanks,
>>> > Ben
>>>
>>> What should be used instead of binary-factory? How do I transform this
>>> link?
>>>
>>>
>>> https://binary-factory.kde.org/view/Windows%2064-bit/job/Kate_Release_win64/1762/artifact/kate-22.08.0-1762-windows-msvc2019_64-cl.exe
>>
>>
>> At this time the Binary Factory is not impacted by this.
>>
>> Regards,
>> Ben
>>
>
> Hi,
>
> I think the issue mentioned by Glen is that this link (and all other
> artifacts from binary-factory) is redirected to
> https://build-artifacts.kde.org/binary-factory/Kate_Release_win64/1762/kate-22.08.0-1762-windows-msvc2019_64-cl.exe
> which does not exist.
>

Oops. That is an oversight on my part - apologies - and has now been
corrected (although the URLs will have changed)

Cheers,
Ben


>
>
> Cheers,
> Johnny
>

Re: Gitlab CI Dashboards and retirement of build.kde.org

[Ben Cooksley]

On Sun, Sep 4, 2022 at 2:13 AM Michael Reeves  wrote:

> I now have no way to even test macosx builds for kdiff3, I have no access
> to a 64bit Intel mac. What are the plans for this and Windows
> builds. I have a functional windows based craft installed locally.
>

At this time the Binary Factory is unaffected by these changes, however
steps will be made in the coming weeks/months to migrate away from the
Binary Factory to equivalent Gitlab jobs (although they won't be available
for Merge Requests due to various technical limitations)

Regards,
Ben


>
>
> Sep 3, 2022 12:47:06 AM Ben Cooksley :
>
> On Sat, Aug 27, 2022 at 9:44 PM Ben Cooksley  wrote:
>
>> Hi all,
>>
>> This evening I completed the necessary setup required to complete our
>> Gitlab CI dashboards, which can now be found at
>> https://metrics.kde.org/dashboards/f/aNxvXJW4k/gitlab-ci (KDE Developer
>> account login required)
>>
>> These allow any developer to get a view on the current CI status of
>> projects and groups of projects on a branch and platform basis - and should
>> hopefully provide useful insight into the overall status that can currently
>> be obtained from Jenkins.
>>
>> As this was the last thing holding us back from shutting down
>> build.kde.org, i'd like to proceed with retiring and shutting down
>> build.kde.org as soon as possible so the capacity it occupies can be
>> released and reallocated to Gitlab.
>>
>
> As previously indicated, I have now shutdown build.kde.org along with the
> domain that supported it's version of the CI tooling.
> The repository containing that tooling has now also been archived, and the
> former build.kde.org domain has been redirected to metrics.kde.org.
>
> The server which was acting as a builder for build.kde.org will be
> rebuilt in the coming days and reallocated to support Gitlab CI workloads.
>
>
>>
>> If anyone would like to experiment with customised views for their own
>> purposes (where the above provided ones are insufficient) please file a
>> Sysadmin ticket.
>>
>> Please let me know if there are any questions on the above.
>>
>> Thanks,
>> Ben
>>
>
> Thanks,
> Ben
>
>

Re: Gitlab CI Dashboards and retirement of build.kde.org

[Ben Cooksley]

On Sat, Sep 3, 2022 at 9:29 PM Gleb Popov <6year...@gmail.com> wrote:

> On Sat, Sep 3, 2022 at 7:46 AM Ben Cooksley  wrote:
> >
> > As previously indicated, I have now shutdown build.kde.org along with
> the domain that supported it's version of the CI tooling.
> > The repository containing that tooling has now also been archived, and
> the former build.kde.org domain has been redirected to metrics.kde.org.
> >
> > The server which was acting as a builder for build.kde.org will be
> rebuilt in the coming days and reallocated to support Gitlab CI workloads.
> >
> > Thanks,
> > Ben
>
> What should be used instead of binary-factory? How do I transform this
> link?
>
>
> https://binary-factory.kde.org/view/Windows%2064-bit/job/Kate_Release_win64/1762/artifact/kate-22.08.0-1762-windows-msvc2019_64-cl.exe


At this time the Binary Factory is not impacted by this.

Regards,
Ben

Re: Gitlab CI Dashboards and retirement of build.kde.org

[Ben Cooksley]

On Sat, Aug 27, 2022 at 9:44 PM Ben Cooksley  wrote:

> Hi all,
>
> This evening I completed the necessary setup required to complete our
> Gitlab CI dashboards, which can now be found at
> https://metrics.kde.org/dashboards/f/aNxvXJW4k/gitlab-ci (KDE Developer
> account login required)
>
> These allow any developer to get a view on the current CI status of
> projects and groups of projects on a branch and platform basis - and should
> hopefully provide useful insight into the overall status that can currently
> be obtained from Jenkins.
>
> As this was the last thing holding us back from shutting down
> build.kde.org, i'd like to proceed with retiring and shutting down
> build.kde.org as soon as possible so the capacity it occupies can be
> released and reallocated to Gitlab.
>

As previously indicated, I have now shutdown build.kde.org along with the
domain that supported it's version of the CI tooling.
The repository containing that tooling has now also been archived, and the
former build.kde.org domain has been redirected to metrics.kde.org.

The server which was acting as a builder for build.kde.org will be rebuilt
in the coming days and reallocated to support Gitlab CI workloads.


>
> If anyone would like to experiment with customised views for their own
> purposes (where the above provided ones are insufficient) please file a
> Sysadmin ticket.
>
> Please let me know if there are any questions on the above.
>
> Thanks,
> Ben
>

Thanks,
Ben

Re: Git commit content

[Ben Cooksley]

On Mon, Aug 29, 2022 at 6:24 PM Helio Chissini de Castro 
wrote:

> Honest question
>

Hey Helio,


>
> Since these files are generated, and create huge hurdles to commit, was
> not make sense to generate this on build time and avoid such huge commits ?
>
> Or we have a technical block that maybe we can help to solve ?
>

These files have usually been the result of generation processes in either
our own repositories (develop.kde.org), or from other external datasets
(kstars).
As such they're not really suitable to generate at build time.


>
>
> []'s
>
>
Cheers,
Ben


> On Sun, Aug 28, 2022 at 1:17 PM Ben Cooksley  wrote:
>
>> Hi all,
>>
>> Over the past couple of months we have had several incidents where people
>> have needed/attempted to push large text files into KDE Git repositories.
>>
>> While this does not seem immediately problematic, it is something that
>> unfortunately the email sending components of our Git hooks are unable to
>> handle (likely due to them embedding commit diffs into the body of emails).
>>
>> This results in them consuming an entire CPU core of the server until
>> Gitlab times out on them and kills them (failing the push/merge in the
>> process).
>>
>> For the most part these have been programmatically generated data
>> sources, making the diffs of little use.
>>
>> Where possible it is recommended not to commit these sort of artifacts to
>> KDE Git repositories, but where it is not avoidable please ensure that:
>> a) The data is not all on a single line (JSON pretty print where
>> possible);
>> b) That the file has been flagged as binary data using .gitattributes
>>
>> This should assist the email generating components of the hooks in more
>> easily handling the content.
>>
>> Thanks,
>> Ben
>>
>

Git commit content

[Ben Cooksley]

Hi all,

Over the past couple of months we have had several incidents where people
have needed/attempted to push large text files into KDE Git repositories.

While this does not seem immediately problematic, it is something that
unfortunately the email sending components of our Git hooks are unable to
handle (likely due to them embedding commit diffs into the body of emails).

This results in them consuming an entire CPU core of the server until
Gitlab times out on them and kills them (failing the push/merge in the
process).

For the most part these have been programmatically generated data sources,
making the diffs of little use.

Where possible it is recommended not to commit these sort of artifacts to
KDE Git repositories, but where it is not avoidable please ensure that:
a) The data is not all on a single line (JSON pretty print where possible);
b) That the file has been flagged as binary data using .gitattributes

This should assist the email generating components of the hooks in more
easily handling the content.

Thanks,
Ben

Re: Gitlab CI Dashboards and retirement of build.kde.org

[Ben Cooksley]

On Sun, Aug 28, 2022 at 4:40 AM Albert Astals Cid  wrote:

> El dissabte, 27 d’agost de 2022, a les 11:44:47 (CEST), Ben Cooksley va
> escriure:
> > Hi all,
> >
> > This evening I completed the necessary setup required to complete our
> > Gitlab CI dashboards, which can now be found at
> > https://metrics.kde.org/dashboards/f/aNxvXJW4k/gitlab-ci (KDE Developer
> > account login required)
> >
> > These allow any developer to get a view on the current CI status of
> > projects and groups of projects on a branch and platform basis - and
> should
> > hopefully provide useful insight into the overall status that can
> currently
> > be obtained from Jenkins.
> >
> > As this was the last thing holding us back from shutting down
> build.kde.org,
> > i'd like to proceed with retiring and shutting down build.kde.org as
> soon
> > as possible so the capacity it occupies can be released and reallocated
> to
> > Gitlab.
> >
> > If anyone would like to experiment with customised views for their own
> > purposes (where the above provided ones are insufficient) please file a
> > Sysadmin ticket.
> >
> > Please let me know if there are any questions on the above.
>
> Looks great.
>

Yay!


>
> One thing that i'm not sure i understand correctly, currently in the
> General
> Overview, it says that there are 3 projects currently failing kwin,
> kpackage
> and kphotoalbum, but then if i go to the Per Platform View i get that
> rkward
> is failing on windows. Shouldn't rkward also be listed as failing on the
> general overview?
>

That is a rather curious bug, caused by the fact it was looking at things
on a Pipeline vs. Job basis.

The query you were looking at was looking at the list of most recent
pipeline runs on a per project basis, which in the case of rkward means the
last push by scripty - which was skipped (so not a failure).
I've tweaked the query to look at things on a per job basis now, which
skips over that issue.


>
> Cheers,
>   Albert
>

Cheers,
Ben


>
> >
> > Thanks,
> > Ben
>
>
>
>
>

Gitlab CI Dashboards and retirement of build.kde.org

[Ben Cooksley]

Hi all,

This evening I completed the necessary setup required to complete our
Gitlab CI dashboards, which can now be found at
https://metrics.kde.org/dashboards/f/aNxvXJW4k/gitlab-ci (KDE Developer
account login required)

These allow any developer to get a view on the current CI status of
projects and groups of projects on a branch and platform basis - and should
hopefully provide useful insight into the overall status that can currently
be obtained from Jenkins.

As this was the last thing holding us back from shutting down build.kde.org,
i'd like to proceed with retiring and shutting down build.kde.org as soon
as possible so the capacity it occupies can be released and reallocated to
Gitlab.

If anyone would like to experiment with customised views for their own
purposes (where the above provided ones are insufficient) please file a
Sysadmin ticket.

Please let me know if there are any questions on the above.

Thanks,
Ben

Re: Move Licentia to KDEREVIEW

[Ben Cooksley]

On Mon, Jul 11, 2022 at 9:10 PM David Redondo  wrote:

> Am Freitag, 8. Juli 2022, 07:36:57 CEST schrieb Felipe Kinoshita:
> > Hey, I'd like to move Licentia  to
> > KDEREVIEW.
> >
> > Licentia is a simple app targeted at developers/creators who need to
> decide
> > which license is suited to their projects, Licentia helps with that by
> > listing a bunch of licenses, which with it's permissions, conditions and
> > limitations, instructions about how to add a license to your project and
> a
> > list of known projects that use said license.
> >
> > Thanks,
> > Felipe.
>
> I am not entirely happy about the Instruction to use a license being 'put
> a
> LICENSE file in the root directory' when it's a pattern we are going away
> from
> and towards reuse compliance.
> Maybe display the SPDX tag instead?
>

Please note that Gitlab and other places still do rely on LICENSE (or
COPYING) files being present (and I guarantee there is distribution tooling
out there that does the same too) so I wouldn't be too quick to shoot down
LICENSE files in the root of repositories.


>
> David
>
>
>
Cheers,
Ben

[sdk/ikona] data: Remove direct link to invent.kde.org.

[Ben Cooksley]

Git commit ebb24a7118581ff27dbf66c3dc32da3bd6645d30 by Ben Cooksley.
Committed on 18/06/2022 at 09:11.
Pushed by bcooksley into branch 'master'.

Remove direct link to invent.kde.org.

Linking to raw files on invent.kde.org is *strictly* and *absolutely* 
prohibited in the strongest possible terms and is something that Sysadmin does 
not permit in any form.
This is particularly the case when those files will be fetched by end user 
systems, as it creates the risk of an unintentional Distributed Denial of 
Service attack on invent.kde.org.

CCMAIL: sysad...@kde.org
CCMAIL: kde-core-devel@kde.org
CCMAIL: community...@kde.org

M  +0-1data/org.kde.Ikona.appdata.xml

https://invent.kde.org/sdk/ikona/commit/ebb24a7118581ff27dbf66c3dc32da3bd6645d30

diff --git a/data/org.kde.Ikona.appdata.xml b/data/org.kde.Ikona.appdata.xml
index c214f59..fe98e6c 100644
--- a/data/org.kde.Ikona.appdata.xml
+++ b/data/org.kde.Ikona.appdata.xml
@@ -2,7 +2,6 @@
 
 
   org.kde.Ikona
-  https://invent.kde.org/sdk/ikona/raw/master/data/org.kde.Ikona.svg
   Ikona
   أيقونة
   Ikona

Issues with mail delivery

[Ben Cooksley]

Hi all,

It has been brought to our attention that for the past several days our
systems have been unable to deliver email to a number of people due to a
blacklisting issue with Spamhaus.

The issue commenced approximately May 11 13:00 UTC, and has only just been
resolved now. Unfortunately all affected email has been permanently lost.

This issue affected people whose providers advertise use of IPv6 and make
use of Spamhaus for spam filtering.

This issue occurred due to our service provider (Bytemark, now IOMart)
allocating out addresses from IPv6 netblocks individually. This breaks an
assumption made by Spamhaus that each customer is allocated their own /64
IPv6 netblock and led to them blacklisting our system for the bad behaviour
of another customer.

A complaint concerning this has been filed with Bytemark/IOMart.

As an interim solution I have now reconfigured Letterbox to prefer IPv4 for
delivery of email which side steps the issue, at the cost of not making use
of IPv6.

This outage affected all KDE.org hosted mailing lists, as well as @kde.org
and @kdemail.net aliases.

Apologies for the inconvenience.

Regards,
Ben Cooksley
KDE Sysadmin

Re: Submitting KTextTemplate for KF6

[Ben Cooksley]

On Sun, Mar 6, 2022 at 6:00 AM Stephen Kelly  wrote:

>
> On 26/02/2022 10:38, Volker Krause wrote:
> > On Sonntag, 20. Februar 2022 16:02:59 CET Stephen Kelly wrote:
> >> Hello,
> >>
> >> The Qt 5 based Grantlee libraries were depended on by some KDE
> applications.
> >>
> >> For Qt 6, I've created a separate repo for KTextTemplate for one of the
> >> Grantlee libraries. The other library is separate and can be dealt with
> >> separately.
> >>
> >> Currently the code lives here:
> >>
> >> https://github.com/steveire/ktexttemplate
> >>
> >> I'd like to get the process moving on getting this reviewed and into
> KF6.
> >>
> >> It's not clear to me what the next step is. I tried googling terms like
> >> "how to submit a library to kde frameworks", but didn't find anything
> >> useful.
> >>
> >> https://techbase.kde.org/KDE_Frameworks describes how to build
> >> frameworks, but not how to submit a new one.
> >>
> >> I've checked and I can still use invent.kde.org, so it would be good if
> >> there is a standard process I can follow for the rest.
> > I don't think the process of integrating a so far external project into
> > Frameworks happened often enough for this to be documented as a
> streamlined
> > process. I'd suggest the following steps:
> >
> > (1) Import the repo into invent.kde.org
> > (2) Go through the KDE Review process
> > (3) Request inclusion into Frameworks
> >
> > Happy to see this moving :)
>
>
> Ok. I was thinking there might be some 'review' area of invent.k.o, but
> I've just made
>
> https://invent.kde.org/skelly/ktexttemplate
>
> Does that satisfy (1)?
>

It starts the process yes.


>
> I think there are changes like CamelCase headers to conform to KF6
> norms. I'm not sure if others can help with that now. Can any KDE
> account push to the above repo?
>

No, only you are able to push to that repository, but people can send you
merge requests.

To give people access to the repository you can either:
a) Invite the teams/kde-developers group to the repository as a 'Developer'
which will grant access to it; or
b) Ask Sysadmin to move the repository into libraries/ which is where it
would go prior to becoming a Framework (which it can only do when it
completes KDE Review)


> Thanks,
>
> Stephen.
>
>
Cheers,
Ben

Re: KDiff3 Missing translation in craft

[Ben Cooksley]

On Sun, Jan 23, 2022 at 6:08 AM Michael Reeves  wrote:

> What would be needed to build from a tar ball on Windows. I have a craft
> setup locally but it likes to use git.
>

That is governed by your Craft Blueprint.
See
https://invent.kde.org/packaging/craft-blueprints-kde/-/blob/master/kde/frameworks/version.ini
for an example of how to declare versions that can come from tarballs
instead of Git branches/tags.

Cheers,
Ben


>
>
> Jan 21, 2022 2:10:19 PM Ben Cooksley :
>
> On Sat, Jan 22, 2022 at 7:52 AM Michael Reeves 
> wrote:
>
>> This is being cross posted because I am not subscribed to kde-window and
>> replies have not shown up in mail archives when I contacted that list for
>> other issues. The bug in question is here:
>>
>> https://bugs.kde.org/show_bug.cgi?id=442607
>>
>> As far as I can see the setup for kdiff3 is correct but does not pull
>> translations.
>>
>
> Hi Michael,
>
> To my understanding Craft does not include explicit support for fetching
> translations if you are building directly from the Git repositories.
> Instead it relies on the translations being bundled in the
> repository/archive it is trying to build.
>
> For release builds which are done from tarballs this should work fine,
> however it does mean nightlies from the Binary Factory will not have
> translations included.
>
> The partial translations you are seeing will be because Craft will be
> using release versions of Frameworks - which will therefore have
> translations included in them.
>
> The fix for this is for scripty to synchronise copies of the translations
> into our Git repositories - something which will ensure a number of other
> translation related items work more smoothly as well.
> This is something which I believe is being worked on and tested in a small
> handful of repositories currently.
>
> Cheers,
> Ben
>
>

Re: KDiff3 Missing translation in craft

[Ben Cooksley]

On Sat, Jan 22, 2022 at 7:52 AM Michael Reeves  wrote:

> This is being cross posted because I am not subscribed to kde-window and
> replies have not shown up in mail archives when I contacted that list for
> other issues. The bug in question is here:
>
> https://bugs.kde.org/show_bug.cgi?id=442607
>
> As far as I can see the setup for kdiff3 is correct but does not pull
> translations.
>

Hi Michael,

To my understanding Craft does not include explicit support for fetching
translations if you are building directly from the Git repositories.
Instead it relies on the translations being bundled in the
repository/archive it is trying to build.

For release builds which are done from tarballs this should work fine,
however it does mean nightlies from the Binary Factory will not have
translations included.

The partial translations you are seeing will be because Craft will be using
release versions of Frameworks - which will therefore have translations
included in them.

The fix for this is for scripty to synchronise copies of the translations
into our Git repositories - something which will ensure a number of other
translation related items work more smoothly as well.
This is something which I believe is being worked on and tested in a small
handful of repositories currently.

Cheers,
Ben

Re: Gitlab Bugzilla integration

[Ben Cooksley]

On Thu, Jan 13, 2022 at 5:19 AM Sandro Knauß  wrote:

> Hey,
>

Hi there,


> Great, one missing feature less!
>
> > This functionality can be seen in action on
> >
> https://invent.kde.org/graphics/digikam/-/commit/7ca7bf03b11ae72842006af8a66
> > ed02894a4a605 (note that content posted prior to this being enabled may
> not
> > feature links due to caching in Gitlab)
>
> The commit using CCBUGS, that is not specified according to. So the best
> is
> either update the documentation or make the Gitlab more strict. Or even
> better, if Gitlab could warn the user about wrong commit metadata.
>

We are using the same regexp that the hooks accept to ensure compatibility
- if the commit hooks fire on it then Gitlab should link it as well.


> https://community.kde.org/Policies/
> Commit_Policy#Special_keywords_in_GIT_and_SVN_log_messages
>
> > Use of this functionality requires the setup of some metadata within
> > sysadmin/repo-metadata which a number of projects have already completed.
> > Once added it will be synced to Gitlab overnight and will become
> available
> > the next day.
>
> What metadata is needed within the repo, the bugzilla information in the
> projects metadata?
>

The "bugzilla:" section is required, with a product subkey being the name
of the product on Bugzilla.
If a specific component should be targeted then a key "component" may also
be added.


> Regards
>
> hefee
>
>
>
Cheers,
Ben

Re: Incoming availability of Windows CI for Gitlab and deprecation of build.kde.org

[Ben Cooksley]

On Mon, Jan 10, 2022 at 3:53 AM Michael Reeves  wrote:

> What is the status for MacOS builds?
>

No work is being made on those at this time, in large part because work is
focused on getting us off Jenkins currently.

Regards,
Ben


>
>
> Jan 9, 2022 4:34:38 AM Ben Cooksley :
>
> Hi all,
>
> Over the past week substantial work has meant that we now have a
> successfully compiling seed job for Frameworks on Windows under Gitlab.
>
> This is a significant step on the road to announcing general availability
> for Windows CI, which I expect projects that use Frameworks only should be
> able to begin enabling in the coming week (there are a few finishing
> touches i'd like to make first)
>
> If your project makes use of Windows CI currently (or would like to begin
> doing so) and has non-Frameworks dependencies it would be appreciated if
> you could please reply with a list of those KDE projects you use so we can
> keep them in mind while working on the seed jobs for Independent Releases
> and Release Service.
>
> Following this we will be able to make Windows CI generally available on
> Gitlab for projects to adopt as needed, and as this represents the
> migration of the final platform over to Gitlab this will also come with an
> EOL announcement for build.kde.org and it's corresponding infrastructure.
>
> If your project has not yet migrated to Gitlab native CI for any platform
> then you should do so *immediately* and without delay. Projects making use
> of legacy jobs on Gitlab (which will have references to sysadmin/ci-tooling
> instead of sysadmin/ci-utilities and will not have a .kde-ci.yml file) are
> not using Gitlab native CI and need to take action.
>
> If you have tooling which interacts with resources provided by
> build.kde.org (including build-artifacts.kde.org and the repository
> sysadmin/ci-tooling) then please contact Sysadmin as soon as possible so we
> can discuss what the best steps are for you to minimize disruption. My
> understanding is that this includes apps.kde.org as well as the Android
> SDK Docker image.
>
> Following the completion of this the dependency metadata located in
> sysadmin/repo-metadata should be considered unmaintained and no longer in
> use. We will remove this information in due course to ensure only a single
> source of truth (being the .kde-ci.yml files now included in repositories).
> It is expected that this will require updates to kdesrc-build.
>
> Should anyone have any questions on the above please let us know.
>
> Thanks,
> Ben Cooksley
> KDE Sysadmin
>
>

Incoming availability of Windows CI for Gitlab and deprecation of build.kde.org

[Ben Cooksley]

Hi all,

Over the past week substantial work has meant that we now have a
successfully compiling seed job for Frameworks on Windows under Gitlab.

This is a significant step on the road to announcing general availability
for Windows CI, which I expect projects that use Frameworks only should be
able to begin enabling in the coming week (there are a few finishing
touches i'd like to make first)

If your project makes use of Windows CI currently (or would like to begin
doing so) and has non-Frameworks dependencies it would be appreciated if
you could please reply with a list of those KDE projects you use so we can
keep them in mind while working on the seed jobs for Independent Releases
and Release Service.

Following this we will be able to make Windows CI generally available on
Gitlab for projects to adopt as needed, and as this represents the
migration of the final platform over to Gitlab this will also come with an
EOL announcement for build.kde.org and it's corresponding infrastructure.

If your project has not yet migrated to Gitlab native CI for any platform
then you should do so *immediately* and without delay. Projects making use
of legacy jobs on Gitlab (which will have references to sysadmin/ci-tooling
instead of sysadmin/ci-utilities and will not have a .kde-ci.yml file) are
not using Gitlab native CI and need to take action.

If you have tooling which interacts with resources provided by build.kde.org
(including build-artifacts.kde.org and the repository sysadmin/ci-tooling)
then please contact Sysadmin as soon as possible so we can discuss what the
best steps are for you to minimize disruption. My understanding is that
this includes apps.kde.org as well as the Android SDK Docker image.

Following the completion of this the dependency metadata located in
sysadmin/repo-metadata should be considered unmaintained and no longer in
use. We will remove this information in due course to ensure only a single
source of truth (being the .kde-ci.yml files now included in repositories).
It is expected that this will require updates to kdesrc-build.

Should anyone have any questions on the above please let us know.

Thanks,
Ben Cooksley
KDE Sysadmin

Gitlab Bugzilla integration

[Ben Cooksley]

Hi all,

This afternoon work has been completed to better link Gitlab with Bugzilla,
allowing people to get from Gitlab more quickly to Bugzilla and also
enabling us to link CCBUG/BUG references in commits, issues and merge
requests from Gitlab over to Bugzilla.

This functionality can be seen in action on
https://invent.kde.org/graphics/digikam/-/commit/7ca7bf03b11ae72842006af8a66ed02894a4a605
(note that content posted prior to this being enabled may not feature links
due to caching in Gitlab)

This integration does not change in any way the use of Gitlab Issues, which
continue to remain intended for contributor collaboration and project
planning.

Use of this functionality requires the setup of some metadata within
sysadmin/repo-metadata which a number of projects have already completed.
Once added it will be synced to Gitlab overnight and will become available
the next day.

Thanks,
Ben Cooksley
KDE Sysadmin

Reminder - Please do not use noreply domains from Gitlab.com/Github.com

[Ben Cooksley]

Hi all,

A while back Sysadmin encountered a series of issues with the DNS resolvers
that support the noreply email addresses offered by one of
Github.com/Gitlab.com to their users.
These issues led to our hooks refusing to accept the email addresses in
commits because the domain does not exist (because their DNS servers are
flaky/faulty).

As these domains are essentially anonymous and not usable in the long run
(and in the case of one of them having faulty DNS servers) the decision was
made to blacklist them.

Recently however we have had an incident where it appears, either someone
has either by accident or otherwise evaded this blacklist by using the
domain users.noreply.github.com.
That domain has now been added to the blacklist.

For those of you using repository forks to prepare your merge requests for
plasma/plasma-desktop, if it was created more than 5 days ago i'm afraid
your fork is now unable to be rebased and will need to be destroyed and
recreated.
(as the commit hooks will refuse to admit the commits from the noreply
domains - especially now that i've corrected the blacklist omission).

My apologies for the inconvenience caused.

Regards,
Ben Cooksley
KDE Sysadmin

Re: Announcing General Availability of Gitlab CI for Linux, FreeBSD and Android

[Ben Cooksley]

On Fri, Dec 3, 2021 at 8:53 AM Michael Reeves  wrote:

> Where are the artifacts being uploaded to? They don't seem to show in
> gitlab. In particular the coverage report is of interest to me.
>

They are uploaded to Gitlab itself, but as reports so only some parts of
Gitlab's functionality show them with a download option.
You should be able to find the necessary links on the Pipelines page for a
project - https://invent.kde.org/sdk/kdiff3/-/pipelines

Cheers,
Ben


> On Sat, Nov 27, 2021 at 1:31 PM Ben Cooksley  wrote:
>
>> Hi all,
>>
>> As mentioned in the subject, i'm happy to announce the general
>> availability of native Gitlab CI for all KDE projects for Linux, FreeBSD
>> and Android.
>>
>> For those who would like to get their projects setup, please ensure you
>> first have a .kde-ci.yml file in the root of your repository specifiying
>> the necessary dependencies of your project and any options your project
>> requires to build and conduct tests. You can find a list of all the
>> supported options at
>> https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/config-template.yml
>> along with guidelines on how to specify your dependencies.
>>
>> Please note the following when specifying the branch to use:
>> - @same should be used for projects which are released together and which
>> follow the same branching scheme (release/21.08 for instance)
>> - @stable should be used to refer to the current stable branch
>> - @latest should be used to refer to the current development branch
>>
>> While the system has been seeded based on the current dependency
>> metadata, if this was incorrect for your project then you may find that the
>> system gives an error that it cannot locate the dependency when first run.
>> If this happens to you then please look into submitting a merge request to
>> https://invent.kde.org/sysadmin/ci-management/-/tree/master/seeds so
>> that this dependency can be included in future seed runs.
>>
>> Once you have your .kde-ci.yml file setup you can then setup your
>> .gitlab-ci.yml to include the appropriate platform files from
>> https://invent.kde.org/sysadmin/ci-utilities/-/tree/master/gitlab-templates
>> .
>>
>> Please note that currently this is limited to builds for master and other
>> development branches, as the necessary metadata for @stable is not yet
>> fully in place. This information can be found at
>> https://invent.kde.org/sysadmin/repo-metadata/-/blob/master/branch-rules.yml
>> - it would be appreciated if developers could please look into updating
>> this for their projects at the same time.
>>
>> With regards to Windows builds, they should become available soon - there
>> is still some infrastructural work that needs to be completed on our Docker
>> image, as well as the facilities to build those images which needs to be
>> completed prior to these being available. We will be transitioning to using
>> Docker for our Windows builds which will hopefully eliminate many of the
>> issues we've had with lingering processes that have plagued the Jenkins
>> system.
>>
>> With the availability of these Gitlab builds we are now entering the
>> twilight phase of our Jenkins based system at build.kde.org so it is
>> imperative that developers setup their CI using the new tooling. For those
>> using the legacy Jenkins system with Gitlab (using the templates at
>> https://invent.kde.org/sysadmin/ci-tooling/-/tree/master/invent) these
>> will be discontinued at the same time as build.kde.org, so you will need
>> to migrate to the new tooling as well.
>>
>> Please let us know if you have any questions on the above.
>>
>> Thanks,
>> Ben
>>
>

Re: Should we consider project basenames unique?

[Ben Cooksley]

On Thu, Nov 25, 2021 at 1:20 AM  wrote:

> > I have no idea why this is all of a sudden a problem for Arch Linux.
>
> In makepkg we do not have origin with project path (see comments in
> pkgbuild linked in
> https://invent.kde.org/sdk/releaseme/-/merge_requests/13). We have
> directory path to which the project was cloned, and it may (and man not) be
> named as project name. In pkgbuild I want to download and build
> translations for project, for that I use KDE_L10N_SYNC_TRANSLATIONS=ON. And
> then cmake module wants to call fetchpo.rb from releaseme. And as it was
> unable to detect project path (because origin is local path, not matching
> regex to extract it), it fallbacks to cmake project name:
> https://invent.kde.org/frameworks/extra-cmake-modules/-/blob/9f519983d628c7c2a68ee1a8dc7a1cbb83c8f535/kde-modules/KDECMakeSettings.cmake#L321.
> And currently fetchpo.rb fails to determine the project by that.
>

I see. It sounds like the basename of the folder is not even guaranteed
based on the above statement, in which case it wouldn't even matter if the
repository names themselves were unique (instead of just our identifier for
them) as your distribution tooling would potentially discard them.

Given you are wanting to include translations, it sounds like the real fix
is something we discussed in a meeting at some point where scripty would
maintain a copy of the latest translations for a project in the actual
project repository.
I think that was being tested in one of our repositories but not sure on
the status of this.


> I wanted to get rid of that function, and use directly cmake project name
> https://invent.kde.org/frameworks/extra-cmake-modules/-/merge_requests/199#note_339860.
> Then modification of releaseme is required (that I created mr 13 for). Can
> we use cmake_project_name as project identifiers? If yes, then I see it is
> no problem to add a from_identifier() to releaseme::project. And then the
> problem will be solved.
>
>
Cheers,
Ben


> 24.11.2021, 13:02, "Ben Cooksley" :
>
> On Wed, Nov 24, 2021 at 10:28 PM Harald Sitter  wrote:
>
> On Wed, Nov 24, 2021 at 10:02 AM Ben Cooksley  wrote:
> > Which means you either provide the path (plasma-mobile/plasma-dialer) or
> you need to go look in the metadata anyway.
>
> If names are unique (not persistent, just unique) and plasma-dialer is
> what I want to release then I know plasma-dialer is called
> plasma-dialer because I'm a plasma-dialer dev. I can then call
> releasme with the argument  'plasma-dialer' and releaseme can work out
> the path from that because the name is global unique so there is only
> one plasma-dialer and that will be what I want to release.
>
>
> In the specific case of releaseme for 99% of projects the situation you've
> described is true, so what you are talking about is a non-issue.
> There are only a small handful of projects where identifier != repository
> name, and the developers in charge of those projects should be able to
> handle that and be aware of the difference - usually because they requested
> it.
>
> I have no idea why this is all of a sudden a problem for Arch Linux.
>
>
> HS
>
>
> Regards,
> Ben
>
>

Re: Announcing General Availability of Gitlab CI for Linux, FreeBSD and Android

[Ben Cooksley]

On Mon, Nov 29, 2021 at 9:11 PM Ingo Klöcker  wrote:

> On Sonntag, 28. November 2021 23:07:50 CET Albert Astals Cid wrote:
> > El dissabte, 27 de novembre de 2021, a les 19:30:08 (CET), Ben Cooksley
> va
> escriure:
> > > As mentioned in the subject, i'm happy to announce the general
> > > availability
> > > of native Gitlab CI for all KDE projects for Linux, FreeBSD and
> Android.
>
> That's great news! Thanks, Ben and other admins!
>
> > Do we have any plan for "category pages" [1] like in Jenkins?
> > [1] i.e. something like
> >
> https://build.kde.org/job/Applications/view/Platform%20-%20WindowsMSVCQt5.15/
>
> At my old workplace I had set up
> https://github.com/heikkipora/gitlab-radiator
> The maintainer was quite responsive to my contributions.
>
> It talks to the GitLab API and uses caching to avoid hammering the GitLab
> server with requests if many people use it. It uses npm which may not be
> the
> best/most secure backend.
>
> I'm sure there are other Free Software GitLab monitors. GitLab seems to
> have
> something similar, but not in the Community Edition.
>

Ingo is correct that GitLab itself (at least in the Community Edition)
lacks this functionality.

I can confirm that this is on our radar and is something we do plan on
addressing.


> Regards,
> Ingo
>

Cheers,
Ben

Announcing General Availability of Gitlab CI for Linux, FreeBSD and Android

[Ben Cooksley]

Hi all,

As mentioned in the subject, i'm happy to announce the general availability
of native Gitlab CI for all KDE projects for Linux, FreeBSD and Android.

For those who would like to get their projects setup, please ensure you
first have a .kde-ci.yml file in the root of your repository specifiying
the necessary dependencies of your project and any options your project
requires to build and conduct tests. You can find a list of all the
supported options at
https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/config-template.yml
along with guidelines on how to specify your dependencies.

Please note the following when specifying the branch to use:
- @same should be used for projects which are released together and which
follow the same branching scheme (release/21.08 for instance)
- @stable should be used to refer to the current stable branch
- @latest should be used to refer to the current development branch

While the system has been seeded based on the current dependency metadata,
if this was incorrect for your project then you may find that the system
gives an error that it cannot locate the dependency when first run. If this
happens to you then please look into submitting a merge request to
https://invent.kde.org/sysadmin/ci-management/-/tree/master/seeds so that
this dependency can be included in future seed runs.

Once you have your .kde-ci.yml file setup you can then setup your
.gitlab-ci.yml to include the appropriate platform files from
https://invent.kde.org/sysadmin/ci-utilities/-/tree/master/gitlab-templates.

Please note that currently this is limited to builds for master and other
development branches, as the necessary metadata for @stable is not yet
fully in place. This information can be found at
https://invent.kde.org/sysadmin/repo-metadata/-/blob/master/branch-rules.yml
- it would be appreciated if developers could please look into updating
this for their projects at the same time.

With regards to Windows builds, they should become available soon - there
is still some infrastructural work that needs to be completed on our Docker
image, as well as the facilities to build those images which needs to be
completed prior to these being available. We will be transitioning to using
Docker for our Windows builds which will hopefully eliminate many of the
issues we've had with lingering processes that have plagued the Jenkins
system.

With the availability of these Gitlab builds we are now entering the
twilight phase of our Jenkins based system at build.kde.org so it is
imperative that developers setup their CI using the new tooling. For those
using the legacy Jenkins system with Gitlab (using the templates at
https://invent.kde.org/sysadmin/ci-tooling/-/tree/master/invent) these will
be discontinued at the same time as build.kde.org, so you will need to
migrate to the new tooling as well.

Please let us know if you have any questions on the above.

Thanks,
Ben

Re: Should we consider project basenames unique?

[Ben Cooksley]

On Wed, Nov 24, 2021 at 10:28 PM Harald Sitter  wrote:

> On Wed, Nov 24, 2021 at 10:02 AM Ben Cooksley  wrote:
> > Which means you either provide the path (plasma-mobile/plasma-dialer) or
> you need to go look in the metadata anyway.
>
> If names are unique (not persistent, just unique) and plasma-dialer is
> what I want to release then I know plasma-dialer is called
> plasma-dialer because I'm a plasma-dialer dev. I can then call
> releasme with the argument  'plasma-dialer' and releaseme can work out
> the path from that because the name is global unique so there is only
> one plasma-dialer and that will be what I want to release.
>

In the specific case of releaseme for 99% of projects the situation you've
described is true, so what you are talking about is a non-issue.
There are only a small handful of projects where identifier != repository
name, and the developers in charge of those projects should be able to
handle that and be aware of the difference - usually because they requested
it.

I have no idea why this is all of a sudden a problem for Arch Linux.


> HS
>

Regards,
Ben

Re: Should we consider project basenames unique?

[Ben Cooksley]

On Wed, Nov 24, 2021 at 9:46 PM Harald Sitter  wrote:

> On Wed, Nov 24, 2021 at 6:00 AM Ben Cooksley  wrote:
> >
> > On Wed, Nov 24, 2021 at 2:29 PM Aleix Pol  wrote:
> >>
> >> On Wed, Nov 24, 2021 at 2:10 AM ash...@linuxcomp.ru <
> ash...@linuxcomp.ru> wrote:
> >> >
> >> > Hello.
> >> > There is currently a problem that projects are identified by path in
> invent.kde.org. But in pkgbuilds in Arch Linux the makepkg clones repo
> twice, first time to the dir with pkgbuild, and second time to the srcdir
> for build package. And that second repo has first repo as origin, i.e. its
> origin is path to the first repo. And this leads to breakage of determining
> project identifier in cmake module. To workaround, I either need to clone
> to the path that matches the regexp, such as crutch.kde.org:pim/zanshin.git
> or fixing the origin of the problem.
> >> >
> >> > I wanted to fix it by identifying the project by its name. See
> https://invent.kde.org/sdk/releaseme/-/merge_requests/13.
> >> >
> >> > Now, the question is, could we consider project names unique? In that
> case, we can merge my mr, and path crutch will not be needed anymore.
> >>
> >> I agree it would be a problem to have different repositories with the
> >> same name. I'm not sure how we can enforce it though.
> >
> >
> > This is something that Sysadmin already encountered and resolved.
> >
> > Please see the 'identifier' key in the repository metadata (located at
> https://invent.kde.org/sysadmin/repo-metadata/ in the projects/
> subfolder) which we guarantee to be unique.
> > This is already used and relied upon by scripty as well as our internal
> systems (such as commits.kde.org) when referencing repositories.
>
> That is the problem. We, the humans, do not know the identifier nor
> should we care. When I want to make a release of plasma-dialer I
> should be able to pass plasma-dialer as argument because I know the
> repo/project is called plasma-dialer.
>

Sorry i'm not seeing your point here.

First, there is a fairly small group of projects where the identifier does
not equal the repository name. Those projects have good valid reasons (as
outlined below) for that difference.

Second, regardless of what you pass to releaseme, it still has to locate
the project on invent.kde.org, which means it needs to know the group it is
in - otherwise it cannot clone the repository.
Which means you either provide the path (plasma-mobile/plasma-dialer) or
you need to go look in the metadata anyway.

If you have an existing checkout then you shouldn't need to provide an
argument because releaseme should be able to infer it for you from the URL
in .git/config of the clone - and can make reference to the metadata (it is
an automated tool after all)


> > The repository name itself (stripped of any folders) is not guaranteed
> to be unique.
>
> That is what is proposed to get changed.
>

The whole reason why we went with identifiers is because we already had
requests from people in certain namespaces (maui/ in particular, but others
as well if memory serves) who were agitating about their repositories being
prefixed with the overall project name due to the extremely generic names
of their repositories.

Those prefixes also read very strange when taking into account the group
name. If this change were to go through, then you end up with URLs such as
https://invent.kde.org/maui/maui-pix - compared with what we have currently
which is https://invent.kde.org/maui/pix


> HS
>

Regards,
Ben

Re: Should we consider project basenames unique?

[Ben Cooksley]

On Wed, Nov 24, 2021 at 2:29 PM Aleix Pol  wrote:

> On Wed, Nov 24, 2021 at 2:10 AM ash...@linuxcomp.ru 
> wrote:
> >
> > Hello.
> > There is currently a problem that projects are identified by path in
> invent.kde.org. But in pkgbuilds in Arch Linux the makepkg clones repo
> twice, first time to the dir with pkgbuild, and second time to the srcdir
> for build package. And that second repo has first repo as origin, i.e. its
> origin is path to the first repo. And this leads to breakage of determining
> project identifier in cmake module. To workaround, I either need to clone
> to the path that matches the regexp, such as crutch.kde.org:pim/zanshin.git
> or fixing the origin of the problem.
> >
> > I wanted to fix it by identifying the project by its name. See
> https://invent.kde.org/sdk/releaseme/-/merge_requests/13.
> >
> > Now, the question is, could we consider project names unique? In that
> case, we can merge my mr, and path crutch will not be needed anymore.
>
> I agree it would be a problem to have different repositories with the
> same name. I'm not sure how we can enforce it though.
>

This is something that Sysadmin already encountered and resolved.

Please see the 'identifier' key in the repository metadata (located at
https://invent.kde.org/sysadmin/repo-metadata/ in the projects/ subfolder)
which we guarantee to be unique.
This is already used and relied upon by scripty as well as our internal
systems (such as commits.kde.org) when referencing repositories.

The repository name itself (stripped of any folders) is not guaranteed to
be unique.


> Aleix
>

Cheers,
Ben

CI System Issues

[Ben Cooksley]

Hi all,

Over the past 24-48 hours we have been experiencing significant issues with
both of the Jenkins' instances which operate build.kde.org and
binary-factory.kde.org.

After significant investigative work these issues have been traced to a
bug/regression within Docker itself, which we have now applied a workaround
for. This issue is tracked at upstream bug
https://github.com/moby/moby/issues/42375 which unfortunately has received
little attention thus far.

With the resolution of this issue we've now initiated Global Rebuilds
across Jenkins to restore normal service. As a consequence of this it may
take several hours for the CI system to attend to builds that have been
initiated, including updates to our websites.

Due to their unreliability in the build chain I will shortly also be
terminating support for building Python bindings on Linux, as they are
interfering in the ability of the Dependency Builds - and thus the Global
Rebuild - to complete.

Apologies for the disruption to service.

Thanks,
Ben Cooksley
KDE Sysadmin

Re: OCS Providers File - High Numbers of Requests

[Ben Cooksley]

On Mon, Sep 27, 2021 at 7:04 AM Aleix Pol  wrote:

> On Fri, Sep 24, 2021 at 8:26 PM Ben Cooksley  wrote:
> >
> > On Sat, Sep 25, 2021 at 12:34 AM Aleix Pol  wrote:
> >>
> >> On Fri, Sep 24, 2021 at 1:32 AM Nicolás Alvarez
> >>  wrote:
> >> >
> >> > El jue, 23 de sep. de 2021 a la(s) 19:31, Aleix Pol (aleix...@kde.org)
> escribió:
> >> > >
> >> > > On Thu, Sep 23, 2021 at 10:12 PM Nicolás Alvarez
> >> > >  wrote:
> >> > > >
> >> > > > El jue, 23 de sep. de 2021 a la(s) 08:55, Aleix Pol (
> aleix...@kde.org) escribió:
> >> > > > >
> >> > > > > On Thu, Sep 23, 2021 at 11:52 AM Ben Cooksley <
> bcooks...@kde.org> wrote:
> >> > > > > >
> >> > > > > > Hi all,
> >> > > > > >
> >> > > > > > It has recently come to our attention that the number of
> queries being handled for the endpoint
> https://autoconfig.kde.org/ocs/providers.xml on a day to day basis has
> gotten to the point where it is causing issues with server responsiveness
> to other traffic. This is perhaps best summarised by the following:
> >> > > > > >
> >> > > > > > root@nicoda /var/log/apache2 # ls -lah ...
> >> > > > > > -rw-r- 1 root adm 458M Sep 23 06:25
> autoconfig.kde.org.log.1
> >> > > > > > -rw-r- 1 root adm 381M Sep 23 06:25
> networkcheck.kde.org.log.1
> >> > > > > > -rw-r- 1 root adm 143M Sep 23 06:25 www.kde.org.log.1
> >> > > > > >
> >> > > > > > root@nicoda /var/log/apache2 # cat autoconfig.kde.org.log.1
> | wc -l
> >> > > > > > 4,222,343
> >> > > > > >
> >> > > > > > Based on those numbers we're looking at 48-49 requests per
> second (on average - peaks are much higher by many magnitudes), which seems
> extremely excessive given that this file is only supposed to be retrieved
> by KDE software when GHNS functionality is triggered. That is supported by
> the substantial size difference it has over networkcheck.kde.org - which
> is used by plasma-nm and NetworkManager (on Neon) to check for whether they
> have a working internet connection - which i'd expect to be the site
> receiving the most traffic.
> >> > > > > >
> >> > > > > > As such, I therefore suspect we have bug(s) in software that
> makes use of GHNS functionality.
> >> > > > > >
> >> > > > > > It would therefore be appreciated if we could please review
> the software in question to determine whether it is operating correctly.
> Given that it usually runs in the background on user systems, i'd
> especially appreciate it if a detailed review could be conducted on
> Discover and other software that conducts package management operations or
> assists in managing updates.
> >> > > > > >
> >> > > > > > Unfortunately all these applications submit a fairly useless
> user agent (Mozilla/5.0) so it is impossible for Sysadmin to ascertain any
> further information. If we could get information on the software that is
> originating the request added to the user agent to assist in investigating
> these issues in the future that would be extremely helpful.
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Ben
> >> > > > >
> >> > > > > That's correct. Discover fetches them at startup. It's
> necessary to be
> >> > > > > able to check if there are updates on KNS-provided resources.
> >> > > > >
> >> > > > > Incidentally,  I was looking into this yesterday incidentally.
> We
> >> > > > > could see if caching is broken somehow. A request will still be
> needed
> >> > > > > though to check if the cache is out of date.
> >> > > >
> >> > > > Caching seems to be working, since the vast majority of the
> requests
> >> > > > are returning 304 Not Modified.
> >> > > >
> >> > > > However in *many* cases I see a single IP making multiple
> requests in
> >> > > > the same second, and doing it again the next minute. Here's one IP
> >> > > > address picked randomly:
> >> > > >
> >> > > > [22/Sep/2021:06:25:41 +] "GET /ocs/providers.xml HTTP/1.1&quo

Re: OCS Providers File - High Numbers of Requests

[Ben Cooksley]

On Sat, Sep 25, 2021 at 12:34 AM Aleix Pol  wrote:

> On Fri, Sep 24, 2021 at 1:32 AM Nicolás Alvarez
>  wrote:
> >
> > El jue, 23 de sep. de 2021 a la(s) 19:31, Aleix Pol (aleix...@kde.org)
> escribió:
> > >
> > > On Thu, Sep 23, 2021 at 10:12 PM Nicolás Alvarez
> > >  wrote:
> > > >
> > > > El jue, 23 de sep. de 2021 a la(s) 08:55, Aleix Pol (
> aleix...@kde.org) escribió:
> > > > >
> > > > > On Thu, Sep 23, 2021 at 11:52 AM Ben Cooksley 
> wrote:
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > It has recently come to our attention that the number of queries
> being handled for the endpoint
> https://autoconfig.kde.org/ocs/providers.xml on a day to day basis has
> gotten to the point where it is causing issues with server responsiveness
> to other traffic. This is perhaps best summarised by the following:
> > > > > >
> > > > > > root@nicoda /var/log/apache2 # ls -lah ...
> > > > > > -rw-r- 1 root adm 458M Sep 23 06:25 autoconfig.kde.org.log.1
> > > > > > -rw-r- 1 root adm 381M Sep 23 06:25
> networkcheck.kde.org.log.1
> > > > > > -rw-r- 1 root adm 143M Sep 23 06:25 www.kde.org.log.1
> > > > > >
> > > > > > root@nicoda /var/log/apache2 # cat autoconfig.kde.org.log.1 |
> wc -l
> > > > > > 4,222,343
> > > > > >
> > > > > > Based on those numbers we're looking at 48-49 requests per
> second (on average - peaks are much higher by many magnitudes), which seems
> extremely excessive given that this file is only supposed to be retrieved
> by KDE software when GHNS functionality is triggered. That is supported by
> the substantial size difference it has over networkcheck.kde.org - which
> is used by plasma-nm and NetworkManager (on Neon) to check for whether they
> have a working internet connection - which i'd expect to be the site
> receiving the most traffic.
> > > > > >
> > > > > > As such, I therefore suspect we have bug(s) in software that
> makes use of GHNS functionality.
> > > > > >
> > > > > > It would therefore be appreciated if we could please review the
> software in question to determine whether it is operating correctly. Given
> that it usually runs in the background on user systems, i'd especially
> appreciate it if a detailed review could be conducted on Discover and other
> software that conducts package management operations or assists in managing
> updates.
> > > > > >
> > > > > > Unfortunately all these applications submit a fairly useless
> user agent (Mozilla/5.0) so it is impossible for Sysadmin to ascertain any
> further information. If we could get information on the software that is
> originating the request added to the user agent to assist in investigating
> these issues in the future that would be extremely helpful.
> > > > > >
> > > > > > Thanks,
> > > > > > Ben
> > > > >
> > > > > That's correct. Discover fetches them at startup. It's necessary
> to be
> > > > > able to check if there are updates on KNS-provided resources.
> > > > >
> > > > > Incidentally,  I was looking into this yesterday incidentally. We
> > > > > could see if caching is broken somehow. A request will still be
> needed
> > > > > though to check if the cache is out of date.
> > > >
> > > > Caching seems to be working, since the vast majority of the requests
> > > > are returning 304 Not Modified.
> > > >
> > > > However in *many* cases I see a single IP making multiple requests in
> > > > the same second, and doing it again the next minute. Here's one IP
> > > > address picked randomly:
> > > >
> > > > [22/Sep/2021:06:25:41 +] "GET /ocs/providers.xml HTTP/1.1" 304
> > > > [22/Sep/2021:06:25:41 +] "GET /ocs/providers.xml HTTP/1.1" 304
> > > > [22/Sep/2021:06:25:41 +] "GET /ocs/providers.xml HTTP/1.1" 304
> > > > [22/Sep/2021:06:25:41 +] "GET /ocs/providers.xml HTTP/1.1" 304
> > > > [22/Sep/2021:06:27:57 +] "GET /ocs/providers.xml HTTP/1.1" 304
> > > > [22/Sep/2021:06:27:58 +] "GET /ocs/providers.xml HTTP/1.1" 304
> > > > [22/Sep/2021:06:27:58 +] "GET /ocs/providers.xml HTTP/1.1" 304
> > > > [22/Sep/2021:06:28:32 +] 

Re: OCS Providers File - High Numbers of Requests

[Ben Cooksley]

On Fri, Sep 24, 2021 at 6:34 PM Shantanu Tushar Jha 
wrote:

> Hi,
>

Hi Shantanu,


> Just an idea - do we have enough logs on the server to see the requests
> history by date? If so, one can identify if there was a spike of the
> requests after a particular set of dates (which can in turn give us a hint
> about which release might contain a bug that's making more calls than
> expected). Of course this is not useful if it's apparent that the request
> count increased gradually instead of a spike.
>

I'm afraid not - we only keep raw logs for 2 weeks after which they are
discarded in accordance with our privacy policy.

This issue is one that has existed for sometime, although it is hard to
tell when the situation started to deteriorate significantly and what the
cause of that was.
(Whether it was due to more user systems, or due to changing software
behaviour/bugs)


> Shantanu
>

Cheers,
Ben


> On Thu, 23 Sept 2021 at 22:13, Nicolás Alvarez 
> wrote:
>
>> El jue, 23 de sep. de 2021 a la(s) 08:55, Aleix Pol (aleix...@kde.org)
>> escribió:
>> >
>> > On Thu, Sep 23, 2021 at 11:52 AM Ben Cooksley 
>> wrote:
>> > >
>> > > Hi all,
>> > >
>> > > It has recently come to our attention that the number of queries
>> being handled for the endpoint
>> https://autoconfig.kde.org/ocs/providers.xml on a day to day basis has
>> gotten to the point where it is causing issues with server responsiveness
>> to other traffic. This is perhaps best summarised by the following:
>> > >
>> > > root@nicoda /var/log/apache2 # ls -lah ...
>> > > -rw-r- 1 root adm 458M Sep 23 06:25 autoconfig.kde.org.log.1
>> > > -rw-r- 1 root adm 381M Sep 23 06:25 networkcheck.kde.org.log.1
>> > > -rw-r- 1 root adm 143M Sep 23 06:25 www.kde.org.log.1
>> > >
>> > > root@nicoda /var/log/apache2 # cat autoconfig.kde.org.log.1 | wc -l
>> > > 4,222,343
>> > >
>> > > Based on those numbers we're looking at 48-49 requests per second (on
>> average - peaks are much higher by many magnitudes), which seems extremely
>> excessive given that this file is only supposed to be retrieved by KDE
>> software when GHNS functionality is triggered. That is supported by the
>> substantial size difference it has over networkcheck.kde.org - which is
>> used by plasma-nm and NetworkManager (on Neon) to check for whether they
>> have a working internet connection - which i'd expect to be the site
>> receiving the most traffic.
>> > >
>> > > As such, I therefore suspect we have bug(s) in software that makes
>> use of GHNS functionality.
>> > >
>> > > It would therefore be appreciated if we could please review the
>> software in question to determine whether it is operating correctly. Given
>> that it usually runs in the background on user systems, i'd especially
>> appreciate it if a detailed review could be conducted on Discover and other
>> software that conducts package management operations or assists in managing
>> updates.
>> > >
>> > > Unfortunately all these applications submit a fairly useless user
>> agent (Mozilla/5.0) so it is impossible for Sysadmin to ascertain any
>> further information. If we could get information on the software that is
>> originating the request added to the user agent to assist in investigating
>> these issues in the future that would be extremely helpful.
>> > >
>> > > Thanks,
>> > > Ben
>> >
>> > That's correct. Discover fetches them at startup. It's necessary to be
>> > able to check if there are updates on KNS-provided resources.
>> >
>> > Incidentally,  I was looking into this yesterday incidentally. We
>> > could see if caching is broken somehow. A request will still be needed
>> > though to check if the cache is out of date.
>>
>> Caching seems to be working, since the vast majority of the requests
>> are returning 304 Not Modified.
>>
>> However in *many* cases I see a single IP making multiple requests in
>> the same second, and doing it again the next minute. Here's one IP
>> address picked randomly:
>>
>> [22/Sep/2021:06:25:41 +] "GET /ocs/providers.xml HTTP/1.1" 304
>> [22/Sep/2021:06:25:41 +] "GET /ocs/providers.xml HTTP/1.1" 304
>> [22/Sep/2021:06:25:41 +] "GET /ocs/providers.xml HTTP/1.1" 304
>> [22/Sep/2021:06:25:41 +] "GET /ocs/providers.xml HTTP/1.1" 304
>> [22/Sep/2021:06:27:57 +] "GET /ocs/providers.xml HTTP/1.1" 304

OCS Providers File - High Numbers of Requests

[Ben Cooksley]

Hi all,

It has recently come to our attention that the number of queries being
handled for the endpoint https://autoconfig.kde.org/ocs/providers.xml on a
day to day basis has gotten to the point where it is causing issues with
server responsiveness to other traffic. This is perhaps best summarised by
the following:

root@nicoda /var/log/apache2 # ls -lah ...
-rw-r- 1 root adm 458M Sep 23 06:25 autoconfig.kde.org.log.1
-rw-r- 1 root adm 381M Sep 23 06:25 networkcheck.kde.org.log.1
-rw-r- 1 root adm 143M Sep 23 06:25 www.kde.org.log.1

root@nicoda /var/log/apache2 # cat autoconfig.kde.org.log.1 | wc -l
4,222,343

Based on those numbers we're looking at 48-49 requests per second (on
average - peaks are much higher by many magnitudes), which seems extremely
excessive given that this file is only supposed to be retrieved by KDE
software when GHNS functionality is triggered. That is supported by the
substantial size difference it has over networkcheck.kde.org - which is
used by plasma-nm and NetworkManager (on Neon) to check for whether they
have a working internet connection - which i'd expect to be the site
receiving the most traffic.

As such, I therefore suspect we have bug(s) in software that makes use of
GHNS functionality.

It would therefore be appreciated if we could please review the software in
question to determine whether it is operating correctly. Given that it
usually runs in the background on user systems, i'd especially appreciate
it if a detailed review could be conducted on Discover and other software
that conducts package management operations or assists in managing updates.

Unfortunately all these applications submit a fairly useless user agent
(Mozilla/5.0) so it is impossible for Sysadmin to ascertain any further
information. If we could get information on the software that is
originating the request added to the user agent to assist in investigating
these issues in the future that would be extremely helpful.

Thanks,
Ben

Re: Towards Excellent Defect Management

[Ben Cooksley]

On Wed, Sep 15, 2021 at 7:06 AM Albert Astals Cid  wrote:

> El dimarts, 14 de setembre de 2021, a les 20:35:40 (CEST), Ben Cooksley va
> escriure:
> > On Wed, Sep 15, 2021 at 5:35 AM Albert Astals Cid  wrote:
> >
> > > El dimarts, 14 de setembre de 2021, a les 17:23:01 (CEST), Harald
> Sitter
> > > va escriure:
> > > > It is  practically free software as far as we are concerned
> > >
> > > I guess this means it's not actually Free Software?
> > >
> >
> > Please see https://open.sentry.io/licensing/
> >
> > The tl;dr is that the license it is provided under (BSL 1.1) is not OSI
> > approved due to the restriction on it's use by cloud vendors (ie. it has
> an
> > anti-AWS clause).
> > That restriction lapses after 36 months, at which point it is Apache 2.0
> > compatible.
> >
> > Based on what Harald has written earlier, it looks like Sentry is the
> only
> > suitable game in town for what we need - the only question is whether we
> > are happy to make use of BSL 1.1 licensed software given that we have
> > traditionally only deployed 100% open source software to our systems
> (which
> > is why we use Gitlab CE over Gitlab EE)
>
> Oh i feel we already discussed this in the past, right?
>
> Sorry for bringing it up again then :)
>

I've checked my mail archives and there was a public discussion regarding
Sentry/BSL earlier yes - however it didn't reach a conclusion, it was
deferred to a BoF.
That was for server/infrastructure event monitoring as well, rather than
crash handling.


> Cheers,
>   Albert
>

Cheers,
Ben


> >
> >
> > > Cheers,
> > >   Albert
> > >
> > >
> > >
> > Cheers,
> > Ben
> >
>
>
>
>
>

Re: Towards Excellent Defect Management

[Ben Cooksley]

On Wed, Sep 15, 2021 at 5:35 AM Albert Astals Cid  wrote:

> El dimarts, 14 de setembre de 2021, a les 17:23:01 (CEST), Harald Sitter
> va escriure:
> > It is  practically free software as far as we are concerned
>
> I guess this means it's not actually Free Software?
>

Please see https://open.sentry.io/licensing/

The tl;dr is that the license it is provided under (BSL 1.1) is not OSI
approved due to the restriction on it's use by cloud vendors (ie. it has an
anti-AWS clause).
That restriction lapses after 36 months, at which point it is Apache 2.0
compatible.

Based on what Harald has written earlier, it looks like Sentry is the only
suitable game in town for what we need - the only question is whether we
are happy to make use of BSL 1.1 licensed software given that we have
traditionally only deployed 100% open source software to our systems (which
is why we use Gitlab CE over Gitlab EE)


> Cheers,
>   Albert
>
>
>
Cheers,
Ben

Re: Gitlab CI - Inbound

[Ben Cooksley]

On Tue, Sep 7, 2021 at 9:09 PM David Edmundson 
wrote:

> Excellent news!! Thanks very much
>
> > Once the scripts have been proven successfully for Frameworks, we will
> look at extending them to projects that depend only on Frameworks and
> repositories
>
> Does this mean we would like Plasma to wait a while before merging?
> Is it worth us creating the kde-cli files and not merging them so we
> have some test cases ready?
>

You can certainly get the .kde-ci.yml files in position - I do not expect
the format of them to change at this stage.

The big thing that will delay the rollout is determining how best to handle
the situation of when two projects want different versions of the same
dependency.
I'm going to have to figure that out sooner than anticipated in any event
due to Phonon and plasma-wayland-protocols which are both dependencies of
Frameworks (yet which both also require Frameworks).


> David
>

Cheers,
Ben

Re: Gitlab CI - Inbound

[Ben Cooksley]

On Tue, Sep 7, 2021 at 6:20 AM Johnny Jazeix  wrote:

> Hi Ben,
> not sure on which priority it is regarding the KDE Frameworks but I've
> added one on GCompris (
> https://invent.kde.org/education/gcompris/-/commit/67c9839d7970b360b5d6b0ec928b492f9003d07d)
> if it can help on more tests.
>

Thanks for getting that landed Johnny.

Please note that you've specified no dependencies, so your builds won't
even have ECM available so you may wish to fix that.


> Cheers,
>
> Johnny
>

Cheers,
Ben


> Le dim. 5 sept. 2021 à 12:11, Ben Cooksley  a écrit :
>
>> On Sun, Sep 5, 2021 at 6:13 PM Ben Cooksley  wrote:
>>
>>> Hi all,
>>>
>>
>> Hi all,
>>
>>
>>> This morning after much work i'm happy to announce that the new
>>> generation CI scripts intended for use with Gitlab CI successfully
>>> completed their first build (of ECM, and then subsequently of KCoreAddons).
>>>
>>> This begins our first steps towards transferring CI over from Jenkins to
>>> Gitlab.
>>>
>>> These scripts are 'Gitlab native' and are designed to work in an
>>> environment where they will be running on merge requests, tags as well as
>>> all branches of our repositories - something the existing scripts were
>>> completely incompatible with. While there are still some infrastructural
>>> elements to put in place (such as 'seed jobs' to mass rebuild all projects
>>> after substantial changes and 'cleanup jobs' to remove old builds from the
>>> Package Registry) the core elements needed for initial testing of these
>>> scripts are now ready.
>>>
>>
>> As an update, an initial version of the seed job tooling is now ready,
>> however testing that tooling requires that dependency information be
>> available.
>> This requires that the .kde-ci.yml files be populated in repositories.
>>
>> It would be appreciated if people could please work on getting these
>> files populated in Frameworks (as everyone needs those) as well as in their
>> own repositories as they are required before we can proceed much further.
>>
>>
>>> For those curious, the results of those initials runs can be found at
>>> https://invent.kde.org/groups/teams/ci-artifacts/-/packages
>>>
>>> Due to the challenges associated with having to handle all of the
>>> different forms of build and the versioning of this information, these
>>> scripts also represent a radical change in how CI builds will be conducted
>>> - with a large part of the configuration of the jobs themselves, including
>>> information on project dependencies now shifting to files located within
>>> the repository themselves. Those who monitor commits to Frameworks
>>> repositories will have noticed the appearance of these '.kde-ci.yml' files
>>> in some repositories.
>>>
>>> To assist in the future rollout of the CI system it would be appreciated
>>> if projects could please begin setting up these files within their
>>> respective repositories.
>>> You can find a template detailing the available options at
>>> https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config-template.yml
>>>
>>> Where possible please avoid overriding the system defaults except where
>>> needed by your project (ie. don't just copy the template in full)
>>> The defaults mirror the template and can be found at
>>> https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config/global.yml
>>>
>>> In terms of the format of the 'Dependencies' section, please bear in
>>> mind the following:
>>> - For the 'on' section, the special value '@all' can be used to mean
>>> "All platforms" to avoid needing to update the file in the event additional
>>> platforms are added in the future
>>> - For the 'require' section:
>>>   1) Please only include the projects you *explicitly* depend on.
>>> Dependencies of your dependencies will be included automatically
>>>   2) When specifying a project, you must use the repository path on
>>> Gitlab. Legacy project paths (such as kde/workspace/plasma-desktop) are no
>>> longer in use and will not work.
>>>   3) There are three special values for the branch name - '@same',
>>> '@latest' and '@stable' which can be used to refer to the branch/tag of a
>>> dependency.
>>>   '@same' means the branch name should be the same as that of the
>>> project being built and should be used when declaring dependencies among
>>&g

Re: Gitlab CI - Inbound

[Ben Cooksley]

On Tue, Sep 7, 2021 at 1:04 AM Tom Zander  wrote:

> On maandag 6 september 2021 11:48:39 CEST Ben Cooksley wrote:
> > > Pushing everything into required is likely not scalable,
> > > causing projects too wait too long for compile.
> > > Avoiding the optional ones means you lack coverage of compile
> > > and testing failures due to changes in libs.
> >
> > The CI system has reused the results of previous builds of
> > dependencies since the very first generation of the system
>
> We seem to be talking about two slightly different topics.
>
> When the (for instance) KIO repo changes, then the CI will
> obviously rebuild that repo and will pull in all the things that
> kio depends on.
>
> What many CIs do is additionally trigger rebuilds of projects
> that _use_ KIO, by them marking kio as a required dependency.
>

Our CI system has never done this, in part due to difficulties in
communicating this to Jenkins and also because of the build storm it would
create (which I believe is what you are referring to).
If we were to limit it to select projects, then we would need to 'pick'
those projects which would raise questions of favouritism which I'd rather
avoid.


> Imagine a small extragear app that uses some KIO stuff and it has
> some unit tests that would break as a result of the KIO change.
> When the KDE CI triggers a rebuild of projects that mark KIO as a
> required dependency, this little app would show its breaking as a
> result of the KIO push. Helping the KIO dev to realize the
> fallout as well.
> Without such a feature the app would show breakage at a random
> time in the future after a new push was made in that repo. Losing
> lots of dev time and compromising quality.
>
> Now, the optional requirements would help diminish the effect of
> changes in frameworks paralysing the build system by limiting the
> apps that gets scheduled for a rebuild.
>

> This kind of functionality becomes pretty easy to add to gen5.1
> of the CI, provided that at this time the dependencies are
> already split since doing it later is going to be an uphill
> battle.
>

Cheers,
Ben

Re: Gitlab CI - Inbound

[Ben Cooksley]

On Mon, Sep 6, 2021 at 9:00 PM Tom Zander  wrote:

> On zondag 5 september 2021 08:13:09 CEST Ben Cooksley wrote:
> > In terms of the format of the 'Dependencies' section,
>
> Playing with kde-build script and noticing the fast growing
> dependency trees we have today, I think it may be beneficial to
> have two types of compile dependencies in this setup.
>
> 1. required-to-build.
>   Which means that if something in the parent tree changes, this
>   one is scheduled for re-build.
>
> 2. optional. Equivalent of the cmake feature, if its not there
>   some code is not compiled.
>   At least once before a release the full dependencies can be
>   compiled to see if it fully compiles.
>

>From the perspective of the CI system there is basically no difference in
terms of making a dependency available or not as all dependencies are
always satisfied using previously built binaries.
If a dependency is not available your build fails.

We also have to make a hard choice - we either bring in optional
dependencies or we don't.

If we were to randomise whether we brought them in - or just brought them
in at certain times - then we would make the system state deterministic.
This could in some cases cause builds to break if it was random whether or
not we included optional dependencies. This would occur if the dependency
of a dependency of a project were rebuilt without an optional dependency
being present which in turn caused it's API interface to change. That would
then cause the project to fail as the dependency would now be broken.


>
> Pushing everything into required is likely not scalable, causing
> projects too wait too long for compile.
> Avoiding the optional ones means you lack coverage of compile and
> testing failures due to changes in libs.
>

The CI system has reused the results of previous builds of dependencies
since the very first generation of the system (this would be generation 5)
so this is not a problem facing the CI system.
For individual developers, my understanding is that kdesrc-build makes use
of incremental builds which eliminates most of the issue as well.


> What do people think, is it useful to have an 'optional' category
> in future there?
> Maybe useful to think that far ahead now people are populating
> their dependencies :-)
>
>
>
>
Thanks,
Ben

Re: Gitlab CI - Inbound

[Ben Cooksley]

On Mon, Sep 6, 2021 at 12:18 AM David Faure  wrote:

> On dimanche 5 septembre 2021 12:26:50 CEST Ben Cooksley wrote:
> > On Sun, Sep 5, 2021 at 10:22 PM David Faure  wrote:
> > > For frameworks, I think we should be able to write a one-time script
> that
> > > generates .kde-ci.yml files using the dependencies listed in kde-build-
> > > metadata (and the platforms listed in metainfo.yaml) ?
> >
> > Yes, that should work nicely (although that information now lives in
> > sysadmin/repo-metadata, dependencies folder)
>
> All done for Frameworks.
>
> The script that collects platforms from metainfo.yaml won't be useful for
> other KDE modules, but the script that collects deps from
> dependency-data-kf5-
> qt5 is attached, in case it's useful to anyone else.
>

Thanks for sorting this David, very much appreciated.


> > Once we've transitioned across to the .kde-ci.yml files the existing
> > dependency metadata and branch group files will no longer be required.
> > (We'll need to work out a way of exporting this information for easy
> > consumption by kdesrc-build and other clients before it can be removed
> > though)
>
> OK. Duplication is bad so I agree :)
>

Cheers,
Ben


> --
> David Faure, fa...@kde.org, http://www.davidfaure.fr
> Working on KDE Frameworks 5
>

Re: Gitlab CI - Inbound

[Ben Cooksley]

On Mon, Sep 6, 2021 at 12:46 AM Nicolas Fella  wrote:

> On 05.09.21 08:13, Ben Cooksley wrote:
> > Hi all,
> >
> > This morning after much work i'm happy to announce that the new
> > generation CI scripts intended for use with Gitlab CI successfully
> > completed their first build (of ECM, and then subsequently of
> > KCoreAddons).
> >
> > This begins our first steps towards transferring CI over from Jenkins
> > to Gitlab.
> >
> > These scripts are 'Gitlab native' and are designed to work in an
> > environment where they will be running on merge requests, tags as well
> > as all branches of our repositories - something the existing scripts
> > were completely incompatible with. While there are still some
> > infrastructural elements to put in place (such as 'seed jobs' to mass
> > rebuild all projects after substantial changes and 'cleanup jobs' to
> > remove old builds from the Package Registry) the core elements needed
> > for initial testing of these scripts are now ready.
> >
> > For those curious, the results of those initials runs can be found at
> > https://invent.kde.org/groups/teams/ci-artifacts/-/packages
> > <https://invent.kde.org/groups/teams/ci-artifacts/-/packages>
> >
> > Due to the challenges associated with having to handle all of the
> > different forms of build and the versioning of this information, these
> > scripts also represent a radical change in how CI builds will be
> > conducted - with a large part of the configuration of the jobs
> > themselves, including information on project dependencies now shifting
> > to files located within the repository themselves. Those who monitor
> > commits to Frameworks repositories will have noticed the appearance of
> > these '.kde-ci.yml' files in some repositories.
> >
> > To assist in the future rollout of the CI system it would be
> > appreciated if projects could please begin setting up these files
> > within their respective repositories.
> > You can find a template detailing the available options at
> >
> https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config-template.yml
> > <
> https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config-template.yml
> >
> >
> > Where possible please avoid overriding the system defaults except
> > where needed by your project (ie. don't just copy the template in full)
> > The defaults mirror the template and can be found at
> >
> https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config/global.yml
> > <
> https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config/global.yml
> >
> >
> > In terms of the format of the 'Dependencies' section, please bear in
> > mind the following:
> > - For the 'on' section, the special value '@all' can be used to mean
> > "All platforms" to avoid needing to update the file in the event
> > additional platforms are added in the future
> > - For the 'require' section:
> >   1) Please only include the projects you *explicitly* depend on.
> > Dependencies of your dependencies will be included automatically
> >   2) When specifying a project, you must use the repository path on
> > Gitlab. Legacy project paths (such as kde/workspace/plasma-desktop)
> > are no longer in use and will not work.
> >   3) There are three special values for the branch name - '@same',
> > '@latest' and '@stable' which can be used to refer to the branch/tag
> > of a dependency.
> >   '@same' means the branch name should be the same as that of the
> > project being built and should be used when declaring dependencies
> > among projects in a release group.
> >   '@latest' and '@stable' refer to the corresponding branches
> > defined in the branch-rules.yml file, which can be found in
> > sysadmin/repo-metadata
> >
> > As a general rule, unless you require the absolute latest version of
> > another project in another release unit, it is recommended that you
> > use @stable.
> > Please avoid using explicit branch names unless absolutely necessary.
> >
> > At this time it is expected that the first set of Gitlab CI builds
> > using the new scripts may be conducted for Frameworks within the next
> > week or so, assuming the build of the seed jobs goes according to plan.
> >
> > Once the scripts have been proven successfully for Frameworks, we will
> > look at extending them to projects that depend only on Frameworks and
> > repositories within their release unit and finally will extend

Re: Gitlab CI - Inbound

[Ben Cooksley]

On Mon, Sep 6, 2021 at 11:03 AM Michael Reeves  wrote:

> How do we get a visual on exactly which lines are covered by auto testing
> and which aren't?
>

Please see
https://docs.gitlab.com/ee/user/project/merge_requests/test_coverage_visualization.html
for more details on how this works on Merge Requests.
To my knowledge this isn't exposed outside of Merge Requests.

Cheers,
Ben

Re: Gitlab CI - Inbound

[Ben Cooksley]

On Sun, Sep 5, 2021 at 10:22 PM David Faure  wrote:

> On dimanche 5 septembre 2021 12:11:05 CEST Ben Cooksley wrote:
> > It would be appreciated if people could please work on getting these
> files
> > populated in Frameworks (as everyone needs those) as well as in their own
> > repositories as they are required before we can proceed much further.
>
> Like this
> https://invent.kde.org/frameworks/kcoreaddons/-/merge_requests/135 ?
>

Yes - just two small tweaks needed for that one but it is otherwise good to
ship.


> For frameworks, I think we should be able to write a one-time script that
> generates .kde-ci.yml files using the dependencies listed in kde-build-
> metadata (and the platforms listed in metainfo.yaml) ?
>

Yes, that should work nicely (although that information now lives in
sysadmin/repo-metadata, dependencies folder)

Once we've transitioned across to the .kde-ci.yml files the existing
dependency metadata and branch group files will no longer be required.
(We'll need to work out a way of exporting this information for easy
consumption by kdesrc-build and other clients before it can be removed
though)


> --
> David Faure, fa...@kde.org, http://www.davidfaure.fr
> Working on KDE Frameworks 5
>
>
Thanks,
Ben

Re: Gitlab CI - Inbound

[Ben Cooksley]

On Sun, Sep 5, 2021 at 6:13 PM Ben Cooksley  wrote:

> Hi all,
>

Hi all,


> This morning after much work i'm happy to announce that the new generation
> CI scripts intended for use with Gitlab CI successfully completed their
> first build (of ECM, and then subsequently of KCoreAddons).
>
> This begins our first steps towards transferring CI over from Jenkins to
> Gitlab.
>
> These scripts are 'Gitlab native' and are designed to work in an
> environment where they will be running on merge requests, tags as well as
> all branches of our repositories - something the existing scripts were
> completely incompatible with. While there are still some infrastructural
> elements to put in place (such as 'seed jobs' to mass rebuild all projects
> after substantial changes and 'cleanup jobs' to remove old builds from the
> Package Registry) the core elements needed for initial testing of these
> scripts are now ready.
>

As an update, an initial version of the seed job tooling is now ready,
however testing that tooling requires that dependency information be
available.
This requires that the .kde-ci.yml files be populated in repositories.

It would be appreciated if people could please work on getting these files
populated in Frameworks (as everyone needs those) as well as in their own
repositories as they are required before we can proceed much further.


> For those curious, the results of those initials runs can be found at
> https://invent.kde.org/groups/teams/ci-artifacts/-/packages
>
> Due to the challenges associated with having to handle all of the
> different forms of build and the versioning of this information, these
> scripts also represent a radical change in how CI builds will be conducted
> - with a large part of the configuration of the jobs themselves, including
> information on project dependencies now shifting to files located within
> the repository themselves. Those who monitor commits to Frameworks
> repositories will have noticed the appearance of these '.kde-ci.yml' files
> in some repositories.
>
> To assist in the future rollout of the CI system it would be appreciated
> if projects could please begin setting up these files within their
> respective repositories.
> You can find a template detailing the available options at
> https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config-template.yml
>
> Where possible please avoid overriding the system defaults except where
> needed by your project (ie. don't just copy the template in full)
> The defaults mirror the template and can be found at
> https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config/global.yml
>
> In terms of the format of the 'Dependencies' section, please bear in mind
> the following:
> - For the 'on' section, the special value '@all' can be used to mean "All
> platforms" to avoid needing to update the file in the event additional
> platforms are added in the future
> - For the 'require' section:
>   1) Please only include the projects you *explicitly* depend on.
> Dependencies of your dependencies will be included automatically
>   2) When specifying a project, you must use the repository path on
> Gitlab. Legacy project paths (such as kde/workspace/plasma-desktop) are no
> longer in use and will not work.
>   3) There are three special values for the branch name - '@same',
> '@latest' and '@stable' which can be used to refer to the branch/tag of a
> dependency.
>   '@same' means the branch name should be the same as that of the
> project being built and should be used when declaring dependencies among
> projects in a release group.
>   '@latest' and '@stable' refer to the corresponding branches defined
> in the branch-rules.yml file, which can be found in sysadmin/repo-metadata
>
> As a general rule, unless you require the absolute latest version of
> another project in another release unit, it is recommended that you use
> @stable.
> Please avoid using explicit branch names unless absolutely necessary.
>
> At this time it is expected that the first set of Gitlab CI builds using
> the new scripts may be conducted for Frameworks within the next week or so,
> assuming the build of the seed jobs goes according to plan.
>
> Once the scripts have been proven successfully for Frameworks, we will
> look at extending them to projects that depend only on Frameworks and
> repositories within their release unit and finally will extend them to
> projects with more complex dependency requirements. It is expected that the
> switch will be flipped on the Frameworks builds sometime in the next week
> or two.
>
> Please let me know if you have any questions on the above.
>
> Thanks,
> Ben Cooksley
> KDE Sysadmin
>

Thanks,
Ben