[sysadmin/ci-management] /: Try to align the number of folders between seed jobs and normal CI jobs.

2022-04-06 Thread Ben Cooksley
Git commit 707e016918c0174235b1dc19883620f96f363572 by Ben Cooksley.
Committed on 07/04/2022 at 05:41.
Pushed by bcooksley into branch 'master'.

Try to align the number of folders between seed jobs and normal CI jobs.
This only affects Windows (guess everywhere else the path is absolute while on 
Windows it is relative) despite all our other jobs having the same layout.

CCMAIL: kde-frameworks-devel@kde.org

M  +8-4.gitlab-ci.yml

https://invent.kde.org/sysadmin/ci-management/commit/707e016918c0174235b1dc19883620f96f363572

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 96c828d..225ecf3 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -154,7 +154,8 @@ frameworks_windows_qt515:
   - .windows_qt515
   script:
 - . ci-utilities/resources/setup-msvc-env.ps1
-- python -u ci-utilities/seed-package-registry.py --seed-file 
latest/frameworks.yml --platform Windows
+- cd ..
+- python -u ci-management/ci-utilities/seed-package-registry.py 
--seed-file ci-management/latest/frameworks.yml --platform Windows
 
 ## Frameworks 6 jobs
 frameworks_suse_tumbleweed_qt62:
@@ -206,7 +207,8 @@ release_service_windows_qt515:
   - .windows_qt515
   script:
 - . ci-utilities/resources/setup-msvc-env.ps1
-- python -u ci-utilities/seed-package-registry.py --seed-file 
latest/release-service.yml --platform Windows
+- cd ..
+- python -u ci-management/ci-utilities/seed-package-registry.py 
--seed-file ci-management/latest/release-service.yml --platform Windows
 
 ## Plasma jobs
 
@@ -276,7 +278,8 @@ independent_release_windows_qt515:
   - .windows_qt515
   script:
 - . ci-utilities/resources/setup-msvc-env.ps1
-- python -u ci-utilities/seed-package-registry.py --seed-file 
latest/independent-release.yml --platform Windows
+- cd ..
+- python -u ci-management/ci-utilities/seed-package-registry.py 
--seed-file ci-management/latest/independent-release.yml --platform Windows
 
 ## PIM
 
@@ -311,4 +314,5 @@ pim_windows_qt515:
   - .windows_qt515
   script:
 - . ci-utilities/resources/setup-msvc-env.ps1
-- python -u ci-utilities/seed-package-registry.py --seed-file 
latest/pim.yml --platform Windows
+- cd ..
+- python -u ci-management/ci-utilities/seed-package-registry.py 
--seed-file ci-management/latest/pim.yml --platform Windows


Re: Unit tests all pass in Jenkins on Linux

2022-03-21 Thread Ben Cooksley
On Mon, Mar 21, 2022 at 9:43 AM David Faure  wrote:

> On dimanche 13 mars 2022 17:53:13 CET Ben Cooksley wrote:
> > On Mon, Mar 14, 2022 at 4:40 AM David Faure  wrote:
> > > After the recent discussions on state of CI, I fixed the last unittest
> > > failures (kio, purpose... + apol fixed ECM) so that
> > > https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/
> > > is all green^H^Hblue again.
> > > Please keep it that way!
> >
> > Thanks for looking into and fixing all of these David.
>
> Now I'd like to fix the remaining unittest failures on FreeBSD.
>
> I just fixed kcrash by reading the unittest code.
> However for the remaining ones, I need to actually debug on FreeBSD.
> Is there a FreeBSD virtual machine with the full setup already done for
> building KDE Frameworks, that I could either run locally or log into?
>

I believe Tobias may have some instructions on how to assemble a machine
(as these are what he used to assemble the VM images currently running the
FreeBSD environment on Gitlab and Jenkins).


>
> --
> David Faure, fa...@kde.org, http://www.davidfaure.fr
> Working on KDE Frameworks 5
>
>
Cheers,
Ben


Re: Windows unittests: KConfig

2022-03-21 Thread Ben Cooksley
On Mon, Mar 21, 2022 at 11:42 AM David Faure  wrote:

> On dimanche 20 mars 2022 22:13:17 CET Christoph Cullmann (cullmann.io)
> wrote:
> > On 2022-03-20 22:07, David Faure wrote:
> > > The KConfig unittests rely on DBus nowadays (for change notification).
> > > This is turned off on Android, and is a cmake option elsewhere,
> > > defaulting to ON.
> > >
> > > On Windows, I'm sure a lot of other modules rely on DBus, so I suppose
> > > it's just a matter of starting the dbus daemon for those modules that
> > > need it?
> > > Currently the kconfig tests fail for lack of dbus:
> > >
> > > 21:55:23  ERROR: The process "dbus-daemon.exe" not found.
> > >
> https://build.kde.org/job/Frameworks/view/Platform%20-%20WindowsMSVCQt5.15
> > > /job/kconfig/job/kf5-qt5%20WindowsMSVCQt5.15/217/console
> > Hi,
> >
> > actually, if one can just disable that on Windows, I would be rather in
> > favor of that.
> > Any dbus stuff is just a pain there and at least Okular/Kate/... as
> > packaged for Windows store avoid the use of any dbus calls.
>
> Makes sense.
>
> I was thinking "akonadi needs dbus anyway", but indeed, that doesn't apply
> to
> standalone apps, and the DBus stuff in KConfig seems to be mostly for
> workspace-level notifications (color theme changed, etc.).
>
> Made a merge request to turn this off on Windows by default:
> https://invent.kde.org/frameworks/kconfig/-/merge_requests/120


If we could head in the direction of being free of D-Bus on Windows (and
Mac as well I guess) then that would definitely be preferrable.
D-Bus makes no sense on those platforms and has only been a cause of issues.

Cheers,
Ben


>
>
> --
> David Faure, fa...@kde.org, http://www.davidfaure.fr
> Working on KDE Frameworks 5
>
>
>
>


Re: Unit tests all pass in Jenkins on Linux

2022-03-14 Thread Ben Cooksley
On Mon, Mar 14, 2022 at 4:44 PM Eduardo de Souza Cruz <
eduardo.c...@kdemail.net> wrote:

> Hi,
>
> Regarding the krunner timer-based test which I authored:
>
> We could just delete the few lines that are enforcing those upper-limit
> timeouts and add some comments explaining that in real life those timeouts
> should have been met, I don't think we need to delete the entire test. I
> was explicitly asked to write a test when I submitted the MR to enforce
> that this functionally wouldn't regress in the future.
>
> Also, I'm wondering... if we happen to have a compilation directive that's
> #defined in this CI environment only, or some framework function that
> returns weather we are in the CI environment or not, we could put some
> #ifndef (or if's) on those few upper-limit timeout lines and avoid
> compiling/running just those lines in the CI environment. I'm not familiar
> with this environment so I don't know if there is such a thing, I'm just
> wondering...
>

Not at this time i'm afraid.

There are some environment variables you can look to but those differ
between our Jenkins and Gitlab setups.


>
> I'm available to submit a quick MR if you'd like me to, just give me some
> directions for what you'd like me to do.
>
> [],
> Eduardo
>

Cheers,
Ben


>
> --
> *From:* Ben Cooksley 
> *Sent:* Sunday, March 13, 2022 1:53 PM
> *To:* KDE Frameworks 
> *Cc:* eduardo.c...@kdemail.net 
> *Subject:* Re: Unit tests all pass in Jenkins on Linux
>
> On Mon, Mar 14, 2022 at 4:40 AM David Faure  wrote:
>
> After the recent discussions on state of CI, I fixed the last unittest
> failures (kio, purpose... + apol fixed ECM) so that
> https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/
> is all green^H^Hblue again.
> Please keep it that way!
>
>
> Thanks for looking into and fixing all of these David.
>
>
>
> Note however that
>
> * kwayland has a flaky test:
>
>
> https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/job/kwayland/job/kf5-qt5%20SUSEQt5.15/171/testReport/junit/projectroot.autotests/client/kwayland_testDataDevice/
>
> FAIL!  : TestDataDevice::testReplaceSource() Compared values are not the
> same
>Actual   (selectionOfferedSpy.count()): 1
>Expected (2)  : 2
>Loc: [autotests/client/test_datadevice.cpp(557)]
>
> Who can look at this one? git log mostly shows Martin Flöser <
> mgraess...@kde.org>
> who I think isn't active anymore?
>
>
> Not sure if it applies to KWayland as well, but I know that KWin has load
> sensitive tests (which is why the Gitlab .kde-ci.yml files support the flag
> tests-load-sensitive)
> If this test appears to be flaky, then it is quite possible that it is
> load sensitive as well.
>
>
>
> * krunner has a flaky test [2] because it measures time spent and expects
> small values like 65ms
> (I changed that one to 100ms), 250ms, 300ms. With only 10% safety margins.
> On a busy CI system,
> this is bound to fail regularly, even with bigger safety margins. In my
> experience this kind of test
> is just not possible (we're not running on a real time OS), I vote for
> removing the test.
> CC'ing Eduardo.
>
>
> https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/job/krunner/job/kf5-qt5%20SUSEQt5.15/325/testReport/junit/projectroot/autotests/runnermanagertest/
>
>
> Yes, that will definitely fail more often than not - your only way to make
> sure tests like this pass on our CI system is to
> set tests-load-sensitive=True (in Gitlab CI)
> Note however that option should be avoided where possible as it means your
> build will stop and wait for load to fall to low levels before proceeding
> with running tests - which blocks a CI worker slot from being used by
> another project.
>
> I'd also be in favour of removing this test.
>
>
>
> --
> David Faure, fa...@kde.org, http://www.davidfaure.fr
> Working on KDE Frameworks 5
>
>
>
>
> Cheers,
> Ben
>


Re: KDE CI: Frameworks » kquickcharts » kf5-qt5 WindowsMSVCQt5.15 - Build # 101 - Still Failing!

2022-03-14 Thread Ben Cooksley
Hi all,

Can someone please confirm whether it is expected that KQuickCharts is
Linux/FreeBSD only given that it is QtQuick based (and therefore should be
fairly platform agnostic?)

Cheers,
Ben

On Mon, Mar 14, 2022 at 6:01 PM CI System  wrote:

> *BUILD FAILURE*
> Build URL
> https://build.kde.org/job/Frameworks/job/kquickcharts/job/kf5-qt5%20WindowsMSVCQt5.15/101/
> Project: kf5-qt5 WindowsMSVCQt5.15
> Date of build: Mon, 14 Mar 2022 05:00:46 +
> Build duration: 23 sec and counting
> * CONSOLE OUTPUT *
> [...truncated 140 lines...]
> [2022-03-14T05:01:02.160Z] PROCESSOR_REVISION = '0102'
> [2022-03-14T05:01:02.160Z] PROGRAMDATA = 'C:\ProgramData'
> [2022-03-14T05:01:02.160Z] PROGRAMFILES = 'C:\Program Files'
> [2022-03-14T05:01:02.160Z] PROGRAMFILES(X86) = 'C:\Program Files (x86)'
> [2022-03-14T05:01:02.160Z] PROGRAMW6432 = 'C:\Program Files'
> [2022-03-14T05:01:02.160Z] PROMPT = '$P$G'
> [2022-03-14T05:01:02.160Z] PSMODULEPATH =
> '%ProgramFiles%\WindowsPowerShell\Modules;C:\WINDOWS\system32\WindowsPowerShell\v1.0\Modules'
> [2022-03-14T05:01:02.160Z] PUBLIC = 'C:\Users\Public'
> [2022-03-14T05:01:02.160Z] RUN_ARTIFACTS_DISPLAY_URL = '
> https://build.kde.org/job/Frameworks/job/kquickcharts/job/kf5-qt5%20WindowsMSVCQt5.15/101/display/redirect?page=artifacts
> '
> [2022-03-14T05:01:02.160Z] RUN_CHANGES_DISPLAY_URL = '
> https://build.kde.org/job/Frameworks/job/kquickcharts/job/kf5-qt5%20WindowsMSVCQt5.15/101/display/redirect?page=changes
> '
> [2022-03-14T05:01:02.160Z] RUN_DISPLAY_URL = '
> https://build.kde.org/job/Frameworks/job/kquickcharts/job/kf5-qt5%20WindowsMSVCQt5.15/101/display/redirect
> '
> [2022-03-14T05:01:02.160Z] RUN_TESTS_DISPLAY_URL = '
> https://build.kde.org/job/Frameworks/job/kquickcharts/job/kf5-qt5%20WindowsMSVCQt5.15/101/display/redirect?page=tests
> '
> [2022-03-14T05:01:02.160Z] STAGE_NAME = 'Configuring Build'
> [2022-03-14T05:01:02.160Z] SYSTEMDRIVE = 'C:'
> [2022-03-14T05:01:02.160Z] SYSTEMROOT = 'C:\WINDOWS'
> [2022-03-14T05:01:02.160Z] TEMP = 'C:\Users\Jenkins\AppData\Local\Temp'
> [2022-03-14T05:01:02.160Z] TMP = 'C:\Users\Jenkins\AppData\Local\Temp'
> [2022-03-14T05:01:02.160Z] UCRTVERSION = '10.0.19041.0'
> [2022-03-14T05:01:02.160Z] UNIVERSALCRTSDKDIR = 'C:\Program Files
> (x86)\Windows Kits\10\'
> [2022-03-14T05:01:02.160Z] USERDOMAIN = 'DESKTOP-9TVNRIT'
> [2022-03-14T05:01:02.160Z] USERNAME = 'Jenkins'
> [2022-03-14T05:01:02.160Z] USERPROFILE = 'C:\Users\Jenkins'
> [2022-03-14T05:01:02.160Z] VCIDEINSTALLDIR = 'C:\Program Files
> (x86)\Microsoft Visual Studio\2019\Professional\Common7\IDE\VC\'
> [2022-03-14T05:01:02.160Z] VCINSTALLDIR = 'C:\Program Files
> (x86)\Microsoft Visual Studio\2019\Professional\VC\'
> [2022-03-14T05:01:02.160Z] VCTOOLSINSTALLDIR = 'C:\Program Files
> (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.29.30037\'
> [2022-03-14T05:01:02.160Z] VCTOOLSREDISTDIR = 'C:\Program Files
> (x86)\Microsoft Visual Studio\2019\Professional\VC\Redist\MSVC\14.29.30036\'
> [2022-03-14T05:01:02.160Z] VCTOOLSVERSION = '14.29.30037'
> [2022-03-14T05:01:02.160Z] VISUALSTUDIOVERSION = '16.0'
> [2022-03-14T05:01:02.160Z] VS160COMNTOOLS = 'C:\Program Files
> (x86)\Microsoft Visual Studio\2019\Professional\Common7\Tools\'
> [2022-03-14T05:01:02.160Z] VSCMD_ARG_APP_PLAT = 'Desktop'
> [2022-03-14T05:01:02.160Z] VSCMD_ARG_HOST_ARCH = 'x64'
> [2022-03-14T05:01:02.160Z] VSCMD_ARG_TGT_ARCH = 'x64'
> [2022-03-14T05:01:02.160Z] VSCMD_VER = '16.10.2'
> [2022-03-14T05:01:02.160Z] VSINSTALLDIR = 'C:\Program Files
> (x86)\Microsoft Visual Studio\2019\Professional\'
> [2022-03-14T05:01:02.160Z] WINDIR = 'C:\WINDOWS'
> [2022-03-14T05:01:02.160Z] WINDOWSLIBPATH = 'C:\Program Files
> (x86)\Windows Kits\10\UnionMetadata\10.0.19041.0;C:\Program Files
> (x86)\Windows Kits\10\References\10.0.19041.0'
> [2022-03-14T05:01:02.160Z] WINDOWSSDKBINPATH = 'C:\Program Files
> (x86)\Windows Kits\10\bin\'
> [2022-03-14T05:01:02.160Z] WINDOWSSDKDIR = 'C:\Program Files (x86)\Windows
> Kits\10\'
> [2022-03-14T05:01:02.160Z] WINDOWSSDKLIBVERSION = '10.0.19041.0\'
> [2022-03-14T05:01:02.161Z] WINDOWSSDKVERBINPATH = 'C:\Program Files
> (x86)\Windows Kits\10\bin\10.0.19041.0\'
> [2022-03-14T05:01:02.161Z] WINDOWSSDKVERSION = '10.0.19041.0\'
> [2022-03-14T05:01:02.161Z] WORKSPACE = 'C:/CI/Job Build'
> [2022-03-14T05:01:02.161Z] WORKSPACE_TMP = 'C:/CI/Job Build@tmp'
> [2022-03-14T05:01:02.161Z] __DEVINIT_PATH = 'C:\Program Files
> (x86)\Microsoft Visual
> Studio\2019\Professional\Common7\Tools\devinit\devinit.exe'
> [2022-03-14T05:01:02.161Z] __DOTNET_ADD_64BIT = '1'
> [2022-03-14T05:01:02.161Z] __DOTNET_PREFERRED_BITNESS = '64'
> [2022-03-14T05:01:02.161Z] __VSCMD_PREINIT_PATH = 'C:\Program Files
> (x86)\Common Files\Oracle\Java\javapath;C:\Program
> Files\Python38\Scripts\;C:\Program
> Files\Python38\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program
> 

Re: Unit tests all pass in Jenkins on Linux

2022-03-13 Thread Ben Cooksley
On Mon, Mar 14, 2022 at 4:40 AM David Faure  wrote:

> After the recent discussions on state of CI, I fixed the last unittest
> failures (kio, purpose... + apol fixed ECM) so that
> https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/
> is all green^H^Hblue again.
> Please keep it that way!
>

Thanks for looking into and fixing all of these David.


>
> Note however that
>
> * kwayland has a flaky test:
>
>
> https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/job/kwayland/job/kf5-qt5%20SUSEQt5.15/171/testReport/junit/projectroot.autotests/client/kwayland_testDataDevice/
>
> FAIL!  : TestDataDevice::testReplaceSource() Compared values are not the
> same
>Actual   (selectionOfferedSpy.count()): 1
>Expected (2)  : 2
>Loc: [autotests/client/test_datadevice.cpp(557)]
>
> Who can look at this one? git log mostly shows Martin Flöser <
> mgraess...@kde.org>
> who I think isn't active anymore?
>

Not sure if it applies to KWayland as well, but I know that KWin has load
sensitive tests (which is why the Gitlab .kde-ci.yml files support the flag
tests-load-sensitive)
If this test appears to be flaky, then it is quite possible that it is load
sensitive as well.


>
> * krunner has a flaky test [2] because it measures time spent and expects
> small values like 65ms
> (I changed that one to 100ms), 250ms, 300ms. With only 10% safety margins.
> On a busy CI system,
> this is bound to fail regularly, even with bigger safety margins. In my
> experience this kind of test
> is just not possible (we're not running on a real time OS), I vote for
> removing the test.
> CC'ing Eduardo.
>
>
> https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/job/krunner/job/kf5-qt5%20SUSEQt5.15/325/testReport/junit/projectroot/autotests/runnermanagertest/


Yes, that will definitely fail more often than not - your only way to make
sure tests like this pass on our CI system is to
set tests-load-sensitive=True (in Gitlab CI)
Note however that option should be avoided where possible as it means your
build will stop and wait for load to fall to low levels before proceeding
with running tests - which blocks a CI worker slot from being used by
another project.

I'd also be in favour of removing this test.


>
> --
> David Faure, fa...@kde.org, http://www.davidfaure.fr
> Working on KDE Frameworks 5
>
>
>
>
Cheers,
Ben


Re: Hard dependency on plasma-framework in Alkimia

2022-03-10 Thread Ben Cooksley
On Thu, Mar 10, 2022 at 9:12 AM Thomas Baumgart  wrote:

> Hi Ben,
>

Hi Thomas,


>
> On Mittwoch, 9. März 2022 11:09:53 CET Ben Cooksley wrote:
>
> > Hi Thomas,
> >
> > Recently some changes were introduced to Frameworks which means that they
> > now enforce more rigorously the platforms on which they build.
> >
> > This means that Plasma Framework is no longer available on Windows -
> > unfortunately though it looks like Alkimia has a mandatory dependency on
> > Plasma Framework.
> >
> > Are you able to make this optional or should we disable Windows CI builds
> > for Alkimia (and the projects that depend on it)?
>
> I just accepted https://invent.kde.org/office/alkimia/-/merge_requests/17
> so you
> can re-enable Alkimia builds once this is merged.
>

Thanks for getting that merged in.


>
> p.s. quite a short time frame from notice until cut-off for my taste.
>

That was a temporary measure only as I wanted to make sure there weren't
any other issues in other projects that also needed addressing.
I wanted to get this sorted as soon as reasonably possible to allow for
progress to be made on Windows CI on Gitlab.


>
> --
>
> Regards
>
> Thomas Baumgart
>
> https://www.signal.org/   Signal, the better WhatsApp
> -
> Linux, because rebooting is for adding new hardware ...
> -
>

Cheers,
Ben


[sysadmin/ci-tooling] local-metadata: Re-enable Alkimia CI on Windows now that the hard dependency issues have been resolved.

2022-03-10 Thread Ben Cooksley
Git commit 9ef1b7c455e1eeba332593eb8cd3e722bca1d33a by Ben Cooksley.
Committed on 10/03/2022 at 08:42.
Pushed by bcooksley into branch 'master'.

Re-enable Alkimia CI on Windows now that the hard dependency issues have been 
resolved.

CCMAIL: kde-frameworks-devel@kde.org
CCMAIL: kmymoney-de...@kde.org

M  +0-1local-metadata/project-ignore-rules.yaml

https://invent.kde.org/sysadmin/ci-tooling/commit/9ef1b7c455e1eeba332593eb8cd3e722bca1d33a

diff --git a/local-metadata/project-ignore-rules.yaml 
b/local-metadata/project-ignore-rules.yaml
index e42d203..5d7ecfd 100644
--- a/local-metadata/project-ignore-rules.yaml
+++ b/local-metadata/project-ignore-rules.yaml
@@ -22,7 +22,6 @@
 - 'kde/workspace/libksysguard'
 - 'kde/kdenetwork/kaccounts-integration'
 - 'extragear/libs/pulseaudio-qt'
-- 'extragear/office/alkimia'
 - "kde/pim/libkleo"
 
 'FreeBSDQt5.15':


[sysadmin/ci-tooling] local-metadata: Alkimia has a hard dependency on Plasma Framework by default on Windows, and Plasma Framework is no longer available on Windows.

2022-03-09 Thread Ben Cooksley
Git commit 9b120b06156140e0c2657de5f9306620812f7d40 by Ben Cooksley.
Committed on 09/03/2022 at 17:39.
Pushed by bcooksley into branch 'master'.

Alkimia has a hard dependency on Plasma Framework by default on Windows, and 
Plasma Framework is no longer available on Windows.
Therefore we have to disable Alkimia builds on Windows.

CCMAIL: kde-frameworks-devel@kde.org
CCMAIL: kmymoney-de...@kde.org

M  +1-0local-metadata/project-ignore-rules.yaml

https://invent.kde.org/sysadmin/ci-tooling/commit/9b120b06156140e0c2657de5f9306620812f7d40

diff --git a/local-metadata/project-ignore-rules.yaml 
b/local-metadata/project-ignore-rules.yaml
index 5d7ecfd..e42d203 100644
--- a/local-metadata/project-ignore-rules.yaml
+++ b/local-metadata/project-ignore-rules.yaml
@@ -22,6 +22,7 @@
 - 'kde/workspace/libksysguard'
 - 'kde/kdenetwork/kaccounts-integration'
 - 'extragear/libs/pulseaudio-qt'
+- 'extragear/office/alkimia'
 - "kde/pim/libkleo"
 
 'FreeBSDQt5.15':


[sysadmin/ci-tooling] local-metadata: KTextEditor has severed it's dependency on KAuth, so we can restore it's Windows builds.

2022-03-09 Thread Ben Cooksley
Git commit 38521ec9142faa672be942ffe9f0419414d35588 by Ben Cooksley.
Committed on 09/03/2022 at 17:37.
Pushed by bcooksley into branch 'master'.

KTextEditor has severed it's dependency on KAuth, so we can restore it's 
Windows builds.

CCMAIL: kde-frameworks-devel@kde.org

M  +0-1local-metadata/project-ignore-rules.yaml

https://invent.kde.org/sysadmin/ci-tooling/commit/38521ec9142faa672be942ffe9f0419414d35588

diff --git a/local-metadata/project-ignore-rules.yaml 
b/local-metadata/project-ignore-rules.yaml
index e8eab67..5d7ecfd 100644
--- a/local-metadata/project-ignore-rules.yaml
+++ b/local-metadata/project-ignore-rules.yaml
@@ -16,7 +16,6 @@
 - 'frameworks/kglobalaccel'
 - 'frameworks/kded'
 - 'frameworks/kdelibs4support'
-- 'frameworks/ktexteditor'
 - 'frameworks/plasma-framework'
 - 'frameworks/krunner'
 - 'kde/applications/baloo-widgets'



Hard dependency on plasma-framework in Alkimia

2022-03-09 Thread Ben Cooksley
Hi Thomas,

Recently some changes were introduced to Frameworks which means that they
now enforce more rigorously the platforms on which they build.

This means that Plasma Framework is no longer available on Windows -
unfortunately though it looks like Alkimia has a mandatory dependency on
Plasma Framework.

Are you able to make this optional or should we disable Windows CI builds
for Alkimia (and the projects that depend on it)?

Cheers,
Ben


[sysadmin/ci-tooling] local-metadata: KRunner requires Plasma Framework, so it effectively has a hard dependency on KGlobalAccel too.

2022-03-09 Thread Ben Cooksley
Git commit 6fd7e3b8adb9789c842fdb4f3361b95db53949b3 by Ben Cooksley.
Committed on 09/03/2022 at 09:31.
Pushed by bcooksley into branch 'master'.

KRunner requires Plasma Framework, so it effectively has a hard dependency on 
KGlobalAccel too.
Disable it as well on Windows.

CCMAIL: kde-frameworks-devel@kde.org

M  +1-0local-metadata/project-ignore-rules.yaml

https://invent.kde.org/sysadmin/ci-tooling/commit/6fd7e3b8adb9789c842fdb4f3361b95db53949b3

diff --git a/local-metadata/project-ignore-rules.yaml 
b/local-metadata/project-ignore-rules.yaml
index e3c22b2..e8eab67 100644
--- a/local-metadata/project-ignore-rules.yaml
+++ b/local-metadata/project-ignore-rules.yaml
@@ -18,6 +18,7 @@
 - 'frameworks/kdelibs4support'
 - 'frameworks/ktexteditor'
 - 'frameworks/plasma-framework'
+- 'frameworks/krunner'
 - 'kde/applications/baloo-widgets'
 - 'kde/workspace/libksysguard'
 - 'kde/kdenetwork/kaccounts-integration'


[sysadmin/ci-tooling] local-metadata: Seems that lots of things require KGlobalAccel - also disable the build of Plasma Framework on Windows.

2022-03-09 Thread Ben Cooksley
Git commit 7671c8eb1f7bb2dcab74048adf03aceeafab336d by Ben Cooksley.
Committed on 09/03/2022 at 08:45.
Pushed by bcooksley into branch 'master'.

Seems that lots of things require KGlobalAccel - also disable the build of 
Plasma Framework on Windows.

CCMAIL: kde-frameworks-devel@kde.org

M  +1-0local-metadata/project-ignore-rules.yaml

https://invent.kde.org/sysadmin/ci-tooling/commit/7671c8eb1f7bb2dcab74048adf03aceeafab336d

diff --git a/local-metadata/project-ignore-rules.yaml 
b/local-metadata/project-ignore-rules.yaml
index dc0496c..e3c22b2 100644
--- a/local-metadata/project-ignore-rules.yaml
+++ b/local-metadata/project-ignore-rules.yaml
@@ -17,6 +17,7 @@
 - 'frameworks/kded'
 - 'frameworks/kdelibs4support'
 - 'frameworks/ktexteditor'
+- 'frameworks/plasma-framework'
 - 'kde/applications/baloo-widgets'
 - 'kde/workspace/libksysguard'
 - 'kde/kdenetwork/kaccounts-integration'


[sysadmin/ci-tooling] local-metadata: KTextEditor has a hard dependency on KAuth which is no longer available on Windows.

2022-03-09 Thread Ben Cooksley
Git commit be4b7627c94e4c51e43e4adf28909bdfbd9cbecc by Ben Cooksley.
Committed on 09/03/2022 at 08:02.
Pushed by bcooksley into branch 'master'.

KTextEditor has a hard dependency on KAuth which is no longer available on 
Windows.
Disable it on Windows as well.

CCMAIL: kde-frameworks-devel@kde.org
CCMAIL: kwrite-de...@kde.org

M  +1-0local-metadata/project-ignore-rules.yaml

https://invent.kde.org/sysadmin/ci-tooling/commit/be4b7627c94e4c51e43e4adf28909bdfbd9cbecc

diff --git a/local-metadata/project-ignore-rules.yaml 
b/local-metadata/project-ignore-rules.yaml
index 33dd1e3..dc0496c 100644
--- a/local-metadata/project-ignore-rules.yaml
+++ b/local-metadata/project-ignore-rules.yaml
@@ -16,6 +16,7 @@
 - 'frameworks/kglobalaccel'
 - 'frameworks/kded'
 - 'frameworks/kdelibs4support'
+- 'frameworks/ktexteditor'
 - 'kde/applications/baloo-widgets'
 - 'kde/workspace/libksysguard'
 - 'kde/kdenetwork/kaccounts-integration'


KStars on Windows

2022-03-08 Thread Ben Cooksley
Hi Jasem,

Recently some changes were introduced to Frameworks which means that they
now enforce more rigorously the platforms on which they build.

This means that KAuth is no longer available on Windows - unfortunately
though it looks like KStars has a mandatory dependency on KAuth.

Are you able to make this optional or should we disable Windows CI builds
for KStars?

Cheers,
Ben


[sysadmin/ci-tooling] local-metadata: kdelibs4support has a hard dependency on kglobalaccel (not sure why) which is no longer available on Windows.

2022-03-08 Thread Ben Cooksley
Git commit 8e43c6c798e16e45e5cc0ad8f148c0c8df6d5fd9 by Ben Cooksley.
Committed on 09/03/2022 at 07:27.
Pushed by bcooksley into branch 'master'.

kdelibs4support has a hard dependency on kglobalaccel (not sure why) which is 
no longer available on Windows.
Therefore blacklist it on Windows.

CCMAIL: kde-frameworks-devel@kde.org

M  +1-0local-metadata/project-ignore-rules.yaml

https://invent.kde.org/sysadmin/ci-tooling/commit/8e43c6c798e16e45e5cc0ad8f148c0c8df6d5fd9

diff --git a/local-metadata/project-ignore-rules.yaml 
b/local-metadata/project-ignore-rules.yaml
index 8f7bf1b..33dd1e3 100644
--- a/local-metadata/project-ignore-rules.yaml
+++ b/local-metadata/project-ignore-rules.yaml
@@ -15,6 +15,7 @@
 - 'frameworks/baloo'
 - 'frameworks/kglobalaccel'
 - 'frameworks/kded'
+- 'frameworks/kdelibs4support'
 - 'kde/applications/baloo-widgets'
 - 'kde/workspace/libksysguard'
 - 'kde/kdenetwork/kaccounts-integration'


[sysadmin/ci-tooling] local-metadata: Block frameworks/kded on Windows too.

2022-03-08 Thread Ben Cooksley
Git commit 712ed90054285fa02a67c1a6fb92b66fc440146f by Ben Cooksley.
Committed on 08/03/2022 at 17:48.
Pushed by bcooksley into branch 'master'.

Block frameworks/kded on Windows too.

CCMAIL: kde-frameworks-devel@kde.org

M  +1-0local-metadata/project-ignore-rules.yaml

https://invent.kde.org/sysadmin/ci-tooling/commit/712ed90054285fa02a67c1a6fb92b66fc440146f

diff --git a/local-metadata/project-ignore-rules.yaml 
b/local-metadata/project-ignore-rules.yaml
index 2f93f05..1430b83 100644
--- a/local-metadata/project-ignore-rules.yaml
+++ b/local-metadata/project-ignore-rules.yaml
@@ -14,6 +14,7 @@
 - 'frameworks/kactivities-stats'
 - 'frameworks/baloo'
 - 'frameworks/kglobalaccel'
+- 'frameworks/kded'
 - 'kde/applications/baloo-widgets'
 - 'kde/workspace/libksysguard'
 - 'kde/kdenetwork/kaccounts-integration'


Re: CI Repairs

2022-03-08 Thread Ben Cooksley
On Tue, Mar 8, 2022 at 11:20 PM Volker Krause  wrote:

> On Dienstag, 8. März 2022 08:54:38 CET Ben Cooksley wrote:
> > This evening i've repaired several issues that were causing builds to
> fail
> > on the main Jenkins CI system. This includes a broken Windows builder
> > (causing Windows builds to periodically fail) and a hung FreeBSD builder
> > (which was consuming half a CPU and preventing KWin CI jobs from
> completing)
>
> Thank you! Would that also explain the problems we are seeing with the
> FreeBSD
> seed job?
>

Unfortunately no.

Most of the issues i've seen with the seed jobs on FreeBSD/Windows have
been due to CMake erroring out as a consequence of the platform not being
supported.
I've been fixing those as we hit them (by disabling the build of that
project on that platform)

Looks like FreeBSD passes now.


>
> > Replacement runs have been initiated for all projects.
> >
> > So far all appears well, however a number of projects appear to have CI
> > regressions on one or more platforms due to:
> > - Use of exceptions (KMail)
>
> For this I'm not finding an explanation, it started after a completely
> unrelated merge commit and the exception using code is in an Akonadi
> header
> that is used all over the place.
>
> > - Use of an ECM version that does not exist (print-manager)
>
> Fixed.
>
> > - Use of C++ functionality that is not enabled (okular on Windows)
>
> https://invent.kde.org/graphics/okular/-/merge_requests/582
>
> > - Something to do with qobject_cast (akonadiconsole)
>
> We had similar issues in other modules over the past two weeks or so due
> to
> the include install layout changes not being propagated fully yet. That's
> what
> made me initially look at the FreeBSD seed job.
>
> Regards,
> Volker


Thanks,
Ben


[sysadmin/ci-tooling] local-metadata: Block KGlobalAccel on Windows too.

2022-03-08 Thread Ben Cooksley
Git commit 8d376b50b67f572dab60519a9ce7b3ba3a9f744c by Ben Cooksley.
Committed on 08/03/2022 at 17:13.
Pushed by bcooksley into branch 'master'.

Block KGlobalAccel on Windows too.

CCMAIL: kde-frameworks-devel@kde.org

M  +1-0local-metadata/project-ignore-rules.yaml

https://invent.kde.org/sysadmin/ci-tooling/commit/8d376b50b67f572dab60519a9ce7b3ba3a9f744c

diff --git a/local-metadata/project-ignore-rules.yaml 
b/local-metadata/project-ignore-rules.yaml
index f970914..2f93f05 100644
--- a/local-metadata/project-ignore-rules.yaml
+++ b/local-metadata/project-ignore-rules.yaml
@@ -13,6 +13,7 @@
 - 'frameworks/kwayland'
 - 'frameworks/kactivities-stats'
 - 'frameworks/baloo'
+- 'frameworks/kglobalaccel'
 - 'kde/applications/baloo-widgets'
 - 'kde/workspace/libksysguard'
 - 'kde/kdenetwork/kaccounts-integration'


[sysadmin/ci-tooling] local-metadata: Block KAuth on Windows.

2022-03-08 Thread Ben Cooksley
Git commit 1322a5f4ae7335bf31a288189a455dff4c34c83c by Ben Cooksley.
Committed on 08/03/2022 at 09:36.
Pushed by bcooksley into branch 'master'.

Block KAuth on Windows.

CCMAIL: kde-frameworks-devel@kde.org

M  +1-0local-metadata/project-ignore-rules.yaml

https://invent.kde.org/sysadmin/ci-tooling/commit/1322a5f4ae7335bf31a288189a455dff4c34c83c

diff --git a/local-metadata/project-ignore-rules.yaml 
b/local-metadata/project-ignore-rules.yaml
index 36b34c4..f970914 100644
--- a/local-metadata/project-ignore-rules.yaml
+++ b/local-metadata/project-ignore-rules.yaml
@@ -7,6 +7,7 @@
 - 'frameworks/networkmanager-qt'
 - 'frameworks/modemmanager-qt'
 - 'frameworks/bluez-qt'
+- 'frameworks/kauth'
 - 'frameworks/kdesu'
 - 'frameworks/kpty'
 - 'frameworks/kwayland'


[sysadmin/ci-tooling] local-metadata: Ensure we do not use frameworks/bluez-qt on FreeBSD

2022-03-08 Thread Ben Cooksley
Git commit fc4c56fed4466c1adf26b570b000edb1791e5e43 by Ben Cooksley.
Committed on 08/03/2022 at 08:59.
Pushed by bcooksley into branch 'master'.

Ensure we do not use frameworks/bluez-qt on FreeBSD

CCMAIL: kde-frameworks-devel@kde.org

M  +1-0local-metadata/project-ignore-rules.yaml

https://invent.kde.org/sysadmin/ci-tooling/commit/fc4c56fed4466c1adf26b570b000edb1791e5e43

diff --git a/local-metadata/project-ignore-rules.yaml 
b/local-metadata/project-ignore-rules.yaml
index aa3ac42..36b34c4 100644
--- a/local-metadata/project-ignore-rules.yaml
+++ b/local-metadata/project-ignore-rules.yaml
@@ -22,6 +22,7 @@
 - 'kdesupport/polkit-qt-1'
 - 'frameworks/networkmanager-qt'
 - 'frameworks/modemmanager-qt'
+- 'frameworks/bluez-qt'
 - 'kde/workspace/plymouth-kcm'
 - 'kde/workspace/plasma-nm'
 - 'kde/workspace/plasma-vault'


Re: Critical Denial of Service bugs in Discover

2022-03-08 Thread Ben Cooksley
On Mon, Mar 7, 2022 at 1:16 PM Aleix Pol  wrote:

>
> On Sat, Mar 5, 2022 at 8:36 AM Ben Cooksley  wrote:
>
>> On Fri, Mar 4, 2022 at 12:49 AM Aleix Pol  wrote:
>>
>>> I'd say wireshark is too low level for what the problem is here. We are
>>> talking about having too many HTTP requests for specific URLs.
>>>
>>
>> Correct, I guess the difference in our approaches comes from a "before
>> release" to a "monitor after release" angle to things.
>> I'd like to see increased scrutiny during the development process as well
>> to make sure that we release code that operates properly from Day 1.
>>
>
> A way to do this could be using commit hooks that do not allow to reach
> certain services. (which we discussed in private chat).
> We could also analyse at cmake time the knsrc files we install, but this
> has a very limited and specific scope.
>

I've now applied two checks as part of the hooks which will hopefully catch
anything new being introduced.
We still need to ensure that anything pre-existing is sorted out of course.


>
>
>> I can think two main measures:
>>> - Trigger an alarm (an e-mail notification?) if there's a specific
>>> UserAgent that has a specific portion of the queries we have in a specific
>>> day in the services we care about.
>>> - Offer plots to see how queries by UserAgent evolve over the last
>>> couple of months (or couple of years).
>>>
>>
>> At the moment our ability to analyse our logs is somewhat limited by our
>> Privacy Policy - https://kde.org/privacypolicy/
>> Currently we don't have any provision for long term storage of
>> this information even on an aggregated basis - so we would need to update
>> this first.
>>
>
> Hopefully the NDA should help here and it doesn't seem all that far away.
> I know Neofytos and Ade have been working on it lately.
>

The privacy policy will still need to be updated, but that can form part of
the puzzle yes.


>
> The second issue there is that we are transitioning users to contact a CDN
>> based endpoint (which is substantially more scalable).
>> This does mean we lose visibility on data such as User Agents and the
>> URLs being impacted though as we only get aggregated data unless we ask for
>> raw logs - which makes implementing something like what you've described
>> much harder.
>>
>
> That does seem like a stopper. Still, it seems like it's not that big of a
> problem when there is a CDN, so we better worry about the other cases.
>

We should still be reasonable to the CDN of course, but it makes it much
more managable yes.


>
> Aleix
>

Cheers,
Ben


[sysadmin/repo-management] hooks: Implement two additional checks as part of our hooks:

2022-03-08 Thread Ben Cooksley
Git commit 919f7163102835d46c81593251fd0689fea71640 by Ben Cooksley.
Committed on 08/03/2022 at 08:13.
Pushed by bcooksley into branch 'master'.

Implement two additional checks as part of our hooks:

1) Require that all *.knsrc file changes be reviewed by a Sysadmin if landing 
in a non-work branch
2) Alert Sysadmin if anyone mentions download.kde.org or files.kde.org in the 
text of their code.

CCMAIL: kde-frameworks-devel@kde.org
CCMAIL: plasma-de...@kde.org

M  +14   -0hooks/hooklib.py
M  +16   -2hooks/invent.pre-receive

https://invent.kde.org/sysadmin/repo-management/commit/919f7163102835d46c81593251fd0689fea71640

diff --git a/hooks/hooklib.py b/hooks/hooklib.py
index 062b0e3..df04d96 100644
--- a/hooks/hooklib.py
+++ b/hooks/hooklib.py
@@ -706,6 +706,10 @@ class CommitEmailNotifier:
 if self.checker and (self.checker.license_problem or 
self.checker.commit_problem):
 cc_addresses.append( self.commit.committer_email )
 
+# Add Sysadmin if infrastructure problems have been found
+if self.checker and self.checker.infra_problem):
+cc_addresses.append( 'sysad...@kde.org' )
+
 if self.keywords['email_gui']:
 cc_addresses.append( 'kde-doc-engl...@kde.org' )
 
@@ -1002,6 +1006,10 @@ class CommitChecker:
 def commit_problem(self):
 return self._commit_problem
 
+@property
+def infra_problem(self):
+return self._infra_problem
+
 @property
 def commit_notes(self):
 return self._commit_notes
@@ -1219,6 +1227,7 @@ class CommitChecker:
 
 # Initialise
 self._license_problem = False
+self._infra_problem = False
 self._commit_problem = False
 self._commit_notes = defaultdict(list)
 
@@ -1261,6 +1270,11 @@ class CommitChecker:
 self._commit_notes[filename].append(note)
 self._commit_problem = True
 
+# Check for references to KDE.org infrastructure which are being 
added without permission
+if re.search(".*(download|files)\.kde\.org.*", line) and 
line.startswith("+"):
+self._commit_notes[filename].append( "[INFRASTRUCTURE]" )
+self._infra_problem = True
+
 # Store the diff
 filediff.append(line)
 
diff --git a/hooks/invent.pre-receive b/hooks/invent.pre-receive
index 75dda6a..537d104 100755
--- a/hooks/invent.pre-receive
+++ b/hooks/invent.pre-receive
@@ -99,6 +99,9 @@ translation_file_rules = [
 '^poqm/.*'
 ]
 
+# These users are authorised to review changes to *.knsrc files
+knsrc_reviewers = ['bcooksley', 'bshah', 'nalvarez']
+
 # For these users we always skip notifications
 notification_user_exceptions = ["scripty"]
 
@@ -355,8 +358,8 @@ for changeset in repository.changesets.values():
 if not os.path.exists(repository_config + "/skip-author-email-checks"):
 auditor.audit_emails_in_metadata( changeset, email_domains_blocked )
 
-   # Depending on who we are, we may also need to check to see whether we are 
changing translations that have been mirrored into the repository
-   # Only specific users are allowed to change these as they are maintained by 
scripty
+# Depending on who we are, we may also need to check to see whether we are 
changing translations that have been mirrored into the repository
+# Only specific users are allowed to change these as they are maintained 
by scripty
 if not os.path.exists(repository_config + "/skip-translation-protections") 
and push_user not in translation_mirror_maintainers:
 # Review each commit for changes to files...
 for commit in changeset.commits.values():
@@ -368,6 +371,17 @@ for changeset in repository.changesets.values():
 if re.match(restriction, filename):
 auditor.log_failure(commit.sha1, "Translations 
maintained separately: " + filename)
 
+# Depending on who we are, we may also need to check to see whether we are 
impacting on a KNSRC file
+# Only specific users are allowed to change these as they can have 
substantial infrastructure implications
+if not os.path.exists(repository_config + "/skip-knsrc-protections") and 
push_user not in knsrc_reviewers and changeset.ref_type is not 
RefType.WorkBranch:
+# Review each commit for changes to files...
+for commit in changeset.commits.values():
+# Now check each file that was changed in that commit...
+for filename in commit.files_changed:
+# Did we change a KNSRC file?
+if re.match(".*knsrc.*", filename):
+auditor.log_failure(commit.sha1, "KNewStuff configuration 
must be Sysadmin reviewed: " + filename)
+
 # Did we have any commit audit failures?
 if auditor.audit_failed:
 print("Push declined - commits failed audit")


CI Repairs

2022-03-07 Thread Ben Cooksley
Hi all,

This evening i've repaired several issues that were causing builds to fail
on the main Jenkins CI system. This includes a broken Windows builder
(causing Windows builds to periodically fail) and a hung FreeBSD builder
(which was consuming half a CPU and preventing KWin CI jobs from completing)

Replacement runs have been initiated for all projects.

So far all appears well, however a number of projects appear to have CI
regressions on one or more platforms due to:
- Use of exceptions (KMail)
- Use of an ECM version that does not exist (print-manager)
- Use of C++ functionality that is not enabled (okular on Windows)
- Something to do with qobject_cast (akonadiconsole)

If developers could please fix their breakages that would be appreciated.

Thanks,
Ben


Re: Critical Denial of Service bugs in Discover

2022-03-04 Thread Ben Cooksley
On Fri, Mar 4, 2022 at 12:49 AM Aleix Pol  wrote:

> I'd say wireshark is too low level for what the problem is here. We are
> talking about having too many HTTP requests for specific URLs.
>

Correct, I guess the difference in our approaches comes from a "before
release" to a "monitor after release" angle to things.
I'd like to see increased scrutiny during the development process as well
to make sure that we release code that operates properly from Day 1.


>
> I can think two main measures:
> - Trigger an alarm (an e-mail notification?) if there's a specific
> UserAgent that has a specific portion of the queries we have in a specific
> day in the services we care about.
> - Offer plots to see how queries by UserAgent evolve over the last couple
> of months (or couple of years).
>

At the moment our ability to analyse our logs is somewhat limited by our
Privacy Policy - https://kde.org/privacypolicy/
Currently we don't have any provision for long term storage of
this information even on an aggregated basis - so we would need to update
this first.

The second issue there is that we are transitioning users to contact a CDN
based endpoint (which is substantially more scalable).
This does mean we lose visibility on data such as User Agents and the URLs
being impacted though as we only get aggregated data unless we ask for
raw logs - which makes implementing something like what you've described
much harder.


>
> Aleix
>

Cheers,
Ben


>
>
> On Thu, Mar 3, 2022 at 9:59 AM Ben Cooksley  wrote:
>
>> On Thu, Mar 3, 2022 at 8:41 AM Aleix Pol  wrote:
>>
>>> (dropping the distros list)
>>>
>>> @sysadmin have you been able to look into any tools we devs can have to
>>> make sure this situation doesn't repeat in the future?
>>>
>>
>> Hi Aleix,
>>
>> To be honest i've been struggling to think of ways that we could detect
>> this on the server side prior to it becoming a massive issue.
>> By the time an issue is evident server side it is usually much too late.
>>
>> The main tools i'd usually recommend would be the standard tools you
>> would use for monitoring the network activity of any application - such as
>> Wireshark.
>>
>> Is there something you were thinking of specifically in terms of us being
>> able to provide?
>>
>> Thanks,
>> Ben
>>
>>
>>>
>>> Aleix
>>>
>>> On Thu, Feb 10, 2022 at 1:10 PM Aleix Pol  wrote:
>>>
>>>> On Thu, Feb 10, 2022 at 11:05 AM Ben Cooksley 
>>>> wrote:
>>>> >
>>>> >
>>>> >
>>>> > On Thu, Feb 10, 2022 at 8:20 AM Aleix Pol  wrote:
>>>> >>
>>>> >> [Snip]
>>>> >>
>>>> >> We still haven't discussed here is how to prevent this problem from
>>>> >> happening again.
>>>> >>
>>>> >> If we don't have information about what is happening, we cannot fix
>>>> problems.
>>>> >
>>>> >
>>>> > Part of the issue here is that the problem only came to Sysadmin
>>>> attention very recently, when the system ran out of disk space as a result
>>>> of growing log files.
>>>> > It was at that point we realised we had a serious problem.
>>>> >
>>>> > Prior to that the system load hadn't climbed to dangerous levels (>
>>>> number of CPU cores) and Apache was keeping up with the traffic, so none of
>>>> our other monitoring was tripped.
>>>> >
>>>> > If you have any thoughts on what sort of information you are thinking
>>>> of that would be helpful.
>>>>
>>>> We could have plots of the amount of queries we get with a KNewStuff/*
>>>> user-agent over time and their distribution.
>>>>
>>>> > It would definitely be helpful though to know when new software is
>>>> going to be released that will be interacting with the servers as we will
>>>> then be able to monitor for abnormalities.
>>>>
>>>> We make big announcements of every Plasma release... (?)
>>>>
>>>> >> Is there anything that could be done in this front? The issue here
>>>> >> could have been addressed months ago, we just never knew it was
>>>> >> happening.
>>>> >
>>>> >
>>>> > One possibility that did occur to me today would be for us to
>>>> integrate some kind of killswitch that our applications would check on
>>>> first initialisation of functionality that talks to KDE.org servers.
>>>> > This would allow us to disable the functionality in question on user
>>>> systems.
>>>> >
>>>> > The check would only be done on first initialization to keep load
>>>> low, while still ensuring all users eventually are affected by the
>>>> killswitch (as they will eventually need to logout/reboot for some reason
>>>> or another).
>>>> >
>>>> > The killswitch would probably work best if it had some kind of
>>>> version check in it so we could specify which versions are disabled.
>>>> > That would allow for subsequent updates - once delivered by
>>>> distributions - to restore the functionality (while leaving it disabled for
>>>> those who haven't updated).
>>>>
>>>> The file we are serving here effectively is the kill switch to all of
>>>> KNewStuff.
>>>>
>>>> Aleix
>>>>
>>>


Re: Critical Denial of Service bugs in Discover

2022-03-03 Thread Ben Cooksley
On Thu, Mar 3, 2022 at 8:41 AM Aleix Pol  wrote:

> (dropping the distros list)
>
> @sysadmin have you been able to look into any tools we devs can have to
> make sure this situation doesn't repeat in the future?
>

Hi Aleix,

To be honest i've been struggling to think of ways that we could detect
this on the server side prior to it becoming a massive issue.
By the time an issue is evident server side it is usually much too late.

The main tools i'd usually recommend would be the standard tools you would
use for monitoring the network activity of any application - such as
Wireshark.

Is there something you were thinking of specifically in terms of us being
able to provide?

Thanks,
Ben


>
> Aleix
>
> On Thu, Feb 10, 2022 at 1:10 PM Aleix Pol  wrote:
>
>> On Thu, Feb 10, 2022 at 11:05 AM Ben Cooksley  wrote:
>> >
>> >
>> >
>> > On Thu, Feb 10, 2022 at 8:20 AM Aleix Pol  wrote:
>> >>
>> >> [Snip]
>> >>
>> >> We still haven't discussed here is how to prevent this problem from
>> >> happening again.
>> >>
>> >> If we don't have information about what is happening, we cannot fix
>> problems.
>> >
>> >
>> > Part of the issue here is that the problem only came to Sysadmin
>> attention very recently, when the system ran out of disk space as a result
>> of growing log files.
>> > It was at that point we realised we had a serious problem.
>> >
>> > Prior to that the system load hadn't climbed to dangerous levels (>
>> number of CPU cores) and Apache was keeping up with the traffic, so none of
>> our other monitoring was tripped.
>> >
>> > If you have any thoughts on what sort of information you are thinking
>> of that would be helpful.
>>
>> We could have plots of the amount of queries we get with a KNewStuff/*
>> user-agent over time and their distribution.
>>
>> > It would definitely be helpful though to know when new software is
>> going to be released that will be interacting with the servers as we will
>> then be able to monitor for abnormalities.
>>
>> We make big announcements of every Plasma release... (?)
>>
>> >> Is there anything that could be done in this front? The issue here
>> >> could have been addressed months ago, we just never knew it was
>> >> happening.
>> >
>> >
>> > One possibility that did occur to me today would be for us to integrate
>> some kind of killswitch that our applications would check on first
>> initialisation of functionality that talks to KDE.org servers.
>> > This would allow us to disable the functionality in question on user
>> systems.
>> >
>> > The check would only be done on first initialization to keep load low,
>> while still ensuring all users eventually are affected by the killswitch
>> (as they will eventually need to logout/reboot for some reason or another).
>> >
>> > The killswitch would probably work best if it had some kind of version
>> check in it so we could specify which versions are disabled.
>> > That would allow for subsequent updates - once delivered by
>> distributions - to restore the functionality (while leaving it disabled for
>> those who haven't updated).
>>
>> The file we are serving here effectively is the kill switch to all of
>> KNewStuff.
>>
>> Aleix
>>
>


Re: Critical Denial of Service bugs in Discover

2022-02-25 Thread Ben Cooksley
On Fri, Feb 25, 2022 at 10:09 PM Harald Sitter  wrote:

> On Mon, Feb 21, 2022 at 11:05 AM Ben Cooksley  wrote:
> >
> > On Mon, Feb 21, 2022 at 10:01 PM Harald Sitter  wrote:
> >>
> >> On Thu, Feb 10, 2022 at 1:11 PM Aleix Pol  wrote:
> >> >
> >> > On Thu, Feb 10, 2022 at 11:05 AM Ben Cooksley 
> wrote:
> >> > >
> >> > >
> >> > >
> >> > > On Thu, Feb 10, 2022 at 8:20 AM Aleix Pol  wrote:
> >> > >>
> >> > >> [Snip]
> >> > >>
> >> > >> We still haven't discussed here is how to prevent this problem from
> >> > >> happening again.
> >> > >>
> >> > >> If we don't have information about what is happening, we cannot
> fix problems.
> >> > >
> >> > >
> >> > > Part of the issue here is that the problem only came to Sysadmin
> attention very recently, when the system ran out of disk space as a result
> of growing log files.
> >> > > It was at that point we realised we had a serious problem.
> >> > >
> >> > > Prior to that the system load hadn't climbed to dangerous levels (>
> number of CPU cores) and Apache was keeping up with the traffic, so none of
> our other monitoring was tripped.
> >> > >
> >> > > If you have any thoughts on what sort of information you are
> thinking of that would be helpful.
> >> >
> >> > We could have plots of the amount of queries we get with a KNewStuff/*
> >> > user-agent over time and their distribution.
> >> >
> >> > > It would definitely be helpful though to know when new software is
> going to be released that will be interacting with the servers as we will
> then be able to monitor for abnormalities.
> >> >
> >> > We make big announcements of every Plasma release... (?)
> >> >
> >> > >> Is there anything that could be done in this front? The issue here
> >> > >> could have been addressed months ago, we just never knew it was
> >> > >> happening.
> >> > >
> >> > >
> >> > > One possibility that did occur to me today would be for us to
> integrate some kind of killswitch that our applications would check on
> first initialisation of functionality that talks to KDE.org servers.
> >> > > This would allow us to disable the functionality in question on
> user systems.
> >> > >
> >> > > The check would only be done on first initialization to keep load
> low, while still ensuring all users eventually are affected by the
> killswitch (as they will eventually need to logout/reboot for some reason
> or another).
> >> > >
> >> > > The killswitch would probably work best if it had some kind of
> version check in it so we could specify which versions are disabled.
> >> > > That would allow for subsequent updates - once delivered by
> distributions - to restore the functionality (while leaving it disabled for
> those who haven't updated).
> >> >
> >> > The file we are serving here effectively is the kill switch to all of
> KNewStuff.
> >>
> >> I'm a bit late to the party but for future reference I think this
> >> was/is an architectural scaling problem on the server side as much as
> >> a bug on the client. If just https load is the problem then the
> >> "hotfix" is to use a HTTP load balancer until fixes make it into the
> >> clients, killing the clients is like the last resort ever. I'm sure we
> >> have the money to afford a bunch of cloud nodes serving as selective
> >> proxy caches for a month to balance out the KNS load on the canonical
> >> server.
> >
> >
> > This was a multi-fold bug:
> >
> > 1) Sysadmin allowing a compatibility endpoint to remain alive for years
> after we told people to stop using it and to use the new one (which is on a
> CDN and which would have handled this whole issue much better)
> > 2) Developers writing code to talk to KDE.org infrastructure without
> consulting Sysadmin, especially where it deviated from previously
> established patterns.
> >
> > In terms of scalability I disagree - the system is not being used here
> in a manner for which it was not designed.
> >
> > This system is intended to serve downloads of KDE software and
> associated data files to distributors and end users. These are actions that
> are expected to:
> > a) Be undertaken on an infre

Re: Critical Denial of Service bugs in Discover

2022-02-21 Thread Ben Cooksley
On Mon, Feb 21, 2022 at 10:01 PM Harald Sitter  wrote:

> On Thu, Feb 10, 2022 at 1:11 PM Aleix Pol  wrote:
> >
> > On Thu, Feb 10, 2022 at 11:05 AM Ben Cooksley  wrote:
> > >
> > >
> > >
> > > On Thu, Feb 10, 2022 at 8:20 AM Aleix Pol  wrote:
> > >>
> > >> [Snip]
> > >>
> > >> We still haven't discussed here is how to prevent this problem from
> > >> happening again.
> > >>
> > >> If we don't have information about what is happening, we cannot fix
> problems.
> > >
> > >
> > > Part of the issue here is that the problem only came to Sysadmin
> attention very recently, when the system ran out of disk space as a result
> of growing log files.
> > > It was at that point we realised we had a serious problem.
> > >
> > > Prior to that the system load hadn't climbed to dangerous levels (>
> number of CPU cores) and Apache was keeping up with the traffic, so none of
> our other monitoring was tripped.
> > >
> > > If you have any thoughts on what sort of information you are thinking
> of that would be helpful.
> >
> > We could have plots of the amount of queries we get with a KNewStuff/*
> > user-agent over time and their distribution.
> >
> > > It would definitely be helpful though to know when new software is
> going to be released that will be interacting with the servers as we will
> then be able to monitor for abnormalities.
> >
> > We make big announcements of every Plasma release... (?)
> >
> > >> Is there anything that could be done in this front? The issue here
> > >> could have been addressed months ago, we just never knew it was
> > >> happening.
> > >
> > >
> > > One possibility that did occur to me today would be for us to
> integrate some kind of killswitch that our applications would check on
> first initialisation of functionality that talks to KDE.org servers.
> > > This would allow us to disable the functionality in question on user
> systems.
> > >
> > > The check would only be done on first initialization to keep load low,
> while still ensuring all users eventually are affected by the killswitch
> (as they will eventually need to logout/reboot for some reason or another).
> > >
> > > The killswitch would probably work best if it had some kind of version
> check in it so we could specify which versions are disabled.
> > > That would allow for subsequent updates - once delivered by
> distributions - to restore the functionality (while leaving it disabled for
> those who haven't updated).
> >
> > The file we are serving here effectively is the kill switch to all of
> KNewStuff.
>
> I'm a bit late to the party but for future reference I think this
> was/is an architectural scaling problem on the server side as much as
> a bug on the client. If just https load is the problem then the
> "hotfix" is to use a HTTP load balancer until fixes make it into the
> clients, killing the clients is like the last resort ever. I'm sure we
> have the money to afford a bunch of cloud nodes serving as selective
> proxy caches for a month to balance out the KNS load on the canonical
> server.
>

This was a multi-fold bug:

1) Sysadmin allowing a compatibility endpoint to remain alive for years
after we told people to stop using it and to use the new one (which is on a
CDN and which would have handled this whole issue much better)
2) Developers writing code to talk to KDE.org infrastructure without
consulting Sysadmin, especially where it deviated from previously
established patterns.

In terms of scalability I disagree - the system is not being used here in a
manner for which it was not designed.

This system is intended to serve downloads of KDE software and associated
data files to distributors and end users. These are actions that are
expected to:
a) Be undertaken on an infrequent basis; and
b) Be undertaken as a result of user initiated action (such as clicking a
download link)

It was never intended to be used to serve configuration data files to end
user systems. We have autoconfig.kde.org for that.

The system in question is handling the load extremely well and far beyond
my expectations - it is fairly unfathomable that download.kde.org and
files.kde.org would receive traffic on the order of 500-600 requests per
second.
During this time the highest load I have seen has been around 8 - and
despite this being uncomfortably busy it has not fallen over or dropped the
ball for both it's BAU activity as well as the abuse it has taken.
(My extreme level of concern on this matter has been because I knew that if
we hit a major release such a

Re: Dropping dead(?) Python bindings generation code?

2022-02-12 Thread Ben Cooksley
On Sun, Feb 13, 2022 at 6:36 AM Friedrich W. H. Kossebau 
wrote:

> Hi,
>

Hi there,


>
> trying to ensure some changes do not break the Python binding generation,
> I
> actually tried to activate that, but found at least on current openSUSE TW
> there seem to be no longer any working dependencies. Also the openSUSE TW
> packages of the KF modules seem to also be build without bindings, for the
> samples I checked.
>

> Then I found that on both gitlab & jenkins CI the binding generation is
> also
> skipt (at least for KCoreAddons on all platforms, but seems also
> everywhere
> else).
> Some related commit removing the support talks about "deterministic"
> builds
> though:
> https://invent.kde.org/sysadmin/ci-tooling/-/commit/
> 6a92fdf747990d2e074e92b2bdc224efc9b08740
>

Not sure if SUSE hit the same issues we did, but the build of these on KDE
CI has been disabled for a long time because the Python bindings did not
build reliably.
This was caused by dependency sequencing issues from my understanding
within CMake.

Consequently we would get builds falling over periodically for no reason
other than the timing within the build itself.
This usually made it pretty difficult for Dependency Builds to complete as
at least one Framework would invariably fall over.

Without that being fixed, we would continue to have the Python bindings
support disabled on the CI system regardless of anything else being fixed.


>
> Then on #kde-devel I was told that"pyqt5 5.15.6 + sip4" do no more go
> together, referencing
> https://www.riverbankcomputing.com/pipermail/pyqt/2021-November/044346.html
> :
> > It wasn't an intentional breakage but it's not something I'm going to
> rush
> to fix.
>
> Who feels in charge of the Python binding support? Is there a chance
> someone
> will work on this soonish?
>
> Or could we drop it now, and save everyone the cmake warning messages they
> cannot fix and also the bad feeling to change things that might break
> binding
> generation support even further?
>
> It was suggested that "the only reasonable way forward it to port to
> modern
> sip" "but that requires an almost full rewrite".
> Which sounds as if any future system will need a rework of ECM's
> PythonModuleGeneration as well, thus keeping the current CMake code in KF
> modules around in chance they might get used again as they are in the
> future
> would not make sense.
>
> Reference removal of the Python binding generation support up as
> https://invent.kde.org/frameworks/kcoreaddons/-/merge_requests/198
> to serve as example for the discussion.
>
> Cheers
> Friedrich
>

Cheers,
Ben


Re: Critical Denial of Service bugs in Discover

2022-02-12 Thread Ben Cooksley
On Fri, Feb 11, 2022 at 10:22 AM Fabian Vogt  wrote:

> Moin,
>
> Am Sonntag, 6. Februar 2022, 21:54:13 CET schrieb Fabian Vogt:
> > Am Sonntag, 6. Februar 2022, 19:27:11 CET schrieb Ben Cooksley:
> > > On Sun, Feb 6, 2022 at 1:07 PM Fabian Vogt 
> wrote:
> > > > The first URL is used by kfontinst.knsrc from plasma-workspace:
> > > > ProvidersUrl=
> https://distribute.kde.org/khotnewstuff/fonts-providers.xml
> > > >
> > > > The second URL is used by multiple knsrc files in my VM:
> > > > aurorae.knsrc:ProvidersUrl=
> https://download.kde.org/ocs/providers.xml
> > > > comic.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml
> > > > kwineffect.knsrc:ProvidersUrl=
> https://download.kde.org/ocs/providers.xml
> > > > kwinscripts.knsrc:ProvidersUrl=
> https://download.kde.org/ocs/providers.xml
> > > > kwinswitcher.knsrc:ProvidersUrl=
> https://download.kde.org/ocs/providers.xml
> > > > wallpaperplugin.knsrc:ProvidersUrl=
> > > > https://download.kde.org/ocs/providers.xml
> > >
> > > This makes me incredibly sad. We had a push to eliminate all usage of
> the
> > > legacy download.kde.org endpoint many years ago...
> > > I have now resolved the majority of these - if distributions could
> please
> > > pick up those patches that would be appreciated.
> > >
> > > Please note that I have now terminated the support on the server that
> was
> > > making these legacy endpoints work, so those patches are necessary to
> > > restore functionality.
> > ...
> > It's also possible that the requests aren't actually caused by Discover
> at all,
> > but just something which imitates it in a DDoS attack. In that case we
> couldn't
> > do anything on the client-side anyway. I don't think this is very
> likely, but
> > until the issue was reproduced with disover it's a possibility.
>
> I think I have a plausible explanation for what could've caused this.
> While testing a MR for the notifier, I noticed odd behaviour: It always ran
> plasma-discover-update twice!
> https://invent.kde.org/plasma/discover/-/merge_requests/254#note_394584
>
> The reason for that is that after the update process finishes, the notifier
> realizes that it's idle again and if updates are available, it will
> immediately
> trigger another update after the 15min idle time. Now here's the catch: If
> the
> system has already been idle for >=15min (which is very likely at that
> point),
> the idle timeout will immediately fire! This process repeats unlimited and
> without delay, until the system is no longer idle or there aren't updates
> available anymore. Here I have plasma-discover-update running approx. every
> second, which amounts to ~4 req/s to download.kde.org.
>
> This is mostly mitigated by the introduction of the 3h delay between
> updates
> by d607e0c6f9, but not entirely. The check is only effective after the
> second
> iteration, which is what I observed in my testing. (One of the commits in
> my MR
> should address that as well.)
>
> One of the conditions for running into this bug is that after the automatic
> updater ran, there still have to be updates available to trigger the next
> run.
> Initially I thought that this can mostly happen if updates fail to
> download or
> install, this is unfortunately not true. The notifier by default counts all
> available updates, but the updater only installs offline updates. So if
> there
> is even a single non-offline update available, the loop continues.
>

Continues infinitely I assume?


>
> So this probably affected a lot of users who enabled automatic
> installation of
> updates :-/
>

Do we know if any distributions flipped that switch?


>
> Cheers,
> Fabian
>
>
Regards,
Ben


Re: Critical Denial of Service bugs in Discover

2022-02-10 Thread Ben Cooksley
On Thu, Feb 10, 2022 at 8:20 AM Aleix Pol  wrote:

> [Snip]
>
> We still haven't discussed here is how to prevent this problem from
> happening again.
>
> If we don't have information about what is happening, we cannot fix
> problems.
>

Part of the issue here is that the problem only came to Sysadmin attention
very recently, when the system ran out of disk space as a result of growing
log files.
It was at that point we realised we had a serious problem.

Prior to that the system load hadn't climbed to dangerous levels (> number
of CPU cores) and Apache was keeping up with the traffic, so none of our
other monitoring was tripped.

If you have any thoughts on what sort of information you are thinking of
that would be helpful.

It would definitely be helpful though to know when new software is going to
be released that will be interacting with the servers as we will then be
able to monitor for abnormalities.
(This would have allowed us to advise on the User-Agent stuff prior to
September, as well as point out potential issues with caching


> Is there anything that could be done in this front? The issue here
> could have been addressed months ago, we just never knew it was
> happening.


One possibility that did occur to me today would be for us to integrate
some kind of killswitch that our applications would check on first
initialisation of functionality that talks to KDE.org servers.
This would allow us to disable the functionality in question on user
systems.

The check would only be done on first initialization to keep load low,
while still ensuring all users eventually are affected by the killswitch
(as they will eventually need to logout/reboot for some reason or another).

The killswitch would probably work best if it had some kind of version
check in it so we could specify which versions are disabled.
That would allow for subsequent updates - once delivered by distributions -
to restore the functionality (while leaving it disabled for those who
haven't updated).


>
> Aleix
>

Thanks,
Ben


Re: Critical Denial of Service bugs in Discover

2022-02-08 Thread Ben Cooksley
On Tue, Feb 8, 2022 at 4:24 AM Aleix Pol  wrote:

> On Sat, Feb 5, 2022 at 10:16 PM Ben Cooksley  wrote:
> >
> > Hi all,
> >
> > Over the past week or so Sysadmin has been dealing with an extremely
> high volume of traffic directed towards both download.kde.org and
> distribute.kde.org.
> >
> > This traffic volume is curious in so far that it is directed at two
> paths specifically:
> > - distribute.kde.org/khotnewstuff/fonts-providers.xml
> > - download.kde.org/ocs/providers.xml
> >
> > The first path is an "internal only" host which we were redirecting a
> legacy path to prior to the resource being relocated to cdn.kde.org. The
> second path has been legacy for numerous years now (more than 5) and is
> replaced by autoconfig.kde.org.
> > It is of extreme concern that these paths are still in use - especially
> the ocs/providers.xml one.
> >
> > The volume of traffic has reached an extent that to prevent the server
> disk filling up we have had to disable logging for those two sites. Whilst
> dependent on the time of day the server is currently dealing with the
> current volume of requests, which is far outside normal specifications:
> >
> > 555 requests/sec - 4.5 MB/second - 8.3 kB/request - .739199
> ms/request
> >
> > Analysis of a fragment of logs (comprising just a few minutes of
> traffic) reveals the following:
> >
> >  63 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-"
> "KNewStuff/5.89.0-discoverupdate/5.23.5"
> >  64 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-"
> "KNewStuff/5.89.0-discoverupdate/5.23.4"
> > 104 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-"
> "KNewStuff/5.90.0-discoverupdate/5.23.90"
> > 105 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-"
> "KNewStuff/5.88.0-discoverupdate/5.23.5"
> >1169 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-"
> "KNewStuff/5.86.0-plasma-discover-update/"
> >1256 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-"
> "KNewStuff/5.90.0-discoverupdate/5.23.5"
> >2905 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" "Mozilla/5.0"
> >
> >  86 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 200 6773 "-"
> "Mozilla/5.0"
> > 130 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-"
> "KNewStuff/5.89.0-discoverupdate/5.23.5"
> > 136 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-"
> "KNewStuff/5.89.0-discoverupdate/5.23.4"
> > 197 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-"
> "KNewStuff/5.88.0-discoverupdate/5.23.5"
> > 199 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-"
> "KNewStuff/5.90.0-discoverupdate/5.23.90"
> >2624 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-"
> "KNewStuff/5.86.0-plasma-discover-update/"
> >2642 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-"
> "KNewStuff/5.90.0-discoverupdate/5.23.5"
> >6117 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-"
> "Mozilla/5.0"
> >
> > This indicates that the bug lies solely within Plasma's Discover
> component - more precisely it's updater.
> >
> > Examining the origin of these requests has indicated that some clients
> are making requests to these paths well in excess of several times a minute
> with a number of IP addresses appearing more 60 times in a 1 minute sized
> sample window.
> >
> > Given that Sysadmin has raised issues with this component and it's
> behaviour in the past, it appears that issues regarding the behaviour of
> the OCS componentry within Discover remain unresolved.
> >
> > Due to the level of distress this is causing our systems, I am therefore
> left with no other option other than to direct the Plasma Discover
> developers to create and release without delay patches for all versions in
> support, as well as for all those currently present in any actively
> maintained distributions, that disable all OCS functionality in the
> Discover updater. Distributions are requested to treat these patches as
> security patches and to distribute them to users without delay.
> >
> > In 24 hours time Sysadmin will be making a posting to kde-announce
> requesting that users immediately cease use of the Discover update clie

Re: KF 5.91: 24 modules with failing unit tests (Re: Please fix failing unit tests with Windows platform)

2022-02-07 Thread Ben Cooksley
On Mon, Feb 7, 2022 at 10:56 PM Christoph Cullmann (cullmann.io) <
christ...@cullmann.io> wrote:

> On 2022-02-07 10:35, Ben Cooksley wrote:
> > On Sun, Feb 6, 2022 at 10:40 PM Friedrich W. H. Kossebau
> >  wrote:
> >
> >> Am Montag, 24. Januar 2022, 01:06:40 CET schrieb Friedrich W. H.
> >> Kossebau:
> >>> Hi,
> >>>
> >>> since a long time there are lots of failing unit tests across
> >> multiple
> >>> repositories. Could the Windows platform maintainers/stakeholders
> >> please
> >>> look soonish into either fixing those tests or properly marking
> >> them as
> >>> expected to fail, so the resources the KDE CI spends on running
> >> the tests
> >>> every hour, day and week make some sense again, as well as having
> >> something
> >>> usable to diff results again, to notice any new regressions?
> >>>
> >>> Please see
> >>>
> >>>
> >>
> >
> https://build.kde.org/job/Frameworks/view/Platform%20-%20WindowsMSVCQt5.15/
> >>> (best sort by "S" build status to get a list what need
> >>
> >> And those who believe in the broken windows theory also would claim
> >> this
> >> slacking now resulted in the regressions in the openSUSE builds,
> >> where 5
> >> modules now have failing unit tests at time of release tagging, when
> >> it once
> >> was 0 thanks to hard work of David F. and others. :(
> >>
> >> Is it time to remove
> >> https://community.kde.org/Frameworks/
> >> Policies#Frameworks_CI_failures_are_treated_as_stop_the_line_events
> >> because seemingly this is just old dead pixels on a web page and not
> >> the
> >> spirit these days?
> >
> > Not sure that is the ideal outcome here - preferrably our tests would
> > continue to all pass.
> >
> > I know some tests on certain platforms have been flaky and switch
> > between failing/passing - are we sure that isn't the driver of people
> > ignoring the results?
>
> One thing that sometimes lead me to ignore stuff with KTextEditor is
> that the UI
> tests are often very unstable.
>
> e.g. they just fail for me locally but then work perfectly in the CI or
> the other
> way around.
>
> Not sure how to improve that.
>

Interesting to note that it is GUI/UI tests that are causing issues. I
would have thought that the setup on the CI for those would be a carbon
copy almost every time which makes these failures interesting.
Out of curiosity, what are the tests trying to accomplish and where is it
failing?


>
> For all non-UI tests naturally no such problems exist for KTextEditor
> and they are easy to keep
> working.
>
> KSyntaxHighlighting only has non-UI test and that is very easy to keep
> in a consistent shape.
>
> Greetings
> Christoph
>

Thanks,
Ben


>
> >
> > Cheers,
> > Ben
> >
> >> Friedrich
>
> --
> Ignorance is bliss...
> https://cullmann.io | https://kate-editor.org
>


Re: KF 5.91: 24 modules with failing unit tests (Re: Please fix failing unit tests with Windows platform)

2022-02-07 Thread Ben Cooksley
On Sun, Feb 6, 2022 at 10:40 PM Friedrich W. H. Kossebau 
wrote:

> Am Montag, 24. Januar 2022, 01:06:40 CET schrieb Friedrich W. H. Kossebau:
> > Hi,
> >
> > since a long time there are lots of failing unit tests across multiple
> > repositories. Could the Windows platform maintainers/stakeholders please
> > look soonish into either fixing those tests or properly marking them as
> > expected to fail, so the resources the KDE CI spends on running the tests
> > every hour, day and week make some sense again, as well as having
> something
> > usable to diff results again, to notice any new regressions?
> >
> > Please see
> >
> >
> https://build.kde.org/job/Frameworks/view/Platform%20-%20WindowsMSVCQt5.15/
> > (best sort by "S" build status to get a list what need
>
> And those who believe in the broken windows theory also would claim this
> slacking now resulted in the regressions in the openSUSE builds, where 5
> modules now have failing unit tests at time of release tagging, when it
> once
> was 0 thanks to hard work of David F. and others. :(
>
> Is it time to remove
> https://community.kde.org/Frameworks/
> Policies#Frameworks_CI_failures_are_treated_as_stop_the_line_events
> because seemingly this is just old dead pixels on a web page and not the
> spirit these days?
>

Not sure that is the ideal outcome here - preferrably our tests would
continue to all pass.

I know some tests on certain platforms have been flaky and switch between
failing/passing - are we sure that isn't the driver of people ignoring the
results?

Cheers,
Ben


>
> Friedrich
>
>
>
>


Re: Critical Denial of Service bugs in Discover

2022-02-06 Thread Ben Cooksley
On Sun, Feb 6, 2022 at 1:07 PM Fabian Vogt  wrote:

> Hi,
>
> Am Samstag, 5. Februar 2022, 22:16:28 CET schrieb Ben Cooksley:
> > Hi all,
> >
> > Over the past week or so Sysadmin has been dealing with an extremely high
> > volume of traffic directed towards both download.kde.org and
> > distribute.kde.org.
> >
> > This traffic volume is curious in so far that it is directed at two paths
> > specifically:
> > - distribute.kde.org/khotnewstuff/fonts-providers.xml
> > - download.kde.org/ocs/providers.xml
> >
> > The first path is an "internal only" host which we were redirecting a
> > legacy path to prior to the resource being relocated to cdn.kde.org. The
> > second path has been legacy for numerous years now (more than 5) and is
> > replaced by autoconfig.kde.org.
> > It is of extreme concern that these paths are still in use - especially
> the
> > ocs/providers.xml one.
> >
> >...
> >
> > This indicates that the bug lies solely within Plasma's Discover
> component
> > - more precisely it's updater.
> >
> > Examining the origin of these requests has indicated that some clients
> are
> > making requests to these paths well in excess of several times a minute
> > with a number of IP addresses appearing more 60 times in a 1 minute sized
> > sample window.
>
> FWICT, this is caused by plasma-discover-update, which is triggered by the
> DiscoverNotifier service if automatic updates are enabled in kcm_updates,
> updates are available and the system idle for >=15min.
>
> // If the system is untouched for 1 hour, trigger the unattened update
> using namespace std::chrono_literals;
>
> KIdleTime::instance()->addIdleTimeout(int(std::chrono::milliseconds(15min).count()));
>
> (I wonder whether there's a bug about calling addIdleTimeout more than
> once.
> It will then invoke triggerUpdate multiple times after 15min of idle.)
>

That may explain why we are seeing so many requests from some IPs and very
few from others.


>
> The Discover KNS backend creates instances for all available knsrc files,
> which on construction call KNSReviews::setProviderUrl with the URL defined
> in
> those files, triggering the requests.
>

That does not sound scalable, and would certainly explain why not too long
ago we found that the traffic received by autoconfig.kde.org had grown to
such an extent we had to shift it to being handled by a CDN.
At the time I chalked the problem up to increasing popularity of our
software that included KNS functionality.


>
> The first URL is used by kfontinst.knsrc from plasma-workspace:
> ProvidersUrl=https://distribute.kde.org/khotnewstuff/fonts-providers.xml
>
> The second URL is used by multiple knsrc files in my VM:
> aurorae.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml
> comic.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml
> kwineffect.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml
> kwinscripts.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml
> kwinswitcher.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml
> wallpaperplugin.knsrc:ProvidersUrl=
> https://download.kde.org/ocs/providers.xml


This makes me incredibly sad. We had a push to eliminate all usage of the
legacy download.kde.org endpoint many years ago...
I have now resolved the majority of these - if distributions could please
pick up those patches that would be appreciated.

Please note that I have now terminated the support on the server that was
making these legacy endpoints work, so those patches are necessary to
restore functionality.


>
> > Given that Sysadmin has raised issues with this component and it's
> > behaviour in the past, it appears that issues regarding the behaviour of
> > the OCS componentry within Discover remain unresolved.
> >
> > Due to the level of distress this is causing our systems, I am therefore
> > left with no other option other than to direct the Plasma Discover
> > developers to create and release without delay patches for all versions
> in
> > support, as well as for all those currently present in any actively
> > maintained distributions, that disable all OCS functionality in the
> > Discover updater. Distributions are requested to treat these patches as
> > security patches and to distribute them to users without delay.
>
> Emergency workarounds for distributions might be to either not ship the KNS
> backend by not building kns-backend.so or deleting it afterwards, or
> disabling
> the discover notifier
> (/etc/xdg/autostart/org.kde.discover.notifier.desktop)
> completely.
>

I have now committed patches to Discover going back to Plasma/5.18 which

Critical Denial of Service bugs in Discover

2022-02-05 Thread Ben Cooksley
Hi all,

Over the past week or so Sysadmin has been dealing with an extremely high
volume of traffic directed towards both download.kde.org and
distribute.kde.org.

This traffic volume is curious in so far that it is directed at two paths
specifically:
- distribute.kde.org/khotnewstuff/fonts-providers.xml
- download.kde.org/ocs/providers.xml

The first path is an "internal only" host which we were redirecting a
legacy path to prior to the resource being relocated to cdn.kde.org. The
second path has been legacy for numerous years now (more than 5) and is
replaced by autoconfig.kde.org.
It is of extreme concern that these paths are still in use - especially the
ocs/providers.xml one.

The volume of traffic has reached an extent that to prevent the server disk
filling up we have had to disable logging for those two sites. Whilst
dependent on the time of day the server is currently dealing with the
current volume of requests, which is far outside normal specifications:

555 requests/sec - 4.5 MB/second - 8.3 kB/request - .739199
ms/request

Analysis of a fragment of logs (comprising just a few minutes of traffic)
reveals the following:

 63 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-"
"KNewStuff/5.89.0-discoverupdate/5.23.5"
 64 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-"
"KNewStuff/5.89.0-discoverupdate/5.23.4"
104 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-"
"KNewStuff/5.90.0-discoverupdate/5.23.90"
105 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-"
"KNewStuff/5.88.0-discoverupdate/5.23.5"
   1169 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-"
"KNewStuff/5.86.0-plasma-discover-update/"
   1256 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-"
"KNewStuff/5.90.0-discoverupdate/5.23.5"
   2905 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" "Mozilla/5.0"

 86 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 200 6773 "-"
"Mozilla/5.0"
130 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-"
"KNewStuff/5.89.0-discoverupdate/5.23.5"
136 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-"
"KNewStuff/5.89.0-discoverupdate/5.23.4"
197 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-"
"KNewStuff/5.88.0-discoverupdate/5.23.5"
199 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-"
"KNewStuff/5.90.0-discoverupdate/5.23.90"
   2624 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-"
"KNewStuff/5.86.0-plasma-discover-update/"
   2642 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-"
"KNewStuff/5.90.0-discoverupdate/5.23.5"
   6117 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-"
"Mozilla/5.0"

This indicates that the bug lies solely within Plasma's Discover component
- more precisely it's updater.

Examining the origin of these requests has indicated that some clients are
making requests to these paths well in excess of several times a minute
with a number of IP addresses appearing more 60 times in a 1 minute sized
sample window.

Given that Sysadmin has raised issues with this component and it's
behaviour in the past, it appears that issues regarding the behaviour of
the OCS componentry within Discover remain unresolved.

Due to the level of distress this is causing our systems, I am therefore
left with no other option other than to direct the Plasma Discover
developers to create and release without delay patches for all versions in
support, as well as for all those currently present in any actively
maintained distributions, that disable all OCS functionality in the
Discover updater. Distributions are requested to treat these patches as
security patches and to distribute them to users without delay.

In 24 hours time Sysadmin will be making a posting to kde-announce
requesting that users immediately cease use of the Discover update client
as it is creating a Denial of Service attack on our infrastructure.

Regards,
Ben Cooksley
KDE Sysadmin


Re: Maintainers of KDE Frameworks for the Windows platform?

2022-01-24 Thread Ben Cooksley
On Mon, Jan 24, 2022 at 10:48 PM Christoph Cullmann (cullmann.io) <
christ...@cullmann.io> wrote:

> On 2022-01-24 01:00, Friedrich W. H. Kossebau wrote:
> > Hi,
> >
> > in the past it was hard to find someone to fix things for KDE
> > Frameworks on
> > Windows, and too often people not interested in Windows had instead to
> > spend
> > their costly leisure time to solve problems, e.g. by debugging via CI
> > runs.
> >
> > I do not think we can expect from every contributor/patch author they
> > are
> > capable to understand and to solve things on all platforms. For one as
> > this
> > does not scale, and even more when the platform is a proprietary one
> > that
> > otherwise works against the mission of KDE and people rather avoid to
> > have to
> > know about it.
> >
> > So we need dedicated maintainer teams for each platform IMHO. And if
> > that team
> > is empty, have to drop the official support for that platform, instead
> > of e.g.
> > having it a "broken windows theory" thing on CI (pun intended).
> >
> > Given Linux (default, all the usual suspect contributors), FreeBSD
> > (Tobias,
> > Adriaan), and Android (some other usual suspect contributors) are
> > covered,
> > there is a reaction time the same day often, when help is needed with
> > those.
> > Other than for Windows (and macOS once it makes it to CI).
> >
> > Who would be available as contact person for KF @ Windows, so could be
> > reliably called in to solve code issues appearing in new work or
> > regressions
> > by external influences? Either by a to be created @teams tag or as
> > highly
> > available individuals?
> >
> > If we do not have enough people who can provide at least, say, weekly
> > work on
> > the Windows platform, I would propose to drop the official support, as
> > it is
> > an annoying burden to those who have no stakes on that platform.
> > And also harms the reputation of the KF product, because being badly
> > maintained and thus partially broken makes it into the developer/user
> > experience on those platforms, which then is mapped onto the whole
> > product
> > (rightfully), not just the support on that platform.
>
> I don't agree with that mindset.
>
> Naturally, as you point out in your other mail,
> the unit tests must be fixed.
>
> But beside that, I see Windows like any other platform,
> you need to ensure your changes don't kill it.
>
> It is not acceptable to commit stuff that breaks the e.g. FreeBSD
> CI, the same rule can be there for Windows, too.
>
> If you need help, you can ping people like me for Windows or we could
> create
> some @teams/windows or whatever.
>
> Beside that, I think in most cases, our code is on a level that doesn't
> really have that much operating specific parts.
>

I concur with Christoph's points here.


>
> There are special cases like baloo and Co., I actually would propose to
> not support such stuff on Windows (or non Linux) at all,
> not sure if it should be a Framework at all in that case.
>

A comprehensive list of what we currently support some form of CI for can
be found at
https://invent.kde.org/sysadmin/ci-management/-/blob/master/seeds/frameworks-latest.yml

Windows CI on Gitlab is not too far away - Frameworks is actually ready to
go as it were I just need a chance to try to run the seed jobs.
Alas other matters keep getting in the way (both within and outside of KDE)


>
> Greetings
> Christoph
>

Cheers,
Ben


>
> --
> Ignorance is bliss...
> https://cullmann.io | https://kate-editor.org
>


Re: Gitlab CI: failed unit tests vs. currently passing CI

2022-01-23 Thread Ben Cooksley
On Mon, Jan 24, 2022 at 12:56 AM Albert Astals Cid  wrote:

> El diumenge, 23 de gener de 2022, a les 1:59:01 (CET), Ben Cooksley va
> escriure:
> > On Sun, Jan 23, 2022 at 12:29 PM Albert Astals Cid 
> wrote:
> >
> > > El diumenge, 23 de gener de 2022, a les 0:09:23 (CET), Ben Cooksley va
> > > escriure:
> > > > On Sun, Jan 23, 2022 at 11:29 AM Albert Astals Cid 
> > > wrote:
> > > >
> > > > > El dissabte, 22 de gener de 2022, a les 6:11:29 (CET), Ben
> Cooksley va
> > > > > escriure:
> > > > > > EXCLUDE_DEPRECATED_BEFORE_AND_ATOn Sat, Jan 22, 2022 at 1:31 PM
> > > Friedrich
> > > > > > W. H. Kossebau  wrote:
> > > > > >
> > > > > > > Hi,
> > > > > >
> > > > > >
> > > > > > > seems that Gitlab CI is currently configured to show the green
> > > > > "Success"
> > > > > > > checkmark for pipeline runs even if unit tests are failing.
> > > > > > >
> > > > > >
> > > > > > That is correct, only compilation or other internal failures
> cause
> > > the
> > > > > > build to show a failure result.
> > > > > >
> > > > > >
> > > > > > > Reasons seems to be that there Gitlab only knows Yay or Nay,
> > > without
> > > > > the
> > > > > > > warning state level known from Jenkins.
> > > > > > >
> > > > > >
> > > > > > Also correct.
> > > > > >
> > > > > >
> > > > > > > And given that quite some projects (sadly) maintain a few
> long-time
> > > > > > > failing
> > > > > > > unit tests, having the pipeline fail on unit tests seems to
> have
> > > been
> > > > > seen
> > > > > > > as
> > > > > > > too aggressive
> > > > > >
> > > > > >
> > > > > > Correct again.
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > This of course harms the purpose of the unit tests, when their
> > > failures
> > > > > > > are
> > > > > > > only noticed weeks later, not e.g. at MR discussion time.
> > > > > > >
> > > > > >
> > > > > > Gitlab does note changes in the test suite as can currently be
> seen
> > > on
> > > > > > https://invent.kde.org/frameworks/kio/-/merge_requests/708
> > > > > > Quoting the page:  "Test summary contained 33 failed and 16 fixed
> > > test
> > > > > > results out of 205 total tests"
> > > > > >
> > > > > > It does the same thing for Code Quality - "Code quality scanning
> > > detected
> > > > > > 51 changes in merged results"
> > > > >
> > > > > Don't want to derail the confirmation, but those results are
> terrible,
> > > > > they always say things changed in places not touched by the code of
> > > the MR,
> > > > > any idea why?
> > > > >
> > > >
> > > > Unfortunately not - my only guess would be that cppcheck reports
> results
> > > > slightly differently which Gitlab has issues interpreting.
> > > >
> > > >
> > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > Seeing how at least in KDE Frameworks first regressions
> sneaked in
> > > > > without
> > > > > > > someone noticing (nobody looks at logs when the surface shows a
> > > green
> > > > > > > checkmark, e.g. kcoreaddons, kwidgetsaddons, kio, purpose,
> krunner
> > > on
> > > > > > > openSUSE
> > > > > > > and possibly more have regressed in recent weeks, see
> > > build.kde.org)
> > > > > this
> > > > > > > should be something to deal with better, right?
> > > > > >
> > > > > >
> > > > > > > Bhushan gave two first ideas just now on #kde-sysadmin:
> > > > > > > > Well we can add a switch that repos can commit to saying test
> > >

Re: Gitlab CI: failed unit tests vs. currently passing CI

2022-01-22 Thread Ben Cooksley
On Sun, Jan 23, 2022 at 12:38 PM Albert Astals Cid  wrote:

> El diumenge, 23 de gener de 2022, a les 0:09:23 (CET), Ben Cooksley va
> escriure:
> > On Sun, Jan 23, 2022 at 11:29 AM Albert Astals Cid 
> wrote:
> >
> > > El dissabte, 22 de gener de 2022, a les 6:11:29 (CET), Ben Cooksley va
> > > escriure:
> > > > EXCLUDE_DEPRECATED_BEFORE_AND_ATOn Sat, Jan 22, 2022 at 1:31 PM
> Friedrich
> > > > W. H. Kossebau  wrote:
> > > >
> > > > > Hi,
> > > >
> > > >
> > > > > seems that Gitlab CI is currently configured to show the green
> > > "Success"
> > > > > checkmark for pipeline runs even if unit tests are failing.
> > > > >
> > > >
> > > > That is correct, only compilation or other internal failures cause
> the
> > > > build to show a failure result.
> > > >
> > > >
> > > > > Reasons seems to be that there Gitlab only knows Yay or Nay,
> without
> > > the
> > > > > warning state level known from Jenkins.
> > > > >
> > > >
> > > > Also correct.
> > > >
> > > >
> > > > > And given that quite some projects (sadly) maintain a few long-time
> > > > > failing
> > > > > unit tests, having the pipeline fail on unit tests seems to have
> been
> > > seen
> > > > > as
> > > > > too aggressive
> > > >
> > > >
> > > > Correct again.
> > > >
> > > >
> > > > >
> > > > >
> > > > > This of course harms the purpose of the unit tests, when their
> failures
> > > > > are
> > > > > only noticed weeks later, not e.g. at MR discussion time.
> > > > >
> > > >
> > > > Gitlab does note changes in the test suite as can currently be seen
> on
> > > > https://invent.kde.org/frameworks/kio/-/merge_requests/708
> > > > Quoting the page:  "Test summary contained 33 failed and 16 fixed
> test
> > > > results out of 205 total tests"
> > > >
> > > > It does the same thing for Code Quality - "Code quality scanning
> detected
> > > > 51 changes in merged results"
> > >
> > > Don't want to derail the confirmation, but those results are terrible,
> > > they always say things changed in places not touched by the code of
> the MR,
> > > any idea why?
> > >
> >
> > Unfortunately not - my only guess would be that cppcheck reports results
> > slightly differently which Gitlab has issues interpreting.
>
> Can we just disable it?
>

Various things can be configured on a per-project basis. cppcheck is one of
them.
See
https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/config-template.yml#L21


>
> Look at the results here
> https://invent.kde.org/graphics/okular/-/merge_requests/544
>
> Major - Either the condition 'printDialog' is redundant or there is
> possible null pointer dereference: printDialog. (CWE-476)
> in part/part.cpp:3341
>
> Fixed: Major - Either the condition 'printDialog' is redundant or there is
> possible null pointer dereference: printDialog. (CWE-476)
> in part/part.cpp:3340
>
> gitlab my friend, don't you think that maybe, just maybe this is the same
> code and you shouldn't complain to me about it since the only change to
> that file is 3000 lines away from it?
>

This is possibly cppcheck's fault, but yes not terribly good work there on
fuzzing for line changes.


>
> I find it confusing, it always makes me sad lowering my productivity.
>
> Cheers,
>   Albert
>
>
>
Cheers,
Ben


Re: Gitlab CI: failed unit tests vs. currently passing CI

2022-01-22 Thread Ben Cooksley
On Sun, Jan 23, 2022 at 12:29 PM Albert Astals Cid  wrote:

> El diumenge, 23 de gener de 2022, a les 0:09:23 (CET), Ben Cooksley va
> escriure:
> > On Sun, Jan 23, 2022 at 11:29 AM Albert Astals Cid 
> wrote:
> >
> > > El dissabte, 22 de gener de 2022, a les 6:11:29 (CET), Ben Cooksley va
> > > escriure:
> > > > EXCLUDE_DEPRECATED_BEFORE_AND_ATOn Sat, Jan 22, 2022 at 1:31 PM
> Friedrich
> > > > W. H. Kossebau  wrote:
> > > >
> > > > > Hi,
> > > >
> > > >
> > > > > seems that Gitlab CI is currently configured to show the green
> > > "Success"
> > > > > checkmark for pipeline runs even if unit tests are failing.
> > > > >
> > > >
> > > > That is correct, only compilation or other internal failures cause
> the
> > > > build to show a failure result.
> > > >
> > > >
> > > > > Reasons seems to be that there Gitlab only knows Yay or Nay,
> without
> > > the
> > > > > warning state level known from Jenkins.
> > > > >
> > > >
> > > > Also correct.
> > > >
> > > >
> > > > > And given that quite some projects (sadly) maintain a few long-time
> > > > > failing
> > > > > unit tests, having the pipeline fail on unit tests seems to have
> been
> > > seen
> > > > > as
> > > > > too aggressive
> > > >
> > > >
> > > > Correct again.
> > > >
> > > >
> > > > >
> > > > >
> > > > > This of course harms the purpose of the unit tests, when their
> failures
> > > > > are
> > > > > only noticed weeks later, not e.g. at MR discussion time.
> > > > >
> > > >
> > > > Gitlab does note changes in the test suite as can currently be seen
> on
> > > > https://invent.kde.org/frameworks/kio/-/merge_requests/708
> > > > Quoting the page:  "Test summary contained 33 failed and 16 fixed
> test
> > > > results out of 205 total tests"
> > > >
> > > > It does the same thing for Code Quality - "Code quality scanning
> detected
> > > > 51 changes in merged results"
> > >
> > > Don't want to derail the confirmation, but those results are terrible,
> > > they always say things changed in places not touched by the code of
> the MR,
> > > any idea why?
> > >
> >
> > Unfortunately not - my only guess would be that cppcheck reports results
> > slightly differently which Gitlab has issues interpreting.
> >
> >
> > >
> > > >
> > > >
> > > > >
> > > > > Seeing how at least in KDE Frameworks first regressions sneaked in
> > > without
> > > > > someone noticing (nobody looks at logs when the surface shows a
> green
> > > > > checkmark, e.g. kcoreaddons, kwidgetsaddons, kio, purpose, krunner
> on
> > > > > openSUSE
> > > > > and possibly more have regressed in recent weeks, see
> build.kde.org)
> > > this
> > > > > should be something to deal with better, right?
> > > >
> > > >
> > > > > Bhushan gave two first ideas just now on #kde-sysadmin:
> > > > > > Well we can add a switch that repos can commit to saying test
> > > failure is
> > > > > build failure
> > > > > > Another alternative is we use bot to write a comment on MR
> > > > >
> > > > > IMHO, to give unit tests the purpose they have, we should by
> default to
> > > > > let
> > > > > test failures be build failures. And have projects opt out if they
> > > need to
> > > > > have some unit tests keep failing, instead of e.g. tagging them
> with
> > > > > expected
> > > > > failures or handling any special environment they run into on the
> CI.
> > > > >
> > > > > Your opinions?
> > > > >
> > > >
> > > > The switch will need to be around the other way i'm afraid as there
> are
> > > > simply too many projects with broken tests right now.
> > > > The best place for that switch will be in .kde-ci.yml.
> > > >
> > > > My only concern however would be abuse of this switch, much in the
> way
> > > that
> > > > 

Re: Gitlab CI: failed unit tests vs. currently passing CI

2022-01-22 Thread Ben Cooksley
On Sun, Jan 23, 2022 at 11:29 AM Albert Astals Cid  wrote:

> El dissabte, 22 de gener de 2022, a les 6:11:29 (CET), Ben Cooksley va
> escriure:
> > EXCLUDE_DEPRECATED_BEFORE_AND_ATOn Sat, Jan 22, 2022 at 1:31 PM Friedrich
> > W. H. Kossebau  wrote:
> >
> > > Hi,
> >
> >
> > > seems that Gitlab CI is currently configured to show the green
> "Success"
> > > checkmark for pipeline runs even if unit tests are failing.
> > >
> >
> > That is correct, only compilation or other internal failures cause the
> > build to show a failure result.
> >
> >
> > > Reasons seems to be that there Gitlab only knows Yay or Nay, without
> the
> > > warning state level known from Jenkins.
> > >
> >
> > Also correct.
> >
> >
> > > And given that quite some projects (sadly) maintain a few long-time
> > > failing
> > > unit tests, having the pipeline fail on unit tests seems to have been
> seen
> > > as
> > > too aggressive
> >
> >
> > Correct again.
> >
> >
> > >
> > >
> > > This of course harms the purpose of the unit tests, when their failures
> > > are
> > > only noticed weeks later, not e.g. at MR discussion time.
> > >
> >
> > Gitlab does note changes in the test suite as can currently be seen on
> > https://invent.kde.org/frameworks/kio/-/merge_requests/708
> > Quoting the page:  "Test summary contained 33 failed and 16 fixed test
> > results out of 205 total tests"
> >
> > It does the same thing for Code Quality - "Code quality scanning detected
> > 51 changes in merged results"
>
> Don't want to derail the confirmation, but those results are terrible,
> they always say things changed in places not touched by the code of the MR,
> any idea why?
>

Unfortunately not - my only guess would be that cppcheck reports results
slightly differently which Gitlab has issues interpreting.


>
> >
> >
> > >
> > > Seeing how at least in KDE Frameworks first regressions sneaked in
> without
> > > someone noticing (nobody looks at logs when the surface shows a green
> > > checkmark, e.g. kcoreaddons, kwidgetsaddons, kio, purpose, krunner on
> > > openSUSE
> > > and possibly more have regressed in recent weeks, see build.kde.org)
> this
> > > should be something to deal with better, right?
> >
> >
> > > Bhushan gave two first ideas just now on #kde-sysadmin:
> > > > Well we can add a switch that repos can commit to saying test
> failure is
> > > build failure
> > > > Another alternative is we use bot to write a comment on MR
> > >
> > > IMHO, to give unit tests the purpose they have, we should by default to
> > > let
> > > test failures be build failures. And have projects opt out if they
> need to
> > > have some unit tests keep failing, instead of e.g. tagging them with
> > > expected
> > > failures or handling any special environment they run into on the CI.
> > >
> > > Your opinions?
> > >
> >
> > The switch will need to be around the other way i'm afraid as there are
> > simply too many projects with broken tests right now.
> > The best place for that switch will be in .kde-ci.yml.
> >
> > My only concern however would be abuse of this switch, much in the way
> that
> > certain projects abuse EXCLUDE_DEPRECATED_BEFORE_AND_AT.
> > The last thing we would want would be for people to flip this switch and
> > then leave their CI builds in a failing state - meaning that actual
> > compilation failures would be missed (and then lead to CI maintenance
> > issues)
> >
> > Thoughts on that?
>
> Test failing should mark the CI as failed, anything other than that
> doesn't make sense. The CI did fail marking it as passed is lying to
> ourselves.


> We can *still* merge failed MR with failed CI, the Merge button is just
> red, but it will work.
>

There is a big difference between "this doesn't compile" (because someone
forgot to commit a header/etc, dependency change that isn't in place or
because of a platform specific issue) and "some tests failed".
What that encourages is for people to ignore the results from the CI system
- as they'll get used to ignoring the CI system saying something is failing.

While this is not such a big deal for Linux, it is a massive deal for the
smaller platforms that far less people run.

Saying you can merge when the CI says it is failing is setting ourselves up
for failure.


> Maybe this red button will convince people to fix their tests. (one can
> hope, right?)
>
> Of course if we do this change it should happen after we've done that
> change that fixes the test failing because of however gitlab CI is set-up
> (you mentioned we had to wait for Jenkins to be disabled for that)
>
> Cheers,
>   Albert
>

Regards,
Ben


>
> >
> >
> > >
> > > Cheers
> > > Friedrich
> > >
> >
> > Cheers,
> > Ben
> >
>
>
>
>
>


Re: Gitlab CI: failed unit tests vs. currently passing CI

2022-01-21 Thread Ben Cooksley
EXCLUDE_DEPRECATED_BEFORE_AND_ATOn Sat, Jan 22, 2022 at 1:31 PM Friedrich
W. H. Kossebau  wrote:

> Hi,


> seems that Gitlab CI is currently configured to show the green "Success"
> checkmark for pipeline runs even if unit tests are failing.
>

That is correct, only compilation or other internal failures cause the
build to show a failure result.


> Reasons seems to be that there Gitlab only knows Yay or Nay, without the
> warning state level known from Jenkins.
>

Also correct.


> And given that quite some projects (sadly) maintain a few long-time
> failing
> unit tests, having the pipeline fail on unit tests seems to have been seen
> as
> too aggressive


Correct again.


>
>
> This of course harms the purpose of the unit tests, when their failures
> are
> only noticed weeks later, not e.g. at MR discussion time.
>

Gitlab does note changes in the test suite as can currently be seen on
https://invent.kde.org/frameworks/kio/-/merge_requests/708
Quoting the page:  "Test summary contained 33 failed and 16 fixed test
results out of 205 total tests"

It does the same thing for Code Quality - "Code quality scanning detected
51 changes in merged results"


>
> Seeing how at least in KDE Frameworks first regressions sneaked in without
> someone noticing (nobody looks at logs when the surface shows a green
> checkmark, e.g. kcoreaddons, kwidgetsaddons, kio, purpose, krunner on
> openSUSE
> and possibly more have regressed in recent weeks, see build.kde.org) this
> should be something to deal with better, right?


> Bhushan gave two first ideas just now on #kde-sysadmin:
> > Well we can add a switch that repos can commit to saying test failure is
> build failure
> > Another alternative is we use bot to write a comment on MR
>
> IMHO, to give unit tests the purpose they have, we should by default to
> let
> test failures be build failures. And have projects opt out if they need to
> have some unit tests keep failing, instead of e.g. tagging them with
> expected
> failures or handling any special environment they run into on the CI.
>
> Your opinions?
>

The switch will need to be around the other way i'm afraid as there are
simply too many projects with broken tests right now.
The best place for that switch will be in .kde-ci.yml.

My only concern however would be abuse of this switch, much in the way that
certain projects abuse EXCLUDE_DEPRECATED_BEFORE_AND_AT.
The last thing we would want would be for people to flip this switch and
then leave their CI builds in a failing state - meaning that actual
compilation failures would be missed (and then lead to CI maintenance
issues)

Thoughts on that?


>
> Cheers
> Friedrich
>

Cheers,
Ben


[sysadmin/ci-management] seeds: Due to Plasma Framework having a hard dependency on KGlobalAccel we are forced to build it on Windows as well.

2022-01-05 Thread Ben Cooksley
Git commit b22096531b29c572197a6c5e78f0248ab5e84274 by Ben Cooksley.
Committed on 05/01/2022 at 18:37.
Pushed by bcooksley into branch 'master'.

Due to Plasma Framework having a hard dependency on KGlobalAccel we are forced 
to build it on Windows as well.
Just like KAuth/KTextEditor this falls into the category of no-op so we should 
probably look at a way of eliminating this hard dependency (unless 
plasma-framework makes no sense on Windows either of course...)

CCMAIL: kde-frameworks-devel@kde.org

M  +1-1seeds/frameworks-latest.yml

https://invent.kde.org/sysadmin/ci-management/commit/b22096531b29c572197a6c5e78f0248ab5e84274

diff --git a/seeds/frameworks-latest.yml b/seeds/frameworks-latest.yml
index 4b86e04..90ce94a 100644
--- a/seeds/frameworks-latest.yml
+++ b/seeds/frameworks-latest.yml
@@ -57,6 +57,7 @@
 "frameworks/kdoctools": "master"
 "frameworks/kemoticons": "master"
 "frameworks/kfilemetadata": "master"
+"frameworks/kglobalaccel": "master"
 "frameworks/khtml": "master"
 "frameworks/kidletime": "master"
 "frameworks/kinit": "master"
@@ -85,7 +86,6 @@
 "frameworks/baloo": "master"
 "frameworks/kactivities-stats": "master"
 "frameworks/kdesu": "master"
-"frameworks/kglobalaccel": "master"
 "frameworks/kpty": "master"
 "frameworks/kwayland": "master"
 


[sysadmin/ci-management] seeds: Due to KTextEditor having a hard dependency on KAuth on Windows, we need to include KAuth in the seed for build on Windows.

2022-01-04 Thread Ben Cooksley
Git commit 63591f66d36d7914b847e471ee6f3e789cbcb4cf by Ben Cooksley.
Committed on 05/01/2022 at 03:58.
Pushed by bcooksley into branch 'master'.

Due to KTextEditor having a hard dependency on KAuth on Windows, we need to 
include KAuth in the seed for build on Windows.

CCMAIL: kde-frameworks-devel@kde.org

M  +1-2seeds/frameworks-latest.yml

https://invent.kde.org/sysadmin/ci-management/commit/63591f66d36d7914b847e471ee6f3e789cbcb4cf

diff --git a/seeds/frameworks-latest.yml b/seeds/frameworks-latest.yml
index 3837da2..4b86e04 100644
--- a/seeds/frameworks-latest.yml
+++ b/seeds/frameworks-latest.yml
@@ -47,6 +47,7 @@
 "frameworks/breeze-icons": "master"
 "frameworks/frameworkintegration": "master"
 "frameworks/kactivities": "master"
+"frameworks/kauth": "master"
 "frameworks/kdbusaddons": "master"
 "frameworks/kdeclarative": "master"
 "frameworks/kdelibs4support": "master"
@@ -81,10 +82,8 @@
   'require':
 "libraries/plasma-wayland-protocols": "master"
 "libraries/polkit-qt-1": "master"
-
 "frameworks/baloo": "master"
 "frameworks/kactivities-stats": "master"
-"frameworks/kauth": "master"
 "frameworks/kdesu": "master"
 "frameworks/kglobalaccel": "master"
 "frameworks/kpty": "master"


[frameworks/ktexteditor] /: KTextEditor has a hard dependency on KAuth - ensure it is available.

2022-01-04 Thread Ben Cooksley
Git commit b83f75b7be0c289a55e4afc1d6281c2f5c9fbffa by Ben Cooksley.
Committed on 05/01/2022 at 03:56.
Pushed by bcooksley into branch 'master'.

KTextEditor has a hard dependency on KAuth - ensure it is available.
On Linux/FreeBSD this is normally received via KIO - however it only requires 
KAuth on those two platforms meaning the build will fail on Windows without 
this explicit dependency.

Given that KAuth is a no-op on Windows it would be worthwhile investigating if 
KTextEditor can have an optional dependency on KAuth as well.

CCMAIL: kde-frameworks-devel@kde.org

M  +1-0.kde-ci.yml

https://invent.kde.org/frameworks/ktexteditor/commit/b83f75b7be0c289a55e4afc1d6281c2f5c9fbffa

diff --git a/.kde-ci.yml b/.kde-ci.yml
index f58a20f6..280bc31b 100644
--- a/.kde-ci.yml
+++ b/.kde-ci.yml
@@ -3,6 +3,7 @@ Dependencies:
   'require':
 'frameworks/extra-cmake-modules': '@same'
 'frameworks/karchive' : '@same'
+'frameworks/kauth': '@same'
 'frameworks/kconfig' : '@same'
 'frameworks/kguiaddons' : '@same'
 'frameworks/ki18n' : '@same'


Re: Gitlab CI for Windows

2022-01-04 Thread Ben Cooksley
On Wed, Jan 5, 2022 at 8:53 AM Christoph Cullmann (cullmann.io) <
christ...@cullmann.io> wrote:

> On 2022-01-04 20:23, Ben Cooksley wrote:
> > On Wed, Jan 5, 2022 at 6:36 AM Christoph Cullmann (cullmann.io [1])
> >  wrote:
> >
> >> On 2022-01-04 18:24, Ben Cooksley wrote:
> >>> Hi all,
> >>>
> >>> Next update in this saga appears to be a defect in KDeclarative,
> >> which
> >>> apparently has a hard dependency on KGlobalAccel.
> >>> https://invent.kde.org/sysadmin/ci-management/-/jobs/195039
> >>>
> >>> While this is something that we have previously built on Windows,
> >> from
> >>> my understanding it is essentially a no-op that does nothing so we
> >>> should probably skip building it.
> >>>
> >>> Can someone please take a look into this and advise whether
> >>> KDeclarative can also make it optional?
> >>
> >> Hi,
> >>
> >> I can take a look.
> >
> > If you could, that would be much appreciated.
>
> Nicolas was faster :=)
>
> I would assume master should already build.
>
> I have some additional small patch here
>
>
> https://invent.kde.org/frameworks/kdeclarative/-/commit/7200ad3d518f199ac040afcaf8d3330fd3f79ab7
>
> Btw., it seems the unit tests fail in the classic CI.
> Is it possible that some data/ dir is created in bin/ for Windows?
> This seems to confuse the auto tests where to find their input.
>

Yes, that is to be expected.

On Windows QStandardPaths expects to find resources in the folder data/
immediately relative to the executable.
As our executables are installed into bin/ we therefore have to install
those resources (which on a OSS Unix system would be in $prefix/share/)
into $prefix/bin/data/

The CI Tooling also does some additional tweaking in that department by
copying in the resources from the libraries made available via Craft to
that location as well (otherwise things like shared-mime-database won't be
found)


> Greetings
> Christoph
>

Cheers,
Ben


> >
> >> Greetings
> >> Christoph
> >
> > Cheers,
> > Ben
> >
> >>>
> >>> Thanks,
> >>> Ben
> >>>
> >>> On Tue, Jan 4, 2022 at 7:51 AM Ben Cooksley 
> >> wrote:
> >>>
> >>>> On Mon, Jan 3, 2022 at 9:00 AM Ben Cooksley 
> >>>> wrote:
> >>>>
> >>>>> Hi all,
> >>>>>
> >>>>> Over the past few days substantial progress has been made in
> >>>>> getting Windows builds running under Gitlab, to the point where
> >>>>> some Frameworks are now successfully compiling.
> >>>>>
> >>>>> Unfortunately we've run into a little issue with breeze-icons as
> >>>>> can be seen at
> >>>>> https://invent.kde.org/sysadmin/ci-management/-/jobs/193039
> >>>>
> >>>> Following investigation and some testing by Harald we've
> >> confirmed
> >>>> that this is a CMake bug - with it being unable to handle
> >> symlinks
> >>>> on Windows correctly.
> >>>> For now I shall workaround the issue by disabling use of symlinks
> >> on
> >>>> Windows in Git (git config --system core.symlinks false) however
> >>>> that is not an ideal long term fix.
> >>>>
> >>>> Do we have any contacts at CMake we can escalate this bug to?
> >>>>
> >>>> As for why this didn't show up earlier - it seems our Windows
> >>>> builders for Jenkins have symlinks disabled (indicating that
> >> either
> >>>> the feature was still too experimental back then or that we did
> >> hit
> >>>> this back then and worked around it then as well)
> >>>>
> >>>>> Any ideas?
> >>>>>
> >>>>> Thanks,
> >>>>> Ben
> >>>>
> >>>> Cheers,
> >>>> Ben
> >>
> >> --
> >> Ignorance is bliss...
> >> https://cullmann.io | https://kate-editor.org
> >
> >
> > Links:
> > --
> > [1] http://cullmann.io
>
> --
> Ignorance is bliss...
> https://cullmann.io | https://kate-editor.org
>


Re: Gitlab CI for Windows

2022-01-04 Thread Ben Cooksley
On Wed, Jan 5, 2022 at 6:36 AM Christoph Cullmann (cullmann.io) <
christ...@cullmann.io> wrote:

> On 2022-01-04 18:24, Ben Cooksley wrote:
> > Hi all,
> >
> > Next update in this saga appears to be a defect in KDeclarative, which
> > apparently has a hard dependency on KGlobalAccel.
> > https://invent.kde.org/sysadmin/ci-management/-/jobs/195039
> >
> > While this is something that we have previously built on Windows, from
> > my understanding it is essentially a no-op that does nothing so we
> > should probably skip building it.
> >
> > Can someone please take a look into this and advise whether
> > KDeclarative can also make it optional?
>
> Hi,
>
> I can take a look.
>

If you could, that would be much appreciated.


> Greetings
> Christoph
>

Cheers,
Ben


> >
> > Thanks,
> > Ben
> >
> > On Tue, Jan 4, 2022 at 7:51 AM Ben Cooksley  wrote:
> >
> >> On Mon, Jan 3, 2022 at 9:00 AM Ben Cooksley 
> >> wrote:
> >>
> >>> Hi all,
> >>>
> >>> Over the past few days substantial progress has been made in
> >>> getting Windows builds running under Gitlab, to the point where
> >>> some Frameworks are now successfully compiling.
> >>>
> >>> Unfortunately we've run into a little issue with breeze-icons as
> >>> can be seen at
> >>> https://invent.kde.org/sysadmin/ci-management/-/jobs/193039
> >>
> >> Following investigation and some testing by Harald we've confirmed
> >> that this is a CMake bug - with it being unable to handle symlinks
> >> on Windows correctly.
> >> For now I shall workaround the issue by disabling use of symlinks on
> >> Windows in Git (git config --system core.symlinks false) however
> >> that is not an ideal long term fix.
> >>
> >> Do we have any contacts at CMake we can escalate this bug to?
> >>
> >> As for why this didn't show up earlier - it seems our Windows
> >> builders for Jenkins have symlinks disabled (indicating that either
> >> the feature was still too experimental back then or that we did hit
> >> this back then and worked around it then as well)
> >>
> >>> Any ideas?
> >>>
> >>> Thanks,
> >>> Ben
> >>
> >> Cheers,
> >> Ben
>
> --
> Ignorance is bliss...
> https://cullmann.io | https://kate-editor.org
>


Re: Gitlab CI for Windows

2022-01-04 Thread Ben Cooksley
Hi all,

Next update in this saga appears to be a defect in KDeclarative, which
apparently has a hard dependency on KGlobalAccel.
https://invent.kde.org/sysadmin/ci-management/-/jobs/195039

While this is something that we have previously built on Windows, from my
understanding it is essentially a no-op that does nothing so we should
probably skip building it.

Can someone please take a look into this and advise whether KDeclarative
can also make it optional?

Thanks,
Ben


On Tue, Jan 4, 2022 at 7:51 AM Ben Cooksley  wrote:

> On Mon, Jan 3, 2022 at 9:00 AM Ben Cooksley  wrote:
>
>> Hi all,
>>
>> Over the past few days substantial progress has been made in getting
>> Windows builds running under Gitlab, to the point where some Frameworks are
>> now successfully compiling.
>>
>> Unfortunately we've run into a little issue with breeze-icons as can be
>> seen at https://invent.kde.org/sysadmin/ci-management/-/jobs/193039
>>
>
> Following investigation and some testing by Harald we've confirmed that
> this is a CMake bug - with it being unable to handle symlinks on Windows
> correctly.
> For now I shall workaround the issue by disabling use of symlinks on
> Windows in Git (git config --system core.symlinks false) however that is
> not an ideal long term fix.
>
> Do we have any contacts at CMake we can escalate this bug to?
>
> As for why this didn't show up earlier - it seems our Windows builders for
> Jenkins have symlinks disabled (indicating that either the feature was
> still too experimental back then or that we did hit this back then and
> worked around it then as well)
>
>
>> Any ideas?
>>
>> Thanks,
>> Ben
>>
>
> Cheers,
> Ben
>


Re: Gitlab CI for Windows

2022-01-03 Thread Ben Cooksley
On Mon, Jan 3, 2022 at 9:00 AM Ben Cooksley  wrote:

> Hi all,
>
> Over the past few days substantial progress has been made in getting
> Windows builds running under Gitlab, to the point where some Frameworks are
> now successfully compiling.
>
> Unfortunately we've run into a little issue with breeze-icons as can be
> seen at https://invent.kde.org/sysadmin/ci-management/-/jobs/193039
>

Following investigation and some testing by Harald we've confirmed that
this is a CMake bug - with it being unable to handle symlinks on Windows
correctly.
For now I shall workaround the issue by disabling use of symlinks on
Windows in Git (git config --system core.symlinks false) however that is
not an ideal long term fix.

Do we have any contacts at CMake we can escalate this bug to?

As for why this didn't show up earlier - it seems our Windows builders for
Jenkins have symlinks disabled (indicating that either the feature was
still too experimental back then or that we did hit this back then and
worked around it then as well)


> Any ideas?
>
> Thanks,
> Ben
>

Cheers,
Ben


Gitlab CI for Windows

2022-01-02 Thread Ben Cooksley
Hi all,

Over the past few days substantial progress has been made in getting
Windows builds running under Gitlab, to the point where some Frameworks are
now successfully compiling.

Unfortunately we've run into a little issue with breeze-icons as can be
seen at https://invent.kde.org/sysadmin/ci-management/-/jobs/193039

Any ideas?

Thanks,
Ben


Re: Announcing General Availability of Gitlab CI for Linux, FreeBSD and Android

2021-12-03 Thread Ben Cooksley
On Fri, Dec 3, 2021 at 8:53 AM Michael Reeves  wrote:

> Where are the artifacts being uploaded to? They don't seem to show in
> gitlab. In particular the coverage report is of interest to me.
>

They are uploaded to Gitlab itself, but as reports so only some parts of
Gitlab's functionality show them with a download option.
You should be able to find the necessary links on the Pipelines page for a
project - https://invent.kde.org/sdk/kdiff3/-/pipelines

Cheers,
Ben


> On Sat, Nov 27, 2021 at 1:31 PM Ben Cooksley  wrote:
>
>> Hi all,
>>
>> As mentioned in the subject, i'm happy to announce the general
>> availability of native Gitlab CI for all KDE projects for Linux, FreeBSD
>> and Android.
>>
>> For those who would like to get their projects setup, please ensure you
>> first have a .kde-ci.yml file in the root of your repository specifiying
>> the necessary dependencies of your project and any options your project
>> requires to build and conduct tests. You can find a list of all the
>> supported options at
>> https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/config-template.yml
>> along with guidelines on how to specify your dependencies.
>>
>> Please note the following when specifying the branch to use:
>> - @same should be used for projects which are released together and which
>> follow the same branching scheme (release/21.08 for instance)
>> - @stable should be used to refer to the current stable branch
>> - @latest should be used to refer to the current development branch
>>
>> While the system has been seeded based on the current dependency
>> metadata, if this was incorrect for your project then you may find that the
>> system gives an error that it cannot locate the dependency when first run.
>> If this happens to you then please look into submitting a merge request to
>> https://invent.kde.org/sysadmin/ci-management/-/tree/master/seeds so
>> that this dependency can be included in future seed runs.
>>
>> Once you have your .kde-ci.yml file setup you can then setup your
>> .gitlab-ci.yml to include the appropriate platform files from
>> https://invent.kde.org/sysadmin/ci-utilities/-/tree/master/gitlab-templates
>> .
>>
>> Please note that currently this is limited to builds for master and other
>> development branches, as the necessary metadata for @stable is not yet
>> fully in place. This information can be found at
>> https://invent.kde.org/sysadmin/repo-metadata/-/blob/master/branch-rules.yml
>> - it would be appreciated if developers could please look into updating
>> this for their projects at the same time.
>>
>> With regards to Windows builds, they should become available soon - there
>> is still some infrastructural work that needs to be completed on our Docker
>> image, as well as the facilities to build those images which needs to be
>> completed prior to these being available. We will be transitioning to using
>> Docker for our Windows builds which will hopefully eliminate many of the
>> issues we've had with lingering processes that have plagued the Jenkins
>> system.
>>
>> With the availability of these Gitlab builds we are now entering the
>> twilight phase of our Jenkins based system at build.kde.org so it is
>> imperative that developers setup their CI using the new tooling. For those
>> using the legacy Jenkins system with Gitlab (using the templates at
>> https://invent.kde.org/sysadmin/ci-tooling/-/tree/master/invent) these
>> will be discontinued at the same time as build.kde.org, so you will need
>> to migrate to the new tooling as well.
>>
>> Please let us know if you have any questions on the above.
>>
>> Thanks,
>> Ben
>>
>


Announcing General Availability of Gitlab CI for Linux, FreeBSD and Android

2021-11-27 Thread Ben Cooksley
Hi all,

As mentioned in the subject, i'm happy to announce the general availability
of native Gitlab CI for all KDE projects for Linux, FreeBSD and Android.

For those who would like to get their projects setup, please ensure you
first have a .kde-ci.yml file in the root of your repository specifiying
the necessary dependencies of your project and any options your project
requires to build and conduct tests. You can find a list of all the
supported options at
https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/config-template.yml
along with guidelines on how to specify your dependencies.

Please note the following when specifying the branch to use:
- @same should be used for projects which are released together and which
follow the same branching scheme (release/21.08 for instance)
- @stable should be used to refer to the current stable branch
- @latest should be used to refer to the current development branch

While the system has been seeded based on the current dependency metadata,
if this was incorrect for your project then you may find that the system
gives an error that it cannot locate the dependency when first run. If this
happens to you then please look into submitting a merge request to
https://invent.kde.org/sysadmin/ci-management/-/tree/master/seeds so that
this dependency can be included in future seed runs.

Once you have your .kde-ci.yml file setup you can then setup your
.gitlab-ci.yml to include the appropriate platform files from
https://invent.kde.org/sysadmin/ci-utilities/-/tree/master/gitlab-templates.

Please note that currently this is limited to builds for master and other
development branches, as the necessary metadata for @stable is not yet
fully in place. This information can be found at
https://invent.kde.org/sysadmin/repo-metadata/-/blob/master/branch-rules.yml
- it would be appreciated if developers could please look into updating
this for their projects at the same time.

With regards to Windows builds, they should become available soon - there
is still some infrastructural work that needs to be completed on our Docker
image, as well as the facilities to build those images which needs to be
completed prior to these being available. We will be transitioning to using
Docker for our Windows builds which will hopefully eliminate many of the
issues we've had with lingering processes that have plagued the Jenkins
system.

With the availability of these Gitlab builds we are now entering the
twilight phase of our Jenkins based system at build.kde.org so it is
imperative that developers setup their CI using the new tooling. For those
using the legacy Jenkins system with Gitlab (using the templates at
https://invent.kde.org/sysadmin/ci-tooling/-/tree/master/invent) these will
be discontinued at the same time as build.kde.org, so you will need to
migrate to the new tooling as well.

Please let us know if you have any questions on the above.

Thanks,
Ben


Re: Fwd: KDE CI: Administration » Dependency Build Applications kf5-qt5 FreeBSDQt5.15 - Build # 127 - Still Failing!

2021-11-18 Thread Ben Cooksley
On Fri, Nov 19, 2021 at 6:39 AM Volker Krause  wrote:

> I looked into this and it seems the problem had already been addressed
> prior
> to your email, so all I ended up doing is pressing the rebuild button.
>
>
> The change starting this was https://invent.kde.org/frameworks/ki18n/-/
> merge_requests/21
> <https://invent.kde.org/frameworks/ki18n/-/merge_requests/21>, by me.
> What actually caused the breakage however was the
> way deprecation versions are managed. Not the first time, and not entirely
> surprising, the MR comments even mention that risk.
>
> There's two approaches on how to handle such changes without breaking the
> build:
>
> (1) Change KF, port apps after the next KF release, deprecate old KF API
> in a
> subsequent release once porting has been completed.
> (2) Change KF and deprecate old API immediately, port apps after the next
> KF
> release and then bump the deprecation version once that has been completed.
>
> Both have been rejected previously and I have yet to see an alternative
> workflow that allows KF changes/deprecation while avoiding breakage like
> we
> have seen here. I very much share your frustration in that regard.
>

Is this the same option that causes API to simply disappear if certain
compiler flags have been set?
I recall it causing substantial fallout in the past.


> Regards,
> Volker
>

Cheers,
Ben


>
>
> On Donnerstag, 18. November 2021 10:24:18 CET Ben Cooksley wrote:
> > Hi PIM Developers,
> >
> > Please see the below breakage in your code, which is impacting both Linux
> > and FreeBSD.
> > This breakage is preventing us from rebuilding the images, which is
> > blocking the web team from deploying changes to apps.kde.org, and also
> > preventing the rollout of new packages requested by the Frameworks
> > developers.
> >
> > Please note that once again it is a PIM breakage which is limiting our
> > ability to work on the CI system, which is both frustrating and quite sad
> > as this should not be happening.
> >
> > Regards,
> > Ben
> >
> > -- Forwarded message -
> > From: CI System 
> > Date: Thu, Nov 18, 2021 at 12:10 AM
> > Subject: KDE CI: Administration » Dependency Build Applications kf5-qt5
> > FreeBSDQt5.15 - Build # 127 - Still Failing!
> > To: 
> >
> >
> > *BUILD FAILURE*
> > Build URL
> >
> https://build.kde.org/job/Administration/job/Dependency%20Build%20Applicatio
> > ns%20kf5-qt5%20FreeBSDQt5.15/127/ Project: Dependency Build Applications
> > kf5-qt5 FreeBSDQt5.15
> > Date of build: Wed, 17 Nov 2021 08:56:29 +
> > Build duration: 2 hr 13 min and counting
> > * CONSOLE OUTPUT *
> > [...truncated 165753 lines...]
> > [2021-11-17T11:08:36.898Z] [ 43%] Linking CXX executable
> > ../../../../bin/testspellchecklineedit
> > [2021-11-17T11:08:37.185Z] [ 43%] Built target translator_gui
> > [2021-11-17T11:08:37.185Z] [ 43%] Building CXX object
> >
> src/pimcommon/autocorrection/tests/CMakeFiles/autocorrection_gui.dir/autocor
> > rection_gui_autogen/mocs_compilation.cpp.o [2021-11-17T11:08:37.465Z] [
> 43%]
> > Built target migratefileinfotest [2021-11-17T11:08:37.465Z] [ 44%]
> Building
> > CXX object
> >
> src/pimcommon/customtools/autotests/CMakeFiles/customtoolswidgetngtest.dir/c
> > ustomtoolswidgetngtest_autogen/mocs_compilation.cpp.o
> > [2021-11-17T11:08:37.465Z] [ 44%] Built target testspellchecklineedit
> > [2021-11-17T11:08:37.465Z] [ 44%] Building CXX object
> >
> src/pimcommon/customtools/autotests/CMakeFiles/customtoolswidgetngtest.dir/c
> > ustomtoolswidgetngtest.cpp.o [2021-11-17T11:08:37.465Z] [ 44%] Building
> CXX
> > object
> >
> src/pimcommon/widgets/tests/CMakeFiles/customtoolswidgetng_gui.dir/customtoo
> > lswidgetng_gui.cpp.o [2021-11-17T11:08:38.086Z] [ 44%] Linking CXX
> > executable
> > ../../../../bin/richtexteditwithautocorrection_gui
> > [2021-11-17T11:08:38.357Z] [ 44%] Linking CXX executable
> > ../../../../bin/shareserviceurlmanagertest
> > [2021-11-17T11:08:38.628Z] [ 44%] Built target
> > richtexteditwithautocorrection_gui
> > [2021-11-17T11:08:38.628Z] [ 44%] Building CXX object
> >
> src/pimcommon/autotests/CMakeFiles/regularexpressiontests.dir/regularexpress
> > iontests_autogen/mocs_compilation.cpp.o [2021-11-17T11:08:38.927Z] [ 44%]
> > Built target shareserviceurlmanagertest [2021-11-17T11:08:38.927Z] [ 44%]
> > Building CXX object
> >
> src/pimcommon/logactivities/autotests/CMakeFiles/logactivitieswidgettest.dir
> > /logactivitieswidgettest_autogen/mocs_compilation.cpp.o
> > [2021-11-17

Fwd: KDE CI: Administration » Dependency Build Applications kf5-qt5 FreeBSDQt5.15 - Build # 127 - Still Failing!

2021-11-18 Thread Ben Cooksley
Hi PIM Developers,

Please see the below breakage in your code, which is impacting both Linux
and FreeBSD.
This breakage is preventing us from rebuilding the images, which is
blocking the web team from deploying changes to apps.kde.org, and also
preventing the rollout of new packages requested by the Frameworks
developers.

Please note that once again it is a PIM breakage which is limiting our
ability to work on the CI system, which is both frustrating and quite sad
as this should not be happening.

Regards,
Ben

-- Forwarded message -
From: CI System 
Date: Thu, Nov 18, 2021 at 12:10 AM
Subject: KDE CI: Administration » Dependency Build Applications kf5-qt5
FreeBSDQt5.15 - Build # 127 - Still Failing!
To: 


*BUILD FAILURE*
Build URL
https://build.kde.org/job/Administration/job/Dependency%20Build%20Applications%20kf5-qt5%20FreeBSDQt5.15/127/
Project: Dependency Build Applications kf5-qt5 FreeBSDQt5.15
Date of build: Wed, 17 Nov 2021 08:56:29 +
Build duration: 2 hr 13 min and counting
* CONSOLE OUTPUT *
[...truncated 165753 lines...]
[2021-11-17T11:08:36.898Z] [ 43%] Linking CXX executable
../../../../bin/testspellchecklineedit
[2021-11-17T11:08:37.185Z] [ 43%] Built target translator_gui
[2021-11-17T11:08:37.185Z] [ 43%] Building CXX object
src/pimcommon/autocorrection/tests/CMakeFiles/autocorrection_gui.dir/autocorrection_gui_autogen/mocs_compilation.cpp.o
[2021-11-17T11:08:37.465Z] [ 43%] Built target migratefileinfotest
[2021-11-17T11:08:37.465Z] [ 44%] Building CXX object
src/pimcommon/customtools/autotests/CMakeFiles/customtoolswidgetngtest.dir/customtoolswidgetngtest_autogen/mocs_compilation.cpp.o
[2021-11-17T11:08:37.465Z] [ 44%] Built target testspellchecklineedit
[2021-11-17T11:08:37.465Z] [ 44%] Building CXX object
src/pimcommon/customtools/autotests/CMakeFiles/customtoolswidgetngtest.dir/customtoolswidgetngtest.cpp.o
[2021-11-17T11:08:37.465Z] [ 44%] Building CXX object
src/pimcommon/widgets/tests/CMakeFiles/customtoolswidgetng_gui.dir/customtoolswidgetng_gui.cpp.o
[2021-11-17T11:08:38.086Z] [ 44%] Linking CXX executable
../../../../bin/richtexteditwithautocorrection_gui
[2021-11-17T11:08:38.357Z] [ 44%] Linking CXX executable
../../../../bin/shareserviceurlmanagertest
[2021-11-17T11:08:38.628Z] [ 44%] Built target
richtexteditwithautocorrection_gui
[2021-11-17T11:08:38.628Z] [ 44%] Building CXX object
src/pimcommon/autotests/CMakeFiles/regularexpressiontests.dir/regularexpressiontests_autogen/mocs_compilation.cpp.o
[2021-11-17T11:08:38.927Z] [ 44%] Built target shareserviceurlmanagertest
[2021-11-17T11:08:38.927Z] [ 44%] Building CXX object
src/pimcommon/logactivities/autotests/CMakeFiles/logactivitieswidgettest.dir/logactivitieswidgettest_autogen/mocs_compilation.cpp.o
[2021-11-17T11:08:38.927Z] [ 44%] Building CXX object
src/pimcommon/autocorrection/tests/CMakeFiles/autocorrection_gui.dir/autocorrection_gui.cpp.o
[2021-11-17T11:08:39.514Z] [ 45%] Building CXX object
src/pimcommon/autotests/CMakeFiles/regularexpressiontests.dir/regularexpressiontests.cpp.o
[2021-11-17T11:08:39.774Z] [ 45%] Linking CXX executable
../../../../bin/customtoolswidgetng_gui
[2021-11-17T11:08:40.046Z] [ 45%] Linking CXX executable
../../../../bin/customtoolswidgetngtest
[2021-11-17T11:08:40.046Z] [ 46%] Building CXX object
src/pimcommon/logactivities/autotests/CMakeFiles/logactivitieswidgettest.dir/logactivitieswidgettest.cpp.o
[2021-11-17T11:08:40.319Z] [ 46%] Built target customtoolswidgetng_gui
[2021-11-17T11:08:40.319Z] [ 47%] Building CXX object
src/pimcommon/logactivities/autotests/CMakeFiles/logactivitiesdialogtest.dir/logactivitiesdialogtest_autogen/mocs_compilation.cpp.o
[2021-11-17T11:08:40.319Z] [ 47%] Building CXX object
src/pimcommon/logactivities/autotests/CMakeFiles/logactivitiesmanagertest.dir/logactivitiesmanagertest_autogen/mocs_compilation.cpp.o
[2021-11-17T11:08:40.653Z] [ 47%] Built target customtoolswidgetngtest
[2021-11-17T11:08:40.653Z] [ 47%] Building CXX object
src/pimcommon/logactivities/autotests/CMakeFiles/logactivitiesdialogtest.dir/logactivitiesdialogtest.cpp.o
[2021-11-17T11:08:41.692Z] [ 48%] Linking CXX executable
../../../../bin/autocorrection_gui
[2021-11-17T11:08:41.967Z] [ 48%] Building CXX object
src/pimcommon/logactivities/autotests/CMakeFiles/logactivitiesmanagertest.dir/logactivitiesmanagertest.cpp.o
[2021-11-17T11:08:41.967Z] [ 48%] Building CXX object
src/pimcommon/logactivities/tests/CMakeFiles/logactivities_gui.dir/logactivities_gui_autogen/mocs_compilation.cpp.o
[2021-11-17T11:08:42.275Z] [ 48%] Built target autocorrection_gui
[2021-11-17T11:08:42.275Z] [ 49%] Building CXX object
src/pimcommon/configureplugins/autotests/CMakeFiles/configurepluginslistwidgettest.dir/configurepluginslistwidgettest_autogen/mocs_compilation.cpp.o
[2021-11-17T11:08:42.897Z] [ 49%] Linking CXX executable
../../../../bin/logactivitieswidgettest
[2021-11-17T11:08:42.897Z] [ 49%] Linking CXX executable
../../../bin/regularexpressiontests
[2021-11-17T11:08:43.182Z] [ 49%] 

[sysadmin/ci-tooling] system-images/suse-qt515: Strip out all packages relating to Python bindings support from our SUSE CI images.

2021-11-09 Thread Ben Cooksley
Git commit 6a92fdf747990d2e074e92b2bdc224efc9b08740 by Ben Cooksley.
Committed on 10/11/2021 at 07:37.
Pushed by bcooksley into branch 'master'.

Strip out all packages relating to Python bindings support from our SUSE CI 
images.
The CMake target dependencies aren't sufficiently setup within the Frameworks 
for the Python bindings, which means our builds for those Frameworks with 
complicated bindings - like KConfig - are now highly deterministic.
This poses severe issues for our Dependency Builds causing them to fail quite 
often, making it very difficult to perform maintenance on the CI system.

CCMAIL: kde-frameworks-devel@kde.org

M  +0-2system-images/suse-qt515/Dockerfile

https://invent.kde.org/sysadmin/ci-tooling/commit/6a92fdf747990d2e074e92b2bdc224efc9b08740

diff --git a/system-images/suse-qt515/Dockerfile 
b/system-images/suse-qt515/Dockerfile
index 812adb0..02cb243 100755
--- a/system-images/suse-qt515/Dockerfile
+++ b/system-images/suse-qt515/Dockerfile
@@ -75,8 +75,6 @@ RUN zypper --non-interactive in --allow-vendor-change \
 aspell-devel \
 hunspell-devel \
 libvoikko-devel \
-# Python bindings in Frameworks
-python38-qt5-sip python38-qt5-devel python3-sip4 python38-qt5 
python38-sip4-devel python3-clang \
 # kio-extras and krdc
 libssh-devel \
 # plasma-pa


CI System Issues

2021-11-09 Thread Ben Cooksley
Hi all,

Over the past 24-48 hours we have been experiencing significant issues with
both of the Jenkins' instances which operate build.kde.org and
binary-factory.kde.org.

After significant investigative work these issues have been traced to a
bug/regression within Docker itself, which we have now applied a workaround
for. This issue is tracked at upstream bug
https://github.com/moby/moby/issues/42375 which unfortunately has received
little attention thus far.

With the resolution of this issue we've now initiated Global Rebuilds
across Jenkins to restore normal service. As a consequence of this it may
take several hours for the CI system to attend to builds that have been
initiated, including updates to our websites.

Due to their unreliability in the build chain I will shortly also be
terminating support for building Python bindings on Linux, as they are
interfering in the ability of the Dependency Builds - and thus the Global
Rebuild - to complete.

Apologies for the disruption to service.

Thanks,
Ben Cooksley
KDE Sysadmin


Re: KDE CI: Plasma » plasma-framework » kf5-qt5 FreeBSDQt5.15 - Build # 644 - Still Failing!

2021-10-18 Thread Ben Cooksley
On Tue, Oct 19, 2021 at 5:52 AM Albert Astals Cid  wrote:

> El dilluns, 18 d’octubre de 2021, a les 13:59:24 (CEST), Vlad Zahorodnii
> va escriure:
> > On 10/18/21 14:25, Aleix Pol wrote:
> > > Does anyone know why we are having this problem all of a sudden? I
> > > cannot reproduce on my system.
> > > Plasma Workspace is failing the same:
> > >
> https://build.kde.org/job/Plasma/job/plasma-workspace/job/kf5-qt5%20SUSEQt5.15/1800/
> > > <
> https://build.kde.org/job/Plasma/job/plasma-workspace/job/kf5-qt5%20SUSEQt5.15/1800/
> >
> >
> > It might be that plasma-framework is built with older kcoreaddons.
>
> And kconfig
>
> GHNS is not a member of KAuthorized
> 13:17:20198 | if (KAuthorized::authorize(KAuthorized::GHNS)) {
>
> I've triggered a new build of
>
> https://build.kde.org/job/Administration/job/Dependency%20Build%20Plasma%20kf5-qt5%20SUSEQt5.15/
>
> Which as far as I understand should update the dependencies used by that
> build.
>

That is correct - we also have a failure on FreeBSD though so i've
triggered the corresponding FreeBSD job too.


> Let's see if this build succeeds (the previous one failed on some
> python/sip)
>
> Cheers,
>   Albert
>

Cheers,
Ben


> >
> >  >
> >
> /usr/home/jenkins/install-prefix/include/KF5/KCoreAddons/kpluginmetadata.h:443:54:
> >  > note: passing argument to parameter 'defaultValue' here
> >  > [2021-10-18T11:17:43.890Z] QString value(const QString , const
> >  > QString  = QString()) const;
> >
> > It appears like the compiler chooses wrong overload.
> >
>
>
>
>
>


Re: KDE Frameworks 5.87.0

2021-10-14 Thread Ben Cooksley
On Thu, Oct 14, 2021 at 9:27 AM Luca Beltrame  wrote:

> In data mercoledì 13 ottobre 2021 19:24:48 CEST, Ben Cooksley ha scritto:
>
> > It would appear that the necessary dependencies between targets aren't
> set
> > up properly?
>
> They aren't - one of the reasons I had a hard time packaging the bindings
> for
> openSUSE.
>

Sad to hear this - that might explain why I vaguely recall us adding the
necessary SIP packages at one point to build these bindings.
The unreliability introduced by their presence would have likely led me to
remove them as we would need the builds to be reliable.

Any interest from those who use these bindings to fix the CMake
dependencies here?
Otherwise i'll need to remove the SIP packages as the need for reliable
builds outweighs the value provided from having them present (i'd suggest
we disable them from building by default even)

Cheers,
Ben


> --
> Luca Beltrame - KDE Forums team
> GPG key ID: A29D259B


Re: KDE Frameworks 5.87.0

2021-10-13 Thread Ben Cooksley
On Tue, Oct 5, 2021 at 8:46 PM Ben Cooksley  wrote:

> On Mon, Oct 4, 2021 at 9:28 PM Antonio Rojas  wrote:
>
>> El lunes, 4 de octubre de 2021 10:10:02 (CEST), Ben Cooksley escribió:
>>
>> > If someone could confirm the SIP version we need that would be awesome.
>> >
>> > Cheers,
>> > Ben
>> >
>>
>> Yes, KF5 doesn't support anything newer than sip 4.
>>
>
> Thanks for confirming - i've now introduced this to our Linux image.
>
> https://invent.kde.org/sysadmin/ci-tooling/commit/7c9b1c2aa28536a9f99c165abbfccc347257f468
>

I now remember why we removed those packages - because it made our builds
unreliable.
Please see https://invent.kde.org/sysadmin/ci-management/-/jobs/141503

It would appear that the necessary dependencies between targets aren't set
up properly?


> Cheers,
> Ben
>

Thanks,
Ben


Re: Rollout of Gitlab CI

2021-10-03 Thread Ben Cooksley
On Wed, Sep 29, 2021 at 10:27 PM Ben Cooksley  wrote:

> Hi all,
>
> As those of you who watch and work on Frameworks repositories will be
> aware, we've just rolled out the first set of native Gitlab CI builds.
>
> These builds are at this time Linux only, but do include support for both
> regular branch builds as well as for Merge Requests. It is anticipated that
> Windows, FreeBSD and Android builds will follow in the near future - there
> are a few extra things we need to get completed first before they can be
> rolled out. As part of carrying out the build the scripts will also gather
> Code Coverage and Code Quality information using gcovr and cppcheck
> respectively, and this will be made available to you within the Gitlab
> interface.
>
> With regards to availability to projects outside Frameworks, projects that
> depend only on Frameworks (and no other KDE project) may enable CI for
> their project by adding the necessary .kde-ci.yml and .gitlab-ci.yml files
> to their repositories. I anticipate that once the necessary changes have
> been made to the "seed" jobs used to provision the initial artifacts it
> should be possible for all projects to rollout support (although i'd like
> to add ccache support to the system to ensure larger project builds
> complete promptly first).
>
> If anyone would like to help out with getting the seed jobs ready please
> let me know as this is definitely something that the community can assist
> with (and will also help with easing the rollout of Windows, FreeBSD and
> Android builds).
>
> For those projects that do go ahead with enabling Gitlab CI support please
> ensure you add the necessary .kde-ci.yml file beforehand, specifying in it
> the necessary Dependencies, as well as the necessary Options your project
> needs to customise for their build. A list of all the possible values in a
> .kde-ci.yml file can be found at
> https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/config-template.yml#L10
>
> For those projects that are not yet able to enable the system, it is
> strongly encouraged that you go ahead and get the .kde-ci.yml files ready
> in your repository now - especially if other projects depend on your
> project. This will allow the rollout of the CI system to those projects to
> proceed much more smoothly (otherwise we will need to get them added to
> those projects which have other projects depending on them).
>

I have now prepared a list of all the projects that fall under this
umbrella of being the dependency of another project -
https://invent.kde.org/sysadmin/ci-management/-/merge_requests/3
It would be appreciated if people could please review this for any issues.

Please note that it is intentional Frameworks is missing as that will be
built by the 'Frameworks' seed.
With regards to the PIM seed, please note that Zanshin, Kube and SInk have
all been included in this, to allow for the Independent Releases seed to
minimize the number of PIM projects it builds.

If your project is included in any of those and you have not added a
.kde-ci.yml file, it is imperative you do it *now*. If you do not then you
will block the rollout of Gitlab CI for non-Frameworks projects.

At some point in the near future we will also transition all of the
references to 'master' in the seed files to '@latest' to allow for those
branches to be resolved and taken from
https://invent.kde.org/sysadmin/repo-metadata/-/blob/master/branch-rules.yml


> Please note that this new infrastructure replaces all existing Gitlab CI
> jobs we provided in the initial testing phase, and those should be removed
> (much like how I did for Frameworks) when switching to the new setup.
>
> Thanks,
> Ben
>

Thanks,
Ben


Re: Rollout of Gitlab CI

2021-10-03 Thread Ben Cooksley
On Wed, Sep 29, 2021 at 10:27 PM Ben Cooksley  wrote:

> Hi all,
>
> As those of you who watch and work on Frameworks repositories will be
> aware, we've just rolled out the first set of native Gitlab CI builds.
>
> These builds are at this time Linux only, but do include support for both
> regular branch builds as well as for Merge Requests. It is anticipated that
> Windows, FreeBSD and Android builds will follow in the near future - there
> are a few extra things we need to get completed first before they can be
> rolled out. As part of carrying out the build the scripts will also gather
> Code Coverage and Code Quality information using gcovr and cppcheck
> respectively, and this will be made available to you within the Gitlab
> interface.
>
> With regards to availability to projects outside Frameworks, projects that
> depend only on Frameworks (and no other KDE project) may enable CI for
> their project by adding the necessary .kde-ci.yml and .gitlab-ci.yml files
> to their repositories. I anticipate that once the necessary changes have
> been made to the "seed" jobs used to provision the initial artifacts it
> should be possible for all projects to rollout support (although i'd like
> to add ccache support to the system to ensure larger project builds
> complete promptly first).
>
> If anyone would like to help out with getting the seed jobs ready please
> let me know as this is definitely something that the community can assist
> with (and will also help with easing the rollout of Windows, FreeBSD and
> Android builds).
>

As an update to this, work on FreeBSD and Android support is progressing
well and should be made available within the next week all going well.


> For those projects that do go ahead with enabling Gitlab CI support please
> ensure you add the necessary .kde-ci.yml file beforehand, specifying in it
> the necessary Dependencies, as well as the necessary Options your project
> needs to customise for their build. A list of all the possible values in a
> .kde-ci.yml file can be found at
> https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/config-template.yml#L10
>
> For those projects that are not yet able to enable the system, it is
> strongly encouraged that you go ahead and get the .kde-ci.yml files ready
> in your repository now - especially if other projects depend on your
> project. This will allow the rollout of the CI system to those projects to
> proceed much more smoothly (otherwise we will need to get them added to
> those projects which have other projects depending on them).
>
> Please note that this new infrastructure replaces all existing Gitlab CI
> jobs we provided in the initial testing phase, and those should be removed
> (much like how I did for Frameworks) when switching to the new setup.
>
> Thanks,
> Ben
>

Cheers,
Ben


Re: Rollout of Gitlab CI

2021-09-30 Thread Ben Cooksley
On Thu, Sep 30, 2021 at 8:04 PM Laurent Montel  wrote:

> On jeudi 30 septembre 2021 08:40:04 CEST Ben Cooksley wrote:
> > On Thu, Sep 30, 2021 at 7:10 PM Laurent Montel  wrote:
> > > Hi,
> >
> > Hi Laurent,
> >
> > I added in .kde-ci.yml
> >
> > > 'kde/pim/akonadi-contacts': '@latest'
> > > 'frameworks/kcontacts': '@stable'
> > >
> > > for kgpg and it doesn’t find akonadi-contacts. Do you have an idea ?
> > > What I missed ?
> > > “https://invent.kde.org/utilities/kgpg/-/jobs/134772/raw”
> >
> > As mentioned in my email, this is only available for projects that depend
> > only on KDE Frameworks at the moment.
> Ah ? I didn’t see this line :( too bad.
>
> >
> > KGpg does not meet this criteria as it depends on akonadi-contacts -
> which
> > is in KDE PIM / Release Service, not Frameworks.
>
> For sure.
> I will comment .kde-ci.yml for the moment.
>

The .kde-ci.yml file should remain in place unchanged.

The .gitlab-ci.yml file certainly can be commented out yes.

Cheers,
Ben


> >
> > > Regards
> >
> > Cheers,
> > Ben
> >
> > > On mercredi 29 septembre 2021 11:27:08 CEST Ben Cooksley wrote:
> > > > Hi all,
> > >
> > > --
> > > Laurent Montel | laurent.mon...@kdab.com | KDE/Qt Senior Software
> > > Engineer
> > > KDAB (France) S.A.S., a KDAB Group company
> > > Tel: France +33 (0)4 90 84 08 53, http://www.kdab.fr
> > > KDAB - The Qt, C++ and OpenGL Experts
>
>
> --
> Laurent Montel | laurent.mon...@kdab.com | KDE/Qt Senior Software
> Engineer
> KDAB (France) S.A.S., a KDAB Group company
> Tel: France +33 (0)4 90 84 08 53, http://www.kdab.fr
> KDAB - The Qt, C++ and OpenGL Experts
>
>
>


Re: Rollout of Gitlab CI

2021-09-30 Thread Ben Cooksley
On Thu, Sep 30, 2021 at 7:10 PM Laurent Montel  wrote:

> Hi,
>

Hi Laurent,

I added in .kde-ci.yml
> 'kde/pim/akonadi-contacts': '@latest'
> 'frameworks/kcontacts': '@stable'
>
> for kgpg and it doesn’t find akonadi-contacts. Do you have an idea ?
> What I missed ?
> “https://invent.kde.org/utilities/kgpg/-/jobs/134772/raw”
>

As mentioned in my email, this is only available for projects that depend
only on KDE Frameworks at the moment.

KGpg does not meet this criteria as it depends on akonadi-contacts - which
is in KDE PIM / Release Service, not Frameworks.


> Regards
>

Cheers,
Ben


> On mercredi 29 septembre 2021 11:27:08 CEST Ben Cooksley wrote:
> > Hi all,
>
> --
> Laurent Montel | laurent.mon...@kdab.com | KDE/Qt Senior Software
> Engineer
> KDAB (France) S.A.S., a KDAB Group company
> Tel: France +33 (0)4 90 84 08 53, http://www.kdab.fr
> KDAB - The Qt, C++ and OpenGL Experts
>
>
>


Re: Rollout of Gitlab CI

2021-09-29 Thread Ben Cooksley
On Thu, Sep 30, 2021 at 3:41 AM Johnny Jazeix  wrote:

>
>
> Le mer. 29 sept. 2021 à 11:27, Ben Cooksley  a écrit :
>
>> Hi all,
>>
>> As those of you who watch and work on Frameworks repositories will be
>> aware, we've just rolled out the first set of native Gitlab CI builds.
>>
>> These builds are at this time Linux only, but do include support for both
>> regular branch builds as well as for Merge Requests. It is anticipated that
>> Windows, FreeBSD and Android builds will follow in the near future - there
>> are a few extra things we need to get completed first before they can be
>> rolled out. As part of carrying out the build the scripts will also gather
>> Code Coverage and Code Quality information using gcovr and cppcheck
>> respectively, and this will be made available to you within the Gitlab
>> interface.
>>
>> With regards to availability to projects outside Frameworks, projects
>> that depend only on Frameworks (and no other KDE project) may enable CI for
>> their project by adding the necessary .kde-ci.yml and .gitlab-ci.yml files
>> to their repositories. I anticipate that once the necessary changes have
>> been made to the "seed" jobs used to provision the initial artifacts it
>> should be possible for all projects to rollout support (although i'd like
>> to add ccache support to the system to ensure larger project builds
>> complete promptly first).
>>
>> If anyone would like to help out with getting the seed jobs ready please
>> let me know as this is definitely something that the community can assist
>> with (and will also help with easing the rollout of Windows, FreeBSD and
>> Android builds).
>>
>> For those projects that do go ahead with enabling Gitlab CI support
>> please ensure you add the necessary .kde-ci.yml file beforehand, specifying
>> in it the necessary Dependencies, as well as the necessary Options your
>> project needs to customise for their build. A list of all the possible
>> values in a .kde-ci.yml file can be found at
>> https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/config-template.yml#L10
>>
>> For those projects that are not yet able to enable the system, it is
>> strongly encouraged that you go ahead and get the .kde-ci.yml files ready
>> in your repository now - especially if other projects depend on your
>> project. This will allow the rollout of the CI system to those projects to
>> proceed much more smoothly (otherwise we will need to get them added to
>> those projects which have other projects depending on them).
>>
>> Please note that this new infrastructure replaces all existing Gitlab CI
>> jobs we provided in the initial testing phase, and those should be removed
>> (much like how I did for Frameworks) when switching to the new setup.
>>
>> Thanks,
>> Ben
>>
>
> Hi,
>
> is there a template for the .gitlab-ci.yml or should we use the same as
> https://invent.kde.org/frameworks/extra-cmake-modules/-/commit/63bec5521d245e3a1ea50109b86a2b716966938e
> for example?
>

You can use the same as that file.

We're not providing a .gitlab-ci.yml template as there will be Windows,
FreeBSD and Android builds as well (and potentially in the future macOS as
well) which are options that won't necessarily apply to all projects.


> Cheers,
>
> Johnny
>

Cheers,
Ben


Rollout of Gitlab CI

2021-09-29 Thread Ben Cooksley
Hi all,

As those of you who watch and work on Frameworks repositories will be
aware, we've just rolled out the first set of native Gitlab CI builds.

These builds are at this time Linux only, but do include support for both
regular branch builds as well as for Merge Requests. It is anticipated that
Windows, FreeBSD and Android builds will follow in the near future - there
are a few extra things we need to get completed first before they can be
rolled out. As part of carrying out the build the scripts will also gather
Code Coverage and Code Quality information using gcovr and cppcheck
respectively, and this will be made available to you within the Gitlab
interface.

With regards to availability to projects outside Frameworks, projects that
depend only on Frameworks (and no other KDE project) may enable CI for
their project by adding the necessary .kde-ci.yml and .gitlab-ci.yml files
to their repositories. I anticipate that once the necessary changes have
been made to the "seed" jobs used to provision the initial artifacts it
should be possible for all projects to rollout support (although i'd like
to add ccache support to the system to ensure larger project builds
complete promptly first).

If anyone would like to help out with getting the seed jobs ready please
let me know as this is definitely something that the community can assist
with (and will also help with easing the rollout of Windows, FreeBSD and
Android builds).

For those projects that do go ahead with enabling Gitlab CI support please
ensure you add the necessary .kde-ci.yml file beforehand, specifying in it
the necessary Dependencies, as well as the necessary Options your project
needs to customise for their build. A list of all the possible values in a
.kde-ci.yml file can be found at
https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/config-template.yml#L10

For those projects that are not yet able to enable the system, it is
strongly encouraged that you go ahead and get the .kde-ci.yml files ready
in your repository now - especially if other projects depend on your
project. This will allow the rollout of the CI system to those projects to
proceed much more smoothly (otherwise we will need to get them added to
those projects which have other projects depending on them).

Please note that this new infrastructure replaces all existing Gitlab CI
jobs we provided in the initial testing phase, and those should be removed
(much like how I did for Frameworks) when switching to the new setup.

Thanks,
Ben


Re: OCS Providers File - High Numbers of Requests

2021-09-27 Thread Ben Cooksley
On Mon, Sep 27, 2021 at 7:04 AM Aleix Pol  wrote:

> On Fri, Sep 24, 2021 at 8:26 PM Ben Cooksley  wrote:
> >
> > On Sat, Sep 25, 2021 at 12:34 AM Aleix Pol  wrote:
> >>
> >> On Fri, Sep 24, 2021 at 1:32 AM Nicolás Alvarez
> >>  wrote:
> >> >
> >> > El jue, 23 de sep. de 2021 a la(s) 19:31, Aleix Pol (aleix...@kde.org)
> escribió:
> >> > >
> >> > > On Thu, Sep 23, 2021 at 10:12 PM Nicolás Alvarez
> >> > >  wrote:
> >> > > >
> >> > > > El jue, 23 de sep. de 2021 a la(s) 08:55, Aleix Pol (
> aleix...@kde.org) escribió:
> >> > > > >
> >> > > > > On Thu, Sep 23, 2021 at 11:52 AM Ben Cooksley <
> bcooks...@kde.org> wrote:
> >> > > > > >
> >> > > > > > Hi all,
> >> > > > > >
> >> > > > > > It has recently come to our attention that the number of
> queries being handled for the endpoint
> https://autoconfig.kde.org/ocs/providers.xml on a day to day basis has
> gotten to the point where it is causing issues with server responsiveness
> to other traffic. This is perhaps best summarised by the following:
> >> > > > > >
> >> > > > > > root@nicoda /var/log/apache2 # ls -lah ...
> >> > > > > > -rw-r- 1 root adm 458M Sep 23 06:25
> autoconfig.kde.org.log.1
> >> > > > > > -rw-r- 1 root adm 381M Sep 23 06:25
> networkcheck.kde.org.log.1
> >> > > > > > -rw-r- 1 root adm 143M Sep 23 06:25 www.kde.org.log.1
> >> > > > > >
> >> > > > > > root@nicoda /var/log/apache2 # cat autoconfig.kde.org.log.1
> | wc -l
> >> > > > > > 4,222,343
> >> > > > > >
> >> > > > > > Based on those numbers we're looking at 48-49 requests per
> second (on average - peaks are much higher by many magnitudes), which seems
> extremely excessive given that this file is only supposed to be retrieved
> by KDE software when GHNS functionality is triggered. That is supported by
> the substantial size difference it has over networkcheck.kde.org - which
> is used by plasma-nm and NetworkManager (on Neon) to check for whether they
> have a working internet connection - which i'd expect to be the site
> receiving the most traffic.
> >> > > > > >
> >> > > > > > As such, I therefore suspect we have bug(s) in software that
> makes use of GHNS functionality.
> >> > > > > >
> >> > > > > > It would therefore be appreciated if we could please review
> the software in question to determine whether it is operating correctly.
> Given that it usually runs in the background on user systems, i'd
> especially appreciate it if a detailed review could be conducted on
> Discover and other software that conducts package management operations or
> assists in managing updates.
> >> > > > > >
> >> > > > > > Unfortunately all these applications submit a fairly useless
> user agent (Mozilla/5.0) so it is impossible for Sysadmin to ascertain any
> further information. If we could get information on the software that is
> originating the request added to the user agent to assist in investigating
> these issues in the future that would be extremely helpful.
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Ben
> >> > > > >
> >> > > > > That's correct. Discover fetches them at startup. It's
> necessary to be
> >> > > > > able to check if there are updates on KNS-provided resources.
> >> > > > >
> >> > > > > Incidentally,  I was looking into this yesterday incidentally.
> We
> >> > > > > could see if caching is broken somehow. A request will still be
> needed
> >> > > > > though to check if the cache is out of date.
> >> > > >
> >> > > > Caching seems to be working, since the vast majority of the
> requests
> >> > > > are returning 304 Not Modified.
> >> > > >
> >> > > > However in *many* cases I see a single IP making multiple
> requests in
> >> > > > the same second, and doing it again the next minute. Here's one IP
> >> > > > address picked randomly:
> >> > > >
> >> > > > [22/Sep/2021:06:25:41 +] "GET /ocs/providers.xml HTTP/1.1&quo

Re: OCS Providers File - High Numbers of Requests

2021-09-24 Thread Ben Cooksley
On Sat, Sep 25, 2021 at 12:34 AM Aleix Pol  wrote:

> On Fri, Sep 24, 2021 at 1:32 AM Nicolás Alvarez
>  wrote:
> >
> > El jue, 23 de sep. de 2021 a la(s) 19:31, Aleix Pol (aleix...@kde.org)
> escribió:
> > >
> > > On Thu, Sep 23, 2021 at 10:12 PM Nicolás Alvarez
> > >  wrote:
> > > >
> > > > El jue, 23 de sep. de 2021 a la(s) 08:55, Aleix Pol (
> aleix...@kde.org) escribió:
> > > > >
> > > > > On Thu, Sep 23, 2021 at 11:52 AM Ben Cooksley 
> wrote:
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > It has recently come to our attention that the number of queries
> being handled for the endpoint
> https://autoconfig.kde.org/ocs/providers.xml on a day to day basis has
> gotten to the point where it is causing issues with server responsiveness
> to other traffic. This is perhaps best summarised by the following:
> > > > > >
> > > > > > root@nicoda /var/log/apache2 # ls -lah ...
> > > > > > -rw-r- 1 root adm 458M Sep 23 06:25 autoconfig.kde.org.log.1
> > > > > > -rw-r- 1 root adm 381M Sep 23 06:25
> networkcheck.kde.org.log.1
> > > > > > -rw-r- 1 root adm 143M Sep 23 06:25 www.kde.org.log.1
> > > > > >
> > > > > > root@nicoda /var/log/apache2 # cat autoconfig.kde.org.log.1 |
> wc -l
> > > > > > 4,222,343
> > > > > >
> > > > > > Based on those numbers we're looking at 48-49 requests per
> second (on average - peaks are much higher by many magnitudes), which seems
> extremely excessive given that this file is only supposed to be retrieved
> by KDE software when GHNS functionality is triggered. That is supported by
> the substantial size difference it has over networkcheck.kde.org - which
> is used by plasma-nm and NetworkManager (on Neon) to check for whether they
> have a working internet connection - which i'd expect to be the site
> receiving the most traffic.
> > > > > >
> > > > > > As such, I therefore suspect we have bug(s) in software that
> makes use of GHNS functionality.
> > > > > >
> > > > > > It would therefore be appreciated if we could please review the
> software in question to determine whether it is operating correctly. Given
> that it usually runs in the background on user systems, i'd especially
> appreciate it if a detailed review could be conducted on Discover and other
> software that conducts package management operations or assists in managing
> updates.
> > > > > >
> > > > > > Unfortunately all these applications submit a fairly useless
> user agent (Mozilla/5.0) so it is impossible for Sysadmin to ascertain any
> further information. If we could get information on the software that is
> originating the request added to the user agent to assist in investigating
> these issues in the future that would be extremely helpful.
> > > > > >
> > > > > > Thanks,
> > > > > > Ben
> > > > >
> > > > > That's correct. Discover fetches them at startup. It's necessary
> to be
> > > > > able to check if there are updates on KNS-provided resources.
> > > > >
> > > > > Incidentally,  I was looking into this yesterday incidentally. We
> > > > > could see if caching is broken somehow. A request will still be
> needed
> > > > > though to check if the cache is out of date.
> > > >
> > > > Caching seems to be working, since the vast majority of the requests
> > > > are returning 304 Not Modified.
> > > >
> > > > However in *many* cases I see a single IP making multiple requests in
> > > > the same second, and doing it again the next minute. Here's one IP
> > > > address picked randomly:
> > > >
> > > > [22/Sep/2021:06:25:41 +] "GET /ocs/providers.xml HTTP/1.1" 304
> > > > [22/Sep/2021:06:25:41 +] "GET /ocs/providers.xml HTTP/1.1" 304
> > > > [22/Sep/2021:06:25:41 +] "GET /ocs/providers.xml HTTP/1.1" 304
> > > > [22/Sep/2021:06:25:41 +] "GET /ocs/providers.xml HTTP/1.1" 304
> > > > [22/Sep/2021:06:27:57 +] "GET /ocs/providers.xml HTTP/1.1" 304
> > > > [22/Sep/2021:06:27:58 +] "GET /ocs/providers.xml HTTP/1.1" 304
> > > > [22/Sep/2021:06:27:58 +] "GET /ocs/providers.xml HTTP/1.1" 304
> > > > [22/Sep/2021:06:28:32 +] 

Re: OCS Providers File - High Numbers of Requests

2021-09-24 Thread Ben Cooksley
On Fri, Sep 24, 2021 at 6:34 PM Shantanu Tushar Jha 
wrote:

> Hi,
>

Hi Shantanu,


> Just an idea - do we have enough logs on the server to see the requests
> history by date? If so, one can identify if there was a spike of the
> requests after a particular set of dates (which can in turn give us a hint
> about which release might contain a bug that's making more calls than
> expected). Of course this is not useful if it's apparent that the request
> count increased gradually instead of a spike.
>

I'm afraid not - we only keep raw logs for 2 weeks after which they are
discarded in accordance with our privacy policy.

This issue is one that has existed for sometime, although it is hard to
tell when the situation started to deteriorate significantly and what the
cause of that was.
(Whether it was due to more user systems, or due to changing software
behaviour/bugs)


> Shantanu
>

Cheers,
Ben


> On Thu, 23 Sept 2021 at 22:13, Nicolás Alvarez 
> wrote:
>
>> El jue, 23 de sep. de 2021 a la(s) 08:55, Aleix Pol (aleix...@kde.org)
>> escribió:
>> >
>> > On Thu, Sep 23, 2021 at 11:52 AM Ben Cooksley 
>> wrote:
>> > >
>> > > Hi all,
>> > >
>> > > It has recently come to our attention that the number of queries
>> being handled for the endpoint
>> https://autoconfig.kde.org/ocs/providers.xml on a day to day basis has
>> gotten to the point where it is causing issues with server responsiveness
>> to other traffic. This is perhaps best summarised by the following:
>> > >
>> > > root@nicoda /var/log/apache2 # ls -lah ...
>> > > -rw-r- 1 root adm 458M Sep 23 06:25 autoconfig.kde.org.log.1
>> > > -rw-r- 1 root adm 381M Sep 23 06:25 networkcheck.kde.org.log.1
>> > > -rw-r- 1 root adm 143M Sep 23 06:25 www.kde.org.log.1
>> > >
>> > > root@nicoda /var/log/apache2 # cat autoconfig.kde.org.log.1 | wc -l
>> > > 4,222,343
>> > >
>> > > Based on those numbers we're looking at 48-49 requests per second (on
>> average - peaks are much higher by many magnitudes), which seems extremely
>> excessive given that this file is only supposed to be retrieved by KDE
>> software when GHNS functionality is triggered. That is supported by the
>> substantial size difference it has over networkcheck.kde.org - which is
>> used by plasma-nm and NetworkManager (on Neon) to check for whether they
>> have a working internet connection - which i'd expect to be the site
>> receiving the most traffic.
>> > >
>> > > As such, I therefore suspect we have bug(s) in software that makes
>> use of GHNS functionality.
>> > >
>> > > It would therefore be appreciated if we could please review the
>> software in question to determine whether it is operating correctly. Given
>> that it usually runs in the background on user systems, i'd especially
>> appreciate it if a detailed review could be conducted on Discover and other
>> software that conducts package management operations or assists in managing
>> updates.
>> > >
>> > > Unfortunately all these applications submit a fairly useless user
>> agent (Mozilla/5.0) so it is impossible for Sysadmin to ascertain any
>> further information. If we could get information on the software that is
>> originating the request added to the user agent to assist in investigating
>> these issues in the future that would be extremely helpful.
>> > >
>> > > Thanks,
>> > > Ben
>> >
>> > That's correct. Discover fetches them at startup. It's necessary to be
>> > able to check if there are updates on KNS-provided resources.
>> >
>> > Incidentally,  I was looking into this yesterday incidentally. We
>> > could see if caching is broken somehow. A request will still be needed
>> > though to check if the cache is out of date.
>>
>> Caching seems to be working, since the vast majority of the requests
>> are returning 304 Not Modified.
>>
>> However in *many* cases I see a single IP making multiple requests in
>> the same second, and doing it again the next minute. Here's one IP
>> address picked randomly:
>>
>> [22/Sep/2021:06:25:41 +] "GET /ocs/providers.xml HTTP/1.1" 304
>> [22/Sep/2021:06:25:41 +] "GET /ocs/providers.xml HTTP/1.1" 304
>> [22/Sep/2021:06:25:41 +] "GET /ocs/providers.xml HTTP/1.1" 304
>> [22/Sep/2021:06:25:41 +] "GET /ocs/providers.xml HTTP/1.1" 304
>> [22/Sep/2021:06:27:57 +] "GET /ocs/providers.xml HTTP/1.1" 304

OCS Providers File - High Numbers of Requests

2021-09-23 Thread Ben Cooksley
Hi all,

It has recently come to our attention that the number of queries being
handled for the endpoint https://autoconfig.kde.org/ocs/providers.xml on a
day to day basis has gotten to the point where it is causing issues with
server responsiveness to other traffic. This is perhaps best summarised by
the following:

root@nicoda /var/log/apache2 # ls -lah ...
-rw-r- 1 root adm 458M Sep 23 06:25 autoconfig.kde.org.log.1
-rw-r- 1 root adm 381M Sep 23 06:25 networkcheck.kde.org.log.1
-rw-r- 1 root adm 143M Sep 23 06:25 www.kde.org.log.1

root@nicoda /var/log/apache2 # cat autoconfig.kde.org.log.1 | wc -l
4,222,343

Based on those numbers we're looking at 48-49 requests per second (on
average - peaks are much higher by many magnitudes), which seems extremely
excessive given that this file is only supposed to be retrieved by KDE
software when GHNS functionality is triggered. That is supported by the
substantial size difference it has over networkcheck.kde.org - which is
used by plasma-nm and NetworkManager (on Neon) to check for whether they
have a working internet connection - which i'd expect to be the site
receiving the most traffic.

As such, I therefore suspect we have bug(s) in software that makes use of
GHNS functionality.

It would therefore be appreciated if we could please review the software in
question to determine whether it is operating correctly. Given that it
usually runs in the background on user systems, i'd especially appreciate
it if a detailed review could be conducted on Discover and other software
that conducts package management operations or assists in managing updates.

Unfortunately all these applications submit a fairly useless user agent
(Mozilla/5.0) so it is impossible for Sysadmin to ascertain any further
information. If we could get information on the software that is
originating the request added to the user agent to assist in investigating
these issues in the future that would be extremely helpful.

Thanks,
Ben


Re: Re : KApiDox move from dedicated server to Jenkins

2021-09-11 Thread Ben Cooksley
On Sun, Sep 12, 2021 at 7:07 AM Frederik Schwarzer 
wrote:

> Hi,
>

Hi all,


> On 9/10/21 21:17, Ben Cooksley wrote:
> > On Sat, Sep 11, 2021 at 5:40 AM Carl Schwan  wrote:
>
> >> We also are losing the krita/kmymoney/other app private api generation,
> >> but that maybe can be generated in another ci pipeline later. Not sure
> how
> >> much thses apps' developers are using it.
> >>
> >
> > This can likely be rectified if needed by including them within
> >
> https://invent.kde.org/sysadmin/binary-factory-tooling/-/blob/master/apidocs/repos-to-process
>
> So, we would need to decide whether we want to have applications that
> only have private APIs documented?
>

That is correct.


> Where do we draw the line currently?
>

To date we've not really drawn a line anywhere, but in general I think it
would be best for us to only include API Documentation for projects that
people can actually use externally of the project itself.
(so for Applications, they would need to provide a plugin interface, or a
library that lets people reuse components of it)

Otherwise people might be confused as to what they're looking at.


> In general I would say, just generate everything that has a Doxyfile ...
> but there might be good reasons against that regarding processing power?
>

In the past that has meant that build runs of API Documentation took
several hours.

I'd also recommend taking a look at it from a maintainability point of view
as this is something that will continue to run for some time, so we need to
ensure that if projects do have special requirements that they do not
create issues from that perspective.


> Cheers,
> Frederik
>

Cheers,
Ben


Re: KApiDox move from dedicated server to Jenkins

2021-09-10 Thread Ben Cooksley
On Sat, Sep 11, 2021 at 5:24 AM Frederik Schwarzer 
wrote:

> Hi,
>

Hi all,


> we have been working on getting KApiDox to run on Jenkins. This work has
> been taken way longer than I expected but has now reached a state close
> to finished. :)
>
> So I would like to invite you to check https://api-dev.kde.org/ for any
> show stoppers.
>
> Known issues (not show stoppers):
>
> - For now the Maintainers field defaults to "KDE Developers" for
> potential GDPR violation reasons, which needs to be figured out later.
>
> - Some modules contain formatting issues regarding markdown code blocks.
> These are also there in the current system and need to be checked at
> some point.
>
> - The "Older versions" links are broken. Since those docs are not
> generated anymore, we need to figure out a way to have them available
> statically.
>

I've now completed transferring a copy of all of the Older API
Documentation to https://api-dev.kde.org/legacy/
This is aliased in from elsewhere on the server, so it's contents won't be
disturbed by the automatic generation process we have for new KApiDox based
documentation.

I also included the legacy CMake documentation when setting that up.

If we could please update the front page to point there that should resolve
this point.


> If we do not see any bigger issues, I would like to go live with the new
> system in a week or two.
>
> Thanks for your help.
>
> CHeers,
> Frederik
>

Cheers,
Ben


Re: KApiDox move from dedicated server to Jenkins

2021-09-10 Thread Ben Cooksley
On Sat, Sep 11, 2021 at 7:43 AM Friedrich W. H. Kossebau 
wrote:

> Am Freitag, 10. September 2021, 19:23:47 CEST schrieb Frederik Schwarzer:
> > Hi,
> >
> > we have been working on getting KApiDox to run on Jenkins. This work has
> > been taken way longer than I expected but has now reached a state close
> > to finished. :)
>
> Congrats on moving forwards, thanks for the work.
>
> Linking to external docs (mainly the Qt ones) seems missing. For that
> respective doxygen tags files need to be added. Qt generates such files
> during
> the build, so for the current api.kde.org people copied those tags files
> from
> their distribution's package.
>
> Here I did for updating to Qt 5.15:
> https://invent.kde.org/websites/quality-kde-org/-/commit/
> 8a18c9033751cb41548361aea5c8ba2de3499dd7
>
> with the files found in /usr/share/doc/packages/qt5/*/*.tags, provided by
> the
> package libqt5-qtdoc-devel on openSUSE TW.
>

> To see how kapidox is instructed about those tags files, as that seems
> missing
> from its own docs, see the usage example in the current api.kde.org
> scripts:
> https://invent.kde.org/websites/quality-kde-org/-/blob/master/apidox/src/
> kapidoxgendox.sh#L33
> 
>

Thanks for the details on this.

I've now copied over those same tags and tweaked the pipeline we run on
Jenkins which should solve that.


> CHeers
> Friedrich
>
>
Cheers,
Ben


Re: Re : KApiDox move from dedicated server to Jenkins

2021-09-10 Thread Ben Cooksley
On Sat, Sep 11, 2021 at 5:40 AM Carl Schwan  wrote:

> Le vendredi 10 septembre 2021 à 7:23 PM, Frederik Schwarzer <
> schwar...@kde.org> a écrit :
>
> > Hi,
>
> Hi :D
>

Hey Carl,


> > we have been working on getting KApiDox to run on Jenkins. This work has
> > been taken way longer than I expected but has now reached a state close
> > to finished. :)
> >
> > So I would like to invite you to check https://api-dev.kde.org/ for any
> > show stoppers.
>
> Great work :D
>
> I see on an issue that I would qualify as blocking and it's the lack of the
> ECM generated doc: https://api-dev.kde.org/ecm. We are also losing the
> kube/sink doc (located at api.kde.org/doc/sink) but it's also available
> in readthedocs and ihmo it should be in Doxygen format.
>

My understanding is that ECM uses something other than Doxygen for
generating it's API Documentation, which is why it is absent.

Would anyone on the list know if it is possible to get this process to use
Doxygen, or if we need to put in place a mechanism explicitly for ECM?


> We also are losing the krita/kmymoney/other app private api generation,
> but that maybe can be generated in another ci pipeline later. Not sure how
> much thses apps' developers are using it.
>

This can likely be rectified if needed by including them within
https://invent.kde.org/sysadmin/binary-factory-tooling/-/blob/master/apidocs/repos-to-process


> On the upside, I see that mauikit doc is finally correctly generated using
> qdoc. Yeah \o/
>

Yes, the new setup allows us to provide this now - and being Docker image
based for the generation phase should make future updates easier as well as
needed.


> Cheers,
> Carl
>

Thanks,
Ben

>
> > Known issues (not show stoppers):
> >
> > -   For now the Maintainers field defaults to "KDE Developers" for
> > potential GDPR violation reasons, which needs to be figured out
> later.
> > -   Some modules contain formatting issues regarding markdown code
> blocks.
> > These are also there in the current system and need to be checked at
> > some point.
> > -   The "Older versions" links are broken. Since those docs are not
> > generated anymore, we need to figure out a way to have them available
> > statically.
> >
> > If we do not see any bigger issues, I would like to go live with the
> new
> > system in a week or two.
> >
> > Thanks for your help.
> >
> > CHeers,
> > Frederik
>


Re: KApiDox move from dedicated server to Jenkins

2021-09-10 Thread Ben Cooksley
On Sat, Sep 11, 2021 at 5:24 AM Frederik Schwarzer 
wrote:

> Hi,
>
> we have been working on getting KApiDox to run on Jenkins. This work has
> been taken way longer than I expected but has now reached a state close
> to finished. :)
>
> So I would like to invite you to check https://api-dev.kde.org/ for any
> show stoppers.
>
> Known issues (not show stoppers):
>
> - For now the Maintainers field defaults to "KDE Developers" for
> potential GDPR violation reasons, which needs to be figured out later.
>
> - Some modules contain formatting issues regarding markdown code blocks.
> These are also there in the current system and need to be checked at
> some point.
>
> - The "Older versions" links are broken. Since those docs are not
> generated anymore, we need to figure out a way to have them available
> statically.
>

I already have a plan for this - don't worry about that one :)


> If we do not see any bigger issues, I would like to go live with the new
> system in a week or two.
>
> Thanks for your help.
>
> CHeers,
> Frederik
>

Cheers,
Ben


Re: Gitlab CI - Inbound

2021-09-07 Thread Ben Cooksley
On Tue, Sep 7, 2021 at 10:28 PM Vlad Zahorodnii 
wrote:

> On 9/7/21 1:21 PM, Ben Cooksley wrote:
> > On Tue, Sep 7, 2021 at 10:14 PM Vlad Zahorodnii  > <mailto:vlad.zahorod...@kde.org>> wrote:
> >
> > On 9/7/21 12:22 PM, Ben Cooksley wrote:
> >  > On Tue, Sep 7, 2021 at 8:48 PM Vlad Zahorodnii
> > mailto:vlad.zahorod...@kde.org>
> >  > <mailto:vlad.zahorod...@kde.org
> > <mailto:vlad.zahorod...@kde.org>>> wrote:
> >  >
> >  > On 9/5/21 3:18 PM, David Faure wrote:
> >  >  > On dimanche 5 septembre 2021 12:26:50 CEST Ben Cooksley
> wrote:
> >  >  >> On Sun, Sep 5, 2021 at 10:22 PM David Faure
> > mailto:fa...@kde.org>
> >  > <mailto:fa...@kde.org <mailto:fa...@kde.org>>> wrote:
> >  >  >>> For frameworks, I think we should be able to write a
> > one-time
> >  > script that
> >  >  >>> generates .kde-ci.yml files using the dependencies
> listed in
> >  > kde-build-
> >  >  >>> metadata (and the platforms listed in metainfo.yaml) ?
> >  >  >>
> >  >  >> Yes, that should work nicely (although that information
> > now lives in
> >  >  >> sysadmin/repo-metadata, dependencies folder)
> >  >  >
> >  >  > All done for Frameworks.
> >  >  >
> >  >  > The script that collects platforms from metainfo.yaml
> won't be
> >  > useful for
> >  >  > other KDE modules, but the script that collects deps from
> >  > dependency-data-kf5-
> >  >  > qt5 is attached, in case it's useful to anyone else.
> >  >
> >  > Is there a file that maps legacy project paths to gitlab
> > project paths?
> >  > dependency-data-kf5-qt5 lists projects with their legacy
> paths.
> >  >
> >  >
> >  > The individual project YAML files in sysadmin/repo-metadata
> > contain this
> >  > information.
> >  > The legacy project path can be found under 'projectpath' while the
> >  > Gitlab paths are under 'repopath'
> >
> > Thanks! I've attached a quick and dirty python script that parses
> > project dependencies from repo-metadata and prints corresponding
> gitlab
> > project paths.
> >
> > Example usage
> >
> > python project-deps.py --repo-metadata
> > /data/projects/src/repo-metadata/ kde/workspace/kwin
> >
> > Output
> >
> > Dependencies:
> > 'requires':
> >   'frameworks/extra-cmake-modules': '@stable'
> >   'plasma/kdecoration': '@stable'
> >   'plasma/kscreenlocker': '@stable'
> >   'plasma/kwayland-integration': '@stable'
> >   'plasma/breeze': '@stable'
> >   'plasma/kwayland-server': '@stable'
> >
> >
> > Please note that the above is missing a 'on' section as required for
> > each Dependencies section in the .kde-ci.yml file.
> Yeah... Is there a database or something that can be queried to fill in
> the "on" section?
>

I'm afraid this is information we have not really collected elsewhere at
this time.

Valid values are: Linux, FreeBSD, Windows, macOS, Android
(capitalisation sensitive)

Cheers,
Ben


Re: Gitlab CI - Inbound

2021-09-07 Thread Ben Cooksley
On Tue, Sep 7, 2021 at 10:14 PM Vlad Zahorodnii 
wrote:

> On 9/7/21 12:22 PM, Ben Cooksley wrote:
> > On Tue, Sep 7, 2021 at 8:48 PM Vlad Zahorodnii  > <mailto:vlad.zahorod...@kde.org>> wrote:
> >
> > On 9/5/21 3:18 PM, David Faure wrote:
> >  > On dimanche 5 septembre 2021 12:26:50 CEST Ben Cooksley wrote:
> >  >> On Sun, Sep 5, 2021 at 10:22 PM David Faure  > <mailto:fa...@kde.org>> wrote:
> >  >>> For frameworks, I think we should be able to write a one-time
> > script that
> >  >>> generates .kde-ci.yml files using the dependencies listed in
> > kde-build-
> >  >>> metadata (and the platforms listed in metainfo.yaml) ?
> >  >>
> >  >> Yes, that should work nicely (although that information now
> lives in
> >  >> sysadmin/repo-metadata, dependencies folder)
> >  >
> >  > All done for Frameworks.
> >  >
> >  > The script that collects platforms from metainfo.yaml won't be
> > useful for
> >  > other KDE modules, but the script that collects deps from
> > dependency-data-kf5-
> >  > qt5 is attached, in case it's useful to anyone else.
> >
> > Is there a file that maps legacy project paths to gitlab project
> paths?
> > dependency-data-kf5-qt5 lists projects with their legacy paths.
> >
> >
> > The individual project YAML files in sysadmin/repo-metadata contain this
> > information.
> > The legacy project path can be found under 'projectpath' while the
> > Gitlab paths are under 'repopath'
>
> Thanks! I've attached a quick and dirty python script that parses
> project dependencies from repo-metadata and prints corresponding gitlab
> project paths.
>
> Example usage
>
>python project-deps.py --repo-metadata
> /data/projects/src/repo-metadata/ kde/workspace/kwin
>
> Output
>
> Dependencies:
>'requires':
>  'frameworks/extra-cmake-modules': '@stable'
>  'plasma/kdecoration': '@stable'
>  'plasma/kscreenlocker': '@stable'
>  'plasma/kwayland-integration': '@stable'
>  'plasma/breeze': '@stable'
>  'plasma/kwayland-server': '@stable'
>

Please note that the above is missing a 'on' section as required for each
Dependencies section in the .kde-ci.yml file.


> Cheers,
> Vlad
>

Cheers,
Ben


> >
> >
> > Cheers,
> > Ben
> >
> >
> >  >> Once we've transitioned across to the .kde-ci.yml files the
> existing
> >  >> dependency metadata and branch group files will no longer be
> > required.
> >  >> (We'll need to work out a way of exporting this information for
> easy
> >  >> consumption by kdesrc-build and other clients before it can be
> > removed
> >  >> though)
> >  >
> >  > OK. Duplication is bad so I agree :)
> >  >
> >
>
>


Re: Gitlab CI - Inbound

2021-09-07 Thread Ben Cooksley
On Tue, Sep 7, 2021 at 8:48 PM Vlad Zahorodnii 
wrote:

> On 9/5/21 3:18 PM, David Faure wrote:
> > On dimanche 5 septembre 2021 12:26:50 CEST Ben Cooksley wrote:
> >> On Sun, Sep 5, 2021 at 10:22 PM David Faure  wrote:
> >>> For frameworks, I think we should be able to write a one-time script
> that
> >>> generates .kde-ci.yml files using the dependencies listed in kde-build-
> >>> metadata (and the platforms listed in metainfo.yaml) ?
> >>
> >> Yes, that should work nicely (although that information now lives in
> >> sysadmin/repo-metadata, dependencies folder)
> >
> > All done for Frameworks.
> >
> > The script that collects platforms from metainfo.yaml won't be useful for
> > other KDE modules, but the script that collects deps from
> dependency-data-kf5-
> > qt5 is attached, in case it's useful to anyone else.
>
> Is there a file that maps legacy project paths to gitlab project paths?
> dependency-data-kf5-qt5 lists projects with their legacy paths.
>

The individual project YAML files in sysadmin/repo-metadata contain this
information.
The legacy project path can be found under 'projectpath' while the Gitlab
paths are under 'repopath'


> vlad
>

Cheers,
Ben


> >> Once we've transitioned across to the .kde-ci.yml files the existing
> >> dependency metadata and branch group files will no longer be required.
> >> (We'll need to work out a way of exporting this information for easy
> >> consumption by kdesrc-build and other clients before it can be removed
> >> though)
> >
> > OK. Duplication is bad so I agree :)
> >
>
>


Re: Gitlab CI - Inbound

2021-09-07 Thread Ben Cooksley
On Tue, Sep 7, 2021 at 9:09 PM David Edmundson 
wrote:

> Excellent news!! Thanks very much
>
> > Once the scripts have been proven successfully for Frameworks, we will
> look at extending them to projects that depend only on Frameworks and
> repositories
>
> Does this mean we would like Plasma to wait a while before merging?
> Is it worth us creating the kde-cli files and not merging them so we
> have some test cases ready?
>

You can certainly get the .kde-ci.yml files in position - I do not expect
the format of them to change at this stage.

The big thing that will delay the rollout is determining how best to handle
the situation of when two projects want different versions of the same
dependency.
I'm going to have to figure that out sooner than anticipated in any event
due to Phonon and plasma-wayland-protocols which are both dependencies of
Frameworks (yet which both also require Frameworks).


> David
>

Cheers,
Ben


Re: Gitlab CI - Inbound

2021-09-07 Thread Ben Cooksley
On Tue, Sep 7, 2021 at 6:20 AM Johnny Jazeix  wrote:

> Hi Ben,
> not sure on which priority it is regarding the KDE Frameworks but I've
> added one on GCompris (
> https://invent.kde.org/education/gcompris/-/commit/67c9839d7970b360b5d6b0ec928b492f9003d07d)
> if it can help on more tests.
>

Thanks for getting that landed Johnny.

Please note that you've specified no dependencies, so your builds won't
even have ECM available so you may wish to fix that.


> Cheers,
>
> Johnny
>

Cheers,
Ben


> Le dim. 5 sept. 2021 à 12:11, Ben Cooksley  a écrit :
>
>> On Sun, Sep 5, 2021 at 6:13 PM Ben Cooksley  wrote:
>>
>>> Hi all,
>>>
>>
>> Hi all,
>>
>>
>>> This morning after much work i'm happy to announce that the new
>>> generation CI scripts intended for use with Gitlab CI successfully
>>> completed their first build (of ECM, and then subsequently of KCoreAddons).
>>>
>>> This begins our first steps towards transferring CI over from Jenkins to
>>> Gitlab.
>>>
>>> These scripts are 'Gitlab native' and are designed to work in an
>>> environment where they will be running on merge requests, tags as well as
>>> all branches of our repositories - something the existing scripts were
>>> completely incompatible with. While there are still some infrastructural
>>> elements to put in place (such as 'seed jobs' to mass rebuild all projects
>>> after substantial changes and 'cleanup jobs' to remove old builds from the
>>> Package Registry) the core elements needed for initial testing of these
>>> scripts are now ready.
>>>
>>
>> As an update, an initial version of the seed job tooling is now ready,
>> however testing that tooling requires that dependency information be
>> available.
>> This requires that the .kde-ci.yml files be populated in repositories.
>>
>> It would be appreciated if people could please work on getting these
>> files populated in Frameworks (as everyone needs those) as well as in their
>> own repositories as they are required before we can proceed much further.
>>
>>
>>> For those curious, the results of those initials runs can be found at
>>> https://invent.kde.org/groups/teams/ci-artifacts/-/packages
>>>
>>> Due to the challenges associated with having to handle all of the
>>> different forms of build and the versioning of this information, these
>>> scripts also represent a radical change in how CI builds will be conducted
>>> - with a large part of the configuration of the jobs themselves, including
>>> information on project dependencies now shifting to files located within
>>> the repository themselves. Those who monitor commits to Frameworks
>>> repositories will have noticed the appearance of these '.kde-ci.yml' files
>>> in some repositories.
>>>
>>> To assist in the future rollout of the CI system it would be appreciated
>>> if projects could please begin setting up these files within their
>>> respective repositories.
>>> You can find a template detailing the available options at
>>> https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config-template.yml
>>>
>>> Where possible please avoid overriding the system defaults except where
>>> needed by your project (ie. don't just copy the template in full)
>>> The defaults mirror the template and can be found at
>>> https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config/global.yml
>>>
>>> In terms of the format of the 'Dependencies' section, please bear in
>>> mind the following:
>>> - For the 'on' section, the special value '@all' can be used to mean
>>> "All platforms" to avoid needing to update the file in the event additional
>>> platforms are added in the future
>>> - For the 'require' section:
>>>   1) Please only include the projects you *explicitly* depend on.
>>> Dependencies of your dependencies will be included automatically
>>>   2) When specifying a project, you must use the repository path on
>>> Gitlab. Legacy project paths (such as kde/workspace/plasma-desktop) are no
>>> longer in use and will not work.
>>>   3) There are three special values for the branch name - '@same',
>>> '@latest' and '@stable' which can be used to refer to the branch/tag of a
>>> dependency.
>>>   '@same' means the branch name should be the same as that of the
>>> project being built and should be used when declaring dependencies among
>>&g

Re: Gitlab CI - Inbound

2021-09-07 Thread Ben Cooksley
On Tue, Sep 7, 2021 at 1:04 AM Tom Zander  wrote:

> On maandag 6 september 2021 11:48:39 CEST Ben Cooksley wrote:
> > > Pushing everything into required is likely not scalable,
> > > causing projects too wait too long for compile.
> > > Avoiding the optional ones means you lack coverage of compile
> > > and testing failures due to changes in libs.
> >
> > The CI system has reused the results of previous builds of
> > dependencies since the very first generation of the system
>
> We seem to be talking about two slightly different topics.
>
> When the (for instance) KIO repo changes, then the CI will
> obviously rebuild that repo and will pull in all the things that
> kio depends on.
>
> What many CIs do is additionally trigger rebuilds of projects
> that _use_ KIO, by them marking kio as a required dependency.
>

Our CI system has never done this, in part due to difficulties in
communicating this to Jenkins and also because of the build storm it would
create (which I believe is what you are referring to).
If we were to limit it to select projects, then we would need to 'pick'
those projects which would raise questions of favouritism which I'd rather
avoid.


> Imagine a small extragear app that uses some KIO stuff and it has
> some unit tests that would break as a result of the KIO change.
> When the KDE CI triggers a rebuild of projects that mark KIO as a
> required dependency, this little app would show its breaking as a
> result of the KIO push. Helping the KIO dev to realize the
> fallout as well.
> Without such a feature the app would show breakage at a random
> time in the future after a new push was made in that repo. Losing
> lots of dev time and compromising quality.
>
> Now, the optional requirements would help diminish the effect of
> changes in frameworks paralysing the build system by limiting the
> apps that gets scheduled for a rebuild.
>

> This kind of functionality becomes pretty easy to add to gen5.1
> of the CI, provided that at this time the dependencies are
> already split since doing it later is going to be an uphill
> battle.
>

Cheers,
Ben


Re: Gitlab CI - Inbound

2021-09-06 Thread Ben Cooksley
On Mon, Sep 6, 2021 at 9:00 PM Tom Zander  wrote:

> On zondag 5 september 2021 08:13:09 CEST Ben Cooksley wrote:
> > In terms of the format of the 'Dependencies' section,
>
> Playing with kde-build script and noticing the fast growing
> dependency trees we have today, I think it may be beneficial to
> have two types of compile dependencies in this setup.
>
> 1. required-to-build.
>   Which means that if something in the parent tree changes, this
>   one is scheduled for re-build.
>
> 2. optional. Equivalent of the cmake feature, if its not there
>   some code is not compiled.
>   At least once before a release the full dependencies can be
>   compiled to see if it fully compiles.
>

>From the perspective of the CI system there is basically no difference in
terms of making a dependency available or not as all dependencies are
always satisfied using previously built binaries.
If a dependency is not available your build fails.

We also have to make a hard choice - we either bring in optional
dependencies or we don't.

If we were to randomise whether we brought them in - or just brought them
in at certain times - then we would make the system state deterministic.
This could in some cases cause builds to break if it was random whether or
not we included optional dependencies. This would occur if the dependency
of a dependency of a project were rebuilt without an optional dependency
being present which in turn caused it's API interface to change. That would
then cause the project to fail as the dependency would now be broken.


>
> Pushing everything into required is likely not scalable, causing
> projects too wait too long for compile.
> Avoiding the optional ones means you lack coverage of compile and
> testing failures due to changes in libs.
>

The CI system has reused the results of previous builds of dependencies
since the very first generation of the system (this would be generation 5)
so this is not a problem facing the CI system.
For individual developers, my understanding is that kdesrc-build makes use
of incremental builds which eliminates most of the issue as well.


> What do people think, is it useful to have an 'optional' category
> in future there?
> Maybe useful to think that far ahead now people are populating
> their dependencies :-)
>
>
>
>
Thanks,
Ben


Re: Gitlab CI - Inbound

2021-09-06 Thread Ben Cooksley
On Mon, Sep 6, 2021 at 12:18 AM David Faure  wrote:

> On dimanche 5 septembre 2021 12:26:50 CEST Ben Cooksley wrote:
> > On Sun, Sep 5, 2021 at 10:22 PM David Faure  wrote:
> > > For frameworks, I think we should be able to write a one-time script
> that
> > > generates .kde-ci.yml files using the dependencies listed in kde-build-
> > > metadata (and the platforms listed in metainfo.yaml) ?
> >
> > Yes, that should work nicely (although that information now lives in
> > sysadmin/repo-metadata, dependencies folder)
>
> All done for Frameworks.
>
> The script that collects platforms from metainfo.yaml won't be useful for
> other KDE modules, but the script that collects deps from
> dependency-data-kf5-
> qt5 is attached, in case it's useful to anyone else.
>

Thanks for sorting this David, very much appreciated.


> > Once we've transitioned across to the .kde-ci.yml files the existing
> > dependency metadata and branch group files will no longer be required.
> > (We'll need to work out a way of exporting this information for easy
> > consumption by kdesrc-build and other clients before it can be removed
> > though)
>
> OK. Duplication is bad so I agree :)
>

Cheers,
Ben


> --
> David Faure, fa...@kde.org, http://www.davidfaure.fr
> Working on KDE Frameworks 5
>


Re: Gitlab CI - Inbound

2021-09-06 Thread Ben Cooksley
On Mon, Sep 6, 2021 at 12:46 AM Nicolas Fella  wrote:

> On 05.09.21 08:13, Ben Cooksley wrote:
> > Hi all,
> >
> > This morning after much work i'm happy to announce that the new
> > generation CI scripts intended for use with Gitlab CI successfully
> > completed their first build (of ECM, and then subsequently of
> > KCoreAddons).
> >
> > This begins our first steps towards transferring CI over from Jenkins
> > to Gitlab.
> >
> > These scripts are 'Gitlab native' and are designed to work in an
> > environment where they will be running on merge requests, tags as well
> > as all branches of our repositories - something the existing scripts
> > were completely incompatible with. While there are still some
> > infrastructural elements to put in place (such as 'seed jobs' to mass
> > rebuild all projects after substantial changes and 'cleanup jobs' to
> > remove old builds from the Package Registry) the core elements needed
> > for initial testing of these scripts are now ready.
> >
> > For those curious, the results of those initials runs can be found at
> > https://invent.kde.org/groups/teams/ci-artifacts/-/packages
> > <https://invent.kde.org/groups/teams/ci-artifacts/-/packages>
> >
> > Due to the challenges associated with having to handle all of the
> > different forms of build and the versioning of this information, these
> > scripts also represent a radical change in how CI builds will be
> > conducted - with a large part of the configuration of the jobs
> > themselves, including information on project dependencies now shifting
> > to files located within the repository themselves. Those who monitor
> > commits to Frameworks repositories will have noticed the appearance of
> > these '.kde-ci.yml' files in some repositories.
> >
> > To assist in the future rollout of the CI system it would be
> > appreciated if projects could please begin setting up these files
> > within their respective repositories.
> > You can find a template detailing the available options at
> >
> https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config-template.yml
> > <
> https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config-template.yml
> >
> >
> > Where possible please avoid overriding the system defaults except
> > where needed by your project (ie. don't just copy the template in full)
> > The defaults mirror the template and can be found at
> >
> https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config/global.yml
> > <
> https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config/global.yml
> >
> >
> > In terms of the format of the 'Dependencies' section, please bear in
> > mind the following:
> > - For the 'on' section, the special value '@all' can be used to mean
> > "All platforms" to avoid needing to update the file in the event
> > additional platforms are added in the future
> > - For the 'require' section:
> >   1) Please only include the projects you *explicitly* depend on.
> > Dependencies of your dependencies will be included automatically
> >   2) When specifying a project, you must use the repository path on
> > Gitlab. Legacy project paths (such as kde/workspace/plasma-desktop)
> > are no longer in use and will not work.
> >   3) There are three special values for the branch name - '@same',
> > '@latest' and '@stable' which can be used to refer to the branch/tag
> > of a dependency.
> >   '@same' means the branch name should be the same as that of the
> > project being built and should be used when declaring dependencies
> > among projects in a release group.
> >   '@latest' and '@stable' refer to the corresponding branches
> > defined in the branch-rules.yml file, which can be found in
> > sysadmin/repo-metadata
> >
> > As a general rule, unless you require the absolute latest version of
> > another project in another release unit, it is recommended that you
> > use @stable.
> > Please avoid using explicit branch names unless absolutely necessary.
> >
> > At this time it is expected that the first set of Gitlab CI builds
> > using the new scripts may be conducted for Frameworks within the next
> > week or so, assuming the build of the seed jobs goes according to plan.
> >
> > Once the scripts have been proven successfully for Frameworks, we will
> > look at extending them to projects that depend only on Frameworks and
> > repositories within their release unit and finally will extend

Re: Gitlab CI - Inbound

2021-09-06 Thread Ben Cooksley
On Mon, Sep 6, 2021 at 11:03 AM Michael Reeves  wrote:

> How do we get a visual on exactly which lines are covered by auto testing
> and which aren't?
>

Please see
https://docs.gitlab.com/ee/user/project/merge_requests/test_coverage_visualization.html
for more details on how this works on Merge Requests.
To my knowledge this isn't exposed outside of Merge Requests.

Cheers,
Ben


Re: Gitlab CI - Inbound

2021-09-05 Thread Ben Cooksley
On Sun, Sep 5, 2021 at 10:22 PM David Faure  wrote:

> On dimanche 5 septembre 2021 12:11:05 CEST Ben Cooksley wrote:
> > It would be appreciated if people could please work on getting these
> files
> > populated in Frameworks (as everyone needs those) as well as in their own
> > repositories as they are required before we can proceed much further.
>
> Like this
> https://invent.kde.org/frameworks/kcoreaddons/-/merge_requests/135 ?
>

Yes - just two small tweaks needed for that one but it is otherwise good to
ship.


> For frameworks, I think we should be able to write a one-time script that
> generates .kde-ci.yml files using the dependencies listed in kde-build-
> metadata (and the platforms listed in metainfo.yaml) ?
>

Yes, that should work nicely (although that information now lives in
sysadmin/repo-metadata, dependencies folder)

Once we've transitioned across to the .kde-ci.yml files the existing
dependency metadata and branch group files will no longer be required.
(We'll need to work out a way of exporting this information for easy
consumption by kdesrc-build and other clients before it can be removed
though)


> --
> David Faure, fa...@kde.org, http://www.davidfaure.fr
> Working on KDE Frameworks 5
>
>
Thanks,
Ben


Re: Gitlab CI - Inbound

2021-09-05 Thread Ben Cooksley
On Sun, Sep 5, 2021 at 6:13 PM Ben Cooksley  wrote:

> Hi all,
>

Hi all,


> This morning after much work i'm happy to announce that the new generation
> CI scripts intended for use with Gitlab CI successfully completed their
> first build (of ECM, and then subsequently of KCoreAddons).
>
> This begins our first steps towards transferring CI over from Jenkins to
> Gitlab.
>
> These scripts are 'Gitlab native' and are designed to work in an
> environment where they will be running on merge requests, tags as well as
> all branches of our repositories - something the existing scripts were
> completely incompatible with. While there are still some infrastructural
> elements to put in place (such as 'seed jobs' to mass rebuild all projects
> after substantial changes and 'cleanup jobs' to remove old builds from the
> Package Registry) the core elements needed for initial testing of these
> scripts are now ready.
>

As an update, an initial version of the seed job tooling is now ready,
however testing that tooling requires that dependency information be
available.
This requires that the .kde-ci.yml files be populated in repositories.

It would be appreciated if people could please work on getting these files
populated in Frameworks (as everyone needs those) as well as in their own
repositories as they are required before we can proceed much further.


> For those curious, the results of those initials runs can be found at
> https://invent.kde.org/groups/teams/ci-artifacts/-/packages
>
> Due to the challenges associated with having to handle all of the
> different forms of build and the versioning of this information, these
> scripts also represent a radical change in how CI builds will be conducted
> - with a large part of the configuration of the jobs themselves, including
> information on project dependencies now shifting to files located within
> the repository themselves. Those who monitor commits to Frameworks
> repositories will have noticed the appearance of these '.kde-ci.yml' files
> in some repositories.
>
> To assist in the future rollout of the CI system it would be appreciated
> if projects could please begin setting up these files within their
> respective repositories.
> You can find a template detailing the available options at
> https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config-template.yml
>
> Where possible please avoid overriding the system defaults except where
> needed by your project (ie. don't just copy the template in full)
> The defaults mirror the template and can be found at
> https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config/global.yml
>
> In terms of the format of the 'Dependencies' section, please bear in mind
> the following:
> - For the 'on' section, the special value '@all' can be used to mean "All
> platforms" to avoid needing to update the file in the event additional
> platforms are added in the future
> - For the 'require' section:
>   1) Please only include the projects you *explicitly* depend on.
> Dependencies of your dependencies will be included automatically
>   2) When specifying a project, you must use the repository path on
> Gitlab. Legacy project paths (such as kde/workspace/plasma-desktop) are no
> longer in use and will not work.
>   3) There are three special values for the branch name - '@same',
> '@latest' and '@stable' which can be used to refer to the branch/tag of a
> dependency.
>   '@same' means the branch name should be the same as that of the
> project being built and should be used when declaring dependencies among
> projects in a release group.
>   '@latest' and '@stable' refer to the corresponding branches defined
> in the branch-rules.yml file, which can be found in sysadmin/repo-metadata
>
> As a general rule, unless you require the absolute latest version of
> another project in another release unit, it is recommended that you use
> @stable.
> Please avoid using explicit branch names unless absolutely necessary.
>
> At this time it is expected that the first set of Gitlab CI builds using
> the new scripts may be conducted for Frameworks within the next week or so,
> assuming the build of the seed jobs goes according to plan.
>
> Once the scripts have been proven successfully for Frameworks, we will
> look at extending them to projects that depend only on Frameworks and
> repositories within their release unit and finally will extend them to
> projects with more complex dependency requirements. It is expected that the
> switch will be flipped on the Frameworks builds sometime in the next week
> or two.
>
> Please let me know if you have any questions on the above.
>
> Thanks,
> Ben Cooksley
> KDE Sysadmin
>

Thanks,
Ben


Gitlab CI - Inbound

2021-09-05 Thread Ben Cooksley
Hi all,

This morning after much work i'm happy to announce that the new generation
CI scripts intended for use with Gitlab CI successfully completed their
first build (of ECM, and then subsequently of KCoreAddons).

This begins our first steps towards transferring CI over from Jenkins to
Gitlab.

These scripts are 'Gitlab native' and are designed to work in an
environment where they will be running on merge requests, tags as well as
all branches of our repositories - something the existing scripts were
completely incompatible with. While there are still some infrastructural
elements to put in place (such as 'seed jobs' to mass rebuild all projects
after substantial changes and 'cleanup jobs' to remove old builds from the
Package Registry) the core elements needed for initial testing of these
scripts are now ready.

For those curious, the results of those initials runs can be found at
https://invent.kde.org/groups/teams/ci-artifacts/-/packages

Due to the challenges associated with having to handle all of the different
forms of build and the versioning of this information, these scripts also
represent a radical change in how CI builds will be conducted - with a
large part of the configuration of the jobs themselves, including
information on project dependencies now shifting to files located within
the repository themselves. Those who monitor commits to Frameworks
repositories will have noticed the appearance of these '.kde-ci.yml' files
in some repositories.

To assist in the future rollout of the CI system it would be appreciated if
projects could please begin setting up these files within their respective
repositories.
You can find a template detailing the available options at
https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config-template.yml

Where possible please avoid overriding the system defaults except where
needed by your project (ie. don't just copy the template in full)
The defaults mirror the template and can be found at
https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config/global.yml

In terms of the format of the 'Dependencies' section, please bear in mind
the following:
- For the 'on' section, the special value '@all' can be used to mean "All
platforms" to avoid needing to update the file in the event additional
platforms are added in the future
- For the 'require' section:
  1) Please only include the projects you *explicitly* depend on.
Dependencies of your dependencies will be included automatically
  2) When specifying a project, you must use the repository path on Gitlab.
Legacy project paths (such as kde/workspace/plasma-desktop) are no longer
in use and will not work.
  3) There are three special values for the branch name - '@same',
'@latest' and '@stable' which can be used to refer to the branch/tag of a
dependency.
  '@same' means the branch name should be the same as that of the
project being built and should be used when declaring dependencies among
projects in a release group.
  '@latest' and '@stable' refer to the corresponding branches defined
in the branch-rules.yml file, which can be found in sysadmin/repo-metadata

As a general rule, unless you require the absolute latest version of
another project in another release unit, it is recommended that you use
@stable.
Please avoid using explicit branch names unless absolutely necessary.

At this time it is expected that the first set of Gitlab CI builds using
the new scripts may be conducted for Frameworks within the next week or so,
assuming the build of the seed jobs goes according to plan.

Once the scripts have been proven successfully for Frameworks, we will look
at extending them to projects that depend only on Frameworks and
repositories within their release unit and finally will extend them to
projects with more complex dependency requirements. It is expected that the
switch will be flipped on the Frameworks builds sometime in the next week
or two.

Please let me know if you have any questions on the above.

Thanks,
Ben Cooksley
KDE Sysadmin


[sysadmin/ci-tooling] local-metadata: The dependency builds for Android starting picking up KDeclarative, and it seems to be broken due to something related to KIO.

2021-08-25 Thread Ben Cooksley
Git commit 7c88db74c67a93883b8af0cbd78dc94287200663 by Ben Cooksley.
Committed on 25/08/2021 at 19:20.
Pushed by bcooksley into branch 'master'.

The dependency builds for Android starting picking up KDeclarative, and it 
seems to be broken due to something related to KIO.
Enable CI for this to allow it to be fixed.

CCMAIL: kde-frameworks-devel@kde.org

M  +1-0local-metadata/product-definitions.yaml

https://invent.kde.org/sysadmin/ci-tooling/commit/7c88db74c67a93883b8af0cbd78dc94287200663

diff --git a/local-metadata/product-definitions.yaml 
b/local-metadata/product-definitions.yaml
index b344361..1f9bd1c 100644
--- a/local-metadata/product-definitions.yaml
+++ b/local-metadata/product-definitions.yaml
@@ -47,6 +47,7 @@
   - "frameworks/kservice"
   - "frameworks/solid"
   - "frameworks/kbookmarks"
+  - "frameworks/kdeclarative"
   platforms:
   - "AndroidQt5.15"
   branchGroups:


[sysadmin/ci-tooling] local-metadata: Enable Android CI for KIO.

2021-08-04 Thread Ben Cooksley
Git commit 19abb08cef0a6a889a5843a9bd749a2363ffa04a by Ben Cooksley.
Committed on 04/08/2021 at 09:27.
Pushed by bcooksley into branch 'master'.

Enable Android CI for KIO.

Ref T14712

CCMAIL: kde-frameworks-devel@kde.org

M  +1-0local-metadata/product-definitions.yaml

https://invent.kde.org/sysadmin/ci-tooling/commit/19abb08cef0a6a889a5843a9bd749a2363ffa04a

diff --git a/local-metadata/product-definitions.yaml 
b/local-metadata/product-definitions.yaml
index 01ce954..d05aeaa 100644
--- a/local-metadata/product-definitions.yaml
+++ b/local-metadata/product-definitions.yaml
@@ -35,6 +35,7 @@
   - "frameworks/kplotting"
   - "frameworks/syndication"
   - "frameworks/kpeople"
+  - "frameworks/kio"
   platforms:
   - "AndroidQt5.15"
   branchGroups:


Re: KDE CI SignTool Error: No certificates were found that met all the given criteria.

2021-07-15 Thread Ben Cooksley
On Thu, Jul 15, 2021 at 11:12 PM Ralf Habacker 
wrote:

> Hi
>

Hi Ralf,


> building kmymoney releases on KDE CI are currently broken by an
> installation issue
>
> https://binary-factory.kde.org/view/Windows%2064-bit/job/KMyMoney_Release_win64/1081/console
> .
> Please fix.
>

That was already fixed yesterday evening.

The current failure is due to a different issue:
https://binary-factory.kde.org/view/Windows%2064-bit/job/KMyMoney_Release_win64/1082/console

Regards,
Ben


> Regards
> Ralf
>


Re: Fwd: KDE CI: Administration » Dependency Build Applications stable-kf5-qt5 WindowsMSVCQt5.15 - Build # 52 - Still Failing!

2021-07-14 Thread Ben Cooksley
On Mon, Jul 12, 2021 at 8:21 PM Ralf Habacker 
wrote:

> Am 12.07.21 um 09:35 schrieb Ralf Habacker:
> >
> >> The fact that you have to apply additional overrides here indicates to
> >> me that your changes to 'kdewin' are also incorrect.
> >
> > It may also be caused by the kdelibs4support package, as I initial
> > stated - I double checked in kdewin git repo, that it exports the
> > correct header and library functions.
> >
>
> At https://build.kde.org/job/Administration/job/Dependency Build
> Extragear stable-kf5-qt5 .15/55/consoleFull you can see
>
> 13:57:34  -- Looking for inet_pton
> 13:57:34  -- Looking for inet_pton - not found
> 13:57:34  -- Looking for inet_ntop
> 13:57:34  -- Looking for inet_ntop - not found
>
> that  cmake does not find inet_ ... function, which *are* available by
> msvc compiler since Windows Vista
> (
> https://docs.microsoft.com/en-us/windows/win32/api/ws2tcpip/nf-ws2tcpip-inet_pton
> )
>

That makes sense - guess it was fixed in the newer Frameworks but never in
kdelibs4support due to it's legacy nature.


>
> I guess using the same approach as done now in kdewin at
>
> https://invent.kde.org/packaging/kdewin/-/merge_requests/1/diffs?commit_id=e7ba87c10528b71ec53865c661dd718447494591#a9f1c36d35a73bf675ad0d1641cad7d3df55284e_27_28
>
> should fix that issue. My earlier mentioned comment related to add the
> -DHAVE_INET_PTON and -DHAVE_INET_NTOP is a valid workaround.
>

It would be preferrable to fix things correctly I think. Please note that I
don't have a developer setup, so i'm unable to prepare/test such a patch.


> Regards
> Ralf
>

Thanks,
Ben

>>
> >>>
> >>> -- Forwarded message -
> >>> From: *CI System* mailto:nore...@kde.org>>
> >>> Date: Sat, Jul 10, 2021 at 7:21 AM
> >>> Subject: KDE CI: Administration » Dependency Build Applications
> >>> stable-kf5-qt5 WindowsMSVCQt5.15 - Build # 52 - Still Failing!
> >>> To: mailto:bcooks...@kde.org>>
> >>>
> >>>
> >>> *BUILD FAILURE*
> >>> Build URL
> >>>
> https://build.kde.org/job/Administration/job/Dependency%20Build%20Applications%20stable-kf5-qt5%20WindowsMSVCQt5.15/52/
> >>> <
> https://build.kde.org/job/Administration/job/Dependency%20Build%20Applications%20stable-kf5-qt5%20WindowsMSVCQt5.15/52/
> >
> >>>
> >>> Project:Dependency Build Applications stable-kf5-qt5
> >>> WindowsMSVCQt5.15
> >>> Date of build:  Fri, 09 Jul 2021 18:58:42 +
> >>> Build duration: 22 min and counting
> >>>
> >>>
> >>> *CONSOLE OUTPUT *
> >>> [...truncated 51900 lines...]
> >>> [2021-07-09T19:20:03.112Z]
> >>>
>  
> C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
> >>> warning C4005: 'Q_OS_WIN': macro redefinition
> >>> [2021-07-09T19:20:03.112Z] ..\src\kdeui\kicon.cpp: note: see
> >>> previous definition of 'Q_OS_WIN'
> >>> [2021-07-09T19:20:03.112Z] [153/533] Building CXX object
> >>> src\CMakeFiles\KF5KDELibs4Support.dir\kdeui\kdialogqueue.cpp.obj
> >>> [2021-07-09T19:20:03.112Z]
> >>>
>  
> C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
> >>> warning C4005: 'Q_OS_WIN': macro redefinition
> >>> [2021-07-09T19:20:03.112Z] ..\src\kdeui\kdialogqueue.cpp: note:
> >>> see previous definition of 'Q_OS_WIN'
> >>> [2021-07-09T19:20:03.112Z] [154/533] Building CXX object
> >>>
>  src\CMakeFiles\KF5KDELibs4Support.dir\kdecore\ktzfiletimezone.cpp.obj
> >>> [2021-07-09T19:20:03.112Z]
> >>>
>  
> C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
> >>> warning C4005: 'Q_OS_WIN': macro redefinition
> >>> [2021-07-09T19:20:03.112Z] ..\src\kdecore\ktzfiletimezone.cpp:
> >>> note: see previous definition of 'Q_OS_WIN'
> >>> [2021-07-09T19:20:03.112Z] [155/533] Building CXX object
> >>> src\CMakeFiles\KF5KDELibs4Support.dir\kdeui\kapplication.cpp.obj
> >>> [2021-07-09T19:20:03.112Z]
> >>>
>  
> C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
> >>> warning C4005: 'Q_OS_WIN': macro redefinition
> >>> [2021-07-09T19:20:03.112Z] ..\src\kdeui\kapplication.cpp: note:
> >>> see previous definition of 'Q_OS_WIN'
> >>> [2021-07-09T19:20:03.684Z] [156/533] Building CXX object
> >>>
>  src\CMakeFiles\KF5KDELibs4Support.dir\kdeui\kstringvalidator.cpp.obj
> >>> [2021-07-09T19:20:03.684Z]
> >>>
>  
> C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
> >>> warning C4005: 'Q_OS_WIN': macro redefinition
> >>> [2021-07-09T19:20:03.684Z] ..\src\kdeui\kstringvalidator.cpp:
> >>> note: see previous definition of 'Q_OS_WIN'
> >>> [2021-07-09T19:20:03.684Z] [157/533] Building CXX object
> >>>
>  src\CMakeFiles\KF5KDELibs4Support.dir\kdeui\kfadewidgeteffect.cpp.obj
> >>> [2021-07-09T19:20:03.684Z]
> >>>
>  
> 

Re: Fwd: KDE CI: Administration » Dependency Build Applications stable-kf5-qt5 WindowsMSVCQt5.15 - Build # 52 - Still Failing!

2021-07-10 Thread Ben Cooksley
On Sat, Jul 10, 2021 at 9:14 PM Ralf Habacker 
wrote:

> Am 09.07.21 um 21:26 schrieb Ben Cooksley:
>
> Hi Ralf,
>
> I'm afraid your previous fixes are insufficient - we are still failing
> with a linking error.
>
> The following is notice that I intend to back all changes out in kdewin
> and restore the repository to 8503ac6e0e07099c4938cbfe60e3f5e25e2ec368 to
> restore the correct function of the CI system in 24 hours time.
>
> Please don't that not - that fix is required for building kdelibs4support
> with mingw headers version 9 and only export replacement functions not
> provided by the os.
>
The platform you are talking about (mingw cross compilation) is not one
generally supported by KDE on Windows - I believe you are the only person
who uses that approach.

The MingW version we do support (native compilation), along with MSVC (as
the principal compiler) do not have the issue you are describing.
This is how all the Binary Factory builds are run, and therefore how every
project on there is built.


> You should either revert the kdewin build script to checkout the source
> from a git tag (v0.6.4) and not master as done by other packages (I'm
> sorry, I cannot do that because of missing knowledge).
>
> At the  mentioned mingw cross build package for kdelib4support I need to
> add  -DHAVE_INET_PTON=1 -DHAVE_INET_NTOP=1 to the cmake configure as
> workaround to the broken inet_xxx function detection inside
> kdeblibs4support source.
>
Sorry, but that is not an option - the CI system builds the same set of
branches that a developer and our release managers build.
That means 'master'.

The fact that you have to apply additional overrides here indicates to me
that your changes to 'kdewin' are also incorrect.

Once the CI system has settled down following the Release Gear tagging i'll
be performing the revert as previously mentioned unless a fix to this is
pushed.


> Ralf
>
> Regards,
> Ben
>
>
Regards,
Ben


> -- Forwarded message -
> From: CI System 
> Date: Sat, Jul 10, 2021 at 7:21 AM
> Subject: KDE CI: Administration » Dependency Build Applications
> stable-kf5-qt5 WindowsMSVCQt5.15 - Build # 52 - Still Failing!
> To: 
>
>
> *BUILD FAILURE*
> Build URL
> https://build.kde.org/job/Administration/job/Dependency%20Build%20Applications%20stable-kf5-qt5%20WindowsMSVCQt5.15/52/
> Project: Dependency Build Applications stable-kf5-qt5 WindowsMSVCQt5.15
> Date of build: Fri, 09 Jul 2021 18:58:42 +
> Build duration: 22 min and counting
> * CONSOLE OUTPUT *
> [...truncated 51900 lines...]
> [2021-07-09T19:20:03.112Z]
> C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
> warning C4005: 'Q_OS_WIN': macro redefinition
> [2021-07-09T19:20:03.112Z] ..\src\kdeui\kicon.cpp: note: see previous
> definition of 'Q_OS_WIN'
> [2021-07-09T19:20:03.112Z] [153/533] Building CXX object
> src\CMakeFiles\KF5KDELibs4Support.dir\kdeui\kdialogqueue.cpp.obj
> [2021-07-09T19:20:03.112Z]
> C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
> warning C4005: 'Q_OS_WIN': macro redefinition
> [2021-07-09T19:20:03.112Z] ..\src\kdeui\kdialogqueue.cpp: note: see
> previous definition of 'Q_OS_WIN'
> [2021-07-09T19:20:03.112Z] [154/533] Building CXX object
> src\CMakeFiles\KF5KDELibs4Support.dir\kdecore\ktzfiletimezone.cpp.obj
> [2021-07-09T19:20:03.112Z]
> C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
> warning C4005: 'Q_OS_WIN': macro redefinition
> [2021-07-09T19:20:03.112Z] ..\src\kdecore\ktzfiletimezone.cpp: note: see
> previous definition of 'Q_OS_WIN'
> [2021-07-09T19:20:03.112Z] [155/533] Building CXX object
> src\CMakeFiles\KF5KDELibs4Support.dir\kdeui\kapplication.cpp.obj
> [2021-07-09T19:20:03.112Z]
> C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
> warning C4005: 'Q_OS_WIN': macro redefinition
> [2021-07-09T19:20:03.112Z] ..\src\kdeui\kapplication.cpp: note: see
> previous definition of 'Q_OS_WIN'
> [2021-07-09T19:20:03.684Z] [156/533] Building CXX object
> src\CMakeFiles\KF5KDELibs4Support.dir\kdeui\kstringvalidator.cpp.obj
> [2021-07-09T19:20:03.684Z]
> C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
> warning C4005: 'Q_OS_WIN': macro redefinition
> [2021-07-09T19:20:03.684Z] ..\src\kdeui\kstringvalidator.cpp: note: see
> previous definition of 'Q_OS_WIN'
> [2021-07-09T19:20:03.684Z] [157/533] Building CXX object
> src\CMakeFiles\KF5KDELibs4Support.dir\kdeui\kfadewidgeteffect.cpp.obj
> [2021-07-09T19:20:03.684Z]
> C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
> warning C4005: 'Q_OS_WIN': macro redefinition
&

Fwd: KDE CI: Administration » Dependency Build Applications stable-kf5-qt5 WindowsMSVCQt5.15 - Build # 52 - Still Failing!

2021-07-09 Thread Ben Cooksley
Hi Ralf,

I'm afraid your previous fixes are insufficient - we are still failing with
a linking error.

The following is notice that I intend to back all changes out in kdewin and
restore the repository to 8503ac6e0e07099c4938cbfe60e3f5e25e2ec368 to
restore the correct function of the CI system in 24 hours time.

Regards,
Ben

-- Forwarded message -
From: CI System 
Date: Sat, Jul 10, 2021 at 7:21 AM
Subject: KDE CI: Administration » Dependency Build Applications
stable-kf5-qt5 WindowsMSVCQt5.15 - Build # 52 - Still Failing!
To: 


*BUILD FAILURE*
Build URL
https://build.kde.org/job/Administration/job/Dependency%20Build%20Applications%20stable-kf5-qt5%20WindowsMSVCQt5.15/52/
Project: Dependency Build Applications stable-kf5-qt5 WindowsMSVCQt5.15
Date of build: Fri, 09 Jul 2021 18:58:42 +
Build duration: 22 min and counting
* CONSOLE OUTPUT *
[...truncated 51900 lines...]
[2021-07-09T19:20:03.112Z]
C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
warning C4005: 'Q_OS_WIN': macro redefinition
[2021-07-09T19:20:03.112Z] ..\src\kdeui\kicon.cpp: note: see previous
definition of 'Q_OS_WIN'
[2021-07-09T19:20:03.112Z] [153/533] Building CXX object
src\CMakeFiles\KF5KDELibs4Support.dir\kdeui\kdialogqueue.cpp.obj
[2021-07-09T19:20:03.112Z]
C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
warning C4005: 'Q_OS_WIN': macro redefinition
[2021-07-09T19:20:03.112Z] ..\src\kdeui\kdialogqueue.cpp: note: see
previous definition of 'Q_OS_WIN'
[2021-07-09T19:20:03.112Z] [154/533] Building CXX object
src\CMakeFiles\KF5KDELibs4Support.dir\kdecore\ktzfiletimezone.cpp.obj
[2021-07-09T19:20:03.112Z]
C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
warning C4005: 'Q_OS_WIN': macro redefinition
[2021-07-09T19:20:03.112Z] ..\src\kdecore\ktzfiletimezone.cpp: note: see
previous definition of 'Q_OS_WIN'
[2021-07-09T19:20:03.112Z] [155/533] Building CXX object
src\CMakeFiles\KF5KDELibs4Support.dir\kdeui\kapplication.cpp.obj
[2021-07-09T19:20:03.112Z]
C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
warning C4005: 'Q_OS_WIN': macro redefinition
[2021-07-09T19:20:03.112Z] ..\src\kdeui\kapplication.cpp: note: see
previous definition of 'Q_OS_WIN'
[2021-07-09T19:20:03.684Z] [156/533] Building CXX object
src\CMakeFiles\KF5KDELibs4Support.dir\kdeui\kstringvalidator.cpp.obj
[2021-07-09T19:20:03.684Z]
C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
warning C4005: 'Q_OS_WIN': macro redefinition
[2021-07-09T19:20:03.684Z] ..\src\kdeui\kstringvalidator.cpp: note: see
previous definition of 'Q_OS_WIN'
[2021-07-09T19:20:03.684Z] [157/533] Building CXX object
src\CMakeFiles\KF5KDELibs4Support.dir\kdeui\kfadewidgeteffect.cpp.obj
[2021-07-09T19:20:03.684Z]
C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
warning C4005: 'Q_OS_WIN': macro redefinition
[2021-07-09T19:20:03.684Z] ..\src\kdeui\kfadewidgeteffect.cpp: note: see
previous definition of 'Q_OS_WIN'
[2021-07-09T19:20:03.945Z] [158/533] Building CXX object
src\CMakeFiles\KF5KDELibs4Support.dir\kdeui\kbuttongroup.cpp.obj
[2021-07-09T19:20:03.945Z]
C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
warning C4005: 'Q_OS_WIN': macro redefinition
[2021-07-09T19:20:03.945Z] ..\src\kdeui\kbuttongroup.cpp: note: see
previous definition of 'Q_OS_WIN'
[2021-07-09T19:20:04.205Z] [159/533] Building CXX object
src\CMakeFiles\KF5KDELibs4Support.dir\kdeui\kaction.cpp.obj
[2021-07-09T19:20:04.205Z]
C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
warning C4005: 'Q_OS_WIN': macro redefinition
[2021-07-09T19:20:04.205Z] ..\src\kdeui\kaction.cpp: note: see previous
definition of 'Q_OS_WIN'
[2021-07-09T19:20:04.205Z] [160/533] Building CXX object
src\CMakeFiles\KF5KDELibs4Support.dir\kdeui\ksessionmanager.cpp.obj
[2021-07-09T19:20:04.205Z]
C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
warning C4005: 'Q_OS_WIN': macro redefinition
[2021-07-09T19:20:04.205Z] ..\src\kdeui\ksessionmanager.cpp: note: see
previous definition of 'Q_OS_WIN'
[2021-07-09T19:20:04.205Z] [161/533] Building CXX object
src\CMakeFiles\KF5KDELibs4Support.dir\kdeui\kmenu.cpp.obj
[2021-07-09T19:20:04.205Z]
C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
warning C4005: 'Q_OS_WIN': macro redefinition
[2021-07-09T19:20:04.205Z] ..\src\kdeui\kmenu.cpp: note: see previous
definition of 'Q_OS_WIN'
[2021-07-09T19:20:04.466Z] [162/533] Building CXX object
src\CMakeFiles\KF5KDELibs4Support.dir\kdeui\kdialog.cpp.obj
[2021-07-09T19:20:04.466Z]
C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
warning C4005: 'Q_OS_WIN': macro redefinition
[2021-07-09T19:20:04.466Z] ..\src\kdeui\kdialog.cpp: 

Re: KDE CI: Administration » Dependency Build Extragear stable-kf5-qt5 WindowsMSVCQt5.15 - Build # 53 - Still Failing!

2021-07-09 Thread Ben Cooksley
On Sat, Jul 10, 2021 at 5:48 AM Ralf Habacker 
wrote:

> Am 08.07.21 um 20:17 schrieb Ben Cooksley:
>
> Hi Ben,
>

HI Ralf,

On Fri, Jul 9, 2021 at 1:28 AM Harald Sitter  wrote:
>
>> comparing the failing build to the last successful one it appears kdewin
>> no longer installs arpa/inet.h which lead me to this commit
>> https://invent.kde.org/packaging/kdewin/-/commit/25a254ab3ab0369e235db7418ac9b95226445a38
>> which seems a bit folly because it makes the installation conditional on
>> inet_ntop being available with a **different** header rendering the change
>> not particularly backwards compatible because of course kdelibs4support is
>> using the old header. One way forward would certainly be to port
>> kdelibs4support to an ifdef q_os_win32 that then includes winsock2 instead
>> of the posix headers. Ralf will probably know best what to do.
>>
>
> Ralf, please note that this needs to be fixed with urgency as it is
> preventing the Windows CI system from being maintainable.
>
> I fixed that issue yesterday, tagged a new release 0.6.6 (see
> https://invent.kde.org/packaging/kdewin/-/tags/v0.6.6) and got successful
> builds with that (see for example
> https://build.opensuse.org/package/live_build_log/windows:mingw:win64/mingw64-kdelibs4support/openSUSE_Leap_15.2/x86_64
> )
>

Those builds are on cross-compiled mingw, which is a completely different
environment.

If these changes have not reached KDE CI, it could be because the cached
> kdewin binary package is outdated and needs to be renewed. But I can't tell
> you how to do that.
>

The Dependency Build jobs always start from scratch - including the build
of 'kdewin'.
I have now started fresh runs of those - hopefully they complete
successfully.

Please be more careful when making changes such as these as that project is
essentially only for life support of kdelibs4support at this time (nothing
else uses it)

Regards
> Ralf
>

Regards,
Ben

Regards,
> Ben
>
>
>> On Thu, Jul 8, 2021 at 8:46 AM Ben Cooksley  wrote:
>>
>>> Hi all,
>>>
>>> Please find below a build log from a Windows dependency build job which
>>> is currently failing in kdelibs4support.
>>>
>>> Any ideas what may have caused this?
>>>
>>> Thanks,
>>> Ben
>>>
>>> -- Forwarded message -
>>> From: CI System 
>>> Date: Thu, Jul 8, 2021 at 8:46 AM
>>> Subject: KDE CI: Administration » Dependency Build Extragear
>>> stable-kf5-qt5 WindowsMSVCQt5.15 - Build # 53 - Still Failing!
>>> To: 
>>>
>>>
>>> *BUILD FAILURE*
>>> Build URL
>>> https://build.kde.org/job/Administration/job/Dependency%20Build%20Extragear%20stable-kf5-qt5%20WindowsMSVCQt5.15/53/
>>> Project: Dependency Build Extragear stable-kf5-qt5 WindowsMSVCQt5.15
>>> Date of build: Wed, 07 Jul 2021 18:00:04 +
>>> Build duration: 2 hr 46 min and counting
>>> * CONSOLE OUTPUT *
>>> [...truncated 53095 lines...]
>>> [2021-07-07T20:42:36.730Z] [84/533] Building CXX object
>>> src\CMakeFiles\KF5KDELibs4Support.dir\kdecore\kcurrencycode.cpp.obj
>>> [2021-07-07T20:42:36.730Z]
>>> C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
>>> warning C4005: 'Q_OS_WIN': macro redefinition
>>> [2021-07-07T20:42:36.730Z] ..\src\kdecore\kcurrencycode.cpp: note: see
>>> previous definition of 'Q_OS_WIN'
>>> [2021-07-07T20:42:37.710Z] [85/533] Building CXX object
>>> src\CMakeFiles\KF5KDELibs4Support.dir\kdecore\kdebugdbusiface.cpp.obj
>>> [2021-07-07T20:42:37.710Z]
>>> C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
>>> warning C4005: 'Q_OS_WIN': macro redefinition
>>> [2021-07-07T20:42:37.710Z] ..\src\kdecore\kdebugdbusiface.cpp: note: see
>>> previous definition of 'Q_OS_WIN'
>>> [2021-07-07T20:42:37.974Z] [86/533] Building CXX object
>>> src\CMakeFiles\KF5KDELibs4Support.dir\kdecore\kdebug.cpp.obj
>>> [2021-07-07T20:42:37.974Z]
>>> C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
>>> warning C4005: 'Q_OS_WIN': macro redefinition
>>> [2021-07-07T20:42:37.974Z] ..\src\kdecore\kdebug.cpp: note: see previous
>>> definition of 'Q_OS_WIN'
>>> [2021-07-07T20:42:39.393Z] [87/533] Building CXX object
>>> src\CMakeFiles\KF5KDELibs4Support.dir\kdecore\klibrary.cpp.obj
>>> [2021-07-07T20:42:39.393Z]
>>> C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
>>> warning C4005: 'Q_OS_WIN': macr

Re: KDE CI: Administration » Dependency Build Extragear stable-kf5-qt5 WindowsMSVCQt5.15 - Build # 53 - Still Failing!

2021-07-08 Thread Ben Cooksley
On Fri, Jul 9, 2021 at 1:28 AM Harald Sitter  wrote:

> comparing the failing build to the last successful one it appears kdewin
> no longer installs arpa/inet.h which lead me to this commit
> https://invent.kde.org/packaging/kdewin/-/commit/25a254ab3ab0369e235db7418ac9b95226445a38
> which seems a bit folly because it makes the installation conditional on
> inet_ntop being available with a **different** header rendering the change
> not particularly backwards compatible because of course kdelibs4support is
> using the old header. One way forward would certainly be to port
> kdelibs4support to an ifdef q_os_win32 that then includes winsock2 instead
> of the posix headers. Ralf will probably know best what to do.
>

Ralf, please note that this needs to be fixed with urgency as it is
preventing the Windows CI system from being maintainable.

Regards,
Ben


> On Thu, Jul 8, 2021 at 8:46 AM Ben Cooksley  wrote:
>
>> Hi all,
>>
>> Please find below a build log from a Windows dependency build job which
>> is currently failing in kdelibs4support.
>>
>> Any ideas what may have caused this?
>>
>> Thanks,
>> Ben
>>
>> -- Forwarded message -
>> From: CI System 
>> Date: Thu, Jul 8, 2021 at 8:46 AM
>> Subject: KDE CI: Administration » Dependency Build Extragear
>> stable-kf5-qt5 WindowsMSVCQt5.15 - Build # 53 - Still Failing!
>> To: 
>>
>>
>> *BUILD FAILURE*
>> Build URL
>> https://build.kde.org/job/Administration/job/Dependency%20Build%20Extragear%20stable-kf5-qt5%20WindowsMSVCQt5.15/53/
>> Project: Dependency Build Extragear stable-kf5-qt5 WindowsMSVCQt5.15
>> Date of build: Wed, 07 Jul 2021 18:00:04 +
>> Build duration: 2 hr 46 min and counting
>> * CONSOLE OUTPUT *
>> [...truncated 53095 lines...]
>> [2021-07-07T20:42:36.730Z] [84/533] Building CXX object
>> src\CMakeFiles\KF5KDELibs4Support.dir\kdecore\kcurrencycode.cpp.obj
>> [2021-07-07T20:42:36.730Z]
>> C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
>> warning C4005: 'Q_OS_WIN': macro redefinition
>> [2021-07-07T20:42:36.730Z] ..\src\kdecore\kcurrencycode.cpp: note: see
>> previous definition of 'Q_OS_WIN'
>> [2021-07-07T20:42:37.710Z] [85/533] Building CXX object
>> src\CMakeFiles\KF5KDELibs4Support.dir\kdecore\kdebugdbusiface.cpp.obj
>> [2021-07-07T20:42:37.710Z]
>> C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
>> warning C4005: 'Q_OS_WIN': macro redefinition
>> [2021-07-07T20:42:37.710Z] ..\src\kdecore\kdebugdbusiface.cpp: note: see
>> previous definition of 'Q_OS_WIN'
>> [2021-07-07T20:42:37.974Z] [86/533] Building CXX object
>> src\CMakeFiles\KF5KDELibs4Support.dir\kdecore\kdebug.cpp.obj
>> [2021-07-07T20:42:37.974Z]
>> C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
>> warning C4005: 'Q_OS_WIN': macro redefinition
>> [2021-07-07T20:42:37.974Z] ..\src\kdecore\kdebug.cpp: note: see previous
>> definition of 'Q_OS_WIN'
>> [2021-07-07T20:42:39.393Z] [87/533] Building CXX object
>> src\CMakeFiles\KF5KDELibs4Support.dir\kdecore\klibrary.cpp.obj
>> [2021-07-07T20:42:39.393Z]
>> C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
>> warning C4005: 'Q_OS_WIN': macro redefinition
>> [2021-07-07T20:42:39.394Z] ..\src\kdecore\klibrary.cpp: note: see
>> previous definition of 'Q_OS_WIN'
>> [2021-07-07T20:42:39.394Z] [88/533] Building CXX object
>> src\CMakeFiles\KF5KDELibs4Support.dir\kdecore\k4aboutdata.cpp.obj
>> [2021-07-07T20:42:39.394Z]
>> C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
>> warning C4005: 'Q_OS_WIN': macro redefinition
>> [2021-07-07T20:42:39.394Z] ..\src\kdecore\k4aboutdata.cpp: note: see
>> previous definition of 'Q_OS_WIN'
>> [2021-07-07T20:42:39.394Z] KDE5 FIXME: This code must be replaced by
>> something with KLocalizedString
>> [2021-07-07T20:42:39.394Z] KDE5 TODO: What about this code ?
>> [2021-07-07T20:42:40.178Z] [89/533] Building CXX object
>> src\CMakeFiles\KF5KDELibs4Support.dir\kdecore\kkernel_win.cpp.obj
>> [2021-07-07T20:42:40.178Z]
>> C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
>> warning C4005: 'Q_OS_WIN': macro redefinition
>> [2021-07-07T20:42:40.178Z] ..\src\kdecore\kkernel_win.cpp: note: see
>> previous definition of 'Q_OS_WIN'
>> [2021-07-07T20:42:40.459Z] [90/533] Building CXX object
>> src\CMakeFiles\KF5KDELibs4Support.dir\kdecore\ktemporaryfile.cpp.obj
>> [2021-07-07T

Fwd: KDE CI: Administration » Dependency Build Extragear stable-kf5-qt5 WindowsMSVCQt5.15 - Build # 53 - Still Failing!

2021-07-08 Thread Ben Cooksley
Hi all,

Please find below a build log from a Windows dependency build job which is
currently failing in kdelibs4support.

Any ideas what may have caused this?

Thanks,
Ben

-- Forwarded message -
From: CI System 
Date: Thu, Jul 8, 2021 at 8:46 AM
Subject: KDE CI: Administration » Dependency Build Extragear stable-kf5-qt5
WindowsMSVCQt5.15 - Build # 53 - Still Failing!
To: 


*BUILD FAILURE*
Build URL
https://build.kde.org/job/Administration/job/Dependency%20Build%20Extragear%20stable-kf5-qt5%20WindowsMSVCQt5.15/53/
Project: Dependency Build Extragear stable-kf5-qt5 WindowsMSVCQt5.15
Date of build: Wed, 07 Jul 2021 18:00:04 +
Build duration: 2 hr 46 min and counting
* CONSOLE OUTPUT *
[...truncated 53095 lines...]
[2021-07-07T20:42:36.730Z] [84/533] Building CXX object
src\CMakeFiles\KF5KDELibs4Support.dir\kdecore\kcurrencycode.cpp.obj
[2021-07-07T20:42:36.730Z]
C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
warning C4005: 'Q_OS_WIN': macro redefinition
[2021-07-07T20:42:36.730Z] ..\src\kdecore\kcurrencycode.cpp: note: see
previous definition of 'Q_OS_WIN'
[2021-07-07T20:42:37.710Z] [85/533] Building CXX object
src\CMakeFiles\KF5KDELibs4Support.dir\kdecore\kdebugdbusiface.cpp.obj
[2021-07-07T20:42:37.710Z]
C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
warning C4005: 'Q_OS_WIN': macro redefinition
[2021-07-07T20:42:37.710Z] ..\src\kdecore\kdebugdbusiface.cpp: note: see
previous definition of 'Q_OS_WIN'
[2021-07-07T20:42:37.974Z] [86/533] Building CXX object
src\CMakeFiles\KF5KDELibs4Support.dir\kdecore\kdebug.cpp.obj
[2021-07-07T20:42:37.974Z]
C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
warning C4005: 'Q_OS_WIN': macro redefinition
[2021-07-07T20:42:37.974Z] ..\src\kdecore\kdebug.cpp: note: see previous
definition of 'Q_OS_WIN'
[2021-07-07T20:42:39.393Z] [87/533] Building CXX object
src\CMakeFiles\KF5KDELibs4Support.dir\kdecore\klibrary.cpp.obj
[2021-07-07T20:42:39.393Z]
C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
warning C4005: 'Q_OS_WIN': macro redefinition
[2021-07-07T20:42:39.394Z] ..\src\kdecore\klibrary.cpp: note: see previous
definition of 'Q_OS_WIN'
[2021-07-07T20:42:39.394Z] [88/533] Building CXX object
src\CMakeFiles\KF5KDELibs4Support.dir\kdecore\k4aboutdata.cpp.obj
[2021-07-07T20:42:39.394Z]
C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
warning C4005: 'Q_OS_WIN': macro redefinition
[2021-07-07T20:42:39.394Z] ..\src\kdecore\k4aboutdata.cpp: note: see
previous definition of 'Q_OS_WIN'
[2021-07-07T20:42:39.394Z] KDE5 FIXME: This code must be replaced by
something with KLocalizedString
[2021-07-07T20:42:39.394Z] KDE5 TODO: What about this code ?
[2021-07-07T20:42:40.178Z] [89/533] Building CXX object
src\CMakeFiles\KF5KDELibs4Support.dir\kdecore\kkernel_win.cpp.obj
[2021-07-07T20:42:40.178Z]
C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
warning C4005: 'Q_OS_WIN': macro redefinition
[2021-07-07T20:42:40.178Z] ..\src\kdecore\kkernel_win.cpp: note: see
previous definition of 'Q_OS_WIN'
[2021-07-07T20:42:40.459Z] [90/533] Building CXX object
src\CMakeFiles\KF5KDELibs4Support.dir\kdecore\ktemporaryfile.cpp.obj
[2021-07-07T20:42:40.459Z]
C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
warning C4005: 'Q_OS_WIN': macro redefinition
[2021-07-07T20:42:40.459Z] ..\src\kdecore\ktemporaryfile.cpp: note: see
previous definition of 'Q_OS_WIN'
[2021-07-07T20:42:41.888Z] [91/533] Building CXX object
src\CMakeFiles\KF5KDELibs4Support.dir\KF5KDELibs4Support_autogen\mocs_compilation.cpp.obj
[2021-07-07T20:42:41.888Z]
C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
warning C4005: 'Q_OS_WIN': macro redefinition
[2021-07-07T20:42:41.888Z]
src\KF5KDELibs4Support_autogen\mocs_compilation.cpp: note: see previous
definition of 'Q_OS_WIN'
[2021-07-07T20:42:41.888Z] Port to Qt5 native filter
[2021-07-07T20:42:41.888Z] [92/533] Building CXX object
src\CMakeFiles\KF5KDELibs4Support.dir\kdecore\ktempdir.cpp.obj
[2021-07-07T20:42:41.888Z]
C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
warning C4005: 'Q_OS_WIN': macro redefinition
[2021-07-07T20:42:41.888Z] ..\src\kdecore\ktempdir.cpp: note: see previous
definition of 'Q_OS_WIN'
[2021-07-07T20:42:41.888Z] [93/533] Building CXX object
src\CMakeFiles\KF5KDELibs4Support.dir\kdecore\kmd5.cpp.obj
[2021-07-07T20:42:41.888Z]
C:\Craft\CI-Qt515\windows-msvc2019_64-cl-debug\include\qt5\QtCore/qsystemdetection.h(189):
warning C4005: 'Q_OS_WIN': macro redefinition
[2021-07-07T20:42:41.888Z] ..\src\kdecore\kmd5.cpp: note: see previous
definition of 'Q_OS_WIN'
[2021-07-07T20:42:42.850Z] [94/533] Building CXX object

Re: Proposal: branch ECM into ECM6 for KF6

2021-07-06 Thread Ben Cooksley
On Tue, Jul 6, 2021 at 11:06 AM Friedrich W. H. Kossebau 
wrote:

> Hi,
>
> during the discussion of the MR for a KDEInstallDirs6 variant I learned
> that
> there is no clear idea yet how ECM should be be released once there are
> both
> KF5 and KF6. And more, it seems so far for ECM there is no version-branch
> planned, but instead would it have to support both Qt5/KF5 and Qt6/KF6 at
> the
> same time for the longer future. With no defined plan when support for
> Qt5/KF6
> should be dropped.
>
> Which also brings a related question: when should all the deprecated ECM
> modules and any other deprecated things there be dropped that have
> accumulated
> in the last years since ECM 1.0? While CMake itself has it's policy
> mechanism
> though so far I think they still keep backward compatibility for anything,
> just doing things different by policy default? And might only dump
> backward-
> compat support really on the next major version?
>
> So when would ECM dump backward-compat support for anything? And, due to
> not
> being co-installable to older version of itself as of now, risk breaking
> builds of older Qt5/KF5 software or forcing the use of dir links
> switching?
> Hello Debian and other LTS systems and trying to develop with more recent
> software at the same time.
>
> IMHO ECM should rather be branched into a major-version-postfixed variant
> as
> well for KF6 times, e.g. "ECM6". And allow a clear cut of legacy things,
> as
> well as some redesign where considered useful. And perhaps even be
> renamed,
> into KFCM, KDE Frameworks CMake Modules (yes, I see the unfortune brand
> name
> conflict while typing, so perhaps something else or no change at all)
>
> Having a version postfix in the name should allow co-installability. Yes,
> there will be some duplication, but Qt5 and Qt6 or Python 2 and Python3
> libraries also duplicate lots of logic ;) and its not that the ECM modules
> are
> MB of installation.
> Maintenance to ensure backward ports of bug fixes between ECM & ECM6 is an
> issue, but might be less trouble than having to support Qt5/KF5 & Qt6/KF6
> at
> the same time (including any cached variables on switching build deps) and
> then the challenge when to annoy some people by dropping legacy support
> they
> rely on.
>
> And those who want to support builds against Qt5/KF5 & Qt6/KF6 from the
> same
> code (which I personally in general recommend against, compromises needed
> just
> hurt), they then would have to do a bit more code branches in cmake code
> as
> well. But they are fine with that in the C++ code, so should they be in
> the
> cmake code (or just copy over any newer CMake modules temporarily if more
> simple).
>
> Also think of test setup - having to test both against Qt5 & Qt6 might be
> more
> troublesome, locally and on CI. More easy if there is just one Qt flavour
> to
> keep in mind.
>

The CI system uses the concept of "platforms" which means a given
combination of Operating System, Compiler and Qt version.
As such, it is impossible (and will remain impossible) to test against two
Qt versions in the same build job.

(Coinstallability doesn't matter here, builds have a terrible habit of
getting mixed/contaminated between the two which just creates trouble -
those who work on Android will be familiar with how a rebuild is required
for ECM changes to be properly picked up)


> My 2 cents as old cmake code writer (who might not be involved in the near
> future)
>
> Cheers
> Friedrich
>

Cheers,
Ben


Upcoming update to GItLab 14.0

2021-06-29 Thread Ben Cooksley
Hi all,

As some of you will have noticed, GitLab recently released 14.0, which
includes a number of new features and other changes. Details on these can
be found at
https://about.gitlab.com/releases/2021/06/22/gitlab-14-0-released/

While most of these won't have an impact on us, there are a number of user
interface changes which may impact your workflows.

At this time we plan to perform the update over the weekend.

Please let us know if you have any questions.

Many thanks,
Ben Cooksley
KDE Sysadmin


Re: Notice of withdrawal of CI services: KDevelop and KDE Connect

2021-06-18 Thread Ben Cooksley
On Fri, Jun 18, 2021 at 7:33 AM Milian Wolff  wrote:

> On Donnerstag, 17. Juni 2021 11:25:15 CEST Ben Cooksley wrote:
> > On Thu, Jun 17, 2021 at 8:44 AM Milian Wolff  wrote:
> > > On Mittwoch, 16. Juni 2021 20:28:25 CEST Ben Cooksley wrote:
> > > > Hi all,
> > > >
> > > >
> > > > The following is notice that I intend to withdraw CI services from
> the
> > > > following two KDE projects due to faults in their code or build
> system
> > > > which are having a significant adverse impact on the CI system and
> > > > negatively impacting on other projects:
> > > >
> > > > - KDevelop
> > > > - KDE Connect
> > > >
> > > > This withdrawal will be applied to all platforms.
> > > >
> > > > In the case of KDevelop, it has a series of unit tests which on
> FreeBSD
> > > > cause gdb to hang and consume an entire CPU core indefinitely. This
> > > > slows
> > > > down builds for all other projects using that CI server, and also
> > >
> > > prevents
> > >
> > > > KWin tests from proceeding - completely blocking it's jobs. This
> fault
> > > > is
> > > > in the debuggee_slow family of tests.
> > >
> > > Hey Ben,
> >
> > Hi Milian,
> >
> > > first time I hear of this issue. I simply don't have the capacity to
> track
> > > kdevelop CI mails, so if anything really bad happens, I would really
> > > appreciate if anyone who notices that sents a mail to the kdevelop
> mailing
> > > list so we can act on it. I still use kdevelop daily, but don't put any
> > > other
> > > work in it currently due to lack of time...
> >
> > The issue in question was noted on #kdevelop (prior to the whole Freenode
> > meltdown) on a few occasions.
>
> Even before that meltdown, I only looked at this when I was directly
> pinged.
> Please prefer to use the mailing lists to get our attention.
>

Certainly.


>
> > > That said: can we just disable it on the FreeBSD platform, if that's
> the
> > > only
> > > one that exhibits this failure? Alternatively I'm obviously fine with
> > > skipping
> > > that test on that platform, can you give me a link to a failure please
> so
> > > I
> > > can see which tests exactly are affected? Then I'll add the QSKIP +
> > > platform
> > > ifdef checks to skip it on freebsd.
> >
> > I'm afraid I only have records of the hung test processes (and the gdb
> > processes above them using an entire CPU core indefinitely each).
> >
> > jenkins60635   0.0  0.0   12120   2256  0  TXs+ 02:44 0:00.00
> > /usr/home/jenkins/workspace/KDevelop/kdevelop/kf5-qt5
> >
> FreeBSDQt5.15/build/plugins/debuggercommon/tests/debuggees/debuggee_debugees
> > low (debuggee_debugeeslo)
> > jenkins74167   0.0  0.0   12120   2260  1  TXs+ 02:55 0:00.00
> > /usr/home/jenkins/workspace/KDevelop/kdevelop/kf5-qt5
> >
> FreeBSDQt5.15/build/plugins/debuggercommon/tests/debuggees/debuggee_debugees
> > low (debuggee_debugeeslo)
> >
> > Hopefully that is enough to identify the tests that are broken?
>
> I have now added code to skip the tests on FreeBSD wherever debuggeeslow
> is
> used. Unsure which test of the many is actually triggering it, but I have
> no
> time to invest on this either.
>
> So, is that acceptable for you for now?
>

That should be perfectly fine - thanks for skipping those tests on FreeBSD.

Please note that it is possible the issue also occurs on Linux however
we've no way of knowing if this is the case as the Docker containers are
used for one build only and then thrown away (which is also why it isn't an
issue there if it is happening, as the process of throwing away the
container will kill off any leftover processes)

Cheers,
Ben


> Cheers
>
> --
> Milian Wolff
> m...@milianw.de
> http://milianw.de


Re: Notice of withdrawal of CI services: KDevelop and KDE Connect

2021-06-17 Thread Ben Cooksley
On Thu, Jun 17, 2021 at 7:26 AM Avamander  wrote:

> Hi,
>

Hi there,


>
> This is indeed an issue that has occurred previously, and actually
> I've written to you, Ben, once before about this. Now I'm asking again, why
> is it necessary to email *FIVE* mailing lists for issues that are
> primarily solved by a much much smaller subgroup of people? Maybe send it
> to a few C++ lists as well, maybe someone'll jump out and fix the CI 浪
>

According to my email archives we've not previously corresponded - and the
name "Avamander" is certainly new to me.

The five lists in question were chosen deliberately for the following
reasons:
1) kdeconn...@kde.org as the matter related to KDE Connect
2) kdevelop-de...@kde.org as the matter related to KDevelop
3) kde-frameworks-devel@kde.org as the email explained why Frameworks had
experienced a substantial number of Windows CI failures lately
4) kde-de...@kde.org as the email explained to various Extragear projects
why they had experienced Windows CI failures as well
5) sysad...@kde.org as the matter related to our infrastructure


> On a very related topic, for the third time, send the kdeconnect CI status
> emails to the actual developers and contributors for whom they are
> actionable, not the entire kdeconnect mailing list. It's just noise for the
> majority of non-code contributors, non-code contributors that might be able
> to help with support questions but who are very likely ignoring the list
> due to horrifically bad SNR.
>

Please direct your complaints regarding the content and moderation of the
kdeconn...@kde.org to the list administrators - kdeconnect-ow...@kde.org -
in the first instance or raise a thread on that mailing list. With regards
to the CI status emails, these were setup at the request of the KDE Connect
developers in https://phabricator.kde.org/D13794


>
>
> Have an extra day,
> Avamander
>

Regards,
Ben


>
> On Wed, Jun 16, 2021 at 9:29 PM Ben Cooksley  wrote:
>
>> Hi all,
>>
>> The following is notice that I intend to withdraw CI services from the
>> following two KDE projects due to faults in their code or build system
>> which are having a significant adverse impact on the CI system and
>> negatively impacting on other projects:
>>
>> - KDevelop
>> - KDE Connect
>>
>> This withdrawal will be applied to all platforms.
>>
>> In the case of KDevelop, it has a series of unit tests which on FreeBSD
>> cause gdb to hang and consume an entire CPU core indefinitely. This slows
>> down builds for all other projects using that CI server, and also prevents
>> KWin tests from proceeding - completely blocking it's jobs. This fault is
>> in the debuggee_slow family of tests.
>>
>> As this issue has persisted for some time now despite being mentioned, I
>> require that the family of tests in question be disabled across all
>> platforms.
>>
>> In the case of KDE Connect, as part of it's configure process it runs an
>> external command that results in dbus-daemon being launched. This process
>> persists following the build and would only be cleaned up by our tooling
>> that runs tests. Should the compilation fail (as it does currently) it will
>> result in the CI builder being broken - which is why we have had so many
>> Windows builds fail lately.
>>
>> As this is an issue that has occurred previously, I require that the
>> configure time check which is launching dbus-daemon to be expunged from KDE
>> Connect.
>>
>> As a reminder to all projects, running commands that interact with
>> dbus-daemon or kdeinit is not permitted during configure or build time.
>>
>> Regards,
>> Ben Cooksley
>> KDE Sysadmin
>>
>


  1   2   3   4   5   6   7   8   9   >