This is great, thanks so much for taking initiative to do it!
Nate
On 5/11/26 7:12 AM, Kristen McWilliam wrote:
Hello, and welcome to the new Plasma CI Health Check!
You may be familiar with the "KDE Gear projects with failing CI"
project/emails that Albert has been doing for some time — I will be
doing similar for Plasma, to ensure that our CI is healthy and improve
our testing strategy.
We'll be looking at a few interesting metrics, for example whether:
• CI is passing or failing on master
If CI is failing on master its usually a bad sign, and something we want
to fix ASAP.
• Tests are being automatically retried
Having the CI configured to auto retry tests is usually a sign of a
workaround for flaky tests, and flaky tests are bad tests; at best they
are frustrating and a waste of CI resources, at worst they give a false
sense of security that code is actually reliable. A test should either
pass or fail — if it doesn't do so reliably it should be fixed or removed.
In the .kde-ci.yml this will look something like:
```yaml
Options:
ctest-arguments: '--repeat until-pass:5'
```
We should prefer to remove this option and fix the tests.
• Tests are configured as required passing for the repo
With this enabled, if tests fail in the CI the associated merge request
will show this as a nice obvious warning; if it isn't enabled, the MR
will report the pipeline is completely green — not good!
We definitely want it to be obvious if the tests failed, otherwise
people are likely to merge it without even being aware there is an issue.
In the .kde-ci.yml this will look like:
```yaml
Options:
require-passing-tests-on: ['@all']
```
Knowing if the tests are passing or failing is super important, so we
need to work towards this being enabled for all our repos.
---------------------------
There will be additional things to track to assess our CI and test
health, but this is a good place to start.
Here are our initial metrics:
• 86 monitored repos
• 10 failing CI in master
• 3 configured for auto-repeating tests
• 14% of repos configured to require passing tests
Failing CI:
• plasma/qqc2-breeze-style
• plasma/plasma-desktop
• plasma/kmenuedit
• plasma/discover
• plasma/lightdm-kde-greeter
• plasma/kde-gtk-config
• plasma/plasma-sdk
• plasma/plasma-systemmonitor
• plasma/latte-dock
• plasma-mobile/spacebar
Auto-retrying tests:
• kwin-x11
• kwin
• plasma-desktop
I hacked together a simple dashboard to help visualize the state of
things, which also includes links to the relevant CI run for each repo;
if interested in checking, it lives here for the moment:
https://ci-healthcheck-5e5c80.local-kde.org/
Thank you for helping to improve the reliability of our software, and
reducing wasted CI resources.
