This is great work! I love the dashboard. I've started doing a few repos
changing them to:
Options:
require-passing-tests-on:['@all']
Regards,
Justin
On 11/5/26 22:42, Kristen McWilliam wrote:
Hello, and welcome to the new Plasma CI Health Check!
You may be familiar with the "KDE Gear projects with failing CI"
project/emails that Albert has been doing for some time — I will be
doing similar for Plasma, to ensure that our CI is healthy and improve
our testing strategy.
We'll be looking at a few interesting metrics, for example whether:
• CI is passing or failing on master
If CI is failing on master its usually a bad sign, and something we
want to fix ASAP.
• Tests are being automatically retried
Having the CI configured to auto retry tests is usually a sign of a
workaround for flaky tests, and flaky tests are bad tests; at best
they are frustrating and a waste of CI resources, at worst they give a
false sense of security that code is actually reliable. A test should
either pass or fail — if it doesn't do so reliably it should be fixed
or removed.
In the .kde-ci.yml this will look something like:
```yaml
Options:
ctest-arguments: '--repeat until-pass:5'
```
We should prefer to remove this option and fix the tests.
• Tests are configured as required passing for the repo
With this enabled, if tests fail in the CI the associated merge
request will show this as a nice obvious warning; if it isn't enabled,
the MR will report the pipeline is completely green — not good!
We definitely want it to be obvious if the tests failed, otherwise
people are likely to merge it without even being aware there is an issue.
In the .kde-ci.yml this will look like:
```yaml
Options:
require-passing-tests-on: ['@all']
```
Knowing if the tests are passing or failing is super important, so we
need to work towards this being enabled for all our repos.
---------------------------
There will be additional things to track to assess our CI and test
health, but this is a good place to start.
Here are our initial metrics:
• 86 monitored repos
• 10 failing CI in master
• 3 configured for auto-repeating tests
• 14% of repos configured to require passing tests
Failing CI:
• plasma/qqc2-breeze-style
• plasma/plasma-desktop
• plasma/kmenuedit
• plasma/discover
• plasma/lightdm-kde-greeter
• plasma/kde-gtk-config
• plasma/plasma-sdk
• plasma/plasma-systemmonitor
• plasma/latte-dock
• plasma-mobile/spacebar
Auto-retrying tests:
• kwin-x11
• kwin
• plasma-desktop
I hacked together a simple dashboard to help visualize the state of
things, which also includes links to the relevant CI run for each
repo; if interested in checking, it lives here for the moment:
https://ci-healthcheck-5e5c80.local-kde.org/
Thank you for helping to improve the reliability of our software, and
reducing wasted CI resources.
