Hi, > > Thanks, that's exactly how I am thinking. If developers are not quick > > to fix or disable failing jobs, they will develop "alert fatigue" and > > stop caring when the CI is red. The longer it remains red, the less > > likely it is to ever turn green again. > > Do you think this is perhaps a bit too strong? "alert fatigue" might > affect some people, but not everyone is the same. And it could apply to > bug reports as well: do large numbers of open bug reports indicate poor > quality and that the package should be kept out of testing longer?
Alert fatigue is a well known phenomenon. The purpose of a CI system is to be 99% green. In the 1% of cases where it fails, it should cause development to stop so developers can research why, and fix a regression before it goes into production or before new feature development continues. The number of bugs is not a testing system by itself nor does tracking bug counts directly prevent regressions from propagating. > making problems, like failing tests, visible is also a good thing - > fixing failing tests should be a good way to get new contributions. My point is specifically that the main|master|debian/* branch must be kept "green", and in particular the very commit that gets tagged and uploaded to Debian should have had the Salsa CI pass first (if CI is being used). Otherwise the point of the CI is moot. Look for example at https://salsa.debian.org/pkg-voip-team/rtpengine/-/commits/master - the CI in all recent uploads failed and the developer ignored it, and the autopkgtest failure went into Debian. If the CI had been passing earlier the regression would have been obvious. Now it wasn't because "keeping it green" wasn't the practice in this example. Sure, fixing failing tests is a good way to get new contributors. But those same contributors might prefer working on packages where the mainline is green and the work to fix or develop something is a separate thing you enable/add separately. The package's CI should not serve as a reminder that something is broken. Bug reports seem like a much better place for that. Bug reports also have other benefits like ownership and history trail that make them much better suited at tracking work items than broken tests on the package mainline branch. I strongly recommend everyone keep the CI green, and use bug reports or other systems to record issues or TODO items. We can see that Salsa CI usage is increasing at https://trends.debian.net/#continuous-integration-on-salsa but the stats from vcswatch are a bit worrying, telling that out of the 25 thousand packages or so that use Salsa CI about 5 thousand were uploaded to Debian *despite* failing CI: udd=> WITH latest AS ( SELECT source, ci_status FROM vcswatch ORDER BY last_scan DESC NULLS LAST LIMIT 1000000 ) SELECT COUNT(*) AS total_packages, COUNT(DISTINCT source) AS unique_packages, COUNT(*) FILTER (WHERE ci_status = 'success') AS passing_ci, COUNT(*) FILTER (WHERE ci_status <> 'success' AND ci_status IS NOT NULL) AS failing_ci, COUNT(*) FILTER (WHERE ci_status IS NULL) AS no_ci_status FROM latest; total_packages | unique_packages | passing_ci | failing_ci | no_ci_status ----------------+-----------------+------------+------------+-------------- 40405 | 40405 | 19222 | 5318 | 15865 This seems wasteful and counterproductive as I wrote in the three points in my original message. I wish people would consider turning off the CI if they don't care about looking at the results. That would basically increase the CI capacity by 25% for packages where CI is maintained and properly used to catch regressions.
