Hello Dmitry, Thank you for raising this topic. Without a good, reliable CI, life gets harder. In that regard, pre-ci.cassandra.apache.org was very convenient to use.
I’m getting used to running tests on my own hardware. Here’s a brief overview on how it’s going. The CI workload appears to be primarily CPU-bound rather than memory-bound (8 cores, 16 threads, 96 GB RAM), and running the full pre-commit test suite for a single JDK version takes about ~9 hours. I also had to write a simple orchestrator [1] to balance the load evenly and collect results for a summary. Here is how the look like: uv run cassandra-remote-ci status -H ubuntu run 20261019-2006-cassandra-5182b2b-jdk17 phase done, unit inactive elapsed 8h32m jobs 196 of 196 done, 1 crashed GROUP JOBS CLASSES TESTS FAIL ERR SKIP ================== ======= =========== ============== ==== === ==== fqltool-test 1/1 3 27/~27 0 0 0 sstableloader-test 1/1 1 13/~13 0 0 0 test 20/20 1,738/1,738 17,188/~17,188 0 1 91 test-latest 20/20 1,738/1,738 17,167/~17,167 0 0 262 stress-test 1/1 8 22/~22 0 0 0 test-burn 5/5 11/11 59/~59 0 0 1 jvm-dtest 16/16 441/441 2,908/~2,908 4 2 53 dtest 64/64 64 1,107/1,107 1 0 276 dtest-latest 64/64 64 1,106/1,106 0 0 274 microbench-test 4/4 0 0 0 0 0 ------------------ ------- ----------- -------------- ---- --- ---- total 196/196 3,928/3,928 39,597 5 3 957 [1] https://github.com/Mmuzaf/cassandra-remote-ci#example On Tue, 20 Oct 2026 at 14:59, Dmitry Konstantinov <[email protected]> wrote: > Hi, I think there's one thing that makes us unhappy, especially for new > contributors to the project: the health of our CI. We probably see it green > about as often as a solar eclipse. :-) > > It wastes time and makes it harder and more frustrating to check CI > results for patches and to plan releases with confidence. > > We've also developed a kind of broken window mentality towards CI > failures: we tend to ignore them, even though some of them may actually > highlight real issues. > > I've already done some work to reduce the number of failing tests, and now > I'm going to invest a bit more time in this area. > > My current goal is to get the 6.0 CI running without test failures, at > least for a while. > 1. Fix JMH test reporting (done). For a few days, our CI was failing > because of a JMH test failure that actually highlighted a real issue. JMH > test failures are now classified and reported as JUnit test failures, > without aborting the whole build: > https://issues.apache.org/jira/browse/CASSANDRA-21757 > 2. Work through the existing test failures. I'll go through the tests that > are currently failing or have been reported as flaky and fix them, or reach > out to the relevant committers for help. > For example: https://issues.apache.org/jira/browse/CASSANDRA-21769 > > Another area we can improve in parallel is CI execution time. The current > turnaround time for 6.0 CI runs (8–9 hours) is too long, and we're > increasingly tempted to avoid or minimise CI runs for patches because of > the cost. > > It also creates a barrier for contributors who don't have enough hardware > to run a large test suite locally. It also quickly consumes our donated CI > credits. > > Here are three areas I'd like to focus on: > 1. Optimise test scopes. Reduce the number of duplicate tests that don't > provide additional coverage, particularly in the Python dtests. > 2. Reduce the number of JDK versions we need to support and test. As we > agreed in our JDK upgrade policy some time ago, we can propagate support > for newer JDKs to older Cassandra releases, allowing us to drop support for > older JDKs in trunk. > 3. Optimise test execution itself. In particular, improve test and > test-class startup and shutdown times. See also: > https://issues.apache.org/jira/browse/CASSANDRA-21383 for more > details/ideas. > > The ultimate goal is to make CI more reliable, faster, and less expensive > to run, so that we can spend less time dealing with CI failures and more > time developing Cassandra with CI as a proper safety net. > > -- > Dmitry >
