> > > I think if we want to do this, it should be extremely easy - by which I > mean automatic, really. This shouldn’t be too tricky I think? We just need > to produce a diff of new test classes and methods within existing classes.
Having a CircleCI job that automatically runs all new/modified tests would be a great way to prevent most of the new flakies. We would still miss some cases, like unmodified tests that turn flaky after changing the tested code, but I'd say that's not as usual. > I can probably help out by putting together something to output @Test > annotated methods within a source tree, if others are able to turn this > into a part of the CircleCI pre-commit task (i.e. to pick the common > ancestor with trunk, 4.1 etc, and run this task for each of the outputs) I think we would need a bash/sh shell script taking a diff file and test directory, and returning the file path and qualified class name of every modified test class. I'd say we don't need the method names for Java tests because quite often we see flaky tests that only fail when running their entire class, so it's probably better to repeatedly run entire test classes instead of particular methods. We would also need a similar script for Python dtests. We would probably want it to provide the full path of the modified tests (as in cqlsh_tests/test_cqlsh.py::TestCqlshSmoke::test_create_index) because those tests can be quite resource-intensive. I think once we have those scripts we could plug their output to the CircleCI commands for repeating tests. Putting together all this seems relatively involved, so it can take us some time to get it ready. In the meantime, I think it's a good practice to just manually include any new/modified tests into the CircleCI config. Doing so only requires to pass a few additional options to the script that generates the config, which doesn't seem to require too much effort. On Wed, 10 Aug 2022 at 19:47, Brandon Williams <dri...@gmail.com> wrote: > > Side note, Butler is reporting CASSANDRA-17348 as open (it's resolved as > a duplicate). > > This is fixed. >
