Thanks for the exhaustive response, Alex :) Let me bring my point of view:
1. Since long tests are just unit tests that take a long time to run, it makes sense to separate them for efficient parallelization in CI. Since we are adding new tests, modifying the existing ones, etc., that should be something maintainable; otherwise, the distinction makes no sense to me. For example - adjust timeouts on CI to 1 minute per test class for "short" tests and more for "long" tests. To satisfy CI, the contributor will have to either make the test run faster or move it to the "long" tests. The opposite enforcement could be more difficult, though it is doable as well - failing the "long" test if it takes too little time and should be qualified as a regular unit test. As I'm reading what I've just written, it sounds stupid :/ We should get rid of long-running unit tests altogether. They should run faster or be split. 2. I'm still confused about the distinction between burn and fuzz tests - it seems to me that fuzz tests are just modern burn tests - should we refactor the existing burn tests to use the new framework? 3. Simulation tests - since you say they provide a way to execute a test deterministically, it should be a property of unit tests - well, a unit test is either deterministic or a fuzz test. Is the simulation framework usable for CQLTester-based tests? (side question here: I've noticed many "sleeps" in the tests - is it possible with simulation tests to artificially move the clock forward by, say, 5 seconds instead of sleeping just to test, for example whether TTL works?) 4. Yeah, running a complete suite for each artificially crafted configuration brings little value compared to the maintenance and infrastructure costs. It feels like we are running all tests a bit blindly, hoping we catch something accidentally. I agree this is not the purpose of the unit tests and should be covered instead by fuzz. For features like CDC, compression, different sstable formats, trie memtable, commit log compression/encryption, system directory keyspace, etc... we should have dedicated tests that verify just that functionality With more or more functionality offered by Cassandra, they will become a significant pain shortly. Let's start thinking about concrete actions. Also, as we start refactoring the tests, it will be an excellent opportunity to move to JUnit 5. thanks, Jacek
