[
https://issues.apache.org/jira/browse/CASSANDRA-21008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ling Mao reassigned CASSANDRA-21008:
------------------------------------
Assignee: Ling Mao
> Optimizing Randomized Testing for Reproducibility
> -------------------------------------------------
>
> Key: CASSANDRA-21008
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21008
> Project: Apache Cassandra
> Issue Type: Improvement
> Components: Test/unit
> Reporter: Ling Mao
> Assignee: Ling Mao
> Priority: Normal
>
> h4. Goals and Motivation
>
> Random Test is designed to help with testing systems that rely on randomness,
> such as those involving probabilistic behavior, concurrency, or randomized
> algorithms. It enables automated, repeatable testing of code that behaves
> differently on each run, ensuring more comprehensive test coverage.
> Key benefits:
> Test Uncertainty: It provides tools for testing systems where inputs,
> behaviors, or results vary across test runs.
> Reproducibility: It helps ensure tests can be reliably reproduced, even with
> randomness involved.
> Stress Testing: It is ideal for identifying edge cases and bugs that might
> not be easily found in deterministic tests.
> Better Test Coverage: Randomized tests provide more varied input
> combinations, leading to more thorough testing.
> In short, Random Test ensures the robustness and reliability of systems
> dealing with randomness, making it essential for testing complex or
> non-deterministic behavior.
> h4. Approach
> {code:java}
> private static final String RANDOMIZED_CONTEXT_FQN =
> "com.carrotsearch.randomizedtesting.RandomizedContext";
> Class.forName(RANDOMIZED_CONTEXT_FQN); // sets isRandomTestMode = true{code}
> We use this approach to determine whether we are in Random Test Mode or
> Normal Mode (if we can find the FQN, we activate the test infrastructure).
> {code:java}
> Randomness.java
> public static Random get() {
> if (isRandomTestMode()) {
> return our ThreadLocal seedable random implementation;
> } else {
> return java native Random implementation;
> }
> } {code}
> We centralize all random operations in "Randomness.java" to make them
> repeatable, unified, and aggregated (when in Random Test Mode, all randomness
> becomes reproducible and deterministic). For example:
> Java Random APIs (SecureRandom, Collections.shuffle, UUID.randomUUID(),
> Math.random(), etc.)
> Additional random utilities (randomUTF8, randomLocale, randomDate,
> randomAsciiLetters, etc.)
> We print the seed when a unit test fails:
> {code:java}
> java.lang.RuntimeException: java.lang.ExceptionInInitializerError
> at
> __randomizedtesting.SeedInfo.seed([FFB4961AF9935703:3AFD18F7B24FCA34]:0)
> at
> org.apache.cassandra.distributed.impl.Instance.lambda$startup$7(Instance.java:675)
> ... {code}
> Then use the following command to reproduce the failure:
> {code:java}
> ant testsome -Dtest.name=org.apache.cassandra.db.marshal.MyRandomizedTest
> -Dtests.seed=FFB4961AF9935703 {code}
> We can also print JVM system properties, memory, and CPU information for
> additional debugging context.
> h4. Proposed Changes
> {code:java}
> // Before
> double randDbl = random.nextDouble();
> // After
> double randDbl = Randomness.get().nextDouble(); {code}
> h4. Test Plan
> We will refactor all random test suites. Existing test suites will verify
> these changes. Additionally, we will add unit tests for Randomness.
> h4. Rejected Alternatives: Why not use carrotsearch randomizedtesting
> directly?
> Elasticsearch and Lucene have successfully used randomizedtesting (Apache-2.0
> license) for decades([7]), and it's already present in Cassandra. However, we
> reject this approach for the following reasons:
> # {*}Initialization timing issues{*}: Using Java reflection to call
> {{RandomizedContext.current().getRandom()}} causes
> {{java.lang.ExceptionInInitializerError}} due to class initialization timing
> at startup.
> # {*}Boundary limitations{*}: randomizedtesting accesses random contexts
> within {{@BeforeClass}} and {{@AfterClass}} boundaries, but static test class
> initializers cannot access random contexts. However, we need random behavior
> during server startup.
> # {*}Distributed JVM incompatibility{*}: randomizedtesting is not compatible
> with distributed JVM tests. Our native Java approach works seamlessly with
> distributed JVM tests, simulators, and QuickTheories.
>
> *Existing test patterns:*
> For unit tests:
>
> {code:java}
> public class UUIDTypeTest extends RandomTestSuite@Listeners({
> ReproduceInfoPrinter.class })
> @ThreadLeakScope(ThreadLeakScope.Scope.NONE)
> @SeedDecorators({ MixWithSuiteName.class })
> @TestMethodProviders(value = { JUnit4MethodProvider.class })
> public abstract class RandomTestSuite extends RandomizedTest{code}
>
> For distributed JVM tests:
>
> {code:java}
> @RunWith(RandomizedRunner.class)
> @Listeners({ ReproduceInfoPrinter.class })
> @ThreadLeakScope(ThreadLeakScope.Scope.NONE)
> @SeedDecorators({ MixWithSuiteName.class })
> public class HintDataReappearingTest extends AbstractHintWindowTest{code}
>
> *Limitations of existing approaches:* # Some distributed JVM tests cannot
> obtain {{RandomizedContext}}
> # Tests using {{@RunWith(Parameterized.class)}} or
> {{@RunWith(BMUnitRunner.class)}} cannot work with
> {{@RunWith(RandomizedRunner.class)}} since only one {{@RunWith}} annotation
> is allowed per test
> # Extending {{RandomTestSuite}} or using
> {{@RunWith(RandomizedRunner.class)}} requires a base test class, causing
> extensive code changes that are difficult to review
> h4. Reference
> [1] https://labs.carrotsearch.com/randomizedtesting-concept.html
> [2]
> https://get.carrotsearch.com/presentations/dweiss-barcelona-testing-2011.pdf
> [3]
> https://spinscale.de/posts/2020-04-22-testing-and-releasing-elasticsearch-and-the-elastic-stack.html
> [4] https://github.com/randomizedtesting/randomizedtesting
> [5] https://labs.carrotsearch.com/randomizedtesting.html
> [6]
> https://github.com/randomizedtesting/randomizedtesting/?tab=Apache-2.0-1-ov-file#
> [7]https://github.com/elastic/elasticsearch/blob/004fab6627b0ae65cf939e0d69a4db8dbd175798/server/src/main/java/org/elasticsearch/common/Randomness.java#L93
> [8] https://www.youtube.com/watch?v=zD57QKzqdCw
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
