Mmuzaf commented on code in PR #3102:
URL: https://github.com/apache/cassandra/pull/3102#discussion_r1714168592
##########
test/unit/org/apache/cassandra/metrics/DecayingEstimatedHistogramReservoirTest.java:
##########
@@ -604,6 +614,115 @@ public void testSize()
assertEquals(2, toSnapshot.apply(histogram).size());
}
+ /**
+ * This is a test that exposes CASSANDRA-19365 race condition
+ * The idea is to update a histogram from multiple threads
+ * and observe if the reported p99 doesn't get stuck at a low value or
p50 at a high value
+ * due to update with high weight being inserted after the buckets are
rescaled
+ * <p>
+ * The load has 95% of 42, and 5% of the time it's 1109. Despite that
the histogram may be convinced for a long time
+ * that p99 is 42 or that p50 is 1109.
+ * The reason may be seen in the snapshot dump, where after rescale
the bucket values may get
+ * very big due to the race condition and too big weight of the
inserted samples.
+ * The values were picked to match bucket boundaries, but that's only
for aesthetics.
+ * <p>
+ * In production the rescale happens every 30 minutes. In this test
time flies roughly 1000 times faster to
+ * hit the condition in a reasonable time.
+ */
+ @Ignore("This test exposes a specific CASSANDRA-19365 race condition;
it doesn't make sense to run it in CI")
+ @Test
+ public void testConcurrentUpdateAndRescale() throws
InterruptedException
+ {
+ int maxTestDurationMillis = 60_000;
+ AtomicBoolean stop = new AtomicBoolean(false);
+ AtomicBoolean failed = new AtomicBoolean(false);
+ int NUM_UPDATE_THREADS = 60;
+ TestClock clock = new TestClock();
+
+ DecayingEstimatedHistogramReservoir histogram = new
DecayingEstimatedHistogramReservoir(clock);
+
+ ExecutorService executors = Executors.newFixedThreadPool(2 +
NUM_UPDATE_THREADS);
+ // NUM_UPDATE_THREADS threads updating the histogram
+ for (int i = 0; i < NUM_UPDATE_THREADS; i++)
+ {
+ executors.submit(() -> {
+ while (!stop.get())
Review Comment:
It would be a good idea to check `isInterrupted` as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
