Dmitry Konstantinov created CASSANDRA-21534:
-----------------------------------------------
Summary: Reduce disk space usage by CommitLogSegmentManagerCDCTest
Key: CASSANDRA-21534
URL: https://issues.apache.org/jira/browse/CASSANDRA-21534
Project: Apache Cassandra
Issue Type: Improvement
Components: Test/unit
Reporter: Dmitry Konstantinov
Problem
-------
A single run of CommitLogSegmentManagerCDCTest transiently consumes ~9 GB of
local disk under build/test/cassandra (peak measured on JDK 11) and causes
periodical pre-ci run failures due to lack of ephemeral storage on nodes.. This
matters
for CI: the test-agent pods are evicted when the node's ephemeral-storage free
space drops below the eviction threshold (~5 GB), so one test needing ~9 GB can
by itself trip the threshold and abort the in-flight test split (observed as
pod evictions / "Retryable interruption" failures on cassandra-6.0 runs).
Root cause
----------
1. The test extends CQLTester, whose afterTest() only clears schema-tracking
state - it does not truncate or drop tables. Each of the 10 test methods
bulk-writes enough data to fill the CDC space limit, flushing GBs of SSTables
that accumulate in data/ across the whole suite (nothing reclaims them until
the suite ends).
2. bulkWrite() loops a hard-coded 1000 times. In non-blocking mode the loop is
not stopped by CDCWriteException, so it writes ~1000 x
(commitlog_segment_size / 3)
of mutations per method. With the 5 MiB test segment size that is ~1.7 GB of
SSTables per non-blocking method.
Changes (test-only, CommitLogSegmentManagerCDCTest)
---------------------------------------------------
1. Override afterTest() to reclaim data between methods: truncate the per-test
keyspace's tables with ColumnFamilyStore.truncateBlockingWithoutSnapshot(),
then LifecycleTransaction.waitForDeletions(), then super.afterTest(). The
WithoutSnapshot variant is required - a plain TRUNCATE/DROP hard-links the
SSTables into snapshots/ under the default auto_snapshot=true and frees
nothing.
2. Shrink the commit log segment to the 1 MiB minimum in @BeforeClass via
DatabaseDescriptor.setCommitLogSegmentSize(1). Mutation size is segment/3,
so this reduces per-write data ~5x. Test-local only; production default is
unchanged.
3. Replace the magic 1000 in bulkWrite() with a value derived from config:
int writes = 2 * (int) (DatabaseDescriptor.getCDCTotalSpace() /
mutationSize);
Blocking mode still stops early on the first CDCWriteException; non-blocking
mode writes 2x the fill count. This removes the fragile "1000 writes fills
the CDC space" assumption and stops the small-CDC cases from over-writing.
4. Reduce the default CDC total space from 1024 MiB to 100 MiB in @BeforeClass.
Only testSwitchingCDCWriteModes relies on the default, and with the computed
write count it no longer needs to be large.
Result
------
Peak build/test/cassandra during the run (JDK 11, all 10 tests passing):
Step Peak disk Wall time
baseline (as-is) 9.15 GB ~95 s
+ afterTest truncate cleanup ~6.5 GB ~92 s
+ 1 MiB commit-log segments 1.03 GB ~35 s
+ computed write count 0.82 GB ~30 s
+ CDC total space 100 MiB 0.29 GB ~26 s
Net: ~31x less peak disk (9.15 GB -> ~292 MB) and ~3.6x faster
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
