[
https://issues.apache.org/jira/browse/CASSANDRA-21534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmitry Konstantinov updated CASSANDRA-21534:
--------------------------------------------
Fix Version/s: 6.x
7.x
> Reduce disk space usage by CommitLogSegmentManagerCDCTest
> ----------------------------------------------------------
>
> Key: CASSANDRA-21534
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21534
> Project: Apache Cassandra
> Issue Type: Improvement
> Components: Test/unit
> Reporter: Dmitry Konstantinov
> Priority: Normal
> Fix For: 6.x, 7.x
>
>
> Problem
> -------
> A single run of CommitLogSegmentManagerCDCTest transiently consumes ~9 GB of
> local disk under build/test/cassandra (peak measured on JDK 11) and causes
> periodical pre-ci run failures due to lack of ephemeral storage on nodes..
> This matters
> for CI: the test-agent pods are evicted when the node's ephemeral-storage free
> space drops below the eviction threshold (~5 GB), so one test needing ~9 GB
> can
> by itself trip the threshold and abort the in-flight test split (observed as
> pod evictions / "Retryable interruption" failures on cassandra-6.0 runs).
> Root cause
> ----------
> 1. The test extends CQLTester, whose afterTest() only clears schema-tracking
> state - it does not truncate or drop tables. Each of the 10 test methods
> bulk-writes enough data to fill the CDC space limit, flushing GBs of
> SSTables
> that accumulate in data/ across the whole suite (nothing reclaims them
> until
> the suite ends).
> 2. bulkWrite() loops a hard-coded 1000 times. In non-blocking mode the loop is
> not stopped by CDCWriteException, so it writes ~1000 x
> (commitlog_segment_size / 3)
> of mutations per method. With the 5 MiB test segment size that is ~1.7 GB
> of
> SSTables per non-blocking method.
> Changes (test-only, CommitLogSegmentManagerCDCTest)
> ---------------------------------------------------
> 1. Override afterTest() to reclaim data between methods: truncate the per-test
> keyspace's tables with ColumnFamilyStore.truncateBlockingWithoutSnapshot(),
> then LifecycleTransaction.waitForDeletions(), then super.afterTest(). The
> WithoutSnapshot variant is required - a plain TRUNCATE/DROP hard-links the
> SSTables into snapshots/ under the default auto_snapshot=true and frees
> nothing.
> 2. Shrink the commit log segment to the 1 MiB minimum in @BeforeClass via
> DatabaseDescriptor.setCommitLogSegmentSize(1). Mutation size is segment/3,
> so this reduces per-write data ~5x. Test-local only; production default is
> unchanged.
> 3. Replace the magic 1000 in bulkWrite() with a value derived from config:
> int writes = 2 * (int) (DatabaseDescriptor.getCDCTotalSpace() /
> mutationSize);
> Blocking mode still stops early on the first CDCWriteException;
> non-blocking
> mode writes 2x the fill count. This removes the fragile "1000 writes fills
> the CDC space" assumption and stops the small-CDC cases from over-writing.
> 4. Reduce the default CDC total space from 1024 MiB to 100 MiB in
> @BeforeClass.
> Only testSwitchingCDCWriteModes relies on the default, and with the
> computed
> write count it no longer needs to be large.
> Result
> ------
> Peak build/test/cassandra during the run (JDK 11, all 10 tests passing):
> Step Peak disk Wall time
> baseline (as-is) 9.15 GB ~95 s
> + afterTest truncate cleanup ~6.5 GB ~92 s
> + 1 MiB commit-log segments 1.03 GB ~35 s
> + computed write count 0.82 GB ~30 s
> + CDC total space 100 MiB 0.29 GB ~26 s
> Net: ~31x less peak disk (9.15 GB -> ~292 MB) and ~3.6x faster
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
