[
https://issues.apache.org/jira/browse/CASSANDRA-21534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmitry Konstantinov updated CASSANDRA-21534:
--------------------------------------------
Description:
Problem
-------
A single run of CommitLogSegmentManagerCDCTest transiently consumes ~9 GB of
local disk under build/test/cassandra (peak measured on JDK 11) and causes
periodical pre-ci run failures due to lack of ephemeral storage on nodes.. This
matters
for CI: the test-agent pods are evicted when the node's ephemeral-storage free
space drops below the eviction threshold (~5 GB), so one test needing ~9 GB can
by itself trip the threshold and abort the in-flight test split (observed as
pod evictions / "Retryable interruption" failures on cassandra-6.0 runs).
Root cause
----------
1. The test extends CQLTester, whose afterTest() only clears schema-tracking
state - it does not truncate or drop tables. Each of the 10 test methods
bulk-writes enough data to fill the CDC space limit, flushing GBs of SSTables
that accumulate in data/ across the whole suite (nothing reclaims them until
the suite ends).
2. bulkWrite() loops a hard-coded 1000 times. In non-blocking mode the loop is
not stopped by CDCWriteException, so it writes ~1000 x
(commitlog_segment_size / 3)
of mutations per method. With the 5 MiB test segment size that is ~1.7 GB of
SSTables per non-blocking method.
Changes (test-only, CommitLogSegmentManagerCDCTest)
---------------------------------------------------
1. Override afterTest() to reclaim data between methods: truncate the per-test
keyspace's tables with ColumnFamilyStore.truncateBlockingWithoutSnapshot(),
then LifecycleTransaction.waitForDeletions(), then super.afterTest(). The
WithoutSnapshot variant is required - a plain TRUNCATE/DROP hard-links the
SSTables into snapshots/ under the default auto_snapshot=true and frees
nothing.
2. Shrink the commit log segment to the 1 MiB minimum in @BeforeClass via
DatabaseDescriptor.setCommitLogSegmentSize(1). Mutation size is segment/3,
so this reduces per-write data ~5x. Test-local only; production default is
unchanged.
3. Replace the magic 1000 in bulkWrite() with a value derived from config:
int writes = 2 * (int) (DatabaseDescriptor.getCDCTotalSpace() /
mutationSize);
Blocking mode still stops early on the first CDCWriteException; non-blocking
mode writes 2x the fill count. This removes the fragile "1000 writes fills
the CDC space" assumption and stops the small-CDC cases from over-writing.
4. Reduce the default CDC total space from 1024 MiB to 100 MiB in @BeforeClass.
Only testSwitchingCDCWriteModes relies on the default, and with the computed
write count it no longer needs to be large.
Result
------
Peak build/test/cassandra during the run (JDK 11, all 10 tests passing):
{code}
Step Peak disk Wall time
baseline (as-is) 9.15 GB ~95 s
+ afterTest truncate cleanup ~6.5 GB ~92 s
+ 1 MiB commit-log segments 1.03 GB ~35 s
+ computed write count 0.82 GB ~30 s
+ CDC total space 100 MiB 0.29 GB ~26 s
{code}
Net: ~31x less peak disk (9.15 GB -> ~292 MB) and ~3.6x faster
was:
Problem
-------
A single run of CommitLogSegmentManagerCDCTest transiently consumes ~9 GB of
local disk under build/test/cassandra (peak measured on JDK 11) and causes
periodical pre-ci run failures due to lack of ephemeral storage on nodes.. This
matters
for CI: the test-agent pods are evicted when the node's ephemeral-storage free
space drops below the eviction threshold (~5 GB), so one test needing ~9 GB can
by itself trip the threshold and abort the in-flight test split (observed as
pod evictions / "Retryable interruption" failures on cassandra-6.0 runs).
Root cause
----------
1. The test extends CQLTester, whose afterTest() only clears schema-tracking
state - it does not truncate or drop tables. Each of the 10 test methods
bulk-writes enough data to fill the CDC space limit, flushing GBs of SSTables
that accumulate in data/ across the whole suite (nothing reclaims them until
the suite ends).
2. bulkWrite() loops a hard-coded 1000 times. In non-blocking mode the loop is
not stopped by CDCWriteException, so it writes ~1000 x
(commitlog_segment_size / 3)
of mutations per method. With the 5 MiB test segment size that is ~1.7 GB of
SSTables per non-blocking method.
Changes (test-only, CommitLogSegmentManagerCDCTest)
---------------------------------------------------
1. Override afterTest() to reclaim data between methods: truncate the per-test
keyspace's tables with ColumnFamilyStore.truncateBlockingWithoutSnapshot(),
then LifecycleTransaction.waitForDeletions(), then super.afterTest(). The
WithoutSnapshot variant is required - a plain TRUNCATE/DROP hard-links the
SSTables into snapshots/ under the default auto_snapshot=true and frees
nothing.
2. Shrink the commit log segment to the 1 MiB minimum in @BeforeClass via
DatabaseDescriptor.setCommitLogSegmentSize(1). Mutation size is segment/3,
so this reduces per-write data ~5x. Test-local only; production default is
unchanged.
3. Replace the magic 1000 in bulkWrite() with a value derived from config:
int writes = 2 * (int) (DatabaseDescriptor.getCDCTotalSpace() /
mutationSize);
Blocking mode still stops early on the first CDCWriteException; non-blocking
mode writes 2x the fill count. This removes the fragile "1000 writes fills
the CDC space" assumption and stops the small-CDC cases from over-writing.
4. Reduce the default CDC total space from 1024 MiB to 100 MiB in @BeforeClass.
Only testSwitchingCDCWriteModes relies on the default, and with the computed
write count it no longer needs to be large.
Result
------
Peak build/test/cassandra during the run (JDK 11, all 10 tests passing):
Step Peak disk Wall time
baseline (as-is) 9.15 GB ~95 s
+ afterTest truncate cleanup ~6.5 GB ~92 s
+ 1 MiB commit-log segments 1.03 GB ~35 s
+ computed write count 0.82 GB ~30 s
+ CDC total space 100 MiB 0.29 GB ~26 s
Net: ~31x less peak disk (9.15 GB -> ~292 MB) and ~3.6x faster
> Reduce disk space usage by CommitLogSegmentManagerCDCTest
> ----------------------------------------------------------
>
> Key: CASSANDRA-21534
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21534
> Project: Apache Cassandra
> Issue Type: Improvement
> Components: Test/unit
> Reporter: Dmitry Konstantinov
> Assignee: Dmitry Konstantinov
> Priority: Normal
> Fix For: 6.x, 7.x
>
>
> Problem
> -------
> A single run of CommitLogSegmentManagerCDCTest transiently consumes ~9 GB of
> local disk under build/test/cassandra (peak measured on JDK 11) and causes
> periodical pre-ci run failures due to lack of ephemeral storage on nodes..
> This matters
> for CI: the test-agent pods are evicted when the node's ephemeral-storage free
> space drops below the eviction threshold (~5 GB), so one test needing ~9 GB
> can
> by itself trip the threshold and abort the in-flight test split (observed as
> pod evictions / "Retryable interruption" failures on cassandra-6.0 runs).
> Root cause
> ----------
> 1. The test extends CQLTester, whose afterTest() only clears schema-tracking
> state - it does not truncate or drop tables. Each of the 10 test methods
> bulk-writes enough data to fill the CDC space limit, flushing GBs of
> SSTables
> that accumulate in data/ across the whole suite (nothing reclaims them
> until
> the suite ends).
> 2. bulkWrite() loops a hard-coded 1000 times. In non-blocking mode the loop is
> not stopped by CDCWriteException, so it writes ~1000 x
> (commitlog_segment_size / 3)
> of mutations per method. With the 5 MiB test segment size that is ~1.7 GB
> of
> SSTables per non-blocking method.
> Changes (test-only, CommitLogSegmentManagerCDCTest)
> ---------------------------------------------------
> 1. Override afterTest() to reclaim data between methods: truncate the per-test
> keyspace's tables with ColumnFamilyStore.truncateBlockingWithoutSnapshot(),
> then LifecycleTransaction.waitForDeletions(), then super.afterTest(). The
> WithoutSnapshot variant is required - a plain TRUNCATE/DROP hard-links the
> SSTables into snapshots/ under the default auto_snapshot=true and frees
> nothing.
> 2. Shrink the commit log segment to the 1 MiB minimum in @BeforeClass via
> DatabaseDescriptor.setCommitLogSegmentSize(1). Mutation size is segment/3,
> so this reduces per-write data ~5x. Test-local only; production default is
> unchanged.
> 3. Replace the magic 1000 in bulkWrite() with a value derived from config:
> int writes = 2 * (int) (DatabaseDescriptor.getCDCTotalSpace() /
> mutationSize);
> Blocking mode still stops early on the first CDCWriteException;
> non-blocking
> mode writes 2x the fill count. This removes the fragile "1000 writes fills
> the CDC space" assumption and stops the small-CDC cases from over-writing.
> 4. Reduce the default CDC total space from 1024 MiB to 100 MiB in
> @BeforeClass.
> Only testSwitchingCDCWriteModes relies on the default, and with the
> computed
> write count it no longer needs to be large.
> Result
> ------
> Peak build/test/cassandra during the run (JDK 11, all 10 tests passing):
> {code}
> Step Peak disk Wall time
> baseline (as-is) 9.15 GB ~95 s
> + afterTest truncate cleanup ~6.5 GB ~92 s
> + 1 MiB commit-log segments 1.03 GB ~35 s
> + computed write count 0.82 GB ~30 s
> + CDC total space 100 MiB 0.29 GB ~26 s
> {code}
> Net: ~31x less peak disk (9.15 GB -> ~292 MB) and ~3.6x faster
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
