[
https://issues.apache.org/jira/browse/CASSANDRA-21800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18130567#comment-18130567
]
Dmitry Konstantinov commented on CASSANDRA-21800:
-------------------------------------------------
CI log analysis:
{code:java}
- The 900 s timeout covers the whole test class. testSequentialDelivery took
3m17s on this agent, versus ~30 s on the jdk11/17 runs. testConcurrentCommit
then ran for 11m41s and was killed at epoch 2428 of the ~3017 it needs. Commits
were still being made up to the kill.
- The test makes 3000 commits. Each epoch calls forceBlockingFlush from
SystemKeyspaceStorage.append
(src/java/org/apache/cassandra/tcm/log/SystemKeyspaceStorage.java:81). The next
commit can't land until the previous epoch has been applied locally, so the
flush time caps throughput at about 3.5 epochs/s on this agent. The same build
passed on jdk17 in 48 s.
- The time is all in one part of the flush:
┌─────────────────────────────────┬─────────────────────┬───────────────────────────────────────────────────┐
│ Agent (data disk) │ Writing the sstable │ Commit step
("Completed flushing" → "Flushed to") │
├─────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────┤
│ jdk17, 099f1687ae8b (/dev/md2) │ ~2 ms │ p50 2 ms, p99 11 ms
│
├─────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────┤
│ jdk21, 8246b18f2010 (/dev/sda4) │ ~2 ms │ p50 270 ms, p99 ~390
ms │
└─────────────────────────────────┴─────────────────────┴───────────────────────────────────────────────────┘
Compactions on the slow agent show the same pattern (313 ms vs 11 ms), and so
do idle system.local and schema flushes at startup (~105 ms vs 3–11 ms).
Why the commit step hits the disk despite skip_sync
skip_sync=true was confirmed in the log ("Skip fsync enabled"), but these three
calls go straight to FileChannel.force(true) instead of through SyncUtil:
- Bloom filter (Filter.db): FilterComponent.java:80 calls stream.sync(), which
is FileOutputStreamPlus.sync() at FileOutputStreamPlus.java:62.
- Compression metadata (CompressionInfo.db): CompressionMetadata.java:567 calls
out.sync(), the same method.
- Digest file (Digest.crc32): ChecksumWriter.java:111 calls
fos.getChannel().force(true).
{code}
> Flaky test - distributed.test.log.DistributedLogTest
> ----------------------------------------------------
>
> Key: CASSANDRA-21800
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21800
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Test/dtest/java
> Reporter: Dmitry Konstantinov
> Assignee: Dmitry Konstantinov
> Priority: Normal
>
> Example:
> https://ci-cassandra.apache.org/job/Cassandra-6.0/25/testReport/junit/org.apache.cassandra.distributed.test.log/DistributedLogTest/Tests___jvm_dtest_jdk21_1_16___testConcurrentCommit_cassandra_testtag_IS_UNDEFINED/
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
