[
https://issues.apache.org/jira/browse/CASSANDRA-21150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18057191#comment-18057191
]
Dmitry Konstantinov commented on CASSANDRA-21150:
-------------------------------------------------
I've re-run tests locally for a set of commits in the suspicious range:
ad3f0e8f - PASSED (Ensure that gossip state for LEFT nodes is expired
eventually)
66a7a366 - PASSED (Merge branch 'cassandra-5.0' into trunk: Switch lz4-java to
at.yawk.lz4 version due to CVE)
ff78780d - FAILED (Add cursor based optimized compaction path)
{code}
testclasslist-compression:
[junit-timeout] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was
deprecated in version 9.0 and will likely be removed in a future release.
[junit-timeout] Testsuite:
org.apache.cassandra.io.sstable.VerifyTest-compression_jdk11
[junit-timeout] Testsuite:
org.apache.cassandra.io.sstable.VerifyTest-compression_jdk11 Tests run: 26,
Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.727 sec
[junit-timeout]
[junit-timeout] Testcase:
testVerifyCorruptRowCorrectDigest(org.apache.cassandra.io.sstable.VerifyTest)-compression_jdk11:
FAILED
[junit-timeout] Simple verify should have succeeded as digest matched
[junit-timeout] junit.framework.AssertionFailedError: Simple verify should have
succeeded as digest matched
[junit-timeout] at
org.apache.cassandra.io.sstable.VerifyTest.testVerifyCorruptRowCorrectDigest(VerifyTest.java:398)
[junit-timeout] at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit-timeout] at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit-timeout] at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit-timeout]
[junit-timeout]
[junit-timeout] Test org.apache.cassandra.io.sstable.VerifyTest FAILED
{code}
to run the tests I used:
{code:java}
./.build/run-tests.sh -t VerifyTest -a test-compression {code}
The failure can be reproduced in IDE as well using extra system property from
testclasslist-compression settings: -Dcassandra.test.compression=true
The exception is:
{code}
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
/Users/dmitry/IdeaProjects/cassandra-trunk/build/test/cassandra/data/Keyspace1/Corrupt2-11759ac4051811f187174f92e9f7cfa5/pa-1-big-Data.db
at
org.apache.cassandra.io.util.CompressedChunkReader$Standard.readChunk(CompressedChunkReader.java:306)
at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:166)
at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:46)
at
com.github.benmanes.caffeine.cache.LocalLoadingCache.lambda$newMappingFunction$3(LocalLoadingCache.java:183)
at
com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$2(LocalCache.java:167)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2688)
at
java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1908)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2686)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2669)
at
com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:112)
at
com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:58)
at
org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:240)
at
org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:219)
at
org.apache.cassandra.io.util.WrappingRebufferer.rebuffer(WrappingRebufferer.java:52)
at
org.apache.cassandra.io.util.LimitingRebufferer.rebuffer(LimitingRebufferer.java:52)
at
org.apache.cassandra.io.util.RandomAccessReader.reBufferAt(RandomAccessReader.java:67)
at
org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:209)
at
org.apache.cassandra.io.sstable.format.big.BigTableVerifier.deserializeIndex(BigTableVerifier.java:153)
at
org.apache.cassandra.io.sstable.format.SortedTableVerifier.verifyIndex(SortedTableVerifier.java:381)
at
org.apache.cassandra.io.sstable.format.big.BigTableVerifier.verifyIndex(BigTableVerifier.java:107)
at
org.apache.cassandra.io.sstable.format.SortedTableVerifier.verify(SortedTableVerifier.java:152)
at
org.apache.cassandra.io.sstable.VerifyTest.testVerifyCorruptRowCorrectDigest(VerifyTest.java:393)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
at
com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
at
com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
at
com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
at
com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)
Caused by: org.apache.cassandra.io.compress.CorruptBlockException:
(/Users/dmitry/IdeaProjects/cassandra-trunk/build/test/cassandra/data/Keyspace1/Corrupt2-11759ac4051811f187174f92e9f7cfa5/pa-1-big-Data.db):
corruption detected, chunk at 0 of length 53.
at
org.apache.cassandra.io.util.CompressedChunkReader$RandomAccessCompressedReader.read(CompressedChunkReader.java:149)
at
org.apache.cassandra.io.util.CompressedChunkReader$Standard.readChunk(CompressedChunkReader.java:271)
... 46 common frames omitted
ERROR [main] 2026-02-08T18:00:36,524 VerifyTest.java:397 - Unexpected exception
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
/Users/dmitry/IdeaProjects/cassandra-trunk/build/test/cassandra/data/Keyspace1/Corrupt2-11759ac4051811f187174f92e9f7cfa5/pa-1-big-Data.db
at
org.apache.cassandra.io.sstable.format.SortedTableVerifier.markAndThrow(SortedTableVerifier.java:141)
at
org.apache.cassandra.io.sstable.format.SortedTableVerifier.markAndThrow(SortedTableVerifier.java:122)
at
org.apache.cassandra.io.sstable.format.SortedTableVerifier.verifyIndex(SortedTableVerifier.java:386)
at
org.apache.cassandra.io.sstable.format.big.BigTableVerifier.verifyIndex(BigTableVerifier.java:107)
at
org.apache.cassandra.io.sstable.format.SortedTableVerifier.verify(SortedTableVerifier.java:152)
at
org.apache.cassandra.io.sstable.VerifyTest.testVerifyCorruptRowCorrectDigest(VerifyTest.java:393)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
at
com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
at
com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
at
com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
at
com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)
Caused by: java.lang.Exception: Invalid SSTable
/Users/dmitry/IdeaProjects/cassandra-trunk/build/test/cassandra/data/Keyspace1/Corrupt2-11759ac4051811f187174f92e9f7cfa5/pa-1-big-Data.db,
please force a full repair
at
org.apache.cassandra.io.sstable.format.SortedTableVerifier.markAndThrow(SortedTableVerifier.java:139)
... 30 common frames omitted
Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
/Users/dmitry/IdeaProjects/cassandra-trunk/build/test/cassandra/data/Keyspace1/Corrupt2-11759ac4051811f187174f92e9f7cfa5/pa-1-big-Data.db
at
org.apache.cassandra.io.util.CompressedChunkReader$Standard.readChunk(CompressedChunkReader.java:306)
at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:166)
at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:46)
at
com.github.benmanes.caffeine.cache.LocalLoadingCache.lambda$newMappingFunction$3(LocalLoadingCache.java:183)
at
com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$2(LocalCache.java:167)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2688)
at
java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1908)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2686)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2669)
at
com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:112)
at
com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:58)
at
org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:240)
at
org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:219)
at
org.apache.cassandra.io.util.WrappingRebufferer.rebuffer(WrappingRebufferer.java:52)
at
org.apache.cassandra.io.util.LimitingRebufferer.rebuffer(LimitingRebufferer.java:52)
at
org.apache.cassandra.io.util.RandomAccessReader.reBufferAt(RandomAccessReader.java:67)
at
org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:209)
at
org.apache.cassandra.io.sstable.format.big.BigTableVerifier.deserializeIndex(BigTableVerifier.java:153)
at
org.apache.cassandra.io.sstable.format.SortedTableVerifier.verifyIndex(SortedTableVerifier.java:381)
... 28 common frames omitted
Caused by: org.apache.cassandra.io.compress.CorruptBlockException:
(/Users/dmitry/IdeaProjects/cassandra-trunk/build/test/cassandra/data/Keyspace1/Corrupt2-11759ac4051811f187174f92e9f7cfa5/pa-1-big-Data.db):
corruption detected, chunk at 0 of length 53.
at
org.apache.cassandra.io.util.CompressedChunkReader$RandomAccessCompressedReader.read(CompressedChunkReader.java:149)
at
org.apache.cassandra.io.util.CompressedChunkReader$Standard.readChunk(CompressedChunkReader.java:271)
... 46 common frames omitted
{code}
> Test failure:
> org.apache.cassandra.io.sstable.VerifyTest.testVerifyCorruptRowCorrectDigest
> ------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-21150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21150
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Local/SSTable
> Reporter: Dmitry Konstantinov
> Assignee: Arvind Kandpal
> Priority: Normal
> Attachments: image-2026-01-31-16-46-01-322.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> [butler link
> |https://butler.cassandra.apache.org/#/ci/upstream/workflow/Cassandra-trunk/failure/org.apac[…]o.sstable/VerifyTest/testVerifyCorruptRowCorrectDigest]
> this test looks like has an explicit breaking point + it fails only for
> compression config profile +
> https://ci-cassandra.apache.org/job/Cassandra-trunk/2367/
> so an issue could be between
> https://github.com/apache/cassandra/commit/ec3b425c38d92d20d77d3a87c782ed9c072e1cd9
> (exclusive) and
> https://github.com/apache/cassandra/commit/3c69bd23673caa40b22f3500a3e3eecabc25c8e5
> commits
> !image-2026-01-31-16-46-01-322.png|width=600!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
