[
https://issues.apache.org/jira/browse/CASSANDRA-17469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andres de la Peña updated CASSANDRA-17469:
------------------------------------------
Description:
Intermitent failures in
{{{}org.apache.cassandra.db.commitlog.GroupCommitLogTest{}}}:
*
[testTruncateWithoutSnapshotNonDurable|https://ci-cassandra.apache.org/job/Cassandra-trunk/1024/testReport/org.apache.cassandra.db.commitlog/GroupCommitLogTest/testTruncateWithoutSnapshotNonDurable_4__cdc_2/]
*
[testRecoveryWithEmptyLog|https://ci-cassandra.apache.org/job/Cassandra-trunk/1024/testReport/org.apache.cassandra.db.commitlog/GroupCommitLogTest/testTruncateWithoutSnapshotNonDurable_4__cdc_2/]
*
[testOutOfOrderLogDiscardWithCompaction|https://ci-cassandra.apache.org/job/Cassandra-trunk/1024/testReport/org.apache.cassandra.db.commitlog/GroupCommitLogTest/testTruncateWithoutSnapshotNonDurable_4__cdc_2/]
*
[testExceedRecordLimitWithMultiplePartitions|https://ci-cassandra.apache.org/job/Cassandra-trunk/1024/testReport/org.apache.cassandra.db.commitlog/GroupCommitLogTest/testExceedRecordLimitWithMultiplePartitions_5__cdc_2/]
*
[testRecoveryWithShortMutationSize|https://ci-cassandra.apache.org/job/Cassandra-trunk/1024/testReport/org.apache.cassandra.db.commitlog/GroupCommitLogTest/testRecoveryWithShortMutationSize_4__cdc_2/]
{code:java}
Failed 2 times in the last 11 runs. Flakiness: 30%, Stability: 81%
Stacktrace
java.io.UncheckedIOException
at
org.apache.cassandra.io.util.PathUtils.propagateUnchecked(PathUtils.java:768)
at
org.apache.cassandra.io.util.PathUtils.propagateUnchecked(PathUtils.java:753)
at org.apache.cassandra.io.util.PathUtils.delete(PathUtils.java:255)
at org.apache.cassandra.io.util.PathUtils.delete(PathUtils.java:297)
at org.apache.cassandra.io.util.PathUtils.delete(PathUtils.java:304)
at org.apache.cassandra.io.util.File.delete(File.java:158)
at org.apache.cassandra.io.util.File.delete(File.java:167)
at
org.apache.cassandra.db.commitlog.CommitLogSegmentManagerCDC.discard(CommitLogSegmentManagerCDC.java:75)
at
org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager.closeAndDeleteSegmentUnsafe(AbstractCommitLogSegmentManager.java:479)
at
org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager.stopUnsafe(AbstractCommitLogSegmentManager.java:452)
at
org.apache.cassandra.db.commitlog.CommitLog.stopUnsafe(CommitLog.java:504)
at
org.apache.cassandra.db.commitlog.CommitLog.resetUnsafe(CommitLog.java:470)
at
org.apache.cassandra.db.commitlog.CommitLogTest.beforeTest(CommitLogTest.java:184)
at jdk.internal.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Caused by: java.nio.file.NoSuchFileException:
build/test/cassandra/commitlog/CommitLog-7-1647704321215.log
at
java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at
java.base/sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:249)
at
java.base/sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:105)
at java.base/java.nio.file.Files.delete(Files.java:1142)
at org.apache.cassandra.io.util.PathUtils.delete(PathUtils.java:250)
Standard Output
DEBUG [main] 2022-03-19 15:41:04,702 InternalLoggerFactory.java:63 - Using
SLF4J as the default logging framework
DEBUG [main] 2022-03-19 15:41:04,712 InternalThreadLocalMap.java:83 -
-Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
DEBUG [main] 2022-03-19 15:41:04,712 InternalThreadLocalMap.java:86 -
-Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
INFO [main] 2022-03-19 15:41:04,713 CipherFactory.java:68 - initializing
CipherFactory
INFO [main] 2022-03-19 15:41:04,713 JKSKeyPro
...[truncated 1879728 chars]...
tLog-7-1647704466908.log
INFO [main] 2022-03-19 15:41:26,711 CommitLog.java:175 - No commitlog files
found; skipping replay
DEBUG [main] 2022-03-19 15:41:28,150 CommitLogReader.java:266 - Reading
build/test/cassandra/commitlog/CommitLog-7-1647704466910.log (CL version 7,
messaging version 12, compression
org.apache.cassandra.io.compress.ZstdCompressor{})
INFO [main] 2022-03-19 15:41:28,178 CommitLogReader.java:257 - Finished
reading build/test/cassandra/commitlog/CommitLog-7-1647704466910.log
{code}
All these failures belong to the same single run, so it could be an environment
issue for that particular run. However, it is possible to hit that failure with
the CircleCI test multiplexer in current trunk:
*
[https://app.circleci.com/pipelines/github/adelapena/cassandra/1403/workflows/c02df2b9-732b-478b-825f-f4a0491c0867]
*
[https://app.circleci.com/pipelines/github/adelapena/cassandra/1404/workflows/8acda383-78e6-472b-8a1e-f72684992d6b]
The above runs have 500 and 2000 iterations, and they hit the failure once and
twice each, so flakiness seems really low.
These tests looks ok in 4.0, as suggested by Butler and [this repeated Circle
run|https://app.circleci.com/pipelines/github/adelapena/cassandra/1402/workflows/5c6ee580-d8c8-4f44-a26c-e8a0db5b008b].
was:
Intermitent failures in
{{{}org.apache.cassandra.db.commitlog.GroupCommitLogTest{}}}:
*
[testTruncateWithoutSnapshotNonDurable|https://ci-cassandra.apache.org/job/Cassandra-trunk/1024/testReport/org.apache.cassandra.db.commitlog/GroupCommitLogTest/testTruncateWithoutSnapshotNonDurable_4__cdc_2/]
*
[testRecoveryWithEmptyLog|https://ci-cassandra.apache.org/job/Cassandra-trunk/1024/testReport/org.apache.cassandra.db.commitlog/GroupCommitLogTest/testTruncateWithoutSnapshotNonDurable_4__cdc_2/]
*
[testOutOfOrderLogDiscardWithCompaction|https://ci-cassandra.apache.org/job/Cassandra-trunk/1024/testReport/org.apache.cassandra.db.commitlog/GroupCommitLogTest/testTruncateWithoutSnapshotNonDurable_4__cdc_2/]
*
[testExceedRecordLimitWithMultiplePartitions|https://ci-cassandra.apache.org/job/Cassandra-trunk/1024/testReport/org.apache.cassandra.db.commitlog/GroupCommitLogTest/testExceedRecordLimitWithMultiplePartitions_5__cdc_2/]
*
[testRecoveryWithShortMutationSize|https://ci-cassandra.apache.org/job/Cassandra-trunk/1024/testReport/org.apache.cassandra.db.commitlog/GroupCommitLogTest/testRecoveryWithShortMutationSize_4__cdc_2/]
{code:java}
Failed 2 times in the last 11 runs. Flakiness: 30%, Stability: 81%
Stacktrace
java.io.UncheckedIOException
at
org.apache.cassandra.io.util.PathUtils.propagateUnchecked(PathUtils.java:768)
at
org.apache.cassandra.io.util.PathUtils.propagateUnchecked(PathUtils.java:753)
at org.apache.cassandra.io.util.PathUtils.delete(PathUtils.java:255)
at org.apache.cassandra.io.util.PathUtils.delete(PathUtils.java:297)
at org.apache.cassandra.io.util.PathUtils.delete(PathUtils.java:304)
at org.apache.cassandra.io.util.File.delete(File.java:158)
at org.apache.cassandra.io.util.File.delete(File.java:167)
at
org.apache.cassandra.db.commitlog.CommitLogSegmentManagerCDC.discard(CommitLogSegmentManagerCDC.java:75)
at
org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager.closeAndDeleteSegmentUnsafe(AbstractCommitLogSegmentManager.java:479)
at
org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager.stopUnsafe(AbstractCommitLogSegmentManager.java:452)
at
org.apache.cassandra.db.commitlog.CommitLog.stopUnsafe(CommitLog.java:504)
at
org.apache.cassandra.db.commitlog.CommitLog.resetUnsafe(CommitLog.java:470)
at
org.apache.cassandra.db.commitlog.CommitLogTest.beforeTest(CommitLogTest.java:184)
at jdk.internal.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Caused by: java.nio.file.NoSuchFileException:
build/test/cassandra/commitlog/CommitLog-7-1647704321215.log
at
java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at
java.base/sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:249)
at
java.base/sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:105)
at java.base/java.nio.file.Files.delete(Files.java:1142)
at org.apache.cassandra.io.util.PathUtils.delete(PathUtils.java:250)
Standard Output
DEBUG [main] 2022-03-19 15:41:04,702 InternalLoggerFactory.java:63 - Using
SLF4J as the default logging framework
DEBUG [main] 2022-03-19 15:41:04,712 InternalThreadLocalMap.java:83 -
-Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
DEBUG [main] 2022-03-19 15:41:04,712 InternalThreadLocalMap.java:86 -
-Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
INFO [main] 2022-03-19 15:41:04,713 CipherFactory.java:68 - initializing
CipherFactory
INFO [main] 2022-03-19 15:41:04,713 JKSKeyPro
...[truncated 1879728 chars]...
tLog-7-1647704466908.log
INFO [main] 2022-03-19 15:41:26,711 CommitLog.java:175 - No commitlog files
found; skipping replay
DEBUG [main] 2022-03-19 15:41:28,150 CommitLogReader.java:266 - Reading
build/test/cassandra/commitlog/CommitLog-7-1647704466910.log (CL version 7,
messaging version 12, compression
org.apache.cassandra.io.compress.ZstdCompressor{})
INFO [main] 2022-03-19 15:41:28,178 CommitLogReader.java:257 - Finished
reading build/test/cassandra/commitlog/CommitLog-7-1647704466910.log
{code}
All these failures belong to the same single run, so it could be an environment
issue for that particular run. However, it is possible to hit that failure with
the CircleCI test multiplexer in current trunk:
*
[https://app.circleci.com/pipelines/github/adelapena/cassandra/1403/workflows/c02df2b9-732b-478b-825f-f4a0491c0867]
*
[https://app.circleci.com/pipelines/github/adelapena/cassandra/1404/workflows/8acda383-78e6-472b-8a1e-f72684992d6b]
The above runs have 500 and 2000 iterations, and they hit the failure once and
twice each, so flakiness seems really low.
> Test Failure: org.apache.cassandra.db.commitlog.GroupCommitLogTest
> ------------------------------------------------------------------
>
> Key: CASSANDRA-17469
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17469
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
> Reporter: Andres de la Peña
> Priority: Normal
>
> Intermitent failures in
> {{{}org.apache.cassandra.db.commitlog.GroupCommitLogTest{}}}:
> *
> [testTruncateWithoutSnapshotNonDurable|https://ci-cassandra.apache.org/job/Cassandra-trunk/1024/testReport/org.apache.cassandra.db.commitlog/GroupCommitLogTest/testTruncateWithoutSnapshotNonDurable_4__cdc_2/]
> *
> [testRecoveryWithEmptyLog|https://ci-cassandra.apache.org/job/Cassandra-trunk/1024/testReport/org.apache.cassandra.db.commitlog/GroupCommitLogTest/testTruncateWithoutSnapshotNonDurable_4__cdc_2/]
> *
> [testOutOfOrderLogDiscardWithCompaction|https://ci-cassandra.apache.org/job/Cassandra-trunk/1024/testReport/org.apache.cassandra.db.commitlog/GroupCommitLogTest/testTruncateWithoutSnapshotNonDurable_4__cdc_2/]
> *
> [testExceedRecordLimitWithMultiplePartitions|https://ci-cassandra.apache.org/job/Cassandra-trunk/1024/testReport/org.apache.cassandra.db.commitlog/GroupCommitLogTest/testExceedRecordLimitWithMultiplePartitions_5__cdc_2/]
> *
> [testRecoveryWithShortMutationSize|https://ci-cassandra.apache.org/job/Cassandra-trunk/1024/testReport/org.apache.cassandra.db.commitlog/GroupCommitLogTest/testRecoveryWithShortMutationSize_4__cdc_2/]
> {code:java}
> Failed 2 times in the last 11 runs. Flakiness: 30%, Stability: 81%
> Stacktrace
> java.io.UncheckedIOException
> at
> org.apache.cassandra.io.util.PathUtils.propagateUnchecked(PathUtils.java:768)
> at
> org.apache.cassandra.io.util.PathUtils.propagateUnchecked(PathUtils.java:753)
> at org.apache.cassandra.io.util.PathUtils.delete(PathUtils.java:255)
> at org.apache.cassandra.io.util.PathUtils.delete(PathUtils.java:297)
> at org.apache.cassandra.io.util.PathUtils.delete(PathUtils.java:304)
> at org.apache.cassandra.io.util.File.delete(File.java:158)
> at org.apache.cassandra.io.util.File.delete(File.java:167)
> at
> org.apache.cassandra.db.commitlog.CommitLogSegmentManagerCDC.discard(CommitLogSegmentManagerCDC.java:75)
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager.closeAndDeleteSegmentUnsafe(AbstractCommitLogSegmentManager.java:479)
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager.stopUnsafe(AbstractCommitLogSegmentManager.java:452)
> at
> org.apache.cassandra.db.commitlog.CommitLog.stopUnsafe(CommitLog.java:504)
> at
> org.apache.cassandra.db.commitlog.CommitLog.resetUnsafe(CommitLog.java:470)
> at
> org.apache.cassandra.db.commitlog.CommitLogTest.beforeTest(CommitLogTest.java:184)
> at jdk.internal.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Caused by: java.nio.file.NoSuchFileException:
> build/test/cassandra/commitlog/CommitLog-7-1647704321215.log
> at
> java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
> at
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
> at
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
> at
> java.base/sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:249)
> at
> java.base/sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:105)
> at java.base/java.nio.file.Files.delete(Files.java:1142)
> at org.apache.cassandra.io.util.PathUtils.delete(PathUtils.java:250)
> Standard Output
> DEBUG [main] 2022-03-19 15:41:04,702 InternalLoggerFactory.java:63 - Using
> SLF4J as the default logging framework
> DEBUG [main] 2022-03-19 15:41:04,712 InternalThreadLocalMap.java:83 -
> -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
> DEBUG [main] 2022-03-19 15:41:04,712 InternalThreadLocalMap.java:86 -
> -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
> INFO [main] 2022-03-19 15:41:04,713 CipherFactory.java:68 - initializing
> CipherFactory
> INFO [main] 2022-03-19 15:41:04,713 JKSKeyPro
> ...[truncated 1879728 chars]...
> tLog-7-1647704466908.log
> INFO [main] 2022-03-19 15:41:26,711 CommitLog.java:175 - No commitlog files
> found; skipping replay
> DEBUG [main] 2022-03-19 15:41:28,150 CommitLogReader.java:266 - Reading
> build/test/cassandra/commitlog/CommitLog-7-1647704466910.log (CL version 7,
> messaging version 12, compression
> org.apache.cassandra.io.compress.ZstdCompressor{})
> INFO [main] 2022-03-19 15:41:28,178 CommitLogReader.java:257 - Finished
> reading build/test/cassandra/commitlog/CommitLog-7-1647704466910.log
> {code}
> All these failures belong to the same single run, so it could be an
> environment issue for that particular run. However, it is possible to hit
> that failure with the CircleCI test multiplexer in current trunk:
> *
> [https://app.circleci.com/pipelines/github/adelapena/cassandra/1403/workflows/c02df2b9-732b-478b-825f-f4a0491c0867]
> *
> [https://app.circleci.com/pipelines/github/adelapena/cassandra/1404/workflows/8acda383-78e6-472b-8a1e-f72684992d6b]
> The above runs have 500 and 2000 iterations, and they hit the failure once
> and twice each, so flakiness seems really low.
> These tests looks ok in 4.0, as suggested by Butler and [this repeated Circle
> run|https://app.circleci.com/pipelines/github/adelapena/cassandra/1402/workflows/5c6ee580-d8c8-4f44-a26c-e8a0db5b008b].
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
