[
https://issues.apache.org/jira/browse/CASSANDRA-20665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tyler Schade updated CASSANDRA-20665:
-------------------------------------
Description:
Noticing some strange behavior today with a Cassandra 5.0.4 Docker Container
that recently likely encountered a forced shutdown. This is a single-node
"cluster".
{noformat}
INFO [main] 2025-05-20 14:49:37,065 CommitLogReader.java:103 - Skipping
playback of empty log: CommitLog-7-1747751996251.log
INFO [main] 2025-05-20 14:49:37,065 CommitLogReader.java:103 - Skipping
playback of empty log: CommitLog-7-1747752061807.log
INFO [main] 2025-05-20 14:49:37,065 CommitLogReader.java:103 - Skipping
playback of empty log: CommitLog-7-1747752126176.log
INFO [main] 2025-05-20 14:49:37,065 CommitLogReader.java:103 - Skipping
playback of empty log: CommitLog-7-1747752190016.log
INFO [main] 2025-05-20 14:49:37,065 CommitLogReader.java:103 - Skipping
playback of empty log: CommitLog-7-1747752253591.log
INFO [main] 2025-05-20 14:49:37,065 CommitLogReader.java:103 - Skipping
playback of empty log: CommitLog-7-1747752317293.log
INFO [main] 2025-05-20 14:49:37,065 CommitLogReader.java:103 - Skipping
playback of empty log: CommitLog-7-1747752380740.log
INFO [main] 2025-05-20 14:49:37,066 CommitLogReader.java:103 - Skipping
playback of empty log: CommitLog-7-1747752445680.log
INFO [main] 2025-05-20 14:49:37,066 CommitLogReader.java:103 - Skipping
playback of empty log: CommitLog-7-1747752511283.log
ERROR [main] 2025-05-20 14:49:37,067 JVMStabilityInspector.java:201 - Exiting
due to error while processing commit log during
initialization.org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException:
Could not read commit log descriptor in file
/opt/cassandra/data/commitlog/CommitLog-7-1747377011833.log
at
org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:196)
at
org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:147)
at
org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:205)
at
org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:229)
at
org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:208)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:345)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:727)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:865){noformat}
I investigated a little with some help from [~jjirsa] and found that the
segment referenced was empty:
{noformat}
root@09f46103e3e2:/var/lib/cassandra/commitlog# hexdump
CommitLog-7-1747377011833.log
0000000 0000 0000 0000 0000 0000 0000 0000 0000
*
2000000{noformat}
The subsequent segment was _also_ empty:
{noformat}
root@09f46103e3e2:/var/lib/cassandra/commitlog# md5sum $(ls | head)
58f06dd588d8ffb3beb46ada6309436b CommitLog-7-1747377011833.log
58f06dd588d8ffb3beb46ada6309436b CommitLog-7-1747377011834.log
1f9bd5ab1f623fd921282713bf3e34cc CommitLog-7-1747377250204.log
08ed83631a94bdb902ddeb600e8f562a CommitLog-7-1747377250205.log
1093718121a69c2843dd6ed204dc6563 CommitLog-7-1747377266512.log
2a48581389c3bb68a58be3d0d14eb6f5 CommitLog-7-1747377266513.log
24ad307b55b8d5d1d33ddcc4e21395e8 CommitLog-7-1747377272735.log
1e4a7628f5f141c587d2cc94eda3c22b CommitLog-7-1747377272736.log
06f498bc5d35e2691761e39e0a6d5476 CommitLog-7-1747377276330.log
10f9e64659b333ae3b8b3e9424a4468b CommitLog-7-1747377276331.log{noformat}
However, there are 8 subsequent segments with data:
{noformat}
root@09f46103e3e2:/var/lib/cassandra/commitlog# ls -l | head
total 143136
-rw-r--r-- 1 999 999 33554432 May 16 06:30 CommitLog-7-1747377011833.log
-rw-r--r-- 1 999 999 33554432 May 16 06:30 CommitLog-7-1747377011834.log
-rw-r--r-- 1 999 999 33554432 May 16 06:34 CommitLog-7-1747377250204.log
-rw-r--r-- 1 999 999 33554432 May 16 06:34 CommitLog-7-1747377250205.log
-rw-r--r-- 1 999 999 33554432 May 16 06:34 CommitLog-7-1747377266512.log
-rw-r--r-- 1 999 999 33554432 May 16 06:34 CommitLog-7-1747377266513.log
-rw-r--r-- 1 999 999 33554432 May 16 06:34 CommitLog-7-1747377272735.log
-rw-r--r-- 1 999 999 33554432 May 16 06:34 CommitLog-7-1747377272736.log
-rw-r--r-- 1 999 999 33554432 May 16 06:34
CommitLog-7-1747377276330.log{noformat}
was:
Noticing some strange behavior today with a Cassandra 5.0.4 Docker Container
that recently likely encountered a forced shutdown. This is a single-node
"cluster".
{noformat}
INFO [main] 2025-05-20 14:49:37,065 CommitLogReader.java:103 - Skipping
playback of empty log: CommitLog-7-1747752126176.logINFO [main] 2025-05-20
14:49:37,065 CommitLogReader.java:103 - Skipping playback of empty log:
CommitLog-7-1747752190016.logINFO [main] 2025-05-20 14:49:37,065
CommitLogReader.java:103 - Skipping playback of empty log:
CommitLog-7-1747752253591.logINFO [main] 2025-05-20 14:49:37,065
CommitLogReader.java:103 - Skipping playback of empty log:
CommitLog-7-1747752317293.logINFO [main] 2025-05-20 14:49:37,065
CommitLogReader.java:103 - Skipping playback of empty log:
CommitLog-7-1747752380740.logINFO [main] 2025-05-20 14:49:37,066
CommitLogReader.java:103 - Skipping playback of empty log:
CommitLog-7-1747752445680.logINFO [main] 2025-05-20 14:49:37,066
CommitLogReader.java:103 - Skipping playback of empty log:
CommitLog-7-1747752511283.logERROR [main] 2025-05-20 14:49:37,067
JVMStabilityInspector.java:201 - Exiting due to error while processing commit
log during
initialization.org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException:
Could not read commit log descriptor in file
/opt/cassandra/data/commitlog/CommitLog-7-1747377011833.log at
org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:196)
at
org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:147)
at
org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:205)
at
org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:229)
at
org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:208)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:345)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:727)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:865){noformat}
I investigated a little with some help from [~jjirsa] and found that the
segment referenced was empty:
{noformat}
root@09f46103e3e2:/var/lib/cassandra/commitlog# hexdump
CommitLog-7-1747377011833.log
0000000 0000 0000 0000 0000 0000 0000 0000 0000
*
2000000{noformat}
The subsequent segment was _also_ empty:
{noformat}
root@09f46103e3e2:/var/lib/cassandra/commitlog# md5sum $(ls | head)
58f06dd588d8ffb3beb46ada6309436b CommitLog-7-1747377011833.log
58f06dd588d8ffb3beb46ada6309436b CommitLog-7-1747377011834.log
1f9bd5ab1f623fd921282713bf3e34cc CommitLog-7-1747377250204.log
08ed83631a94bdb902ddeb600e8f562a CommitLog-7-1747377250205.log
1093718121a69c2843dd6ed204dc6563 CommitLog-7-1747377266512.log
2a48581389c3bb68a58be3d0d14eb6f5 CommitLog-7-1747377266513.log
24ad307b55b8d5d1d33ddcc4e21395e8 CommitLog-7-1747377272735.log
1e4a7628f5f141c587d2cc94eda3c22b CommitLog-7-1747377272736.log
06f498bc5d35e2691761e39e0a6d5476 CommitLog-7-1747377276330.log
10f9e64659b333ae3b8b3e9424a4468b CommitLog-7-1747377276331.log{noformat}
However, there are 8 subsequent segments with data:
{noformat}
root@09f46103e3e2:/var/lib/cassandra/commitlog# ls -l | head
total 143136
-rw-r--r-- 1 999 999 33554432 May 16 06:30 CommitLog-7-1747377011833.log
-rw-r--r-- 1 999 999 33554432 May 16 06:30 CommitLog-7-1747377011834.log
-rw-r--r-- 1 999 999 33554432 May 16 06:34 CommitLog-7-1747377250204.log
-rw-r--r-- 1 999 999 33554432 May 16 06:34 CommitLog-7-1747377250205.log
-rw-r--r-- 1 999 999 33554432 May 16 06:34 CommitLog-7-1747377266512.log
-rw-r--r-- 1 999 999 33554432 May 16 06:34 CommitLog-7-1747377266513.log
-rw-r--r-- 1 999 999 33554432 May 16 06:34 CommitLog-7-1747377272735.log
-rw-r--r-- 1 999 999 33554432 May 16 06:34 CommitLog-7-1747377272736.log
-rw-r--r-- 1 999 999 33554432 May 16 06:34
CommitLog-7-1747377276330.log{noformat}
> Cassandra 5.0.4 crashing repeatedly on restart
> ----------------------------------------------
>
> Key: CASSANDRA-20665
> URL: https://issues.apache.org/jira/browse/CASSANDRA-20665
> Project: Apache Cassandra
> Issue Type: Bug
> Reporter: Tyler Schade
> Priority: Normal
>
> Noticing some strange behavior today with a Cassandra 5.0.4 Docker Container
> that recently likely encountered a forced shutdown. This is a single-node
> "cluster".
> {noformat}
> INFO [main] 2025-05-20 14:49:37,065 CommitLogReader.java:103 - Skipping
> playback of empty log: CommitLog-7-1747751996251.log
> INFO [main] 2025-05-20 14:49:37,065 CommitLogReader.java:103 - Skipping
> playback of empty log: CommitLog-7-1747752061807.log
> INFO [main] 2025-05-20 14:49:37,065 CommitLogReader.java:103 - Skipping
> playback of empty log: CommitLog-7-1747752126176.log
> INFO [main] 2025-05-20 14:49:37,065 CommitLogReader.java:103 - Skipping
> playback of empty log: CommitLog-7-1747752190016.log
> INFO [main] 2025-05-20 14:49:37,065 CommitLogReader.java:103 - Skipping
> playback of empty log: CommitLog-7-1747752253591.log
> INFO [main] 2025-05-20 14:49:37,065 CommitLogReader.java:103 - Skipping
> playback of empty log: CommitLog-7-1747752317293.log
> INFO [main] 2025-05-20 14:49:37,065 CommitLogReader.java:103 - Skipping
> playback of empty log: CommitLog-7-1747752380740.log
> INFO [main] 2025-05-20 14:49:37,066 CommitLogReader.java:103 - Skipping
> playback of empty log: CommitLog-7-1747752445680.log
> INFO [main] 2025-05-20 14:49:37,066 CommitLogReader.java:103 - Skipping
> playback of empty log: CommitLog-7-1747752511283.log
> ERROR [main] 2025-05-20 14:49:37,067 JVMStabilityInspector.java:201 - Exiting
> due to error while processing commit log during
> initialization.org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException:
> Could not read commit log descriptor in file
> /opt/cassandra/data/commitlog/CommitLog-7-1747377011833.log
> at
> org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:196)
> at
> org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:147)
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:205)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:229)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:208)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:345)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:727)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:865){noformat}
> I investigated a little with some help from [~jjirsa] and found that the
> segment referenced was empty:
> {noformat}
> root@09f46103e3e2:/var/lib/cassandra/commitlog# hexdump
> CommitLog-7-1747377011833.log
> 0000000 0000 0000 0000 0000 0000 0000 0000 0000
> *
> 2000000{noformat}
> The subsequent segment was _also_ empty:
> {noformat}
> root@09f46103e3e2:/var/lib/cassandra/commitlog# md5sum $(ls | head)
> 58f06dd588d8ffb3beb46ada6309436b CommitLog-7-1747377011833.log
> 58f06dd588d8ffb3beb46ada6309436b CommitLog-7-1747377011834.log
> 1f9bd5ab1f623fd921282713bf3e34cc CommitLog-7-1747377250204.log
> 08ed83631a94bdb902ddeb600e8f562a CommitLog-7-1747377250205.log
> 1093718121a69c2843dd6ed204dc6563 CommitLog-7-1747377266512.log
> 2a48581389c3bb68a58be3d0d14eb6f5 CommitLog-7-1747377266513.log
> 24ad307b55b8d5d1d33ddcc4e21395e8 CommitLog-7-1747377272735.log
> 1e4a7628f5f141c587d2cc94eda3c22b CommitLog-7-1747377272736.log
> 06f498bc5d35e2691761e39e0a6d5476 CommitLog-7-1747377276330.log
> 10f9e64659b333ae3b8b3e9424a4468b CommitLog-7-1747377276331.log{noformat}
> However, there are 8 subsequent segments with data:
> {noformat}
> root@09f46103e3e2:/var/lib/cassandra/commitlog# ls -l | head
> total 143136
> -rw-r--r-- 1 999 999 33554432 May 16 06:30 CommitLog-7-1747377011833.log
> -rw-r--r-- 1 999 999 33554432 May 16 06:30 CommitLog-7-1747377011834.log
> -rw-r--r-- 1 999 999 33554432 May 16 06:34 CommitLog-7-1747377250204.log
> -rw-r--r-- 1 999 999 33554432 May 16 06:34 CommitLog-7-1747377250205.log
> -rw-r--r-- 1 999 999 33554432 May 16 06:34 CommitLog-7-1747377266512.log
> -rw-r--r-- 1 999 999 33554432 May 16 06:34 CommitLog-7-1747377266513.log
> -rw-r--r-- 1 999 999 33554432 May 16 06:34 CommitLog-7-1747377272735.log
> -rw-r--r-- 1 999 999 33554432 May 16 06:34 CommitLog-7-1747377272736.log
> -rw-r--r-- 1 999 999 33554432 May 16 06:34
> CommitLog-7-1747377276330.log{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
