[
https://issues.apache.org/jira/browse/CASSANDRA-10315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743623#comment-14743623
]
David Loegering commented on CASSANDRA-10315:
---------------------------------------------
This is very easy to reproduce and is happening on multiple clusters on all
nodes at random times. Just starting thrift and gossip without restarting
Cassandra seems to resolve the issue. I have very complete trace data.
> Cassandra nodes shutting down on COMMIT-LOG-ALLOCATOR error
> -----------------------------------------------------------
>
> Key: CASSANDRA-10315
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10315
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: 16 GB, Cassandra 2.0.15,
> Reporter: David Loegering
>
> After migrating from 2.0.9 to 2.0.15 all nodes on multiple clusters
> Cassandra nodes shutting themselves down every 24-48 hours. The error
> reported is:
> {code}
> ERROR [COMMIT-LOG-ALLOCATOR] 2015-09-11 17:14:48,034 CommitLog.java (line
> 420) Failed to allocate new commit log segments. Commit disk failure policy
> is stop; terminating thread
> java.lang.AssertionError: attempted to delete non-existing file
> CommitLog-3-1441961724221.log
> at
> org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:113)
> at
> org.apache.cassandra.db.commitlog.CommitLogSegment.discard(CommitLogSegment.java:161)
> at
> org.apache.cassandra.db.commitlog.CommitLogAllocator$4.run(CommitLogAllocator.java:228)
> at
> org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:99)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at java.lang.Thread.run(Unknown Source)
> {code}
> We’ve seen this error before, but now with DEBUG / TRACE on, we see that this
> file has 18 entries starting at 5:14pm that look something like this:
> {code}
> DEBUG [COMMIT-LOG-WRITER] 2015-09-11 17:14:45,372 CommitLog.java (line 245)
> Not safe to delete commit log segment
> CommitLogSegment(/opt/osi/monarch/chronus/resources/cassandra/commitlog/CommitLog-3-1441961724221.log);
> dirty is soe (a77b7765-1e3b-30eb-9f46-2cac8dfe1ac7), min_max_avg_hourly
> (f86973a2-f5e6-36b6-9d7f-7fc4e109fb6e),
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
