[
https://issues.apache.org/jira/browse/CASSANDRA-11995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15644627#comment-15644627
]
Jose Garcia commented on CASSANDRA-11995:
-----------------------------------------
Mine is running on Ubuntu. What I found was that the specific node had a bad
memory stick. The machine in question was crashing constantly, every few hours.
I think when the crash happens at the wrong moment, there is a log being
created and initialized with NULL characters, or probably just allocated to be
sure it gets enough space. When the crash happens at that time, the log stays
allocated with NULL characters and that prevents the cassandra node from
starting, as its format is incorrect. I think what is needed after reboot is to
check for that condition and realize the failire and roll it back. and restart.
In our specific case, we replaced the memory stick, the machine stopped
crashing and I've never seen the error again. Probably the condition is rare
and only happens during bad timing of a crash, but loosing power on a large
cluster would make such condition appear with some frequency.
> Commitlog replaced with all NULs
> --------------------------------
>
> Key: CASSANDRA-11995
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11995
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Windows 10 Enterprise 1511
> DataStax Cassandra Community Server 2.2.3
> Reporter: James Howe
>
> I noticed this morning that Cassandra was failing to start, after being shut
> down on Friday.
> {code}
> ERROR 09:13:37 Exiting due to error while processing commit log during
> initialization.
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
> Could not read commit log descriptor in file C:\Program Files\DataStax
> Community\data\commitlog\CommitLog-5-1465571056722.log
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:622)
> [apache-cassandra-2.2.3.jar:2.2.3]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:302)
> [apache-cassandra-2.2.3.jar:2.2.3]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:147)
> [apache-cassandra-2.2.3.jar:2.2.3]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189)
> [apache-cassandra-2.2.3.jar:2.2.3]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169)
> [apache-cassandra-2.2.3.jar:2.2.3]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:273)
> [apache-cassandra-2.2.3.jar:2.2.3]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:513)
> [apache-cassandra-2.2.3.jar:2.2.3]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:622)
> [apache-cassandra-2.2.3.jar:2.2.3]
> {code}
> Checking the referenced file reveals it comprises 33,554,432 (32 * 1024 *
> 1024) NUL bytes.
> No logs (stdout, stderr, prunsrv) from the shutdown show any other issues and
> appear exactly as normal.
> Is installed as a service via DataStax's distribution.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
