If you already have a regular cadence of repair then you can set "commit_failure_policy" to ignore in cassandra.yaml. So that C* process does not crash on corrupt commit log.
On Fri, Jul 7, 2017 at 2:10 AM, Hannu Kröger <hkro...@gmail.com> wrote: > Hello, > > yes, that’s what we do when things like this happen. > > My thinking is just that when commit log is corrupted, you cannot really > do anything else but exactly those steps. Delete corrupted file and run > repair after starting. At least I haven’t heard of any tools for salvaging > commit log sections. > > Current behaviour gives DBA control over when to do those things and of > course DBA realizes this way that things didn’t go ok but that’s about it. > There is no alternative way of healing the system or anything. > > Hannu > > On 7 July 2017 at 12:03:06, benjamin roth (brs...@gmail.com) wrote: > > Hi Hannu, > > I remember there have been discussions about this in the past. Most > probably there is already a JIRA for this. > I roughly remember a consense like that: > - Default behaviour should remain > - It should be configurable to the needs and preferences of the DBA > - It should at least spit out errors in the logs > > ... of course it would be even better to have the underlying issue fixed > that commit logs should not be corrupt but I remember that this is not so > easy due to some "architectural implications" of Cassandra. IIRC Ed > Capriolo posted something related to that some months ago. > > For a quick fix, I'd recommend: > - Delete the affected log file > - Start the node > - Run a full-range (not -pr) repair on that node > > 2017-07-07 10:57 GMT+02:00 Hannu Kröger <hkro...@gmail.com>: > >> Hello, >> >> We had a test server crashing for some reason (not related to Cassandra >> probably) and now when trying to start cassandra, it gives following error: >> >> ERROR [main] 2017-07-06 09:29:56,140 JVMStabilityInspector.java:82 - >> Exiting due to error while processing commit log during initialization. >> org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException: >> Mutation checksum failure at 24240116 in Next section at 24239690 in >> CommitLog-6-1498576271195.log >> at >> org.apache.cassandra.db.commitlog.CommitLogReader.readSection(CommitLogReader.java:332) >> [apache-cassandra-3.10.jar:3.10] >> at org.apache.cassandra.db.commitlog.CommitLogReader.readCommit >> LogSegment(CommitLogReader.java:201) [apache-cassandra-3.10.jar:3.10] >> at >> org.apache.cassandra.db.commitlog.CommitLogReader.readAllFiles(CommitLogReader.java:84) >> [apache-cassandra-3.10.jar:3.10] >> at org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFi >> les(CommitLogReplayer.java:140) [apache-cassandra-3.10.jar:3.10] >> at >> org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:177) >> [apache-cassandra-3.10.jar:3.10] >> at >> org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:158) >> [apache-cassandra-3.10.jar:3.10] >> at >> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:326) >> [apache-cassandra-3.10.jar:3.10] >> at >> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:601) >> [apache-cassandra-3.10.jar:3.10] >> at >> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:735) >> [apache-cassandra-3.10.jar:3.10] >> >> Shouldn’t Cassandra tolerate this situation? >> >> Of course we can delete commit logs and life goes on. But isn’t this a >> bug or something? >> >> Hannu >> >> >
