Re: Startup fails - 4.1.0

2023-02-03 Thread Joe Obernberger 

Thank you Sean.  I had to remove two of the files and then it started.
Cheers!

-Joe

On 2/3/2023 3:52 PM, Durity, Sean R via user wrote:


In most cases, I would delete the corrupt commit log file and restart. 
Then run repairs on that node. I have seen cases where multiple files 
are corrupted and it is easier to remove all commit log files to get 
the node restarted.


Sean R. Durity

*From:*Joe Obernberger 
*Sent:* Friday, February 3, 2023 3:15 PM
*To:* user@cassandra.apache.org
*Subject:* [EXTERNAL] Startup fails - 4.1.0

Hi all - cluster had a power outage and one of the nodes in a 14 nodes 
cluster isn't starting with: DEBUG [MemtableFlushWriter: 1] 2023-02-03 
13: 52: 45,468 ColumnFamilyStore. java: 1329 - Flushed to 
[BigTableReader(path='/data/2/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8407-big-Data. db'),


INTERNAL USE

Hi all - cluster had a power outage and one of the nodes in a 14 nodes
cluster isn't starting with:
DEBUG [MemtableFlushWriter:1] 2023-02-03 13:52:45,468
ColumnFamilyStore.java:1329 - Flushed to
[BigTableReader(path='/data/2/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8407-big-Data.db'), 

BigTableReader(path='/data/3/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8408-big-Data.db'), 

BigTableReader(path='/data/4/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8409-big-Data.db'), 

BigTableReader(path='/data/5/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8410-big-Data.db'), 

BigTableReader(path='/data/6/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8411-big-Data.db'), 

BigTableReader(path='/data/8/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8412-big-Data.db'), 

BigTableReader(path='/data/9/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8413-big-Data.db')] 


(7 sstables, 92.858MiB), biggest 15.420MiB, smallest 10.307MiB
INFO  [main] 2023-02-03 13:52:45,621 CommitLogReader.java:257 - Finished
reading /var/lib/cassandra/commitlog/CommitLog-7-1674161126163.log
DEBUG [main] 2023-02-03 13:52:45,622 CommitLogReader.java:266 - Reading
/var/lib/cassandra/commitlog/CommitLog-7-1674161126164.log (CL version
7, messaging version 12, compression null)
INFO  [main] 2023-02-03 13:52:46,811 CommitLogReader.java:257 - Finished
reading /var/lib/cassandra/commitlog/CommitLog-7-1674161126164.log
DEBUG [main] 2023-02-03 13:52:46,811 CommitLogReader.java:266 - Reading
/var/lib/cassandra/commitlog/CommitLog-7-1674161126165.log (CL version
7, messaging version 12, compression null)
INFO  [main] 2023-02-03 13:52:47,985 CommitLogReader.java:257 - Finished
reading /var/lib/cassandra/commitlog/CommitLog-7-1674161126165.log
DEBUG [main] 2023-02-03 13:52:47,986 CommitLogReader.java:266 - Reading
/var/lib/cassandra/commitlog/CommitLog-7-1674161126166.log (CL version
7, messaging version 12, compression null)
INFO  [main] 2023-02-03 13:52:49,282 CommitLogReader.java:257 - Finished
reading /var/lib/cassandra/commitlog/CommitLog-7-1674161126166.log
DEBUG [main] 2023-02-03 13:52:49,283 CommitLogReader.java:266 - Reading
/var/lib/cassandra/commitlog/CommitLog-7-1674161126167.log (CL version
7, messaging version 12, compression null)
ERROR [main] 2023-02-03 13:52:49,651 JVMStabilityInspector.java:196 -
Exiting due to error while processing commit log during initialization.
org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException: 


Mutation checksum failure at 11231154 in Next section at 11230925 in
CommitLog-7-1674161126167.log
    at
org.apache.cassandra.db.commitlog.CommitLogReader.readSection(CommitLogReader.java:387)
    at
org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:244)
    at
org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:147)
    at
org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:191)
    at
org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:200)
    at
org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:181)
    at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:357)
    at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:752)
    at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:876)
How to proceed?
Thank you!
-Joe
--
This email has been checked for viruses by AVG antivirus software.
https://urldefense.com/v3/__http://www.avg.com__;!!M-nmYVHPHQ!Js4daAIOfZ53leb5CNpzGIRZEtXKw9d4dmRxBCGsYe-51J4i2NkrYuazzFt8O5U-KdQ3HCo9xu4_AeqVYBNySKpz31KzFe0cOQ$ 
<https://urldefense.com/v3/__http:/www.avg.com__;!!M-nmYVHPHQ!Js4daAIOfZ53leb5CNpzGIRZEtXKw9d4dmRxBCGsYe-51J4i2NkrYuazzFt8O5U-KdQ3HCo9xu4_AeqVYBNySKpz31KzFe0cOQ$>

RE: Startup fails - 4.1.0

2023-02-03 Thread Durity, Sean R via user 
In most cases, I would delete the corrupt commit log file and restart. Then run 
repairs on that node. I have seen cases where multiple files are corrupted and 
it is easier to remove all commit log files to get the node restarted.

Sean R. Durity
From: Joe Obernberger 
Sent: Friday, February 3, 2023 3:15 PM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Startup fails - 4.1.0

Hi all - cluster had a power outage and one of the nodes in a 14 nodes cluster 
isn't starting with: DEBUG [MemtableFlushWriter: 1] 2023-02-03 13: 52: 45,468 
ColumnFamilyStore. java: 1329 - Flushed to 
[BigTableReader(path='/data/2/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8407-big-Data.
 db'),



INTERNAL USE

Hi all - cluster had a power outage and one of the nodes in a 14 nodes

cluster isn't starting with:



DEBUG [MemtableFlushWriter:1] 2023-02-03 13:52:45,468

ColumnFamilyStore.java:1329 - Flushed to

[BigTableReader(path='/data/2/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8407-big-Data.db'),

BigTableReader(path='/data/3/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8408-big-Data.db'),

BigTableReader(path='/data/4/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8409-big-Data.db'),

BigTableReader(path='/data/5/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8410-big-Data.db'),

BigTableReader(path='/data/6/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8411-big-Data.db'),

BigTableReader(path='/data/8/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8412-big-Data.db'),

BigTableReader(path='/data/9/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8413-big-Data.db')]

(7 sstables, 92.858MiB), biggest 15.420MiB, smallest 10.307MiB

INFO  [main] 2023-02-03 13:52:45,621 CommitLogReader.java:257 - Finished

reading /var/lib/cassandra/commitlog/CommitLog-7-1674161126163.log

DEBUG [main] 2023-02-03 13:52:45,622 CommitLogReader.java:266 - Reading

/var/lib/cassandra/commitlog/CommitLog-7-1674161126164.log (CL version

7, messaging version 12, compression null)

INFO  [main] 2023-02-03 13:52:46,811 CommitLogReader.java:257 - Finished

reading /var/lib/cassandra/commitlog/CommitLog-7-1674161126164.log

DEBUG [main] 2023-02-03 13:52:46,811 CommitLogReader.java:266 - Reading

/var/lib/cassandra/commitlog/CommitLog-7-1674161126165.log (CL version

7, messaging version 12, compression null)

INFO  [main] 2023-02-03 13:52:47,985 CommitLogReader.java:257 - Finished

reading /var/lib/cassandra/commitlog/CommitLog-7-1674161126165.log

DEBUG [main] 2023-02-03 13:52:47,986 CommitLogReader.java:266 - Reading

/var/lib/cassandra/commitlog/CommitLog-7-1674161126166.log (CL version

7, messaging version 12, compression null)

INFO  [main] 2023-02-03 13:52:49,282 CommitLogReader.java:257 - Finished

reading /var/lib/cassandra/commitlog/CommitLog-7-1674161126166.log

DEBUG [main] 2023-02-03 13:52:49,283 CommitLogReader.java:266 - Reading

/var/lib/cassandra/commitlog/CommitLog-7-1674161126167.log (CL version

7, messaging version 12, compression null)

ERROR [main] 2023-02-03 13:52:49,651 JVMStabilityInspector.java:196 -

Exiting due to error while processing commit log during initialization.

org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException:

Mutation checksum failure at 11231154 in Next section at 11230925 in

CommitLog-7-1674161126167.log

 at

org.apache.cassandra.db.commitlog.CommitLogReader.readSection(CommitLogReader.java:387)

 at

org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:244)

 at

org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:147)

 at

org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:191)

 at

org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:200)

 at

org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:181)

 at

org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:357)

 at

org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:752)

 at

org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:876)



How to proceed?

Thank you!



-Joe





--

This email has been checked for viruses by AVG antivirus software.

https://urldefense.com/v3/__http://www.avg.com__;!!M-nmYVHPHQ!Js4daAIOfZ53leb5CNpzGIRZEtXKw9d4dmRxBCGsYe-51J4i2NkrYuazzFt8O5U-KdQ3HCo9xu4_AeqVYBNySKpz31KzFe0cOQ$<https://urldefense.com/v3/__http:/www.avg.com__;!!M-nmYVHPHQ!Js4daAIOfZ53leb5CNpzGIRZEtXKw9d4dmRxBCGsYe-51J4i2NkrYuazzFt8O5U-KdQ3HCo9xu4_AeqVYBNySKpz31KzFe0cOQ$>

Startup fails - 4.1.0

2023-02-03 Thread Joe Obernberger 
Hi all - cluster had a power outage and one of the nodes in a 14 nodes 
cluster isn't starting with:


DEBUG [MemtableFlushWriter:1] 2023-02-03 13:52:45,468 
ColumnFamilyStore.java:1329 - Flushed to 
[BigTableReader(path='/data/2/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8407-big-Data.db'), 
BigTableReader(path='/data/3/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8408-big-Data.db'), 
BigTableReader(path='/data/4/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8409-big-Data.db'), 
BigTableReader(path='/data/5/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8410-big-Data.db'), 
BigTableReader(path='/data/6/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8411-big-Data.db'), 
BigTableReader(path='/data/8/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8412-big-Data.db'), 
BigTableReader(path='/data/9/cassandra/data/doc/correlation_counts-ca4e8c0080b311edbd6d4d9b3bfd78b2/nb-8413-big-Data.db')] 
(7 sstables, 92.858MiB), biggest 15.420MiB, smallest 10.307MiB
INFO  [main] 2023-02-03 13:52:45,621 CommitLogReader.java:257 - Finished 
reading /var/lib/cassandra/commitlog/CommitLog-7-1674161126163.log
DEBUG [main] 2023-02-03 13:52:45,622 CommitLogReader.java:266 - Reading 
/var/lib/cassandra/commitlog/CommitLog-7-1674161126164.log (CL version 
7, messaging version 12, compression null)
INFO  [main] 2023-02-03 13:52:46,811 CommitLogReader.java:257 - Finished 
reading /var/lib/cassandra/commitlog/CommitLog-7-1674161126164.log
DEBUG [main] 2023-02-03 13:52:46,811 CommitLogReader.java:266 - Reading 
/var/lib/cassandra/commitlog/CommitLog-7-1674161126165.log (CL version 
7, messaging version 12, compression null)
INFO  [main] 2023-02-03 13:52:47,985 CommitLogReader.java:257 - Finished 
reading /var/lib/cassandra/commitlog/CommitLog-7-1674161126165.log
DEBUG [main] 2023-02-03 13:52:47,986 CommitLogReader.java:266 - Reading 
/var/lib/cassandra/commitlog/CommitLog-7-1674161126166.log (CL version 
7, messaging version 12, compression null)
INFO  [main] 2023-02-03 13:52:49,282 CommitLogReader.java:257 - Finished 
reading /var/lib/cassandra/commitlog/CommitLog-7-1674161126166.log
DEBUG [main] 2023-02-03 13:52:49,283 CommitLogReader.java:266 - Reading 
/var/lib/cassandra/commitlog/CommitLog-7-1674161126167.log (CL version 
7, messaging version 12, compression null)
ERROR [main] 2023-02-03 13:52:49,651 JVMStabilityInspector.java:196 - 
Exiting due to error while processing commit log during initialization.
org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException: 
Mutation checksum failure at 11231154 in Next section at 11230925 in 
CommitLog-7-1674161126167.log
    at 
org.apache.cassandra.db.commitlog.CommitLogReader.readSection(CommitLogReader.java:387)
    at 
org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:244)
    at 
org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:147)
    at 
org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:191)
    at 
org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:200)
    at 
org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:181)
    at 
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:357)
    at 
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:752)
    at 
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:876)


How to proceed?
Thank you!

-Joe


--
This email has been checked for viruses by AVG antivirus software.
www.avg.com