[
https://issues.apache.org/jira/browse/CASSANDRA-8665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kishan Karunaratne updated CASSANDRA-8665:
------------------------------------------
Description:
During a ruby driver endurance/duration test, the following error occurred:
{noformat}
/mnt/systemlogs/system.log:ERROR [main] 2015-01-17 21:18:25,780
CassandraDaemon.java:482 - Exception encountered during startup
/mnt/systemlogs/system.log-java.lang.NullPointerException: null
/mnt/systemlogs/system.log- at
org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:573)
~[main/:na]
/mnt/systemlogs/system.log- at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:249)
[main/:na]
/mnt/systemlogs/system.log- at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:465)
[main/:na]
/mnt/systemlogs/system.log- at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:554)
[main/:na]
{noformat}
Here is the system.log leading up to the error:
{noformat}
INFO [main] 2015-01-17 21:18:24,581 ColumnFamilyStore.java:278 - Initializing
system.peers
INFO [SSTableBatchOpen:1] 2015-01-17 21:18:24,593 SSTableReader.java:392 -
Opening
/srv/performance/cass/data/system/peers-37f71aca7dc2383ba70672528af04d4f/system-peers-ka-169
(10533 bytes)
INFO [SSTableBatchOpen:1] 2015-01-17 21:18:24,597 SSTableReader.java:392 -
Opening
/srv/performance/cass/data/system/peers-37f71aca7dc2383ba70672528af04d4f/system-peers-ka-171
(10572 bytes)
INFO [SSTableBatchOpen:1] 2015-01-17 21:18:24,598 SSTableReader.java:392 -
Opening
/srv/performance/cass/data/system/peers-37f71aca7dc2383ba70672528af04d4f/system-peers-ka-170
(10581 bytes)
INFO [main] 2015-01-17 21:18:24,609 ColumnFamilyStore.java:278 - Initializing
system.local
INFO [SSTableBatchOpen:1] 2015-01-17 21:18:24,613 SSTableReader.java:392 -
Opening
/srv/performance/cass/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-679
(5257 bytes)
INFO [SSTableBatchOpen:1] 2015-01-17 21:18:24,616 SSTableReader.java:392 -
Opening
/srv/performance/cass/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-678
(5679 bytes)
{noformat}
Cassandra attempted to restart twice unsuccessfully (this error occurred twice)
and then gave up; it seems like a corrupt Data.db file? The endurance test
consists of a chaos rhino which randomly rolling restarts a node.
The only other significant error is TombstoneOverwhelmingException and is
probably unrelated:
{noformat}
/mnt/systemlogs/system.log:ERROR [HintedHandoff:2] 2015-01-17 12:46:32,378
SliceQueryFilter.java:218 - Scanned over 100000 tombstones in system.hints;
query aborted (see tombstone_failure_threshold)
/mnt/systemlogs/system.log:ERROR [HintedHandoff:2] 2015-01-17 12:46:32,416
CassandraDaemon.java:170 - Exception in thread Thread[HintedHandoff:2,1,main]
/mnt/systemlogs/system.log-org.apache.cassandra.db.filter.TombstoneOverwhelmingException:
null
/mnt/systemlogs/system.log- at
org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:220)
~[main/:na]
/mnt/systemlogs/system.log- at
org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:107)
~[main/:na]
/mnt/systemlogs/system.log- at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:81)
~[main/:na]
/mnt/systemlogs/system.log- at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:69)
~[main/:na]
/mnt/systemlogs/system.log- at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:320)
~[main/:na]
/mnt/systemlogs/system.log- at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:62)
~[main/:na]
/mnt/systemlogs/system.log- at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1885)
~[main/:na]
/mnt/systemlogs/system.log- at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1693)
~[main/:na]
/mnt/systemlogs/system.log- at
org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:378)
~[main/:na]
/mnt/systemlogs/system.log- at
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:337)
~[main/:na]
/mnt/systemlogs/system.log- at
org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:88)
~[main/:na]
/mnt/systemlogs/system.log- at
org.apache.cassandra.db.HintedHandOffManager$5.run(HintedHandOffManager.java:548)
~[main/:na]
/mnt/systemlogs/system.log- at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_25]
/mnt/systemlogs/system.log- at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
~[na:1.7.0_25]
/mnt/systemlogs/system.log- at
java.util.concurrent.FutureTask.run(FutureTask.java:166) ~[na:1.7.0_25]
/mnt/systemlogs/system.log- at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
~[na:1.7.0_25]
/mnt/systemlogs/system.log- at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
~[na:1.7.0_25]
/mnt/systemlogs/system.log- at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_25]
/mnt/systemlogs/system.log- at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_25]
/mnt/systemlogs/system.log- at java.lang.Thread.run(Thread.java:724)
~[na:1.7.0_25]
{noformat}
was:
During a ruby driver endurance/duration test, the following error occurred:
{noformat}
/mnt/systemlogs/system.log:ERROR [main] 2015-01-17 21:18:25,780
CassandraDaemon.java:482 - Exception encountered during startup
/mnt/systemlogs/system.log-java.lang.NullPointerException: null
/mnt/systemlogs/system.log- at
org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:573)
~[main/:na]
/mnt/systemlogs/system.log- at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:249)
[main/:na]
/mnt/systemlogs/system.log- at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:465)
[main/:na]
/mnt/systemlogs/system.log- at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:554)
[main/:na]
{noformat}
Here is the system.log leading up to the error:
{noformat}
INFO [main] 2015-01-17 21:18:24,581 ColumnFamilyStore.java:278 - Initializing
system.peers
INFO [SSTableBatchOpen:1] 2015-01-17 21:18:24,593 SSTableReader.java:392 -
Opening
/srv/performance/cass/data/system/peers-37f71aca7dc2383ba70672528af04d4f/system-peers-ka-169
(10533 bytes)
INFO [SSTableBatchOpen:1] 2015-01-17 21:18:24,597 SSTableReader.java:392 -
Opening
/srv/performance/cass/data/system/peers-37f71aca7dc2383ba70672528af04d4f/system-peers-ka-171
(10572 bytes)
INFO [SSTableBatchOpen:1] 2015-01-17 21:18:24,598 SSTableReader.java:392 -
Opening
/srv/performance/cass/data/system/peers-37f71aca7dc2383ba70672528af04d4f/system-peers-ka-170
(10581 bytes)
INFO [main] 2015-01-17 21:18:24,609 ColumnFamilyStore.java:278 - Initializing
system.local
INFO [SSTableBatchOpen:1] 2015-01-17 21:18:24,613 SSTableReader.java:392 -
Opening
/srv/performance/cass/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-679
(5257 bytes)
INFO [SSTableBatchOpen:1] 2015-01-17 21:18:24,616 SSTableReader.java:392 -
Opening
/srv/performance/cass/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-678
(5679 bytes)
{noformat}
Cassandra attempted to restart twice unsuccessfully (this error occurred twice)
and then gave up; it seems like a corrupt Data.db file? The endurance test
consists of a chaos rhino which randomly rolling restarts a node.
> Cassandra does not start with NPE in
> ColumnFamilyStore.removeUnfinishedCompactionLeftovers
> ------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-8665
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8665
> Project: Cassandra
> Issue Type: Bug
> Environment: C* 2.1.2 | ruby-driver 1.2
> Reporter: Kishan Karunaratne
> Fix For: 2.1.3
>
>
> During a ruby driver endurance/duration test, the following error occurred:
> {noformat}
> /mnt/systemlogs/system.log:ERROR [main] 2015-01-17 21:18:25,780
> CassandraDaemon.java:482 - Exception encountered during startup
> /mnt/systemlogs/system.log-java.lang.NullPointerException: null
> /mnt/systemlogs/system.log- at
> org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:573)
> ~[main/:na]
> /mnt/systemlogs/system.log- at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:249)
> [main/:na]
> /mnt/systemlogs/system.log- at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:465)
> [main/:na]
> /mnt/systemlogs/system.log- at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:554)
> [main/:na]
> {noformat}
> Here is the system.log leading up to the error:
> {noformat}
> INFO [main] 2015-01-17 21:18:24,581 ColumnFamilyStore.java:278 -
> Initializing system.peers
> INFO [SSTableBatchOpen:1] 2015-01-17 21:18:24,593 SSTableReader.java:392 -
> Opening
> /srv/performance/cass/data/system/peers-37f71aca7dc2383ba70672528af04d4f/system-peers-ka-169
> (10533 bytes)
> INFO [SSTableBatchOpen:1] 2015-01-17 21:18:24,597 SSTableReader.java:392 -
> Opening
> /srv/performance/cass/data/system/peers-37f71aca7dc2383ba70672528af04d4f/system-peers-ka-171
> (10572 bytes)
> INFO [SSTableBatchOpen:1] 2015-01-17 21:18:24,598 SSTableReader.java:392 -
> Opening
> /srv/performance/cass/data/system/peers-37f71aca7dc2383ba70672528af04d4f/system-peers-ka-170
> (10581 bytes)
> INFO [main] 2015-01-17 21:18:24,609 ColumnFamilyStore.java:278 -
> Initializing system.local
> INFO [SSTableBatchOpen:1] 2015-01-17 21:18:24,613 SSTableReader.java:392 -
> Opening
> /srv/performance/cass/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-679
> (5257 bytes)
> INFO [SSTableBatchOpen:1] 2015-01-17 21:18:24,616 SSTableReader.java:392 -
> Opening
> /srv/performance/cass/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-678
> (5679 bytes)
> {noformat}
> Cassandra attempted to restart twice unsuccessfully (this error occurred
> twice) and then gave up; it seems like a corrupt Data.db file? The endurance
> test consists of a chaos rhino which randomly rolling restarts a node.
> The only other significant error is TombstoneOverwhelmingException and is
> probably unrelated:
> {noformat}
> /mnt/systemlogs/system.log:ERROR [HintedHandoff:2] 2015-01-17 12:46:32,378
> SliceQueryFilter.java:218 - Scanned over 100000 tombstones in system.hints;
> query aborted (see tombstone_failure_threshold)
> /mnt/systemlogs/system.log:ERROR [HintedHandoff:2] 2015-01-17 12:46:32,416
> CassandraDaemon.java:170 - Exception in thread Thread[HintedHandoff:2,1,main]
> /mnt/systemlogs/system.log-org.apache.cassandra.db.filter.TombstoneOverwhelmingException:
> null
> /mnt/systemlogs/system.log- at
> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:220)
> ~[main/:na]
> /mnt/systemlogs/system.log- at
> org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:107)
> ~[main/:na]
> /mnt/systemlogs/system.log- at
> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:81)
> ~[main/:na]
> /mnt/systemlogs/system.log- at
> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:69)
> ~[main/:na]
> /mnt/systemlogs/system.log- at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:320)
> ~[main/:na]
> /mnt/systemlogs/system.log- at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:62)
> ~[main/:na]
> /mnt/systemlogs/system.log- at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1885)
> ~[main/:na]
> /mnt/systemlogs/system.log- at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1693)
> ~[main/:na]
> /mnt/systemlogs/system.log- at
> org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:378)
> ~[main/:na]
> /mnt/systemlogs/system.log- at
> org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:337)
> ~[main/:na]
> /mnt/systemlogs/system.log- at
> org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:88)
> ~[main/:na]
> /mnt/systemlogs/system.log- at
> org.apache.cassandra.db.HintedHandOffManager$5.run(HintedHandOffManager.java:548)
> ~[main/:na]
> /mnt/systemlogs/system.log- at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[na:1.7.0_25]
> /mnt/systemlogs/system.log- at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> ~[na:1.7.0_25]
> /mnt/systemlogs/system.log- at
> java.util.concurrent.FutureTask.run(FutureTask.java:166) ~[na:1.7.0_25]
> /mnt/systemlogs/system.log- at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> ~[na:1.7.0_25]
> /mnt/systemlogs/system.log- at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> ~[na:1.7.0_25]
> /mnt/systemlogs/system.log- at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[na:1.7.0_25]
> /mnt/systemlogs/system.log- at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[na:1.7.0_25]
> /mnt/systemlogs/system.log- at java.lang.Thread.run(Thread.java:724)
> ~[na:1.7.0_25]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
