Nilson Pontello created CASSANDRA-12152:
-------------------------------------------
Summary: Unknown exception caught while attempting to update
MaterializedView: AssertionError: Flags = 128
Key: CASSANDRA-12152
URL: https://issues.apache.org/jira/browse/CASSANDRA-12152
Project: Cassandra
Issue Type: Bug
Reporter: Nilson Pontello
I have a single DC with 3 cassandra nodes. After a restart today, none of them
were capable of processing the commitlog while starting up. The exception
doesn't contains enough information about what is going on, please check bellow:
{code}
ERROR [SharedPool-Worker-21] 2016-07-08 12:42:12,866 Keyspace.java:521 -
Unknown exception caught while attempting to update MaterializedView!
data_monitor.user_timeline
java.lang.AssertionError: Flags = 128
at
org.apache.cassandra.db.ClusteringPrefix$Deserializer.prepare(ClusteringPrefix.java:421)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.prepareNext(UnfilteredDeserializer.java:172)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.hasNext(UnfilteredDeserializer.java:153)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.handlePreSliceData(SSTableIterator.java:96)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:141)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:354)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:229)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:32)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:93)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:25)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:419)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:279)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:70)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.SinglePartitionReadCommand.withSSTablesIterated(SinglePartitionReadCommand.java:637)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:586)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:463)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:325)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:366)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.ReadCommand.executeInternal(ReadCommand.java:397)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.service.pager.AbstractQueryPager.fetchPageInternal(AbstractQueryPager.java:77)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.service.pager.SinglePartitionPager.fetchPageInternal(SinglePartitionPager.java:34)
~[apache-cassandra-3.5.jar:3.5]
at org.apache.cassandra.db.view.View.readLocalRows(View.java:565)
~[apache-cassandra-3.5.jar:3.5]
at org.apache.cassandra.db.view.View.getTemporalRowSet(View.java:621)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.view.ViewManager.pushViewReplicaUpdates(ViewManager.java:136)
~[apache-cassandra-3.5.jar:3.5]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:516)
[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.Keyspace.applyFromCommitLog(Keyspace.java:409)
[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.commitlog.CommitLogReplayer$MutationInitiator$1.runMayThrow(CommitLogReplayer.java:157)
[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
[apache-cassandra-3.5.jar:3.5]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_91]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
[apache-cassandra-3.5.jar:3.5]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-3.5.jar:3.5]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
WARN [SharedPool-Worker-21] 2016-07-08 12:42:12,867
AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
Thread[SharedPool-Worker-21,5,main]: {}
java.lang.AssertionError: Flags = 128
at
org.apache.cassandra.db.ClusteringPrefix$Deserializer.prepare(ClusteringPrefix.java:421)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.prepareNext(UnfilteredDeserializer.java:172)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.hasNext(UnfilteredDeserializer.java:153)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.handlePreSliceData(SSTableIterator.java:96)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:141)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:354)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:229)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:32)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:93)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:25)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:419)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:279)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:70)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.SinglePartitionReadCommand.withSSTablesIterated(SinglePartitionReadCommand.java:637)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:586)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:463)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:325)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:366)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.ReadCommand.executeInternal(ReadCommand.java:397)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.service.pager.AbstractQueryPager.fetchPageInternal(AbstractQueryPager.java:77)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.service.pager.SinglePartitionPager.fetchPageInternal(SinglePartitionPager.java:34)
~[apache-cassandra-3.5.jar:3.5]
at org.apache.cassandra.db.view.View.readLocalRows(View.java:565)
~[apache-cassandra-3.5.jar:3.5]
at org.apache.cassandra.db.view.View.getTemporalRowSet(View.java:621)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.view.ViewManager.pushViewReplicaUpdates(ViewManager.java:136)
~[apache-cassandra-3.5.jar:3.5]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:516)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.Keyspace.applyFromCommitLog(Keyspace.java:409)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.commitlog.CommitLogReplayer$MutationInitiator$1.runMayThrow(CommitLogReplayer.java:157)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-3.5.jar:3.5]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_91]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.5.jar:3.5]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-3.5.jar:3.5]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
ERROR [main] 2016-07-08 12:42:12,875 CassandraDaemon.java:697 - Exception
encountered during startup
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.AssertionError: Flags = 128
at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:384)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.commitlog.CommitLogReplayer.replayMutation(CommitLogReplayer.java:592)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:521)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:407)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:236)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:192)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:172)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:283)
[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551)
[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:680)
[apache-cassandra-3.5.jar:3.5]
Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError:
Flags = 128
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.get(AbstractLocalAwareExecutorService.java:200)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:380)
~[apache-cassandra-3.5.jar:3.5]
... 9 common frames omitted
Caused by: java.lang.AssertionError: Flags = 128
at
org.apache.cassandra.db.ClusteringPrefix$Deserializer.prepare(ClusteringPrefix.java:421)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.prepareNext(UnfilteredDeserializer.java:172)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.hasNext(UnfilteredDeserializer.java:153)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.handlePreSliceData(SSTableIterator.java:96)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:141)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:354)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:229)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:32)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:93)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:25)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:419)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:279)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:70)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.SinglePartitionReadCommand.withSSTablesIterated(SinglePartitionReadCommand.java:637)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:586)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:463)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:325)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:366)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.ReadCommand.executeInternal(ReadCommand.java:397)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.service.pager.AbstractQueryPager.fetchPageInternal(AbstractQueryPager.java:77)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.service.pager.SinglePartitionPager.fetchPageInternal(SinglePartitionPager.java:34)
~[apache-cassandra-3.5.jar:3.5]
at org.apache.cassandra.db.view.View.readLocalRows(View.java:565)
~[apache-cassandra-3.5.jar:3.5]
at org.apache.cassandra.db.view.View.getTemporalRowSet(View.java:621)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.view.ViewManager.pushViewReplicaUpdates(ViewManager.java:136)
~[apache-cassandra-3.5.jar:3.5]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:516)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.Keyspace.applyFromCommitLog(Keyspace.java:409)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.db.commitlog.CommitLogReplayer$MutationInitiator$1.runMayThrow(CommitLogReplayer.java:157)
~[apache-cassandra-3.5.jar:3.5]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-3.5.jar:3.5]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_91]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.5.jar:3.5]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
~[apache-cassandra-3.5.jar:3.5]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_91]
{code}
I have played with *commit_failure_policy* hoping that this could be ignored
but it wasn't. I don't mind loosing this materialized view, I can create it
over again if needed.
I am opening this bug because the error message is not informative and I have
no clues of what should be done in order to put my cluster live again...
Also, my system administrator did stop cassandra without running the bellow
commands:
{code}
nodetool disablethrift
nodetool disablegossip
nodetool drain
{code}
I am not sure if this could have affected the pending commitlogs somehow.
Any help will be appreciated
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
