[
https://issues.apache.org/jira/browse/CASSANDRA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17022861#comment-17022861
]
Ankitha commented on CASSANDRA-15263:
-------------------------------------
Hi Benedict,
After a month of upgrade to 3.11.4 on our production systems , we are still
seeing some exceptions in the log. Some of the issues are:
# When we are bringing up the Cassandra process it is sometimes hanging at
reading the Key-cache. We are clearing the key - cache and starting it back up
as a work around.
# During start up mutation stage is logging following warning :
{code:java}
WARN [MutationStage-12] 2020-01-24 02:52:41,264
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[MutationStage-12,5,main]: {}
java.lang.AssertionError: null
at
org.apache.cassandra.utils.memory.AbstractAllocator.clone(AbstractAllocator.java:35)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.RangeTombstoneList.clone(RangeTombstoneList.java:130)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.RangeTombstoneList.copy(RangeTombstoneList.java:119)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.MutableDeletionInfo.copy(MutableDeletionInfo.java:90)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.MutableDeletionInfo.copy(MutableDeletionInfo.java:33)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.partitions.AtomicBTreePartition.addAllWithSizeDelta(AtomicBTreePartition.java:141)
~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.db.Memtable.put(Memtable.java:282)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1352)
~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:626)
~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:470)
~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.db.Mutation.apply(Mutation.java:227)
~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.db.Mutation.apply(Mutation.java:232)
~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.db.Mutation.apply(Mutation.java:241)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.ReadRepairVerbHandler.doVerb(ReadRepairVerbHandler.java:28)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0-internal]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114)
[apache-cassandra-3.11.4.jar:3.11.4]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0-internal]
WARN [MutationStage-18] 2020-01-24 02:52:44,625
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[MutationStage-18,5,main]: {}
java.lang.AssertionError: null
{code}
# We are also seeing the following WARN for read stage:
{code:java}
WARN [ReadStage-88] 2019-12-03 06:42:04,289
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-88,5,main]: {}java.lang.IllegalStateException:
UnfilteredRowIterator for SAL.sal_purge has an open RT bound as its last item
at
org.apache.cassandra.db.transform.RTBoundCloser$RowsTransformation.moreContents(RTBoundCloser.java:109)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.db.transform.RTBoundCloser$RowsTransformation.moreContents(RTBoundCloser.java:63)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.db.transform.BaseIterator.tryGetMoreContents(BaseIterator.java:121)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.db.transform.BaseIterator.hasMoreContents(BaseIterator.java:111)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:159)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.db.rows.UnfilteredRowIterators.digest(UnfilteredRowIterators.java:204)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators.digest(UnfilteredPartitionIterators.java:263)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.db.ReadResponse.makeDigest(ReadResponse.java:140)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.db.ReadResponse.createDigestResponse(ReadResponse.java:87)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:352)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:50)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
~[apache-cassandra-3.11.4.jar:3.11.4] at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_131] at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114)
[apache-cassandra-3.11.4.jar:3.11.4] at
java.lang.Thread.run(Thread.java:748) [na:1.8.0_131] Suppressed:
java.lang.IllegalStateException: PROCESSED UnfilteredRowIterator for
SAL.sal_purge has an illegal RT bounds sequence: expected all RTs to be closed,
but the last one is open at
org.apache.cassandra.db.transform.RTBoundValidator$RowsTransformation.ise(RTBoundValidator.java:120)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.db.transform.RTBoundValidator$RowsTransformation.onPartitionClose(RTBoundValidator.java:113)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.db.transform.BaseRows.runOnClose(BaseRows.java:91)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:86)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators.digest(UnfilteredPartitionIterators.java:264)
~[apache-cassandra-3.11.4.jar:3.11.4] ... 10 common frames
omitted
{code}
# In some cases like below , gossip is shutting down and when trying to bring
up the node it is getting stuck:
#
{code:java}
[StorageServiceShutdownHook] 2020-01-24 02:19:30,586 HintsService.java:209 -
Paused hints dispatch
INFO [HintsDispatcher:14410] 2020-01-24 02:19:30,593
HintsDispatchExecutor.java:289 - Finished hinted handoff of file
594f065a-3134-4a39-b00d-b87b0e4625ff-1579831570632-1.hints to endpoint
/10.177.56.125: 594f065a-3134-4a39-b00d-b87b0e4625ff, partially
INFO [StorageServiceShutdownHook] 2020-01-24 02:19:30,623 Server.java:176 -
Stop listening for CQL clients
INFO [StorageServiceShutdownHook] 2020-01-24 02:19:30,624 Gossiper.java:1551 -
Announcing shutdown
INFO [StorageServiceShutdownHook] 2020-01-24 02:19:30,625
StorageService.java:2327 - Node ont-dce-cass-sal05-priv/10.103.56.25 state jump
to shutdown
INFO [HintsDispatcher:14411] 2020-01-24 02:19:30,682
HintsDispatchExecutor.java:289 - Finished hinted handoff of file
b224b069-5b16-42f9-971b-6ae8f8bbcf23-1579829423673-1.hints to endpoint
/10.177.56.116: b224b069-5b16-42f9-971b-6ae8f8bbcf23, partially
INFO [StorageServiceShutdownHook] 2020-01-24 02:19:32,629
MessagingService.java:981 - Waiting for messaging service to quiesce
INFO [ACCEPT-ont-dce-cass-sal05-priv/10.103.56.25] 2020-01-24 02:19:32,631
MessagingService.java:1336 - MessagingService has terminated the accept() thread
INFO [main] 2020-01-24 02:20:06,214 YamlConfigurationLoader.java:89 -
Configuration location:
file:/opt/cass/apache-cassandra-3.11.4/conf/cassandra.yamlWARN
[MutationStage-12] 2020-01-24 02:24:57,608
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[MutationStage-12,5,main]: {}
java.lang.AssertionError: null
at
org.apache.cassandra.utils.memory.AbstractAllocator.clone(AbstractAllocator.java:35)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.RangeTombstoneList.clone(RangeTombstoneList.java:130)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.RangeTombstoneList.copy(RangeTombstoneList.java:119)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.MutableDeletionInfo.copy(MutableDeletionInfo.java:90)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.MutableDeletionInfo.copy(MutableDeletionInfo.java:33)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.partitions.AtomicBTreePartition.addAllWithSizeDelta(AtomicBTreePartition.java:141)
~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.db.Memtable.put(Memtable.java:282)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1352)
~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:626)
~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:470)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.commitlog.CommitLogReplayer$MutationInitiator$1.runMayThrow(CommitLogReplayer.java:224)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0-internal]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114)
[apache-cassandra-3.11.4.jar:3.11.4]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0-internal]
ERROR [main] 2020-01-24 02:24:57,611 CassandraDaemon.java:749 - Exception
encountered during startup
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.AssertionError
{code}
> LegacyLayout RangeTombstoneList throws java.lang.NullPointerException: null
> ---------------------------------------------------------------------------
>
> Key: CASSANDRA-15263
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15263
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Schema
> Reporter: feroz shaik
> Assignee: Benedict Elliott Smith
> Priority: Normal
> Labels: 2.1.16, 3.11.4
> Attachments: sample.system.log, schema.txt,
> sstabledump_sal_purge_d03.json, sstablemetadata_sal_purge_d03,
> stack_trace.txt, system.log, system.log, system.log, system.log,
> system_latest.log
>
>
> We have hit a problem today while upgrading from 2.1.16 to 3.11.4.
> we encountered this as soon as the first node started up with 3.11.4
> The full error stack is attached - [^stack_trace.txt]
>
> The below errors continued in the log file as long as the process was up.
> ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778
> ErrorMessage.java:384 - Unexpected exception during request
> java.lang.NullPointerException: null
> ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454
>
> The nodetool version says 3.11.4 and the no of connections on native por t-
> 9042 was similar to other nodes. The exceptions were scary that we had to
> call off the change. Any help and insights to this problem from the community
> is appreciated.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
