[
https://issues.apache.org/jira/browse/CASSANDRA-15529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pooja Nair updated CASSANDRA-15529:
-----------------------------------
Description:
Hello Team,
We have cluster running on cassandra 3.11.4
Following is the table schema of the tables that is being used in our system.
{code:java}
cqlsh> desc KEYSPACE "SAL"
CREATE KEYSPACE "SAL" WITH replication = {'class': 'NetworkTopologyStrategy',
'DC_EAST': '3', 'DC_WEST': '3'} AND durable_writes = true;
CREATE TABLE "SAL".sal_purge (
key text,
column1 text,
column2 text,
value text,
PRIMARY KEY (key, column1, column2)
) WITH COMPACT STORAGE
AND CLUSTERING ORDER BY (column1 ASC, column2 ASC)
AND bloom_filter_fp_chance = 0.1
AND caching = '{"keys":"NONE", "rows_per_partition":"NONE"}'
AND comment = 'Holds items to be removed as [shardid][salid][timestamp].
The table records SALIDs to be deleted along with their deletion times (which
may be modified)'
AND compaction = {'class':
'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
AND compression = {'chunk_length_kb': '64', 'sstable_compression':
'org.apache.cassandra.io.compress.SnappyCompressor'}
AND dclocal_read_repair_chance = 0.0
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.1
AND speculative_retry = '99.0PERCENTILE';
CREATE TABLE "SAL".sal_ref (
key text,
column1 text,
column2 text,
value text,
PRIMARY KEY (key, column1, column2)
) WITH COMPACT STORAGE
AND CLUSTERING ORDER BY (column1 ASC, column2 ASC)
AND bloom_filter_fp_chance = 0.025
AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
AND comment = 'Holds owner references to content as [salid][lcid/opid]'
AND compaction = {'sstable_size_in_mb': '180', 'class':
'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
AND compression = {'chunk_length_kb': '64', 'sstable_compression':
'org.apache.cassandra.io.compress.SnappyCompressor'}
AND dclocal_read_repair_chance = 0.0
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99.0PERCENTILE';
{code}
Things to note:
# The column2 is always passed a null value during insertion
# column2 is a part of primary key
# Range select and Range delete is done through our app.
Iniatally the cluster was on casssandra version 2.1.16 and have been recently
upgraded to 3.11.4 post the upgrade, we see that the nodes are going down, and
log the below exceptions during startup and even after node is up. This one
node is causing the whole cluster to behave improperly.
{code:java}
WARN [Native-Transport-Requests-47] 2020-01-29 13:49:05,190
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[Native-Transport-Requests-47,5,main]: {} java.lang.RuntimeException:
java.lang.IllegalStateException: UnfilteredRowIterator for SAL.sal_purge has an
open RT bound as its last item at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2588)
~[apache-cassandra-3.11.4.jar:3.11.4] at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0-internal] at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.concurrent.SEPExecutor.maybeExecuteImmediately(SEPExecutor.java:194)
[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.service.AbstractReadExecutor.makeRequests(AbstractReadExecutor.java:117)
[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.service.AbstractReadExecutor.makeDataRequests(AbstractReadExecutor.java:85)
[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.service.AbstractReadExecutor$SpeculatingReadExecutor.executeAsync(AbstractReadExecutor.java:271)
[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.doInitialQueries(StorageProxy.java:1778)
[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1731)
[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1671)
[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1586)
[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:1209)
[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:315)
[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:285)
[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:117)
[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:225)
[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:532)
[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:509)
[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:146)
[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:566)
[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410)
[apache-cassandra-3.11.4.jar:3.11.4] at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
[netty-all-4.0.44.Final.jar:4.0.44.Final] at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
[netty-all-4.0.44.Final.jar:4.0.44.Final] at
io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)
[netty-all-4.0.44.Final.jar:4.0.44.Final] at
io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348)
[netty-all-4.0.44.Final.jar:4.0.44.Final] at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0-internal] at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114)
[apache-cassandra-3.11.4.jar:3.11.4] at java.lang.Thread.run(Thread.java:748)
[na:1.8.0-internal] Caused by: java.lang.IllegalStateException:
UnfilteredRowIterator for SAL.sal_purge has an open RT bound as its last item
at
org.apache.cassandra.db.transform.RTBoundCloser$RowsTransformation.moreContents(RTBoundCloser.java:109)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.db.transform.RTBoundCloser$RowsTransformation.moreContents(RTBoundCloser.java:63)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.db.transform.BaseIterator.tryGetMoreContents(BaseIterator.java:121)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.db.transform.BaseIterator.hasMoreContents(BaseIterator.java:111)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:159)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:136)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:92)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:79)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:308)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:187)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:180)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:176)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:353)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1876)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2584)
~[apache-cassandra-3.11.4.jar:3.11.4] ... 29 common frames omitted Suppressed:
java.lang.IllegalStateException: PROCESSED UnfilteredRowIterator for
SAL.sal_purge has an illegal RT bounds sequence: expected all RTs to be closed,
but the last one is open at
org.apache.cassandra.db.transform.RTBoundValidator$RowsTransformation.ise(RTBoundValidator.java:120)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.db.transform.RTBoundValidator$RowsTransformation.onPartitionClose(RTBoundValidator.java:113)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.db.transform.BaseRows.runOnClose(BaseRows.java:91)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:86)
~[apache-cassandra-3.11.4.jar:3.11.4] at
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:309)
~[apache-cassandra-3.11.4.jar:3.11.4] ... 36 common frames omitted
{code}
was:
Hello Team,
For the background of this problem, kindly refer this ticket
https://issues.apache.org/jira/browse/CASSANDRA-15263
++
It was proposed that after the upgrade of the cluster from 2.1.16 to 3.11.4, we
should no longer be seeing any type of exceptions (WARN/ERROR).
But even after a month of upgrade we still see the below exceptions
{code:java}
WARN [ReadStage-231] 2020-01-23 02:29:11,137
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-231,5,main]: {}
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
[apache-cassandra-3.11.4.jar:3.11.4]
{code}
*and* **
{code:java}
WARN [MutationStage-36] 2020-01-21 19:31:03,343
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[MutationStage-36,5,main]: {}
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
[apache-cassandra-3.11.4.jar:3.11.4] .{code}
*Problem:*
If one of the node goes down, and we try to bring it up, the node hangs at a
stage of reading Key-cache. Sometimes the node comes up after a long period of
pause with the above *Mutation-stage* warnings. Other times , the node does not
come up and we have to manually clear the Key-cache to bring up the node.
Also,once the node is up we see lot of above *Read-Stage* warnings.
In some cases like below , gossip is shutting down and when trying to bring up
the node it is getting stuck:
{code:java}
[StorageServiceShutdownHook] 2020-01-24 02:19:30,586 HintsService.java:209 -
Paused hints dispatch
INFO [HintsDispatcher:14410] 2020-01-24 02:19:30,593
HintsDispatchExecutor.java:289 - Finished hinted handoff of file
594f065a-3134-4a39-b00d-b87b0e4625ff-1579831570632-1.hints to endpoint
/10.177.56.125: 594f065a-3134-4a39-b00d-b87b0e4625ff, partially
INFO [StorageServiceShutdownHook] 2020-01-24 02:19:30,623 Server.java:176 -
Stop listening for CQL clients
INFO [StorageServiceShutdownHook] 2020-01-24 02:19:30,624 Gossiper.java:1551 -
Announcing shutdown
INFO [StorageServiceShutdownHook] 2020-01-24 02:19:30,625
StorageService.java:2327 - Node ont-dce-cass-sal05-priv/10.103.56.25 state jump
to shutdown
INFO [HintsDispatcher:14411] 2020-01-24 02:19:30,682
HintsDispatchExecutor.java:289 - Finished hinted handoff of file
b224b069-5b16-42f9-971b-6ae8f8bbcf23-1579829423673-1.hints to endpoint
/10.177.56.116: b224b069-5b16-42f9-971b-6ae8f8bbcf23, partially
INFO [StorageServiceShutdownHook] 2020-01-24 02:19:32,629
MessagingService.java:981 - Waiting for messaging service to quiesce
INFO [ACCEPT-ont-dce-cass-sal05-priv/10.103.56.25] 2020-01-24 02:19:32,631
MessagingService.java:1336 - MessagingService has terminated the accept() thread
INFO [main] 2020-01-24 02:20:06,214 YamlConfigurationLoader.java:89 -
Configuration location:
file:/opt/cass/apache-cassandra-3.11.4/conf/cassandra.yamlWARN
[MutationStage-12] 2020-01-24 02:24:57,608
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[MutationStage-12,5,main]: {}
java.lang.AssertionError: null at
org.apache.cassandra.utils.memory.AbstractAllocator.clone(AbstractAllocator.java:35)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.RangeTombstoneList.clone(RangeTombstoneList.java:130)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.RangeTombstoneList.copy(RangeTombstoneList.java:119)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.MutableDeletionInfo.copy(MutableDeletionInfo.java:90)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.MutableDeletionInfo.copy(MutableDeletionInfo.java:33)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.partitions.AtomicBTreePartition.addAllWithSizeDelta(AtomicBTreePartition.java:141)
~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.db.Memtable.put(Memtable.java:282)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1352)
~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:626)
~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:470)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.commitlog.CommitLogReplayer$MutationInitiator$1.runMayThrow(CommitLogReplayer.java:224)
~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-3.11.4.jar:3.11.4]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0-internal]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114)
[apache-cassandra-3.11.4.jar:3.11.4]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0-internal]
ERROR [main] 2020-01-24 02:24:57,611 CassandraDaemon.java:749 - Exception
encountered during startup
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.AssertionError{code}
> AbstractLocalAwareExecutorService.java exceptions after upgrade from 2.1.16
> to 3.11.4
> -------------------------------------------------------------------------------------
>
> Key: CASSANDRA-15529
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15529
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Schema
> Reporter: Pooja Nair
> Priority: Urgent
> Labels: 2.1.16, 3.11.4
>
> Hello Team,
> We have cluster running on cassandra 3.11.4
> Following is the table schema of the tables that is being used in our system.
> {code:java}
> cqlsh> desc KEYSPACE "SAL"
>
> CREATE KEYSPACE "SAL" WITH replication = {'class':
> 'NetworkTopologyStrategy', 'DC_EAST': '3', 'DC_WEST': '3'} AND
> durable_writes = true;
>
> CREATE TABLE "SAL".sal_purge (
> key text,
> column1 text,
> column2 text,
> value text,
> PRIMARY KEY (key, column1, column2)
> ) WITH COMPACT STORAGE
> AND CLUSTERING ORDER BY (column1 ASC, column2 ASC)
> AND bloom_filter_fp_chance = 0.1
> AND caching = '{"keys":"NONE", "rows_per_partition":"NONE"}'
> AND comment = 'Holds items to be removed as
> [shardid][salid][timestamp]. The table records SALIDs to be deleted along
> with their deletion times (which may be modified)'
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'chunk_length_kb': '64', 'sstable_compression':
> 'org.apache.cassandra.io.compress.SnappyCompressor'}
> AND dclocal_read_repair_chance = 0.0
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.1
> AND speculative_retry = '99.0PERCENTILE';
>
> CREATE TABLE "SAL".sal_ref (
> key text,
> column1 text,
> column2 text,
> value text,
> PRIMARY KEY (key, column1, column2)
> ) WITH COMPACT STORAGE
> AND CLUSTERING ORDER BY (column1 ASC, column2 ASC)
> AND bloom_filter_fp_chance = 0.025
> AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
> AND comment = 'Holds owner references to content as [salid][lcid/opid]'
> AND compaction = {'sstable_size_in_mb': '180', 'class':
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'chunk_length_kb': '64', 'sstable_compression':
> 'org.apache.cassandra.io.compress.SnappyCompressor'}
> AND dclocal_read_repair_chance = 0.0
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99.0PERCENTILE';
> {code}
> Things to note:
> # The column2 is always passed a null value during insertion
> # column2 is a part of primary key
> # Range select and Range delete is done through our app.
> Iniatally the cluster was on casssandra version 2.1.16 and have been
> recently upgraded to 3.11.4 post the upgrade, we see that the nodes are going
> down, and log the below exceptions during startup and even after node is up.
> This one node is causing the whole cluster to behave improperly.
> {code:java}
> WARN [Native-Transport-Requests-47] 2020-01-29 13:49:05,190
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[Native-Transport-Requests-47,5,main]: {} java.lang.RuntimeException:
> java.lang.IllegalStateException: UnfilteredRowIterator for SAL.sal_purge has
> an open RT bound as its last item at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2588)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0-internal] at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.concurrent.SEPExecutor.maybeExecuteImmediately(SEPExecutor.java:194)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.service.AbstractReadExecutor.makeRequests(AbstractReadExecutor.java:117)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.service.AbstractReadExecutor.makeDataRequests(AbstractReadExecutor.java:85)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.service.AbstractReadExecutor$SpeculatingReadExecutor.executeAsync(AbstractReadExecutor.java:271)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.doInitialQueries(StorageProxy.java:1778)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1731)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1671)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1586)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:1209)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:315)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:285)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:117)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:225)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:532)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:509)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:146)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:566)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> [netty-all-4.0.44.Final.jar:4.0.44.Final] at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
> [netty-all-4.0.44.Final.jar:4.0.44.Final] at
> io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)
> [netty-all-4.0.44.Final.jar:4.0.44.Final] at
> io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348)
> [netty-all-4.0.44.Final.jar:4.0.44.Final] at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0-internal] at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114)
> [apache-cassandra-3.11.4.jar:3.11.4] at java.lang.Thread.run(Thread.java:748)
> [na:1.8.0-internal] Caused by: java.lang.IllegalStateException:
> UnfilteredRowIterator for SAL.sal_purge has an open RT bound as its last item
> at
> org.apache.cassandra.db.transform.RTBoundCloser$RowsTransformation.moreContents(RTBoundCloser.java:109)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.transform.RTBoundCloser$RowsTransformation.moreContents(RTBoundCloser.java:63)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.transform.BaseIterator.tryGetMoreContents(BaseIterator.java:121)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.transform.BaseIterator.hasMoreContents(BaseIterator.java:111)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:159)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:136)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:92)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:79)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:308)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:187)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:180)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:176)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:353)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1876)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2584)
> ~[apache-cassandra-3.11.4.jar:3.11.4] ... 29 common frames omitted
> Suppressed: java.lang.IllegalStateException: PROCESSED UnfilteredRowIterator
> for SAL.sal_purge has an illegal RT bounds sequence: expected all RTs to be
> closed, but the last one is open at
> org.apache.cassandra.db.transform.RTBoundValidator$RowsTransformation.ise(RTBoundValidator.java:120)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.transform.RTBoundValidator$RowsTransformation.onPartitionClose(RTBoundValidator.java:113)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.transform.BaseRows.runOnClose(BaseRows.java:91)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:86)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:309)
> ~[apache-cassandra-3.11.4.jar:3.11.4] ... 36 common frames omitted
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
