[
https://issues.apache.org/jira/browse/CASSANDRA-15529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029481#comment-17029481
]
feroz shaik edited comment on CASSANDRA-15529 at 2/4/20 2:35 AM:
-----------------------------------------------------------------
Hi [~ifesdjeen] , I am a colleague of [~Pooja_nair] and would like to answer
your questions. # We have all the nodes upgraded to 3.11.4 and also finished
upgradesstables. There are no previous sstables present on any of the
nodes/cluster.
# We have no clue whether the affected node is cordinator or not as it is too
random and the affected node behaves rogue potentially impacting whole cluster.
# The query patterns are like below:
static String INSERT_SCHEDULED_DELETION_QUERY = "INSERT INTO \"sal_purge\"
(key,column1,value) VALUES (?,?,?) USING TIMESTAMP
?;"; static String SELECT_SCHEDULED_DELETION_QUERY = "SELECT column1, value
FROM sal_purge where key=? AND column1>=? LIMIT
?;"; static String DELETE_SCHEDULED_DELETION_QUERY = "DELETE FROM
\"sal_purge\" USING TIMESTAMP ? WHERE key=? AND column1=?;";
I will try to upload the sstable dump soon to this, Appreciate your support
[~ifesdjeen]
was (Author: ferozshaik...@gmail.com):
Hi [~ifesdjeen] , I am a colleague of [~Pooja_nair] and would like to answer
your questions. # We have all the nodes upgraded to 3.11.4 and also finished
upgradesstables. There are no previous sstables present on any of the
nodes/cluster.
# We have no clue whether the affected node is cordinator or not as it is too
random and the affected node behaves rogue potentially bring impacting whole
cluster.
# The query patterns are like below:
static String INSERT_SCHEDULED_DELETION_QUERY = "INSERT INTO \"sal_purge\"
(key,column1,value) VALUES (?,?,?) USING TIMESTAMP
?;"; static String SELECT_SCHEDULED_DELETION_QUERY = "SELECT column1, value
FROM sal_purge where key=? AND column1>=? LIMIT
?;"; static String DELETE_SCHEDULED_DELETION_QUERY = "DELETE FROM \"sal_purge\"
USING TIMESTAMP ? WHERE key=? AND column1=?;";
I will try to upload the sstable dump soon to this, Appreciate your support
[~ifesdjeen]
> AbstractLocalAwareExecutorService.java exceptions after upgrade from 2.1.16
> to 3.11.4
> -------------------------------------------------------------------------------------
>
> Key: CASSANDRA-15529
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15529
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Schema
> Reporter: Pooja Nair
> Priority: Urgent
> Labels: 2.1.16, 3.11.4
>
> Hello Team,
> We have cluster running on cassandra 3.11.4
> Following is the table schema of the tables that is being used in our system.
> {code:java}
> cqlsh> desc KEYSPACE "SAL"
>
> CREATE KEYSPACE "SAL" WITH replication = {'class':
> 'NetworkTopologyStrategy', 'DC_EAST': '3', 'DC_WEST': '3'} AND
> durable_writes = true;
>
> CREATE TABLE "SAL".sal_purge (
> key text,
> column1 text,
> column2 text,
> value text,
> PRIMARY KEY (key, column1, column2)
> ) WITH COMPACT STORAGE
> AND CLUSTERING ORDER BY (column1 ASC, column2 ASC)
> AND bloom_filter_fp_chance = 0.1
> AND caching = '{"keys":"NONE", "rows_per_partition":"NONE"}'
> AND comment = 'Holds items to be removed as
> [shardid][salid][timestamp]. The table records SALIDs to be deleted along
> with their deletion times (which may be modified)'
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'chunk_length_kb': '64', 'sstable_compression':
> 'org.apache.cassandra.io.compress.SnappyCompressor'}
> AND dclocal_read_repair_chance = 0.0
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.1
> AND speculative_retry = '99.0PERCENTILE';
>
> CREATE TABLE "SAL".sal_ref (
> key text,
> column1 text,
> column2 text,
> value text,
> PRIMARY KEY (key, column1, column2)
> ) WITH COMPACT STORAGE
> AND CLUSTERING ORDER BY (column1 ASC, column2 ASC)
> AND bloom_filter_fp_chance = 0.025
> AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
> AND comment = 'Holds owner references to content as [salid][lcid/opid]'
> AND compaction = {'sstable_size_in_mb': '180', 'class':
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'chunk_length_kb': '64', 'sstable_compression':
> 'org.apache.cassandra.io.compress.SnappyCompressor'}
> AND dclocal_read_repair_chance = 0.0
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99.0PERCENTILE';
> {code}
> Things to note:
> # The column2 is always passed a null value during insertion
> # column2 is a part of primary key
> # Range select and Range delete is done through our app.
> Iniatally the cluster was on casssandra version 2.1.16 and have been
> recently upgraded to 3.11.4 post the upgrade, we see that the nodes are going
> down, and log the below exceptions during startup and even after node is up.
> This one node is causing the whole cluster to behave improperly.
> {code:java}
> WARN [Native-Transport-Requests-47] 2020-01-29 13:49:05,190
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[Native-Transport-Requests-47,5,main]: {} java.lang.RuntimeException:
> java.lang.IllegalStateException: UnfilteredRowIterator for SAL.sal_purge has
> an open RT bound as its last item at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2588)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0-internal] at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.concurrent.SEPExecutor.maybeExecuteImmediately(SEPExecutor.java:194)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.service.AbstractReadExecutor.makeRequests(AbstractReadExecutor.java:117)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.service.AbstractReadExecutor.makeDataRequests(AbstractReadExecutor.java:85)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.service.AbstractReadExecutor$SpeculatingReadExecutor.executeAsync(AbstractReadExecutor.java:271)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.doInitialQueries(StorageProxy.java:1778)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1731)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1671)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1586)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:1209)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:315)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:285)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:117)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:225)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:532)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:509)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:146)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:566)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> [netty-all-4.0.44.Final.jar:4.0.44.Final] at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
> [netty-all-4.0.44.Final.jar:4.0.44.Final] at
> io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)
> [netty-all-4.0.44.Final.jar:4.0.44.Final] at
> io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348)
> [netty-all-4.0.44.Final.jar:4.0.44.Final] at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0-internal] at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> [apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114)
> [apache-cassandra-3.11.4.jar:3.11.4] at java.lang.Thread.run(Thread.java:748)
> [na:1.8.0-internal] Caused by: java.lang.IllegalStateException:
> UnfilteredRowIterator for SAL.sal_purge has an open RT bound as its last item
> at
> org.apache.cassandra.db.transform.RTBoundCloser$RowsTransformation.moreContents(RTBoundCloser.java:109)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.transform.RTBoundCloser$RowsTransformation.moreContents(RTBoundCloser.java:63)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.transform.BaseIterator.tryGetMoreContents(BaseIterator.java:121)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.transform.BaseIterator.hasMoreContents(BaseIterator.java:111)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:159)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:136)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:92)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:79)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:308)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:187)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:180)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:176)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:353)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1876)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2584)
> ~[apache-cassandra-3.11.4.jar:3.11.4] ... 29 common frames omitted
> Suppressed: java.lang.IllegalStateException: PROCESSED UnfilteredRowIterator
> for SAL.sal_purge has an illegal RT bounds sequence: expected all RTs to be
> closed, but the last one is open at
> org.apache.cassandra.db.transform.RTBoundValidator$RowsTransformation.ise(RTBoundValidator.java:120)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.transform.RTBoundValidator$RowsTransformation.onPartitionClose(RTBoundValidator.java:113)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.transform.BaseRows.runOnClose(BaseRows.java:91)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:86)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:309)
> ~[apache-cassandra-3.11.4.jar:3.11.4] ... 36 common frames omitted
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
