[jira] [Commented] (CASSANDRA-15336) LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException When Running Range Queries
[
https://issues.apache.org/jira/browse/CASSANDRA-15336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948205#comment-16948205
]
Shalom commented on CASSANDRA-15336:
Hi All,
I'll just update for anyone that might be experiencing the same issue, that
after all the binaries were upgraded, the issue was gone as it came.
Moreover, we did some preliminary actions to mitigate this issue by running
major compactions on "hazardous" tables, thus dropping potentially affecting
tombstones.
There were still errors, but not as many as before.
Thanks [~benedict] for your assistance on this matter!
> LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException When Running
> Range Queries
> ---
>
> Key: CASSANDRA-15336
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15336
> Project: Cassandra
> Issue Type: Bug
>Reporter: Shalom
>Assignee: Benedict Elliott Smith
>Priority: Normal
>
> Hi All,
> This bug is similar to https://issues.apache.org/jira/browse/CASSANDRA-15172
> but relates specifically to range queries running over range tombstones.
>
>
> *+Steps to Reproduce:
> +*
> CREATE KEYSPACE ks1 WITH replication = \{'class': 'NetworkTopologyStrategy',
> 'DC1': '3'} AND durable_writes = true;
> +*TABLE:*+
> CREATE TABLE ks1.table1 (
> col1 text,
> col2 text,
> col3 text,
> col4 text,
> col5 text,
> col6 timestamp,
> data text,
> PRIMARY KEY ((col1, col2, col3), col4, col5, col6)
> );
>
> Inserted ~4 million rows and created range tombstones by deleting ~1 million
> rows.
>
> +*Create Data*+
> _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data )
> VALUES ( '1', '11', '21', '1', 'a', 1231231230, 'data');_
> _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data )
> VALUES ( '1', '11', '21', '2', 'a', 1231231230, 'data');_
> _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data )
> VALUES ( '1', '11', '21', '3', 'a', 1231231230, 'data');_
> _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data )
> VALUES ( '1', '11', '21', '4', 'a', 1231231230, 'data');_
> _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data )
> VALUES ( '1', '11', '21', '5', 'a', 1231231230, 'data');_
>
> +*Create Range Tombstones*+
> delete from ks1.table1 where col1='1' and col2='11' and col3='21' and
> col4='1';
>
> +*Query Live Rows (no tombstones)*+
> _select * from ks1.table1 where col1='1' and col2='201' and col3='21' and
> col4='1' and col5='a' and *col6>1231231230*;_
> No issues found, everything is running properly.
>
> +*Query Range Tombstones*+
> _select * from ks1.table1 where col1='1' and col2='11' and col3='21' and
> col4='1' and col5='a' and *col6=1231231230*;_
> No issues found, everything is running properly.
>
> +BUT when running range queries:+
> _select * from ks1.table1 where col1='1' and col2='11' and col3='21' and
> col4='1' and col5='a' and *col6>1231231220;*_
> WARN [ReadStage-1] 2019-09-23 14:17:10,281
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[ReadStage-1,5,main]: {}
> java.lang.ArrayIndexOutOfBoundsException: 2
> at
> org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522)
> at
> org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352)
> at org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171)
> at
> org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:77)
> at
> org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:802)
> at
> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:953)
> at
> org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:929)
> at
> org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:62)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> at
>
[jira] [Commented] (CASSANDRA-15336) LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException When Running Range Queries
[
https://issues.apache.org/jira/browse/CASSANDRA-15336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943812#comment-16943812
]
Shalom commented on CASSANDRA-15336:
Hi [~benedict]
I tried to use the 3.11 branch and unfortunately was able to reproduce the bug.
I've double and triple checked that the fix was there and that I used the
correct branch (pasted it below).
I'm a bit stressed about this matter because I have a planned upgrade this
coming Sunday and will be super happy if it works flawlessly.
I believe that once all nodes are upgraded though, the issue will be gone. Is
my assumption correct?
Thanks!
...
for (int i = 0; i < this.size; i++)
{
LegacyBound start = starts[i];
LegacyBound end = ends[i];
CompositeType.Builder startBuilder = type.builder();
CompositeType.Builder endBuilder = type.builder();
*{color:#FF}for (int j = 0; j < start.bound.size(); j++){color}*
*{color:#FF}startBuilder.add(start.bound.get(j));{color}*
*{color:#FF}for (int j = 0; j < end.bound.size(); j++){color}*
*{color:#FF}endBuilder.add(end.bound.get(j));{color}*
if (start.collectionName != null)
startBuilder.add(start.collectionName.name.bytes);
if (end.collectionName != null)
endBuilder.add(end.collectionName.name.bytes);
size += ByteBufferUtil.serializedSizeWithShortLength(startBuilder.build());
size +=
ByteBufferUtil.serializedSizeWithShortLength(endBuilder.buildAsEndOfRange());
...
> LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException When Running
> Range Queries
> ---
>
> Key: CASSANDRA-15336
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15336
> Project: Cassandra
> Issue Type: Bug
>Reporter: Shalom
>Assignee: Benedict Elliott Smith
>Priority: Normal
>
> Hi All,
> This bug is similar to https://issues.apache.org/jira/browse/CASSANDRA-15172
> but relates specifically to range queries running over range tombstones.
>
>
> *+Steps to Reproduce:
> +*
> CREATE KEYSPACE ks1 WITH replication = \{'class': 'NetworkTopologyStrategy',
> 'DC1': '3'} AND durable_writes = true;
> +*TABLE:*+
> CREATE TABLE ks1.table1 (
> col1 text,
> col2 text,
> col3 text,
> col4 text,
> col5 text,
> col6 timestamp,
> data text,
> PRIMARY KEY ((col1, col2, col3), col4, col5, col6)
> );
>
> Inserted ~4 million rows and created range tombstones by deleting ~1 million
> rows.
>
> +*Create Data*+
> _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data )
> VALUES ( '1', '11', '21', '1', 'a', 1231231230, 'data');_
> _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data )
> VALUES ( '1', '11', '21', '2', 'a', 1231231230, 'data');_
> _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data )
> VALUES ( '1', '11', '21', '3', 'a', 1231231230, 'data');_
> _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data )
> VALUES ( '1', '11', '21', '4', 'a', 1231231230, 'data');_
> _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data )
> VALUES ( '1', '11', '21', '5', 'a', 1231231230, 'data');_
>
> +*Create Range Tombstones*+
> delete from ks1.table1 where col1='1' and col2='11' and col3='21' and
> col4='1';
>
> +*Query Live Rows (no tombstones)*+
> _select * from ks1.table1 where col1='1' and col2='201' and col3='21' and
> col4='1' and col5='a' and *col6>1231231230*;_
> No issues found, everything is running properly.
>
> +*Query Range Tombstones*+
> _select * from ks1.table1 where col1='1' and col2='11' and col3='21' and
> col4='1' and col5='a' and *col6=1231231230*;_
> No issues found, everything is running properly.
>
> +BUT when running range queries:+
> _select * from ks1.table1 where col1='1' and col2='11' and col3='21' and
> col4='1' and col5='a' and *col6>1231231220;*_
> WARN [ReadStage-1] 2019-09-23 14:17:10,281
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[ReadStage-1,5,main]: {}
> java.lang.ArrayIndexOutOfBoundsException: 2
> at
> org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522)
> at
> org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352)
> at org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171)
> at
>
[jira] [Commented] (CASSANDRA-15336) LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException When Running Range Queries
[
https://issues.apache.org/jira/browse/CASSANDRA-15336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942902#comment-16942902
]
Shalom commented on CASSANDRA-15336:
[~benedict] I'm running v2.1.21 and wish to upgrade to v3.11.4.
I'm pretty sure I upgraded using the patch but I'll double check.
One question though, I see that the branch you used is 3.0 (
https://github.com/belliottsmith/cassandra/blob/{color:#FF}*15172-3.0*{color}/src/java/org/apache/cassandra/db/LegacyLayout.java)
Is this the correct branch or should it be a 3.11 branch?
> LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException When Running
> Range Queries
> ---
>
> Key: CASSANDRA-15336
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15336
> Project: Cassandra
> Issue Type: Bug
>Reporter: Shalom
>Assignee: Benedict Elliott Smith
>Priority: Normal
>
> Hi All,
> This bug is similar to https://issues.apache.org/jira/browse/CASSANDRA-15172
> but relates specifically to range queries running over range tombstones.
>
>
> *+Steps to Reproduce:
> +*
> CREATE KEYSPACE ks1 WITH replication = \{'class': 'NetworkTopologyStrategy',
> 'DC1': '3'} AND durable_writes = true;
> +*TABLE:*+
> CREATE TABLE ks1.table1 (
> col1 text,
> col2 text,
> col3 text,
> col4 text,
> col5 text,
> col6 timestamp,
> data text,
> PRIMARY KEY ((col1, col2, col3), col4, col5, col6)
> );
>
> Inserted ~4 million rows and created range tombstones by deleting ~1 million
> rows.
>
> +*Create Data*+
> _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data )
> VALUES ( '1', '11', '21', '1', 'a', 1231231230, 'data');_
> _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data )
> VALUES ( '1', '11', '21', '2', 'a', 1231231230, 'data');_
> _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data )
> VALUES ( '1', '11', '21', '3', 'a', 1231231230, 'data');_
> _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data )
> VALUES ( '1', '11', '21', '4', 'a', 1231231230, 'data');_
> _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data )
> VALUES ( '1', '11', '21', '5', 'a', 1231231230, 'data');_
>
> +*Create Range Tombstones*+
> delete from ks1.table1 where col1='1' and col2='11' and col3='21' and
> col4='1';
>
> +*Query Live Rows (no tombstones)*+
> _select * from ks1.table1 where col1='1' and col2='201' and col3='21' and
> col4='1' and col5='a' and *col6>1231231230*;_
> No issues found, everything is running properly.
>
> +*Query Range Tombstones*+
> _select * from ks1.table1 where col1='1' and col2='11' and col3='21' and
> col4='1' and col5='a' and *col6=1231231230*;_
> No issues found, everything is running properly.
>
> +BUT when running range queries:+
> _select * from ks1.table1 where col1='1' and col2='11' and col3='21' and
> col4='1' and col5='a' and *col6>1231231220;*_
> WARN [ReadStage-1] 2019-09-23 14:17:10,281
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[ReadStage-1,5,main]: {}
> java.lang.ArrayIndexOutOfBoundsException: 2
> at
> org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522)
> at
> org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352)
> at org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171)
> at
> org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:77)
> at
> org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:802)
> at
> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:953)
> at
> org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:929)
> at
> org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:62)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
[jira] [Commented] (CASSANDRA-15172) LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/CASSANDRA-15172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937903#comment-16937903
]
Shalom commented on CASSANDRA-15172:
Hi [~benedict]
Done and assigned to you! https://issues.apache.org/jira/browse/CASSANDRA-15336
Enjoy your holiday. (the more the merrier) :)
> LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException
>
>
> Key: CASSANDRA-15172
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15172
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Other
>Reporter: Shalom
>Assignee: Benedict
>Priority: Normal
> Fix For: 3.0.19, 3.11.5
>
>
> Hi All,
> This is the first time I open an issue, so apologies if I'm not following the
> rules properly.
>
> After upgrading a node from version 2.1.21 to 3.11.4, we've started seeing a
> lot of AbstractLocalAwareExecutorService exceptions. This happened right
> after the node successfully started up with the new 3.11.4 binaries.
> {noformat}
> INFO [main] 2019-06-05 04:41:37,730 Gossiper.java:1715 - No gossip backlog;
> proceeding
> INFO [main] 2019-06-05 04:41:38,036 NativeTransportService.java:70 - Netty
> using native Epoll event loop
> INFO [main] 2019-06-05 04:41:38,117 Server.java:155 - Using Netty Version:
> [netty-buffer=netty-buffer-4.0.44.Final.452812a,
> netty-codec=netty-codec-4.0.44.Final.452812a,
> netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a,
> netty-codec-http=netty-codec-http-4.0.44.Final.452812a,
> netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a,
> netty-common=netty-common-4.0.44.Final.452812a,
> netty-handler=netty-handler-4.0.44.Final.452812a,
> netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb,
> netty-transport=netty-transport-4.0.44.Final.452812a,
> netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a,
> netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a,
> netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a,
> netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a]
> INFO [main] 2019-06-05 04:41:38,118 Server.java:156 - Starting listening for
> CQL clients on /0.0.0.0:9042 (unencrypted)...
> INFO [main] 2019-06-05 04:41:38,179 CassandraDaemon.java:556 - Not starting
> RPC server as requested. Use JMX (StorageService->startRPCServer()) or
> nodetool (enablethrift) to start it
> INFO [Native-Transport-Requests-21] 2019-06-05 04:41:39,145
> AuthCache.java:161 - (Re)initializing PermissionsCache (validity
> period/update interval/max entries) (2000/2000/1000)
> INFO [OptionalTasks:1] 2019-06-05 04:41:39,729 CassandraAuthorizer.java:409
> - Converting legacy permissions data
> INFO [HANDSHAKE-/10.10.10.8] 2019-06-05 04:41:39,808
> OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.8
> INFO [HANDSHAKE-/10.10.10.9] 2019-06-05 04:41:39,808
> OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.9
> INFO [HANDSHAKE-dc1_02/10.10.10.6] 2019-06-05 04:41:39,809
> OutboundTcpConnection.java:561 - Handshaking version with dc1_02/10.10.10.6
> WARN [ReadStage-2] 2019-06-05 04:41:39,857
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[ReadStage-2,5,main]: {}
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522)
> at
> org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352)
> at
> org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171)
> at
> org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:77)
> at
> org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:802)
> at
> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:953)
> at
> org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:929)
> at
> org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:62)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at
>
[jira] [Created] (CASSANDRA-15336) LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException When Running Range Queries
Shalom created CASSANDRA-15336:
--
Summary: LegacyLayout RangeTombstoneList throws
IndexOutOfBoundsException When Running Range Queries
Key: CASSANDRA-15336
URL: https://issues.apache.org/jira/browse/CASSANDRA-15336
Project: Cassandra
Issue Type: Bug
Reporter: Shalom
Assignee: Benedict
Hi All,
This bug is similar to https://issues.apache.org/jira/browse/CASSANDRA-15172
but relates specifically to range queries running over range tombstones.
*+Steps to Reproduce:
+*
CREATE KEYSPACE ks1 WITH replication = \{'class': 'NetworkTopologyStrategy',
'DC1': '3'} AND durable_writes = true;
+*TABLE:*+
CREATE TABLE ks1.table1 (
col1 text,
col2 text,
col3 text,
col4 text,
col5 text,
col6 timestamp,
data text,
PRIMARY KEY ((col1, col2, col3), col4, col5, col6)
);
Inserted ~4 million rows and created range tombstones by deleting ~1 million
rows.
+*Create Data*+
_insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data ) VALUES
( '1', '11', '21', '1', 'a', 1231231230, 'data');_
_insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data )
VALUES ( '1', '11', '21', '2', 'a', 1231231230, 'data');_
_insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data )
VALUES ( '1', '11', '21', '3', 'a', 1231231230, 'data');_
_insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data )
VALUES ( '1', '11', '21', '4', 'a', 1231231230, 'data');_
_insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data )
VALUES ( '1', '11', '21', '5', 'a', 1231231230, 'data');_
+*Create Range Tombstones*+
delete from ks1.table1 where col1='1' and col2='11' and col3='21' and col4='1';
+*Query Live Rows (no tombstones)*+
_select * from ks1.table1 where col1='1' and col2='201' and col3='21' and
col4='1' and col5='a' and *col6>1231231230*;_
No issues found, everything is running properly.
+*Query Range Tombstones*+
_select * from ks1.table1 where col1='1' and col2='11' and col3='21' and
col4='1' and col5='a' and *col6=1231231230*;_
No issues found, everything is running properly.
+BUT when running range queries:+
_select * from ks1.table1 where col1='1' and col2='11' and col3='21' and
col4='1' and col5='a' and *col6>1231231220;*_
WARN [ReadStage-1] 2019-09-23 14:17:10,281
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-1,5,main]: {}
java.lang.ArrayIndexOutOfBoundsException: 2
at
org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55)
at
org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545)
at
org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522)
at
org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565)
at
org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446)
at
org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352)
at org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171)
at
org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:77)
at
org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:802)
at
org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:953)
at
org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:929)
at
org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:62)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114)
at java.lang.Thread.run(Thread.java:745)
This WARN is constantly generated until I stop the range queries script.
Hope this helps..
Thanks!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15172) LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/CASSANDRA-15172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937526#comment-16937526
]
Shalom commented on CASSANDRA-15172:
For users that happen to stumble upon this use case too, I'd like to emphasize
that the bug occurs only during mixed versions.
Once the cluster got fully upgraded (binaries only), the issue was gone.
> LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException
>
>
> Key: CASSANDRA-15172
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15172
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Other
>Reporter: Shalom
>Assignee: Benedict
>Priority: Normal
> Fix For: 3.0.19, 3.11.5
>
>
> Hi All,
> This is the first time I open an issue, so apologies if I'm not following the
> rules properly.
>
> After upgrading a node from version 2.1.21 to 3.11.4, we've started seeing a
> lot of AbstractLocalAwareExecutorService exceptions. This happened right
> after the node successfully started up with the new 3.11.4 binaries.
> {noformat}
> INFO [main] 2019-06-05 04:41:37,730 Gossiper.java:1715 - No gossip backlog;
> proceeding
> INFO [main] 2019-06-05 04:41:38,036 NativeTransportService.java:70 - Netty
> using native Epoll event loop
> INFO [main] 2019-06-05 04:41:38,117 Server.java:155 - Using Netty Version:
> [netty-buffer=netty-buffer-4.0.44.Final.452812a,
> netty-codec=netty-codec-4.0.44.Final.452812a,
> netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a,
> netty-codec-http=netty-codec-http-4.0.44.Final.452812a,
> netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a,
> netty-common=netty-common-4.0.44.Final.452812a,
> netty-handler=netty-handler-4.0.44.Final.452812a,
> netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb,
> netty-transport=netty-transport-4.0.44.Final.452812a,
> netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a,
> netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a,
> netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a,
> netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a]
> INFO [main] 2019-06-05 04:41:38,118 Server.java:156 - Starting listening for
> CQL clients on /0.0.0.0:9042 (unencrypted)...
> INFO [main] 2019-06-05 04:41:38,179 CassandraDaemon.java:556 - Not starting
> RPC server as requested. Use JMX (StorageService->startRPCServer()) or
> nodetool (enablethrift) to start it
> INFO [Native-Transport-Requests-21] 2019-06-05 04:41:39,145
> AuthCache.java:161 - (Re)initializing PermissionsCache (validity
> period/update interval/max entries) (2000/2000/1000)
> INFO [OptionalTasks:1] 2019-06-05 04:41:39,729 CassandraAuthorizer.java:409
> - Converting legacy permissions data
> INFO [HANDSHAKE-/10.10.10.8] 2019-06-05 04:41:39,808
> OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.8
> INFO [HANDSHAKE-/10.10.10.9] 2019-06-05 04:41:39,808
> OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.9
> INFO [HANDSHAKE-dc1_02/10.10.10.6] 2019-06-05 04:41:39,809
> OutboundTcpConnection.java:561 - Handshaking version with dc1_02/10.10.10.6
> WARN [ReadStage-2] 2019-06-05 04:41:39,857
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[ReadStage-2,5,main]: {}
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522)
> at
> org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352)
> at
> org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171)
> at
> org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:77)
> at
> org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:802)
> at
> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:953)
> at
> org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:929)
> at
> org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:62)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
> at
>
[jira] [Commented] (CASSANDRA-15172) LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/CASSANDRA-15172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936164#comment-16936164
]
Shalom commented on CASSANDRA-15172:
Hi [~benedict]
I tried the patch. It didn't do the trick, however, I was able to fully
reproduce the bug.
tl;dr
The bug does occur when running queries on range tombstones, but on top of
that, those queries have to specifically be range queries.
*+Steps to Reproduce:
+*
CREATE KEYSPACE ks1 WITH replication = \{'class': 'NetworkTopologyStrategy',
'DC1': '3'} AND durable_writes = true;
+*TABLE:*+
CREATE TABLE ks1.table1 (
col1 text,
col2 text,
col3 text,
col4 text,
col5 text,
col6 timestamp,
data text,
PRIMARY KEY ((col1, col2, col3), col4, col5, col6)
);
Inserted ~4 million rows and created range tombstones by deleting ~1 million
rows.
+*Create Data*+
_insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data ) VALUES
( '1', '11', '21', '1', 'a', 1231231230, 'data');_
_insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data ) VALUES
( '1', '11', '21', '2', 'a', 1231231230, 'data');_
_insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data ) VALUES
( '1', '11', '21', '3', 'a', 1231231230, 'data');_
_insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data ) VALUES
( '1', '11', '21', '4', 'a', 1231231230, 'data');_
_insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data ) VALUES
( '1', '11', '21', '5', 'a', 1231231230, 'data');_
+*Create Range Tombstones*+
delete from ks1.table1 where col1='1' and col2='11' and col3='21' and col4='1';
+*Query Live Rows (no tombstones)*+
_select * from ks1.table1 where col1='1' and col2='201' and col3='21' and
col4='1' and col5='a' and *col6>1231231230*;_
No issues found, everything is running properly.
+*Query Range Tombstones*+
_select * from ks1.table1 where col1='1' and col2='11' and col3='21' and
col4='1' and col5='a' and *col6=1231231230*;_
No issues found, everything is running properly.
+BUT when running range queries:+
_select * from ks1.table1 where col1='1' and col2='11' and col3='21' and
col4='1' and col5='a' and *col6>1231231220;*_
WARN [ReadStage-1] 2019-09-23 14:17:10,281
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-1,5,main]: {}
java.lang.ArrayIndexOutOfBoundsException: 2
at
org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55)
at
org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545)
at
org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522)
at
org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565)
at
org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446)
at
org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352)
at org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171)
at
org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:77)
at
org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:802)
at
org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:953)
at
org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:929)
at
org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:62)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114)
at java.lang.Thread.run(Thread.java:745)
This WARN is constantly generated until I stop the range queries script.
Hope this helps..
Thanks!
> LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException
>
>
> Key: CASSANDRA-15172
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15172
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Other
>Reporter: Shalom
>Assignee: Benedict
>Priority: Normal
> Fix For: 3.0.19, 3.11.5
>
>
> Hi All,
> This is the first time I open an issue, so apologies if I'm not following the
> rules properly.
>
> After upgrading a node from version 2.1.21 to 3.11.4, we've started seeing a
> lot of
[jira] [Commented] (CASSANDRA-15172) LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/CASSANDRA-15172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918545#comment-16918545
]
Shalom commented on CASSANDRA-15172:
Thanks a lot for further clarifying [~benedict]. (I hope you enjoyed your
vacation :) )
Just to set my mind straight, the issue is when there are mixed versions in the
cluster, so if I upgraded all binaries to 3.11, it won't recur even if I
haven't upgraded the SSTables yet. Is my assumption correct?
> LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException
>
>
> Key: CASSANDRA-15172
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15172
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Other
>Reporter: Shalom
>Assignee: Benedict
>Priority: Normal
> Fix For: 3.0.19, 3.11.5
>
>
> Hi All,
> This is the first time I open an issue, so apologies if I'm not following the
> rules properly.
>
> After upgrading a node from version 2.1.21 to 3.11.4, we've started seeing a
> lot of AbstractLocalAwareExecutorService exceptions. This happened right
> after the node successfully started up with the new 3.11.4 binaries.
> {noformat}
> INFO [main] 2019-06-05 04:41:37,730 Gossiper.java:1715 - No gossip backlog;
> proceeding
> INFO [main] 2019-06-05 04:41:38,036 NativeTransportService.java:70 - Netty
> using native Epoll event loop
> INFO [main] 2019-06-05 04:41:38,117 Server.java:155 - Using Netty Version:
> [netty-buffer=netty-buffer-4.0.44.Final.452812a,
> netty-codec=netty-codec-4.0.44.Final.452812a,
> netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a,
> netty-codec-http=netty-codec-http-4.0.44.Final.452812a,
> netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a,
> netty-common=netty-common-4.0.44.Final.452812a,
> netty-handler=netty-handler-4.0.44.Final.452812a,
> netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb,
> netty-transport=netty-transport-4.0.44.Final.452812a,
> netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a,
> netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a,
> netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a,
> netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a]
> INFO [main] 2019-06-05 04:41:38,118 Server.java:156 - Starting listening for
> CQL clients on /0.0.0.0:9042 (unencrypted)...
> INFO [main] 2019-06-05 04:41:38,179 CassandraDaemon.java:556 - Not starting
> RPC server as requested. Use JMX (StorageService->startRPCServer()) or
> nodetool (enablethrift) to start it
> INFO [Native-Transport-Requests-21] 2019-06-05 04:41:39,145
> AuthCache.java:161 - (Re)initializing PermissionsCache (validity
> period/update interval/max entries) (2000/2000/1000)
> INFO [OptionalTasks:1] 2019-06-05 04:41:39,729 CassandraAuthorizer.java:409
> - Converting legacy permissions data
> INFO [HANDSHAKE-/10.10.10.8] 2019-06-05 04:41:39,808
> OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.8
> INFO [HANDSHAKE-/10.10.10.9] 2019-06-05 04:41:39,808
> OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.9
> INFO [HANDSHAKE-dc1_02/10.10.10.6] 2019-06-05 04:41:39,809
> OutboundTcpConnection.java:561 - Handshaking version with dc1_02/10.10.10.6
> WARN [ReadStage-2] 2019-06-05 04:41:39,857
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[ReadStage-2,5,main]: {}
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522)
> at
> org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352)
> at
> org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171)
> at
> org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:77)
> at
> org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:802)
> at
> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:953)
> at
> org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:929)
> at
> org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:62)
> at
>
[jira] [Commented] (CASSANDRA-15172) LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/CASSANDRA-15172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916953#comment-16916953
]
Shalom commented on CASSANDRA-15172:
Thanks for clarifying [~michaelsembwever]!
> LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException
>
>
> Key: CASSANDRA-15172
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15172
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Other
>Reporter: Shalom
>Assignee: Benedict
>Priority: Normal
> Fix For: 3.0.19, 3.11.5
>
>
> Hi All,
> This is the first time I open an issue, so apologies if I'm not following the
> rules properly.
>
> After upgrading a node from version 2.1.21 to 3.11.4, we've started seeing a
> lot of AbstractLocalAwareExecutorService exceptions. This happened right
> after the node successfully started up with the new 3.11.4 binaries.
> {noformat}
> INFO [main] 2019-06-05 04:41:37,730 Gossiper.java:1715 - No gossip backlog;
> proceeding
> INFO [main] 2019-06-05 04:41:38,036 NativeTransportService.java:70 - Netty
> using native Epoll event loop
> INFO [main] 2019-06-05 04:41:38,117 Server.java:155 - Using Netty Version:
> [netty-buffer=netty-buffer-4.0.44.Final.452812a,
> netty-codec=netty-codec-4.0.44.Final.452812a,
> netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a,
> netty-codec-http=netty-codec-http-4.0.44.Final.452812a,
> netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a,
> netty-common=netty-common-4.0.44.Final.452812a,
> netty-handler=netty-handler-4.0.44.Final.452812a,
> netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb,
> netty-transport=netty-transport-4.0.44.Final.452812a,
> netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a,
> netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a,
> netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a,
> netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a]
> INFO [main] 2019-06-05 04:41:38,118 Server.java:156 - Starting listening for
> CQL clients on /0.0.0.0:9042 (unencrypted)...
> INFO [main] 2019-06-05 04:41:38,179 CassandraDaemon.java:556 - Not starting
> RPC server as requested. Use JMX (StorageService->startRPCServer()) or
> nodetool (enablethrift) to start it
> INFO [Native-Transport-Requests-21] 2019-06-05 04:41:39,145
> AuthCache.java:161 - (Re)initializing PermissionsCache (validity
> period/update interval/max entries) (2000/2000/1000)
> INFO [OptionalTasks:1] 2019-06-05 04:41:39,729 CassandraAuthorizer.java:409
> - Converting legacy permissions data
> INFO [HANDSHAKE-/10.10.10.8] 2019-06-05 04:41:39,808
> OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.8
> INFO [HANDSHAKE-/10.10.10.9] 2019-06-05 04:41:39,808
> OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.9
> INFO [HANDSHAKE-dc1_02/10.10.10.6] 2019-06-05 04:41:39,809
> OutboundTcpConnection.java:561 - Handshaking version with dc1_02/10.10.10.6
> WARN [ReadStage-2] 2019-06-05 04:41:39,857
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[ReadStage-2,5,main]: {}
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522)
> at
> org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352)
> at
> org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171)
> at
> org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:77)
> at
> org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:802)
> at
> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:953)
> at
> org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:929)
> at
> org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:62)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
[jira] [Commented] (CASSANDRA-15172) LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/CASSANDRA-15172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916910#comment-16916910
]
Shalom commented on CASSANDRA-15172:
Hi [~benedict] ,
I'm not sure if my comments/tickets are somehow not getting filed, but I hope
not :)
Is the above error (previous comment) related to this bug or is it a different
one?
Thanks!
> LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException
>
>
> Key: CASSANDRA-15172
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15172
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Other
>Reporter: Shalom
>Assignee: Benedict
>Priority: Normal
> Fix For: 3.0.19, 3.11.5
>
>
> Hi All,
> This is the first time I open an issue, so apologies if I'm not following the
> rules properly.
>
> After upgrading a node from version 2.1.21 to 3.11.4, we've started seeing a
> lot of AbstractLocalAwareExecutorService exceptions. This happened right
> after the node successfully started up with the new 3.11.4 binaries.
> {noformat}
> INFO [main] 2019-06-05 04:41:37,730 Gossiper.java:1715 - No gossip backlog;
> proceeding
> INFO [main] 2019-06-05 04:41:38,036 NativeTransportService.java:70 - Netty
> using native Epoll event loop
> INFO [main] 2019-06-05 04:41:38,117 Server.java:155 - Using Netty Version:
> [netty-buffer=netty-buffer-4.0.44.Final.452812a,
> netty-codec=netty-codec-4.0.44.Final.452812a,
> netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a,
> netty-codec-http=netty-codec-http-4.0.44.Final.452812a,
> netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a,
> netty-common=netty-common-4.0.44.Final.452812a,
> netty-handler=netty-handler-4.0.44.Final.452812a,
> netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb,
> netty-transport=netty-transport-4.0.44.Final.452812a,
> netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a,
> netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a,
> netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a,
> netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a]
> INFO [main] 2019-06-05 04:41:38,118 Server.java:156 - Starting listening for
> CQL clients on /0.0.0.0:9042 (unencrypted)...
> INFO [main] 2019-06-05 04:41:38,179 CassandraDaemon.java:556 - Not starting
> RPC server as requested. Use JMX (StorageService->startRPCServer()) or
> nodetool (enablethrift) to start it
> INFO [Native-Transport-Requests-21] 2019-06-05 04:41:39,145
> AuthCache.java:161 - (Re)initializing PermissionsCache (validity
> period/update interval/max entries) (2000/2000/1000)
> INFO [OptionalTasks:1] 2019-06-05 04:41:39,729 CassandraAuthorizer.java:409
> - Converting legacy permissions data
> INFO [HANDSHAKE-/10.10.10.8] 2019-06-05 04:41:39,808
> OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.8
> INFO [HANDSHAKE-/10.10.10.9] 2019-06-05 04:41:39,808
> OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.9
> INFO [HANDSHAKE-dc1_02/10.10.10.6] 2019-06-05 04:41:39,809
> OutboundTcpConnection.java:561 - Handshaking version with dc1_02/10.10.10.6
> WARN [ReadStage-2] 2019-06-05 04:41:39,857
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[ReadStage-2,5,main]: {}
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522)
> at
> org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352)
> at
> org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171)
> at
> org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:77)
> at
> org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:802)
> at
> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:953)
> at
> org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:929)
> at
> org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:62)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
> at
>
[jira] [Commented] (CASSANDRA-15172) LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/CASSANDRA-15172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915197#comment-16915197
]
Shalom commented on CASSANDRA-15172:
Hi [~benedict]
Sorry for the late reply...
Could you also confirm if you utilise thrift, or CQL schema? It's possible this
is a compatibility issue specific to thrift.
I do see there are a few thrift counter tables that were created with the WITH
COMPACT STORAGE attribute.
I also see that [~ferozshaik...@gmail.com] had seen WARN [ReadStage-4]
2019-08-06 02:57:57,408 AbstractLocalAwareExecutorService.java:167 - Uncaught
exception on thread Thread[ReadStage-4,5,main]: {}
java.lang.NullPointerException: null
What I mainly saw was AbstractLocalAwareExecutorService.java:167 - Uncaught
exception on thread Thread[ReadStage-9,5,main]: {}
java.lang.ArrayIndexOutOfBoundsException: null
Does the fix fit the same scenario as Feroz's? Could my issue be thrift related?
Sorry again for the late reply.
> LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException
>
>
> Key: CASSANDRA-15172
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15172
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Other
>Reporter: Shalom
>Assignee: Benedict
>Priority: Normal
> Fix For: 3.0.19, 3.11.5
>
>
> Hi All,
> This is the first time I open an issue, so apologies if I'm not following the
> rules properly.
>
> After upgrading a node from version 2.1.21 to 3.11.4, we've started seeing a
> lot of AbstractLocalAwareExecutorService exceptions. This happened right
> after the node successfully started up with the new 3.11.4 binaries.
> {noformat}
> INFO [main] 2019-06-05 04:41:37,730 Gossiper.java:1715 - No gossip backlog;
> proceeding
> INFO [main] 2019-06-05 04:41:38,036 NativeTransportService.java:70 - Netty
> using native Epoll event loop
> INFO [main] 2019-06-05 04:41:38,117 Server.java:155 - Using Netty Version:
> [netty-buffer=netty-buffer-4.0.44.Final.452812a,
> netty-codec=netty-codec-4.0.44.Final.452812a,
> netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a,
> netty-codec-http=netty-codec-http-4.0.44.Final.452812a,
> netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a,
> netty-common=netty-common-4.0.44.Final.452812a,
> netty-handler=netty-handler-4.0.44.Final.452812a,
> netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb,
> netty-transport=netty-transport-4.0.44.Final.452812a,
> netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a,
> netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a,
> netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a,
> netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a]
> INFO [main] 2019-06-05 04:41:38,118 Server.java:156 - Starting listening for
> CQL clients on /0.0.0.0:9042 (unencrypted)...
> INFO [main] 2019-06-05 04:41:38,179 CassandraDaemon.java:556 - Not starting
> RPC server as requested. Use JMX (StorageService->startRPCServer()) or
> nodetool (enablethrift) to start it
> INFO [Native-Transport-Requests-21] 2019-06-05 04:41:39,145
> AuthCache.java:161 - (Re)initializing PermissionsCache (validity
> period/update interval/max entries) (2000/2000/1000)
> INFO [OptionalTasks:1] 2019-06-05 04:41:39,729 CassandraAuthorizer.java:409
> - Converting legacy permissions data
> INFO [HANDSHAKE-/10.10.10.8] 2019-06-05 04:41:39,808
> OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.8
> INFO [HANDSHAKE-/10.10.10.9] 2019-06-05 04:41:39,808
> OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.9
> INFO [HANDSHAKE-dc1_02/10.10.10.6] 2019-06-05 04:41:39,809
> OutboundTcpConnection.java:561 - Handshaking version with dc1_02/10.10.10.6
> WARN [ReadStage-2] 2019-06-05 04:41:39,857
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[ReadStage-2,5,main]: {}
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522)
> at
> org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352)
> at
> org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171)
> at
>
[jira] [Commented] (CASSANDRA-15172) AbstractLocalAwareExecutorService Exception During Upgrade to 3.11.4
[
https://issues.apache.org/jira/browse/CASSANDRA-15172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16888040#comment-16888040
]
Shalom commented on CASSANDRA-15172:
Trying to push this up a bit because I still want to upgrade to 3.11.4 but I
fear that this issue may recur.
If someone has any idea on what happened here or how to mitigate it, that'd be
awesome!
Thanks!
> AbstractLocalAwareExecutorService Exception During Upgrade to 3.11.4
>
>
> Key: CASSANDRA-15172
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15172
> Project: Cassandra
> Issue Type: Bug
>Reporter: Shalom
>Priority: Normal
>
> Hi All,
> This is the first time I open an issue, so apologies if I'm not following the
> rules properly.
>
> After upgrading a node from version 2.1.21 to 3.11.4, we've started seeing a
> lot of AbstractLocalAwareExecutorService exceptions. This happened right
> after the node successfully started up with the new 3.11.4 binaries.
> INFO [main] 2019-06-05 04:41:37,730 Gossiper.java:1715 - No gossip backlog;
> proceeding
> INFO [main] 2019-06-05 04:41:38,036 NativeTransportService.java:70 - Netty
> using native Epoll event loop
> INFO [main] 2019-06-05 04:41:38,117 Server.java:155 - Using Netty Version:
> [netty-buffer=netty-buffer-4.0.44.Final.452812a,
> netty-codec=netty-codec-4.0.44.Final.452812a,
> netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a,
> netty-codec-http=netty-codec-http-4.0.44.Final.452812a,
> netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a,
> netty-common=netty-common-4.0.44.Final.452812a,
> netty-handler=netty-handler-4.0.44.Final.452812a,
> netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb,
> netty-transport=netty-transport-4.0.44.Final.452812a,
> netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a,
> netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a,
> netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a,
> netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a]
> INFO [main] 2019-06-05 04:41:38,118 Server.java:156 - Starting listening for
> CQL clients on /0.0.0.0:9042 (unencrypted)...
> INFO [main] 2019-06-05 04:41:38,179 CassandraDaemon.java:556 - Not starting
> RPC server as requested. Use JMX (StorageService->startRPCServer()) or
> nodetool (enablethrift) to start it
> INFO [Native-Transport-Requests-21] 2019-06-05 04:41:39,145
> AuthCache.java:161 - (Re)initializing PermissionsCache (validity
> period/update interval/max entries) (2000/2000/1000)
> INFO [OptionalTasks:1] 2019-06-05 04:41:39,729 CassandraAuthorizer.java:409
> - Converting legacy permissions data
> INFO [HANDSHAKE-/10.10.10.8] 2019-06-05 04:41:39,808
> OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.8
> INFO [HANDSHAKE-/10.10.10.9] 2019-06-05 04:41:39,808
> OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.9
> INFO [HANDSHAKE-dc1_02/10.10.10.6] 2019-06-05 04:41:39,809
> OutboundTcpConnection.java:561 - Handshaking version with dc1_02/10.10.10.6
> WARN [ReadStage-2] 2019-06-05 04:41:39,857
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[ReadStage-2,5,main]: {}
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522)
> at
> org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352)
> at
> org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171)
> at
> org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:77)
> at
> org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:802)
> at
> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:953)
> at
> org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:929)
> at
> org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:62)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at
>
[jira] [Created] (CASSANDRA-15172) AbstractLocalAwareExecutorService Exception During Upgrade to 3.11.4
Shalom created CASSANDRA-15172:
--
Summary: AbstractLocalAwareExecutorService Exception During
Upgrade to 3.11.4
Key: CASSANDRA-15172
URL: https://issues.apache.org/jira/browse/CASSANDRA-15172
Project: Cassandra
Issue Type: Bug
Reporter: Shalom
Hi All,
This is the first time I open an issue, so apologies if I'm not following the
rules properly.
After upgrading a node from version 2.1.21 to 3.11.4, we've started seeing a
lot of AbstractLocalAwareExecutorService exceptions. This happened right after
the node successfully started up with the new 3.11.4 binaries.
INFO [main] 2019-06-05 04:41:37,730 Gossiper.java:1715 - No gossip backlog;
proceeding
INFO [main] 2019-06-05 04:41:38,036 NativeTransportService.java:70 - Netty
using native Epoll event loop
INFO [main] 2019-06-05 04:41:38,117 Server.java:155 - Using Netty Version:
[netty-buffer=netty-buffer-4.0.44.Final.452812a,
netty-codec=netty-codec-4.0.44.Final.452812a,
netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a,
netty-codec-http=netty-codec-http-4.0.44.Final.452812a,
netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a,
netty-common=netty-common-4.0.44.Final.452812a,
netty-handler=netty-handler-4.0.44.Final.452812a,
netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb,
netty-transport=netty-transport-4.0.44.Final.452812a,
netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a,
netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a,
netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a,
netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a]
INFO [main] 2019-06-05 04:41:38,118 Server.java:156 - Starting listening for
CQL clients on /0.0.0.0:9042 (unencrypted)...
INFO [main] 2019-06-05 04:41:38,179 CassandraDaemon.java:556 - Not starting
RPC server as requested. Use JMX (StorageService->startRPCServer()) or nodetool
(enablethrift) to start it
INFO [Native-Transport-Requests-21] 2019-06-05 04:41:39,145 AuthCache.java:161
- (Re)initializing PermissionsCache (validity period/update interval/max
entries) (2000/2000/1000)
INFO [OptionalTasks:1] 2019-06-05 04:41:39,729 CassandraAuthorizer.java:409 -
Converting legacy permissions data
INFO [HANDSHAKE-/10.10.10.8] 2019-06-05 04:41:39,808
OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.8
INFO [HANDSHAKE-/10.10.10.9] 2019-06-05 04:41:39,808
OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.9
INFO [HANDSHAKE-dc1_02/10.10.10.6] 2019-06-05 04:41:39,809
OutboundTcpConnection.java:561 - Handshaking version with dc1_02/10.10.10.6
WARN [ReadStage-2] 2019-06-05 04:41:39,857
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-2,5,main]: {}
java.lang.ArrayIndexOutOfBoundsException: 1
at
org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55)
at
org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545)
at
org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522)
at
org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565)
at
org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446)
at
org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352)
at org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171)
at
org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:77)
at
org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:802)
at
org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:953)
at
org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:929)
at
org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:62)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114)
at java.lang.Thread.run(Thread.java:745)
After several of the above warnings, the following warning appeared as well:
WARN [ReadStage-9] 2019-06-05 04:42:04,369
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-9,5,main]: {}
java.lang.ArrayIndexOutOfBoundsException: null
