[jira] [Commented] (CASSANDRA-15172) AbstractLocalAwareExecutorService Exception During Upgrade to 3.11.4
[
https://issues.apache.org/jira/browse/CASSANDRA-15172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901010#comment-16901010
]
Benedict commented on CASSANDRA-15172:
--
[~Sagges], sorry for the slow response - I missed the original filing of this
ticket.
[~ferozshaik...@gmail.com] it looks like your bug, while very similar, presents
differently. It would be great if you could file a separate ticket.
Both of these look to be among the category of 2.1->3.0 upgrade bugs involving
range deletions. To best investigate and diagnose, it would be great to start
with information about the affected schema, the kinds of range tombstone
deletes you perform, and preferably if you could pin down sstables that are
affected and upload them somewhere private for us to access. This would help
us investigate much more readily.
> AbstractLocalAwareExecutorService Exception During Upgrade to 3.11.4
>
>
> Key: CASSANDRA-15172
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15172
> Project: Cassandra
> Issue Type: Bug
>Reporter: Shalom
>Priority: Normal
>
> Hi All,
> This is the first time I open an issue, so apologies if I'm not following the
> rules properly.
>
> After upgrading a node from version 2.1.21 to 3.11.4, we've started seeing a
> lot of AbstractLocalAwareExecutorService exceptions. This happened right
> after the node successfully started up with the new 3.11.4 binaries.
> INFO [main] 2019-06-05 04:41:37,730 Gossiper.java:1715 - No gossip backlog;
> proceeding
> INFO [main] 2019-06-05 04:41:38,036 NativeTransportService.java:70 - Netty
> using native Epoll event loop
> INFO [main] 2019-06-05 04:41:38,117 Server.java:155 - Using Netty Version:
> [netty-buffer=netty-buffer-4.0.44.Final.452812a,
> netty-codec=netty-codec-4.0.44.Final.452812a,
> netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a,
> netty-codec-http=netty-codec-http-4.0.44.Final.452812a,
> netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a,
> netty-common=netty-common-4.0.44.Final.452812a,
> netty-handler=netty-handler-4.0.44.Final.452812a,
> netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb,
> netty-transport=netty-transport-4.0.44.Final.452812a,
> netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a,
> netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a,
> netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a,
> netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a]
> INFO [main] 2019-06-05 04:41:38,118 Server.java:156 - Starting listening for
> CQL clients on /0.0.0.0:9042 (unencrypted)...
> INFO [main] 2019-06-05 04:41:38,179 CassandraDaemon.java:556 - Not starting
> RPC server as requested. Use JMX (StorageService->startRPCServer()) or
> nodetool (enablethrift) to start it
> INFO [Native-Transport-Requests-21] 2019-06-05 04:41:39,145
> AuthCache.java:161 - (Re)initializing PermissionsCache (validity
> period/update interval/max entries) (2000/2000/1000)
> INFO [OptionalTasks:1] 2019-06-05 04:41:39,729 CassandraAuthorizer.java:409
> - Converting legacy permissions data
> INFO [HANDSHAKE-/10.10.10.8] 2019-06-05 04:41:39,808
> OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.8
> INFO [HANDSHAKE-/10.10.10.9] 2019-06-05 04:41:39,808
> OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.9
> INFO [HANDSHAKE-dc1_02/10.10.10.6] 2019-06-05 04:41:39,809
> OutboundTcpConnection.java:561 - Handshaking version with dc1_02/10.10.10.6
> WARN [ReadStage-2] 2019-06-05 04:41:39,857
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[ReadStage-2,5,main]: {}
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522)
> at
> org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352)
> at
> org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171)
> at
> org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:77)
> at
> org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:802)
> at
> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:953)
>
[jira] [Commented] (CASSANDRA-15172) AbstractLocalAwareExecutorService Exception During Upgrade to 3.11.4
[
https://issues.apache.org/jira/browse/CASSANDRA-15172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900882#comment-16900882
]
feroz shaik commented on CASSANDRA-15172:
-
We have currently isolated this node (cut off thrift , native protocols) and
trying to run upgradesstables to see if it can re-write all the files and stop
logging the below message.
"WARN [ReadStage-6] 2019-08-06 10:44:09,773
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-6,5,main]: {}
java.lang.NullPointerException: null"
I will keep the thread posted about its outcome,
> AbstractLocalAwareExecutorService Exception During Upgrade to 3.11.4
>
>
> Key: CASSANDRA-15172
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15172
> Project: Cassandra
> Issue Type: Bug
>Reporter: Shalom
>Priority: Normal
>
> Hi All,
> This is the first time I open an issue, so apologies if I'm not following the
> rules properly.
>
> After upgrading a node from version 2.1.21 to 3.11.4, we've started seeing a
> lot of AbstractLocalAwareExecutorService exceptions. This happened right
> after the node successfully started up with the new 3.11.4 binaries.
> INFO [main] 2019-06-05 04:41:37,730 Gossiper.java:1715 - No gossip backlog;
> proceeding
> INFO [main] 2019-06-05 04:41:38,036 NativeTransportService.java:70 - Netty
> using native Epoll event loop
> INFO [main] 2019-06-05 04:41:38,117 Server.java:155 - Using Netty Version:
> [netty-buffer=netty-buffer-4.0.44.Final.452812a,
> netty-codec=netty-codec-4.0.44.Final.452812a,
> netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a,
> netty-codec-http=netty-codec-http-4.0.44.Final.452812a,
> netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a,
> netty-common=netty-common-4.0.44.Final.452812a,
> netty-handler=netty-handler-4.0.44.Final.452812a,
> netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb,
> netty-transport=netty-transport-4.0.44.Final.452812a,
> netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a,
> netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a,
> netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a,
> netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a]
> INFO [main] 2019-06-05 04:41:38,118 Server.java:156 - Starting listening for
> CQL clients on /0.0.0.0:9042 (unencrypted)...
> INFO [main] 2019-06-05 04:41:38,179 CassandraDaemon.java:556 - Not starting
> RPC server as requested. Use JMX (StorageService->startRPCServer()) or
> nodetool (enablethrift) to start it
> INFO [Native-Transport-Requests-21] 2019-06-05 04:41:39,145
> AuthCache.java:161 - (Re)initializing PermissionsCache (validity
> period/update interval/max entries) (2000/2000/1000)
> INFO [OptionalTasks:1] 2019-06-05 04:41:39,729 CassandraAuthorizer.java:409
> - Converting legacy permissions data
> INFO [HANDSHAKE-/10.10.10.8] 2019-06-05 04:41:39,808
> OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.8
> INFO [HANDSHAKE-/10.10.10.9] 2019-06-05 04:41:39,808
> OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.9
> INFO [HANDSHAKE-dc1_02/10.10.10.6] 2019-06-05 04:41:39,809
> OutboundTcpConnection.java:561 - Handshaking version with dc1_02/10.10.10.6
> WARN [ReadStage-2] 2019-06-05 04:41:39,857
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[ReadStage-2,5,main]: {}
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522)
> at
> org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352)
> at
> org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171)
> at
> org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:77)
> at
> org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:802)
> at
> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:953)
> at
> org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:929)
> at
> org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:62)
> at
>
[jira] [Commented] (CASSANDRA-15172) AbstractLocalAwareExecutorService Exception During Upgrade to 3.11.4
[
https://issues.apache.org/jira/browse/CASSANDRA-15172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900879#comment-16900879
]
feroz shaik commented on CASSANDRA-15172:
-
What we also know is that the nodes suffered some heavy tombstones recently..
could it be under specific condition like "tombstones" plus reading legacy
version files is hitting this problem. [~Sagges] did you cluster have any
tombstone references at all?
> AbstractLocalAwareExecutorService Exception During Upgrade to 3.11.4
>
>
> Key: CASSANDRA-15172
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15172
> Project: Cassandra
> Issue Type: Bug
>Reporter: Shalom
>Priority: Normal
>
> Hi All,
> This is the first time I open an issue, so apologies if I'm not following the
> rules properly.
>
> After upgrading a node from version 2.1.21 to 3.11.4, we've started seeing a
> lot of AbstractLocalAwareExecutorService exceptions. This happened right
> after the node successfully started up with the new 3.11.4 binaries.
> INFO [main] 2019-06-05 04:41:37,730 Gossiper.java:1715 - No gossip backlog;
> proceeding
> INFO [main] 2019-06-05 04:41:38,036 NativeTransportService.java:70 - Netty
> using native Epoll event loop
> INFO [main] 2019-06-05 04:41:38,117 Server.java:155 - Using Netty Version:
> [netty-buffer=netty-buffer-4.0.44.Final.452812a,
> netty-codec=netty-codec-4.0.44.Final.452812a,
> netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a,
> netty-codec-http=netty-codec-http-4.0.44.Final.452812a,
> netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a,
> netty-common=netty-common-4.0.44.Final.452812a,
> netty-handler=netty-handler-4.0.44.Final.452812a,
> netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb,
> netty-transport=netty-transport-4.0.44.Final.452812a,
> netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a,
> netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a,
> netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a,
> netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a]
> INFO [main] 2019-06-05 04:41:38,118 Server.java:156 - Starting listening for
> CQL clients on /0.0.0.0:9042 (unencrypted)...
> INFO [main] 2019-06-05 04:41:38,179 CassandraDaemon.java:556 - Not starting
> RPC server as requested. Use JMX (StorageService->startRPCServer()) or
> nodetool (enablethrift) to start it
> INFO [Native-Transport-Requests-21] 2019-06-05 04:41:39,145
> AuthCache.java:161 - (Re)initializing PermissionsCache (validity
> period/update interval/max entries) (2000/2000/1000)
> INFO [OptionalTasks:1] 2019-06-05 04:41:39,729 CassandraAuthorizer.java:409
> - Converting legacy permissions data
> INFO [HANDSHAKE-/10.10.10.8] 2019-06-05 04:41:39,808
> OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.8
> INFO [HANDSHAKE-/10.10.10.9] 2019-06-05 04:41:39,808
> OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.9
> INFO [HANDSHAKE-dc1_02/10.10.10.6] 2019-06-05 04:41:39,809
> OutboundTcpConnection.java:561 - Handshaking version with dc1_02/10.10.10.6
> WARN [ReadStage-2] 2019-06-05 04:41:39,857
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[ReadStage-2,5,main]: {}
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522)
> at
> org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352)
> at
> org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171)
> at
> org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:77)
> at
> org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:802)
> at
> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:953)
> at
> org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:929)
> at
> org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:62)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at
>
[jira] [Commented] (CASSANDRA-15172) AbstractLocalAwareExecutorService Exception During Upgrade to 3.11.4
[
https://issues.apache.org/jira/browse/CASSANDRA-15172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900876#comment-16900876
]
feroz shaik commented on CASSANDRA-15172:
-
Full stack trace is as below:
WARN [ReadStage-4] 2019-08-06 02:57:57,408
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-4,5,main]: {}
java.lang.NullPointerException: null
at
org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.updateDigest(LegacyLayout.java:2433)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.LegacyLayout$LegacyUnfilteredPartition.digest(LegacyLayout.java:1479)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.rows.UnfilteredRowIterators.digest(UnfilteredRowIterators.java:182)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators.digest(UnfilteredPartitionIterators.java:263)
~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.db.ReadResponse.makeDigest(ReadResponse.java:140)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.ReadResponse.createDigestResponse(ReadResponse.java:87)
~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:352)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:50)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
~[apache-cassandra-3.11.4.jar:3.11.4]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_131]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
~[apache-cassandra-3.11.4.jar:3.11.4]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114)
[apache-cassandra-3.11.4.jar:3.11.4]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
> AbstractLocalAwareExecutorService Exception During Upgrade to 3.11.4
>
>
> Key: CASSANDRA-15172
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15172
> Project: Cassandra
> Issue Type: Bug
>Reporter: Shalom
>Priority: Normal
>
> Hi All,
> This is the first time I open an issue, so apologies if I'm not following the
> rules properly.
>
> After upgrading a node from version 2.1.21 to 3.11.4, we've started seeing a
> lot of AbstractLocalAwareExecutorService exceptions. This happened right
> after the node successfully started up with the new 3.11.4 binaries.
> INFO [main] 2019-06-05 04:41:37,730 Gossiper.java:1715 - No gossip backlog;
> proceeding
> INFO [main] 2019-06-05 04:41:38,036 NativeTransportService.java:70 - Netty
> using native Epoll event loop
> INFO [main] 2019-06-05 04:41:38,117 Server.java:155 - Using Netty Version:
> [netty-buffer=netty-buffer-4.0.44.Final.452812a,
> netty-codec=netty-codec-4.0.44.Final.452812a,
> netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a,
> netty-codec-http=netty-codec-http-4.0.44.Final.452812a,
> netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a,
> netty-common=netty-common-4.0.44.Final.452812a,
> netty-handler=netty-handler-4.0.44.Final.452812a,
> netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb,
> netty-transport=netty-transport-4.0.44.Final.452812a,
> netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a,
> netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a,
> netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a,
> netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a]
> INFO [main] 2019-06-05 04:41:38,118 Server.java:156 - Starting listening for
> CQL clients on /0.0.0.0:9042 (unencrypted)...
> INFO [main] 2019-06-05 04:41:38,179 CassandraDaemon.java:556 - Not starting
> RPC server as requested. Use JMX (StorageService->startRPCServer()) or
> nodetool (enablethrift) to start it
> INFO [Native-Transport-Requests-21] 2019-06-05 04:41:39,145
> AuthCache.java:161 - (Re)initializing PermissionsCache (validity
> period/update interval/max entries) (2000/2000/1000)
> INFO [OptionalTasks:1] 2019-06-05 04:41:39,729 CassandraAuthorizer.java:409
> - Converting legacy permissions data
> INFO [HANDSHAKE-/10.10.10.8] 2019-06-05 04:41:39,808
> OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.8
> INFO [HANDSHAKE-/10.10.10.9] 2019-06-05 04:41:39,808
> OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.9
> INFO [HANDSHAKE-dc1_02/10.10.10.6] 2019-06-05 04:41:39,809
>
[jira] [Commented] (CASSANDRA-15172) AbstractLocalAwareExecutorService Exception During Upgrade to 3.11.4
[
https://issues.apache.org/jira/browse/CASSANDRA-15172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900581#comment-16900581
]
feroz shaik commented on CASSANDRA-15172:
-
We have also hit this problem today while upgrading from 2.1.16 to 3.11.4/
we encountered this as soon as node started up with 3.11.4
ERROR [Native-Transport-Requests-32] 2019-08-06 02:14:20,353
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
and the below errors continued in the logfile as long as the process was up.
ERROR [Native-Transport-Requests-12] 2019-08-06 03:00:47,135
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:48,778
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-13] 2019-08-06 03:00:57,454
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-11] 2019-08-06 03:00:57,482
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-2] 2019-08-06 03:00:58,543
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-8] 2019-08-06 03:00:58,899
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-17] 2019-08-06 03:00:59,074
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-12] 2019-08-06 03:01:08,123
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-17] 2019-08-06 03:01:19,055
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-4] 2019-08-06 03:01:20,880
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
WARN [ReadStage-13] 2019-08-06 03:01:29,983
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-13,5,main]: {}
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-2] 2019-08-06 03:01:31,119
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-6] 2019-08-06 03:01:46,262
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-15] 2019-08-06 03:01:46,520
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
WARN [ReadStage-2] 2019-08-06 03:01:48,842
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-2,5,main]: {}
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-1] 2019-08-06 03:01:50,351
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-5] 2019-08-06 03:02:06,061
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
WARN [ReadStage-8] 2019-08-06 03:02:07,616
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-8,5,main]: {}
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-17] 2019-08-06 03:02:08,384
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
ERROR [Native-Transport-Requests-5] 2019-08-06 03:02:10,244
ErrorMessage.java:384 - Unexpected exception during request
java.lang.NullPointerException: null
The nodetool version says 3.11.4 and the no of connections on 9042 was similar
to other nodes. The exceptions were scary that we had to call off the change.
Any help and insights to this problem from the community is appreciated.
> AbstractLocalAwareExecutorService Exception During Upgrade to 3.11.4
>
>
> Key: CASSANDRA-15172
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15172
> Project: Cassandra
> Issue Type: Bug
>Reporter: Shalom
>Priority: Normal
>
> Hi All,
> This is the first time I open an issue, so apologies if I'm not following the
> rules properly.
>
> After upgrading a node from version 2.1.21 to 3.11.4, we've started seeing a
> lot of AbstractLocalAwareExecutorService exceptions. This happened right
> after the node successfully started up with the new 3.11.4 binaries.
> INFO [main] 2019-06-05 04:41:37,730 Gossiper.java:1715 - No gossip backlog;
> proceeding
> INFO [main] 2019-06-05 04:41:38,036
[jira] [Commented] (CASSANDRA-15172) AbstractLocalAwareExecutorService Exception During Upgrade to 3.11.4
[
https://issues.apache.org/jira/browse/CASSANDRA-15172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16888040#comment-16888040
]
Shalom commented on CASSANDRA-15172:
Trying to push this up a bit because I still want to upgrade to 3.11.4 but I
fear that this issue may recur.
If someone has any idea on what happened here or how to mitigate it, that'd be
awesome!
Thanks!
> AbstractLocalAwareExecutorService Exception During Upgrade to 3.11.4
>
>
> Key: CASSANDRA-15172
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15172
> Project: Cassandra
> Issue Type: Bug
>Reporter: Shalom
>Priority: Normal
>
> Hi All,
> This is the first time I open an issue, so apologies if I'm not following the
> rules properly.
>
> After upgrading a node from version 2.1.21 to 3.11.4, we've started seeing a
> lot of AbstractLocalAwareExecutorService exceptions. This happened right
> after the node successfully started up with the new 3.11.4 binaries.
> INFO [main] 2019-06-05 04:41:37,730 Gossiper.java:1715 - No gossip backlog;
> proceeding
> INFO [main] 2019-06-05 04:41:38,036 NativeTransportService.java:70 - Netty
> using native Epoll event loop
> INFO [main] 2019-06-05 04:41:38,117 Server.java:155 - Using Netty Version:
> [netty-buffer=netty-buffer-4.0.44.Final.452812a,
> netty-codec=netty-codec-4.0.44.Final.452812a,
> netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a,
> netty-codec-http=netty-codec-http-4.0.44.Final.452812a,
> netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a,
> netty-common=netty-common-4.0.44.Final.452812a,
> netty-handler=netty-handler-4.0.44.Final.452812a,
> netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb,
> netty-transport=netty-transport-4.0.44.Final.452812a,
> netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a,
> netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a,
> netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a,
> netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a]
> INFO [main] 2019-06-05 04:41:38,118 Server.java:156 - Starting listening for
> CQL clients on /0.0.0.0:9042 (unencrypted)...
> INFO [main] 2019-06-05 04:41:38,179 CassandraDaemon.java:556 - Not starting
> RPC server as requested. Use JMX (StorageService->startRPCServer()) or
> nodetool (enablethrift) to start it
> INFO [Native-Transport-Requests-21] 2019-06-05 04:41:39,145
> AuthCache.java:161 - (Re)initializing PermissionsCache (validity
> period/update interval/max entries) (2000/2000/1000)
> INFO [OptionalTasks:1] 2019-06-05 04:41:39,729 CassandraAuthorizer.java:409
> - Converting legacy permissions data
> INFO [HANDSHAKE-/10.10.10.8] 2019-06-05 04:41:39,808
> OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.8
> INFO [HANDSHAKE-/10.10.10.9] 2019-06-05 04:41:39,808
> OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.9
> INFO [HANDSHAKE-dc1_02/10.10.10.6] 2019-06-05 04:41:39,809
> OutboundTcpConnection.java:561 - Handshaking version with dc1_02/10.10.10.6
> WARN [ReadStage-2] 2019-06-05 04:41:39,857
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[ReadStage-2,5,main]: {}
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545)
> at
> org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522)
> at
> org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446)
> at
> org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352)
> at
> org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171)
> at
> org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:77)
> at
> org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:802)
> at
> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:953)
> at
> org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:929)
> at
> org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:62)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at
>
