[
https://issues.apache.org/jira/browse/CASSANDRA-21491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Steinmaurer updated CASSANDRA-21491:
-------------------------------------------
Description:
With Cassandra 4.1.11, from time to time we see the following NPE in production:
{code:java}
ERROR [Messaging-EventLoop-3-9] 2026-07-07 14:31:19,408
InboundMessageHandler.java:298 -
/10.244.123.131:7000->/10.244.18.26:7000-LARGE_MESSAGES-24ff2250 unexpected
exception caught while processing inbound messages; terminating connection
java.lang.NullPointerException: null
at
org.apache.cassandra.metrics.ClientMetrics.unpauseConnection(ClientMetrics.java:91)
at
org.apache.cassandra.net.AbstractMessageHandler.onReserveCapacityRegained(AbstractMessageHandler.java:314)
at
org.apache.cassandra.net.AbstractMessageHandler.onGlobalReserveCapacityRegained(AbstractMessageHandler.java:289)
at
org.apache.cassandra.net.AbstractMessageHandler.access$1600(AbstractMessageHandler.java:134)
at
org.apache.cassandra.net.AbstractMessageHandler$WaitQueue$Ticket.reactivateHandler(AbstractMessageHandler.java:792)
at
org.apache.cassandra.net.AbstractMessageHandler$WaitQueue$Ticket.access$1200(AbstractMessageHandler.java:759)
at
org.apache.cassandra.net.AbstractMessageHandler$WaitQueue$ReactivateHandlers.run(AbstractMessageHandler.java:740)
at
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)
{code}
Throwing Claude (Sonnet 4.6) on it, I get the following (can't judge, if this
makes sense):
---
Root Cause
ClientMetrics.pausedConnections is null when unpauseConnection() is called for
inter-node connections.
The crash chain:
1. pausedConnections is only initialized in ClientMetrics.init()
(ClientMetrics.java:145), which is called from NativeTransportService.java:120
— i.e., only when the CQL native transport starts up.
2. Inter-node messaging (port 7000) starts earlier than the native transport
during Cassandra startup. InboundMessageHandler extends AbstractMessageHandler
and handles these connections.
3. AbstractMessageHandler.onReserveCapacityRegained() (line 314)
unconditionally calls ClientMetrics.instance.unpauseConnection() — for both CQL
client connections (correct) and inter-node LARGE_MESSAGES connections (wrong).
4. If a LARGE_MESSAGE inter-node connection gets throttled due to reserve
capacity pressure and then the capacity is regained, this code path fires
before ClientMetrics.init() has run → pausedConnections is still null → NPE.
---
Fix
The minimal, targeted fix is to initialize pausedConnections at field
declaration rather than lazily inside init():
Claude proposes the following code change:
!image-2026-07-08-10-20-27-313.png!
was:
With Cassandra 4.1.11, from time to time we see the following NPE in production:
{code:java}
ERROR [Messaging-EventLoop-3-9] 2026-07-07 14:31:19,408
InboundMessageHandler.java:298 -
/10.244.123.131:7000->/10.244.18.26:7000-LARGE_MESSAGES-24ff2250 unexpected
exception caught while processing inbound messages; terminating connection
java.lang.NullPointerException: null
at
org.apache.cassandra.metrics.ClientMetrics.unpauseConnection(ClientMetrics.java:91)
at
org.apache.cassandra.net.AbstractMessageHandler.onReserveCapacityRegained(AbstractMessageHandler.java:314)
at
org.apache.cassandra.net.AbstractMessageHandler.onGlobalReserveCapacityRegained(AbstractMessageHandler.java:289)
at
org.apache.cassandra.net.AbstractMessageHandler.access$1600(AbstractMessageHandler.java:134)
at
org.apache.cassandra.net.AbstractMessageHandler$WaitQueue$Ticket.reactivateHandler(AbstractMessageHandler.java:792)
at
org.apache.cassandra.net.AbstractMessageHandler$WaitQueue$Ticket.access$1200(AbstractMessageHandler.java:759)
at
org.apache.cassandra.net.AbstractMessageHandler$WaitQueue$ReactivateHandlers.run(AbstractMessageHandler.java:740)
at
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)
{code}
Throwing Claude (Sonnet 4.6) on it, I get the following:
---
Root Cause
ClientMetrics.pausedConnections is null when unpauseConnection() is called for
inter-node connections.
The crash chain:
1. pausedConnections is only initialized in ClientMetrics.init()
(ClientMetrics.java:145), which is called from NativeTransportService.java:120
— i.e., only when the CQL native transport starts up.
2. Inter-node messaging (port 7000) starts earlier than the native transport
during Cassandra startup. InboundMessageHandler extends AbstractMessageHandler
and handles these connections.
3. AbstractMessageHandler.onReserveCapacityRegained() (line 314)
unconditionally calls ClientMetrics.instance.unpauseConnection() — for both CQL
client connections (correct) and inter-node LARGE_MESSAGES connections (wrong).
4. If a LARGE_MESSAGE inter-node connection gets throttled due to reserve
capacity pressure and then the capacity is regained, this code path fires
before ClientMetrics.init() has run → pausedConnections is still null → NPE.
---
Fix
The minimal, targeted fix is to initialize pausedConnections at field
declaration rather than lazily inside init():
Claude proposes the following code change:
!image-2026-07-08-10-20-27-313.png!
> NPE in ClientMetrics.unpauseConnectionClient
> --------------------------------------------
>
> Key: CASSANDRA-21491
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21491
> Project: Apache Cassandra
> Issue Type: Bug
> Reporter: Thomas Steinmaurer
> Priority: Normal
> Attachments: image-2026-07-08-10-20-27-313.png
>
>
> With Cassandra 4.1.11, from time to time we see the following NPE in
> production:
> {code:java}
> ERROR [Messaging-EventLoop-3-9] 2026-07-07 14:31:19,408
> InboundMessageHandler.java:298 -
> /10.244.123.131:7000->/10.244.18.26:7000-LARGE_MESSAGES-24ff2250 unexpected
> exception caught while processing inbound messages; terminating connection
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.metrics.ClientMetrics.unpauseConnection(ClientMetrics.java:91)
> at
> org.apache.cassandra.net.AbstractMessageHandler.onReserveCapacityRegained(AbstractMessageHandler.java:314)
> at
> org.apache.cassandra.net.AbstractMessageHandler.onGlobalReserveCapacityRegained(AbstractMessageHandler.java:289)
> at
> org.apache.cassandra.net.AbstractMessageHandler.access$1600(AbstractMessageHandler.java:134)
> at
> org.apache.cassandra.net.AbstractMessageHandler$WaitQueue$Ticket.reactivateHandler(AbstractMessageHandler.java:792)
> at
> org.apache.cassandra.net.AbstractMessageHandler$WaitQueue$Ticket.access$1200(AbstractMessageHandler.java:759)
> at
> org.apache.cassandra.net.AbstractMessageHandler$WaitQueue$ReactivateHandlers.run(AbstractMessageHandler.java:740)
> at
> io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Unknown Source)
> {code}
> Throwing Claude (Sonnet 4.6) on it, I get the following (can't judge, if this
> makes sense):
> ---
> Root Cause
> ClientMetrics.pausedConnections is null when unpauseConnection() is called
> for inter-node connections.
> The crash chain:
> 1. pausedConnections is only initialized in ClientMetrics.init()
> (ClientMetrics.java:145), which is called from
> NativeTransportService.java:120 — i.e., only when the CQL native transport
> starts up.
> 2. Inter-node messaging (port 7000) starts earlier than the native transport
> during Cassandra startup. InboundMessageHandler extends
> AbstractMessageHandler and handles these connections.
> 3. AbstractMessageHandler.onReserveCapacityRegained() (line 314)
> unconditionally calls ClientMetrics.instance.unpauseConnection() — for both
> CQL client connections (correct) and inter-node LARGE_MESSAGES connections
> (wrong).
> 4. If a LARGE_MESSAGE inter-node connection gets throttled due to reserve
> capacity pressure and then the capacity is regained, this code path fires
> before ClientMetrics.init() has run → pausedConnections is still null → NPE.
> ---
> Fix
> The minimal, targeted fix is to initialize pausedConnections at field
> declaration rather than lazily inside init():
> Claude proposes the following code change:
> !image-2026-07-08-10-20-27-313.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
