Thomas Steinmaurer created CASSANDRA-21491:
----------------------------------------------
Summary: NPE in ClientMetrics.unpauseConnectionClient
Key: CASSANDRA-21491
URL: https://issues.apache.org/jira/browse/CASSANDRA-21491
Project: Apache Cassandra
Issue Type: Bug
Reporter: Thomas Steinmaurer
Attachments: image-2026-07-08-10-20-27-313.png
With Cassandra 4.1.11, from time to time we see the following NPE in production:
{code:java}
ERROR [Messaging-EventLoop-3-9] 2026-07-07 14:31:19,408
InboundMessageHandler.java:298 -
/10.244.123.131:7000->/10.244.18.26:7000-LARGE_MESSAGES-24ff2250 unexpected
exception caught while processing inbound messages; terminating connection
java.lang.NullPointerException: null
at
org.apache.cassandra.metrics.ClientMetrics.unpauseConnection(ClientMetrics.java:91)
at
org.apache.cassandra.net.AbstractMessageHandler.onReserveCapacityRegained(AbstractMessageHandler.java:314)
at
org.apache.cassandra.net.AbstractMessageHandler.onGlobalReserveCapacityRegained(AbstractMessageHandler.java:289)
at
org.apache.cassandra.net.AbstractMessageHandler.access$1600(AbstractMessageHandler.java:134)
at
org.apache.cassandra.net.AbstractMessageHandler$WaitQueue$Ticket.reactivateHandler(AbstractMessageHandler.java:792)
at
org.apache.cassandra.net.AbstractMessageHandler$WaitQueue$Ticket.access$1200(AbstractMessageHandler.java:759)
at
org.apache.cassandra.net.AbstractMessageHandler$WaitQueue$ReactivateHandlers.run(AbstractMessageHandler.java:740)
at
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)
{code}
Throwing Claude (Sonnet 4.6) on it, I get the following:
---
Root Cause
ClientMetrics.pausedConnections is null when unpauseConnection() is called for
inter-node connections.
The crash chain:
1. pausedConnections is only initialized in ClientMetrics.init()
(ClientMetrics.java:145), which is called from NativeTransportService.java:120
— i.e., only when the CQL native transport starts up.
2. Inter-node messaging (port 7000) starts earlier than the native transport
during Cassandra startup. InboundMessageHandler extends AbstractMessageHandler
and handles these connections.
3. AbstractMessageHandler.onReserveCapacityRegained() (line 314)
unconditionally calls ClientMetrics.instance.unpauseConnection() — for both CQL
client connections (correct) and inter-node LARGE_MESSAGES connections (wrong).
4. If a LARGE_MESSAGE inter-node connection gets throttled due to reserve
capacity pressure and then the capacity is regained, this code path fires
before ClientMetrics.init() has run → pausedConnections is still null → NPE.
---
Fix
The minimal, targeted fix is to initialize pausedConnections at field
declaration rather than lazily inside init():
Claude proposes the following code change:
!image-2026-07-08-10-20-27-313.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
