Runtian Liu created CASSANDRA-21558:
---------------------------------------
Summary: Client connections that complete the CQL handshake after
nodetool disablebinary are never closed
Key: CASSANDRA-21558
URL: https://issues.apache.org/jira/browse/CASSANDRA-21558
Project: Apache Cassandra
Issue Type: Bug
Reporter: Runtian Liu
Assignee: Runtian Liu
A client channel is registered with \{{Server.ConnectionTracker}} only when the
STARTUP
message is processed, not when the TCP connection is accepted:
{\{InitialConnectionHandler}} calls \{{factory.newConnection(...)}}, and the
\{{Connection}}
constructor calls \{{tracker.addConnection(channel, this)}}.
{\{Server.stop()}} -> \{{close()}} -> \{{ConnectionTracker.closeAll()}} closes
only the
channels present in \{{allChannels}} at the instant it runs, and
\{{DefaultChannelGroup}}
does not reject additions made afterwards.
So a connection accepted before \{{disablebinary}} that sends STARTUP after
\{{closeAll()}}
has iterated is inserted into \{{allChannels}} after the sweep, and is never
closed by
anything. It stays open on a node reporting \{{isNativeTransportRunning() ==
false}}.
|| time || client || server ||
| t0 | TCP connect completes | channel accepted, not yet in allChannels |
| t1 | | nodetool disablebinary -> closeAll() closes the channels it can see |
| t2 | sends STARTUP | addConnection() inserts the channel into allChannels |
| t3 | | channel is live, transport is stopped, nothing will ever close it |
The leak is present on 4.0, 4.1, 5.0 and trunk. The visible symptom differs
because
CASSANDRA-19534 added a dispatch-time guard in 5.0:
* 4.0.x / 4.1.x: the leaked connection is a fully usable CQL session on a node
whose
native transport is stopped. We hit this draining a datacenter: after disabling
the
binary protocol and removing that DC from a keyspace's replication, the
surviving
session stayed pinned to a coordinator with RF=0 for that keyspace, so every
LOCAL_QUORUM query on it failed with UnavailableException while the cluster was
healthy.
The session never broke, so the driver never reconnected elsewhere.
* 5.0.x / trunk: \{{Dispatcher.dispatch()}} rejects requests on a non-running
tracker with
OverloadedException("Server is shutting down"), which contains the data-plane
impact but
does not fix the leak. The channel is still never closed, the driver keeps it
in its
pool, and a later \{{enablebinary}} silently turns it back into a serving
session.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
