[
https://issues.apache.org/jira/browse/CASSANDRA-21558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18103456#comment-18103456
]
Runtian Liu commented on CASSANDRA-21558:
-----------------------------------------
4.1 PR: [https://github.com/apache/cassandra/pull/5009]
If this looks good, I can create PR for the other branches.
> Client connections that complete the CQL handshake after nodetool
> disablebinary are never closed
> ------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-21558
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21558
> Project: Apache Cassandra
> Issue Type: Bug
> Reporter: Runtian Liu
> Assignee: Runtian Liu
> Priority: Normal
> Time Spent: 10m
> Remaining Estimate: 0h
>
> A client channel is registered with \{{Server.ConnectionTracker}} only when
> the STARTUP
> message is processed, not when the TCP connection is accepted:
> {\{InitialConnectionHandler}} calls \{{factory.newConnection(...)}}, and the
> \{{Connection}}
> constructor calls \{{tracker.addConnection(channel, this)}}.
>
> {\{Server.stop()}} -> \{{close()}} -> \{{ConnectionTracker.closeAll()}}
> closes only the
> channels present in \{{allChannels}} at the instant it runs, and
> \{{DefaultChannelGroup}}
> does not reject additions made afterwards.
>
> So a connection accepted before \{{disablebinary}} that sends STARTUP after
> \{{closeAll()}}
> has iterated is inserted into \{{allChannels}} after the sweep, and is never
> closed by
> anything. It stays open on a node reporting \{{isNativeTransportRunning() ==
> false}}.
>
> || time || client || server ||
> | t0 | TCP connect completes | channel accepted, not yet in allChannels |
> | t1 | | nodetool disablebinary -> closeAll() closes the channels it can see |
> | t2 | sends STARTUP | addConnection() inserts the channel into allChannels |
> | t3 | | channel is live, transport is stopped, nothing will ever close it |
>
> The leak is present on 4.0, 4.1, 5.0 and trunk. The visible symptom differs
> because
> CASSANDRA-19534 added a dispatch-time guard in 5.0:
>
> * 4.0.x / 4.1.x: the leaked connection is a fully usable CQL session on a
> node whose
> native transport is stopped. We hit this draining a datacenter: after
> disabling the
> binary protocol and removing that DC from a keyspace's replication, the
> surviving
> session stayed pinned to a coordinator with RF=0 for that keyspace, so every
> LOCAL_QUORUM query on it failed with UnavailableException while the cluster
> was healthy.
> The session never broke, so the driver never reconnected elsewhere.
> * 5.0.x / trunk: \{{Dispatcher.dispatch()}} rejects requests on a non-running
> tracker with
> OverloadedException("Server is shutting down"), which contains the data-plane
> impact but
> does not fix the leak. The channel is still never closed, the driver keeps it
> in its
> pool, and a later \{{enablebinary}} silently turns it back into a serving
> session.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
