[
https://issues.apache.org/jira/browse/CASSANDRA-9630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15398338#comment-15398338
]
Paulo Motta commented on CASSANDRA-9630:
----------------------------------------
I noticed we're not closing the socket [if there is an exception while
connecting|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/OutboundTcpConnection.java#L496]
on {{OutboundTcpConnection}}, so a race during a node's shutdown might cause a
failed connection attempt to that node remain in the {{CLOSE_WAIT}} state until
the next GC, which could potentially cause this.
[~farzad.panahi] Are you willing to try out [this
patch|https://github.com/pauloricardomg/cassandra/commit/3f46d414b06afb607b6a97152661b10c53c103e6]
to see if it fixes it? You need to replace your
{{lib/apache-cassandra-3.0.8.jar}} with
[apache-cassandra-3.0.8-SNAPSHOT.jar|https://issues.apache.org/jira/secure/attachment/12820814/apache-cassandra-3.0.8-SNAPSHOT.jar]
and perform a rolling restart on some of the nodes and check if this will fix
the issue in these nodes (if you prefer you can generate your own jar by
cloning [this
branch|https://github.com/pauloricardomg/cassandra/tree/3.0.6-9630] and running
{{ant clean jar}}).
If this does not solve it, it would be nice if you could set the logging level
of the {{org.apache.cassandra.net}} package to {{TRACE}}, either via {{nodetool
setlogginglevel org.apache.cassandra.net TRACE}} or by adding {{<logger
name="org.apache.cassandra.net" level="TRACE"/>}} to the end of your
{{conf/logback.xml}}. After this, please attach the relevant information in the
logs of affected nodes to this ticket for further analysis.
> Killing cassandra process results in unclosed connections
> ---------------------------------------------------------
>
> Key: CASSANDRA-9630
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9630
> Project: Cassandra
> Issue Type: Bug
> Components: Distributed Metadata, Streaming and Messaging
> Reporter: Paulo Motta
> Assignee: Paulo Motta
> Priority: Minor
> Fix For: 3.x
>
> Attachments: apache-cassandra-3.0.8-SNAPSHOT.jar
>
>
> After upgrading from Cassandra from 2.0.12 to 2.0.15, whenever we killed a
> cassandra process (with SIGTERM), some other nodes maintained a connection
> with the killed node in the CLOSE_WAIT state on port 7000 for about 5-20
> minutes.
> So, when we started the killed node again, other nodes could not establish a
> handshake because of the connections on the CLOSE_WAIT state, so they
> remained on the DOWN state to each other until the initial connection expired.
> The problem did not happen if I ran a nodetool disablegossip before killing
> the node.
> I was able to fix this issue by reverting the CASSANDRA-8336 commits
> (including CASSANDRA-9238). After reverting this, cassandra now closes
> connection correctly when killed with -TERM, but leaves connections on
> CLOSE_WAIT state if I run nodetool disablethrift before killing the nodes.
> I did not try to reproduce the problem in a clean environment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
