On 01/31/2013 11:59 AM, Gabriel Hurley wrote:
Even though I don't experience this problem (and prefer nginx to
apache), I can help diagnose:
Connections ending up in CLOSE_WAIT means that the socket isn't being
fully closed, which is controlled by the client lib (in this case
python-keystoneclient) which uses httplib2 under the hood.
Expanding on that a bit. CLOSE_WAIT is the state a TCP endpoint will
enter upon receiving a FINished segment from the remote TCP. When the
FIN arrives, the local application will receive notification of this via
the classic "read return of zero" on a receive/read call against the
socket. The FIN segment "means" "I will be sending you no more data."
Meanwhile, the local TCP will have ACKed the FIN segment, and the remote
TCP will transition to FIN_WAIT_2 upon receipt of that ACK (until then
it will be in FIN_WAIT_1).
Depending on how the remote application triggered the sending of the
FIN, the TCP connection is now in a perfectly valid simplex state
wherein the side in CLOSE_WAIT can continue sending data to the side
which will now be in FIN_WAIT_2. It is exceedingly rare for
applications to want a simplex TCP connection
If such a unidirectional TCP connection is not of any use to an
application, (the common case)then that application should/must also
close the connection upon the read return of zero.
Thus, seeing lots of connections "stuck" in CLOSE_WAIT is an indication
of an application-level (relative to TCP) bug wherein the application on
the "CLOSE_WAIT side" is ignoring the read return of zero.
Such bugs in applications may be "masked" by a few things:
1) If the remote side called close() rather than shutdown(SHUT_WR) then
an attempt on the CLOSE_WAIT side to send data to the remote will cause
the remote TCP to return a RST segment (reset) because there is no
longer anything above TCP to receive the data. This will then cause the
local TCP to terminate the connection. This may also happen if the
local application set SO_KEEPALIVE to enable TCP keepalives.
*) If the local side doesn't send anything, and doesn't have TCP
keepalives set, if the remote TCP has a FIN_WAIT_2 timer of some sort
going (long story involving a hole in the TCP specification and
implementation workarounds, email if you want to hear it) then when that
FIN_WAIT_2 timer expires the remote TCP may sent a RST segment.
RST segments are "best effort" in sending - they don't get retransmitted
explicitly. In case 1 if the RST segment doesn't make it back, the
local TCP will retransmit the data it was sending (because it will not
have received an ACKnowledgement either). It will then either receive
the RST triggered by that retransmission, or if no RSTs ever make it
back, the local TCP will at some point reach its retransmission limit
and terminate the connection. In case 2, if that one RST is lost,
that's it, and the CLOSE_WAIT may remain forever.
Again though, given the rarity of actual application use of a simplex
TCP connection, 99 times out of 10, seeing lots of CLOSE_WAIT
connections building-up implies a buggy application or the libraries
doing work on its behalf.
rick jones
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
