On 26.03.2018 23:36, Shawn Heisey wrote:
On 3/26/2018 2:39 PM, André Warnier (tomcat) wrote:
Just a question, more to satisfy my curiosity : when you have these
hundreds of "pending" connections, in what state are they, TCP/IP-wise ?
Not sure where you got "pending". I don't recall mentioning anything
like that.
The TCP state is ESTABLISHED. Entries in the TCP table on port 3306
with the ESTABLISHED state is how we have Xymon keeping track of the
connection count.
Slightly redacted screenshot:
https://www.dropbox.com/s/4yj8y2hzfrnr2aj/xymon-port-monitoring-mysql.png?dl=0
It's not a low-level networking issue.
Sorry for my use of the vague "pending". I meant "unused", "just sitting there".
I wasn't thinking of a low-level networking issue, but more like a case like the
following. I am not a java expert, so I'll do this in text :
Some code creates a new object, which implicitly opens a connection to a
back-end server.
Then this code returns, and the created object is now out of scope (including its
underlying connection). The unreachable object at some point will be destroyed, giving the
programmer a warm feeling that all is well.
But with java this does not necessarily happen right away : it will happen when there is a
garbage collection. And only at that point will the underlying connection be closed and
disappear.
So if you have a very comfortable Heap, it may take a very long time for these "abandoned"
connections to be closed, as part of the destruction of the objects which contain them.
And in the meantime they accumulate (without being used, because they can only be used via
the object, and the object is now unreachable).
The non-intutive part is that, the bigger the available Heap, the more such connections
accumulate, because the interval between GCs is longer.
In my case, it was a bit more tricky still, in the sense that the connection was closed on
one side and not on the other, so these connections appeared in the "CLOSE_WAIT" state.
But under Linux, when reaching several hundreds such CLOSE_WAIT connections, the TCP/IP
stack for some reason stops being responsive, leading to system paralysis.
(So we had the problem on the front-end client, not on the back-end server).
A forced java garbage collection cleared things up (which gave us a hint about what was
happening), but it nevertheless took us a while to diagnose this properly (we also did not
have the source code of the application doing this, so we had to guess).
The above is probably not your case exactly, but it may give you some hint as to how to
find the underlying issue quicker.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
