Lars Engholm Johansen wrote:
Our company are running several Tomcat 7.0.52 high volume Ubuntu 12.04 production servers. We are using Tomcat WebSockets (JSR356 implementation) heavily with 100M text messages (100GiB) per day.We monitor webserver health by measuring several key parameters every minute, including tomcat connection count using: mBeanServer.getAttribute(threadPool, "connectionCount"); // threadPool is MBean of "type=ThreadPool" ______The problem______ Our sites can run for days without problems, but once in a while the tomcat connection count suddenly starts growing abnormally fast. See this graph: http://imgur.com/s4fOUte netstat shows these TCP connections to be mostly in CLOSE_WAIT state.
And if at that moment, you force the JVM that runs Tomcat to do a Garbage Collection, do you still have these numerous connections in CLOSE_WAIT state after the GC completed ?
Our sites still functions normally with no cpu spikes during this build up until around 60,000 connections, but then the server refuses further connections and a manual Tomcat restart is required. We have no output in tomcat or our logs at the time when this event occurs. The only sign is when comparing full java thread dump with a dump from a newly launched Tomcat: One of http-nio-80-ClientPoller-0 or http-nio-80-ClientPoller-1 is missing/has died. We have observed this problem at least since Tomcat 7.0.48 and can not find indications in Tomcat 7.0.x change logs that it should have been fixed in newer releases. Any help or advises are appreciated, Best regards, Lars Engholm Johansen Our connector configuration: <Connector acceptCount="1500" acceptorThreadCount="4" asyncTimeout="100000" connectionTimeout="60000" connectionUploadTimeout="120000" disableUploadTimeout="false" enableLookups="false" keepAliveTimeout="120000" maxConnections="100000" maxPostSize="3000000" maxThreads="300" port="80" protocol="org.apache.coyote.http11.Http11NioProtocol" socket.soKeepAlive="true" />
--------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
