David Kumar wrote:
Hey,
I'm using:
lsof -u tomcat
example of "netstat" output on one of our own systems right now :
1) first one
vovm1:~# netstat -t -pan | grep -v ESTAB
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
PID/Program name
tcp 0 0 0.0.0.0:32032 0.0.0.0:* LISTEN
2999/usbsrvd
tcp 0 0 0.0.0.0:20224 0.0.0.0:* LISTEN
2965/inetd
tcp 0 0 0.0.0.0:11022 0.0.0.0:* LISTEN
9179/perl
tcp 0 0 127.0.0.1:111 0.0.0.0:* LISTEN
2172/portmap
tcp 0 0 0.0.0.0:54321 0.0.0.0:* LISTEN
18279/python
tcp 0 0 0.0.0.0:4949 0.0.0.0:* LISTEN
23225/munin-node
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
2431/sshd
tcp 0 0 0.0.0.0:23 0.0.0.0:* LISTEN
2965/inetd
tcp 0 0 0.0.0.0:43961 0.0.0.0:* LISTEN
-
tcp 0 0 192.168.20.43:25 0.0.0.0:* LISTEN
2946/exim4
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN
2946/exim4
tcp 0 0 0.0.0.0:11002 0.0.0.0:* LISTEN
2965/inetd
tcp 0 0 0.0.0.0:46911 0.0.0.0:* LISTEN
2183/rpc.statd
tcp 0 0 127.0.0.1:11002 127.0.0.1:40739 TIME_WAIT
-
tcp 0 0 127.0.0.1:11002 127.0.0.1:40735 TIME_WAIT
-
tcp 0 0 127.0.0.1:11002 127.0.0.1:40728 TIME_WAIT
-
tcp 0 0 127.0.0.1:11002 127.0.0.1:40724 TIME_WAIT
-
tcp 1 0 127.0.0.1:54321 127.0.0.1:37916 CLOSE_WAIT
18279/python
tcp 0 0 127.0.0.1:43797 127.0.0.1:8009 TIME_WAIT
-
tcp 0 0 192.168.20.43:56603 192.168.20.80:111 TIME_WAIT
-
tcp6 0 0 :::8009 :::* LISTEN
9206/jsvc
tcp6 0 0 :::80 :::* LISTEN
10730/apache2
tcp6 0 0 :::8180 :::* LISTEN
9206/jsvc
tcp6 0 0 :::22 :::* LISTEN
2431/sshd
tcp6 0 0 :::11100 :::* LISTEN
9133/java
tcp6 0 0 :::11101 :::* LISTEN
9160/java
tcp6 0 0 127.0.0.1:11100 127.0.0.1:38143 TIME_WAIT
-
tcp6 0 0 127.0.0.1:40725 127.0.0.1:11002 TIME_WAIT
-
tcp6 0 0 127.0.0.1:11101 127.0.0.1:54664 TIME_WAIT
-
tcp6 1 0 127.0.0.1:8009 127.0.0.1:35156 CLOSE_WAIT
9206/jsvc
tcp6 0 0 212.85.38.148:80 62.99.208.50:49516 TIME_WAIT
-
tcp6 0 0 212.85.38.148:80 62.155.235.89:50654 TIME_WAIT
-
tcp6 0 0 127.0.0.1:11101 127.0.0.1:54649 TIME_WAIT
-
tcp6 1 0 127.0.0.1:8009 127.0.0.1:33159 CLOSE_WAIT
9206/jsvc
tcp6 1 0 127.0.0.1:8009 127.0.0.1:60682 CLOSE_WAIT
9206/jsvc
tcp6 1 0 127.0.0.1:8009 127.0.0.1:33138 CLOSE_WAIT
9206/jsvc
tcp6 0 0 127.0.0.1:40743 127.0.0.1:11002 TIME_WAIT
-
tcp6 0 0 127.0.0.1:40733 127.0.0.1:11002 TIME_WAIT
-
tcp6 0 0 127.0.0.1:40740 127.0.0.1:11002 TIME_WAIT
-
tcp6 0 0 127.0.0.1:11100 127.0.0.1:38136 TIME_WAIT
-
tcp6 0 0 127.0.0.1:11101 127.0.0.1:54656 TIME_WAIT
-
tcp6 0 0 127.0.0.1:11101 127.0.0.1:54652 TIME_WAIT
-
tcp6 0 0 127.0.0.1:11100 127.0.0.1:50778 TIME_WAIT
-
tcp6 0 0 127.0.0.1:40732 127.0.0.1:11002 TIME_WAIT
-
tcp6 0 0 127.0.0.1:40742 127.0.0.1:11002 TIME_WAIT
-
tcp6 0 0 127.0.0.1:11101 127.0.0.1:54667 TIME_WAIT
-
tcp6 0 0 127.0.0.1:40729 127.0.0.1:11002 TIME_WAIT
-
tcp6 0 0 127.0.0.1:40736 127.0.0.1:11002 TIME_WAIT
-
tcp6 0 0 127.0.0.1:11100 127.0.0.1:38140 TIME_WAIT
-
vovm1:~#
(The "grep -v ESTAB" is to eliminate the "ESTABLISHED" connections, which are maybe
irrelevant here).
The "jsvc" process here is the wrapper which wraps the JVM and Tomcat to allow them to use
ports below 1024 and still run as non-root. For practical purposes, consider it as "tomcat".
As you can see, there are some AJP connections (local port 8009) in the CLOSE_WAIT state
also. It is a normal state of a TCP connection.
What is less normal, is that they would remain in that state for a long time. That means
that one of the sides of the TCP connection is not doing what it should do.
In my experience, under Linux, this can become a problem if you have hundreds
of these.
At some point, the TCP stack becomes unresponsive and does not allow any new
connection.
Sometimes, it can happen because the "client" side in the connection (the one who
initially creates the connection to a server's "listening" socket), discards a Java object
which contains an open low-level OS socket object with a connection still open, without
explicitly "closing" this connection first.
The discarded (and unreachable) Java object is then still sitting in the heap for a while,
and only a GC will actually destroy it and as a side-effect close the underlying connection.
That is why I was asking you to force a GC, to see if this made your CLOSE_WAIT
connections disappear. Apparently in your case it doesn't, which mean that yours is
another case.
Now, instead of restarting Tomcat or doing a GC, have you tried to restart
Apache ?
or maybe just do a /etc/init.d/apache2 reload. That may kill off and restart the Apache
children processes, and clean up their mod_jk connections to Tomcat.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
