Rainer Jung wrote:
Please have a look at your mod_jk log file and look for '[error]'
messages. Around the error messages there should also be additional
info messages giving some mor einfo about the root cause.
mod_jk.log is empty.
OK, you could switch to mod_jk 1.2.26 so we don't have to check
against any old bugs. 1.2.26 is now 3 months old and no major new bugs
are known.
OK, I'll do that this weekend..
That sounds really weird. If you stop and start Apache httpd and
Tomcat (i.e. all processes completely stopped, wait a few seconds and
then started freshly), those processes should not have any more
knowledge of previous problems.
That is what I would have thought.. no joy... I attached the netstat
output so you can see all of the lines for yourself.. Hope that helps..
Possibly related information:
netstat -np --tcp reports dozens of "dead" connections to port 8009,
like the following samples: (count in parens to save space here):
tcp 0 0 127.0.0.1:8009
127.0.0.1:48461 SYN_RECV - (13)
tcp 0 6 127.0.0.1:48459
127.0.0.1:8009 FIN_WAIT1 - (3) tcp 6
0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48191 CLOSE_WAIT
- (50)
The CLOSE_WAIT most likely come from the fact, that you use a
connect_timeout (that's fine) for mod_jk, but have no
connectionTimeout on teh Toncat side. Set connectionTimeout in the
Tomcat sidfe in the AJP connector to 600000 (milliseconds).
Actually I do.. Here is that setting from my server.xml
<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8009"
enableLookups="false"
redirectPort="8443"
protocol="AJP/1.3"
connectionTimeout="60000"
maxThreads="750"
debug="0" />
Is there anything else here that needs changing..?
The SYN_RECV seem to indicate, that your Tomcat is not able to accept
any more connections. Two possible reasons:
- all threads in the thread pool of your AJP connector (Tomcat side)
are already connected with httpd and no more threads are left. E.g. in
your case it could be, that 50 threads are connected and maybe the
size of your thread pool is 50?
- you got an exception during creation of a new thread for the pool,
and as a consequence your Tomcat is not listening any more. Can you
see the LISTEN for 8009 in netstat? If no, the most likely cause is,
that your JVM exceeded the number of threads that it can create (most
32 Bit JVMs: something between 400 and 500).
This is a 64 bit system and using 64 bit JVM... I was seeing this
problem also on a 32 bit system and had hoped that the 64 bit JVM would
give me more "margin of error", apparently not. (See attached netstat
output)
If the problem happens again: check to see, if you can telnet to the
TC port (telnet localhost 8009). If no, then mod_jk also has no chance
of connecting to the Tomcat and we must find out, why Tomcat doesn't
accept any more connections. In that case a thread dump of the Tomcat
java process is helpful (kill -QUIT, result goes to catalin.out).
Good suggestion, I'll give that a try..
Thanks for all of the other suggestions, I made those changes
immediately, and I'll upgrade the connector this weekend. If you are
anyone else have some ideas for me to try, I would REALLY appreciate it..
Thanks again..
John..
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address
State PID/Program name
tcp 0 0 127.0.0.1:8009 127.0.0.1:48461
SYN_RECV -
tcp 0 0 127.0.0.1:8009 127.0.0.1:48457
SYN_RECV -
tcp 0 0 127.0.0.1:8009 127.0.0.1:48455
SYN_RECV -
tcp 0 0 127.0.0.1:8009 127.0.0.1:48449
SYN_RECV -
tcp 0 0 127.0.0.1:8009 127.0.0.1:48459
SYN_RECV -
tcp 0 0 127.0.0.1:8009 127.0.0.1:48460
SYN_RECV -
tcp 0 0 127.0.0.1:8009 127.0.0.1:48456
SYN_RECV -
tcp 0 0 127.0.0.1:8009 127.0.0.1:48450
SYN_RECV -
tcp 0 0 127.0.0.1:8009 127.0.0.1:48448
SYN_RECV -
tcp 0 0 127.0.0.1:8009 127.0.0.1:48452
SYN_RECV -
tcp 0 0 127.0.0.1:8009 127.0.0.1:48454
SYN_RECV -
tcp 0 0 127.0.0.1:8009 127.0.0.1:48451
SYN_RECV -
tcp 0 0 127.0.0.1:8009 127.0.0.1:48458
SYN_RECV -
tcp 0 0 127.0.0.1:8009 127.0.0.1:48453
SYN_RECV -
tcp 0 0 127.0.0.1:8009 127.0.0.1:48447
SYN_RECV -
tcp 0 0 207.137.13.188:139 207.137.13.170:1068
ESTABLISHED 27611/smbd
tcp 0 0 207.137.13.188:445 207.137.13.168:1067
ESTABLISHED 344/smbd
tcp 0 0 207.137.13.188:139 207.137.13.176:52041
ESTABLISHED 18400/smbd
tcp 0 6 127.0.0.1:48459 127.0.0.1:8009
FIN_WAIT1 -
tcp 0 6 127.0.0.1:48458 127.0.0.1:8009
FIN_WAIT1 -
tcp 0 6 127.0.0.1:48461 127.0.0.1:8009
FIN_WAIT1 -
tcp 0 6 127.0.0.1:48460 127.0.0.1:8009
FIN_WAIT1 -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48191
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48190
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48189
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48188
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48187
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48186
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48182
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48181
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48180
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48177
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48223
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48222
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48221
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48220
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48219
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48216
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48215
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48214
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48213
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48212
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48211
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48210
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48209
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48208
CLOSE_WAIT -
tcp 0 0 ::ffff:207.137.13.188:48217 ::ffff:207.137.13.184:3050
ESTABLISHED 11965/java
tcp 0 0 ::ffff:207.137.13.188:48218 ::ffff:207.137.13.184:3050
ESTABLISHED 11965/java
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48203
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48202
CLOSE_WAIT -
tcp 0 0 ::ffff:207.137.13.188:48109 ::ffff:207.137.13.184:3050
ESTABLISHED 11965/java
tcp 0 0 ::ffff:207.137.13.188:48048 ::ffff:207.137.13.184:3050
ESTABLISHED 11965/java
tcp 0 0 ::ffff:207.137.13.188:48047 ::ffff:207.137.13.184:3050
ESTABLISHED 11965/java
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48201
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48200
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48199
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48198
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48197
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48196
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48195
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48194
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48193
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48192
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48240
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48239
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48238
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48237
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48236
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48235
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48234
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48233
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48232
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48231
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48230
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48227
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48226
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48225
CLOSE_WAIT -
tcp 6 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48224
CLOSE_WAIT -
tcp 1 0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:44793
CLOSE_WAIT 11965/java
tcp 0 0 ::ffff:207.137.13.188:44744 ::ffff:207.137.13.183:3050
ESTABLISHED 11965/java
tcp 0 0 ::ffff:207.137.13.188:44745 ::ffff:207.137.13.183:3050
ESTABLISHED 11965/java
tcp 0 0 ::ffff:207.137.13.188:44798 ::ffff:207.137.13.183:3050
ESTABLISHED 11965/java
tcp 0 0 ::ffff:207.137.13.188:47989 ::ffff:207.137.13.183:3050
ESTABLISHED 11965/java
tcp 0 0 ::ffff:207.137.13.188:47988 ::ffff:207.137.13.183:3050
ESTABLISHED 11965/java
tcp 0 0 ::ffff:207.137.13.188:48093 ::ffff:207.137.13.183:3050
ESTABLISHED 11965/java
tcp 0 0 ::ffff:207.137.13.188:48069 ::ffff:207.137.13.183:3050
ESTABLISHED 11965/java
tcp 0 0 ::ffff:207.137.13.188:48068 ::ffff:207.137.13.183:3050
ESTABLISHED 11965/java
tcp 0 0 ::ffff:207.137.13.188:48115 ::ffff:207.137.13.183:3050
ESTABLISHED 11965/java
tcp 0 0 ::ffff:207.137.13.188:48116 ::ffff:207.137.13.183:3050
ESTABLISHED 11965/java
tcp 0 0 ::ffff:207.137.13.188:48124 ::ffff:207.137.13.183:3050
ESTABLISHED 11965/java
tcp 0 0 ::ffff:207.137.13.188:48127 ::ffff:207.137.13.183:3050
ESTABLISHED 11965/java
tcp 0 0 ::ffff:207.137.13.188:48126 ::ffff:207.137.13.183:3050
ESTABLISHED 11965/java
tcp 0 0 ::ffff:207.137.13.188:48111 ::ffff:207.137.13.183:3050
ESTABLISHED 11965/java
tcp 0 0 ::ffff:207.137.13.188:48110 ::ffff:207.137.13.183:3050
ESTABLISHED 11965/java
tcp 0 0 ::ffff:207.137.13.188:48001 ::ffff:207.137.13.183:3050
ESTABLISHED 11965/java
tcp 0 0 ::ffff:207.137.13.188:48002 ::ffff:207.137.13.183:3050
ESTABLISHED 11965/java
tcp 0 0 ::ffff:207.137.13.188:48035 ::ffff:207.137.13.183:3050
ESTABLISHED 11965/java
tcp 0 0 ::ffff:207.137.13.188:48034 ::ffff:207.137.13.183:3050
ESTABLISHED 11965/java
tcp 0 0 ::ffff:207.137.13.188:48043 ::ffff:207.137.13.183:3050
ESTABLISHED 11965/java
tcp 0 0 ::ffff:207.137.13.188:48130 ::ffff:207.137.13.183:3050
ESTABLISHED 11965/java
tcp 0 0 ::ffff:207.137.13.188:48131 ::ffff:207.137.13.183:3050
ESTABLISHED 11965/java
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
