On 28.05.2009 00:25, CrystalCracker wrote: > Apache 2.2 > Tomcat 6 > Mod_jk 1.2.21 > > All of them are running on the same box. > > I have at any given time around 300 active sessions using the site, and upto > 450 at max. Each user on average logs on to the site for around 15 minutes, > and the calls are usually big and slow database or web- service calls to > various backend systems. The session time out is 30 minutes, but a few users > stay logged in for hours. > > My Major settings are: > > //Apache httpd.conf > KeepAlive Off > Timeout 120 > > <IfModule prefork.c> > StartServers 8 > MinSpareServers 5 > MaxSpareServers 20 > ServerLimit 512 > MaxClients 512 > MaxRequestsPerChild 4000 > </IfModule> > > //Tomcat server.xml > <!-- Define an AJP 1.3 Connector on port 8009 --> > <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" > maxThreads="512" minSpareThreads="5" maxSpareThreads="20" > /> > > > //Modjk worker.properties > worker.node1.port=8009 > worker.node1.host= localhost > worker.node1.type=ajp13 > worker.node1.lbfactor=1 > > All other settings are using default values. > > Everything is working fine, except that I see a huge spike in Apache threads > and Tomcat threads (all 512 threads are used) every few days, and it becomes > literally unresponive for 10-15 minutes. The no of requests is as usual and > Garbage collection and memory usage seems to be fine. Sometimes it recovers > itself, and but most of the time I end up restarting the servers. > > Upon looking at the mod_jk logs I see a lot of the following, but there are > no errors on tomcat side: > > [Tue May 26 13:38:45 2009][30302:33088] [error] > ajp_get_reply::jk_ajp_common.c (1580): (node1) Tomcat is down or refused > connection. No response has been sent to the client (yet) > [Tue May 26 13:38:45 2009][30302:33088] [info] ajp_service::jk_ajp_common.c > (1891): (node1) receiving from tomcat failed, recoverable operation > attempt=0 > [Tue May 26 13:38:45 2009][30302:33088] [info] ajp_service::jk_ajp_common.c > (1930): (node1) sending request to tomcat failed, recoverable operation > attempt=1 > [Tue May 26 13:38:46 2009][30305:33088] [error] > ajp_connection_tcp_get_message::jk_ajp_common.c (951): (node1) can't receive > the response message from tomcat, network problems or tomcat (127.0.0.1:800 > 9) is down (errno=104) > [Tue May 26 13:38:46 2009][30305:33088] [error] > ajp_get_reply::jk_ajp_common.c (1580): (node1) Tomcat is down or refused > connection. No response has been sent to the client (yet) > [Tue May 26 13:38:46 2009][30305:33088] [info] ajp_service::jk_ajp_common.c > (1891): (node1) receiving from tomcat failed, recoverable operation > attempt=0 > > Is my configuration suited for the kind of load I have? I think errno=104 > means modjk thinks tomcat is down. Why would that happen? Any advices?
When it happens again, produce thread dumps. You get them (on a Unix/Linux type system) by sending the QUIT signal to the Tomcat process, e.g. "kill -QUIT mypid", where mypid is the process id of Tomcat. You should do about three thread dumps, each of them a few seconds apart from the next. The thread dump is a text listing of all threads in the JVM, together with the program stack they are in at the moment of the dump. The dump goes to STDOUT, which is send by the Tomcat startup script to the file catalina.out. If really all 512 threads are busy doing something, the output will be huge (more than 512 thread stacks), but if you share them with us, we might be able to quickly detect your problem. For the sake of general sanity: - Version 1.2.21 of mod_jk is now more than 2 years old. There was a lot of development since then, so you might want to consider moving to the most recent one (1.2.28). If your update intervals get to big, your risk and chance of running into an incompatibility increases. Note: I'm not expecting your problem to go away because of upgrading. Note: Between 21 and 28 might have been the change, that JkMount is no longer inhertied between VirtualHosts or between the global server and VirtualHosts. If you use VirtualHosts, put JkMount into them or set "JkMountCopy All" in the global server. - Use timeouts. Use connectionTimeout on the AJP connector of Tomcat, and have a look at http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html concerning mod_jk. Some of the Timeouts might not exist for version 1.2.21. The full attribute documentation contains the info, which attribute was created for hich version: http://tomcat.apache.org/connectors-doc/reference/workers.html Regards, Rainer --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
