On 28.05.2009 22:05, CrystalCracker wrote: > The problem has always occured at the oddest hours for me to do a thread > dump. I have done jmeter load test and tried to recreate the problem in > test, but have never able. > I will upgrade to the latest version and then play with the connection > timeouts.
You can also add a little script as a cron job, doing the three dumps like twice an hour. So you have a chance to capture something, at least if the problem lasts long enough. Try to write a timestamp to catalina.out before calling the dumps, so that you can reconstruct, which dump belongs to the observed problem. BTW: If it always occurs at odd hours, then that already might be an indication of the root cause, e.g. some backend system (database, mainframe, whatever) or similar doing nightly maintenance. Regards, Rainer > CrystalCracker wrote: >> Apache 2.2 >> Tomcat 6 >> Mod_jk 1.2.21 >> >> All of them are running on the same box. >> >> I have at any given time around 300 active sessions using the site, and >> upto 450 at max. Each user on average logs on to the site for around 15 >> minutes, and the calls are usually big and slow database or web- service >> calls to various backend systems. The session time out is 30 minutes, but >> a few users stay logged in for hours. >> >> My Major settings are: >> >> //Apache httpd.conf >> KeepAlive Off >> Timeout 120 >> >> <IfModule prefork.c> >> StartServers 8 >> MinSpareServers 5 >> MaxSpareServers 20 >> ServerLimit 512 >> MaxClients 512 >> MaxRequestsPerChild 4000 >> </IfModule> >> >> //Tomcat server.xml >> <!-- Define an AJP 1.3 Connector on port 8009 --> >> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" >> maxThreads="512" minSpareThreads="5" maxSpareThreads="20" >> /> >> >> >> //Modjk worker.properties >> worker.node1.port=8009 >> worker.node1.host= localhost >> worker.node1.type=ajp13 >> worker.node1.lbfactor=1 >> >> All other settings are using default values. >> >> Everything is working fine, except that I see a huge spike in Apache >> threads and Tomcat threads (all 512 threads are used) every few days, and >> it becomes literally unresponive for 10-15 minutes. The no of requests is >> as usual and Garbage collection and memory usage seems to be fine. >> Sometimes it recovers itself, and but most of the time I end up restarting >> the servers. >> >> Upon looking at the mod_jk logs I see a lot of the following, but there >> are no errors on tomcat side: >> >> [Tue May 26 13:38:45 2009][30302:33088] [error] >> ajp_get_reply::jk_ajp_common.c (1580): (node1) Tomcat is down or refused >> connection. No response has been sent to the client (yet) >> [Tue May 26 13:38:45 2009][30302:33088] [info] >> ajp_service::jk_ajp_common.c (1891): (node1) receiving from tomcat failed, >> recoverable operation attempt=0 >> [Tue May 26 13:38:45 2009][30302:33088] [info] >> ajp_service::jk_ajp_common.c (1930): (node1) sending request to tomcat >> failed, recoverable operation attempt=1 >> [Tue May 26 13:38:46 2009][30305:33088] [error] >> ajp_connection_tcp_get_message::jk_ajp_common.c (951): (node1) can't >> receive the response message from tomcat, network problems or tomcat >> (127.0.0.1:800 >> 9) is down (errno=104) >> [Tue May 26 13:38:46 2009][30305:33088] [error] >> ajp_get_reply::jk_ajp_common.c (1580): (node1) Tomcat is down or refused >> connection. No response has been sent to the client (yet) >> [Tue May 26 13:38:46 2009][30305:33088] [info] >> ajp_service::jk_ajp_common.c (1891): (node1) receiving from tomcat failed, >> recoverable operation attempt=0 >> >> Is my configuration suited for the kind of load I have? I think errno=104 >> means modjk thinks tomcat is down. Why would that happen? Any advices? >> >> Thanks a lot. --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
