-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Isaac,
On 2/24/14, 2:27 PM, Isaac Gonzalez wrote: > Hello all, > > I'm running tomcat 6.0.32 on Cent OS 6 with 2 front end apache load > balancers with a firewall in between the tomcat and load balancers > using mod_jk v. 1.2.37 under apache 2.2.10 to connect the backend > tomcat. I have had this running ok for a few years but our user > traffic has increased significantly. A few months ago, the tomcat > server seemed to refuse or not accept any new connections from > either load balancer and required a restart on the tomcat end, even > though I could easily connect to tomcat on port 8080(manager). I > can intermittently telnet to port 8009, but am denied a bit as well > both inside and outside the firewall. > > I proceeded to split the tomcats up into their own instances, > hoping when this issue recurred that it would only affect a > particular tomcat app. It also gave our developers the ability to > patch a single tomcat app without downing all of our apps. > > Unfortunately, this issue has recurred several times and I have > spent most of my days researching and digging for hope of someone > with a similar experience that may have resolved it. Last Friday > the problem was so bad, I had to completely restart the tomcat > server(reboot it). > > So far I am at a loss...I have installed psi-probe on all tomcat > instances to give me more in depth analysis to tomcat threads and > related server metadata when the problem is occuring. I have made a > few modifications to workers.properties, in particular to decrease > the connection timeout as well as the tomcat ajp connector from 10 > minutes to 5 minutes and added the ping timeout and socket timeout. > I also increased my apache prefork MPM client connections to 500 on > each load balancer. Below is my relevant configs...any suggestions > to help remedy this would help... I have also increased threads > from 200 to 500 on all tomcat instances. I'd be interested to see a thread dump on a "stuck" Tomcat to see what it's doing. If it happens again, please take a thread dump (or, better yet, 3 or so maybe 5-10 seconds apart) and post them back to the list. http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F Does restarting the Tomcat instance fix everything, or do you have to also bounce httpd? What happens if you bounce only httpd? After the "split", did both Tomcats appear to lock-up simultaneously, or did only one of them have a problem and the other one stayed up? Do the lock-ups appear to be related to anything you can observe, such as particularly high-load, etc.? > Workers.properties: > > worker.list=jkstatus,server1,server2,server3,server4,server5,server6,server7,server8 > > worker.jkstatus.type=status > > # Let's define some defaults worker.basic.port=8009 > worker.basic.type=ajp13 worker.basic.socket_keepalive=True > worker.basic.connection_pool_timeout=300 > worker.basic.ping_timeout=1000 worker.basic.ping_mode=A > worker.basic.socket_timeout=10 > > worker.lb1.distance=0 worker.lb1.reference=worker.basic > > worker.server1.host= server1hostname > worker.server1.reference=worker.lb1 > worker.server2.host=server2hostname > worker.server2.reference=worker.lb1 > worker.server3.host=server3hostname > worker.server3.reference=worker.lb1 worker.server4.host= > server4hostname worker.server4.reference=worker.lb1 > worker.server5.host= server5hostname > worker.server5.reference=worker.lb1 worker.server6.host= > server6hostname worker.server6.reference=worker.lb1 > worker.server7.host= server7hostname > worker.server7.reference=worker.lb1 worker.server8.host= > server7hostname worker.server8.reference=worker.lb1 You didn't show any JkMounts in your httpd.conf file. What worker are you using? It sounded like you were load-balancing the servers, but your "lb1" worker does not have any balance_workers setting so it doesn't look like it's going to work. > httpd.conf: > > KeepAlive Off MaxKeepAliveRequests 100 KeepAliveTimeout 15 > > > > # prefork MPM # StartServers: number of server processes to start # > MinSpareServers: minimum number of server processes which are kept > spare # MaxSpareServers: maximum number of server processes which > are kept spare # ServerLimit: maximum value for MaxClients for the > lifetime of the server # MaxClients: maximum number of server > processes allowed to start # MaxRequestsPerChild: maximum number of > requests a server process serves <IfModule prefork.c> StartServers > 8 MinSpareServers 5 MaxSpareServers 20 ServerLimit 500 > MaxClients 500 MaxRequestsPerChild 5000 </IfModule> It would be good to see your Jk* setting as well. > Tomcat server.xml: > > <!-- Define an AJP 1.3 Connector on port 8009 --> <Connector > port="8009" address="x.x.x.x" protocol="AJP/1.3" > redirectPort="8443" connectionTimeout="300000" maxThreads="500" /> Why do you both having a connectionTimeout on an AJP connection? httpd should only send a request to you once the request line has been received by the client, so there isn't really any legitimate reason for an AJP request to time out before the request-line comes-through. IMO, you should define an <Executor> and share threads between your AJP connector and your HTTP connector (which you didn't show config for, but mentioned you had one above running on port 8080). Otherwise, your Tomcat configuration looks fine. If you get despirate, you can set mod_jk's log level to DEBUG and watch your disk fill up. You'll get great information if/when things start to go south. - -chris -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJTDDCXAAoJEBzwKT+lPKRYfm0P/jx9Dqimk06Aihjpm1Qv9e6r G7wgpmKjpPYOULNk6TVsz0Gs4YljOTklviDvTH2dL1bHszaJNKR5ZOKXlgXiaWRG 3w6HpH2ujgJQczXxVi43eF3bbYnPHJguGEN6pVG4ig5qYGuwPVHNYsvWQBRjVITS IRxnTdCsBtBOgdfAS8WEJNZKK/Ep5s37nehtcZ4Hq8T/q9TkWGnZ7rABglUN0Mex JKnSJR3/VYS1FFH7efV+GDdtlPRFEebXXzjA+DNAYyo2HH4zV0XAfF4zaMAmTHxK MgvP8sQA6klWH3XSkNQRm+SJMD5Kl32JLsIi3osGP7pLowlg/jvI66lcZnkVKhGi SIhgzUSEHLAgBZOO+wIRyTEBU5ALJ6kW3ySZqdL6fF4wDsON1ZV/gXCshFlByJ9q i+jth+jtovmDsLgiwYhKsZ3zr+lfeVVQWjZ3cY5l0kGxaPYpJt+EhEjLtjSMg8Lg 06ZMSBi8hZ+R+fyz1jBzcGadRMTRmN6aaolf/XZOom4i+5qcjrJ5r0cSXFqVBY1D XBfnDRwAAjOfWLCwiQ9XWhDWfyOQZghJ2mBT2vaQbCnZelcK4e3V0Ix8iYUatnTu o2lKiOxsRlkIXRgBfutXu7z3IpdNK1YaotpFtz4hb4lz0dSmDsNP5llpw3zyuQ9l ZMKc94GOw+ZY2Tt2XFYD =tdbH -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
