Re: Tomcat and IIS losing their link
It appears that my problem with IIS losing its link with Tomcat was actually an application problem, as several of you suggested. The JMX thread dump (very helpful over terminal services) indicated that 100 threads were locked up with Hibernate trying to get a database connection from DBCP. Upon further research I learned that Hibernate (and probably everything else) works better with the c3p0 pooling API, so I switched all my applications over to c3p0 and all is well (for now). Thanks for all the help, Travis Haagen - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat and IIS losing their link
Thanks for the responses thus far. I've been trying to get more information about the problem, but haven't been very successfull. I can't seem to create thread dumps, because I have to use Terminal Services. Is there a way to get a thread dump even though I'm using Terminal Services? Tomcat keeps locking up every afternoon, during the work week, so its definitely related to server traffic. Sometimes it locks up twice in an afternoon. I found this article about IIS 6.0 application health monitoring and I was wondering if anyone know if the items listed in the section 'ISAPI Extensions Can Declare Themselves Unhealthy' are implemented in the ISAPI redirector? http://help.alentus.com/article.aspx?id=10958cNode=7F8F8Y Here's what the ISAPI redirector logs say (the following errors are simply repeated until Tomcat is restarted). My log level is currently set to 'WARN': Mon Dec 17 13:48:34.484 2007] [4340:5744] [error] jk_ajp_common.c (966): (ajp13w) can't receive the response message from tomcat, network problems or tomcat (127.0.0.1:8009) is down (errno=60) [Mon Dec 17 13:48:34.484 2007] [4340:5744] [error] jk_ajp_common.c (1658): (ajp13w) Tomcat is down or refused connection. No response has been sent to the client (yet) [Mon Dec 17 13:48:38.390 2007] [4340:3340] [error] jk_ajp_common.c (966): (ajp13w) can't receive the response message from tomcat, network problems or tomcat (127.0.0.1:8009) is down (errno=60) [Mon Dec 17 13:48:38.390 2007] [4340:3340] [error] jk_ajp_common.c (1658): (ajp13w) Tomcat is down or refused connection. No response has been sent to the client (yet) [Mon Dec 17 13:48:47.015 2007] [4340:5744] [error] jk_ajp_common.c (966): (ajp13w) can't receive the response message from tomcat, network problems or tomcat (127.0.0.1:8009) is down (errno=60) [Mon Dec 17 13:48:47.015 2007] [4340:5744] [error] jk_ajp_common.c (1658): (ajp13w) Tomcat is down or refused connection. No response has been sent to the client (yet) [Mon Dec 17 13:48:49.015 2007] [4340:5744] [error] jk_ajp_common.c (2097): (ajp13w) Connecting to tomcat failed. Tomcat is probably not started or is listening on the wrong port - Original Message - From: Rainer Jung [EMAIL PROTECTED] To: Tomcat Users List users@tomcat.apache.org Sent: Friday, December 14, 2007 9:20 AM Subject: Re: Tomcat and IIS losing their link Travis Haagen wrote: Hello, I've got IIS 6.0 linked with Tomcat 5.5.25 (with JDK 1.5.0_14-b03 and AJP dll) via the ISAPI connector (JK 1.2.25) on Windows 2003 Server, and during high traffic periods, the connection between IIS and Tomcat is being severed. This has been occurring in the mid-afternoon every day since this server went live and to get it running again, I've had to restart Tomcat. The peak connections to IIS during one of the days was just under 100, so this isn't really all that high of traffic. The isapi_redirect.log isn't much help. It has only been logging errors after the connection to Tomcat has been broken. Tomcat itself hasn't logged anything of relevance and I don't actually think that Tomcat is affected when the link is broken, but as I mentioned earlier, I have to restart Tomcat to get it to reconnect with IIS. Nevertheless at least i would be curious to know, what the exact contents of the isapi redirector log files are. We don't need all of it, but the first 100 lines or so could be interesting. What's you log level? Have a look at the other thread running on this list the last few days (ISAPI JK2 ran better than JK, how can that be?). To get a slightly better idea what's happening when, you could also introduce a load balancer and a status worker. Most likely it would help to take Java Thread Dumps and look inside the JVM, what your request threads are doing, when IIS gets stuck. Regards, Rainer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat and IIS losing their link
Travis Haagen schrieb: Thanks for the responses thus far. I've been trying to get more information about the problem, but haven't been very successfull. I can't seem to create thread dumps, because I have to use Terminal Services. Is there a way to get a thread dump even though I'm using Terminal Services? There are a couple of not so nice ways of doing that. One example: - Install a JDK not only a JRE - enable JMX for your java process (adding the following flags to your service definition): -Dcom.sun.management.jmxremote.port=1090 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false (Note: no authentication here) - search for the jar file FullThreadDump.jar, something like JDK_HOME\demo\management\FullThreadDump\FullThreadDump.jar - run java -cp PathToMyJar\FullThreadDump.jar FullThreadDump localhost:1090 C:\temp\ThreadDumps.log Tomcat keeps locking up every afternoon, during the work week, so its definitely related to server traffic. Sometimes it locks up twice in an afternoon. I found this article about IIS 6.0 application health monitoring and I was wondering if anyone know if the items listed in the section 'ISAPI Extensions Can Declare Themselves Unhealthy' are implemented in the ISAPI redirector? http://help.alentus.com/article.aspx?id=10958cNode=7F8F8Y Interesting, but not used at the moment in the isapi redirector. Here's what the ISAPI redirector logs say (the following errors are simply repeated until Tomcat is restarted). My log level is currently set to 'WARN': Those unfortunately don't give us more than what we already expect: that the problem lies inside your Tomcat instance. Try the above recipe for thread dumps. Regards, Rainer - Original Message - From: Rainer Jung [EMAIL PROTECTED] To: Tomcat Users List users@tomcat.apache.org Sent: Friday, December 14, 2007 9:20 AM Subject: Re: Tomcat and IIS losing their link Travis Haagen wrote: Hello, I've got IIS 6.0 linked with Tomcat 5.5.25 (with JDK 1.5.0_14-b03 and AJP dll) via the ISAPI connector (JK 1.2.25) on Windows 2003 Server, and during high traffic periods, the connection between IIS and Tomcat is being severed. This has been occurring in the mid-afternoon every day since this server went live and to get it running again, I've had to restart Tomcat. The peak connections to IIS during one of the days was just under 100, so this isn't really all that high of traffic. The isapi_redirect.log isn't much help. It has only been logging errors after the connection to Tomcat has been broken. Tomcat itself hasn't logged anything of relevance and I don't actually think that Tomcat is affected when the link is broken, but as I mentioned earlier, I have to restart Tomcat to get it to reconnect with IIS. Nevertheless at least i would be curious to know, what the exact contents of the isapi redirector log files are. We don't need all of it, but the first 100 lines or so could be interesting. What's you log level? Have a look at the other thread running on this list the last few days (ISAPI JK2 ran better than JK, how can that be?). To get a slightly better idea what's happening when, you could also introduce a load balancer and a status worker. Most likely it would help to take Java Thread Dumps and look inside the JVM, what your request threads are doing, when IIS gets stuck. Regards, Rainer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat and IIS losing their link
Thanks for the reply Per... Tomcat and IIS are running on the same machine, with no external load balancer or firewall between IIS and Tomcat. When the connection is lost, IIS displays either service is unavailable or a 503 error message. I only have maxThreads=400, because I didn't think it would hurt anything. I was running with far fewer maxThreads and had the same problem. The machine is a quad-core CPU with 4 gigs of ram and the load is very light. I'll look into your other suggestions. Also, I noticed in my original post that the redirectPort was wrong and I fixed that :) Any other ideas out there? -Travis - Original Message - From: Per Johnsson [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Thursday, December 13, 2007 11:52 PM Subject: FW: Tomcat and IIS losing their link (Tomcat maillist bounced my mail so I send it to you directly./per) Hi! Some questions: - Is the tomcat and IIS running on the same machine? - What's the result when the connection is lost? Service unavaible from IIS? - Is there any loadbalancer, firewall between the IIS and tomcat? - You have maxThreads=400, are there 400 concurrant requests ever during high load? We had some similar issues and we didn't find any good solution for it but a lot of small adjustments made it better like: - Lower the maxthreads (we had very cpu consuming requests at some times) we went from like 300 to 20 I beleive. - Installing java 1.6 which is much more efficient. - Using the latest APR and Isapi dll. /Per Jonsson -Original Message- From: Travis Haagen [mailto:[EMAIL PROTECTED] Sent: den 14 december 2007 05:08 To: Tomcat Posts Subject: Tomcat and IIS losing their link Hello, I've got IIS 6.0 linked with Tomcat 5.5.25 (with JDK 1.5.0_14-b03 and AJP dll) via the ISAPI connector (JK 1.2.25) on Windows 2003 Server, and during high traffic periods, the connection between IIS and Tomcat is being severed. This has been occurring in the mid-afternoon every day since this server went live and to get it running again, I've had to restart Tomcat. The peak connections to IIS during one of the days was just under 100, so this isn't really all that high of traffic. The isapi_redirect.log isn't much help. It has only been logging errors after the connection to Tomcat has been broken. Tomcat itself hasn't logged anything of relevance and I don't actually think that Tomcat is affected when the link is broken, but as I mentioned earlier, I have to restart Tomcat to get it to reconnect with IIS. Most of my configuration settings came from this website: http://wiki.jboss.org/wiki/Wiki.jsp?page=OptimalMod_jk1.2Configuration Here are the relevant workers.properties settings: worker.ajp13w.type=ajp13 worker.ajp13w.host=localhost worker.ajp13w.port=8009 worker.ajp13w.lbfactor=1 worker.ajp13w.connect_timeout=1 worker.ajp13w.prepost_timeout=1 worker.ajp13w.socket_timeout=10 worker.ajp13w.connection_pool_timeout=600 worker.ajp13w.connection_pool_size=350 Here are the relevant server.xml settings: Connector port=8443 maxHttpHeaderSize=8192 maxThreads=400 minSpareThreads=50 maxSpareThreads=150 enableLookups=false disableUploadTimeout=true acceptCount=100 scheme=https secure=true SSLEngine=on SSLPassword=removed SSLCertificateFile=removed SSLCertificateKeyFile=removed / Connector port=8009 redirectPort=8444 protocol=AJP/1.3 connectionTimeout=60 enableLookups=false maxThreads=400 minSpareThreads=50 maxSpareThreads=150 / I just configured IIS 6.0 to run in IIS 5.0 isolation mode and that's the end of my ideas for fixing this issue. The only other ISAPI filters that I'm running are ASP.NET_1.1.4322.2307 and ISAPI_Rewrite3 (for URL rewriting). IIS web site connections is set to 'unlimited', connection timeout is 120 seconds, HTTP Keep-Alives is enabled. Until the crazy thing stops working, everything is fine. Any help will be greatly appreciated. Thanks, Travis - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat and IIS losing their link
Travis Haagen wrote: Hello, I've got IIS 6.0 linked with Tomcat 5.5.25 (with JDK 1.5.0_14-b03 and AJP dll) via the ISAPI connector (JK 1.2.25) on Windows 2003 Server, and during high traffic periods, the connection between IIS and Tomcat is being severed. This has been occurring in the mid-afternoon every day since this server went live and to get it running again, I've had to restart Tomcat. The peak connections to IIS during one of the days was just under 100, so this isn't really all that high of traffic. The isapi_redirect.log isn't much help. It has only been logging errors after the connection to Tomcat has been broken. Tomcat itself hasn't logged anything of relevance and I don't actually think that Tomcat is affected when the link is broken, but as I mentioned earlier, I have to restart Tomcat to get it to reconnect with IIS. Nevertheless at least i would be curious to know, what the exact contents of the isapi redirector log files are. We don't need all of it, but the first 100 lines or so could be interesting. What's you log level? Have a look at the other thread running on this list the last few days (ISAPI JK2 ran better than JK, how can that be?). To get a slightly better idea what's happening when, you could also introduce a load balancer and a status worker. Most likely it would help to take Java Thread Dumps and look inside the JVM, what your request threads are doing, when IIS gets stuck. Regards, Rainer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat and IIS losing their link
Travis take a look at this excerpt from Tomcat committer Mladen Turk.. Just like Apache Web server for Windows, Microsoft IIS maintains a separate child process and thread pool for serving concurrent client connections. For non server products like Windows 2000 Professional or Windows XP the number of concurrent connections is limited to 10. This mean that you can not use workstation products for production servers unless the 10 connections limit will fulfil your needs. The server range of products does not impose that 10 connection limit, but just like Apache, the 2000 connections is a limit when the thread context switching will take its share and slow down the effective number of concurrent connections. If you need higher load you will need to deploy additional web servers and use Windows Network Load Balancer (WNLB) in front of Tomcat servers. http://people.apache.org/~mturk/docs/article/ftwai.html this means that Windows 2000 Pro or Windows XP has an artificial limit of 10..2 solutions another (more capable) OS existing OS with WindowsNetworkLoadBalancer Nota Bene: Any diagnosis is incomplete without looking at Java Thread Dumps as Rainier suggested earlier.. Martin- - Original Message - From: Rainer Jung [EMAIL PROTECTED] To: Tomcat Users List users@tomcat.apache.org Sent: Friday, December 14, 2007 12:20 PM Subject: Re: Tomcat and IIS losing their link Travis Haagen wrote: Hello, I've got IIS 6.0 linked with Tomcat 5.5.25 (with JDK 1.5.0_14-b03 and AJP dll) via the ISAPI connector (JK 1.2.25) on Windows 2003 Server, and during high traffic periods, the connection between IIS and Tomcat is being severed. This has been occurring in the mid-afternoon every day since this server went live and to get it running again, I've had to restart Tomcat. The peak connections to IIS during one of the days was just under 100, so this isn't really all that high of traffic. The isapi_redirect.log isn't much help. It has only been logging errors after the connection to Tomcat has been broken. Tomcat itself hasn't logged anything of relevance and I don't actually think that Tomcat is affected when the link is broken, but as I mentioned earlier, I have to restart Tomcat to get it to reconnect with IIS. Nevertheless at least i would be curious to know, what the exact contents of the isapi redirector log files are. We don't need all of it, but the first 100 lines or so could be interesting. What's you log level? Have a look at the other thread running on this list the last few days (ISAPI JK2 ran better than JK, how can that be?). To get a slightly better idea what's happening when, you could also introduce a load balancer and a status worker. Most likely it would help to take Java Thread Dumps and look inside the JVM, what your request threads are doing, when IIS gets stuck. Regards, Rainer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Tomcat and IIS losing their link
Hello, I've got IIS 6.0 linked with Tomcat 5.5.25 (with JDK 1.5.0_14-b03 and AJP dll) via the ISAPI connector (JK 1.2.25) on Windows 2003 Server, and during high traffic periods, the connection between IIS and Tomcat is being severed. This has been occurring in the mid-afternoon every day since this server went live and to get it running again, I've had to restart Tomcat. The peak connections to IIS during one of the days was just under 100, so this isn't really all that high of traffic. The isapi_redirect.log isn't much help. It has only been logging errors after the connection to Tomcat has been broken. Tomcat itself hasn't logged anything of relevance and I don't actually think that Tomcat is affected when the link is broken, but as I mentioned earlier, I have to restart Tomcat to get it to reconnect with IIS. Most of my configuration settings came from this website: http://wiki.jboss.org/wiki/Wiki.jsp?page=OptimalMod_jk1.2Configuration Here are the relevant workers.properties settings: worker.ajp13w.type=ajp13 worker.ajp13w.host=localhost worker.ajp13w.port=8009 worker.ajp13w.lbfactor=1 worker.ajp13w.connect_timeout=1 worker.ajp13w.prepost_timeout=1 worker.ajp13w.socket_timeout=10 worker.ajp13w.connection_pool_timeout=600 worker.ajp13w.connection_pool_size=350 Here are the relevant server.xml settings: Connector port=8443 maxHttpHeaderSize=8192 maxThreads=400 minSpareThreads=50 maxSpareThreads=150 enableLookups=false disableUploadTimeout=true acceptCount=100 scheme=https secure=true SSLEngine=on SSLPassword=removed SSLCertificateFile=removed SSLCertificateKeyFile=removed / Connector port=8009 redirectPort=8444 protocol=AJP/1.3 connectionTimeout=60 enableLookups=false maxThreads=400 minSpareThreads=50 maxSpareThreads=150 / I just configured IIS 6.0 to run in IIS 5.0 isolation mode and that's the end of my ideas for fixing this issue. The only other ISAPI filters that I'm running are ASP.NET_1.1.4322.2307 and ISAPI_Rewrite3 (for URL rewriting). IIS web site connections is set to 'unlimited', connection timeout is 120 seconds, HTTP Keep-Alives is enabled. Until the crazy thing stops working, everything is fine. Any help will be greatly appreciated. Thanks, Travis
