Re: Problem with mod_jk and Tomcat Native Connectors on Solaris
Another similar but not identical case: Solaris 10 on x86 (occurs on both 32/64 bit hardware) Tomcat 6.0.18 Tomcat Native 1.1.16 APR 1.3.3 Apache 2.2.11 Using mod_proxy_ajp and AjpAprProtocol All tools compiled with GCC 3.4.3 (default shipped with Solaris 10). mod_proxy_ajp times out waiting for tomcat to respond with: The timeout specified has expired: ajp_ilink_receive() can't receive header This occurs intermittently while loading a JSP consisting of: %=System.currentTimeMillis()% I've proxied the AJP port, and can see the request being issued from mod_proxy_ajp, but no response is generated. I've added an AccessLogValve, but the requests that timeout are not logged. Switching from AjpAprProtocol to AjpProtocol works around the problem, but we really could benefit from send_file support. jordiang wrote: In Solaris, I think the Tomcat Native library just loses some requests. I concur; likely polling is somehow broken on Solaris. I think we can rule out mod_jk/mod_proxy_ajp, as both are experiencing similar issues. I am available to assist anyone with investigation/debugging - please let me know how I can help. Thanks, --Scott Severtson -- View this message in context: http://www.nabble.com/Problem-with-mod_jk-and-Tomcat-Native-Connectors-on-Solaris-tp21796027p21937252.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Problem with mod_jk and Tomcat Native Connectors on Solaris
On 02.02.2009 20:01, Jorge Medina wrote: I am having problems with the mod_jk module 1.2.26 and Tomcat Native connectors running in Solaris 10. The problem occurs in both processors x86 (64-bit) and sparc (64-bit). (The problem does not occur on RedHat EL5 64-bit). On the mod_jk workers.properties file I am specifying a value for reply_timeout: worker.localworker.reply_timeout=1(10 seconds) I am anticipating that some requests would need more time, so I decided to run my test suite just increasing the value to 3. Oddly, with this value and running the same set of tests I get sporadic errors. The mod_jk log shows the following message when this occurs: [error] ajp_get_reply::jk_ajp_common.c (1730) : (localworker) Timeout with waiting reply from tomcat. Tomcat is down, stopped or nertwork problems (errno=145) I am using the following components: Apache Web Server 2.2.11 APR 1.3.3 APR-util 1.3.4 openssl-0.9.8 mod_jk1.2.26 Before debugging a problem you might consider switching to 1.2.27, because once everything works, you'll most likely not do the upgrade ;) Tomcat 6.0.16 Tomcat Native Connectors 1.1.12 (using same APR version as Apache) Please update to 1.1.16. There have been a couple of fixes in tcnative. jsvc Sun Java JDK 1.6 I also found that the problem only occurs if the Tomcat Native Connectors is enabled in Tomcat. I can't figure why increasing the reply timeout would actually produce the errors it is supposed to prevent!! I don't really understand: why do you think the timeout produces the problem? Setting the timeout means, that any reply taking longer than the timeout will be interrupted inside Apache and logged. So setting no timeout will not produce any log statements (by default the timeout is off), but I expect that you will still have long running requests. To find out about the response times and the requests with long response times, add %D to your Apache LogFormat (response times in microseconds) and activate an access log in tomcat too, with a pattern that also contains %D 8response times in milliseconds). That way you can control during your tests, what tomcat resp. Apache think the response time actually was. After verifying that Tomcat actually needs to long for some requests, you need to start to analse why (locking in application, waiting for backend or database, etc.). You can use Java thread dumps for this (kill -QUIT). Any help is appreciated. I would like to benefit from the Tomcat Native Connectors, but I can't figure what I am configuring wrong. Try 1.1.16. --- Below is my workers.properties ... --- worker.list= wlb, jkwatch, jkmanage # Properties for worker: localworker worker.localworker.type=ajp13 worker.localworker.host=localhost worker.localworker.port=8009 worker.localworker.lbfactor=1 worker.localworker.connection_pool_timeout=600 worker.localworker.socket_keepalive=True worker.localworker.socket_timeout=60 ### this parameter causes trouble if increased to 3 worker.localworker.reply_timeout=1 # # Defining a load balancer (with a single worker, the local worker) worker.wlb.type=lb worker.wlb.balance_workers=localworker worker.wlb.max_reply_timeouts=3 I would add cping/cpong. See http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html --- ... and here is the segment of server.xml --- Connector port=8009 protocol=AJP/1.3 connectionTimeout=60 enableLookups=false bufferSize=32768 maxThreads=3000/ - Jorge - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Problem with mod_jk and Tomcat Native Connectors on Solaris
Hallo Rainer! Danke für dein Hilfe. I will follow your recommendation on upgrading, and I will need to learn how to configure a more verbose log and interpret them. I found out the following: - My client code (a java Axis2 client) had a timeout (I found it was 30 sec, I was under the assumption the default was 60s). When the client timeoust I would get a message Read timed out - The above timeout was too short, the client would timeout before Apache (mod_jk) had a chance to retry the request to Tomcat. I played with both timeouts, making sure the client timeout was always larger than the mod_jk reply_timeout for at least a factor of 2. This works pefectly in RedHat, but not in Solaris 10 (I tried sparc and x86, both compiled in 64-bit) In Solaris, I think the Tomcat Native library just loses some requests. No matter how I configured the timeouts and retries, I always get some sporadic 504 Gateway timeout. Disabling the Tomcat Native library and keeping everything in mod_jk the same works fine. So, for now I just disabled the Tomcat Native Library. Thanks again for your response. -Jorge -Original Message- From: Rainer Jung [mailto:rainer.j...@kippdata.de] Sent: Friday, February 06, 2009 11:38 AM To: Tomcat Users List Subject: Re: Problem with mod_jk and Tomcat Native Connectors on Solaris On 02.02.2009 20:01, Jorge Medina wrote: I am having problems with the mod_jk module 1.2.26 and Tomcat Native connectors running in Solaris 10. The problem occurs in both processors x86 (64-bit) and sparc (64-bit). (The problem does not occur on RedHat EL5 64-bit). On the mod_jk workers.properties file I am specifying a value for reply_timeout: worker.localworker.reply_timeout=1(10 seconds) I am anticipating that some requests would need more time, so I decided to run my test suite just increasing the value to 3. Oddly, with this value and running the same set of tests I get sporadic errors. The mod_jk log shows the following message when this occurs: [error] ajp_get_reply::jk_ajp_common.c (1730) : (localworker) Timeout with waiting reply from tomcat. Tomcat is down, stopped or nertwork problems (errno=145) I am using the following components: Apache Web Server 2.2.11 APR 1.3.3 APR-util 1.3.4 openssl-0.9.8 mod_jk1.2.26 Before debugging a problem you might consider switching to 1.2.27, because once everything works, you'll most likely not do the upgrade ;) Tomcat 6.0.16 Tomcat Native Connectors 1.1.12 (using same APR version as Apache) Please update to 1.1.16. There have been a couple of fixes in tcnative. jsvc Sun Java JDK 1.6 I also found that the problem only occurs if the Tomcat Native Connectors is enabled in Tomcat. I can't figure why increasing the reply timeout would actually produce the errors it is supposed to prevent!! I don't really understand: why do you think the timeout produces the problem? Setting the timeout means, that any reply taking longer than the timeout will be interrupted inside Apache and logged. So setting no timeout will not produce any log statements (by default the timeout is off), but I expect that you will still have long running requests. To find out about the response times and the requests with long response times, add %D to your Apache LogFormat (response times in microseconds) and activate an access log in tomcat too, with a pattern that also contains %D 8response times in milliseconds). That way you can control during your tests, what tomcat resp. Apache think the response time actually was. After verifying that Tomcat actually needs to long for some requests, you need to start to analse why (locking in application, waiting for backend or database, etc.). You can use Java thread dumps for this (kill -QUIT). Any help is appreciated. I would like to benefit from the Tomcat Native Connectors, but I can't figure what I am configuring wrong. Try 1.1.16. --- Below is my workers.properties ... --- worker.list= wlb, jkwatch, jkmanage # Properties for worker: localworker worker.localworker.type=ajp13 worker.localworker.host=localhost worker.localworker.port=8009 worker.localworker.lbfactor=1 worker.localworker.connection_pool_timeout=600 worker.localworker.socket_keepalive=True worker.localworker.socket_timeout=60 ### this parameter causes trouble if increased to 3 worker.localworker.reply_timeout=1 # # Defining a load balancer (with a single worker, the local worker) worker.wlb.type=lb worker.wlb.balance_workers=localworker worker.wlb.max_reply_timeouts=3 I would add cping/cpong.
