Re: URGENT HELP NEEDED: mod_jk loadbalancing issues
Each Tomcat machine is running in 512 MB Max. (We recently increased that to ~850 MB. We are trying not to go up to far to avoid prolonged garbage collector pauses) CPU, I/O utilization under heavy loads is not excessive. 80% max usually 60% Max. thread settings are default. I was thinking to bump it by 35-50% as well as the backlog. BTW - What would be the impoact/side effects of increasing the Connectors max threads and backlog size. Thanks, Edmon _ What is your load average under these circumstance? Is each of the 10 tomcats on a seperate physical machine? What are your max thread settings in Tomcat? Are your tomcat machines CPU bound or I/O bound under full load? Alex On 7/9/06, Edmon Begoli [EMAIL PROTECTED] wrote: Hi, First of all - thanks Rainer for willing to look into this issue. Environment description: Web Server: Apache 2.0.52 mod_jk: 1.2.15 OS: RedHat ES 4.2 App. Server: Tomcat 5.5.15 Cluster topology: 4 webservers with mod_jk doing sticky loadbalancing into 10 tomcats MOD_JK Configuration: Pretty much basic mod_jk and Tomcat settings for AJP13 Description of the issue: Under very heavy loads for our standards (about thousand of concurrent users) we start seeing errors (listed at end botton of this e-mail) in the mod_jk error log. Our concern is mod_jk behavior on the web server. We closely monitor apache and Apache itself is not under heavy stress. However, shortly after we start seeing these messages in the mod_jk's error log, web server becomes unresponsive and we have to re-start it. While the web server server is unresponsive we monitor 8009 port - nothing is going on it. We monitor Tomcats and they are fine. As soon as we re-start web servers traffic resumes. Then under heavy loads things break again We are planning on increasing the number of max threads on the Tomcats' AJP connector to imrpove the throughput of the Tomcats. However, our concern is with the web server side lock ups. It seems that mod_jk cannot recover after this oversaturation. TAIL -F FROM THE MOD_JK LOG RIGHT BEFORE IT DIES __ [Fri Jul 07 09:59:43 2006] [info] ajp_service::jk_ajp_common.c (1721): Receiving from tomcat failed, recoverable operation attempt=0 [Fri Jul 07 09:59:43 2006] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1 [Fri Jul 07 09:59:44 2006] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (IP REMOVED:8009), err=-110 [Fri Jul 07 09:59:44 2006] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1721): Receiving from tomcat failed, recoverable operation attempt=0 [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1 [Fri Jul 07 09:59:44 2006] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (IP REMOVED:8009), err=-110 [Fri Jul 07 09:59:44 2006] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1721): Receiving from tomcat failed, recoverable operation attempt=0 [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1 [Fri Jul 07 09:59:44 2006] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (IP REMOVED:8009), err=-110 [Fri Jul 07 09:59:44 2006] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1721): Receiving from tomcat failed, recoverable operation attempt=0 [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1 [Fri Jul 07 09:59:45 2006] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (IP REMOVED:8009), err=-110 [Fri Jul 07 09:59:45 2006] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Fri Jul 07 09:59:45 2006] [info] ajp_service::jk_ajp_common.c (1721): Receiving from tomcat failed, recoverable operation attempt=0 [Fri Jul 07 09:59:45 2006] [info] ajp_service::jk_ajp_common.c (1749):
Re: URGENT HELP NEEDED: mod_jk loadbalancing issues
Hi, First of all - thanks Rainer for willing to look into this issue. Environment description: Web Server: Apache 2.0.52 mod_jk: 1.2.15 OS: RedHat ES 4.2 App. Server: Tomcat 5.5.15 Cluster topology: 4 webservers with mod_jk doing sticky loadbalancing into 10 tomcats MOD_JK Configuration: Pretty much basic mod_jk and Tomcat settings for AJP13 Description of the issue: Under very heavy loads for our standards (about thousand of concurrent users) we start seeing errors (listed at end botton of this e-mail) in the mod_jk error log. Our concern is mod_jk behavior on the web server. We closely monitor apache and Apache itself is not under heavy stress. However, shortly after we start seeing these messages in the mod_jk's error log, web server becomes unresponsive and we have to re-start it. While the web server server is unresponsive we monitor 8009 port - nothing is going on it. We monitor Tomcats and they are fine. As soon as we re-start web servers traffic resumes. Then under heavy loads things break again We are planning on increasing the number of max threads on the Tomcats' AJP connector to imrpove the throughput of the Tomcats. However, our concern is with the web server side lock ups. It seems that mod_jk cannot recover after this oversaturation. TAIL -F FROM THE MOD_JK LOG RIGHT BEFORE IT DIES __ [Fri Jul 07 09:59:43 2006] [info] ajp_service::jk_ajp_common.c (1721): Receiving from tomcat failed, recoverable operation attempt=0 [Fri Jul 07 09:59:43 2006] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1 [Fri Jul 07 09:59:44 2006] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (IP REMOVED:8009), err=-110 [Fri Jul 07 09:59:44 2006] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1721): Receiving from tomcat failed, recoverable operation attempt=0 [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1 [Fri Jul 07 09:59:44 2006] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (IP REMOVED:8009), err=-110 [Fri Jul 07 09:59:44 2006] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1721): Receiving from tomcat failed, recoverable operation attempt=0 [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1 [Fri Jul 07 09:59:44 2006] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (IP REMOVED:8009), err=-110 [Fri Jul 07 09:59:44 2006] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1721): Receiving from tomcat failed, recoverable operation attempt=0 [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1 [Fri Jul 07 09:59:45 2006] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (IP REMOVED:8009), err=-110 [Fri Jul 07 09:59:45 2006] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Fri Jul 07 09:59:45 2006] [info] ajp_service::jk_ajp_common.c (1721): Receiving from tomcat failed, recoverable operation attempt=0 [Fri Jul 07 09:59:45 2006] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1 [Fri Jul 07 09:59:46 2006] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (IP REMOVED:8009), err=-110 [Fri Jul 07 09:59:46 2006] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Fri Jul 07 09:59:46 2006] [info] ajp_service::jk_ajp_common.c (1721): Receiving from tomcat failed, recoverable operation attempt=0 [Fri Jul 07 09:59:46 2006] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1 [Fri Jul 07 09:59:46 2006] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems
Re: URGENT HELP NEEDED: mod_jk loadbalancing issues
What is your load average under these circumstance? Is each of the 10 tomcats on a seperate physical machine? What are your max thread settings in Tomcat? Are your tomcat machines CPU bound or I/O bound under full load? Alex On 7/9/06, Edmon Begoli [EMAIL PROTECTED] wrote: Hi, First of all - thanks Rainer for willing to look into this issue. Environment description: Web Server: Apache 2.0.52 mod_jk: 1.2.15 OS: RedHat ES 4.2 App. Server: Tomcat 5.5.15 Cluster topology: 4 webservers with mod_jk doing sticky loadbalancing into 10 tomcats MOD_JK Configuration: Pretty much basic mod_jk and Tomcat settings for AJP13 Description of the issue: Under very heavy loads for our standards (about thousand of concurrent users) we start seeing errors (listed at end botton of this e-mail) in the mod_jk error log. Our concern is mod_jk behavior on the web server. We closely monitor apache and Apache itself is not under heavy stress. However, shortly after we start seeing these messages in the mod_jk's error log, web server becomes unresponsive and we have to re-start it. While the web server server is unresponsive we monitor 8009 port - nothing is going on it. We monitor Tomcats and they are fine. As soon as we re-start web servers traffic resumes. Then under heavy loads things break again We are planning on increasing the number of max threads on the Tomcats' AJP connector to imrpove the throughput of the Tomcats. However, our concern is with the web server side lock ups. It seems that mod_jk cannot recover after this oversaturation. TAIL -F FROM THE MOD_JK LOG RIGHT BEFORE IT DIES __ [Fri Jul 07 09:59:43 2006] [info] ajp_service::jk_ajp_common.c (1721): Receiving from tomcat failed, recoverable operation attempt=0 [Fri Jul 07 09:59:43 2006] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1 [Fri Jul 07 09:59:44 2006] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (IP REMOVED:8009), err=-110 [Fri Jul 07 09:59:44 2006] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1721): Receiving from tomcat failed, recoverable operation attempt=0 [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1 [Fri Jul 07 09:59:44 2006] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (IP REMOVED:8009), err=-110 [Fri Jul 07 09:59:44 2006] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1721): Receiving from tomcat failed, recoverable operation attempt=0 [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1 [Fri Jul 07 09:59:44 2006] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (IP REMOVED:8009), err=-110 [Fri Jul 07 09:59:44 2006] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1721): Receiving from tomcat failed, recoverable operation attempt=0 [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1 [Fri Jul 07 09:59:45 2006] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (IP REMOVED:8009), err=-110 [Fri Jul 07 09:59:45 2006] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Fri Jul 07 09:59:45 2006] [info] ajp_service::jk_ajp_common.c (1721): Receiving from tomcat failed, recoverable operation attempt=0 [Fri Jul 07 09:59:45 2006] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1 [Fri Jul 07 09:59:46 2006] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (IP REMOVED:8009), err=-110 [Fri Jul 07 09:59:46 2006] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Fri Jul 07 09:59:46 2006] [info] ajp_service::jk_ajp_common.c (1721): Receiving from tomcat failed, recoverable operation attempt=0 [Fri Jul 07
URGENT HELP NEEDED: mod_jk loadbalancing issues
We are running into some serious load and scalability issues related to the mod_jk. I read some threads that indicate that these issues may have been fixed in post 1.2.15 version. Available downoalds are only for 1.2.15. Some websites are indicating that the latest one should be 1.2.16. Can someone please tell me what is the latest version of mod_jk that we should be using. We are running 1.2.15 complied on RedHat ES 4.2.x. with Apache 2.0.52 -- Thank you, Edmon
Re: URGENT HELP NEEDED: mod_jk loadbalancing issues
Hello, if you would give detailed information about your problems, we could advise you, whether it is known, and whether it has been fixed in some version. The latest release is 1.2.15, we tried to produce a 1.2.16 release, that but it has now been withdrawn because of a regression bug. Another release (1.2.17) will be tried next week. Rainer Edmon Begoli schrieb: We are running into some serious load and scalability issues related to the mod_jk. I read some threads that indicate that these issues may have been fixed in post 1.2.15 version. Available downoalds are only for 1.2.15. Some websites are indicating that the latest one should be 1.2.16. Can someone please tell me what is the latest version of mod_jk that we should be using. We are running 1.2.15 complied on RedHat ES 4.2.x. with Apache 2.0.52 - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
