Re: mod_jk dealy sendng request to backend tomcat
for telnet immediateconnection refused is given telnet 10.xxx.xxx.xx 8911 Trying 10.xx.xx.x... telnet: connect to address 10.xx.x.xx: Connection refused here is the error log for the specifed time [Fri Feb 27 02:26:14.463 2015] [31713:140059770595072] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Fri Feb 27 02:27:30.458 2015] [3471:140059728635648] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:27:30.560 2015] [3471:140059728635648] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:27:30.560 2015] [3471:140059728635648] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Fri Feb 27 02:28:14.585 2015] [31713:140059550308096] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:28:14.686 2015] [31713:140059550308096] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:28:14.687 2015] [31713:140059550308096] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Fri Feb 27 02:29:14.401 2015] [31713:140059728635648] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:29:14.503 2015] [31713:140059728635648] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:29:14.503 2015] [31713:140059728635648] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Fri Feb 27 02:30:40.149 2015] [31713:140059739125504] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:30:40.250 2015] [31713:140059739125504] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:30:40.251 2015] [31713:140059739125504] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Fri Feb 27 02:31:15.442 2015] [31713:140059644716800] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:31:15.543 2015] [31713:140059644716800] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on On Mon, Mar 2, 2015 at 4:28 PM, Rainer Jung rainer.j...@kippdata.de wrote: Am 02.03.2015 um 11:34 schrieb Rajesh Cherukuri: rainer looks like what you said is correct , but not sure why the Mod_jk has to wait for 10 seconds when the backend tomcat servers is down Because your network layer behaves like that. It simply hangs for (more than) 10 seconds. You should be able to observer that yourself e.g. using telnet tomcatserverip tomcatajpport It should hang that long as well. here is the error log i don't see that any place where it is aitng for 20 sec The situation you want to discuss happened at 02:28:14, the log snippet is from 01:28:35 to 01:33:31. So they do not match. error log [Thu Feb 26 01:28:35.432 2015] [19535:140214433396480] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:28:35.534 2015] [19535:140214433396480] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:28:35.534 2015] [19535:140214433396480] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Thu Feb 26 01:30:26.182 2015] [19536:140214611724032] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:30:26.284 2015] [19536:140214611724032] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong
Re: mod_jk dealy sendng request to backend tomcat
Please don't top post, it makes following the communication very hard. Am 02.03.2015 um 12:48 schrieb Rajesh Cherukuri: for telnet immediateconnection refused is given telnet 10.xxx.xxx.xx 8911 Trying 10.xx.xx.x... telnet: connect to address 10.xx.x.xx: Connection refused Then I would expect that mod_jk gets exactly the same quick error. Maybe the network situation between your mod_jk server and the Tomcat worker was different, when the problem occurred. But ... here is the error log for the specifed time the logs looks strange to me. It could be, that mod_jk did not really run into a 10 second timeout. Error 111 on Linux is connection refused, not a timeout. Is that 20 second delay always the case? Could it be that your http server was overloaded? Are you using the latest mod_jk version? [Fri Feb 27 02:26:14.463 2015] [31713:140059770595072] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Fri Feb 27 02:27:30.458 2015] [3471:140059728635648] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:27:30.560 2015] [3471:140059728635648] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:27:30.560 2015] [3471:140059728635648] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Fri Feb 27 02:28:14.585 2015] [31713:140059550308096] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:28:14.686 2015] [31713:140059550308096] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:28:14.687 2015] [31713:140059550308096] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Fri Feb 27 02:29:14.401 2015] [31713:140059728635648] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:29:14.503 2015] [31713:140059728635648] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:29:14.503 2015] [31713:140059728635648] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Fri Feb 27 02:30:40.149 2015] [31713:140059739125504] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:30:40.250 2015] [31713:140059739125504] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:30:40.251 2015] [31713:140059739125504] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Fri Feb 27 02:31:15.442 2015] [31713:140059644716800] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:31:15.543 2015] [31713:140059644716800] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on On Mon, Mar 2, 2015 at 4:28 PM, Rainer Jung rainer.j...@kippdata.de wrote: Am 02.03.2015 um 11:34 schrieb Rajesh Cherukuri: rainer looks like what you said is correct , but not sure why the Mod_jk has to wait for 10 seconds when the backend tomcat servers is down Because your network layer behaves like that. It simply hangs for (more than) 10 seconds. You should be able to observer that yourself e.g. using telnet tomcatserverip tomcatajpport It should hang that long as well. here is the error log i don't see that any place where it is aitng for 20 sec The situation you want to discuss happened at 02:28:14, the log snippet is from 01:28:35 to 01:33:31. So they do not match. error log [Thu Feb 26 01:28:35.432 2015] [19535:140214433396480] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:28:35.534 2015] [19535:140214433396480] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111)
Re: mod_jk dealy sendng request to backend tomcat
Am 02.03.2015 um 10:45 schrieb Rajesh Cherukuri: here is what i can see in modJk log file , we have stopped tomcat 11 a week back , 172.x [27/Feb/2015:*02:28:14* +] GET /xx/deal-finder HTTP/1.1200 15172 20 *lb* tomcat--11** 0 26712 12118 **tomcat-03** 0 820679 4 *20.352845* – worker configuration *here is my work.property file* *worker.lb.common.type=ajp13worker.lb.common.lbfactor=1worker.lb.common.ping_mode=Aworker.lb.common.socket_connect_timeout=1# worker.lb.common.connection_pool_size=6worker.lb.common.connection_pool_timeout=600worker.lb.common.socket_keepalive=True* That is not a production ready config. - update to the recent version 1.2.40 (if your version is older) - build your configuration starting from the *source* download linked on http://tomcat.apache.org/download-connectors.cgi. In the source download (tarball or zip) there's a conf folder containing a workers.properties and a httpd-jk.conf file. Look at those and adjust to your environment. They contain generally useful timeouts etc. The 20 seconds you are noticing are likely 10 seconds (connect_timeout) times 2 (retries). If the server for a Tomcat worker is up and Tomcat itself is down, the server should typically reject a new connect very fast, because nothing is listening to the port. If the server is down it depends on the network situation, whether you get a fast error (e.g. no route to host etc.) or whether it takes long. If the server is up and Tomcat is up, but your Tomcat jvm doesn't respond, it again depends on the details. If you know, that a node is down, then you should set its activation to stopped to let mod_jk know, that it should no longer try to use it. You mod_jk log should show more information than what you posted. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: mod_jk dealy sendng request to backend tomcat
Am 02.03.2015 um 11:34 schrieb Rajesh Cherukuri: rainer looks like what you said is correct , but not sure why the Mod_jk has to wait for 10 seconds when the backend tomcat servers is down Because your network layer behaves like that. It simply hangs for (more than) 10 seconds. You should be able to observer that yourself e.g. using telnet tomcatserverip tomcatajpport It should hang that long as well. here is the error log i don't see that any place where it is aitng for 20 sec The situation you want to discuss happened at 02:28:14, the log snippet is from 01:28:35 to 01:33:31. So they do not match. error log [Thu Feb 26 01:28:35.432 2015] [19535:140214433396480] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:28:35.534 2015] [19535:140214433396480] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:28:35.534 2015] [19535:140214433396480] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Thu Feb 26 01:30:26.182 2015] [19536:140214611724032] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:30:26.284 2015] [19536:140214611724032] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:30:26.284 2015] [19536:140214611724032] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Thu Feb 26 01:30:32.670 2015] [19538:140214548784896] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:30:32.772 2015] [19538:140214548784896] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:30:32.772 2015] [19538:140214548784896] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Thu Feb 26 01:31:37.441 2015] [19535:140214318008064] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:31:37.543 2015] [19535:140214318008064] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:31:37.543 2015] [19535:140214318008064] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Thu Feb 26 01:32:29.323 2015] [19534:140214338987776] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:32:29.425 2015] [19534:140214338987776] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:32:29.425 2015] [19534:140214338987776] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Thu Feb 26 01:33:31.431 2015] [19536:140214569764608] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:33:31.532 2015] [19536:140214569764608] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port ( On Mon, Mar 2, 2015 at 3:47 PM, Rainer Jung rainer.j...@kippdata.de wrote: Am 02.03.2015 um 10:45 schrieb Rajesh Cherukuri: here is what i can see in modJk log file , we have stopped tomcat 11 a week back , 172.x [27/Feb/2015:*02:28:14* +] GET /xx/deal-finder HTTP/1.1200 15172 20 *lb* tomcat--11** 0 26712 12118 **tomcat-03** 0 820679 4 *20.352845* – worker configuration *here is my work.property file* *worker.lb.common.type=ajp13worker.lb.common.lbfactor=1worker.lb.common. ping_mode=Aworker.lb.common.socket_connect_timeout=1# worker.lb.common.connection_pool_size=6worker.lb.common. connection_pool_timeout=600worker.lb.common.socket_keepalive=True* That is not a production ready config. - update to the recent version 1.2.40 (if your version is older) - build your configuration starting from the *source* download linked on
Re: mod_jk dealy sendng request to backend tomcat
rainer looks like what you said is correct , but not sure why the Mod_jk has to wait for 10 seconds when the backend tomcat servers is down here is the error log i don't see that any place where it is aitng for 20 sec error log [Thu Feb 26 01:28:35.432 2015] [19535:140214433396480] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:28:35.534 2015] [19535:140214433396480] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:28:35.534 2015] [19535:140214433396480] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Thu Feb 26 01:30:26.182 2015] [19536:140214611724032] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:30:26.284 2015] [19536:140214611724032] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:30:26.284 2015] [19536:140214611724032] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Thu Feb 26 01:30:32.670 2015] [19538:140214548784896] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:30:32.772 2015] [19538:140214548784896] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:30:32.772 2015] [19538:140214548784896] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Thu Feb 26 01:31:37.441 2015] [19535:140214318008064] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:31:37.543 2015] [19535:140214318008064] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:31:37.543 2015] [19535:140214318008064] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Thu Feb 26 01:32:29.323 2015] [19534:140214338987776] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:32:29.425 2015] [19534:140214338987776] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:32:29.425 2015] [19534:140214338987776] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Thu Feb 26 01:33:31.431 2015] [19536:140214569764608] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:33:31.532 2015] [19536:140214569764608] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port ( On Mon, Mar 2, 2015 at 3:47 PM, Rainer Jung rainer.j...@kippdata.de wrote: Am 02.03.2015 um 10:45 schrieb Rajesh Cherukuri: here is what i can see in modJk log file , we have stopped tomcat 11 a week back , 172.x [27/Feb/2015:*02:28:14* +] GET /xx/deal-finder HTTP/1.1200 15172 20 *lb* tomcat--11** 0 26712 12118 **tomcat-03** 0 820679 4 *20.352845* – worker configuration *here is my work.property file* *worker.lb.common.type=ajp13worker.lb.common.lbfactor=1worker.lb.common. ping_mode=Aworker.lb.common.socket_connect_timeout=1# worker.lb.common.connection_pool_size=6worker.lb.common. connection_pool_timeout=600worker.lb.common.socket_keepalive=True* That is not a production ready config. - update to the recent version 1.2.40 (if your version is older) - build your configuration starting from the *source* download linked on http://tomcat.apache.org/download-connectors.cgi. In the source download (tarball or zip) there's a conf folder containing a workers.properties and a httpd-jk.conf file. Look at those and adjust to your environment. They contain generally useful timeouts etc. The 20 seconds you are noticing are likely 10 seconds (connect_timeout) times 2 (retries). If the server for a Tomcat worker is up
Re: mod_jk dealy sendng request to backend tomcat
here is what i can see in modJk log file , we have stopped tomcat 11 a week back , 172.x [27/Feb/2015:*02:28:14* +] GET /xx/deal-finder HTTP/1.1200 15172 20 *lb* tomcat--11** 0 26712 12118 **tomcat-03** 0 820679 4 *20.352845* – worker configuration *here is my work.property file* *worker.lb.common.type=ajp13worker.lb.common.lbfactor=1worker.lb.common.ping_mode=Aworker.lb.common.socket_connect_timeout=1# worker.lb.common.connection_pool_size=6worker.lb.common.connection_pool_timeout=600worker.lb.common.socket_keepalive=True* On Sun, Mar 1, 2015 at 11:18 PM, Rainer Jung rainer.j...@kippdata.de wrote: Am 01.03.2015 um 17:57 schrieb Rajesh Cherukuri: hi today i have noticed that mod_jk took 20 seconds delay in sending the request to backend tomcat servers from apache , lookslike the request was assigned to a tomcat 11and it has replicate the request to the other tomcat-3 here are my findings Deal Finder request * Apache request access logs * 172.xx - - [27/Feb/2015:*02:28:14* +] GET /x/deal-finder HTTP/1.1 200 15172 - 195..xxx.xx, - - Mod_Jk apache log Mod Jk logs show that the request was assigned to tomcat live 11 but it was reassigned to tomcat live 03 since it was down so there might be delay due to this 172.x [27/Feb/2015:*02:28:14* +] GET /xx/deal-finder HTTP/1.1 200 15172 20 *lb tomcat--11* 0 26712 12118 *tomcat-03* 0 820679 4 20.352845 – *Tomcat access logs - Delay of 20 sec from mod JK * 172.xxx - - [27/Feb/2015:02:*28:34* +] 'GET /xx/deal-finder HTTP/1.1' 200 15172 TP-Processor243 D45BF91DBAF9D2CBFFBC2D6B19D3CC61.tomcat-03 0.127 Tomcat response in 0.127 millseconds quick What's in your mod_jk log file? What is your worker configuration? Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: mod_jk dealy sendng request to backend tomcat
Yes it is always 20 sec always the case when the MOD_jk assigns the request to the tomcat that was down using Mod_jk 1.2.37 version On Mon, Mar 2, 2015 at 8:27 PM, Rainer Jung rainer.j...@kippdata.de wrote: Please don't top post, it makes following the communication very hard. Am 02.03.2015 um 12:48 schrieb Rajesh Cherukuri: for telnet immediateconnection refused is given telnet 10.xxx.xxx.xx 8911 Trying 10.xx.xx.x... telnet: connect to address 10.xx.x.xx: Connection refused Then I would expect that mod_jk gets exactly the same quick error. Maybe the network situation between your mod_jk server and the Tomcat worker was different, when the problem occurred. But ... here is the error log for the specifed time the logs looks strange to me. It could be, that mod_jk did not really run into a 10 second timeout. Error 111 on Linux is connection refused, not a timeout. Is that 20 second delay always the case? Could it be that your http server was overloaded? Are you using the latest mod_jk version? [Fri Feb 27 02:26:14.463 2015] [31713:140059770595072] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Fri Feb 27 02:27:30.458 2015] [3471:140059728635648] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:27:30.560 2015] [3471:140059728635648] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:27:30.560 2015] [3471:140059728635648] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Fri Feb 27 02:28:14.585 2015] [31713:140059550308096] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:28:14.686 2015] [31713:140059550308096] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:28:14.687 2015] [31713:140059550308096] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Fri Feb 27 02:29:14.401 2015] [31713:140059728635648] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:29:14.503 2015] [31713:140059728635648] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:29:14.503 2015] [31713:140059728635648] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Fri Feb 27 02:30:40.149 2015] [31713:140059739125504] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:30:40.250 2015] [31713:140059739125504] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:30:40.251 2015] [31713:140059739125504] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Fri Feb 27 02:31:15.442 2015] [31713:140059644716800] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:31:15.543 2015] [31713:140059644716800] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on On Mon, Mar 2, 2015 at 4:28 PM, Rainer Jung rainer.j...@kippdata.de wrote: Am 02.03.2015 um 11:34 schrieb Rajesh Cherukuri: rainer looks like what you said is correct , but not sure why the Mod_jk has to wait for 10 seconds when the backend tomcat servers is down Because your network layer behaves like that. It simply hangs for (more than) 10 seconds. You should be able to observer that yourself e.g. using telnet tomcatserverip tomcatajpport It should hang that long as well. here is the error log i don't see that any place where it is aitng for 20 sec The situation you want to discuss happened at 02:28:14, the log snippet is from 01:28:35 to 01:33:31. So they do not match. error log [Thu Feb 26 01:28:35.432 2015] [19535:140214433396480] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat
Re: mod_jk dealy sendng request to backend tomcat
Please dont top post. On 03/02/2015 04:07 PM, Rajesh Cherukuri wrote: Yes it is always 20 sec always the case when the MOD_jk assigns the request to the tomcat that was down Are you sure that there's not some other tomcat in the same load balanced worker that's in hang? Didnt had time to check your log file, just a suggestion. using Mod_jk 1.2.37 version On Mon, Mar 2, 2015 at 8:27 PM, Rainer Jung rainer.j...@kippdata.de wrote: Please don't top post, it makes following the communication very hard. Am 02.03.2015 um 12:48 schrieb Rajesh Cherukuri: for telnet immediateconnection refused is given telnet 10.xxx.xxx.xx 8911 Trying 10.xx.xx.x... telnet: connect to address 10.xx.x.xx: Connection refused Then I would expect that mod_jk gets exactly the same quick error. Maybe the network situation between your mod_jk server and the Tomcat worker was different, when the problem occurred. But ... here is the error log for the specifed time the logs looks strange to me. It could be, that mod_jk did not really run into a 10 second timeout. Error 111 on Linux is connection refused, not a timeout. Is that 20 second delay always the case? Could it be that your http server was overloaded? Are you using the latest mod_jk version? [Fri Feb 27 02:26:14.463 2015] [31713:140059770595072] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Fri Feb 27 02:27:30.458 2015] [3471:140059728635648] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:27:30.560 2015] [3471:140059728635648] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:27:30.560 2015] [3471:140059728635648] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Fri Feb 27 02:28:14.585 2015] [31713:140059550308096] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:28:14.686 2015] [31713:140059550308096] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:28:14.687 2015] [31713:140059550308096] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Fri Feb 27 02:29:14.401 2015] [31713:140059728635648] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:29:14.503 2015] [31713:140059728635648] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:29:14.503 2015] [31713:140059728635648] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Fri Feb 27 02:30:40.149 2015] [31713:140059739125504] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:30:40.250 2015] [31713:140059739125504] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:30:40.251 2015] [31713:140059739125504] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Fri Feb 27 02:31:15.442 2015] [31713:140059644716800] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:31:15.543 2015] [31713:140059644716800] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on On Mon, Mar 2, 2015 at 4:28 PM, Rainer Jung rainer.j...@kippdata.de wrote: Am 02.03.2015 um 11:34 schrieb Rajesh Cherukuri: rainer looks like what you said is correct , but not sure why the Mod_jk has to wait for 10 seconds when the backend tomcat servers is down Because your network layer behaves like that. It simply hangs for (more than) 10 seconds. You should be able to observer that yourself e.g. using telnet tomcatserverip tomcatajpport It should hang that long as well. here is the error log i don't see that any place where it is aitng for 20 sec The situation you want to discuss happened at 02:28:14, the log snippet is from 01:28:35 to 01:33:31. So they do not match. error log [Thu Feb 26
Re: mod_jk dealy sendng request to backend tomcat
For an explanation of what we mean by please do not top post, see here : http://www.idallen.com/topposting.html Got it ? Rajesh Cherukuri wrote: Yes it is always 20 sec always the case when the MOD_jk assigns the request to the tomcat that was down using Mod_jk 1.2.37 version On Mon, Mar 2, 2015 at 8:27 PM, Rainer Jung rainer.j...@kippdata.de wrote: Please don't top post, it makes following the communication very hard. Am 02.03.2015 um 12:48 schrieb Rajesh Cherukuri: for telnet immediateconnection refused is given telnet 10.xxx.xxx.xx 8911 Trying 10.xx.xx.x... telnet: connect to address 10.xx.x.xx: Connection refused Then I would expect that mod_jk gets exactly the same quick error. Maybe the network situation between your mod_jk server and the Tomcat worker was different, when the problem occurred. But ... here is the error log for the specifed time the logs looks strange to me. It could be, that mod_jk did not really run into a 10 second timeout. Error 111 on Linux is connection refused, not a timeout. Is that 20 second delay always the case? Could it be that your http server was overloaded? Are you using the latest mod_jk version? [Fri Feb 27 02:26:14.463 2015] [31713:140059770595072] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Fri Feb 27 02:27:30.458 2015] [3471:140059728635648] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:27:30.560 2015] [3471:140059728635648] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:27:30.560 2015] [3471:140059728635648] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Fri Feb 27 02:28:14.585 2015] [31713:140059550308096] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:28:14.686 2015] [31713:140059550308096] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:28:14.687 2015] [31713:140059550308096] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Fri Feb 27 02:29:14.401 2015] [31713:140059728635648] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:29:14.503 2015] [31713:140059728635648] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:29:14.503 2015] [31713:140059728635648] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Fri Feb 27 02:30:40.149 2015] [31713:140059739125504] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:30:40.250 2015] [31713:140059739125504] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:30:40.251 2015] [31713:140059739125504] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Fri Feb 27 02:31:15.442 2015] [31713:140059644716800] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:31:15.543 2015] [31713:140059644716800] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on On Mon, Mar 2, 2015 at 4:28 PM, Rainer Jung rainer.j...@kippdata.de wrote: Am 02.03.2015 um 11:34 schrieb Rajesh Cherukuri: rainer looks like what you said is correct , but not sure why the Mod_jk has to wait for 10 seconds when the backend tomcat servers is down Because your network layer behaves like that. It simply hangs for (more than) 10 seconds. You should be able to observer that yourself e.g. using telnet tomcatserverip tomcatajpport It should hang that long as well. here is the error log i don't see that any place where it is aitng for 20 sec The situation you want to discuss happened at 02:28:14, the log snippet is from 01:28:35 to 01:33:31. So they do not match. error log [Thu Feb 26 01:28:35.432 2015] [19535:140214433396480] [error] ajp_send_request::jk_ajp_common.c (1630):
Re: mod_jk dealy sendng request to backend tomcat
Am 01.03.2015 um 17:57 schrieb Rajesh Cherukuri: hi today i have noticed that mod_jk took 20 seconds delay in sending the request to backend tomcat servers from apache , lookslike the request was assigned to a tomcat 11and it has replicate the request to the other tomcat-3 here are my findings Deal Finder request * Apache request access logs * 172.xx - - [27/Feb/2015:*02:28:14* +] GET /x/deal-finder HTTP/1.1 200 15172 - 195..xxx.xx, - - Mod_Jk apache log Mod Jk logs show that the request was assigned to tomcat live 11 but it was reassigned to tomcat live 03 since it was down so there might be delay due to this 172.x [27/Feb/2015:*02:28:14* +] GET /xx/deal-finder HTTP/1.1 200 15172 20 *lb tomcat--11* 0 26712 12118 *tomcat-03* 0 820679 4 20.352845 – *Tomcat access logs - Delay of 20 sec from mod JK * 172.xxx - - [27/Feb/2015:02:*28:34* +] 'GET /xx/deal-finder HTTP/1.1' 200 15172 TP-Processor243 D45BF91DBAF9D2CBFFBC2D6B19D3CC61.tomcat-03 0.127 Tomcat response in 0.127 millseconds quick What's in your mod_jk log file? What is your worker configuration? Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
