Re: why does mod_jk still try to send the request to a shutdown machine?
David Rees schrieb: So in case the remote host is dead (i.e. it's not only Tomcat not answering or no Tomcat there), we have the problem, that TCP as a reliable problem tries hard to establish a connection with several resends of SYNs in increasing intervals, leading to long waiting times. So with connect_timeout set to 500, mod_jk won't give up on the connection attempt after 500 ms have elapsed? Not in general. It depends on which part of the connect needs long. For the TCP connect it will use the socket_timeout configured. For the following Cping/Cpong the connect_timeout. Mostly I agree, but I would set a timeout for athe connection pool. Perhaps the default configuration and docs could be updated to reflect that instead of setting to zero? I normally use these settings on my servers: socket_keepalive=1 socket_timeout=300 connection_pool_timeout=300 connect_timeout=500 prepost_timeout=500 We never changed the defaults out of compatibility considerations. Most of the timeouts didn't exist from the beginning and that's why they are unfortunately disabled by default. I also normally set the worker maintain and lb worker recover_time to something lower than the default as well so that mod_jk picks up recovering workers more quickly. It would be nice if worker maintenance could be done by a process other than the processes/threads which also process requests! -Dave That on the TODO for the next major release, provisionary named JK3. We will use the APR libs as an infrastructure, so that we can easily use threads. One use case will be a management thread, that does the maintenance and concurrently monitors the backend status. No timeline for that yet. Regards, Rainer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: why does mod_jk still try to send the request to a shutdown machine?
Thanks for all the replies about this question. But I still can not know if it could be solved by tuning the attributes in worker.properties. I read that doc carefully and make some changes to the worker.proerties. There is no improvement. It seems that mod_jk try to establish a TCP connection to the master and failed by timeout (about one minute). Then it forwards the request to the salve/backup machine. The following is the content of worker.properties. # Define list of workers that will be used # for mapping requests worker.list=loadbalancer,status # Define Master worker.master.host=192.168.225.195 worker.master.port=8009 worker.master.type=ajp13 worker.master.lbfactor=1 #socket timeout in seconds worker.master.socket_timeout=5 worker.master.prepost_timeout=250 worker.master.connect_timeout=500 worker.master.recovery_options=3 # Define prefered failover node for master worker.master.redirect=slave # Define Slave worker.slave.host= 192.168.225.116 worker.slave.port=8009 worker.slave.type=ajp13 worker.slave.lbfactor=1 #socket timeout in seconds worker.slave.socket_timeout=5 worker.slave.prepost_timeout=250 worker.slave.connect_timeout=500 worker.slave.recovery_options=3 #Disable slave for all requests except failover worker.slave.activation=d # Load-balancing behaviour worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=master, slave worker.loadbalancer.sticky_session=1 # Status worker for managing load balancer worker.status.type=status -Original Message- From: Rainer Jung [mailto:[EMAIL PROTECTED] Sent: Thursday, August 23, 2007 10:24 PM To: Tomcat Users List Subject: Re: why does mod_jk still try to send the request to a shutdown machine? Have a look at http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html You could lower you prepost_timeout. The Timeouts will help for unplanned downtimes. For planned downtimes you should administratively change the activation attribute of the load balancer members. Regards, Rainer Guofeng Zhang schrieb: Hi, I use Apache as the load balancer and configured mod_jk to work in failover mode. The master and slave run on separate machines. To test how the failover works, I shut down the master machine, then I access my application. But it takes about one minute for me to see the first page. I think the time is used to try to establish the socket connection. I think I need to configure something that I missed in the worker.properties so that mod_jk could detect the status of the master before it sends the request to it. Thanks for your help very much. I use Apache HTTPD 2.2.4, mod_jk 2.2.4, JBoss 4.0.4/Tomcat 5.5. The following the log of mod_jk. [Wed Aug 22 17:24:59 2007][1632:1624] [info] mod_jk.c (2775): mod_jk/1.2.24 initialized [Wed Aug 22 17:25:42 2007][1632:6124] [info] jk_connect.c (473): connect to 192.168.225.195:8009 failed (errno=60) [Wed Aug 22 17:25:42 2007][1632:6124] [info] jk_ajp_common.c (891): Failed opening socket to (192.168.225.195:8009) (errno=60) [Wed Aug 22 17:25:42 2007][1632:6124] [info] jk_ajp_common.c (1311): (master) error connecting to the backend server (errno=60) [Wed Aug 22 17:25:42 2007][1632:6124] [info] jk_ajp_common.c (2071): (master) sending request to tomcat failed, recoverable operation attempt=1 The following is the content of worker.properties. # Define list of workers that will be used # for mapping requests worker.list=loadbalancer,status # Define Master worker.master.host=192.168.225.195 worker.master.port=8009 worker.master.type=ajp13 worker.master.lbfactor=1 worker.master.cachesize=10 worker.master.cache_timeout=600 worker.master.socket_timeout=180 worker.master.recycle_timeout=100 worker.master.prepost_timeout=50 worker.master.recovery_options=0 # Define prefered failover node for master worker.master.redirect=slave # Define Slave worker.slave.host= 192.168.225.116 worker.slave.port=8009 worker.slave.type=ajp13 worker.slave.lbfactor=1 worker.slave.cachesize=10 worker.slave.cache_timeout=600 worker.slave.socket_timeout=180 worker.slave.recycle_timeout=100 worker.slave.prepost_timeout=50 worker.slave.recovery_options=0 #Disable slave for all requests except failover worker.slave.activation=disabled # Load-balancing behaviour worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=master, slave worker.loadbalancer.sticky_session=1 # Status worker for managing load balancer worker.status.type=status - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail:
Re: why does mod_jk still try to send the request to a shutdown machine?
Guofeng Zhang wrote: Thanks for all the replies about this question. But I still can not know if it could be solved by tuning the attributes in worker.properties. I read that doc carefully and make some changes to the worker.proerties. There is no improvement. It seems that mod_jk try to establish a TCP connection to the master and failed by timeout (about one minute). Then it forwards the request to the salve/backup machine. That is correct. The reason for such behavior is because one cannot know if the failed backend is permanently, temporarily disabled or just busy. So it needs to figure out that by sending a request that can take up to 1 second if the node is dead. Later you can setup recover_time for the worker (defaults to 60 seconds) if you need lower recovery frequency. For example worker.xxx.recover_time=600 will try to reconnect to failed worker each 10 minutes. If you know that your backend will be down for designed amout of time, use the status manager page for mod_jk and completely disable desired worker. When it comes online back, re-enable it again manually. In that case there will neither initial nor scheduled retries. Regards, Mladen. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: why does mod_jk still try to send the request to a shutdown machine?
Have a look at http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html You could lower you prepost_timeout. The Timeouts will help for unplanned downtimes. For planned downtimes you should administratively change the activation attribute of the load balancer members. Regards, Rainer Guofeng Zhang schrieb: Hi, I use Apache as the load balancer and configured mod_jk to work in failover mode. The master and slave run on separate machines. To test how the failover works, I shut down the master machine, then I access my application. But it takes about one minute for me to see the first page. I think the time is used to try to establish the socket connection. I think I need to configure something that I missed in the worker.properties so that mod_jk could detect the status of the master before it sends the request to it. Thanks for your help very much. I use Apache HTTPD 2.2.4, mod_jk 2.2.4, JBoss 4.0.4/Tomcat 5.5. The following the log of mod_jk. [Wed Aug 22 17:24:59 2007][1632:1624] [info] mod_jk.c (2775): mod_jk/1.2.24 initialized [Wed Aug 22 17:25:42 2007][1632:6124] [info] jk_connect.c (473): connect to 192.168.225.195:8009 failed (errno=60) [Wed Aug 22 17:25:42 2007][1632:6124] [info] jk_ajp_common.c (891): Failed opening socket to (192.168.225.195:8009) (errno=60) [Wed Aug 22 17:25:42 2007][1632:6124] [info] jk_ajp_common.c (1311): (master) error connecting to the backend server (errno=60) [Wed Aug 22 17:25:42 2007][1632:6124] [info] jk_ajp_common.c (2071): (master) sending request to tomcat failed, recoverable operation attempt=1 The following is the content of worker.properties. # Define list of workers that will be used # for mapping requests worker.list=loadbalancer,status # Define Master worker.master.host=192.168.225.195 worker.master.port=8009 worker.master.type=ajp13 worker.master.lbfactor=1 worker.master.cachesize=10 worker.master.cache_timeout=600 worker.master.socket_timeout=180 worker.master.recycle_timeout=100 worker.master.prepost_timeout=50 worker.master.recovery_options=0 # Define prefered failover node for master worker.master.redirect=slave # Define Slave worker.slave.host= 192.168.225.116 worker.slave.port=8009 worker.slave.type=ajp13 worker.slave.lbfactor=1 worker.slave.cachesize=10 worker.slave.cache_timeout=600 worker.slave.socket_timeout=180 worker.slave.recycle_timeout=100 worker.slave.prepost_timeout=50 worker.slave.recovery_options=0 #Disable slave for all requests except failover worker.slave.activation=disabled # Load-balancing behaviour worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=master, slave worker.loadbalancer.sticky_session=1 # Status worker for managing load balancer worker.status.type=status - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: why does mod_jk still try to send the request to a shutdown machine?
On 8/23/07, Rainer Jung [EMAIL PROTECTED] wrote: Guofeng Zhang schrieb: # Define Master worker.master.host=192.168.225.195 worker.master.port=8009 worker.master.type=ajp13 worker.master.lbfactor=1 worker.master.cachesize=10 worker.master.cache_timeout=600 worker.master.socket_timeout=180 worker.master.recycle_timeout=100 worker.master.prepost_timeout=50 worker.master.recovery_options=0 You could lower you prepost_timeout. The Timeouts will help for unplanned downtimes. For planned downtimes you should administratively change the activation attribute of the load balancer members. He's already using a low prepost_timeout of 50ms (IMO is way too low and should be set to 250-500ms). What he should also set is the connect_timeout setting, again, I would recommend 500ms or higher. Additionally, I see some other settings which are incorrect: cachesize is deprecated and should not be set (or left at 1 unless using a threaded Apache MPM which is unlikely unless running on Windows which the OP does not mention). cache_timeout is also deprecated and should not be set. recycle_timeout is also deprecated and should not be set. -Dave - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: why does mod_jk still try to send the request to a shutdown machine?
David Rees schrieb: On 8/23/07, Rainer Jung [EMAIL PROTECTED] wrote: Guofeng Zhang schrieb: # Define Master worker.master.host=192.168.225.195 worker.master.port=8009 worker.master.type=ajp13 worker.master.lbfactor=1 worker.master.cachesize=10 worker.master.cache_timeout=600 worker.master.socket_timeout=180 worker.master.recycle_timeout=100 worker.master.prepost_timeout=50 worker.master.recovery_options=0 You could lower you prepost_timeout. The Timeouts will help for unplanned downtimes. For planned downtimes you should administratively change the activation attribute of the load balancer members. He's already using a low prepost_timeout of 50ms (IMO is way too low and should be set to 250-500ms). Sorry, I had only little time and directly ran into our old problem, that some timeouts are seconds, and some are milliseconds. Your are right, 50 is way too small, se my new Timeouts docs page :) What he should also set is the connect_timeout setting, again, I would recommend 500ms or higher. Yes, since prepost_timeout is already that small, I would also guess, that the minute he had to wait happened during connection establishment. Unfortunately, the connect_timeout will only help, if the connection can be established fast (the TCP connection), but then one wants to check, if Tomcat can answer fast (or maybe one is connected to something other than Tomcat). So in case the remote host is dead (i.e. it's not only Tomcat not answering or no Tomcat there), we have the problem, that TCP as a reliable problem tries hard to establish a connection with several resends of SYNs in increasing intervals, leading to long waiting times. Once the timeout fires, the load balancer takes the node down, so no more waiting time. Retries will only influence few requests. Additionally, I see some other settings which are incorrect: cachesize is deprecated and should not be set (or left at 1 unless using a threaded Apache MPM which is unlikely unless running on Windows which the OP does not mention). cache_timeout is also deprecated and should not be set. recycle_timeout is also deprecated and should not be set. Mostly I agree, but I would set a timeout for athe connection pool. -Dave Regards, Rainer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: why does mod_jk still try to send the request to a shutdown machine?
He's already using a low prepost_timeout of 50ms (IMO is way too low and should be set to 250-500ms). Sorry, I had only little time and directly ran into our old problem, that some timeouts are seconds, and some are milliseconds. Your are right, 50 is way too small, se my new Timeouts docs page :) Looks good. :-) What he should also set is the connect_timeout setting, again, I would recommend 500ms or higher. Yes, since prepost_timeout is already that small, I would also guess, that the minute he had to wait happened during connection establishment. Unfortunately, the connect_timeout will only help, if the connection can be established fast (the TCP connection), but then one wants to check, if Tomcat can answer fast (or maybe one is connected to something other than Tomcat). So in case the remote host is dead (i.e. it's not only Tomcat not answering or no Tomcat there), we have the problem, that TCP as a reliable problem tries hard to establish a connection with several resends of SYNs in increasing intervals, leading to long waiting times. So with connect_timeout set to 500, mod_jk won't give up on the connection attempt after 500 ms have elapsed? Mostly I agree, but I would set a timeout for athe connection pool. Perhaps the default configuration and docs could be updated to reflect that instead of setting to zero? I normally use these settings on my servers: socket_keepalive=1 socket_timeout=300 connection_pool_timeout=300 connect_timeout=500 prepost_timeout=500 I also normally set the worker maintain and lb worker recover_time to something lower than the default as well so that mod_jk picks up recovering workers more quickly. It would be nice if worker maintenance could be done by a process other than the processes/threads which also process requests! -Dave - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
why does mod_jk still try to send the request to a shutdown machine?
Hi, I use Apache as the load balancer and configured mod_jk to work in failover mode. The master and slave run on separate machines. To test how the failover works, I shut down the master machine, then I access my application. But it takes about one minute for me to see the first page. I think the time is used to try to establish the socket connection. I think I need to configure something that I missed in the worker.properties so that mod_jk could detect the status of the master before it sends the request to it. Thanks for your help very much. I use Apache HTTPD 2.2.4, mod_jk 2.2.4, JBoss 4.0.4/Tomcat 5.5. The following the log of mod_jk. [Wed Aug 22 17:24:59 2007][1632:1624] [info] mod_jk.c (2775): mod_jk/1.2.24 initialized [Wed Aug 22 17:25:42 2007][1632:6124] [info] jk_connect.c (473): connect to 192.168.225.195:8009 failed (errno=60) [Wed Aug 22 17:25:42 2007][1632:6124] [info] jk_ajp_common.c (891): Failed opening socket to (192.168.225.195:8009) (errno=60) [Wed Aug 22 17:25:42 2007][1632:6124] [info] jk_ajp_common.c (1311): (master) error connecting to the backend server (errno=60) [Wed Aug 22 17:25:42 2007][1632:6124] [info] jk_ajp_common.c (2071): (master) sending request to tomcat failed, recoverable operation attempt=1 The following is the content of worker.properties. # Define list of workers that will be used # for mapping requests worker.list=loadbalancer,status # Define Master worker.master.host=192.168.225.195 worker.master.port=8009 worker.master.type=ajp13 worker.master.lbfactor=1 worker.master.cachesize=10 worker.master.cache_timeout=600 worker.master.socket_timeout=180 worker.master.recycle_timeout=100 worker.master.prepost_timeout=50 worker.master.recovery_options=0 # Define prefered failover node for master worker.master.redirect=slave # Define Slave worker.slave.host= 192.168.225.116 worker.slave.port=8009 worker.slave.type=ajp13 worker.slave.lbfactor=1 worker.slave.cachesize=10 worker.slave.cache_timeout=600 worker.slave.socket_timeout=180 worker.slave.recycle_timeout=100 worker.slave.prepost_timeout=50 worker.slave.recovery_options=0 #Disable slave for all requests except failover worker.slave.activation=disabled # Load-balancing behaviour worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=master, slave worker.loadbalancer.sticky_session=1 # Status worker for managing load balancer worker.status.type=status
