Re: reply timeout, connect_timeout and preprost_timeout values
Hi,
Thanks for all the information so far. This communication has really made
apache-tomcat load balancing aspects clear to me.
As you suggested earlier, I am attaching my configuration. I have also added %D
in apache access log and seeing the response time. Accordingly, I will try to
increase reply_timeout. I will also use max_reply_timeout.
Btw, I have added JK_LB_FIRST_NAME, JK_LB_LAST_NAME and so on in the apache
LogFormat as follows and restarted apache.
LogFormat %h %l %u %t \%r\ %s %b %D \%{Referer}i\ \%{User-Agent}i\
\%{cookie}i\ %{JK_WORKER_NAME}n %{JK_LB_FIRST_NAME}n \
%{JK_LB_FIRST_BUSY}n %{JK_LB_FIRST_STATE}n %{JK_LB_LAST_NAME}n
%{JK_LB_LAST_BUSY}n %{JK_LB_LAST_STATE}n mod_jk_log
CustomLog logs/access_log mod_jk_log
But I don't see JK_LB_FIRST_NAME, JK_LB_LAST_NAME etc values in the apache
access log. I am trying to figure out.
This is my workers.properties. Just to keep it short I have mentioned only 2
workers.
*** Config **
# OPTIONS ( very important for jni mode )
workers.tomcat_home=/usr/local/tomcat/apache-tomcat-5.5.20
workers.java_home=/usr/local/java
# You should configure your environment slash... ps=\ on NT and / on UNIX
ps=/
worker.list=loadbalancer,status
worker.status.type=status
#-WORKER DEFINITIONS --
# PRE117_5
worker.pre117_5.port=12009
worker.pre117_5.host=10.0.3.117
worker.pre117_5.type=ajp13
worker.pre117_5.lbfactor=100
worker.pre117_5.ping_mode=A
worker.pre117_5.reply_timeout=6
worker.pre117_5.socket_connect_timeout=5000
worker.pre117_5.recovery_options=3
# PRE117_6
worker.pre117_6.port=13009
worker.pre117_6.host=10.0.3.117
worker.pre117_6.type=ajp13
worker.pre117_6.lbfactor=100
worker.pre117_6.ping_mode=A
worker.pre117_6.reply_timeout=6
worker.pre117_6.socket_connect_timeout=5000
worker.pre117_6.recovery_options=3
# LOADBALANCER
worker.loadbalancer.sticky_session=1
worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=pre117_5,pre117_6
*
Thanks,
Madhuri
--- On Tue, 7/14/09, Rainer Jung rainer.j...@kippdata.de wrote:
From: Rainer Jung rainer.j...@kippdata.de
Subject: Re: reply timeout, connect_timeout and preprost_timeout values
To: Tomcat Users List users@tomcat.apache.org
Date: Tuesday, July 14, 2009, 3:28 PM
On 14.07.2009 10:41, Madhuri
Patwardhan wrote:
As I was reading the timeout howto once again, one
question came to
my mind. What is the difference between socket_timeout
and
socket_connect_timeout? socket_timeout is used for TCP
send/receive
timeouts. Does socket_connect_timeout indicate failure
to establish
socket connection?
The socket connect timeout is only used during
establishment of a new
connection to Tomcat. If there were no timeout, you could
run into long
tcp retransmits in some network problem situations.
After a connection is established, you can use a general
socket timeout.
Some platforms do not support them though (e.g. Solaris)
and mod_jk
silently drops that timeout if it is not supported. I'm not
a fan of a
general socket timeout , because once such a timeout fires,
the
connection cannot be used any more. I do like more the
various ping
timeouts and if applicable to the app a reply timeout.
One question regardig recovery. Every maintainence
period it tries to
recover worker in ERR state. I see the state being
changed to
ERR/REC. What does it exactly do during recovery?
REC means: this worker is now allowed for future requests.
The first
request, which could be balanced to it, will go there (and
set the
status to PRB for probing). During the execution of this
request the
worker will be not available for further request, until
that ne either
returns successfully (then the worker goes into OK), or it
returns with
an error, so the worker goes back to ERR.
Are the real new
requests sent to the worker in ERR/REC state?
Yes, at the moment we can only probe with real requests.
At present, we have
recorvey option, reply_timeout set to default.
ping_mode is not set.
If it does try a real new request with a worker during
ERR/REC state
and if request doesn't succeed then the request is
sent to a good
worker?
It depends on where the request fails. If it fails early:
yes. If it is
a huge POST request and the data has already been sent to
the worker and
it fails during response generation, we will no longer have
the huge
request body at hand to send to some other worker.
But in moist situations the failure will be detectable
either during TCP
connect or the following ping test. So both are very
important to activate!
We have mod_jk load balancer running in production. We
are using load
balancer with 19 mod_jk workers. We noticed that even
though 10 out
of 19 tomcat workers are in good
Re: reply timeout, connect_timeout and preprost_timeout values
On 14.07.2009 04:34, Madhuri Patwardhan wrote: Did you read the docs page about timeouts: http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html If you are very concerned about timeouts, use version 1.2.28, because it has an additional socket_connect_timeout. connect_timeout and preprost_timeout what would be the typical values? something like 5000 or less/more? I wouldn't choose less. The ping timeout usually shouldn't fire. By going to extremes, very small problems might lead to an unacceptable big reaction (taking a worker out of balancing, only because a ping took a second or so). Note, that in 1.2.28 there is an alternative unified way of setting the ping_timeout. I also wonder if I specify prepost_timeout then it would mean now each request would take that much additional time?Before serving each request cping/cpong would be tried with timeout value of prepost_timeout so that adds little delay to serving the request?? Yes, before each request it adds a little latency, but usually only very few milliseconds, because the ping processing is very simple. It adds roughly double the network latency between your Apache and Tomcat, which for a normal LAN will be a single digit millisecond number. I usually rate the improved stability higher than this additional resource use and latency. Also, a typical value for recovery_options would be 3. ... or 7. Concerning reply timeout: Also set max_reply_timeouts. Don't push reply_timeouts to low. In 1.2.28 you can set a general reply_timeout and set more specific ones for individual URL patterns, e.g. if you have some URLs that you expect to respond very sow (report generation etc.). This is done using extensions. See http://tomcat.apache.org/connectors-doc/reference/uriworkermap.html#Rule%20extensions Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: reply timeout, connect_timeout and preprost_timeout values
Thanks for the prompt reply. Yes. I did read the following howto document on timeout and also the mail thread http://www.mailinglistarchive.com/users@tomcat.apache.org/msg28345.html After reading the timeout howto once again, many things are becoming clear. Thanks for such a good document. As I was reading the timeout howto once again, one question came to my mind. What is the difference between socket_timeout and socket_connect_timeout? socket_timeout is used for TCP send/receive timeouts. Does socket_connect_timeout indicate failure to establish socket connection? One question regardig recovery. Every maintainence period it tries to recover worker in ERR state. I see the state being changed to ERR/REC. What does it exactly do during recovery? Are the real new requests sent to the worker in ERR/REC state? At present, we have recorvey option, reply_timeout set to default. ping_mode is not set. If it does try a real new request with a worker during ERR/REC state and if request doesn't succeed then the request is sent to a good worker? We have mod_jk load balancer running in production. We are using load balancer with 19 mod_jk workers. We noticed that even though 10 out of 19 tomcat workers are in good state, some times web site is unresponsive for few minutes. Thats why, I was wondering if somehow the requests are still being sent to 9 workers which are in bad state. Perhaps, this is happening during the transition when a worker goes bad but it takes time for the load balancer to detect that. This was setup by somebody else and prepost_timeout, ping_mode, reply_timeout and recovery_options are not set. I am hoping that after setting these timeout values we will not see these issue. I will be setting these timeout values in production system, so are there anythings I should be careful about? Thanks a lot. Madhuri --- On Tue, 7/14/09, Rainer Jung rainer.j...@kippdata.de wrote: From: Rainer Jung rainer.j...@kippdata.de Subject: Re: reply timeout, connect_timeout and preprost_timeout values To: Tomcat Users List users@tomcat.apache.org Date: Tuesday, July 14, 2009, 2:14 AM On 14.07.2009 04:34, Madhuri Patwardhan wrote: Did you read the docs page about timeouts: http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html If you are very concerned about timeouts, use version 1.2.28, because it has an additional socket_connect_timeout. connect_timeout and preprost_timeout what would be the typical values? something like 5000 or less/more? I wouldn't choose less. The ping timeout usually shouldn't fire. By going to extremes, very small problems might lead to an unacceptable big reaction (taking a worker out of balancing, only because a ping took a second or so). Note, that in 1.2.28 there is an alternative unified way of setting the ping_timeout. I also wonder if I specify prepost_timeout then it would mean now each request would take that much additional time?Before serving each request cping/cpong would be tried with timeout value of prepost_timeout so that adds little delay to serving the request?? Yes, before each request it adds a little latency, but usually only very few milliseconds, because the ping processing is very simple. It adds roughly double the network latency between your Apache and Tomcat, which for a normal LAN will be a single digit millisecond number. I usually rate the improved stability higher than this additional resource use and latency. Also, a typical value for recovery_options would be 3. ... or 7. Concerning reply timeout: Also set max_reply_timeouts. Don't push reply_timeouts to low. In 1.2.28 you can set a general reply_timeout and set more specific ones for individual URL patterns, e.g. if you have some URLs that you expect to respond very sow (report generation etc.). This is done using extensions. See http://tomcat.apache.org/connectors-doc/reference/uriworkermap.html#Rule%20extensions Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: reply timeout, connect_timeout and preprost_timeout values
On 14.07.2009 10:41, Madhuri Patwardhan wrote: As I was reading the timeout howto once again, one question came to my mind. What is the difference between socket_timeout and socket_connect_timeout? socket_timeout is used for TCP send/receive timeouts. Does socket_connect_timeout indicate failure to establish socket connection? The socket connect timeout is only used during establishment of a new connection to Tomcat. If there were no timeout, you could run into long tcp retransmits in some network problem situations. After a connection is established, you can use a general socket timeout. Some platforms do not support them though (e.g. Solaris) and mod_jk silently drops that timeout if it is not supported. I'm not a fan of a general socket timeout , because once such a timeout fires, the connection cannot be used any more. I do like more the various ping timeouts and if applicable to the app a reply timeout. One question regardig recovery. Every maintainence period it tries to recover worker in ERR state. I see the state being changed to ERR/REC. What does it exactly do during recovery? REC means: this worker is now allowed for future requests. The first request, which could be balanced to it, will go there (and set the status to PRB for probing). During the execution of this request the worker will be not available for further request, until that ne either returns successfully (then the worker goes into OK), or it returns with an error, so the worker goes back to ERR. Are the real new requests sent to the worker in ERR/REC state? Yes, at the moment we can only probe with real requests. At present, we have recorvey option, reply_timeout set to default. ping_mode is not set. If it does try a real new request with a worker during ERR/REC state and if request doesn't succeed then the request is sent to a good worker? It depends on where the request fails. If it fails early: yes. If it is a huge POST request and the data has already been sent to the worker and it fails during response generation, we will no longer have the huge request body at hand to send to some other worker. But in moist situations the failure will be detectable either during TCP connect or the following ping test. So both are very important to activate! We have mod_jk load balancer running in production. We are using load balancer with 19 mod_jk workers. We noticed that even though 10 out of 19 tomcat workers are in good state, some times web site is unresponsive for few minutes. Thats why, I was wondering if somehow the requests are still being sent to 9 workers which are in bad state.Perhaps, this is happening during the transition when a worker goes bad but it takes time for the load balancer to detect that. This was setup by somebody else and prepost_timeout, ping_mode, reply_timeout and recovery_options are not set. I am hoping that after setting these timeout values we will not see these issue. I will be setting these timeout values in production system, so are there anythings I should be careful about? You can post your complete new config for check. Most of the timeouts are off by default in order to make newer jk versions behave compatible with older ones, even if that behaviour wasn't optimal. So activate the timeouts (and use version 1.2.28). Finally add %D to your Apache LogFormat. It's the response time in microseconds and lets you fid out, when and for what requests things start to go wrong. Also add a couple of the log notes mentioned in http://tomcat.apache.org/connectors-doc/reference/apache.html (like e.g. JK_LB_FIRST_NAME, JK_LB_LAST_NAME, JK_LB_LAST_ERRORS, JK_LB_LAST_BUSY, JK_LB_LAST_ACCESSED). Regards, Rainer --- On Tue, 7/14/09, Rainer Jung rainer.j...@kippdata.de wrote: From: Rainer Jung rainer.j...@kippdata.de Subject: Re: reply timeout, connect_timeout and preprost_timeout values To: Tomcat Users List users@tomcat.apache.org Date: Tuesday, July 14, 2009, 2:14 AM On 14.07.2009 04:34, Madhuri Patwardhan wrote: Did you read the docs page about timeouts: http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html If you are very concerned about timeouts, use version 1.2.28, because it has an additional socket_connect_timeout. connect_timeout and preprost_timeout what would be the typical values? something like 5000 or less/more? I wouldn't choose less. The ping timeout usually shouldn't fire. By going to extremes, very small problems might lead to an unacceptable big reaction (taking a worker out of balancing, only because a ping took a second or so). Note, that in 1.2.28 there is an alternative unified way of setting the ping_timeout. I also wonder if I specify prepost_timeout then it would mean now each request would take that much additional time?Before serving each request cping/cpong would be tried with timeout value of prepost_timeout so that adds little delay to serving the request?? Yes, before each request
Re: reply timeout, connect_timeout and preprost_timeout values
Thanks a lot. This is very useful information. I will soon post my complete config. Thanks, Madhuri --- On Tue, 7/14/09, Rainer Jung rainer.j...@kippdata.de wrote: From: Rainer Jung rainer.j...@kippdata.de Subject: Re: reply timeout, connect_timeout and preprost_timeout values To: Tomcat Users List users@tomcat.apache.org Date: Tuesday, July 14, 2009, 3:28 PM On 14.07.2009 10:41, Madhuri Patwardhan wrote: As I was reading the timeout howto once again, one question came to my mind. What is the difference between socket_timeout and socket_connect_timeout? socket_timeout is used for TCP send/receive timeouts. Does socket_connect_timeout indicate failure to establish socket connection? The socket connect timeout is only used during establishment of a new connection to Tomcat. If there were no timeout, you could run into long tcp retransmits in some network problem situations. After a connection is established, you can use a general socket timeout. Some platforms do not support them though (e.g. Solaris) and mod_jk silently drops that timeout if it is not supported. I'm not a fan of a general socket timeout , because once such a timeout fires, the connection cannot be used any more. I do like more the various ping timeouts and if applicable to the app a reply timeout. One question regardig recovery. Every maintainence period it tries to recover worker in ERR state. I see the state being changed to ERR/REC. What does it exactly do during recovery? REC means: this worker is now allowed for future requests. The first request, which could be balanced to it, will go there (and set the status to PRB for probing). During the execution of this request the worker will be not available for further request, until that ne either returns successfully (then the worker goes into OK), or it returns with an error, so the worker goes back to ERR. Are the real new requests sent to the worker in ERR/REC state? Yes, at the moment we can only probe with real requests. At present, we have recorvey option, reply_timeout set to default. ping_mode is not set. If it does try a real new request with a worker during ERR/REC state and if request doesn't succeed then the request is sent to a good worker? It depends on where the request fails. If it fails early: yes. If it is a huge POST request and the data has already been sent to the worker and it fails during response generation, we will no longer have the huge request body at hand to send to some other worker. But in moist situations the failure will be detectable either during TCP connect or the following ping test. So both are very important to activate! We have mod_jk load balancer running in production. We are using load balancer with 19 mod_jk workers. We noticed that even though 10 out of 19 tomcat workers are in good state, some times web site is unresponsive for few minutes. Thats why, I was wondering if somehow the requests are still being sent to 9 workers which are in bad state.Perhaps, this is happening during the transition when a worker goes bad but it takes time for the load balancer to detect that. This was setup by somebody else and prepost_timeout, ping_mode, reply_timeout and recovery_options are not set. I am hoping that after setting these timeout values we will not see these issue. I will be setting these timeout values in production system, so are there anythings I should be careful about? You can post your complete new config for check. Most of the timeouts are off by default in order to make newer jk versions behave compatible with older ones, even if that behaviour wasn't optimal. So activate the timeouts (and use version 1.2.28). Finally add %D to your Apache LogFormat. It's the response time in microseconds and lets you fid out, when and for what requests things start to go wrong. Also add a couple of the log notes mentioned in http://tomcat.apache.org/connectors-doc/reference/apache.html (like e.g. JK_LB_FIRST_NAME, JK_LB_LAST_NAME, JK_LB_LAST_ERRORS, JK_LB_LAST_BUSY, JK_LB_LAST_ACCESSED). Regards, Rainer --- On Tue, 7/14/09, Rainer Jung rainer.j...@kippdata.de wrote: From: Rainer Jung rainer.j...@kippdata.de Subject: Re: reply timeout, connect_timeout and preprost_timeout values To: Tomcat Users List users@tomcat.apache.org Date: Tuesday, July 14, 2009, 2:14 AM On 14.07.2009 04:34, Madhuri Patwardhan wrote: Did you read the docs page about timeouts: http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html If you are very concerned about timeouts, use version 1.2.28, because it has an additional socket_connect_timeout. connect_timeout and preprost_timeout what would be the typical values? something like 5000 or less/more? I wouldn't choose less. The ping timeout usually shouldn't fire. By going to extremes, very small
Re: reply timeout, connect_timeout and preprost_timeout values
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Madhuri, On 7/13/2009 9:09 PM, Madhuri Patwardhan wrote: What should be the practical values for connect_timeout, prepost_timeout and reply_timeout in a production environment where traffic is considerable and a load balancer is used with 19 workers? That depends on your requirements. How long should a connection be allowed to sit idle before a request is made on it in your environment? 19 workers, eh? That's an odd (ha!) number of workers to have. Do you mean back-end servers? Or do you mean mod_jk workers. Or do you mean worker threads. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkpb4QcACgkQ9CaO5/Lv0PD+fACeJQJ63Yj7s2DIMZBToLmdHY3V dggAoLMuVfX5+nsJnVFQsiwzCr9zs7qd =35+S -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: reply timeout, connect_timeout and preprost_timeout values
Hi, Thanks for the prompt reply. I mean 19 tomcat instances (mod_jk workers) managed with load balancer worker. I want to configure the timeout values so that hung tomcat is detected and failover happens without users suffering. I completely understand that this would be dependent on the application. However, I just wanted to know the typical values that would be used for these. Thanks, Madhuri --- On Mon, 7/13/09, Christopher Schultz ch...@christopherschultz.net wrote: From: Christopher Schultz ch...@christopherschultz.net Subject: Re: reply timeout, connect_timeout and preprost_timeout values To: Tomcat Users List users@tomcat.apache.org Date: Monday, July 13, 2009, 9:36 PM -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Madhuri, On 7/13/2009 9:09 PM, Madhuri Patwardhan wrote: What should be the practical values for connect_timeout, prepost_timeout and reply_timeout in a production environment where traffic is considerable and a load balancer is used with 19 workers? That depends on your requirements. How long should a connection be allowed to sit idle before a request is made on it in your environment? 19 workers, eh? That's an odd (ha!) number of workers to have. Do you mean back-end servers? Or do you mean mod_jk workers. Or do you mean worker threads. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkpb4QcACgkQ9CaO5/Lv0PD+fACeJQJ63Yj7s2DIMZBToLmdHY3V dggAoLMuVfX5+nsJnVFQsiwzCr9zs7qd =35+S -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: reply timeout, connect_timeout and preprost_timeout values
Thanks for the prompt reply. What is APR? (please pardon my ignorance. I am new
to this.)
connect_timeout and preprost_timeout what would be the typical values?
something like 5000 or less/more?
I also wonder if I specify prepost_timeout then it would mean now each request
would take that much additional time? Before serving each request cping/cpong
would be tried with timeout value of prepost_timeout so that adds little delay
to serving the request??
Also, a typical value for recovery_options would be 3.
Thanks,
Madhuri
--- On Mon, 7/13/09, Martin Gainty mgai...@hotmail.com wrote:
From: Martin Gainty mgai...@hotmail.com
Subject: RE: reply timeout, connect_timeout and preprost_timeout values
To: madhuri_patward...@yahoo.com
Date: Monday, July 13, 2009, 10:05 PM
#yiv2065840069 .hmmessage P
{
margin:0px;padding:0px;}
#yiv2065840069 {
font-size:10pt;font-family:Verdana;}
assume No APR
worker.template.reply-timeout=2
worker.template.socket-connect-timeout=5000
worker.template.ping-mode=A
worker.template.ping-timeout=25000
http://www.pubbs.net/tomcat/200902/18677
Martin Gainty
__
Verzicht und Vertraulichkeitanmerkung
Diese Nachricht ist vertraulich. Sollten Sie nicht der
vorgesehene Empfaenger sein, so bitten wir hoeflich um eine
Mitteilung. Jede unbefugte Weiterleitung oder Fertigung
einer Kopie ist unzulaessig. Diese Nachricht dient lediglich
dem Austausch von Informationen und entfaltet keine
rechtliche Bindungswirkung. Aufgrund der leichten
Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer
den Inhalt uebernehmen.
Date: Mon, 13 Jul 2009 18:09:39 -0700
From: madhuri_patward...@yahoo.com
Subject: reply timeout, connect_timeout and
preprost_timeout values
To: users@tomcat.apache.org
Hi,
What should be the practical values for
connect_timeout, prepost_timeout and reply_timeout in a
production environment where traffic is considerable and a
load balancer is used with 19 workers?
Thanks,
Madhuri
-
To unsubscribe, e-mail:
users-unsubscr...@tomcat.apache.org
For additional commands, e-mail:
users-h...@tomcat.apache.org
Windows Live™ Hotmail®: Spread the word when
you add celeb photos to your e-mails. Check
it out.
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
