Re: Need Help - Mod_jk/Apache - Mutiple Content issue
On 15.09.2009 14:50, balakarthik.baska...@wipro.com wrote: I tried with all the configuration that you defined and am still able to simulate the partial content when I make the code to sleep for a time greater than the time defined in the reply timeout. Yes, that's expected when using a reply_timeout. I understand that once the partail content is written into the client pipe,it would not be feasible to get it back or ignore it.We are working on finding the root cause of what is creating a delay in th app side of more than 3 mins of reply timeout defined in prod. Try using thread dumps. But just looking at a general case,considering a long running transaction/idle thread in the app for a 1 in 1M probability,if the case is going to occur,there will be definitely be a partial content seen.And we are afraid this would be aggravated when akamai comes into play.We did speak to akamai on this regard and they have a opinion that there would be a performance hit when any patching is going to be applied at their end. So they sacrifice correctness for performance. To bad. So,do you think of any fall back approach or a way to eliminate this situation from apache/mod_jk perspective (although the current configuration does not help) (or) do u think/aware of any cases/clients who uses mod_jk/apache where the situation would have been handled better (or) even if everyone who uses mod_jk/apache would be failing with the same issue when such a situation (app code taking a longer time and leading to partial content sent) would be faced by them? 1) You can switch off the reply_timeout But then depending on how often those long running requests happen and how long they actually take, you might slowly loose thread after thread until finally all resources are exhausted. Furthermore other components, like Apache or some other component in front might still interrupt request processing. 2) You could write a filter either in Apache or Tomcat to save alll response data until it is finished and only then send it to the client. This will be complex, error prone and will use a lot of memory ressources. 3) Fix your application Use thread dumps to find out what is happening. If you add %D to the access log, you can find out, what kind of URLs are responsible for the long running requests. You can then use a feature mod_jk allows in version 1.2.28, namely making the reply_timeout depending on the URL. Since you are already using a long timeout (e.g. 300 seconds), you might also need to globally adjust Timeout in Apache. For the URL specific reply_timeout ses http://tomcat.apache.org/connectors-doc/reference/uriworkermap.html#Rule%20extensions I would put all efforts in fixing the app. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Need Help - Mod_jk/Apache - Mutiple Content issue
On 15.09.2009 15:32, balakarthik.baska...@wipro.com wrote: Hi Rainer, The other possible configuration that I think probably we might be lacking/quite different is that we are loadbalancing the requests among 240 worker nodes each pointing to one app instance (all app instances available in the default cluster) and we use Busyness algorithm(for load balancer worker). We have seen partial content from the middle of the pages as well which was quite difficult to simulate in my local setup. Do you think load balancing among 240 workers could be a possibiltiy of messing up with request responses/adding up delay when there is a failover/other load balancing situations? (Or) is there a defined limit of worker nodes to be balanced that have been identified/defined? There is no builtin limit. Adding a lot of workers, makes things a bit slower, though in reality the time needed for the balancing should be very small compared to the time needed to actually server the requests. Using 240 Tomcat nodes will drive up your management cost (what's the state of all nodes etc.). I generally prefer the R method of balancing (by requests) but opinions vary w.r.t. the optimal method. I am not aware of issues concerning partial content from the middle of the pages. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Need Help - Mod_jk/Apache - Mutiple Content issue
Hi Rainer, I tried with all the configuration that you defined and am still able to simulate the partial content when I make the code to sleep for a time greater than the time defined in the reply timeout. I understand that once the partail content is written into the client pipe,it would not be feasible to get it back or ignore it.We are working on finding the root cause of what is creating a delay in th app side of more than 3 mins of reply timeout defined in prod. But just looking at a general case,considering a long running transaction/idle thread in the app for a 1 in 1M probability,if the case is going to occur,there will be definitely be a partial content seen.And we are afraid this would be aggravated when akamai comes into play.We did speak to akamai on this regard and they have a opinion that there would be a performance hit when any patching is going to be applied at their end. So,do you think of any fall back approach or a way to eliminate this situation from apache/mod_jk perspective (although the current configuration does not help) (or) do u think/aware of any cases/clients who uses mod_jk/apache where the situation would have been handled better (or) even if everyone who uses mod_jk/apache would be failing with the same issue when such a situation (app code taking a longer time and leading to partial content sent) would be faced by them? Thanks again for you help. Regards, Bala -Original Message- From: Rainer Jung [mailto:rainer.j...@kippdata.de] Sent: Monday, September 14, 2009 4:29 PM To: Tomcat Users List Subject: Re: Need Help - Mod_jk/Apache - Mutiple Content issue On 14.09.2009 12:02, balakarthik.baska...@wipro.com wrote: But I think you mentioned earlier that you were able to simulate the problem. I didn't simulate, I understodd from reading the code. I also said, it should go away when using recovery_options. So please: - optimize your configuration - see whether you still have problems and let us know if so The configurations will not help with Akamai eventually caching incomplete responses. If a backend doesn't send the full response, we can't fix it with configuration. Regards, Rainer -Original Message- From: Rainer Jung [mailto:rainer.j...@kippdata.de] Sent: Monday, September 14, 2009 3:11 PM To: Tomcat Users List Subject: Re: Need Help - Mod_jk/Apache - Mutiple Content issue On 14.09.2009 11:33, balakarthik.baska...@wipro.com wrote: Thanks Rainer. Just Curious,did these settings helped to overcome the multiple content problem in your local setup? I never had a multiple content problem - maybe due to good configuration. Also,thanks for your pointer on the thread id's for getting the correlations. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org Please do not print this email unless it is absolutely necessary. The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Need Help - Mod_jk/Apache - Mutiple Content issue
Hi Rainer, The other possible configuration that I think probably we might be lacking/quite different is that we are loadbalancing the requests among 240 worker nodes each pointing to one app instance (all app instances available in the default cluster) and we use Busyness algorithm(for load balancer worker). We have seen partial content from the middle of the pages as well which was quite difficult to simulate in my local setup. Do you think load balancing among 240 workers could be a possibiltiy of messing up with request responses/adding up delay when there is a failover/other load balancing situations? (Or) is there a defined limit of worker nodes to be balanced that have been identified/defined? Regards, Bala -Original Message- From: Balakarthik Baskaran (WT01 - Retail, CPG, Transportation Gover Sent: Tuesday, September 15, 2009 6:20 PM To: 'Tomcat Users List' Subject: RE: Need Help - Mod_jk/Apache - Mutiple Content issue Hi Rainer, I tried with all the configuration that you defined and am still able to simulate the partial content when I make the code to sleep for a time greater than the time defined in the reply timeout. I understand that once the partail content is written into the client pipe,it would not be feasible to get it back or ignore it.We are working on finding the root cause of what is creating a delay in th app side of more than 3 mins of reply timeout defined in prod. But just looking at a general case,considering a long running transaction/idle thread in the app for a 1 in 1M probability,if the case is going to occur,there will be definitely be a partial content seen.And we are afraid this would be aggravated when akamai comes into play.We did speak to akamai on this regard and they have a opinion that there would be a performance hit when any patching is going to be applied at their end. So,do you think of any fall back approach or a way to eliminate this situation from apache/mod_jk perspective (although the current configuration does not help) (or) do u think/aware of any cases/clients who uses mod_jk/apache where the situation would have been handled better (or) even if everyone who uses mod_jk/apache would be failing with the same issue when such a situation (app code taking a longer time and leading to partial content sent) would be faced by them? Thanks again for you help. Regards, Bala -Original Message- From: Rainer Jung [mailto:rainer.j...@kippdata.de] Sent: Monday, September 14, 2009 4:29 PM To: Tomcat Users List Subject: Re: Need Help - Mod_jk/Apache - Mutiple Content issue On 14.09.2009 12:02, balakarthik.baska...@wipro.com wrote: But I think you mentioned earlier that you were able to simulate the problem. I didn't simulate, I understodd from reading the code. I also said, it should go away when using recovery_options. So please: - optimize your configuration - see whether you still have problems and let us know if so The configurations will not help with Akamai eventually caching incomplete responses. If a backend doesn't send the full response, we can't fix it with configuration. Regards, Rainer -Original Message- From: Rainer Jung [mailto:rainer.j...@kippdata.de] Sent: Monday, September 14, 2009 3:11 PM To: Tomcat Users List Subject: Re: Need Help - Mod_jk/Apache - Mutiple Content issue On 14.09.2009 11:33, balakarthik.baska...@wipro.com wrote: Thanks Rainer. Just Curious,did these settings helped to overcome the multiple content problem in your local setup? I never had a multiple content problem - maybe due to good configuration. Also,thanks for your pointer on the thread id's for getting the correlations. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org Please do not print this email unless it is absolutely necessary. The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Need Help - Mod_jk/Apache - Mutiple Content issue
Hi Rainer, Pfa the config files in txt format.I tried with the combinatrions available in the settings marked with comments. Also,could you pl let me know the following: - Will the access log contain an entry for each request that is seen in the jk log?I can see a corresponding entry in the accesslog for an entry in the jk log in my local setup.But I coulnt interpret the same from my prod log.I hope getting this will provide some direction on what element is creating the timeout as from the prod log I can see one entry haing the a timeout in the same time perios as that of the other requests.Pl let me know how to interpret the files.I have sent you the prod log file in my prev mail.Pl let me know if you would like to resend the same or any other details.I too the all entries around 13:44:08 from both logs from prod. Regards, Bala -Original Message- From: Rainer Jung [mailto:rainer.j...@kippdata.de] Sent: Thursday, September 10, 2009 10:43 PM To: Tomcat Users List Subject: Re: Need Help - Mod_jk/Apache - Mutiple Content issue On 10.09.2009 14:46, balakarthik.baska...@wipro.com wrote: Hi Rainer, I did tried with a lot of combinations for the settings and can see that when recovery_options is set,no multiple content is seen and partial content is seen followed by an OK message when a huge chunk of data is written. However,the failover case didn't at all work out when the recovery_options is set.Pfa a doc with the scenarios that I tested and the results that I am seeing. However,in our prod envt,we currently have the reply_timeout and the recovery_options setting available and we have reomoved the socket_timeout.The recent behaviour that we are observing is,unlike the duplicate content with the header that we saw earlier with socket_timeout,we are seeing some of the content in the middle of the response being duplicated. We are making use of dynamic includes of the jsp files (with ATG dsp tags) and when I tried having the same in my local setup,I could see the response being written into the socket once the jsp completes.But not able to replicate a scenario of recreating multiple content from the middle of the page/content served by a fragment that is included. Pfa a file with the behaviour that I am noticing from the access and jk log files.For the test app,I could see only one GET request and a corresponding response in the jk log with the response time.I included one image and I can see a corresponding entry for the image as well in both files.However,there are no entries seen for the included jsp files. I am attaching the log entries from our prod envt as well which I took when I was able to see a timeout reply in the jk file.However,from the access log I couldn't interpret a corresponding entry/behaviour.Could you pl help on how the multiple content could have been occuring in the middle and your insights from the log files? (Also,pl note that we have currenlty turned off akamai in prod and the timeout is seen without akamai as well) Let's start in the right way, first fix the configuration, then do and analyze the tests, maybe resulting in more config optimization. Please post your config. I had a quick look at the files you attached, but it looks very much like we first need to improve the config, before analyzing behaviour, that will likely not appear with the right config. Also,reg your explanation about akamai,I can contact akamai to find out if they can handle the prematurely closed data differently.Do you remember any setting to the browser to interpret this invalid data based on content length/premature closure?(just for testing though wouldn't apply to real life scenario) No there is no such browser setting that I'm aware of. But akamai is not a browser, its a server that behaves more like a cache. So they need to solve a server problem, not an end user problem. They should immediately invalidate content, that was delivered over a connection that had a protocol abort. Thanks much for your help.I can give you a call if you would like to discuss this over phone. Lets first check the config, maybe then everything will be fine ;) Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org Please do not print this email unless it is absolutely necessary. The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments
RE: Need Help - Mod_jk/Apache - Mutiple Content issue
Hi Rainer, Also find attached the jk log file of the reply timeout occurrence of today in Prod envt.Please notice the thread 1258649952 which has failed and there are also some successful responses seen in the intermediate duration.I am not sure at this point if this belongs to the same request or due to a parallel thread.At this point,I am not able to enable the debug logs in prod as it might overflow the log bringing down the web instance. Regards, Bala -Original Message- From: Balakarthik Baskaran (WT01 - Retail, CPG, Transportation Gover Sent: Monday, September 14, 2009 12:52 PM To: 'Tomcat Users List' Subject: RE: Need Help - Mod_jk/Apache - Mutiple Content issue Hi Rainer, Pfa the config files in txt format.I tried with the combinatrions available in the settings marked with comments. Also,could you pl let me know the following: - Will the access log contain an entry for each request that is seen in the jk log?I can see a corresponding entry in the accesslog for an entry in the jk log in my local setup.But I coulnt interpret the same from my prod log.I hope getting this will provide some direction on what element is creating the timeout as from the prod log I can see one entry haing the a timeout in the same time perios as that of the other requests.Pl let me know how to interpret the files.I have sent you the prod log file in my prev mail.Pl let me know if you would like to resend the same or any other details.I too the all entries around 13:44:08 from both logs from prod. Regards, Bala -Original Message- From: Rainer Jung [mailto:rainer.j...@kippdata.de] Sent: Thursday, September 10, 2009 10:43 PM To: Tomcat Users List Subject: Re: Need Help - Mod_jk/Apache - Mutiple Content issue On 10.09.2009 14:46, balakarthik.baska...@wipro.com wrote: Hi Rainer, I did tried with a lot of combinations for the settings and can see that when recovery_options is set,no multiple content is seen and partial content is seen followed by an OK message when a huge chunk of data is written. However,the failover case didn't at all work out when the recovery_options is set.Pfa a doc with the scenarios that I tested and the results that I am seeing. However,in our prod envt,we currently have the reply_timeout and the recovery_options setting available and we have reomoved the socket_timeout.The recent behaviour that we are observing is,unlike the duplicate content with the header that we saw earlier with socket_timeout,we are seeing some of the content in the middle of the response being duplicated. We are making use of dynamic includes of the jsp files (with ATG dsp tags) and when I tried having the same in my local setup,I could see the response being written into the socket once the jsp completes.But not able to replicate a scenario of recreating multiple content from the middle of the page/content served by a fragment that is included. Pfa a file with the behaviour that I am noticing from the access and jk log files.For the test app,I could see only one GET request and a corresponding response in the jk log with the response time.I included one image and I can see a corresponding entry for the image as well in both files.However,there are no entries seen for the included jsp files. I am attaching the log entries from our prod envt as well which I took when I was able to see a timeout reply in the jk file.However,from the access log I couldn't interpret a corresponding entry/behaviour.Could you pl help on how the multiple content could have been occuring in the middle and your insights from the log files? (Also,pl note that we have currenlty turned off akamai in prod and the timeout is seen without akamai as well) Let's start in the right way, first fix the configuration, then do and analyze the tests, maybe resulting in more config optimization. Please post your config. I had a quick look at the files you attached, but it looks very much like we first need to improve the config, before analyzing behaviour, that will likely not appear with the right config. Also,reg your explanation about akamai,I can contact akamai to find out if they can handle the prematurely closed data differently.Do you remember any setting to the browser to interpret this invalid data based on content length/premature closure?(just for testing though wouldn't apply to real life scenario) No there is no such browser setting that I'm aware of. But akamai is not a browser, its a server that behaves more like a cache. So they need to solve a server problem, not an end user problem. They should immediately invalidate content, that was delivered over a connection that had a protocol abort. Thanks much for your help.I can give you a call if you would like to discuss this over phone. Lets first check the config, maybe then everything will be fine ;) Regards, Rainer - To unsubscribe, e-mail: users-unsubscr
RE: Need Help - Mod_jk/Apache - Mutiple Content issue
Yes Andre,you are right.We do have akamai in between the client and webserver.But we afraid that there could be a performance hit when we start patching in there.We are looking for a much intuitive solution of solving this at origin or how to handle the situation better.But,do you think all the clients who uses the apapche_mod_jk setup would suffer from this problem or what would be an optimum solution that would work fine or a configuration that would have been working fine for others that you might be aware of? Regards, Bala -Original Message- From: André Warnier [mailto:a...@ice-sa.com] Sent: Thursday, September 10, 2009 8:02 PM To: Tomcat Users List Subject: Re: Need Help - Mod_jk/Apache - Mutiple Content issue balakarthik.baska...@wipro.com wrote: Yes you are right.I think I didn't mention it clearly.I am having this while loop for writing a huge chunk of data followed by a thread that sleeps for some time and that is where I am expecting my reply_timeout to be timed out and the data written till the sleep of the thread is seen in the browser and the log file. Ok, now I do understand better, I think. There is a request. The Tomcat app starts answering (a long chunk of data), then crashes. This is simulated by your test script, which sends a long chunk of data, then sleeps (to simulate a crash). Eventually, some timeout occurs, which (may) cause another element to consider this Tomcat unresponsive, and switch the request to another one. This other one then restarts the request from the beginning. The final result being that the requesting browser first received an incomplete chunk of data from the first webapp, and now starts receiving the same data (again) from another webapp. Again intuitively, I don't know how any server-side scheme is going to solve that. When the first webapp starts answering, that Tomcat has to start sending the answer back to the client. If it then crashes, it is too late to call it back; the data is already on the wire to the client. It would be in my view the client's responsibility to ensure that it discards any incomplete data received. But I don't know of any browser for instance, which does that. They all start to try to display the data as soon as they have the beginning of it. I think that if you really need this, then you would have to insert some other item between Tomcat and the client, which acts as a proxy, and only starts sending the response to the client when it is sure that it has received and buffered a complete response from the server. But then, this item becomes the weak link. It could also crash.. Or else you need an applet at the client side, which does the same thing. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org Please do not print this email unless it is absolutely necessary. The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Need Help - Mod_jk/Apache - Mutiple Content issue
On 14.09.2009 09:22, balakarthik.baska...@wipro.com wrote:
- Will the access log contain an entry for each request that is seen in
the jk log?I can see a corresponding entry in the accesslog for an entry
in the jk log in my local setup.But I coulnt interpret the same from my
prod log.I hope getting this will provide some direction on what element
is creating the timeout as from the prod log I can see one entry haing
the a timeout in the same time perios as that of the other requests.Pl
let me know how to interpret the files.I have sent you the prod log file
in my prev mail.Pl let me know if you would like to resend the same or
any other details.I too the all entries around 13:44:08 from both logs
from prod.
Yes, all requests handled by mod_jk will be logged as usual in the
Apache access log (after finishing the request). If mod_jk needs to do
retries or a failover, you still get only one entry in the access log.
The duration logged is the full duration from beginning working on the
request until logging.
You should add a couple of interesting data to your httpd LogFormat:
%d: duration in microsenconds
%P %{tid}P: process and thread id. Those get logged in the mod_jk log
file to, so your can do correlations between access log and mod_jk log.
Maybe the Session Cookie (incoming and outgoing):
%{JSESSIONID}C %{Set-Cookie}o
and a couple of JK notes (details on
http://tomcat.apache.org/connectors-doc/reference/apache.html):
%{JK_LB_FIRST_NAME}n
%{JK_LB_FIRST_STATE}n
%{JK_LB_LAST_NAME}n
%{JK_LB_LAST_ACCESSED}n
%{JK_LB_LAST_ERRORS}n
%{JK_LB_LAST_BUSY}n
%{JK_LB_LAST_ACTIVATION}n
%{JK_LB_LAST_STATE}n
# Define list of workers that will be used
# for mapping requests
# The configuration directives are valid
# for the mod_jk version 1.2.18 and later
#
worker.list=loadbalancer,status
# Define Node1
# modify the host as your host IP or DNS name.
worker.node1.port=8109
worker.node1.host=localhost
worker.node1.type=ajp13
worker.node1.lbfactor=1
#worker.node1.prepost_timeout=1 #Not required if using ping_mode=A
#worker.node1.connect_timeout=8 #Not required if using ping_mode=A
worker.node1.ping_mode=A #As of mod_jk 1.2.27
#worker.node1.activation=a
#worker.node1.connection_pool_size=55
Let mod_jk decide on the connection_pool_size (when using Apache), do
not activate this line.
worker.node1.socket_keepalive=True
worker.node1.socket_timeout=5
I would not use a socket_timeout (personal preference).
#worker.node1.connection_pool_timeout=30
#worker.node1.recovery_options=7
#worker.node1.ping_timeout=8
#worker.node1.prepost_timeout=8
#worker.node1.max_packet_size=65536
#worker.node1.reply_timeout=5000
worker.node1.retries=2
I would add the following (also to node2):
worker.node1.recovery_options=7
worker.node1.ping_timeout=1
worker.node1.socket_connect_timeout=2000
worker.node1.connection_pool_minsize=0
worker.node1.connection_pool_timeout=600
# CAUTION: you need to set connectionTimeout in your Tomcat
# or JBoss connector to the same value
# as connection_pool_timeout. But for mod_jk it is in
# seconds, for Tomcat or JBoss in milliseconds.
# So add connectionTimeout=60 to your AJP connectors.
Personally I would also add a reply timeout, but that depends on the
application:
worker.node1.reply_timeout=30
and then in order to not make nodes fail because of few reply timeouts:
worker.loadbalancer.max_reply_timeouts=20
Finally I would add the following to the balancer,
which was introduced in 1.2.28:
worker.loadbalancer.error_escalation_time=0
# Define Node2
# modify the host as your host IP or DNS name.
worker.node2.port=8209
worker.node2.host= localhost
worker.node2.type=ajp13
worker.node2.lbfactor=1
#worker.node2.prepost_timeout=1 #Not required if using ping_mode=A
#worker.node2.connect_timeout=8 #Not required if using ping_mode=A
worker.node2.ping_mode=A #As of mod_jk 1.2.27
#worker.node2.activation=a
#worker.node2.connection_pool_size=55
worker.node2.socket_keepalive=True
worker.node2.socket_timeout=5
#worker.node2.connection_pool_timeout=30
#worker.node2.recovery_options=7
#worker.node2.ping_timeout=8
#worker.node2.prepost_timeout=8
#worker.node2.max_packet_size=65536
#worker.node2.reply_timeout=5000
worker.node2.retries=2
# Load-balancing behaviour
worker.loadbalancer.type=lb
worker.loadbalancer.method=B
worker.loadbalancer.balance_workers=node1,node2
# Status worker for managing load balancer
worker.status.type=status
#worker.node1.fail_on_status=304
#worker.node2.fail_on_status=304
#worker.node1.socket_connect_timeout=2000
#worker.node2.socket_connect_timeout=2000
You might also want to use a template like this:
worker.template.host=localhost
worker.template.type=ajp13
worker.template.lbfactor=1
worker.template.ping_mode=A #As of mod_jk 1.2.27
worker.template.socket_keepalive=True
worker.template.retries=2
...
and then remove all of the above attributes form node1 and
RE: Need Help - Mod_jk/Apache - Mutiple Content issue
Thanks Rainer.
Just Curious,did these settings helped to overcome the multiple content
problem in your local setup?
Also,thanks for your pointer on the thread id's for getting the
correlations.
Regards,
Bala
-Original Message-
From: Rainer Jung [mailto:rainer.j...@kippdata.de]
Sent: Monday, September 14, 2009 2:50 PM
To: Tomcat Users List
Subject: Re: Need Help - Mod_jk/Apache - Mutiple Content issue
On 14.09.2009 09:22, balakarthik.baska...@wipro.com wrote:
- Will the access log contain an entry for each request that is seen
in the jk log?I can see a corresponding entry in the accesslog for an
entry in the jk log in my local setup.But I coulnt interpret the same
from my prod log.I hope getting this will provide some direction on
what element is creating the timeout as from the prod log I can see
one entry haing the a timeout in the same time perios as that of the
other requests.Pl let me know how to interpret the files.I have sent
you the prod log file in my prev mail.Pl let me know if you would like
to resend the same or any other details.I too the all entries around
13:44:08 from both logs from prod.
Yes, all requests handled by mod_jk will be logged as usual in the
Apache access log (after finishing the request). If mod_jk needs to do
retries or a failover, you still get only one entry in the access log.
The duration logged is the full duration from beginning working on the
request until logging.
You should add a couple of interesting data to your httpd LogFormat:
%d: duration in microsenconds
%P %{tid}P: process and thread id. Those get logged in the mod_jk log
file to, so your can do correlations between access log and mod_jk log.
Maybe the Session Cookie (incoming and outgoing):
%{JSESSIONID}C %{Set-Cookie}o
and a couple of JK notes (details on
http://tomcat.apache.org/connectors-doc/reference/apache.html):
%{JK_LB_FIRST_NAME}n
%{JK_LB_FIRST_STATE}n
%{JK_LB_LAST_NAME}n
%{JK_LB_LAST_ACCESSED}n
%{JK_LB_LAST_ERRORS}n
%{JK_LB_LAST_BUSY}n
%{JK_LB_LAST_ACTIVATION}n
%{JK_LB_LAST_STATE}n
# Define list of workers that will be used # for mapping requests #
The configuration directives are valid # for the mod_jk version 1.2.18
and later # worker.list=loadbalancer,status
# Define Node1
# modify the host as your host IP or DNS name.
worker.node1.port=8109
worker.node1.host=localhost
worker.node1.type=ajp13
worker.node1.lbfactor=1
#worker.node1.prepost_timeout=1 #Not required if using ping_mode=A
#worker.node1.connect_timeout=8 #Not required if using ping_mode=A
worker.node1.ping_mode=A #As of mod_jk 1.2.27
#worker.node1.activation=a
#worker.node1.connection_pool_size=55
Let mod_jk decide on the connection_pool_size (when using Apache), do
not activate this line.
worker.node1.socket_keepalive=True
worker.node1.socket_timeout=5
I would not use a socket_timeout (personal preference).
#worker.node1.connection_pool_timeout=30
#worker.node1.recovery_options=7
#worker.node1.ping_timeout=8
#worker.node1.prepost_timeout=8
#worker.node1.max_packet_size=65536
#worker.node1.reply_timeout=5000
worker.node1.retries=2
I would add the following (also to node2):
worker.node1.recovery_options=7
worker.node1.ping_timeout=1
worker.node1.socket_connect_timeout=2000
worker.node1.connection_pool_minsize=0
worker.node1.connection_pool_timeout=600
# CAUTION: you need to set connectionTimeout in your Tomcat # or JBoss
connector to the same value # as connection_pool_timeout. But for mod_jk
it is in # seconds, for Tomcat or JBoss in milliseconds.
# So add connectionTimeout=60 to your AJP connectors.
Personally I would also add a reply timeout, but that depends on the
application:
worker.node1.reply_timeout=30
and then in order to not make nodes fail because of few reply timeouts:
worker.loadbalancer.max_reply_timeouts=20
Finally I would add the following to the balancer, which was introduced
in 1.2.28:
worker.loadbalancer.error_escalation_time=0
# Define Node2
# modify the host as your host IP or DNS name.
worker.node2.port=8209
worker.node2.host= localhost
worker.node2.type=ajp13
worker.node2.lbfactor=1
#worker.node2.prepost_timeout=1 #Not required if using ping_mode=A
#worker.node2.connect_timeout=8 #Not required if using ping_mode=A
worker.node2.ping_mode=A #As of mod_jk 1.2.27
#worker.node2.activation=a
#worker.node2.connection_pool_size=55
worker.node2.socket_keepalive=True
worker.node2.socket_timeout=5
#worker.node2.connection_pool_timeout=30
#worker.node2.recovery_options=7
#worker.node2.ping_timeout=8
#worker.node2.prepost_timeout=8
#worker.node2.max_packet_size=65536
#worker.node2.reply_timeout=5000
worker.node2.retries=2
# Load-balancing behaviour
worker.loadbalancer.type=lb
worker.loadbalancer.method=B
worker.loadbalancer.balance_workers=node1,node2
# Status worker for managing load balancer worker.status.type=status
#worker.node1.fail_on_status=304
#worker.node2
Re: Need Help - Mod_jk/Apache - Mutiple Content issue
On 14.09.2009 11:33, balakarthik.baska...@wipro.com wrote: Thanks Rainer. Just Curious,did these settings helped to overcome the multiple content problem in your local setup? I never had a multiple content problem - maybe due to good configuration. Also,thanks for your pointer on the thread id's for getting the correlations. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Need Help - Mod_jk/Apache - Mutiple Content issue
But I think you mentioned earlier that you were able to simulate the problem. Regards, Bala -Original Message- From: Rainer Jung [mailto:rainer.j...@kippdata.de] Sent: Monday, September 14, 2009 3:11 PM To: Tomcat Users List Subject: Re: Need Help - Mod_jk/Apache - Mutiple Content issue On 14.09.2009 11:33, balakarthik.baska...@wipro.com wrote: Thanks Rainer. Just Curious,did these settings helped to overcome the multiple content problem in your local setup? I never had a multiple content problem - maybe due to good configuration. Also,thanks for your pointer on the thread id's for getting the correlations. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org Please do not print this email unless it is absolutely necessary. The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Need Help - Mod_jk/Apache - Mutiple Content issue
On 14.09.2009 12:02, balakarthik.baska...@wipro.com wrote: But I think you mentioned earlier that you were able to simulate the problem. I didn't simulate, I understodd from reading the code. I also said, it should go away when using recovery_options. So please: - optimize your configuration - see whether you still have problems and let us know if so The configurations will not help with Akamai eventually caching incomplete responses. If a backend doesn't send the full response, we can't fix it with configuration. Regards, Rainer -Original Message- From: Rainer Jung [mailto:rainer.j...@kippdata.de] Sent: Monday, September 14, 2009 3:11 PM To: Tomcat Users List Subject: Re: Need Help - Mod_jk/Apache - Mutiple Content issue On 14.09.2009 11:33, balakarthik.baska...@wipro.com wrote: Thanks Rainer. Just Curious,did these settings helped to overcome the multiple content problem in your local setup? I never had a multiple content problem - maybe due to good configuration. Also,thanks for your pointer on the thread id's for getting the correlations. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Need Help - Mod_jk/Apache - Mutiple Content issue
Hi Rainer, I did change the process id configuration,and the following is observed: Access Log: === 3052 1536 127.0.0.1 - - [14/Sep/2009:17:57:04 +0530] GET /test/test.jsp HTTP/1.1 200 61584 3052 1524 127.0.0.1 - - [14/Sep/2009:17:57:05 +0530] GET /test/images/logo.gif HTTP/1.1 304 - But from the modjk log,I can see the following and no relevant pid or thread id is seen: [Mon Sep 14 17:57:05 2009]loadbalancer localhost 0.687500 GET 200 [Mon Sep 14 17:57:05 2009]loadbalancer localhost 0.015625 GET 304 Can you pl let me know what configuration will make the pid to appear in both logs to find out the correlation Regards, Bala -Original Message- From: Rainer Jung [mailto:rainer.j...@kippdata.de] Sent: Monday, September 14, 2009 4:29 PM To: Tomcat Users List Subject: Re: Need Help - Mod_jk/Apache - Mutiple Content issue On 14.09.2009 12:02, balakarthik.baska...@wipro.com wrote: But I think you mentioned earlier that you were able to simulate the problem. I didn't simulate, I understodd from reading the code. I also said, it should go away when using recovery_options. So please: - optimize your configuration - see whether you still have problems and let us know if so The configurations will not help with Akamai eventually caching incomplete responses. If a backend doesn't send the full response, we can't fix it with configuration. Regards, Rainer -Original Message- From: Rainer Jung [mailto:rainer.j...@kippdata.de] Sent: Monday, September 14, 2009 3:11 PM To: Tomcat Users List Subject: Re: Need Help - Mod_jk/Apache - Mutiple Content issue On 14.09.2009 11:33, balakarthik.baska...@wipro.com wrote: Thanks Rainer. Just Curious,did these settings helped to overcome the multiple content problem in your local setup? I never had a multiple content problem - maybe due to good configuration. Also,thanks for your pointer on the thread id's for getting the correlations. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org Please do not print this email unless it is absolutely necessary. The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Need Help - Mod_jk/Apache - Mutiple Content issue
On 14.09.2009 14:40, balakarthik.baska...@wipro.com wrote: Hi Rainer, I did change the process id configuration,and the following is observed: Access Log: === 3052 1536 127.0.0.1 - - [14/Sep/2009:17:57:04 +0530] GET /test/test.jsp HTTP/1.1 200 61584 3052 1524 127.0.0.1 - - [14/Sep/2009:17:57:05 +0530] GET /test/images/logo.gif HTTP/1.1 304 - But from the modjk log,I can see the following and no relevant pid or thread id is seen: [Mon Sep 14 17:57:05 2009]loadbalancer localhost 0.687500 GET 200 [Mon Sep 14 17:57:05 2009]loadbalancer localhost 0.015625 GET 304 Those two lines are the internal jk request log lines. They are not necessary, because you can add all important access log data to the real access log. See my recommendations. That's why I wrote you should simply delete the JkRequestLogFormat. All other log lines in the mod_jk log do have pid and tid. Regards, Rainer -Original Message- From: Rainer Jung [mailto:rainer.j...@kippdata.de] Sent: Monday, September 14, 2009 4:29 PM To: Tomcat Users List Subject: Re: Need Help - Mod_jk/Apache - Mutiple Content issue On 14.09.2009 12:02, balakarthik.baska...@wipro.com wrote: But I think you mentioned earlier that you were able to simulate the problem. I didn't simulate, I understodd from reading the code. I also said, it should go away when using recovery_options. So please: - optimize your configuration - see whether you still have problems and let us know if so The configurations will not help with Akamai eventually caching incomplete responses. If a backend doesn't send the full response, we can't fix it with configuration. Regards, Rainer -Original Message- From: Rainer Jung [mailto:rainer.j...@kippdata.de] Sent: Monday, September 14, 2009 3:11 PM To: Tomcat Users List Subject: Re: Need Help - Mod_jk/Apache - Mutiple Content issue On 14.09.2009 11:33, balakarthik.baska...@wipro.com wrote: Thanks Rainer. Just Curious,did these settings helped to overcome the multiple content problem in your local setup? I never had a multiple content problem - maybe due to good configuration. Also,thanks for your pointer on the thread id's for getting the correlations. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Need Help - Mod_jk/Apache - Mutiple Content issue
On 10.09.2009 06:30, balakarthik.baska...@wipro.com wrote: Hi Rainer, I did try with the recovery_option as 7 and I was able to see a partial content being sent even in that case.Also,this configuration is what we tried in prod as well(we removed socket_timeout,had reply_timeout to 3 mins and recovery_options to 7).But still multiple content was seen.So,hope this partial content was the one that was served to akamai and guess its messed up there. Partial content is possible. We can't take it back, once it is sent to the client, and we can't fix, if the backend doesn't send all the response. Partial content should not lead to dublicate content though. Partial content can be detected by the client, because either the content length is not right, or the chunked encoded transfer ends prematurely. The client (akamai) should then not serve this partial content. Also,when this option is set,the session failover behaviour gets completely ruled out as we make the worker not to retry and this will not try to contact the other worker and thereby the other app instance. No, failover should still work. Pfa the configuration files that I used in my local setup. They didn't make it to the list. You can try to post inline. Regards, Rainer -Original Message- From: Rainer Jung [mailto:rainer.j...@kippdata.de] Sent: Wednesday, September 09, 2009 11:39 PM To: users@tomcat.apache.org Subject: Re: Need Help - Mod_jk/Apache - Mutiple Content issue On 09.09.2009 16:11, balakarthik.baska...@wipro.com wrote: Thanks for your reply.I am able to simulate the problem with 1.2.28 as well.This is the mod_jk version that I used in my local setup.Our prod version has 1.2.27.So regardless of the version,the issue is seen. OK Pfa the detailed debug logs of mod_jk for different combinations that I tried out and when the issue occured. Thanks. The reason is in your configuration. There is one switch that is much recommended and form the logs it seems you already found and tried it: the recovery_options. See: http://tomcat.apache.org/connectors-doc/reference/workers.html and also look carefully at: http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html You should set recovery_options to 7. This switch is exactly for disabling request retries in case Tomcat already replied, or at least the request was send to Tomcat. It should fix your problem. So maybe we are back to the beginning: why don't you post your config, so we can see, if there is more potential for optimization. From the timestamps in the jk log I can see e.g., that you are using a hard configured time stamp format. That's not to good, because that way you don't see the milliseconds, that are logged since a couple of versions by default. The other main scenario apart from the one explained below that I am seeing is when I tried setting up a JBOSS cluster node setup of two nodes with session replication (with socket_timeout not set or default to infinite).Node 1 app code is made to take a longer time to respond.When the node1 has started writing a huge chunk,I abruptly kill the node 1 and I see mod_jk failing over the request to node2.But the end result is the data written by node1 until it has been killed clubbed with the data written by node2 is seen in the browser. Will also be fixed by recovery_options. This is where we are not able to make a decision of even having socket_timeout to be infinite or to a defined limit(as we never know how long/when the root cause of the delay might occur) as there may be lot of possibility of failover and we suscpect the multiple content issue would be coming up when failover happens.The worst part is when akamai comes into picture where I hope this partial content is cached in their edge servers and completely messes up by showing the content to the users.So for all other subsequent requests,the data which is served from akamai cache is the messedup multiple content thereby completely degrading the customer experience. See above. I generally am not in favor of the socket_timeout. Even after your original problem goes away by using recovery_options, the behaviour of the general socket_timeout is not very nice. I would stick with a couple of the other timeouts. Regards, Rainer -Original Message- From: Rainer Jung [mailto:rainer.j...@kippdata.de] Sent: Wednesday, September 09, 2009 7:16 PM To: Tomcat Users List Subject: Re: Need Help - Mod_jk/Apache - Mutiple Content issue Hi, On 09.09.2009 06:56, balakarthik.baska...@wipro.com wrote: Hi, In our production envt,we are making use of a configuration of Apache(2.2.10-1)+Mod_jk(1.2.27)+JBOSS(4.0.5)+ATG 2007.1. We are facing a problem where multiple content is being displayed to the customer.It was observed that the pages were being periodically rendered or rendered multiple times.The page was rendered
Re: Need Help - Mod_jk/Apache - Mutiple Content issue
balakarthik.baska...@wipro.com wrote: ... Just jumping in here, but doesn't that request to /test/test.jsp appear to be looping ? or is that by design ? ReplyTimeout_Analysis.txt : [Thu Sep 10 14:46:47 2009][5024:2952] [debug] jk_ajp_common.c (1259): 00f06D 20 74 65 73 74 69 6E 63 6C 75 64 65 31 20 50 - m.testinclude1.P [Thu Sep 10 14:46:47 2009][5024:2952] [debug] jk_ajp_common.c (1259): 010061 67 65 0D 0A 48 65 6C 6C 6F 6F 6F 6F 6F 20 66 - age..Hello.f [Thu Sep 10 14:46:47 2009][5024:2952] [debug] jk_ajp_common.c (1259): 011072 6F 6D 20 74 65 73 74 69 6E 63 6C 75 64 65 31 - rom.testinclude1 - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Need Help - Mod_jk/Apache - Mutiple Content issue
No I explicitly made it to loop so that I can write a content for a time longer than the reply_timout setting to simulate and look at the behaviour. Regards, Bala -Original Message- From: André Warnier [mailto:a...@ice-sa.com] Sent: Thursday, September 10, 2009 6:54 PM To: Tomcat Users List Subject: Re: Need Help - Mod_jk/Apache - Mutiple Content issue balakarthik.baska...@wipro.com wrote: ... Just jumping in here, but doesn't that request to /test/test.jsp appear to be looping ? or is that by design ? ReplyTimeout_Analysis.txt : [Thu Sep 10 14:46:47 2009][5024:2952] [debug] jk_ajp_common.c (1259): 00f06D 20 74 65 73 74 69 6E 63 6C 75 64 65 31 20 50 - m.testinclude1.P [Thu Sep 10 14:46:47 2009][5024:2952] [debug] jk_ajp_common.c (1259): 010061 67 65 0D 0A 48 65 6C 6C 6F 6F 6F 6F 6F 20 66 - age..Hello.f [Thu Sep 10 14:46:47 2009][5024:2952] [debug] jk_ajp_common.c (1259): 011072 6F 6D 20 74 65 73 74 69 6E 63 6C 75 64 65 31 - rom.testinclude1 - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org Please do not print this email unless it is absolutely necessary. The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Need Help - Mod_jk/Apache - Mutiple Content issue
balakarthik.baska...@wipro.com wrote: No I explicitly made it to loop so that I can write a content for a time longer than the reply_timout setting to simulate and look at the behaviour. Ok, but does a reply timeout really apply in that case ? I would intuitively think that a reply timeout would be irrelevant as soon as the first byte of the reply arrives. How long the response is after that, should not matter. No ? As I mentioned, I was just jumping in in the middle. Disregard my comments if they are not relevant. I was just looking at the logs and trying to figure out what the problem is. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Need Help - Mod_jk/Apache - Mutiple Content issue
Yes you are right.I think I didn't mention it clearly.I am having this while loop for writing a huge chunk of data followed by a thread that sleeps for some time and that is where I am expecting my reply_timeout to be timed out and the data written till the sleep of the thread is seen in the browser and the log file. Regards, Bala -Original Message- From: André Warnier [mailto:a...@ice-sa.com] Sent: Thursday, September 10, 2009 7:06 PM To: Tomcat Users List Subject: Re: Need Help - Mod_jk/Apache - Mutiple Content issue balakarthik.baska...@wipro.com wrote: No I explicitly made it to loop so that I can write a content for a time longer than the reply_timout setting to simulate and look at the behaviour. Ok, but does a reply timeout really apply in that case ? I would intuitively think that a reply timeout would be irrelevant as soon as the first byte of the reply arrives. How long the response is after that, should not matter. No ? As I mentioned, I was just jumping in in the middle. Disregard my comments if they are not relevant. I was just looking at the logs and trying to figure out what the problem is. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org Please do not print this email unless it is absolutely necessary. The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Need Help - Mod_jk/Apache - Mutiple Content issue
balakarthik.baska...@wipro.com wrote: Yes you are right.I think I didn't mention it clearly.I am having this while loop for writing a huge chunk of data followed by a thread that sleeps for some time and that is where I am expecting my reply_timeout to be timed out and the data written till the sleep of the thread is seen in the browser and the log file. Ok, now I do understand better, I think. There is a request. The Tomcat app starts answering (a long chunk of data), then crashes. This is simulated by your test script, which sends a long chunk of data, then sleeps (to simulate a crash). Eventually, some timeout occurs, which (may) cause another element to consider this Tomcat unresponsive, and switch the request to another one. This other one then restarts the request from the beginning. The final result being that the requesting browser first received an incomplete chunk of data from the first webapp, and now starts receiving the same data (again) from another webapp. Again intuitively, I don't know how any server-side scheme is going to solve that. When the first webapp starts answering, that Tomcat has to start sending the answer back to the client. If it then crashes, it is too late to call it back; the data is already on the wire to the client. It would be in my view the client's responsibility to ensure that it discards any incomplete data received. But I don't know of any browser for instance, which does that. They all start to try to display the data as soon as they have the beginning of it. I think that if you really need this, then you would have to insert some other item between Tomcat and the client, which acts as a proxy, and only starts sending the response to the client when it is sure that it has received and buffered a complete response from the server. But then, this item becomes the weak link. It could also crash.. Or else you need an applet at the client side, which does the same thing. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Need Help - Mod_jk/Apache - Mutiple Content issue
On 10.09.2009 14:46, balakarthik.baska...@wipro.com wrote: Hi Rainer, I did tried with a lot of combinations for the settings and can see that when recovery_options is set,no multiple content is seen and partial content is seen followed by an OK message when a huge chunk of data is written. However,the failover case didn't at all work out when the recovery_options is set.Pfa a doc with the scenarios that I tested and the results that I am seeing. However,in our prod envt,we currently have the reply_timeout and the recovery_options setting available and we have reomoved the socket_timeout.The recent behaviour that we are observing is,unlike the duplicate content with the header that we saw earlier with socket_timeout,we are seeing some of the content in the middle of the response being duplicated. We are making use of dynamic includes of the jsp files (with ATG dsp tags) and when I tried having the same in my local setup,I could see the response being written into the socket once the jsp completes.But not able to replicate a scenario of recreating multiple content from the middle of the page/content served by a fragment that is included. Pfa a file with the behaviour that I am noticing from the access and jk log files.For the test app,I could see only one GET request and a corresponding response in the jk log with the response time.I included one image and I can see a corresponding entry for the image as well in both files.However,there are no entries seen for the included jsp files. I am attaching the log entries from our prod envt as well which I took when I was able to see a timeout reply in the jk file.However,from the access log I couldn't interpret a corresponding entry/behaviour.Could you pl help on how the multiple content could have been occuring in the middle and your insights from the log files? (Also,pl note that we have currenlty turned off akamai in prod and the timeout is seen without akamai as well) Let's start in the right way, first fix the configuration, then do and analyze the tests, maybe resulting in more config optimization. Please post your config. I had a quick look at the files you attached, but it looks very much like we first need to improve the config, before analyzing behaviour, that will likely not appear with the right config. Also,reg your explanation about akamai,I can contact akamai to find out if they can handle the prematurely closed data differently.Do you remember any setting to the browser to interpret this invalid data based on content length/premature closure?(just for testing though wouldn't apply to real life scenario) No there is no such browser setting that I'm aware of. But akamai is not a browser, its a server that behaves more like a cache. So they need to solve a server problem, not an end user problem. They should immediately invalidate content, that was delivered over a connection that had a protocol abort. Thanks much for your help.I can give you a call if you would like to discuss this over phone. Lets first check the config, maybe then everything will be fine ;) Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Need Help - Mod_jk/Apache - Mutiple Content issue
Hi, On 09.09.2009 06:56, balakarthik.baska...@wipro.com wrote: Hi, In our production envt,we are making use of a configuration of Apache(2.2.10-1)+Mod_jk(1.2.27)+JBOSS(4.0.5)+ATG 2007.1. We are facing a problem where multiple content is being displayed to the customer.It was observed that the pages were being periodically rendered or rendered multiple times.The page was rendered with a standard navigational header showing two times. Depending on the browser, the duplicate headers was stacked vertically or horizontally. The issue was further exasperated because a particular fragment was cached at Akamai, and once this double header was served up by origin, it was then cached at Akamai and included in customer pages at a very high rate. Our Analysis So far We did a local setup of the above configuration and we are able to recreate the multiple content problem.The setup involves apache configured with mod_jk having two worker nodes balanced by a load balancer worker.The problem occurs when the app code takes a longer time than the time defined in the socket_timeout parameter of the worker nodes. We have set the socket_timeout config of the worker nodes to be 2 secs for both nodes and the appserver jsp logic is made to sleep for 3secs. The behaviour observed is as follows: The first worker node fails after the socket timeout time and the second worker node is able to retreive the response from the app server.However while turning on the debug of mod_jk,we are observing that a partial content has been served by the AJP Connector/Container from the App server for the node 1.Also,a full content has been served for the request initiated by node2.The full content follows the partial content that has been already sent to the client .Hence the browser has a duplicate/multiple content shown to the customers. What are we looking for? The open question at this point that we have is how/what configuration could either limit this partial response (or) send an error response to the client when the worker node is found to be inactive? (or) handling the situation of the app server taking a longer time to respond than the defined timeouts (in mod_jk) and a partial content being writtent into the socket,received by mod_jk and sent to the client? The problem gets aggravated when we akamai comes into picture. Thanks a lot for your very careful analysis. I usually do not recommend socket_timeout, so that's why I never noticed such behaviour. But of course this is a bug which has to be fixed. Is there a possibility of getting a hand on the part of the debug log, where the problem occurs? I'm pretty sure, we can quickly fix it then. If the problem is easy for you to reproduce, would you mind testing once again using version 1.2.28 of mod_jk? There have been some changes there not directly related to the problem, but it would be good to know, whether you can still reproduce with 1.2.28. I will nevertheless investiaget, what's wrong in 1.2.27 and possibly also 1.2.28. Thanks again and a debug log snippet would be very helpful. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Need Help - Mod_jk/Apache - Mutiple Content issue
On 09.09.2009 16:11, balakarthik.baska...@wipro.com wrote: Thanks for your reply.I am able to simulate the problem with 1.2.28 as well.This is the mod_jk version that I used in my local setup.Our prod version has 1.2.27.So regardless of the version,the issue is seen. OK Pfa the detailed debug logs of mod_jk for different combinations that I tried out and when the issue occured. Thanks. The reason is in your configuration. There is one switch that is much recommended and form the logs it seems you already found and tried it: the recovery_options. See: http://tomcat.apache.org/connectors-doc/reference/workers.html and also look carefully at: http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html You should set recovery_options to 7. This switch is exactly for disabling request retries in case Tomcat already replied, or at least the request was send to Tomcat. It should fix your problem. So maybe we are back to the beginning: why don't you post your config, so we can see, if there is more potential for optimization. From the timestamps in the jk log I can see e.g., that you are using a hard configured time stamp format. That's not to good, because that way you don't see the milliseconds, that are logged since a couple of versions by default. The other main scenario apart from the one explained below that I am seeing is when I tried setting up a JBOSS cluster node setup of two nodes with session replication (with socket_timeout not set or default to infinite).Node 1 app code is made to take a longer time to respond.When the node1 has started writing a huge chunk,I abruptly kill the node 1 and I see mod_jk failing over the request to node2.But the end result is the data written by node1 until it has been killed clubbed with the data written by node2 is seen in the browser. Will also be fixed by recovery_options. This is where we are not able to make a decision of even having socket_timeout to be infinite or to a defined limit(as we never know how long/when the root cause of the delay might occur) as there may be lot of possibility of failover and we suscpect the multiple content issue would be coming up when failover happens.The worst part is when akamai comes into picture where I hope this partial content is cached in their edge servers and completely messes up by showing the content to the users.So for all other subsequent requests,the data which is served from akamai cache is the messedup multiple content thereby completely degrading the customer experience. See above. I generally am not in favor of the socket_timeout. Even after your original problem goes away by using recovery_options, the behaviour of the general socket_timeout is not very nice. I would stick with a couple of the other timeouts. Regards, Rainer -Original Message- From: Rainer Jung [mailto:rainer.j...@kippdata.de] Sent: Wednesday, September 09, 2009 7:16 PM To: Tomcat Users List Subject: Re: Need Help - Mod_jk/Apache - Mutiple Content issue Hi, On 09.09.2009 06:56, balakarthik.baska...@wipro.com wrote: Hi, In our production envt,we are making use of a configuration of Apache(2.2.10-1)+Mod_jk(1.2.27)+JBOSS(4.0.5)+ATG 2007.1. We are facing a problem where multiple content is being displayed to the customer.It was observed that the pages were being periodically rendered or rendered multiple times.The page was rendered with a standard navigational header showing two times. Depending on the browser, the duplicate headers was stacked vertically or horizontally. The issue was further exasperated because a particular fragment was cached at Akamai, and once this double header was served up by origin, it was then cached at Akamai and included in customer pages at a very high rate. Our Analysis So far We did a local setup of the above configuration and we are able to recreate the multiple content problem.The setup involves apache configured with mod_jk having two worker nodes balanced by a load balancer worker.The problem occurs when the app code takes a longer time than the time defined in the socket_timeout parameter of the worker nodes. We have set the socket_timeout config of the worker nodes to be 2 secs for both nodes and the appserver jsp logic is made to sleep for 3secs. The behaviour observed is as follows: The first worker node fails after the socket timeout time and the second worker node is able to retreive the response from the app server.However while turning on the debug of mod_jk,we are observing that a partial content has been served by the AJP Connector/Container from the App server for the node 1.Also,a full content has been served for the request initiated by node2.The full content follows the partial content that has been already sent to the client .Hence the browser has a duplicate/multiple content shown to the customers. What are we looking for? The open question at this point that we
RE: Need Help - Mod_jk/Apache - Mutiple Content issue
Hi Rainer, I did try with the recovery_option as 7 and I was able to see a partial content being sent even in that case.Also,this configuration is what we tried in prod as well(we removed socket_timeout,had reply_timeout to 3 mins and recovery_options to 7).But still multiple content was seen.So,hope this partial content was the one that was served to akamai and guess its messed up there. Also,when this option is set,the session failover behaviour gets completely ruled out as we make the worker not to retry and this will not try to contact the other worker and thereby the other app instance. Pfa the configuration files that I used in my local setup. Regards, Bala -Original Message- From: Rainer Jung [mailto:rainer.j...@kippdata.de] Sent: Wednesday, September 09, 2009 11:39 PM To: users@tomcat.apache.org Subject: Re: Need Help - Mod_jk/Apache - Mutiple Content issue On 09.09.2009 16:11, balakarthik.baska...@wipro.com wrote: Thanks for your reply.I am able to simulate the problem with 1.2.28 as well.This is the mod_jk version that I used in my local setup.Our prod version has 1.2.27.So regardless of the version,the issue is seen. OK Pfa the detailed debug logs of mod_jk for different combinations that I tried out and when the issue occured. Thanks. The reason is in your configuration. There is one switch that is much recommended and form the logs it seems you already found and tried it: the recovery_options. See: http://tomcat.apache.org/connectors-doc/reference/workers.html and also look carefully at: http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html You should set recovery_options to 7. This switch is exactly for disabling request retries in case Tomcat already replied, or at least the request was send to Tomcat. It should fix your problem. So maybe we are back to the beginning: why don't you post your config, so we can see, if there is more potential for optimization. From the timestamps in the jk log I can see e.g., that you are using a hard configured time stamp format. That's not to good, because that way you don't see the milliseconds, that are logged since a couple of versions by default. The other main scenario apart from the one explained below that I am seeing is when I tried setting up a JBOSS cluster node setup of two nodes with session replication (with socket_timeout not set or default to infinite).Node 1 app code is made to take a longer time to respond.When the node1 has started writing a huge chunk,I abruptly kill the node 1 and I see mod_jk failing over the request to node2.But the end result is the data written by node1 until it has been killed clubbed with the data written by node2 is seen in the browser. Will also be fixed by recovery_options. This is where we are not able to make a decision of even having socket_timeout to be infinite or to a defined limit(as we never know how long/when the root cause of the delay might occur) as there may be lot of possibility of failover and we suscpect the multiple content issue would be coming up when failover happens.The worst part is when akamai comes into picture where I hope this partial content is cached in their edge servers and completely messes up by showing the content to the users.So for all other subsequent requests,the data which is served from akamai cache is the messedup multiple content thereby completely degrading the customer experience. See above. I generally am not in favor of the socket_timeout. Even after your original problem goes away by using recovery_options, the behaviour of the general socket_timeout is not very nice. I would stick with a couple of the other timeouts. Regards, Rainer -Original Message- From: Rainer Jung [mailto:rainer.j...@kippdata.de] Sent: Wednesday, September 09, 2009 7:16 PM To: Tomcat Users List Subject: Re: Need Help - Mod_jk/Apache - Mutiple Content issue Hi, On 09.09.2009 06:56, balakarthik.baska...@wipro.com wrote: Hi, In our production envt,we are making use of a configuration of Apache(2.2.10-1)+Mod_jk(1.2.27)+JBOSS(4.0.5)+ATG 2007.1. We are facing a problem where multiple content is being displayed to the customer.It was observed that the pages were being periodically rendered or rendered multiple times.The page was rendered with a standard navigational header showing two times. Depending on the browser, the duplicate headers was stacked vertically or horizontally. The issue was further exasperated because a particular fragment was cached at Akamai, and once this double header was served up by origin, it was then cached at Akamai and included in customer pages at a very high rate. Our Analysis So far We did a local setup of the above configuration and we are able to recreate the multiple content problem.The setup involves apache configured with mod_jk having two worker nodes balanced by a load balancer worker.The problem occurs when the app code takes a longer time than
