Re: Apache/mod_jk serves random files from tomcat
On 08/10/2010 04:40, domiguo wrote: Has this thread has a clear answer now? You resurrected a thread which is over a year old. If you have a problem, please start a new email and describe the details and your environment. p 0x62590808.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature
Re: Apache/mod_jk serves random files from tomcat
Has this thread has a clear answer now? -- View this message in context: http://old.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p29912221.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache/mod_jk serves random files from tomcat
Problem found in mod_proxy_ajp too, https://issues.apache.org/bugzilla/show_bug.cgi?id=46949 /Jakob On Fri, Mar 27, 2009 at 10:33 PM, LukeK l...@sce.net wrote: Rainer Jung-3 wrote: Thanks very much for the feedback. Considering the severity of the problem, if you could give us another update at a time you think is appropriate (depending on how often the problem happened before). It's been about a month now since we dropped libtcnative, and the problem has not been reported again. Right now, I've seen enough to suggest it was tcnative. Would you please be so kind to also tell us, which tcnative version you were using? 1.14 though 1.16; perhaps a version or two before that, I cannot say for sure. Cheers! Luke -- View this message in context: http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p22746664.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- Jakob Ericsson, JAKERI AB Tel. +46 704 533 627 - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache/mod_jk serves random files from tomcat
Rainer Jung-3 wrote: Thanks very much for the feedback. Considering the severity of the problem, if you could give us another update at a time you think is appropriate (depending on how often the problem happened before). It's been about a month now since we dropped libtcnative, and the problem has not been reported again. Right now, I've seen enough to suggest it was tcnative. Would you please be so kind to also tell us, which tcnative version you were using? 1.14 though 1.16; perhaps a version or two before that, I cannot say for sure. Cheers! Luke -- View this message in context: http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p22746664.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache/mod_jk serves random files from tomcat
Rainer, Rainer Jung-3 wrote: I guess you mean the lines with the 503 are the bad responses? But those do not indicate, that the probe gets back the page requested by someone else, it shows that the web server or Tomcat throw an HTTP error, namely 503. In this case I would guess, that mod_jk detected an error and put th enode into error status. You should check your mod_jk log file. It might also be good to temporarily activate the access log of Tomcat too, in order to check, whether the 503 already came from there or not. I would expect the develop observation and this one are two different things. Actually I was trying to draw comparison to the original poster, Tim Redding's comments about how the file size on a static page would change. Unfortunately, I gave you a bad example, which included the 503 error. Please ignore that. Basically, in the access_log we see the file size change, because it is serving the wrong page. We had to throw a band aid up, which monitors the file sizes and if they differ, the script bounces Apache. Seems to work ok for now, but its kind of ugly. Rainer Jung-3 wrote: Do both (mod_jk and mod_jk2 show the problems A=develop and B=probes? Yes, after looking closer, we're having the same problem with both connectors. We're currently in the process of upgrading everything-- going to Apache 2.2.11, mod_jk 1.2.27, and Tomcat 5.5.27. I'll let you know if any of these help. -- View this message in context: http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p22628124.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache/mod_jk serves random files from tomcat
We were not using APR. Since this was a live project I had no choice but switch to regular http proxy which doesn't work as well as AJP (speed/functionality) but is consistent. Alas, as much as I'd like to help with solving this issue, I wasn't able to reproduce in a test environment and the production environment is now live so I can't mess with it. Cheers! Yuval Perlov www.r-u-on.com On Mar 10, 2009, at 6:27 PM, Rainer Jung wrote: Hi Yuval, did you find out in the meantime, whether you were using the tcnative (aka APR) connector? Regards, Rainer On 19.02.2009 11:34, Yuval Perlov wrote: Just the swapping responses has me concerned. Thank you so much for the rest of your responses we will put them to good use once we give up on AJP completely. Yuval On Feb 18, 2009, at 8:45 PM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Yuval, On 2/17/2009 1:48 PM, Yuval Perlov wrote: Is APR part of tomcat or apache [httpd]? APR is the Apache Portable Runtime. Technically, it's its own beast and is used by both httpd and Tomcat (optionally). If I am running on linux and have no .so files in my tomcat directory does that mean I have no APR installed? The Tomcat directory isn't the only place .so files could be located. Anywhere in the java.library.path is possible. If you have an AprLifecycleListener configured in your server.xml, then you are attempting to use APR. If you get a message in catalina.out on startup that says something like APR Configured or APR library not found then you have your answer. On a more positive note, we switched to proxy_http (after making the necessary code changes) and everything works now - no more mixed content. Of course we lost a lot of necessary functionality: 1. request.isSecure() doesn't work You can always use https :) 2. we don't know the server name we are hit with (since it is hard coded in httpd.conf) This should be an option in mod_proxy. Is it not? ProxyPreserveHost? 3. we have no access to the source IP (for geo location) Why not use mod_headers to convert the original IP address into an X-Original-IP header. Better yet, use the X-Forwarded-For header that should be set by default by mod_proxy. BTW - Am I the only one that is seriously worried that this kind of problem can even exist on a platform of this maturity? Which problem? The swapping-responses problem or everything else you've outlined about your inadequate configuration? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache/mod_jk serves random files from tomcat
Hi Rainer, Thanks for the reply. Answers to your questions below, Rainer Jung-3 wrote: Jus to make sure, we are talking about the same kind of observation: could you please describe independently, how the observed problem looks like in your case? In development, the developers are getting other people pages. So user1 requests pageA and gets user2's pageB. In production, we don't get user input, but the probe on the load balancer is not getting the response it is looking for, so it thinks the machine its checking is down. The probe is called serverlive.jsp. Here is the accesslog entry during the problem (13 being the primary LB, 14 the backup): xxx.xxx.xxx.14 - - [12/Mar/2009:23:09:53 -0500] GET /serverlive.jsp 200 13 xxx.xxx.xxx.13 - - [12/Mar/2009:23:09:54 -0500] GET /serverlive.jsp 200 13 xxx.xxx.xxx.14 - - [12/Mar/2009:23:09:59 -0500] GET /serverlive.jsp 200 13 xxx.xxx.xxx.13 - - [12/Mar/2009:23:09:59 -0500] GET /serverlive.jsp 200 13 xxx.xxx.xxx.14 - - [12/Mar/2009:23:10:04 -0500] GET /serverlive.jsp 503 1070 xxx.xxx.xxx.13 - - [12/Mar/2009:23:10:05 -0500] GET /serverlive.jsp 503 1070 xxx.xxx.xxx.14 - - [12/Mar/2009:23:10:10 -0500] GET /serverlive.jsp 503 1070 xxx.xxx.xxx.13 - - [12/Mar/2009:23:10:10 -0500] GET /serverlive.jsp 503 1070 xxx.xxx.xxx.14 - - [12/Mar/2009:23:11:00 -0500] GET /serverlive.jsp 503 1070 xxx.xxx.xxx.14 - - [12/Mar/2009:23:12:34 -0500] GET /serverlive.jsp 503 997 xxx.xxx.xxx.13 - - [12/Mar/2009:23:11:46 -0500] GET /serverlive.jsp 503 997 xxx.xxx.xxx.13 - - [12/Mar/2009:23:12:31 -0500] GET /serverlive.jsp 503 997 xxx.xxx.xxx.13 - - [12/Mar/2009:23:11:01 -0500] GET /serverlive.jsp 503 997 I'll try and get some log entries from development. Rainer Jung-3 wrote: Since you see the problem with mod_jk2 and with mod_jk I somehow doubt, that it comes form mod_jk (but hey, I'm involved in mod_jk development, so that might simply be defense. This is the main reason I posted here. If I'm indeed seeing the same problem as the others here, then my case may disprove the mod_jk theory. Or perhaps the issue resides in both my versions? Rainer Jung-3 wrote: What is obvious, your Tomcat is *very* outdated. You are using a no longer supported major version (5.0) and with 5.0 you are using a very old minor version. If you have any chance, upgrade your Tomcat. Yes, I know. I'll see what I can do. Rainer Jung-3 wrote: Apart from that: what else can you tell about the problem? Are there log entries either from mod_jk, Apache httpd or Tomcat associated with these events? Would you be able to snoop traffic between httpd and Tomcat and between httpd and the clients? We haven't been seen any errors, in any logs. I can go through the logs and compare them, and then compare those findings between the enviroments. Not sure whats involved in snooping traffic. I can look into that as well. Rainer Jung-3 wrote: Where did you get your mod_jk from? How was it build? Not sure the answer to that. Both were installed by other people, who either don't recall their orgins, or are no longer employed here. I'm working on building the 1.2.27 from source right now. We're x86, not sparc, by the way. Thanks for your help. Please let me know of anything else I can provide. I will make updates as new information comes up. -- View this message in context: http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p22500565.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache/mod_jk serves random files from tomcat
On 13.03.2009 17:50, SQ wrote: Rainer Jung-3 wrote: Just to make sure, we are talking about the same kind of observation: could you please describe independently, how the observed problem looks like in your case? In development, the developers are getting other people pages. So user1 requests pageA and gets user2's pageB. In production, we don't get user input, but the probe on the load balancer is not getting the response it is looking for, so it thinks the machine its checking is down. The probe is called serverlive.jsp. Here is the accesslog entry during the problem (13 being the primary LB, 14 the backup): xxx.xxx.xxx.14 - - [12/Mar/2009:23:09:53 -0500] GET /serverlive.jsp 200 13 xxx.xxx.xxx.13 - - [12/Mar/2009:23:09:54 -0500] GET /serverlive.jsp 200 13 xxx.xxx.xxx.14 - - [12/Mar/2009:23:09:59 -0500] GET /serverlive.jsp 200 13 xxx.xxx.xxx.13 - - [12/Mar/2009:23:09:59 -0500] GET /serverlive.jsp 200 13 xxx.xxx.xxx.14 - - [12/Mar/2009:23:10:04 -0500] GET /serverlive.jsp 503 1070 xxx.xxx.xxx.13 - - [12/Mar/2009:23:10:05 -0500] GET /serverlive.jsp 503 1070 xxx.xxx.xxx.14 - - [12/Mar/2009:23:10:10 -0500] GET /serverlive.jsp 503 1070 xxx.xxx.xxx.13 - - [12/Mar/2009:23:10:10 -0500] GET /serverlive.jsp 503 1070 xxx.xxx.xxx.14 - - [12/Mar/2009:23:11:00 -0500] GET /serverlive.jsp 503 1070 xxx.xxx.xxx.14 - - [12/Mar/2009:23:12:34 -0500] GET /serverlive.jsp 503 997 xxx.xxx.xxx.13 - - [12/Mar/2009:23:11:46 -0500] GET /serverlive.jsp 503 997 xxx.xxx.xxx.13 - - [12/Mar/2009:23:12:31 -0500] GET /serverlive.jsp 503 997 xxx.xxx.xxx.13 - - [12/Mar/2009:23:11:01 -0500] GET /serverlive.jsp 503 997 I guess you mean the lines with the 503 are the bad responses? But those do not indicate, that the probe gets back the page requested by someone else, it shows that the web server or Tomcat throw an HTTP error, namely 503. In this case I would guess, that mod_jk detected an error and put th enode into error status. You should check your mod_jk log file. It might also be good to temporarily activate the access log of Tomcat too, in order to check, whether the 503 already came from there or not. I would expect the develop observation and this one are two different things. I'll try and get some log entries from development. Rainer Jung-3 wrote: Since you see the problem with mod_jk2 and with mod_jk I somehow doubt, that it comes form mod_jk (but hey, I'm involved in mod_jk development, so that might simply be defense. This is the main reason I posted here. If I'm indeed seeing the same problem as the others here, then my case may disprove the mod_jk theory. Or perhaps the issue resides in both my versions? Do both (mod_jk and mod_jk2 show the problems A=develop and B=probes? Rainer Jung-3 wrote: What is obvious, your Tomcat is *very* outdated. You are using a no longer supported major version (5.0) and with 5.0 you are using a very old minor version. If you have any chance, upgrade your Tomcat. Yes, I know. I'll see what I can do. Rainer Jung-3 wrote: Apart from that: what else can you tell about the problem? Are there log entries either from mod_jk, Apache httpd or Tomcat associated with these events? Would you be able to snoop traffic between httpd and Tomcat and between httpd and the clients? As indicated above: if the system using mod_jk logs status code 503 in the access log (and the 503 is not in the Tomcat access log), it is *very* likely, that mod_jk writes something to its JkLogFile. Set JkLogLevel to info (but info message alone are not relevant; when you get a 503 it should log some error and interesting info messages at the same time). We haven't been seen any errors, in any logs. I can go through the logs and compare them, and then compare those findings between the enviroments. Not sure whats involved in snooping traffic. I can look into that as well. Rainer Jung-3 wrote: Where did you get your mod_jk from? How was it build? Not sure the answer to that. Both were installed by other people, who either don't recall their orgins, or are no longer employed here. I'm working on building the 1.2.27 from source right now. We're x86, not sparc, by the way. OK. For Solaris x86 we never provided bins (I think), so someone might have built them. Under Solaris you might run into some build troubles, in case you are using a Sun provided httpd. Sun often compiles it with the Sun compiler and there is a slight chance, that a gcc compiled mod_jk will crash with a Sun compiled httpd. The Sun compiler is free though. I'm just mentioning this, so you know that it would be best if the compiler used for httpd and used for mod_jk are the same or at least close to each other. Thanks for your help. Please let me know of anything else I can provide. I will make updates as new information comes up. Regards, Rainer - To unsubscribe, e-mail:
Re: Apache/mod_jk serves random files from tomcat
On 13.03.2009 18:14, Rainer Jung wrote: Not sure the answer to that. Both were installed by other people, who either don't recall their orgins, or are no longer employed here. I'm working on building the 1.2.27 from source right now. We're x86, not sparc, by the way. OK. For Solaris x86 we never provided bins (I think) Oups, correction, we did. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache/mod_jk serves random files from tomcat
Good to see others are seeing the same problem that’s been driving us crazy and is slowly become a very serious issue. Admittedly, my knowledge on this whole area is limited, but I’ll try my best to provide as much info as possible to help solve the problem. Here are some specifics: Tomcat serves most of the pages, excluding html pages, which there are very few. We have tried extensively to reproduce the problem, but cannot. Restarting either apache or tomcat clears the problem. We have two different environments that are exhibiting the same problem. In both environments, tomcat and apache are on the same machines. Development Solaris 10 Tomcat 5.0.19 Apache 2.0.49 mod_jk2/2.0.4 Web Solaris 10 Tomcat 5.0.19 Apache 2.2.6 mod_jk/1.2.25 Web servers are load balanced using separate machines. These machines have a probe that runs, checking the health of the web servers. The servers are constantly going up and down depending on the random responses. This is normally how we are alerted of the problem, or user input. Problem happens daily on the web servers, maybe once a week in development. I looked for any signs of APR and found none; I don’t think we’re using it. After glancing over past responses, it appears upgrading mod_jk should be the first step, but it doesn’t seem like that was a guaranteed fix for all. Interestingly enough, we’re using two different versions and getting the same problem on both. Any other suggestions? Any additional info I can provide? -- View this message in context: http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p22478781.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache/mod_jk serves random files from tomcat
On 12.03.2009 16:42, SQ wrote: Good to see others are seeing the same problem that’s been driving us crazy and is slowly become a very serious issue. Admittedly, my knowledge on this whole area is limited, but I’ll try my best to provide as much info as possible to help solve the problem. Here are some specifics: Tomcat serves most of the pages, excluding html pages, which there are very few. We have tried extensively to reproduce the problem, but cannot. Restarting either apache or tomcat clears the problem. We have two different environments that are exhibiting the same problem. In both environments, tomcat and apache are on the same machines. Development Solaris 10 Tomcat 5.0.19 Apache 2.0.49 mod_jk2/2.0.4 Web Solaris 10 Tomcat 5.0.19 Apache 2.2.6 mod_jk/1.2.25 Web servers are load balanced using separate machines. These machines have a probe that runs, checking the health of the web servers. The servers are constantly going up and down depending on the random responses. This is normally how we are alerted of the problem, or user input. Problem happens daily on the web servers, maybe once a week in development. I looked for any signs of APR and found none; I don’t think we’re using it. After glancing over past responses, it appears upgrading mod_jk should be the first step, but it doesn’t seem like that was a guaranteed fix for all. Interestingly enough, we’re using two different versions and getting the same problem on both. Any other suggestions? Any additional info I can provide? Jus to make sure, we are talking about the same kind of observation: could you please describe independently, how the observed problem looks like in your case? Since you see the problem with mod_jk2 and with mod_jk I somehow doubt, that it comes form mod_jk (but hey, I'm involved in mod_jk development, so that might simply be defense. What is obvious, your Tomcat is *very* outdated. You are using a no longer supported major version (5.0) and with 5.0 you are using a very old minor version. If you have any chance, upgrade your Tomcat. Apart from that: what else can you tell about the problem? Are there log entries either from mod_jk, Apache httpd or Tomcat associated with these events? Would you be able to snoop traffic between httpd and Tomcat and between httpd and the clients? Where did you get your mod_jk from? How was it build? Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache/mod_jk serves random files from tomcat
Rainer Jung-3 wrote: did you find out in the meantime, whether you were using the tcnative (aka APR) connector? I was certainly using libtcnative, and removed it at the start of the month. I haven't seen enough to definitively say that it solved the problem, but my experience thus far is certainly consistent with such a hypothesis. Cheers! Luke -- View this message in context: http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p22462521.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache/mod_jk serves random files from tomcat
On 11.03.2009 20:19, LukeK wrote: Rainer Jung-3 wrote: did you find out in the meantime, whether you were using the tcnative (aka APR) connector? I was certainly using libtcnative, and removed it at the start of the month. I haven't seen enough to definitively say that it solved the problem, but my experience thus far is certainly consistent with such a hypothesis. Thanks very much for the feedback. Considering the severity of the problem, if you could give us another update at a time you think is appropriate (depending on how often the problem happened before). Would you please be so kind to also tell us, which tcnative version you were using? Thanks again, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache/mod_jk serves random files from tomcat
Hi Yuval, did you find out in the meantime, whether you were using the tcnative (aka APR) connector? Regards, Rainer On 19.02.2009 11:34, Yuval Perlov wrote: Just the swapping responses has me concerned. Thank you so much for the rest of your responses we will put them to good use once we give up on AJP completely. Yuval On Feb 18, 2009, at 8:45 PM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Yuval, On 2/17/2009 1:48 PM, Yuval Perlov wrote: Is APR part of tomcat or apache [httpd]? APR is the Apache Portable Runtime. Technically, it's its own beast and is used by both httpd and Tomcat (optionally). If I am running on linux and have no .so files in my tomcat directory does that mean I have no APR installed? The Tomcat directory isn't the only place .so files could be located. Anywhere in the java.library.path is possible. If you have an AprLifecycleListener configured in your server.xml, then you are attempting to use APR. If you get a message in catalina.out on startup that says something like APR Configured or APR library not found then you have your answer. On a more positive note, we switched to proxy_http (after making the necessary code changes) and everything works now - no more mixed content. Of course we lost a lot of necessary functionality: 1. request.isSecure() doesn't work You can always use https :) 2. we don't know the server name we are hit with (since it is hard coded in httpd.conf) This should be an option in mod_proxy. Is it not? ProxyPreserveHost? 3. we have no access to the source IP (for geo location) Why not use mod_headers to convert the original IP address into an X-Original-IP header. Better yet, use the X-Forwarded-For header that should be set by default by mod_proxy. BTW - Am I the only one that is seriously worried that this kind of problem can even exist on a platform of this maturity? Which problem? The swapping-responses problem or everything else you've outlined about your inadequate configuration? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache/mod_jk serves random files from tomcat
Just the swapping responses has me concerned. Thank you so much for the rest of your responses we will put them to good use once we give up on AJP completely. Yuval On Feb 18, 2009, at 8:45 PM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Yuval, On 2/17/2009 1:48 PM, Yuval Perlov wrote: Is APR part of tomcat or apache [httpd]? APR is the Apache Portable Runtime. Technically, it's its own beast and is used by both httpd and Tomcat (optionally). If I am running on linux and have no .so files in my tomcat directory does that mean I have no APR installed? The Tomcat directory isn't the only place .so files could be located. Anywhere in the java.library.path is possible. If you have an AprLifecycleListener configured in your server.xml, then you are attempting to use APR. If you get a message in catalina.out on startup that says something like APR Configured or APR library not found then you have your answer. On a more positive note, we switched to proxy_http (after making the necessary code changes) and everything works now - no more mixed content. Of course we lost a lot of necessary functionality: 1. request.isSecure() doesn't work You can always use https :) 2. we don't know the server name we are hit with (since it is hard coded in httpd.conf) This should be an option in mod_proxy. Is it not? ProxyPreserveHost? 3. we have no access to the source IP (for geo location) Why not use mod_headers to convert the original IP address into an X-Original-IP header. Better yet, use the X-Forwarded-For header that should be set by default by mod_proxy. BTW - Am I the only one that is seriously worried that this kind of problem can even exist on a platform of this maturity? Which problem? The swapping-responses problem or everything else you've outlined about your inadequate configuration? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkmcV0AACgkQ9CaO5/Lv0PBw4wCgtvTgf5Jy6z30u9Z3z/8M9ViN stwAn1urDcjts1xtPvSMiSuL00jEMYPV =/Ge+ -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache/mod_jk serves random files from tomcat
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Yuval, On 2/17/2009 1:48 PM, Yuval Perlov wrote: Is APR part of tomcat or apache [httpd]? APR is the Apache Portable Runtime. Technically, it's its own beast and is used by both httpd and Tomcat (optionally). If I am running on linux and have no .so files in my tomcat directory does that mean I have no APR installed? The Tomcat directory isn't the only place .so files could be located. Anywhere in the java.library.path is possible. If you have an AprLifecycleListener configured in your server.xml, then you are attempting to use APR. If you get a message in catalina.out on startup that says something like APR Configured or APR library not found then you have your answer. On a more positive note, we switched to proxy_http (after making the necessary code changes) and everything works now - no more mixed content. Of course we lost a lot of necessary functionality: 1. request.isSecure() doesn't work You can always use https :) 2. we don't know the server name we are hit with (since it is hard coded in httpd.conf) This should be an option in mod_proxy. Is it not? ProxyPreserveHost? 3. we have no access to the source IP (for geo location) Why not use mod_headers to convert the original IP address into an X-Original-IP header. Better yet, use the X-Forwarded-For header that should be set by default by mod_proxy. BTW - Am I the only one that is seriously worried that this kind of problem can even exist on a platform of this maturity? Which problem? The swapping-responses problem or everything else you've outlined about your inadequate configuration? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkmcV0AACgkQ9CaO5/Lv0PBw4wCgtvTgf5Jy6z30u9Z3z/8M9ViN stwAn1urDcjts1xtPvSMiSuL00jEMYPV =/Ge+ -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache/mod_jk serves random files from tomcat
Hi, We had similar behaviour on our server, apache in the front serving tomcat on SSL, mostly servlets. First off, this is what we installed: Tomcat 5.5.27 and Apache 2.2.11 with mod_jk 1.2.26. The server is a 64 bit version of Windows Server 2008 (2 CPUs). We got some strange problems with constant high CPU usage. A thread suggested to change the tcnative.dll to an older version since there was a bug in that, so we did and changed to version 1.1.6.0 (somewhere in the process we have misplaced the tomcat version this file came with..) , and that got rid of the CPU usage problems. But then another problem arose. After uploading documents into our system, we could not download them successfully - this was first seen with PDF files. And the larger the file, the more likely the problem. Then we noticed that going directly towards Tomcat (on 8080) worked like a charm, so did accessing static files under apache - but when trying the same static files under tomcat - going through apache - it failed again. We have tried different versions of mod_jk with no success, and experimented with mod_expires settings etc. Then we got reports of issues where large servlet generated reports got messed up, and it seemed the stream had just left out parts of the produced HTML code, only to continue further down in the expected report. And finally, to top it, we also got data from one user showing up at another user! After having no success we finally decided to try to remove the tcnative DLL, and what do you know, it worked! No more problems with PDFs, and so far, no more cut or mixed servlet response! So what I want to know is, what could have caused this? My knowledge of both Apache server and Tomcat is rather limited, so I hope someone can come up with a good answer :) Yuval Perlov wrote: We started restarting apache on a regular basis but if a user is in mid request (consider a user that just filled a big form and is upload a file). I moved all static content to apache so tomcat is now only delivering the actual jsp file. The result was that the mix up took longer to appear, however when it did USERS STARTED SEEING EACH OTHERS DATA!!! (before that, the mixup was usually with images etc just because there are more of them). I am actually amazed that this can even happen in such a mature version and that such a small number of us are experiencing it. This is slowly killing our project. Trying to move to proxy_ajp did not help which makes the whole thing even more mysterious - these are two separate code bases, no? (BTW - are mod_jk developers reading this?) We are contemplating two approaches: 1) moving to proxy_http. My only concern is that this won't help - maybe the problem is unrelated to AJP? Upgrading has helped some users but not all and the problem exists in both mod_jk and proxy_ajp. 2) getting rid of apache and moving tomcat to the front (much harder to configure but ensures we are rid of this problem). Any Thoughts? Regards, Yuval Perlov On Feb 5, 2009, at 11:27 PM, LukeK wrote: JohnHardin wrote: * Have others (that now seem to be fixed) gotten things to work by updating to the latest mod_jk (1.2.27)? I suspect that it's related to 1.2.27 - I have been playing around with older versions. .24 and .25 have had issues forwarding certain request headers, but so far .26 seems to be working OK. Is periodically restarting apache a suitable (if not hackish) work- around until we can get our production environment upgraded? That'd be my fallback position. Cheers! Luke -- View this message in context: http://www.nabble.com/Apache-mod_jk- serves-random-files-from-tomcat-tp18385568p21861548.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- View this message in context: http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p22059232.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache/mod_jk serves random files from tomcat
Is APR part of tomcat or apache? If I am running on linux and have no .so files in my tomcat directory does that mean I have no APR installed? On a more positive note, we switched to proxy_http (after making the necessary code changes) and everything works now - no more mixed content. Of course we lost a lot of necessary functionality: 1. request.isSecure() doesn't work 2. we don't know the server name we are hit with (since it is hard coded in httpd.conf) 3. we have no access to the source IP (for geo location) 4. We had to some make all client redirection code use the full URL with the server name - turns out client redirect uses the server name from the request so it tries to hit the 8080 port (tomcat) instead of 80 (httpd). BTW - Am I the only one that is seriously worried that this kind of problem can even exist on a platform of this maturity? Yuval Perlov www.r-u-on.com On Feb 17, 2009, at 1:38 AM, dave smith wrote: Sorry for not providing an update sooner. I disabled the APR and the problem went away. On 2/12/09, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Yuval, On 2/12/2009 3:12 AM, Yuval Perlov wrote: I actually upgraded from mod_jk 1.2.26 to 27 to try and make the problem go away. Ha! Okay. Sorry for a bad tip. ;) So, I'm definitely not going to be able to help you from here on out, but I know that folks like Rainer and Mladen could use some more information, so I'll go ahead and ask for some. The mixup occurs only in tomcat originated data - the static stuff coming from httpd stays fine. Good to know. Moreover, in the past I had it setup so the static stuff came from tomcat as well. This naturally resulted in significantly more hits between apache and tomcat which made the problem appear much faster (hence my theory that some resource is being depleted over time). Is this something you can reproduce reliably in a test environment? Does it require heavy load in order for this behavior to manifest itself? Or, is it just after 5M requests everything goes to hell? I'm wondering if concurrency is the problem or maybe something silly like logging or maintaining worker status that somehow corrupts something. It's very odd that responses would be crossed. I don't think any of that stuff is shared between threads/processes in mod_jk/httpd, but I suppose when you overwrite memory (which is the only explanation I can think of), you can't really expect the program to operate properly. Oh, are you using worker or prefork MPM? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkmUplsACgkQ9CaO5/Lv0PANwQCeM7IEsDUu+o8cKjZP3kxAZgXP 7g4AoLyLW2cvmLC7AGGJnEf8jHBzNBvM =E4BT -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache/mod_jk serves random files from tomcat
Sorry for not providing an update sooner. I disabled the APR and the problem went away. On 2/12/09, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Yuval, On 2/12/2009 3:12 AM, Yuval Perlov wrote: I actually upgraded from mod_jk 1.2.26 to 27 to try and make the problem go away. Ha! Okay. Sorry for a bad tip. ;) So, I'm definitely not going to be able to help you from here on out, but I know that folks like Rainer and Mladen could use some more information, so I'll go ahead and ask for some. The mixup occurs only in tomcat originated data - the static stuff coming from httpd stays fine. Good to know. Moreover, in the past I had it setup so the static stuff came from tomcat as well. This naturally resulted in significantly more hits between apache and tomcat which made the problem appear much faster (hence my theory that some resource is being depleted over time). Is this something you can reproduce reliably in a test environment? Does it require heavy load in order for this behavior to manifest itself? Or, is it just after 5M requests everything goes to hell? I'm wondering if concurrency is the problem or maybe something silly like logging or maintaining worker status that somehow corrupts something. It's very odd that responses would be crossed. I don't think any of that stuff is shared between threads/processes in mod_jk/httpd, but I suppose when you overwrite memory (which is the only explanation I can think of), you can't really expect the program to operate properly. Oh, are you using worker or prefork MPM? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkmUplsACgkQ9CaO5/Lv0PANwQCeM7IEsDUu+o8cKjZP3kxAZgXP 7g4AoLyLW2cvmLC7AGGJnEf8jHBzNBvM =E4BT -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache/mod_jk serves random files from tomcat
I actually upgraded from mod_jk 1.2.26 to 27 to try and make the problem go away. I see the mixup in the file sizes so thought a trace was not necessary. The mixup occurs only in tomcat originated data - the static stuff coming from httpd stays fine. Moreover, in the past I had it setup so the static stuff came from tomcat as well. This naturally resulted in significantly more hits between apache and tomcat which made the problem appear much faster (hence my theory that some resource is being depleted over time). Yuval On Feb 11, 2009, at 3:44 PM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Yuval, On 2/11/2009 1:56 AM, Yuval Perlov wrote: What leads me to believe this is unrelated to my application code is that restarting apache makes the problem go away. So, when your site goes crazy, a simple httpd-bounce does the trick? No Tomcat restart or anything required? Existing users and sessions are all preserved and pretty much the problem just magically goes away? Crazy. I see that you are using httpd 2.2.10. Have you tried downgrading to 2.0.x to see if that helps? I've heard some folks having trouble with mod_jk 1.2.27, so you might try downgrading to 1.2.26 unless something vital is in the .27 release that you need. Those are easier fixes than switching to proxy_http or removing httpd altogether. If you watch the network traffic with a TCP sniffer like wireshark, does it look like request A results in response B instead of (expected) response A? When the server goes crazy, can you start sending TRACE requests to see if those get mixed-up? Does all traffic get jumbled, or just the stuff bound for Tomcat? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkmS1lgACgkQ9CaO5/Lv0PBNTwCghqlzDnFDppy0WmgHGTdKjMoQ czQAnijlks4T6XAM72WuC3EgMN1NB+0Q =xzLb -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache/mod_jk serves random files from tomcat
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Yuval, On 2/12/2009 3:12 AM, Yuval Perlov wrote: I actually upgraded from mod_jk 1.2.26 to 27 to try and make the problem go away. Ha! Okay. Sorry for a bad tip. ;) So, I'm definitely not going to be able to help you from here on out, but I know that folks like Rainer and Mladen could use some more information, so I'll go ahead and ask for some. The mixup occurs only in tomcat originated data - the static stuff coming from httpd stays fine. Good to know. Moreover, in the past I had it setup so the static stuff came from tomcat as well. This naturally resulted in significantly more hits between apache and tomcat which made the problem appear much faster (hence my theory that some resource is being depleted over time). Is this something you can reproduce reliably in a test environment? Does it require heavy load in order for this behavior to manifest itself? Or, is it just after 5M requests everything goes to hell? I'm wondering if concurrency is the problem or maybe something silly like logging or maintaining worker status that somehow corrupts something. It's very odd that responses would be crossed. I don't think any of that stuff is shared between threads/processes in mod_jk/httpd, but I suppose when you overwrite memory (which is the only explanation I can think of), you can't really expect the program to operate properly. Oh, are you using worker or prefork MPM? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkmUplsACgkQ9CaO5/Lv0PANwQCeM7IEsDUu+o8cKjZP3kxAZgXP 7g4AoLyLW2cvmLC7AGGJnEf8jHBzNBvM =E4BT -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache/mod_jk serves random files from tomcat
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Yuval, On 2/11/2009 1:56 AM, Yuval Perlov wrote: What leads me to believe this is unrelated to my application code is that restarting apache makes the problem go away. So, when your site goes crazy, a simple httpd-bounce does the trick? No Tomcat restart or anything required? Existing users and sessions are all preserved and pretty much the problem just magically goes away? Crazy. I see that you are using httpd 2.2.10. Have you tried downgrading to 2.0.x to see if that helps? I've heard some folks having trouble with mod_jk 1.2.27, so you might try downgrading to 1.2.26 unless something vital is in the .27 release that you need. Those are easier fixes than switching to proxy_http or removing httpd altogether. If you watch the network traffic with a TCP sniffer like wireshark, does it look like request A results in response B instead of (expected) response A? When the server goes crazy, can you start sending TRACE requests to see if those get mixed-up? Does all traffic get jumbled, or just the stuff bound for Tomcat? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkmS1lgACgkQ9CaO5/Lv0PBNTwCghqlzDnFDppy0WmgHGTdKjMoQ czQAnijlks4T6XAM72WuC3EgMN1NB+0Q =xzLb -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache/mod_jk serves random files from tomcat
We started restarting apache on a regular basis but if a user is in mid request (consider a user that just filled a big form and is upload a file). I moved all static content to apache so tomcat is now only delivering the actual jsp file. The result was that the mix up took longer to appear, however when it did USERS STARTED SEEING EACH OTHERS DATA!!! (before that, the mixup was usually with images etc just because there are more of them). I am actually amazed that this can even happen in such a mature version and that such a small number of us are experiencing it. This is slowly killing our project. Trying to move to proxy_ajp did not help which makes the whole thing even more mysterious - these are two separate code bases, no? (BTW - are mod_jk developers reading this?) We are contemplating two approaches: 1) moving to proxy_http. My only concern is that this won't help - maybe the problem is unrelated to AJP? Upgrading has helped some users but not all and the problem exists in both mod_jk and proxy_ajp. 2) getting rid of apache and moving tomcat to the front (much harder to configure but ensures we are rid of this problem). Any Thoughts? Regards, Yuval Perlov On Feb 5, 2009, at 11:27 PM, LukeK wrote: JohnHardin wrote: * Have others (that now seem to be fixed) gotten things to work by updating to the latest mod_jk (1.2.27)? I suspect that it's related to 1.2.27 - I have been playing around with older versions. .24 and .25 have had issues forwarding certain request headers, but so far .26 seems to be working OK. Is periodically restarting apache a suitable (if not hackish) work- around until we can get our production environment upgraded? That'd be my fallback position. Cheers! Luke -- View this message in context: http://www.nabble.com/Apache-mod_jk- serves-random-files-from-tomcat-tp18385568p21861548.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache/mod_jk serves random files from tomcat
Yuval Perlov wrote: [...] 2) getting rid of apache and moving tomcat to the front (much harder to configure but ensures we are rid of this problem). This being the Tomcat forum, and as these things go, I am sure you are going to get some ringing endorsements for that. But I am less sure they will be unbiased. You should balance that with a similar question on the Apache httpd forum. And yes, the mod_jk developers do lurk around here. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache/mod_jk serves random files from tomcat
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Yuval, On 2/10/2009 3:44 PM, Yuval Perlov wrote: We started restarting apache on a regular basis but if a user is in mid request (consider a user that just filled a big form and is upload a file). So it appears that Apache is, over time, losing track of user identities? That seems odd since neither mod_jk nor Apache httpd actually do anything but forward the identity information from the browser to Tomcat. Either an HTTP cookie or a URL parameter is used to identify sessions, and both are provided with every request. Do you have unusually long URLs? Unusually long request bodies? I'm just trying to think of why any data would be mixed-up. Does this happen seemingly randomly, or only for certain pages on your site? Certain source IP addresses? We had some users that were getting all messed up before we recognized that they were doing through google's cache which was seriously confusing just about everything. Fortunately, we could see from our server logs that some requests came from the /real/ remote user and others came from google's domain. Otherwise, all I can think of is that you have some bug in your application shrug. How are you doing authentication? How about user identification - aside from relying on session data in Tomcat. We are contemplating two approaches: 1) moving to proxy_http. My only concern is that this won't help - maybe the problem is unrelated to AJP? Upgrading has helped some users but not all and the problem exists in both mod_jk and proxy_ajp. Trying mod_proxy_http will certainly give you more information. Can you reproduce this problem in a safe environment? 2) getting rid of apache and moving tomcat to the front (much harder to configure but ensures we are rid of this problem). Are you /sure/ that a Tomcat-only setup doesn't exhibit this problem? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkmSOi0ACgkQ9CaO5/Lv0PBpiwCdH2pRuaVP7TRl7E6tOqZbkUQM yuUAniM9m8+Mo9aWiu2G8XQcZjXf2W/M =l0Xk -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache/mod_jk serves random files from tomcat
Thanks! The problem as far as I can tell is a simple mixup of http requests so user identities don't play into this. It might look like it since user A is getting the results of user B but as far as session management goes it is unaffected by this. The URLs are very short. This happens at random. This only starts to happen after the server has been running for a while - it's as if some resource is being consumed and once it's done this problem starts emerging. When it starts happening it happens to all users. What leads me to believe this is unrelated to my application code is that restarting apache makes the problem go away. User data is managed on the session object and I am not interfering with it in any way (no direct cookie code). This is also the reason I believe tomcat only will work. Also we have been running for sometime in a tomcat only mode and never had this problem (which is not definite evidence, i know). The reason I am not jumping to proxy_http is that the application is currently using IP geo location which I suspect will not be available once we are behind a http proxy. We will be shutting off this functionality just so we can switch to proxy_http but it takes a few day to test. Yuval On Feb 11, 2009, at 4:38 AM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Yuval, On 2/10/2009 3:44 PM, Yuval Perlov wrote: We started restarting apache on a regular basis but if a user is in mid request (consider a user that just filled a big form and is upload a file). So it appears that Apache is, over time, losing track of user identities? That seems odd since neither mod_jk nor Apache httpd actually do anything but forward the identity information from the browser to Tomcat. Either an HTTP cookie or a URL parameter is used to identify sessions, and both are provided with every request. Do you have unusually long URLs? Unusually long request bodies? I'm just trying to think of why any data would be mixed-up. Does this happen seemingly randomly, or only for certain pages on your site? Certain source IP addresses? We had some users that were getting all messed up before we recognized that they were doing through google's cache which was seriously confusing just about everything. Fortunately, we could see from our server logs that some requests came from the /real/ remote user and others came from google's domain. Otherwise, all I can think of is that you have some bug in your application shrug. How are you doing authentication? How about user identification - aside from relying on session data in Tomcat. We are contemplating two approaches: 1) moving to proxy_http. My only concern is that this won't help - maybe the problem is unrelated to AJP? Upgrading has helped some users but not all and the problem exists in both mod_jk and proxy_ajp. Trying mod_proxy_http will certainly give you more information. Can you reproduce this problem in a safe environment? 2) getting rid of apache and moving tomcat to the front (much harder to configure but ensures we are rid of this problem). Are you /sure/ that a Tomcat-only setup doesn't exhibit this problem? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkmSOi0ACgkQ9CaO5/Lv0PBpiwCdH2pRuaVP7TRl7E6tOqZbkUQM yuUAniM9m8+Mo9aWiu2G8XQcZjXf2W/M =l0Xk -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache/mod_jk serves random files from tomcat
LukeK wrote: yuvalperlov wrote: I am having the exact same symptoms with the latest versions of everything: Fedora 10 Tomcat 6.0.18 Apache Apache/2.2.10 mod_jk-1.2.27 (and the same problem with the built-in mod_proxy_ajp). It takes a day or so for the problem to start but once it does it happens more frequently - resources get mixed up. Tomcat logs show that the right resources are loaded in response to each call but on the browser end you can see images loaded in the wrong place. Restarting apache resets the problem for another day or so. This describes my issue as well. SuSE 10.3 64-bit, running Apache 2.2.11, Tomcat 6.0.16, APR 1.3.4 and mod_jk/1.2.27. I have downgraded mod_jk to 1.2.24 to see if this makes a difference. We, too, are experiencing similar issues: CentOS release 5, Apache 2.2.3, Tomcat 6.0.14, and mod_jk 1.2.23. Comparing tomcat and apache logs show discrepancies between tomcat's and apache's content sizes for given requests. We're clearly not up-to-date on any of these components, but we're trying to find the minimum we have to update in order to resolve this issue. * Have others (that now seem to be fixed) gotten things to work by updating to the latest mod_jk (1.2.27)? * Has anyone discovered a way to reproduce this issue? * Is periodically restarting apache a suitable (if not hackish) work-around until we can get our production environment upgraded? Thoughts? Thanks in advance for your assistance. -John Hardin -- View this message in context: http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p21861245.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache/mod_jk serves random files from tomcat
JohnHardin wrote: * Have others (that now seem to be fixed) gotten things to work by updating to the latest mod_jk (1.2.27)? I suspect that it's related to 1.2.27 - I have been playing around with older versions. .24 and .25 have had issues forwarding certain request headers, but so far .26 seems to be working OK. Is periodically restarting apache a suitable (if not hackish) work-around until we can get our production environment upgraded? That'd be my fallback position. Cheers! Luke -- View this message in context: http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p21861548.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache/mod_jk serves random files from tomcat
yuvalperlov wrote: I am having the exact same symptoms with the latest versions of everything: Fedora 10 Tomcat 6.0.18 Apache Apache/2.2.10 mod_jk-1.2.27 (and the same problem with the built-in mod_proxy_ajp). It takes a day or so for the problem to start but once it does it happens more frequently - resources get mixed up. Tomcat logs show that the right resources are loaded in response to each call but on the browser end you can see images loaded in the wrong place. Restarting apache resets the problem for another day or so. This describes my issue as well. SuSE 10.3 64-bit, running Apache 2.2.11, Tomcat 6.0.16, APR 1.3.4 and mod_jk/1.2.27. I have downgraded mod_jk to 1.2.24 to see if this makes a difference. Cheers! Luke -- View this message in context: http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p21812944.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache/mod_jk serves random files from tomcat
Trying to revive this thread: I am having the exact same symptoms with the latest versions of everything: Fedora 10 Tomcat 6.0.18 Apache Apache/2.2.10 mod_jk-1.2.27 (and the same problem with the built-in mod_proxy_ajp). It takes a day or so for the problem to start but once it does it happens more frequently - resources get mixed up. Tomcat logs show that the right resources are loaded in response to each call but on the browser end you can see images loaded in the wrong place. Restarting apache resets the problem for another day or so. Has this been resolved for all who posted? Regards, Yuval Perlov Rainer Jung-3 wrote: Jakob Ericsson schrieb: On Thu, Nov 13, 2008 at 8:17 PM, Rainer Jung rainer.j...@kippdata.de wrote: Jakob Ericsson schrieb: Upgrading to latest version of mod_jk solves the problem. As I said before, this on Windows 2003 Server running both httpd (2.0.59) and tomcat (6.0.13) on the same machine. We upgraded all production machines this morning. Problem is in mod_jk 1.2.22 and is at least and fixed in 1.2.27. We did not do any other fixes to the environment when upgrading mod_jk. Hopefully this will help other people experiencing the same problem. Thanks a lot for reporting back. Let us know, in case the problem shows up again (hopefully not). We had a peak yesterday on our machines and no problems reported. Looks very promising. Do you know (out of curiosity) know which bug in mod_jk that probably caused the problem? Unfortunately I have no idea. As already indicated, there wasn't any change I'm aware of, which should have fixed such a problem on Windows. Regards, Rainer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- View this message in context: http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p21599110.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache/mod_jk serves random files from tomcat
On Thu, Nov 13, 2008 at 8:17 PM, Rainer Jung [EMAIL PROTECTED] wrote: Jakob Ericsson schrieb: Upgrading to latest version of mod_jk solves the problem. As I said before, this on Windows 2003 Server running both httpd (2.0.59) and tomcat (6.0.13) on the same machine. We upgraded all production machines this morning. Problem is in mod_jk 1.2.22 and is at least and fixed in 1.2.27. We did not do any other fixes to the environment when upgrading mod_jk. Hopefully this will help other people experiencing the same problem. Thanks a lot for reporting back. Let us know, in case the problem shows up again (hopefully not). We had a peak yesterday on our machines and no problems reported. Looks very promising. Do you know (out of curiosity) know which bug in mod_jk that probably caused the problem? Regards, Rainer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Apache/mod_jk serves random files from tomcat
Jakob Ericsson schrieb: On Thu, Nov 13, 2008 at 8:17 PM, Rainer Jung [EMAIL PROTECTED] wrote: Jakob Ericsson schrieb: Upgrading to latest version of mod_jk solves the problem. As I said before, this on Windows 2003 Server running both httpd (2.0.59) and tomcat (6.0.13) on the same machine. We upgraded all production machines this morning. Problem is in mod_jk 1.2.22 and is at least and fixed in 1.2.27. We did not do any other fixes to the environment when upgrading mod_jk. Hopefully this will help other people experiencing the same problem. Thanks a lot for reporting back. Let us know, in case the problem shows up again (hopefully not). We had a peak yesterday on our machines and no problems reported. Looks very promising. Do you know (out of curiosity) know which bug in mod_jk that probably caused the problem? Unfortunately I have no idea. As already indicated, there wasn't any change I'm aware of, which should have fixed such a problem on Windows. Regards, Rainer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Apache/mod_jk serves random files from tomcat
Hi, Upgrading to latest version of mod_jk solves the problem. As I said before, this on Windows 2003 Server running both httpd (2.0.59) and tomcat (6.0.13) on the same machine. We upgraded all production machines this morning. Problem is in mod_jk 1.2.22 and is at least and fixed in 1.2.27. We did not do any other fixes to the environment when upgrading mod_jk. Hopefully this will help other people experiencing the same problem. /Jakob On Wed, Nov 12, 2008 at 9:29 PM, Jakob Ericsson [EMAIL PROTECTED] wrote: On Wed, Nov 12, 2008 at 12:21 PM, Rainer Jung [EMAIL PROTECTED] wrote: Jakob Ericsson schrieb: -- Jakob Ericsson +46 704 533 627 11 nov 2008 kl. 22.37 skrev Christopher Schultz [EMAIL PROTECTED]: Jakob, Jakob Ericsson wrote: We are also experiencing this problem. Our setup is running Windows 2003 Server with Apache 2.0.59 (no prefork), mod_jk 1.2.22 and Apache Tomcat/6.0.13. Will upgrading to latest mod_proxy_ajp in Apache httpd 2.2.X solve this problem? If you're willing to donate some time to this, please stick with mod_jk and work with Rainer/Mladen to fix whatever might be wrong. Upgrading to the latest mod_jk is a definite requirement before you continue testing. We are in the process of updating our system to the latest mod_jk. I will give an update if this solves our problem. The underlying problem is hopefully the missed multi thread flag in mod_jk compile. Does anyone know which issue this is in bugzilla? On Woindows I would not expext that to be the problem. mod_jk tries to determine automatically during compile time, whether a multi-threaded environment gets used and then enables thread safe mutexes. On some more exotic platforms like AIX this determination was broken for some time, so some versions ago we decided to compile thread-safe by default and add a new flag to configure to allow compiling without thread support if you give the flag explicitely. On Windows it should have been always thread safe. We have updated a couple of our machines in the production environment to 1.2.27 and it looks quite good. Tomorrow, we will upgrade all machines and hopefully the problem will disappear. I´ll keep you posted. Nevertheless I appreciate you update first. In case you can reproduce the behaviour, it would be extremely helpful to have a JK log file with debug log level. Unfortunately that is a problem for production because of the high log volume. So if you can reproduce it easily, or when only reproducibale under load othen on a test system, a debug level JK log would be extremely helpful. You can make that available also only privately. We have tried to replicate the problem in a test environment but all attempts have been unsuccessful. As you probably understand, we can not enable debug log in the production environment. The thing we see is basically the same as people have said in this thread before. First request's response is served to the second request's response. :-) And it only happens when it is one the same thread in tomcat. I took your mail as a reminder to ask Tim Redding again for his log but did not yet get any response. Regards, Rainer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- Jakob Ericsson, JAKERI AB Tel. +46 704 533 627
Re: Apache/mod_jk serves random files from tomcat
Jakob Ericsson schrieb: Upgrading to latest version of mod_jk solves the problem. As I said before, this on Windows 2003 Server running both httpd (2.0.59) and tomcat (6.0.13) on the same machine. We upgraded all production machines this morning. Problem is in mod_jk 1.2.22 and is at least and fixed in 1.2.27. We did not do any other fixes to the environment when upgrading mod_jk. Hopefully this will help other people experiencing the same problem. Thanks a lot for reporting back. Let us know, in case the problem shows up again (hopefully not). Regards, Rainer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Apache/mod_jk serves random files from tomcat
-- Jakob Ericsson +46 704 533 627 11 nov 2008 kl. 22.37 skrev Christopher Schultz [EMAIL PROTECTED] : -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jakob, Jakob Ericsson wrote: We are also experiencing this problem. Our setup is running Windows 2003 Server with Apache 2.0.59 (no prefork), mod_jk 1.2.22 and Apache Tomcat/6.0.13. Will upgrading to latest mod_proxy_ajp in Apache httpd 2.2.X solve this problem? If you're willing to donate some time to this, please stick with mod_jk and work with Rainer/Mladen to fix whatever might be wrong. Upgrading to the latest mod_jk is a definite requirement before you continue testing. We are in the process of updating our system to the latest mod_jk. I will give an update if this solves our problem. The underlying problem is hopefully the missed multi thread flag in mod_jk compile. Does anyone know which issue this is in bugzilla? Switching to mod_proxy_ajp will switch your configuration around and you'll be using a different package. In that case, your problem may go away (yay!) but the underlying problem might not get fixed (boo!). - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkkZ+yIACgkQ9CaO5/Lv0PCUOQCdHZZH9Fzgjg+brfipzDiSv9KN qdwAnid9AjHkKA0h+VZRZdxNGAB8xSuy =T/6T -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Apache/mod_jk serves random files from tomcat
Jakob Ericsson schrieb: -- Jakob Ericsson +46 704 533 627 11 nov 2008 kl. 22.37 skrev Christopher Schultz [EMAIL PROTECTED]: Jakob, Jakob Ericsson wrote: We are also experiencing this problem. Our setup is running Windows 2003 Server with Apache 2.0.59 (no prefork), mod_jk 1.2.22 and Apache Tomcat/6.0.13. Will upgrading to latest mod_proxy_ajp in Apache httpd 2.2.X solve this problem? If you're willing to donate some time to this, please stick with mod_jk and work with Rainer/Mladen to fix whatever might be wrong. Upgrading to the latest mod_jk is a definite requirement before you continue testing. We are in the process of updating our system to the latest mod_jk. I will give an update if this solves our problem. The underlying problem is hopefully the missed multi thread flag in mod_jk compile. Does anyone know which issue this is in bugzilla? On Woindows I would not expext that to be the problem. mod_jk tries to determine automatically during compile time, whether a multi-threaded environment gets used and then enables thread safe mutexes. On some more exotic platforms like AIX this determination was broken for some time, so some versions ago we decided to compile thread-safe by default and add a new flag to configure to allow compiling without thread support if you give the flag explicitely. On Windows it should have been always thread safe. Nevertheless I appreciate you update first. In case you can reproduce the behaviour, it would be extremely helpful to have a JK log file with debug log level. Unfortunately that is a problem for production because of the high log volume. So if you can reproduce it easily, or when only reproducibale under load othen on a test system, a debug level JK log would be extremely helpful. You can make that available also only privately. I took your mail as a reminder to ask Tim Redding again for his log but did not yet get any response. Regards, Rainer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Apache/mod_jk serves random files from tomcat
On Wed, Nov 12, 2008 at 12:21 PM, Rainer Jung [EMAIL PROTECTED] wrote: Jakob Ericsson schrieb: -- Jakob Ericsson +46 704 533 627 11 nov 2008 kl. 22.37 skrev Christopher Schultz [EMAIL PROTECTED]: Jakob, Jakob Ericsson wrote: We are also experiencing this problem. Our setup is running Windows 2003 Server with Apache 2.0.59 (no prefork), mod_jk 1.2.22 and Apache Tomcat/6.0.13. Will upgrading to latest mod_proxy_ajp in Apache httpd 2.2.X solve this problem? If you're willing to donate some time to this, please stick with mod_jk and work with Rainer/Mladen to fix whatever might be wrong. Upgrading to the latest mod_jk is a definite requirement before you continue testing. We are in the process of updating our system to the latest mod_jk. I will give an update if this solves our problem. The underlying problem is hopefully the missed multi thread flag in mod_jk compile. Does anyone know which issue this is in bugzilla? On Woindows I would not expext that to be the problem. mod_jk tries to determine automatically during compile time, whether a multi-threaded environment gets used and then enables thread safe mutexes. On some more exotic platforms like AIX this determination was broken for some time, so some versions ago we decided to compile thread-safe by default and add a new flag to configure to allow compiling without thread support if you give the flag explicitely. On Windows it should have been always thread safe. We have updated a couple of our machines in the production environment to 1.2.27 and it looks quite good. Tomorrow, we will upgrade all machines and hopefully the problem will disappear. I´ll keep you posted. Nevertheless I appreciate you update first. In case you can reproduce the behaviour, it would be extremely helpful to have a JK log file with debug log level. Unfortunately that is a problem for production because of the high log volume. So if you can reproduce it easily, or when only reproducibale under load othen on a test system, a debug level JK log would be extremely helpful. You can make that available also only privately. We have tried to replicate the problem in a test environment but all attempts have been unsuccessful. As you probably understand, we can not enable debug log in the production environment. The thing we see is basically the same as people have said in this thread before. First request's response is served to the second request's response. :-) And it only happens when it is one the same thread in tomcat. I took your mail as a reminder to ask Tim Redding again for his log but did not yet get any response. Regards, Rainer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- Jakob Ericsson, JAKERI AB Tel. +46 704 533 627
Re: Apache/mod_jk serves random files from tomcat
Hi, We are also experiencing this problem. Our setup is running Windows 2003 Server with Apache 2.0.59 (no prefork), mod_jk 1.2.22 and Apache Tomcat/6.0.13. Will upgrading to latest mod_proxy_ajp in Apache httpd 2.2.X solve this problem? How did other people solve this problem? Regards, Jakob Ericsson Rainer Jung-3 wrote: Hi Tim, Tim Redding schrieb: Hi, We are experiencing intermittent problems with a particular site that is not returning the correct file that is requested. For instance if we request the index.html file we actually get a css file or even an image. From the apache access log you can see that the size of the index.html file grows on the second request. This is because a gif was actually returned. XXX.XXX.XXX.130 - - [10/Jul/2008:15:10:39 +0100] GET /index.html HTTP/1.1 200 1068 XXX.XXX.XXX.130 - - [10/Jul/2008:15:13:10 +0100] GET /index.html HTTP/1.1 200 9526 XXX.XXX.XXX.130 - - [10/Jul/2008:15:13:48 +0100] GET /index.html HTTP/1.1 200 1086 This is pretty serious (I assume that 1086!=1968 was a typo). No error messages are logged in the mode_jk.log file. We have Apache/2.2.3 on the front on a Tomcat 6.0.16 server with mod_jk (version unknown but fairly recent). We have all assets in our war file. When we hit Tomcat directly on port 8080 it serves the correct file. And to fix the problem an apache restart seems to sort things out. On this server with have 2 vhosts. One is a simple nothing fancy static site and the other forwards everything to our Tomcat server. Below I've included our mod_jk config and a snippet of our httpd.conf. Any ideas or things to try would be most appreciated. What's you platform and which httpd MPM (prefork orworker or something else) do you use? For some platforms (e.g. AIX) the detection of multi-threading in httpd during mpod_jk build-time was broken. Starting with 1.2.24 we build always including multi-thread support unless explicitely stated via a configure option. If you 1.2.23 build is not thread safe, but your httpd uses threads (like with worker mpm), then such trouble is possible, although more likely you would see crashes etc. For most platforms like Linux and Solaris the threading detection was OK already before 1.2.24. Another possible (but not very likely) cause could be bug 44494 of Tomcat 6.0.16 which under certain circumstances could leave data in the request object after request handling completed. You could try either downgrding to 6.0.15 or upgrading to the soon to be expected 6.0.17. I would also add the access log on the Tomcat side. If you find the same phenomenon there, then it's unlikely, that httpd/mod_jk are responsible and the reason should be inside Tomcat or the webapp. Can you reproduce the problem on a test system? Regards, Rainer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- View this message in context: http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p20435642.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Apache/mod_jk serves random files from tomcat
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jakob, Jakob Ericsson wrote: We are also experiencing this problem. Our setup is running Windows 2003 Server with Apache 2.0.59 (no prefork), mod_jk 1.2.22 and Apache Tomcat/6.0.13. Will upgrading to latest mod_proxy_ajp in Apache httpd 2.2.X solve this problem? If you're willing to donate some time to this, please stick with mod_jk and work with Rainer/Mladen to fix whatever might be wrong. Upgrading to the latest mod_jk is a definite requirement before you continue testing. Switching to mod_proxy_ajp will switch your configuration around and you'll be using a different package. In that case, your problem may go away (yay!) but the underlying problem might not get fixed (boo!). - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkkZ+yIACgkQ9CaO5/Lv0PCUOQCdHZZH9Fzgjg+brfipzDiSv9KN qdwAnid9AjHkKA0h+VZRZdxNGAB8xSuy =T/6T -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Apache/mod_jk serves random files from tomcat
dave.smith wrote: As I mentioned upgrading to mod_jk 1.2.26 was very easy. Unfortunately, Tomcat is now crashing with An unexpected error has been detected by HotSpot Virtual Machine. # SIGSEGV (0xb) at pc=0xb7aeaf7b, pid=19887, tid=2991246224 # # Java VM: Java HotSpot(TM) Client VM (1.5.0_12-b04 mixed mode, sharing) # Problematic frame: # V [libjvm.so+0x2ccf7b] # There's a lot more. Also, here are the jvm args: jvm_args: -Djava.library.path=/usr/local/apr/lib -Xms512m -Xmx512m -XX:PermSize=256m -XX:MaxPermSize=256m -Djava.util.logging.manager=org.apache.juli.ClassLo aderLogManager -Djava.util.logging.config.file=/usr/share/tomcat5/conf/logging.properties -Djava.endorsed.dirs=/usr/share/tomcat5/common/endorsed -Dcatalina. base=/usr/share/tomcat5 -Dcatalina.home=/usr/share/tomcat5 -Djava.io.tmpdir=/usr/share/tomcat5/temp java_command: org.apache.catalina.startup.Bootstrap start Do you believe these are related to the mod_jk upgrade? It's happened twice (once on each server) since the upgrade that was made 2 weeks ago. Please, let me know if you need anymore information. Thanks, Dave I don't know of any similar case. The full hot spot error file would be useful though. It e.g. contains a stack of the thread were the crash happened. Regards, Rainer Rainer Jung-3 wrote: dave.smith schrieb: Yesterday, I upgraded our dev environment to mod_jk 1.2.26, which couldn't have been easier. It will probably take me a couple of days before I can get this done in production, though. I terminate all HTTPS requests before they get to the web server, so from what you have described, it is probably safe to disable the APR connector. How do I disable it, though? I will look into disabling this after I have updated mod_jk in production. Locate the tcnative shared object file (tcnative.so or tcnative-1.so) and renme it, so that the linker loader does not find it (e.g. add an underscore at the end of the file name). Then during the next startup, Tomcat should emit an info level log message telling you, that it couldn't find the lib. Here's the full stack trace for that exception, displayed in my Tomcat logs: Jul 10, 2008 10:06:50 PM org.apache.catalina.connector.Request parseParameters WARNING: Exception thrown whilst processing POSTed parameters java.io.IOException: Socket read failed at org.apache.coyote.ajp.AjpAprProcessor.read(AjpAprProcessor.java:1037) at org.apache.coyote.ajp.AjpAprProcessor.readMessage(AjpAprProcessor.java:1158) at org.apache.coyote.ajp.AjpAprProcessor.receive(AjpAprProcessor.java:1090) at org.apache.coyote.ajp.AjpAprProcessor$SocketInputBuffer.doRead(AjpAprProcessor.java:1228) at org.apache.coyote.Request.doRead(Request.java:419) at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:265) at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:403) at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:280) at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:193) at org.apache.catalina.connector.Request.readPostBody(Request.java:2400) at org.apache.catalina.connector.Request.parseParameters(Request.java:2379) at org.apache.catalina.connector.Request.getParameterNames(Request.java:1047) at org.apache.catalina.connector.RequestFacade.getParameterNames(RequestFacade.java:369) at org.apache.struts.util.RequestUtils.populate(RequestUtils.java:1225) at org.apache.struts.action.RequestProcessor.processPopulate(RequestProcessor.java:821) at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:254) at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1482) at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:525) at javax.servlet.http.HttpServlet.service(HttpServlet.java:710) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151) at org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:444) at org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:472) at
Re: Apache/mod_jk serves random files from tomcat
. You are sure it is not 5.5.26? Regards, Rainer Rainer Jung-3 wrote: Hi David, dave.smith schrieb: Hi Rainer, Thanks a lot for the reply. I am using Tomcat 5.5.25 (rpm from jpackage.org). CentOS Linux 2.6.18. httpd was compiled in prefork mode. The prefork settings are: StartServers 8 MinSpareServers5 MaxSpareServers 20 ServerLimit 256 MaxClients 256 MaxRequestsPerChild 4000 I have setup JMeter to run against a test environment, but was unable to reproduce. These random responses occur in production about once every week or so/more. The problem will often (temporarily) correct itself, but sometimes I will need to restart httpd if the problem persists -- restarting tomcat also works to temporarily correct the problem. The only thing strange that I see in my logs are in the test_client.log: WARNING: Exception thrown whilst processing POSTed parameters java.io.IOException: Socket read failed at org.apache.coyote.ajp.AjpAprProcessor.read(AjpAprProcessor.java:1037) ... Thanks for the information. What is test_client.log? It looks like a Tomcat log file? Could you also post a larger part of the stack, or do you only get one line? Would you be able to do the following two things, maybe not both at the same time: - disable the apr connector (tcnative.so) - upgrade jk to 1.2.26 Concerning the apr connector: If you are using OpenSSL with apr and Tomcat or you have some similar reason you really need it, then don't switch. But if you use it without a very specific reason, disabling it for a week or two would help us isolate the problem. Concerning mod_jk upgrade: That should be very easy, apart from the following: if your httpd uses VirtualHost in the configuration, you have to include your JkMount inside the VirtualHost, not in the global part, or you add JkMountCopy On to the VirtualHost. Regards, Rainer Rainer Jung-3 wrote: dave.smith schrieb: Wow. That's weird. Is Tomcat serving the file, or is httpd serving it? Not too weird. I am experiencing the same thing with Tomcat 5.5 and mod_jk 1.2.23. I have Tomcat serving everything. I am also using a load balancer that sends an OPTION every 2 seconds to each web server to make sure that the server is alive. This intermittent random response issue is really killing me. Could you please also add some info: Tomcat version? And from my previous mail: What's you platform and which httpd MPM (prefork orworker or something else) do you use? For some platforms (e.g. AIX) the detection of multi-threading in httpd during mpod_jk build-time was broken. Starting with 1.2.24 we build always including multi-thread support unless explicitely stated via a configure option. If you 1.2.23 build is not thread safe, but your httpd uses threads (like with worker mpm), then such trouble is possible, although more likely you would see crashes etc. For most platforms like Linux and Solaris the threading detection was OK already before 1.2.24. Another possible (but not very likely) cause could be bug 44494 of Tomcat 6.0.16/5.5.26 which under certain circumstances could leave data in the request object after request handling completed. You could try either downgrading to 6.0.15/5.5.25 or upgrading to the soon to be expected 6.0.17/5.5.27. I would also add the access log on the Tomcat side. If you find the same phenomenon there, then it's unlikely, that httpd/mod_jk are responsible and the reason should be inside Tomcat or the webapp. Can you reproduce the problem on a test system? Regards, Rainer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- View this message in context: http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p18762047.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Apache/mod_jk serves random files from tomcat
CustomLog logs/default-access.log common alias /logs /var/widgets Location /logs AuthUserFile /var/widgets/.htpasswd AuthName Widgets AuthType Basic Require valid-user /Location Rewriteengine on RewriteRule ^/$ /index.html [R] jkmount /* loadbalancer jkunmount /logs/*.gz loadbalancer /VirtualHost VirtualHost *:80 DocumentRoot /var/www/html/ ServerName widgets.example.co.uk ErrorLog /var/widgets/widget-error.log CustomLog /var/widgets/widgets-access.log common jkunmount /* loadbalancer /VirtualHost === worker.properties == worker.list=loadbalancer,status worker.node1.port=8009 worker.node1.host=127.0.0.1 worker.node1.type=ajp13 worker.node1.lbfactor=1 worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=node1 worker.status.type=status -- View this message in context: http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p18385568.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- kippdata informationstechnologie GmbH Tel: 0228 98549 -0 Bornheimer Str. 33aFax: 0228 98549 -50 53111 Bonn www.kippdata.de HRB 8018 Amtsgericht Bonn / USt.-IdNr. DE 196 457 417 Geschäftsführer: Dr. Thomas Höfer, Rainer Jung, Sven Maurmann === kippdata informationstechnologie GmbH Tel: +49 228 98549 -0 Bornheimer Str. 33aFax: +49 228 98549 -50 D-53111 Bonn www.kippdata.de HRB 8018 Amtsgericht Bonn / USt.-IdNr. DE 196 457 417 Geschäftsführer: Dr. Thomas Höfer, Rainer Jung, Sven Maurmann - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- View this message in context: http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p18565942.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Apache/mod_jk serves random files from tomcat
jkunmount /logs/*.gz loadbalancer /VirtualHost VirtualHost *:80 DocumentRoot /var/www/html/ ServerName widgets.example.co.uk ErrorLog /var/widgets/widget-error.log CustomLog /var/widgets/widgets-access.log common jkunmount /* loadbalancer /VirtualHost === worker.properties == worker.list=loadbalancer,status worker.node1.port=8009 worker.node1.host=127.0.0.1 worker.node1.type=ajp13 worker.node1.lbfactor=1 worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=node1 worker.status.type=status -- View this message in context: http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p18385568.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- kippdata informationstechnologie GmbH Tel: 0228 98549 -0 Bornheimer Str. 33aFax: 0228 98549 -50 53111 Bonn www.kippdata.de HRB 8018 Amtsgericht Bonn / USt.-IdNr. DE 196 457 417 Geschäftsführer: Dr. Thomas Höfer, Rainer Jung, Sven Maurmann === kippdata informationstechnologie GmbH Tel: +49 228 98549 -0 Bornheimer Str. 33aFax: +49 228 98549 -50 D-53111 Bonn www.kippdata.de HRB 8018 Amtsgericht Bonn / USt.-IdNr. DE 196 457 417 Geschäftsführer: Dr. Thomas Höfer, Rainer Jung, Sven Maurmann - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Apache/mod_jk serves random files from tomcat
Tim Redding wrote: Just checked the the mod_jk log file. 2 other files were requested at 12:31:42 in addition to the /css/global.css file. One was index.html which just happened to be 2352 bytes in size. Exactly the same as the mysterious global.css file we got served. I have full debug level log files for mod_jk if interested. Very interested. You can send it or post a download URL to me directly, if you think there are private things in there. Regards, Rainer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Apache/mod_jk serves random files from tomcat
Rainer Jung-3 wrote:
Do you get any info or above log messages in the JK log file close to
this events?
I don't believe there was anything interesting in the JK log file near that
time period.
Rainer Jung-3 wrote:
What is the last column in the log formats? Is it both %D? Then there is
also something else strange, namely execution times of 4, 28 and 99
milliseconds in the Tomcat log, and 14.8, 9 and 8.9 milliseconds on the
httpd side. So the first request takes much longer on the httpd side,
the next two seem to be quicker for httpd than for the backend, which is
impossible unless httpd/mod_jk terminate the request/response cycle
prematurely. In this case I would expect a message either in the JK log
or the httpd error log.
They are timestamps as you guessed and yes they are a bit strange when
correlating the two sets of log entries. I also forgot to mention that we
have app-side logging and we do know that userB's request to /portal/Login
was received and processed correctly by the application. Knowing that
requests make it through to the application it appears to be an issue on the
response side of the cycle.
Rainer Jung-3 wrote:
Do you have some efficient way to detect those errors (scripts comparing
the log files), or do you only detect them on user feedback? If you've
got scripts, you could also add %P and %{tid}P to the access log format,
which will provide process id and thread id, so we get an idea, if the
switch happens on a single thread.
We don't have a script to detect these errors and have, to date, relied on
user reports. I may look into figuring out a script to compare the two sets
of logs and see if we can find these that way. I'll also look into adding
the pid and tid info to the logs.
-Eric
--
View this message in context:
http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p18509694.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: Apache/mod_jk serves random files from tomcat
/VirtualHost === worker.properties == worker.list=loadbalancer,status worker.node1.port=8009 worker.node1.host=127.0.0.1 worker.node1.type=ajp13 worker.node1.lbfactor=1 worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=node1 worker.status.type=status -- View this message in context: http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p18385568.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- View this message in context: http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p18511166.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Apache/mod_jk serves random files from tomcat
Basic Require valid-user /Location Rewriteengine on RewriteRule ^/$ /index.html [R] jkmount /* loadbalancer jkunmount /logs/*.gz loadbalancer /VirtualHost VirtualHost *:80 DocumentRoot /var/www/html/ ServerName widgets.example.co.uk ErrorLog /var/widgets/widget-error.log CustomLog /var/widgets/widgets-access.log common jkunmount /* loadbalancer /VirtualHost === worker.properties == worker.list=loadbalancer,status worker.node1.port=8009 worker.node1.host=127.0.0.1 worker.node1.type=ajp13 worker.node1.lbfactor=1 worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=node1 worker.status.type=status -- View this message in context: http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p18385568.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- View this message in context: http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p18512318.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Apache/mod_jk serves random files from tomcat
We are seeing a similar problem. We have Apache 2.2.6 using prefork connecting to Tomcat 5.5.23 via mod_jk 1.2.25 We see infrequent issues with requests getting swapped from request to Tomcat. After the first time it was reported we added Tomcat access logging to try and help figure out the problem. Here are logs from both Apache and Tomcat where UserB ends up seeing the results of UserA's request for '/portal/content' for their request to '/portal/Login'. You can also see that UserA doesn't actually get a full response for their request to /portal/content. Tomcat Logs: 0.0.0.0 - userA [22/May/2008:21:57:01 -0500] GET /portal/redirect.jsp HTTP/1.1 302 - - Mozilla/4.0 4 0.0.0.0 - userA [22/May/2008:21:57:01 -0500] GET /portal/content HTTP/1.1 200 1154 - Mozilla/4.0 28 1.1.1.1 - userB [22/May/2008:21:57:03 -0500] GET /portal/Login HTTP/1.1 302 - - Mozilla/5.0 99 Apache Logs: 0.0.0.0 - userA [22/May/2008:21:57:01 -0500] GET /portal/redirect.jsp HTTP/1.1 302 - - Mozilla/4.0 14895 0.0.0.0 - userA [22/May/2008:21:57:01 -0500] GET /portal/content HTTP/1.1 200 550 - Mozilla/4.0 9009 1.1.1.1 - userB [22/May/2008:21:57:03 -0500] GET /portal/Login HTTP/1.1 200 1154 - Mozilla/5.0 8946 We do not have the native connector deployed with Tomcat (we're using the distribution from the TC website). -- View this message in context: http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p18497152.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Apache/mod_jk serves random files from tomcat
Eric Dalquist schrieb:
We are seeing a similar problem.
We have Apache 2.2.6 using prefork connecting to Tomcat 5.5.23 via mod_jk
1.2.25
We see infrequent issues with requests getting swapped from request to
Tomcat. After the first time it was reported we added Tomcat access logging
to try and help figure out the problem.
Here are logs from both Apache and Tomcat where UserB ends up seeing the
results of UserA's request for '/portal/content' for their request to
'/portal/Login'. You can also see that UserA doesn't actually get a full
response for their request to /portal/content.
Tomcat Logs:
0.0.0.0 - userA [22/May/2008:21:57:01 -0500] GET /portal/redirect.jsp
HTTP/1.1 302 - - Mozilla/4.0 4
0.0.0.0 - userA [22/May/2008:21:57:01 -0500] GET /portal/content HTTP/1.1
200 1154 - Mozilla/4.0 28
1.1.1.1 - userB [22/May/2008:21:57:03 -0500] GET /portal/Login HTTP/1.1
302 - - Mozilla/5.0 99
Apache Logs:
0.0.0.0 - userA [22/May/2008:21:57:01 -0500] GET /portal/redirect.jsp
HTTP/1.1 302 - - Mozilla/4.0 14895
0.0.0.0 - userA [22/May/2008:21:57:01 -0500] GET /portal/content HTTP/1.1
200 550 - Mozilla/4.0 9009
1.1.1.1 - userB [22/May/2008:21:57:03 -0500] GET /portal/Login HTTP/1.1
200 1154 - Mozilla/5.0 8946
We do not have the native connector deployed with Tomcat (we're using the
distribution from the TC website).
Do you get any info or above log messages in the JK log file close to
this events?
What is the last column in the log formats? Is it both %D? Then there is
also something else strange, namely execution times of 4, 28 and 99
milliseconds in the Tomcat log, and 14.8, 9 and 8.9 milliseconds on the
httpd side. So the first request takes much longer on the httpd side,
the next two seem to be quicker for httpd than for the backend, which is
impossible unless httpd/mod_jk terminate the request/response cycle
prematurely. In this case I would expect a message either in the JK log
or the httpd error log.
Do you have some efficient way to detect those errors (scripts comparing
the log files), or do you only detect them on user feedback? If you've
got scripts, you could also add %P and %{tid}P to the access log format,
which will provide process id and thread id, so we get an idea, if the
switch happens on a single thread.
Regards,
Rainer
-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: Apache/mod_jk serves random files from tomcat
Hi Rainer, Thanks a lot for the reply. I am using Tomcat 5.5.25 (rpm from jpackage.org). CentOS Linux 2.6.18. httpd was compiled in prefork mode. The prefork settings are: StartServers 8 MinSpareServers5 MaxSpareServers 20 ServerLimit 256 MaxClients 256 MaxRequestsPerChild 4000 I have setup JMeter to run against a test environment, but was unable to reproduce. These random responses occur in production about once every week or so/more. The problem will often (temporarily) correct itself, but sometimes I will need to restart httpd if the problem persists -- restarting tomcat also works to temporarily correct the problem. The only thing strange that I see in my logs are in the test_client.log: WARNING: Exception thrown whilst processing POSTed parameters java.io.IOException: Socket read failed at org.apache.coyote.ajp.AjpAprProcessor.read(AjpAprProcessor.java:1037) ... Rainer Jung-3 wrote: dave.smith schrieb: Wow. That's weird. Is Tomcat serving the file, or is httpd serving it? Not too weird. I am experiencing the same thing with Tomcat 5.5 and mod_jk 1.2.23. I have Tomcat serving everything. I am also using a load balancer that sends an OPTION every 2 seconds to each web server to make sure that the server is alive. This intermittent random response issue is really killing me. Could you please also add some info: Tomcat version? And from my previous mail: What's you platform and which httpd MPM (prefork orworker or something else) do you use? For some platforms (e.g. AIX) the detection of multi-threading in httpd during mpod_jk build-time was broken. Starting with 1.2.24 we build always including multi-thread support unless explicitely stated via a configure option. If you 1.2.23 build is not thread safe, but your httpd uses threads (like with worker mpm), then such trouble is possible, although more likely you would see crashes etc. For most platforms like Linux and Solaris the threading detection was OK already before 1.2.24. Another possible (but not very likely) cause could be bug 44494 of Tomcat 6.0.16/5.5.26 which under certain circumstances could leave data in the request object after request handling completed. You could try either downgrading to 6.0.15/5.5.25 or upgrading to the soon to be expected 6.0.17/5.5.27. I would also add the access log on the Tomcat side. If you find the same phenomenon there, then it's unlikely, that httpd/mod_jk are responsible and the reason should be inside Tomcat or the webapp. Can you reproduce the problem on a test system? Regards, Rainer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- View this message in context: http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p18465349.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Apache/mod_jk serves random files from tomcat
Hi David, dave.smith schrieb: Hi Rainer, Thanks a lot for the reply. I am using Tomcat 5.5.25 (rpm from jpackage.org). CentOS Linux 2.6.18. httpd was compiled in prefork mode. The prefork settings are: StartServers 8 MinSpareServers5 MaxSpareServers 20 ServerLimit 256 MaxClients 256 MaxRequestsPerChild 4000 I have setup JMeter to run against a test environment, but was unable to reproduce. These random responses occur in production about once every week or so/more. The problem will often (temporarily) correct itself, but sometimes I will need to restart httpd if the problem persists -- restarting tomcat also works to temporarily correct the problem. The only thing strange that I see in my logs are in the test_client.log: WARNING: Exception thrown whilst processing POSTed parameters java.io.IOException: Socket read failed at org.apache.coyote.ajp.AjpAprProcessor.read(AjpAprProcessor.java:1037) ... Thanks for the information. What is test_client.log? It looks like a Tomcat log file? Could you also post a larger part of the stack, or do you only get one line? Would you be able to do the following two things, maybe not both at the same time: - disable the apr connector (tcnative.so) - upgrade jk to 1.2.26 Concerning the apr connector: If you are using OpenSSL with apr and Tomcat or you have some similar reason you really need it, then don't switch. But if you use it without a very specific reason, disabling it for a week or two would help us isolate the problem. Concerning mod_jk upgrade: That should be very easy, apart from the following: if your httpd uses VirtualHost in the configuration, you have to include your JkMount inside the VirtualHost, not in the global part, or you add JkMountCopy On to the VirtualHost. Regards, Rainer Rainer Jung-3 wrote: dave.smith schrieb: Wow. That's weird. Is Tomcat serving the file, or is httpd serving it? Not too weird. I am experiencing the same thing with Tomcat 5.5 and mod_jk 1.2.23. I have Tomcat serving everything. I am also using a load balancer that sends an OPTION every 2 seconds to each web server to make sure that the server is alive. This intermittent random response issue is really killing me. Could you please also add some info: Tomcat version? And from my previous mail: What's you platform and which httpd MPM (prefork orworker or something else) do you use? For some platforms (e.g. AIX) the detection of multi-threading in httpd during mpod_jk build-time was broken. Starting with 1.2.24 we build always including multi-thread support unless explicitely stated via a configure option. If you 1.2.23 build is not thread safe, but your httpd uses threads (like with worker mpm), then such trouble is possible, although more likely you would see crashes etc. For most platforms like Linux and Solaris the threading detection was OK already before 1.2.24. Another possible (but not very likely) cause could be bug 44494 of Tomcat 6.0.16/5.5.26 which under certain circumstances could leave data in the request object after request handling completed. You could try either downgrading to 6.0.15/5.5.25 or upgrading to the soon to be expected 6.0.17/5.5.27. I would also add the access log on the Tomcat side. If you find the same phenomenon there, then it's unlikely, that httpd/mod_jk are responsible and the reason should be inside Tomcat or the webapp. Can you reproduce the problem on a test system? Regards, Rainer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Apache/mod_jk serves random files from tomcat
Rainer Jung wrote: Hi David, dave.smith schrieb: Hi Rainer, Thanks a lot for the reply. I am using Tomcat 5.5.25 (rpm from jpackage.org). CentOS Linux 2.6.18. Could you be seeing CVE-2007-6286 ? See http://tomcat.apache.org/security-5.html for info. Mark - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Apache/mod_jk serves random files from tomcat
it without a very specific reason, disabling it for a week or two would help us isolate the problem. Concerning mod_jk upgrade: That should be very easy, apart from the following: if your httpd uses VirtualHost in the configuration, you have to include your JkMount inside the VirtualHost, not in the global part, or you add JkMountCopy On to the VirtualHost. Regards, Rainer Rainer Jung-3 wrote: dave.smith schrieb: Wow. That's weird. Is Tomcat serving the file, or is httpd serving it? Not too weird. I am experiencing the same thing with Tomcat 5.5 and mod_jk 1.2.23. I have Tomcat serving everything. I am also using a load balancer that sends an OPTION every 2 seconds to each web server to make sure that the server is alive. This intermittent random response issue is really killing me. Could you please also add some info: Tomcat version? And from my previous mail: What's you platform and which httpd MPM (prefork orworker or something else) do you use? For some platforms (e.g. AIX) the detection of multi-threading in httpd during mpod_jk build-time was broken. Starting with 1.2.24 we build always including multi-thread support unless explicitely stated via a configure option. If you 1.2.23 build is not thread safe, but your httpd uses threads (like with worker mpm), then such trouble is possible, although more likely you would see crashes etc. For most platforms like Linux and Solaris the threading detection was OK already before 1.2.24. Another possible (but not very likely) cause could be bug 44494 of Tomcat 6.0.16/5.5.26 which under certain circumstances could leave data in the request object after request handling completed. You could try either downgrading to 6.0.15/5.5.25 or upgrading to the soon to be expected 6.0.17/5.5.27. I would also add the access log on the Tomcat side. If you find the same phenomenon there, then it's unlikely, that httpd/mod_jk are responsible and the reason should be inside Tomcat or the webapp. Can you reproduce the problem on a test system? Regards, Rainer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- View this message in context: http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p18466059.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Apache/mod_jk serves random files from tomcat
dave.smith schrieb: Yesterday, I upgraded our dev environment to mod_jk 1.2.26, which couldn't have been easier. It will probably take me a couple of days before I can get this done in production, though. I terminate all HTTPS requests before they get to the web server, so from what you have described, it is probably safe to disable the APR connector. How do I disable it, though? I will look into disabling this after I have updated mod_jk in production. Locate the tcnative shared object file (tcnative.so or tcnative-1.so) and renme it, so that the linker loader does not find it (e.g. add an underscore at the end of the file name). Then during the next startup, Tomcat should emit an info level log message telling you, that it couldn't find the lib. Here's the full stack trace for that exception, displayed in my Tomcat logs: Jul 10, 2008 10:06:50 PM org.apache.catalina.connector.Request parseParameters WARNING: Exception thrown whilst processing POSTed parameters java.io.IOException: Socket read failed at org.apache.coyote.ajp.AjpAprProcessor.read(AjpAprProcessor.java:1037) at org.apache.coyote.ajp.AjpAprProcessor.readMessage(AjpAprProcessor.java:1158) at org.apache.coyote.ajp.AjpAprProcessor.receive(AjpAprProcessor.java:1090) at org.apache.coyote.ajp.AjpAprProcessor$SocketInputBuffer.doRead(AjpAprProcessor.java:1228) at org.apache.coyote.Request.doRead(Request.java:419) at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:265) at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:403) at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:280) at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:193) at org.apache.catalina.connector.Request.readPostBody(Request.java:2400) at org.apache.catalina.connector.Request.parseParameters(Request.java:2379) at org.apache.catalina.connector.Request.getParameterNames(Request.java:1047) at org.apache.catalina.connector.RequestFacade.getParameterNames(RequestFacade.java:369) at org.apache.struts.util.RequestUtils.populate(RequestUtils.java:1225) at org.apache.struts.action.RequestProcessor.processPopulate(RequestProcessor.java:821) at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:254) at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1482) at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:525) at javax.servlet.http.HttpServlet.service(HttpServlet.java:710) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151) at org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:444) at org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:472) at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1286) at java.lang.Thread.run(Thread.java:595) OK, thanks. You are sure it is not 5.5.26? Regards, Rainer Rainer Jung-3 wrote: Hi David, dave.smith schrieb: Hi Rainer, Thanks a lot for the reply. I am using Tomcat 5.5.25 (rpm from jpackage.org). CentOS Linux 2.6.18. httpd was compiled in prefork mode. The prefork settings are: StartServers 8 MinSpareServers5 MaxSpareServers 20 ServerLimit 256 MaxClients 256 MaxRequestsPerChild 4000 I have setup JMeter to run against a test environment, but was unable to reproduce. These random responses occur in production about once every week or so/more. The problem will often (temporarily) correct itself, but sometimes I will need to restart httpd if the problem persists -- restarting tomcat also works to temporarily correct the problem. The only thing strange that I see in my logs are in the test_client.log: WARNING: Exception thrown whilst processing POSTed parameters java.io.IOException: Socket read failed at org.apache.coyote.ajp.AjpAprProcessor.read(AjpAprProcessor.java:1037) ... Thanks for the information. What is test_client.log? It looks like a Tomcat log file? Could you also post a larger part of the stack, or do you only get one
Re: Apache/mod_jk serves random files from tomcat
Thanks, Mark. When I first saw that security notice, I thought, this is it!. I don't think it is my problem, though, because I don't allow direct SSL requests to get to the web servers. All HTTPS gets terminated to HTTP at the load balancer. The load balancer sends the HTTP requests to Apache 2 on the web server which sends it to Tomcat via mod_jk. VirtualHost 10.10.1.1:80 # ... JkMount / ajp13 JkMount /* ajp13 DocumentRoot /usr/share/tomcat5/webapps/ROOT /VirtualHost Also, in server.xml, I have the ajp connector on 8009 (protocol AJP/1.3) with redirectPort to 8443, but never define a connector on 8443. Furthermore, I spent the last hour trying reproduce the issue with netcat and was unable to. Thanks, Dave Mark Thomas-18 wrote: Rainer Jung wrote: Hi David, dave.smith schrieb: Hi Rainer, Thanks a lot for the reply. I am using Tomcat 5.5.25 (rpm from jpackage.org). CentOS Linux 2.6.18. Could you be seeing CVE-2007-6286 ? See http://tomcat.apache.org/security-5.html for info. Mark - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- View this message in context: http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p18468376.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Apache/mod_jk serves random files from tomcat
Wow. That's weird. Is Tomcat serving the file, or is httpd serving it? Not too weird. I am experiencing the same thing with Tomcat 5.5 and mod_jk 1.2.23. I have Tomcat serving everything. I am also using a load balancer that sends an OPTION every 2 seconds to each web server to make sure that the server is alive. This intermittent random response issue is really killing me. Christopher Schultz-2 wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Tim, Tim Redding wrote: | We are experiencing intermittent problems with a particular site that is not | returning the correct file that is requested. For instance if we request | the index.html file we actually get a css file or even an image. From the | apache access log you can see that the size of the index.html file grows on | the second request. This is because a gif was actually returned. | | XXX.XXX.XXX.130 - - [10/Jul/2008:15:10:39 +0100] GET /index.html HTTP/1.1 | 200 1068 | XXX.XXX.XXX.130 - - [10/Jul/2008:15:13:10 +0100] GET /index.html HTTP/1.1 | 200 9526 | XXX.XXX.XXX.130 - - [10/Jul/2008:15:13:48 +0100] GET /index.html HTTP/1.1 | 200 1086 Wow. That's weird. Is Tomcat serving the file, or is httpd serving it? Does this happen whether Tomcat is running or not? How about if mod_jk is disabled versus enabled? | We have Apache/2.2.3 on the front on a Tomcat 6.0.16 server with mod_jk | (version unknown but fairly recent). $ strings /path/to/mod_jk.so | grep mod_jk/ mod_jk/1.2.26 (This is what mine returns). Remember that mod_jk 1.2 has been around for ... ever and there's a big difference between 1.2.5 and 1.2.26. Virtually all versions of mod_jk are compatible with all versions of Apache httpd and Tomcat, so you should upgrade to the latest whenever it is convenient for you. | We have all assets in our war file. Does the WAR file get expanded during deployment? | JkMount status Probably not your problem, but is this the correct syntax? I would have expected something like JkMount status_uri status | jkmount /* loadbalancer Looks like everything goes to Tomcat. Try enabling the AccessLogValve to see which requests Tomcat is responding to -- that may help shed a bit of light on the problem. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkh2eEsACgkQ9CaO5/Lv0PDSZwCgjk7OvOHHAZpvDDolD3JAgIdq EVgAnRWBvsQbbNZlSvJsRp+b2dLmT0ml =9ZiZ -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- View this message in context: http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p18450219.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Apache/mod_jk serves random files from tomcat
dave.smith schrieb: Wow. That's weird. Is Tomcat serving the file, or is httpd serving it? Not too weird. I am experiencing the same thing with Tomcat 5.5 and mod_jk 1.2.23. I have Tomcat serving everything. I am also using a load balancer that sends an OPTION every 2 seconds to each web server to make sure that the server is alive. This intermittent random response issue is really killing me. Could you please also add some info: Tomcat version? And from my previous mail: What's you platform and which httpd MPM (prefork orworker or something else) do you use? For some platforms (e.g. AIX) the detection of multi-threading in httpd during mpod_jk build-time was broken. Starting with 1.2.24 we build always including multi-thread support unless explicitely stated via a configure option. If you 1.2.23 build is not thread safe, but your httpd uses threads (like with worker mpm), then such trouble is possible, although more likely you would see crashes etc. For most platforms like Linux and Solaris the threading detection was OK already before 1.2.24. Another possible (but not very likely) cause could be bug 44494 of Tomcat 6.0.16/5.5.26 which under certain circumstances could leave data in the request object after request handling completed. You could try either downgrading to 6.0.15/5.5.25 or upgrading to the soon to be expected 6.0.17/5.5.27. I would also add the access log on the Tomcat side. If you find the same phenomenon there, then it's unlikely, that httpd/mod_jk are responsible and the reason should be inside Tomcat or the webapp. Can you reproduce the problem on a test system? Regards, Rainer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Apache/mod_jk serves random files from tomcat
Hi Tim, Tim Redding schrieb: Hi, We are experiencing intermittent problems with a particular site that is not returning the correct file that is requested. For instance if we request the index.html file we actually get a css file or even an image. From the apache access log you can see that the size of the index.html file grows on the second request. This is because a gif was actually returned. XXX.XXX.XXX.130 - - [10/Jul/2008:15:10:39 +0100] GET /index.html HTTP/1.1 200 1068 XXX.XXX.XXX.130 - - [10/Jul/2008:15:13:10 +0100] GET /index.html HTTP/1.1 200 9526 XXX.XXX.XXX.130 - - [10/Jul/2008:15:13:48 +0100] GET /index.html HTTP/1.1 200 1086 This is pretty serious (I assume that 1086!=1968 was a typo). No error messages are logged in the mode_jk.log file. We have Apache/2.2.3 on the front on a Tomcat 6.0.16 server with mod_jk (version unknown but fairly recent). We have all assets in our war file. When we hit Tomcat directly on port 8080 it serves the correct file. And to fix the problem an apache restart seems to sort things out. On this server with have 2 vhosts. One is a simple nothing fancy static site and the other forwards everything to our Tomcat server. Below I've included our mod_jk config and a snippet of our httpd.conf. Any ideas or things to try would be most appreciated. What's you platform and which httpd MPM (prefork orworker or something else) do you use? For some platforms (e.g. AIX) the detection of multi-threading in httpd during mpod_jk build-time was broken. Starting with 1.2.24 we build always including multi-thread support unless explicitely stated via a configure option. If you 1.2.23 build is not thread safe, but your httpd uses threads (like with worker mpm), then such trouble is possible, although more likely you would see crashes etc. For most platforms like Linux and Solaris the threading detection was OK already before 1.2.24. Another possible (but not very likely) cause could be bug 44494 of Tomcat 6.0.16 which under certain circumstances could leave data in the request object after request handling completed. You could try either downgrding to 6.0.15 or upgrading to the soon to be expected 6.0.17. I would also add the access log on the Tomcat side. If you find the same phenomenon there, then it's unlikely, that httpd/mod_jk are responsible and the reason should be inside Tomcat or the webapp. Can you reproduce the problem on a test system? Regards, Rainer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Apache/mod_jk serves random files from tomcat
Christopher Schultz schrieb: | JkMount status Probably not your problem, but is this the correct syntax? You can use that syntax in Location directives. Then the missing URL is taken from the Location URL, so the mount is valid for each URL in the respective Location. Regards, Rainer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Apache/mod_jk serves random files from tomcat
Thanks for the quick reply. I've enabled the AccessLogValve. I've just gotta wait for it to start playing up again. Could be 2 hours or 2 weeks. I'll reply when I have more info. Tim. Len Popp wrote: That log file is from the httpd server, right? What does the Tomcat log file say? (Turn on AccessLogValve if you haven't already.) Is Tomcat always getting requests for the correct file, or is mod_jk requesting the wrong file sometimes? -- Len On Thu, Jul 10, 2008 at 11:44, Tim Redding [EMAIL PROTECTED] wrote: Hi, We are experiencing intermittent problems with a particular site that is not returning the correct file that is requested. For instance if we request the index.html file we actually get a css file or even an image. From the apache access log you can see that the size of the index.html file grows on the second request. This is because a gif was actually returned. XXX.XXX.XXX.130 - - [10/Jul/2008:15:10:39 +0100] GET /index.html HTTP/1.1 200 1068 XXX.XXX.XXX.130 - - [10/Jul/2008:15:13:10 +0100] GET /index.html HTTP/1.1 200 9526 XXX.XXX.XXX.130 - - [10/Jul/2008:15:13:48 +0100] GET /index.html HTTP/1.1 200 1086 No error messages are logged in the mode_jk.log file. We have Apache/2.2.3 on the front on a Tomcat 6.0.16 server with mod_jk (version unknown but fairly recent). We have all assets in our war file. When we hit Tomcat directly on port 8080 it serves the correct file. And to fix the problem an apache restart seems to sort things out. On this server with have 2 vhosts. One is a simple nothing fancy static site and the other forwards everything to our Tomcat server. Below I've included our mod_jk config and a snippet of our httpd.conf. Any ideas or things to try would be most appreciated. Tim. = mod_jk.conf == # Load mod_jk module # Specify the filename of the mod_jk lib LoadModule jk_module modules/mod_jk.so # Where to find workers.properties JkWorkersFile conf/workers.properties # Where to put jk logs JkLogFile logs/mod_jk.log # Set the jk log level [debug/error/info] JkLogLevel debug # Select the log format JkLogStampFormat [%a %b %d %H:%M:%S %Y] # JkOptions indicates to send SSK KEY SIZE JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories # JkRequestLogFormat JkRequestLogFormat %w %V %T # Add shared memory. # This directive is present with 1.2.10 and # later versions of mod_jk, and is needed for # for load balancing to work properly JkShmFile logs/jk.shm # original URL pass through JkEnvVarORIGINAL_URIw00t # Add jkstatus for managing runtime data Location /jkstatus/ JkMount status Order deny,allow Deny from all Allow from 127.0.0.1 /Location === httpd.conf (our additions to the default file) == # mod_jk include Include conf/mod_jk.conf VirtualHost *:80 DocumentRoot /var/www/html/ ServerName example.co.uk ErrorLog logs/default-error.log CustomLog logs/default-access.log common alias /logs /var/widgets Location /logs AuthUserFile /var/widgets/.htpasswd AuthName Widgets AuthType Basic Require valid-user /Location Rewriteengine on RewriteRule ^/$ /index.html [R] jkmount /* loadbalancer jkunmount /logs/*.gz loadbalancer /VirtualHost VirtualHost *:80 DocumentRoot /var/www/html/ ServerName widgets.example.co.uk ErrorLog /var/widgets/widget-error.log CustomLog /var/widgets/widgets-access.log common jkunmount /* loadbalancer /VirtualHost === worker.properties == worker.list=loadbalancer,status worker.node1.port=8009 worker.node1.host=127.0.0.1 worker.node1.type=ajp13 worker.node1.lbfactor=1 worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=node1 worker.status.type=status -- View this message in context: http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p18385568.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- View this message in context: http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p18405872.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Apache/mod_jk serves random files from tomcat
Hi, Wow. That's weird. Is Tomcat serving the file, or is httpd serving it? All requests are going to tomcat via httpd. Does the WAR file get expanded during deployment? Well tomcat usually extracts the contents to a temporary location so I guess yes. Does this happen whether Tomcat is running or not? How about if mod_jk is disabled versus enabled? When tomcat isn't running I get the standard 503 Service Temporarily Unavailable. As this is a production server I can't really afford to disable mod_jk. We're running mod_jk/1.2.23. Could probably upgrade. But haven't had an issue with this on our other servers. I'll wait and see what the accessLogValve presents. I'm pretty sure the syntax is correct. It follows the sysntax i've seen in many examples on the net I've corrected the case just in case that makes any difference. Yeah I've just enabled the AccessLogValve so I'll know more about problem soon Thanks for your response. Tim. Christopher Schultz-2 wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Tim, Tim Redding wrote: | We are experiencing intermittent problems with a particular site that is not | returning the correct file that is requested. For instance if we request | the index.html file we actually get a css file or even an image. From the | apache access log you can see that the size of the index.html file grows on | the second request. This is because a gif was actually returned. | | XXX.XXX.XXX.130 - - [10/Jul/2008:15:10:39 +0100] GET /index.html HTTP/1.1 | 200 1068 | XXX.XXX.XXX.130 - - [10/Jul/2008:15:13:10 +0100] GET /index.html HTTP/1.1 | 200 9526 | XXX.XXX.XXX.130 - - [10/Jul/2008:15:13:48 +0100] GET /index.html HTTP/1.1 | 200 1086 Wow. That's weird. Is Tomcat serving the file, or is httpd serving it? Does this happen whether Tomcat is running or not? How about if mod_jk is disabled versus enabled? | We have Apache/2.2.3 on the front on a Tomcat 6.0.16 server with mod_jk | (version unknown but fairly recent). $ strings /path/to/mod_jk.so | grep mod_jk/ mod_jk/1.2.26 (This is what mine returns). Remember that mod_jk 1.2 has been around for ... ever and there's a big difference between 1.2.5 and 1.2.26. Virtually all versions of mod_jk are compatible with all versions of Apache httpd and Tomcat, so you should upgrade to the latest whenever it is convenient for you. | We have all assets in our war file. Does the WAR file get expanded during deployment? | JkMount status Probably not your problem, but is this the correct syntax? I would have expected something like JkMount status_uri status | jkmount /* loadbalancer Looks like everything goes to Tomcat. Try enabling the AccessLogValve to see which requests Tomcat is responding to -- that may help shed a bit of light on the problem. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkh2eEsACgkQ9CaO5/Lv0PDSZwCgjk7OvOHHAZpvDDolD3JAgIdq EVgAnRWBvsQbbNZlSvJsRp+b2dLmT0ml =9ZiZ -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- View this message in context: http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p18406520.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Apache/mod_jk serves random files from tomcat
Hi, We are experiencing intermittent problems with a particular site that is not returning the correct file that is requested. For instance if we request the index.html file we actually get a css file or even an image. From the apache access log you can see that the size of the index.html file grows on the second request. This is because a gif was actually returned. XXX.XXX.XXX.130 - - [10/Jul/2008:15:10:39 +0100] GET /index.html HTTP/1.1 200 1068 XXX.XXX.XXX.130 - - [10/Jul/2008:15:13:10 +0100] GET /index.html HTTP/1.1 200 9526 XXX.XXX.XXX.130 - - [10/Jul/2008:15:13:48 +0100] GET /index.html HTTP/1.1 200 1086 No error messages are logged in the mode_jk.log file. We have Apache/2.2.3 on the front on a Tomcat 6.0.16 server with mod_jk (version unknown but fairly recent). We have all assets in our war file. When we hit Tomcat directly on port 8080 it serves the correct file. And to fix the problem an apache restart seems to sort things out. On this server with have 2 vhosts. One is a simple nothing fancy static site and the other forwards everything to our Tomcat server. Below I've included our mod_jk config and a snippet of our httpd.conf. Any ideas or things to try would be most appreciated. Tim. = mod_jk.conf == # Load mod_jk module # Specify the filename of the mod_jk lib LoadModule jk_module modules/mod_jk.so # Where to find workers.properties JkWorkersFile conf/workers.properties # Where to put jk logs JkLogFile logs/mod_jk.log # Set the jk log level [debug/error/info] JkLogLevel debug # Select the log format JkLogStampFormat [%a %b %d %H:%M:%S %Y] # JkOptions indicates to send SSK KEY SIZE JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories # JkRequestLogFormat JkRequestLogFormat %w %V %T # Add shared memory. # This directive is present with 1.2.10 and # later versions of mod_jk, and is needed for # for load balancing to work properly JkShmFile logs/jk.shm # original URL pass through JkEnvVarORIGINAL_URIw00t # Add jkstatus for managing runtime data Location /jkstatus/ JkMount status Order deny,allow Deny from all Allow from 127.0.0.1 /Location === httpd.conf (our additions to the default file) == # mod_jk include Include conf/mod_jk.conf VirtualHost *:80 DocumentRoot /var/www/html/ ServerName example.co.uk ErrorLog logs/default-error.log CustomLog logs/default-access.log common alias /logs /var/widgets Location /logs AuthUserFile /var/widgets/.htpasswd AuthName Widgets AuthType Basic Require valid-user /Location Rewriteengine on RewriteRule ^/$ /index.html [R] jkmount /* loadbalancer jkunmount /logs/*.gz loadbalancer /VirtualHost VirtualHost *:80 DocumentRoot /var/www/html/ ServerName widgets.example.co.uk ErrorLog /var/widgets/widget-error.log CustomLog /var/widgets/widgets-access.log common jkunmount /* loadbalancer /VirtualHost === worker.properties == worker.list=loadbalancer,status worker.node1.port=8009 worker.node1.host=127.0.0.1 worker.node1.type=ajp13 worker.node1.lbfactor=1 worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=node1 worker.status.type=status -- View this message in context: http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p18385568.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Apache/mod_jk serves random files from tomcat
That log file is from the httpd server, right? What does the Tomcat log file say? (Turn on AccessLogValve if you haven't already.) Is Tomcat always getting requests for the correct file, or is mod_jk requesting the wrong file sometimes? -- Len On Thu, Jul 10, 2008 at 11:44, Tim Redding [EMAIL PROTECTED] wrote: Hi, We are experiencing intermittent problems with a particular site that is not returning the correct file that is requested. For instance if we request the index.html file we actually get a css file or even an image. From the apache access log you can see that the size of the index.html file grows on the second request. This is because a gif was actually returned. XXX.XXX.XXX.130 - - [10/Jul/2008:15:10:39 +0100] GET /index.html HTTP/1.1 200 1068 XXX.XXX.XXX.130 - - [10/Jul/2008:15:13:10 +0100] GET /index.html HTTP/1.1 200 9526 XXX.XXX.XXX.130 - - [10/Jul/2008:15:13:48 +0100] GET /index.html HTTP/1.1 200 1086 No error messages are logged in the mode_jk.log file. We have Apache/2.2.3 on the front on a Tomcat 6.0.16 server with mod_jk (version unknown but fairly recent). We have all assets in our war file. When we hit Tomcat directly on port 8080 it serves the correct file. And to fix the problem an apache restart seems to sort things out. On this server with have 2 vhosts. One is a simple nothing fancy static site and the other forwards everything to our Tomcat server. Below I've included our mod_jk config and a snippet of our httpd.conf. Any ideas or things to try would be most appreciated. Tim. = mod_jk.conf == # Load mod_jk module # Specify the filename of the mod_jk lib LoadModule jk_module modules/mod_jk.so # Where to find workers.properties JkWorkersFile conf/workers.properties # Where to put jk logs JkLogFile logs/mod_jk.log # Set the jk log level [debug/error/info] JkLogLevel debug # Select the log format JkLogStampFormat [%a %b %d %H:%M:%S %Y] # JkOptions indicates to send SSK KEY SIZE JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories # JkRequestLogFormat JkRequestLogFormat %w %V %T # Add shared memory. # This directive is present with 1.2.10 and # later versions of mod_jk, and is needed for # for load balancing to work properly JkShmFile logs/jk.shm # original URL pass through JkEnvVarORIGINAL_URIw00t # Add jkstatus for managing runtime data Location /jkstatus/ JkMount status Order deny,allow Deny from all Allow from 127.0.0.1 /Location === httpd.conf (our additions to the default file) == # mod_jk include Include conf/mod_jk.conf VirtualHost *:80 DocumentRoot /var/www/html/ ServerName example.co.uk ErrorLog logs/default-error.log CustomLog logs/default-access.log common alias /logs /var/widgets Location /logs AuthUserFile /var/widgets/.htpasswd AuthName Widgets AuthType Basic Require valid-user /Location Rewriteengine on RewriteRule ^/$ /index.html [R] jkmount /* loadbalancer jkunmount /logs/*.gz loadbalancer /VirtualHost VirtualHost *:80 DocumentRoot /var/www/html/ ServerName widgets.example.co.uk ErrorLog /var/widgets/widget-error.log CustomLog /var/widgets/widgets-access.log common jkunmount /* loadbalancer /VirtualHost === worker.properties == worker.list=loadbalancer,status worker.node1.port=8009 worker.node1.host=127.0.0.1 worker.node1.type=ajp13 worker.node1.lbfactor=1 worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=node1 worker.status.type=status -- View this message in context: http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p18385568.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
