RE: Tomcat copies context.xml to conf/Catalina/localhost/app.xml, but neverupdates it?
Hi, It was always intended, and finally fixed in 6.0.19 (see the changelog). The impetus was this bug report: https://issues.apache.org/bugzilla/show_bug.cgi?id=42747 Thank you very much for the reference. To be helpful to other people who stumble upon this thread when searching for information on this problem, here I found an early comment made by Mark about this issue: Sep 4, 2008: http://markmail.org/message/jhl4hvfdhrrgwjv6 Is there a different handling for packaged webapps (wars) vs. an exploded directory? There is a slight one. For WARs with a context.xml, it will get copied to $CATALINA_HOME/conf/[enginename]/[hostname]/ if a contextpath.xml does not exist. context.xml files in an exploded directory do not get copied. This is an inconsistency that is on my list to fix as part of fixing https://issues.apache.org/bugzilla/show_bug.cgi?id=42747 Another related thread is this one: http://markmail.org/thread/ivrwdxm35cs3p3zs Jul 20, 2009 What I want (as option): I know that developer/packager did it right and I never want to have local configuration. Always use the *context.xml* within the currently deployed application, updated every time I redeploy the app. In this thread, Rainer Frey asks for the same thing, although the difference is that back then it was about .war deployments and not about exploded deployments. The copying change makes it possible to edit conf/Catalina/[host]/[appName].xml, regardless of whether the origin was a .war or a directory. Charles, I understand the benefits of this for a production environment, especially when there is a dedicated deployment team. But why not go for the following simpler procedure: 1. If there is a context descriptor in $CATALINA_BASE/conf/[enginename]/[hostname]/[webappname].xml use this. 2. Otherwise, use the context descriptor $CATALINA_BASE/webapps/[webappname]/META-INF/context.xml If that would be all, wouldn't everybody be happy? System administrators can still override whatever is provided by the web application, but if nobody puts something in $CATALINA_BASE/conf/[enginename]/[hostname]/[webappname].xml then the version of the web application is used and the developer can simply update things like datasources without having to change stuff inside a local Tomcat installation. It seems to me that for development the latter situation is much more desirable. In our case (and I figure in a lot of other cases), the source code for a web application is under version control. This includes the META-INF directory and thus context.xml. If I check in a change to context.xml, for instance if I created an additional datasource for my code, then previously all developers would get this change if they update their workspace. With the recent changes however, I'll have to mail each and every developer on our project to inform them they have to clear their $CATALINA_BASE/conf/[enginename]/[hostname]/ directory, or otherwise they will never get the change I just checked in. We have people working at several different locations and some of those include less technically inclined people who mainly work on texts. This kind of a procedure is quite painful for every team member, but is especially painful or perhaps even impossible for those less technically inclined members. How do others cope with this? I guess the problem would be solved if all existing Tomcat server connectors for every IDE (we use MyEclipse), takes this necessary cleaning of $CATALINA_BASE/conf/[enginename]/[hostname]/ into account. On the other hand, if this cleaning is a requirement for development, then it's basically just a workaround for the fact that at least in development mode, Tomcat might be better off not doing this copying-once-and-holding-on-to-the-copy thing? I might be missing something, so I would be grateful if someone can enlighten me. At the moment this required copying seems plain wrong to me on every possible level, but I fully understand that I might not be seeing the bigger picture yet. As I said, I understand the benefits of doing this, but I don't understand what's bad on (optionally) not doing this copying. But what counts as an undeploy for a static deployment? Using the manager to do a real undeploy; deletion of the expanded directory is only part of it. H, but if you absolutely need the manager to do a proper deploy and undeploy, doesn't that somehow imply that static deployment (just copying and removing from the webapps directory) has become more or less deprecated? I think most IDEs take advantage of the static deployment method, although some of them (like Eclipse/WTP) do indeed have knowledge of the conf/Catalina convention. Maybe it's thus only a MyEclipse problem. Besides the Context element, there are also files under Tomcat's work directory that must be removed. Well, in practice this doesn't seem to be necessary that often. We have several web applications with
RE: Tomcat copies context.xml to, conf/Catalina/localhost/app.xml,but neverupdates it?
Hi, 1. If there is a context descriptor in $CATALINA_BASE/conf/[enginename]/[hostname]/[webappname].xml use this. 2. Otherwise, use the context descriptor $CATALINA_BASE/webapps/[webappname]/META-INF/context.xml That is exactly how it works today. Well, in my humble opinion this is how it worked yesterday. Doesn't it work like this today (for exploded archives): 1. If there is a context descriptor in $CATALINA_BASE/conf/[enginename]/[hostname]/[webappname].xml use this. 2. Otherwise, *copy* the context descriptor $CATALINA_BASE/webapps/[webappname]/META-INF/context.xml to $CATALINA_BASE/conf/[enginename]/[hostname]/[webappname].xml 3. Go 1. I fully understand the need for global overrides, but the procedure of copying the very first encountered context.xml from a web application and then continue to use that as the global override variant, I'm sorry but I just don't see the logic in doing that. Are there any other systems out there using a similar mechanism? I'm thinking broadly here, trying to think of general Linux configuration systems, other web servers, programming languages, etc. I'm not sure that I've ever encountered a system where a local configuration can be overridden by a global configuration, but in absence of a global override the local version is copied and kept indefinitely. It just seems really awkward. Basically what you're saying is that you don't trust the web application to provide a correct context.xml, -except- for the first time. Isn't that a little arbitrary? Either you trust the web application or you don't. If you don't trust the web application the maintainer of a Tomcat instance puts his own context.xml in conf/Catalina, thereby overriding whatever the web application defines. If you do trust the web application, you don't put anything in conf/Catalina. These are basic mechanics that you find in tons of systems where a global configuration overrides a local one or the other way around (depending on what you see as global or local). It's just this copy-from-local-and-use-as-global-but-only-for-the-first-time that just raises questions with me. Sure, I can work around it. I can also use another servlet container or if I'm really desperate patch the Tomcat source code to not do this copying anymore. It's a free world and thankfully nothing is holding anyone to anything here. The reason for prolonging this discussion however is to hopefully get some light behind the rationale of doing this initial copying. Not true; the only reason it becomes an issue in your case is because you're not doing a proper redeployment. You cannot simply copy a new .war file into the appBase directory and expect it to work. I see, that's my bad then. But to my defense, it has worked for many years this way, the manual suggests this is the way to do this and my IDE (not the highest authority of course, but really used a lot and not some obscure little known one) also does it this way and basically nearly every place where I ever came across does it this way. Up until recently I had very little reason to assume this was not the correct way. Maybe the one who writes the Tomcat documentation could make this a little clearer? If I read the documentation here: http://tomcat.apache.org/tomcat-6.0-doc/deployer-howto.html it really says that just copying the exploded or compressed web application to the appBase location should work. It doesn't mention that officially this only works the first time and not the second time. i.e. shouldn't the following: If you are not interested in using the Tomcat Manager, or TCD, then you'll need to deploy your web applications statically to Tomcat, followed by a Tomcat startup. The location you deploy web applications to for this type of deployment is called the |appBase| which is specified per Host. You either copy a so-called /exploded web application/, i.e non-compressed, to this location, or a compressed web application resource .WAR file. Become If you are not interested in using the Tomcat Manager, or TCD, then you'll need to deploy your web applications statically to Tomcat, followed by a Tomcat startup. The location you deploy web applications to for this type of deployment is called the |appBase| which is specified per Host. You either copy a so-called /exploded web application/, i.e non-compressed, to this location, or a compressed web application resource .WAR file. *Warning: This only works the first time. For a redeployment you need to follow the following procedure [...]* Not trying to be smart or anything, just hoping to help out. Kind regards, Arjan Tijms -- It's a cult. If you've coded for any length of time, you've run across someone from this warped brotherhood. Their creed: if you can write complicated code, you must be good.
Re: Tomcat copies context.xml to conf/Catalina/localhost/app.xml, but never updates it?
Hi, Michael wrote: I always just copy my web application statically to Tomcat's webapps directory ($CATALINA_BASE/webapps) for the production server Insufficient, as Chuck has pointed out: use the manager app. Does that, basically, mean that the static deployment method is deprecated? If that's true, why doesn't the Tomcat documentation mention this? Currently the documentation says: If you are not interested in using the Tomcat Manager, or TCD, then you'll need to deploy your web applications statically to Tomcat, followed by a Tomcat startup. See: http://tomcat.apache.org/tomcat-6.0-doc/deployer-howto.html Reading this gives me the impression that I can use it. If I should not use the static deployment mechanism, shouldn't this entry be removed from the documentation? Arjan -- It's a cult. If you've coded for any length of time, you've run across someone from this warped brotherhood. Their creed: if you can write complicated code, you must be good. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat copies context.xml to, conf/Catalina/localhost/app.xml,but neverupdates it?
Len Wrote: Unfortunately it's not that simple. Take for example the most common case, a Resource definition for a JDBC database connection. The app writer has to provide part of the definition (the resource name, e.g. jdbc/myAppDB) and the sysadmin has to provide another part (the address of the DB server). When the application vendor is different from the installer/maintainer, the context.xml has to be customized at install time. That's true and I agree that it's handy that Tomcat has the ability to copy the context.xml file from the war archive and allow that to be edited. It fits the situation you describe nicely. However, in a pure development situation, where I'm dealing with a local Tomcat installation of which I'm my own system administrator and an exploded deployment of which I'm both the developer and the user at that point, this mechanism doesn't make a lot of sense and in fact is more of a liability than an asset. To me it all would make a lot more sense of there was an option to disable this copying-and-holding of the context.xml. For exploded development mode, this option could then be set to false and for production mode it could be set to true. Arjan -- It's a cult. If you've coded for any length of time, you've run across someone from this warped brotherhood. Their creed: if you can write complicated code, you must be good. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat copies context.xml to conf/Catalina/localhost/app.xml but never updates it?
p.s. M4N - Arjan Tijms wrote: I'm running Tomcat 6.0.20 using JDK 6 update 14 on a 64 bits Debian Lenny. I've unpacked a stock Tomcat 6.0.20 and deployed a web module to the webapps directory. After I started and stopped the web module, I noticed that the META-INF/context.xml had been copied to conf/Catalina/localhost/[name of web module].xml. When I try to make the [tomcat install dir]/conf read only, I get the following exception when starting up: INFO: Starting Coyote HTTP/1.1 on http-8080 Aug 6, 2009 6:52:07 PM org.apache.jk.common.ChannelSocket init INFO: JK: ajp13 listening on /0.0.0.0:8009 Aug 6, 2009 6:52:07 PM org.apache.jk.server.JkMain start INFO: Jk running ID=0 time=0/19 config=null Aug 6, 2009 6:52:07 PM org.apache.catalina.startup.Catalina start INFO: Server startup in 137 ms Aug 6, 2009 6:53:17 PM org.apache.catalina.startup.HostConfig deployDirectory SEVERE: Error deploying web application directory my_test java.io.FileNotFoundException: /opt/apache-tomcat-6.0.20/conf/Catalina/localhost/my_test.xml (No such file or directory) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.init(FileOutputStream.java:179) at java.io.FileOutputStream.init(FileOutputStream.java:131) at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:957) at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:909) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:495) at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1274) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:296) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1337) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1601) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1610) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1590) at java.lang.Thread.run(Thread.java:619) It thus appears that Tomcat really wants to make this copy first and then process only this copy. I however very much want this not to happen. Is there any option that allows me to change this? What I need is the Tomcat 6.0.18 and earlier behavior where META-INF/context.xml is used from the webapps/someapp directory as-is, without any copying of any kind. Thanks in advance for all help -- It's a cult. If you've coded for any length of time, you've run across someone from this warped brotherhood. Their creed: if you can write complicated code, you must be good. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat copies context.xml to conf/Catalina/localhost/app.xml but never updates it?
Hi, I'm running Tomcat 6.0.20 using JDK 6 update 14 on a 64 bits Debian Lenny. I've unpacked a stock Tomcat 6.0.20 and deployed a web module to the webapps directory. After I started and stopped the web module, I noticed that the META-INF/context.xml had been copied to conf/Catalina/localhost/[name of web module].xml. I made a change to META-INF/context.xml and started Tomcat again, but the copy in conf/Catalina/localhost/ appeared to be not updated. I stopped Tomcat, removed the web module from webapps and started Tomcat again. The copy in conf/Catalina/localhost still wasn't updated/deleted and now Tomcat started complaining it couldn't find my webapp anymore. I've been a long time Tomcat user and this has never ever been a problem for me. I never noticed conf/Catalina/localhost contained a copy of my context.xml files, but I could always modify context.xml and never needed to explicitly delete the contents of conf/Catalina/localhost. This is a rather scary situation. I'm not 100% sure why Tomcat does this copying in the first place and I'm not at all sure what's going on with the stale copies. Anyone has any idea? -- It's a cult. If you've coded for any length of time, you've run across someone from this warped brotherhood. Their creed: if you can write complicated code, you must be good. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: RE: Occasional NoClassDefFoundError, but disappears after couple of restarts
Hi, Caldarale, Charles R wrote: The problem I have is that occasionally after I restart Tomcat, the startup fails with NoClassDefFoundError or ClassNotFoundException for every servlet listener that's being added from web.xml. This kind of error can be caused by having the same class(es) in multiple locations in a given branch of the classloader tree, such as in both a webapp's WEB-INF/lib and in Tomcat's lib directory (or on the system classpath). Have you checked that? I removed the obvious suspect, a duplicate postgresql driver that appeared in both WEB-INF/lib and [tomcat install dir]/lib. Unfortunately this did not seem to help. Last night I needed to restart the production server for a configuration change and I needed 22(!) restarts for Tomcat to start up correctly. It really seems that there's a race condition going on somehow. I mean, if it was a structural problem then Tomcat would simply never start up, wouldn't it? These are the libraries the web app currently uses: In WEB-INF/lib slf4j-jdk14-1.5.2.jar slf4j-api-1.5.2.jar regexp.jar quartz-1.5.2.jar jnp-client.jar jbosssx-client.jar jbosssx-as-client.jar jboss-security-spi.jar jboss-remoting.jar jboss-logging-spi.jar jboss-j2ee.jar jboss-integration.jar jboss-ejb3-security-client.jar jboss-ejb3-proxy-client.jar jboss-ejb3-core-client.jar jboss-ejb3-common-client.jar jboss-common-core.jar jboss-client.jar jboss-aspect-jdk50-client.jar jboss-aop-client.jar hibernate-annotations.jar concurrent.jar commons-logging-1.0.4.jar commons-digester-1.6.jar commons-collections-3.1.jar commons-beanutils-1.7.0.jar In [tomcat install dir]/lib mail.jar activation.jar postgresql-8.3-603.jdbc3.jar The Jboss libraries are the Jboss client libraries and are needed to consume EJBs from a remote Jboss AS instance. Maybe there are some classes somewhere that conflict with those of Tomcat, but if that would be true it's still hard to understand why starting up fails in production but never on a local workstation. Anyone has any idea? Grtz, Arjan -- It's a cult. If you've coded for any length of time, you've run across someone from this warped brotherhood. Their creed: if you can write complicated code, you must be good.
Occasional NoClassDefFoundError, but disappears after couple of restarts
Hi, I'm running Tomcat 6.0.16 on a 32 bit Debian Etch using Java 6 update 12. The problem I have is that occasionally after I restart Tomcat, the startup fails with NoClassDefFoundError or ClassNotFoundException for every servlet listener that's being added from web.xml. If I restart Tomcat a couple of times, the exceptions mysteriously vanish and startup succeeds. I've seen the exact same exceptions happening when starting up another Tomcat instance I manage, which is a Tomcat 6.0.18 on a 64 bit Debian Lenny also using Java 6 update 12. The Tomcat 6.0.16 instance produces the following strack trace when starting up: Jul 23, 2009 11:32:41 PM org.apache.catalina.core.AprLifecycleListener init INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /opt/jdk1.6.0_12/jre/lib/i386/server:/opt/jdk1.6.0_12/jre/lib/i386:/opt/jdk1.6.0_12/jre/../lib/i386:/usr/java/packages/l ib/i386:/lib:/usr/lib Jul 23, 2009 11:32:41 PM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 506 ms Jul 23, 2009 11:32:41 PM org.apache.catalina.core.StandardService start INFO: Starting service Catalina Jul 23, 2009 11:32:41 PM org.apache.catalina.core.StandardEngine start INFO: Starting Servlet Engine: Apache Tomcat/6.0.16 Jul 23, 2009 11:32:42 PM org.apache.catalina.core.StandardContext addApplicationListener INFO: The listener com.xmx.cyberdyme.system.settings.ApplicationSettings is already configured for this context. The duplicate definition has been ignored. Jul 23, 2009 11:32:42 PM org.apache.catalina.core.StandardContext addApplicationListener INFO: The listener org.quartz.ee.servlet.QuartzInitializerListener is already configured for this context. The duplicate definition has been ignored. Jul 23, 2009 11:32:42 PM org.apache.catalina.core.StandardContext listenerStart SEVERE: Exception sending context initialized event to listener instance of class com.xmx.cyberdyme.system.settings.ApplicationSettings java.lang.NoClassDefFoundError: com/xmx/cyberdyme/system/settings/AbstractWebApplicationSettings at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:621) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124) at java.net.URLClassLoader.defineClass(URLClassLoader.java:260) at java.net.URLClassLoader.access$000(URLClassLoader.java:56) at java.net.URLClassLoader$1.run(URLClassLoader.java:195) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:252) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1275) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1206) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:169) at com.xmx.cyberdyme.system.settings.ApplicationSettings.contextInitialized(ApplicationSettings.java:52) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3843) at org.apache.catalina.core.StandardContext.start(StandardContext.java:4350) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:525) at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:924) at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:887) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:492) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1147) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:311) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053) at org.apache.catalina.core.StandardHost.start(StandardHost.java:719) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045) at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443) at org.apache.catalina.core.StandardService.start(StandardService.java:516) at org.apache.catalina.core.StandardServer.start(StandardServer.java:710) at org.apache.catalina.startup.Catalina.start(Catalina.java:578) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
RE: Occasional NoClassDefFoundError, but disappears after couple of restarts
Hi, This kind of error can be caused by having the same class(es) in multiple locations in a given branch of the classloader tree, such as in both a webapp's WEB-INF/lib and in Tomcat's lib directory (or on the system classpath). Have you checked that? Well, that's indeed a good suggestion. We added 3 libraries to [tomcat install dir]/lib, namely activation.jar, mail.jar and postgresql-8.3-603.jdbc3.jar. As it happens the web application's WEB-INF/lib does contain a postgresql-8.3-603.jdbc4.jar. This last .jar should of course not be there. I have indeed seen applications on Jboss AS fail due to duplicate class definitions. The odd thing is that on Jboss AS it would fail both on localhost and on the production server, but in the case of Tomcat it only fails on the production server. Also, these (nearly) duplicate jar files seem to have been there for some time and restarts only started recently to fail on a regular basis. A recent upgrade from JDK 6 update 6 to update 12 seems to be most likely candidate for causing the sudden change in behavior. I'll remove the duplicate .jar, and hopefully the next restarts will happen smoothly again. Thanks again for your reply. Arjan -- It's a cult. If you've coded for any length of time, you've run across someone from this warped brotherhood. Their creed: if you can write complicated code, you must be good. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat looses POST parameters
Hi, M4N - Arjan Tijms wrote: The problem I'm encountering is that for a percentage of the POST requests, Tomcat seems to loose all parameters. Our application uses a filter that logs the (first few characters of) post parameters. To follow up on this issue, I think that I have probably found the cause of this issue. In the AJP/1.3 connector configuration in server.xml, the connectionTimeout had been set to a very low value (200). In the catalina.[date].log this resulted in entries like: Aug 29, 2007 4:27:21 PM org.apache.jk.common.HandlerRequest invoke INFO: Unknown message 31 Aug 29, 2007 4:27:23 PM org.apache.jk.common.HandlerRequest invoke INFO: Unknown message 31 Aug 29, 2007 4:27:25 PM org.apache.jk.common.HandlerRequest invoke INFO: Unknown message 31 Aug 29, 2007 4:27:26 PM org.apache.jk.common.HandlerRequest invoke INFO: Unknown message 31 Except for message 31, there were also entries for Unknown message 8, 1, 19, 20, 22, 0, 29, 11 and 15, although 31 was the most common. I missed these messages earlier (which is stupid of me, I know). I compared the time stamps of these messages with those of the empty posts in the other log file and found out they matched exactly or almost exactly. Looking up the code of HandlerRequest it appears to switch on a byte, defined in org.apache.jk.common.AjpConstants. It actually switches on a subset of these. The default action is to print the Unknown message line and continue, which happens for the numbers I mentioned above. Perhaps the connector is closing the connection early, but tomcat somehow isn't signalled about that event and switches on a random byte from the message the AJP connector was sending. Anyway, the result is that the post request continues to be processed, but with missing request parameters. Setting the timeout value too low is of course our fault, but perhaps it would be better if Tomcat could just abort the request and throw some exception? Kind regards, Arjan Tijms -- It's a cult. If you've coded for any length of time, you've run across someone from this warped brotherhood. Their creed: if you can write complicated code, you must be good. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat looses POST parameters
Christopher Schultz wrote: Uh, I'm not really sure what you're saying, here. TCs auth code /will/ run before any of your code if you are using it. Of course, but that's exactly what I'm saying. If Tomcat's auth code runs first, than the possible error would be in Tomcat's code. Do you implement authentication and authorization as a filter? Or, do you have something else going on? I'm asking because it's possible your own authentication mechanism has a bug or two that is somehow killing the request parameters. The custom authentication method is filter based. I installed the filter that prints the request parameters to the log as the very first filter in the chain; this means it gets called before any other filter does. So, the logging is the very first piece of my code that gets called when processing a single request. Like I said earlier: in the AJP connector the parameters are there and at the first opportunity that my own code can run they are gone. They get lost somewhere in between. Since none of my code runs in between, it's hard to say what happens. I might try building a debug Tomcat that does some extra logging at key points in the request handling, but for a high volume production server this would really be my last resort. Kind regards, Arjan Tijms -- It's a cult. If you've coded for any length of time, you've run across someone from this warped brotherhood. Their creed: if you can write complicated code, you must be good. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat looses POST parameters
Hi, Martin Gainty wrote: Fiddler lets you construct the request http://www.fiddlertool.com/fiddler/ Thanks for your suggestion, but I already tried that: Using a proxy server to monitor what my browser was sending, I clearly saw in the raw HTTP headers that parameters where being send In my case I was using Burpproxy, which seems to do more or less the same as fiddler. Grtz, Arjan Tijms - Original Message - From: thebugslayer [EMAIL PROTECTED] To: Tomcat Users List users@tomcat.apache.org Sent: Sunday, August 26, 2007 8:06 PM Subject: Re: Tomcat looses POST parameters I would suggest a TCP sniff tool like one found from axis or grinder to peek at your http track to be sure. On 8/24/07, M4N - Arjan Tijms [EMAIL PROTECTED] wrote: Hi, We're hosting a fairly high traffic web application based on Tomcat. It's running on Debian-Etch, JDK 5.0U10 and Tomcat 5.5.20. We're using Apache as a front-end with the AJP connector. The problem I'm encountering is that for a percentage of the POST requests, Tomcat seems to loose all parameters. Our application uses a filter that logs the (first few characters of) post parameters. This filter is installed as the first one in the filter chain, so nothing else can interfere with it. For requests originating from pages which logically can not produce such an empty post request, the log clearly shows there are no parameters. The problem is often fairly random, although I have been able to consistently reproduce it on one occasion. Using a proxy server to monitor what my browser was sending, I clearly saw in the raw HTTP headers that parameters where being send, yet they weren't received in Tomcat. I also enabled TCP/IP packet logging at the server for a while. For requests that appeared with empty parameters in Tomcat, the tcp/ip log showed the parameters did arrive at the server. Next to that I enabled debug logging in the AJP connector, and again the POST parameters were in the HTTP request but not present when the mentioned filter logged the request in Tomcat. I did notice though that the overwhelming majority of the empty post requests concerned Faces requests (we're using MyFaces 1.1.4). We store state on client, so typical Faces HTTP post requests are at least 22KB in size. Nevertheless, thousands of requests from the same pages from all kinds of different browsers arrive with the post parameters intact. I'm at a loss here how to proceed. Naturally I could change JSF to keep state on server, but because of the way some custom components work that's currently not an option. It would also not really solve the underlying problem of course. Any help would be greatly appreciated Kind regards, Arjan Tijms -- It's a cult. If you've coded for any length of time, you've run across someone from this warped brotherhood. Their creed: if you can write complicated code, you must be good. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- /bugslayer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- It's a cult. If you've coded for any length of time, you've run across someone from this warped brotherhood. Their creed: if you can write complicated code, you must be good. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat looses POST parameters
Hi, thebugslayer wrote: I would suggest a TCP sniff tool like one found from axis or grinder to peek at your http track to be sure. Thanks for your reply. The TCP Sniffer from The Grinder is an HTTP proxy, and I already tried that: Using a proxy server to monitor what my browser was sending, I clearly saw in the raw HTTP headers that parameters where being send, yet they weren't received in Tomcat. As I wrote in my original mail, I traced the problem two levels deeper; both at the TCP/IP level at the server and in the connector between Apache and Tomcat. Post parameters are still there, but are gone once in Tomcat. Grtz, Arjan Tijms On 8/24/07, M4N - Arjan Tijms [EMAIL PROTECTED] wrote: Hi, We're hosting a fairly high traffic web application based on Tomcat. It's running on Debian-Etch, JDK 5.0U10 and Tomcat 5.5.20. We're using Apache as a front-end with the AJP connector. The problem I'm encountering is that for a percentage of the POST requests, Tomcat seems to loose all parameters. Our application uses a filter that logs the (first few characters of) post parameters. This filter is installed as the first one in the filter chain, so nothing else can interfere with it. For requests originating from pages which logically can not produce such an empty post request, the log clearly shows there are no parameters. The problem is often fairly random, although I have been able to consistently reproduce it on one occasion. Using a proxy server to monitor what my browser was sending, I clearly saw in the raw HTTP headers that parameters where being send, yet they weren't received in Tomcat. I also enabled TCP/IP packet logging at the server for a while. For requests that appeared with empty parameters in Tomcat, the tcp/ip log showed the parameters did arrive at the server. Next to that I enabled debug logging in the AJP connector, and again the POST parameters were in the HTTP request but not present when the mentioned filter logged the request in Tomcat. I did notice though that the overwhelming majority of the empty post requests concerned Faces requests (we're using MyFaces 1.1.4). We store state on client, so typical Faces HTTP post requests are at least 22KB in size. Nevertheless, thousands of requests from the same pages from all kinds of different browsers arrive with the post parameters intact. I'm at a loss here how to proceed. Naturally I could change JSF to keep state on server, but because of the way some custom components work that's currently not an option. It would also not really solve the underlying problem of course. Any help would be greatly appreciated Kind regards, Arjan Tijms -- It's a cult. If you've coded for any length of time, you've run across someone from this warped brotherhood. Their creed: if you can write complicated code, you must be good. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- It's a cult. If you've coded for any length of time, you've run across someone from this warped brotherhood. Their creed: if you can write complicated code, you must be good. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat looses POST parameters
Hehl, Thomas wrote: I've read other responses and don't know much more about what to tell you. It seems to be the first order of business is to figure out how to consistently re-create the problem. Indeed. We've been trying that for a couple of weeks now but no luck yet. As of now the only thing we can do is indeed fire a high load on the system, and trace back in the log which instances went wrong. The seemingly randomness of the problem is making this quite hard. For instance, at one moment the problem seemed to be reproducable on a certain page in a certain account. However when I plugged the laptop with which I could reproduce the problem to an internet connection at another location, the problem could not be reproduced. If this doesn't re-create the issue, then the problem is probably outside of tomcat's realm. I'm not sure. Even if it would be MyFaces that sometimes causes invalid POST requests to happen, it's Tomcat that drops them without giving any warning or error message. Don't forget, before the request hits the MyFaces filter, we're still in 'plain Tomcat' (for lack of a better term). At the top of the filter chain I installed a servlet filter that only prints the post parameters. There is no MyFaces or anything else active at that point. Kind regards, Arjan Tijms Something's going on in your MyFaces or some such. Sorry not more help. -Original Message- From: M4N - Arjan Tijms [mailto:[EMAIL PROTECTED] Sent: Friday, August 24, 2007 12:43 PM To: users@tomcat.apache.org Subject: Tomcat looses POST parameters Hi, We're hosting a fairly high traffic web application based on Tomcat. It's running on Debian-Etch, JDK 5.0U10 and Tomcat 5.5.20. We're using Apache as a front-end with the AJP connector. The problem I'm encountering is that for a percentage of the POST requests, Tomcat seems to loose all parameters. Our application uses a filter that logs the (first few characters of) post parameters. This filter is installed as the first one in the filter chain, so nothing else can interfere with it. For requests originating from pages which logically can not produce such an empty post request, the log clearly shows there are no parameters. The problem is often fairly random, although I have been able to consistently reproduce it on one occasion. Using a proxy server to monitor what my browser was sending, I clearly saw in the raw HTTP headers that parameters where being send, yet they weren't received in Tomcat. I also enabled TCP/IP packet logging at the server for a while. For requests that appeared with empty parameters in Tomcat, the tcp/ip log showed the parameters did arrive at the server. Next to that I enabled debug logging in the AJP connector, and again the POST parameters were in the HTTP request but not present when the mentioned filter logged the request in Tomcat. I did notice though that the overwhelming majority of the empty post requests concerned Faces requests (we're using MyFaces 1.1.4). We store state on client, so typical Faces HTTP post requests are at least 22KB in size. Nevertheless, thousands of requests from the same pages from all kinds of different browsers arrive with the post parameters intact. I'm at a loss here how to proceed. Naturally I could change JSF to keep state on server, but because of the way some custom components work that's currently not an option. It would also not really solve the underlying problem of course. Any help would be greatly appreciated Kind regards, Arjan Tijms -- It's a cult. If you've coded for any length of time, you've run across someone from this warped brotherhood. Their creed: if you can write complicated code, you must be good. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat looses POST parameters
Hi, Christopher Schultz wrote: The problem I'm encountering is that for a percentage of the POST requests, Tomcat seems to loose all parameters. Are you observing this in log files, or by actually doing it and having it fail. Both actually. I have installed a top-level filter, and the very first thing it does when its doFilter method is being called is calling request.getParameterMap(), iterating over its values, and logging the name and first chars of each parameter. Logging happens to a standard java.Util.logging.Logger. Before this filter is being called, none of my own code has been executed in the context of the current request. After that, if the page required a specific parameter an error is being generated. This indeed happens. Other pages ignore the request if parameters are missing. I have observed this too. I'm wondering if something is happening to the saved request during the login. It's a good thought, and such a thing could have happened. Only in this case the parameters are already missing before any request processing is being done. Also, 99.9% of the requests execute well, only in 0.1 orso the parameters are missing. That doesn't sound like much but 0.1% of a high traffic site is still a lot of requests. (I made the 0.1 number up and haven't calculated it exactly, but it's a small number) Kind regards, Arjan Tijms Just looking for other possibilities. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFG0vPr9CaO5/Lv0PARAkCpAKCjmbuGvcENYdmmeJ0I64AJuBsjlgCgri5D 9BOdjvmdWbarm3liy6EHjfk= =GbyQ -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- It's a cult. If you've coded for any length of time, you've run across someone from this warped brotherhood. Their creed: if you can write complicated code, you must be good. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat looses POST parameters
Hi, Ben Souther wrote: One thing that you've go on your side, as far as isolating the problem is concerned, is that Tomcat is a busy project and has thousands of users. If it had a bug that caused it to miss or drop form parameters hundreds or thousands of apps would be breaking right now and this, the dev list, and bugzilla would be getting flooded with reports. I absolutely agree with you. It's the exact same line of reasoning that went on during our internal discussions about this problem. However, the thing is that by far not all post parameters are dropped. I happened to stumble about it at first when inspecting the log files manually for some other problems. There was a log line that showed an empty post was done, for a page of which I knew this was impossible. I did some grepping on the logs and discovered a few hundred of occasions where it happened. Between thousands upon thousands of requests something like this might go unnoticed for a long time. When post parameters are dropped, most users would either see nothing happening and simply click again, or some error page comes up, they use the back button and click again. The fact that they're not, is a pretty good indicator telling you that the problem is not with Tomcat. If it is a problem with Tomcat, you're doing something very rare to get this bug to surface. Indeed. I wonder what that could be. We're running on a quad core server. Maybe it's some race condition? Kind regards, Arjan Tijms -- It's a cult. If you've coded for any length of time, you've run across someone from this warped brotherhood. Their creed: if you can write complicated code, you must be good. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat looses POST parameters
Hi, c. wrote: I experienced some problems with missing POST parameters. I'm not sure where the problem lay but I was using apache 2, tomcat 5.5, mod_ajp_proxy and dwr. I upgraded to tomcat 6 using mod_proxy and NIO and have not seen the problem since. Chris, this sounds very interesting and is consistent with what I understood about possible race problems in Tomcat 5.5 being (perhaps) fixed in Tomcat 6. Unfortunately I can not upgrade a live production server overnight, certainly not with a whole major version, but I'll definitely will push for an early upgrade. In the mean time I will try if 5.5.23 or better the soon to be released 5.5.25 makes any difference. Grtz, Arjan Tijms -- It's a cult. If you've coded for any length of time, you've run across someone from this warped brotherhood. Their creed: if you can write complicated code, you must be good. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat looses POST parameters
Hi, Christopher Schultz wrote: No, your filter will only be executed after the login has been handled, since Tomcat's authentication is done using a valve, which is processed before any filters. I see. The thing is, I'm not using any container managed authentication at all. All authentication is handled by the application. But even if I used the container's (Tomcat's) authentication, then it still would be Tomcat's code than ran before my code, right? Grtz, Arjan Tijms -- It's a cult. If you've coded for any length of time, you've run across someone from this warped brotherhood. Their creed: if you can write complicated code, you must be good. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Tomcat looses POST parameters
Hi, We're hosting a fairly high traffic web application based on Tomcat. It's running on Debian-Etch, JDK 5.0U10 and Tomcat 5.5.20. We're using Apache as a front-end with the AJP connector. The problem I'm encountering is that for a percentage of the POST requests, Tomcat seems to loose all parameters. Our application uses a filter that logs the (first few characters of) post parameters. This filter is installed as the first one in the filter chain, so nothing else can interfere with it. For requests originating from pages which logically can not produce such an empty post request, the log clearly shows there are no parameters. The problem is often fairly random, although I have been able to consistently reproduce it on one occasion. Using a proxy server to monitor what my browser was sending, I clearly saw in the raw HTTP headers that parameters where being send, yet they weren't received in Tomcat. I also enabled TCP/IP packet logging at the server for a while. For requests that appeared with empty parameters in Tomcat, the tcp/ip log showed the parameters did arrive at the server. Next to that I enabled debug logging in the AJP connector, and again the POST parameters were in the HTTP request but not present when the mentioned filter logged the request in Tomcat. I did notice though that the overwhelming majority of the empty post requests concerned Faces requests (we're using MyFaces 1.1.4). We store state on client, so typical Faces HTTP post requests are at least 22KB in size. Nevertheless, thousands of requests from the same pages from all kinds of different browsers arrive with the post parameters intact. I'm at a loss here how to proceed. Naturally I could change JSF to keep state on server, but because of the way some custom components work that's currently not an option. It would also not really solve the underlying problem of course. Any help would be greatly appreciated Kind regards, Arjan Tijms -- It's a cult. If you've coded for any length of time, you've run across someone from this warped brotherhood. Their creed: if you can write complicated code, you must be good. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
