Re: Can jasper2's validateXml be added back, at least for the lifetime of Tomcat 7?
Got it. Much appreciated! On 2/21/14, 1:11 AM, Mark Thomas ma...@apache.org wrote: On 21/02/2014 01:44, mark_desp...@mcafee.com wrote: Thank you for the quick reply, Mark. Would it be possible to get a sense of which release the validateXml attribute might get added back? E.g. 7.0.53? It will be 7.0.53. See: http://svn.us.apache.org/repos/asf/tomcat/tc7.0.x/trunk/webapps/docs/chang elog.xml I did a quick search through the bugs, but it does not seem to be tracked. Would it be worth submitting something? To be perfectly honest, no. The problem is already fixed. The Tomcat developers don't create a bug report for every issue. We tend to use Bugzilla as a list of things still to fix so if someone reports a bug we know we aren't going to fix straight away, we'll ask them to create a Bugzilla entry for it so it doesn't get lost. In this case all that would happen is that you'd create it and I'd immediately close it as fixed. Mark Thanks! Mark D. On 2/20/14, 12:21 AM, Mark Thomas ma...@apache.org wrote: On 20/02/2014 07:46, mark_desp...@mcafee.com wrote: Hi everyone, My project recently tried upgrading to Tomcat 7.0.52 and has become bit by the renaming of jasper2¹s ³validateXml² to ³validateTld², as described on the thread below. This change has made it more difficult pick up the latest Tomcat 7¹s security fixes since the change breaks build scripts for quite a few projects maintained by different teams. Would it be possible to add back support for ³validateXml² as part of the next minor release of Tomcat 7, as more or less described by Mark Thomas on that thread? http://mail-archives.apache.org/mod_mbox/tomcat-users/201401.mbox/%3Ccd 84 922d-b159-49bd-9c11-6aaa5cb97...@email.android.com%3E This would save us a lot of effort, and we would then plan on switching to the new attribute name as part of adopting Tomcat 8. Thank you for any assistance on this! It looks like that attribute was going to get added back already as, digging into another issue reported on the dev list, Jasper in 7.0.x still parses web.xml in full so it needs to have separate control over validateXml. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Can jasper2's validateXml be added back, at least for the lifetime of Tomcat 7?
Thank you for the quick reply, Mark. Would it be possible to get a sense of which release the validateXml attribute might get added back? E.g. 7.0.53? I did a quick search through the bugs, but it does not seem to be tracked. Would it be worth submitting something? Thanks! Mark D. On 2/20/14, 12:21 AM, Mark Thomas ma...@apache.org wrote: On 20/02/2014 07:46, mark_desp...@mcafee.com wrote: Hi everyone, My project recently tried upgrading to Tomcat 7.0.52 and has become bit by the renaming of jasper2¹s ³validateXml² to ³validateTld², as described on the thread below. This change has made it more difficult pick up the latest Tomcat 7¹s security fixes since the change breaks build scripts for quite a few projects maintained by different teams. Would it be possible to add back support for ³validateXml² as part of the next minor release of Tomcat 7, as more or less described by Mark Thomas on that thread? http://mail-archives.apache.org/mod_mbox/tomcat-users/201401.mbox/%3Ccd84 922d-b159-49bd-9c11-6aaa5cb97...@email.android.com%3E This would save us a lot of effort, and we would then plan on switching to the new attribute name as part of adopting Tomcat 8. Thank you for any assistance on this! It looks like that attribute was going to get added back already as, digging into another issue reported on the dev list, Jasper in 7.0.x still parses web.xml in full so it needs to have separate control over validateXml. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Can jasper2's validateXml be added back, at least for the lifetime of Tomcat 7?
Hi everyone, My project recently tried upgrading to Tomcat 7.0.52 and has become bit by the renaming of jasper2’s “validateXml” to “validateTld”, as described on the thread below. This change has made it more difficult pick up the latest Tomcat 7’s security fixes since the change breaks build scripts for quite a few projects maintained by different teams. Would it be possible to add back support for “validateXml” as part of the next minor release of Tomcat 7, as more or less described by Mark Thomas on that thread? http://mail-archives.apache.org/mod_mbox/tomcat-users/201401.mbox/%3ccd84922d-b159-49bd-9c11-6aaa5cb97...@email.android.com%3E This would save us a lot of effort, and we would then plan on switching to the new attribute name as part of adopting Tomcat 8. Thank you for any assistance on this! Sincerely, Mark DeSpain
RE: SSL Mysterious Self Signed Certificate
Can you clarify on mysterious self-signed certificate displayed within the browser? Also, into what did you import the relevant root certs and SSL cert? The keystore? W is right. If your certificate is was not issued (signed) by a CA that the browser trusts, then the browser will not trust your certificate and will show a warning as a result. If that is your issue, then in order to get that message to go away, you'll either need use a certificate issued by a trusted CA, or import your certificate information into the browser. ~Mark -Original Message- From: Jonathan Mast [mailto:jhmast.develo...@gmail.com] Sent: Thursday, May 07, 2009 9:59 AM To: Tomcat Users List Subject: Re: SSL Mysterious Self Signed Certificate Its my understanding that all Self-signed certs generate the creepy browser messages. Not sure though. Were the imported root certs issued by a well known CA? On Wed, May 6, 2009 at 10:43 PM, Andrews, Wayne wayne.andr...@sap.comwrote: Hi I have an issue whereby on a windows installation of Tomcat; I have a mysterious seflt signed certificate displayed within the browser. Despite the fact that I have created a new keystore and imported the relevant root certs and SSL cert and then redirected server.xml to point to the keystore Any ideas?: W. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Headstart on Resolving OOM-PermGen errors on webapp reload
Yeah, Insane just using reflection and a graph traversal algorithm to get the job done. It looks like this is implemented by org.netbeans.insane.impl.InsaneEngine. Oh, and I found my copy of the Insane source. The third argument to ScannerUtils.scan() should be true since that is what signals to InsaneEngine that static fields should be traversed during the heap walk. ~Mark -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Wednesday, April 22, 2009 9:05 AM To: Tomcat Users List Subject: Re: Headstart on Resolving OOM-PermGen errors on webapp reload -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Mark, On 4/21/2009 10:27 PM, mark_desp...@mcafee.com wrote: Ok, so my wife actually wrote a couple of month ago in Japanese about using strategy for leveraging the Insane library and a continuous integration server in order to prevent webapp classloader leakage issues from creeping in. I'll definitely take a look at this (in English -- tell her thanks!). With this in place, you can then setup your test environment to exercise a given webapp, shut it down, and then invoke your ScannerUtils code to see if that the webapp's classloader is still hanging around. This is super sexy! What a nice job. I'll have to read-up on the Insane library, but my suspicion is that you probably don't really need it... all the RTTI information is available from the objects themselves, and the code should be relatively simple just tons and tons of loops and recursive calls. A word of warning... this is a very heavy weight operation. Heh, you think? That's why this type of testing should be done in development and not in production ; - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAknvQCkACgkQ9CaO5/Lv0PC5OwCeONLPIu7BAaBiwGhEbuYm4caf d/4An2TpoymWDAi2/o4fi/sRwNpqxROy =sL8m -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Headstart on Resolving OOM-PermGen errors on webapp reload
I don't doubt that jmap/jhat would be able to give you more detailed information. My exact goal was to come up with something for automated testing that would help prevent classloader leaks from making it into production. If someone can think of a programmatic way to do that with jmap/jhat, please share! Mark -Original Message- From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com] Sent: Wednesday, April 22, 2009 10:30 AM To: Tomcat Users List Subject: RE: Headstart on Resolving OOM-PermGen errors on webapp reload From: mark_desp...@mcafee.com [mailto:mark_desp...@mcafee.com] Subject: RE: Headstart on Resolving OOM-PermGen errors on webapp reload Yeah, Insane just using reflection and a graph traversal algorithm to get the job done. It looks like this is implemented by org.netbeans.insane.impl.InsaneEngine. Other than being programmable for automated testing purposes, does this provide any more or different information than a jmap/jhat combo? - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
RE: Headstart on Resolving OOM-PermGen errors on webapp reload
Hi Ken (and Mark), Different Mark here... I'm new to this mailing list and am not a Tomcat developer. Forgive me if I'm interrupting your thread, but I am very interested in this topic, since I've spent a fair amount of time debugging OOM-PermGen errors within Tomcat (5.5.x). I would be interested in hearing any tips or tricks regarding OM-PermGen, too, and am happy to share my experiences regarding this. None of the issues I've looked into have never been attributed to Tomcat. They have all been application issues related to an inability to garbage collect a webapp's classloader. This root causes were split between one of the following: * A webapp registering an object with another object that outlives the webapp and forgetting to unregister it webapp shutdown. As a result, that object cannot be garbage collected, which also prevents that webapp's classloader from being garbage collected. Examples of things that outlive a webapp or objects from libraries within Tomcat's shared library folder, Tomcat itself, and the JRE itself. For example, if a webapp registers a JDBC driver or a JCA/JCE provider, it should take care to unregister it on shutdown. * Instantiating a new Thread somehow (e.g. directly via its constructor, or indirectly via a thread pool, a Timer, or a ScheduledExecutorService) in response to a request to a webapp. This is because, by default, the new Thread inherits the parent thread's context classloader, which in Tomcat (5.5.x) is set to that webapp's classloader. If that Thread keeps the context classloader reference and outlives the webapp, the webapp's classloader cannot be garbage collected. As an aside, those causes highlight a reason why people should be weary of putting a new JAR files into one of Tomcat's shared library folders. Doing so requires that a webapp be vigilant about cleaning up any interactions with that library on shutdown. Said library also needs to provide a means to do the cleanup, as well as be sensitive to context classloader issues. When trying to debug these issues, I typically load things up in a profiler, stop the webapp and see of its classloader still resides in memory. When it exists, I use the profiler's find the garbage collection root to help determine why the classloader still resides in memory and figure out how to fix the issue. If using JProfiler (and probably other profilers), I think you have to tell it to go not stop when it reaches the Class object along the reference path. To help pro-actively detect webapp classloader garbage collection issues, I've leveraged the Insane library (a library I found from Netbeans while researching the topic) to write a utility that searches for webapp classloaders that should have been garbage collected. Using the utility in combination with automated tests has been definitely helped catch and diagnose issues. Apologies if this wasn't helpful. Please let me know if I'm wrong, outdated, or if there is a better way! Best regards, Mark DeSpain -Original Message- From: Ken Bowen [mailto:kbo...@als.com] Sent: Tuesday, April 21, 2009 1:33 PM To: Tomcat Users List Subject: Headstart on Resolving OOM-PermGen errors on webapp reload Mark, Any chance we could make a headstart on Resolving OOM-PermGen errors on webapp reload ?? Perhaps some general pointers, guidance etc. [to help you refine the talk in advance :-) ] The manager app is giving me more more of: FAIL - Application at context path /ctx could not be started FAIL - Encountered exception java.lang.OutOfMemoryError: PermGen space This is on a Tomcat 6.0.18 (java 1.5) running on a remote Linux CentOS 5, in a Parallels VM, with httpd + mod_jk, running two versions of the app. Total memory available is 432 MB, with a number of non-system, non-TC processes, including ActiveMQ and 9 Postres processes. But there appears to be about 200MB of unused memory according to the Parallels control panel. I develop locally on a Mac OS/X 10.5.6 box with 4GBmem using (My)Eclipse and Tomcat 6.0.18 directly. With lots lots of reloads, I'm not surprised that I eventually hit OOM PermGen space in this setting, but it happens much less often than on the CentOS box (of course 4GB != 432MB). Thanks...Ken Mark Thomas wrote: I was at a conference recently and rather rashly made the statement OOM PermGen errors on reload are an application issue, not a Tomcat one. Several people came up to me afterwards with variations on a theme of Our app has OOM on reload - put your money where your mouth is and show us where we have gone wrong Having helped several people track down the causes of these errors (and yes they were all application issues) it occurred to me that there may be interest in this at ApacheCon. So my suggestion is: Title:Resolving OOM-PermGen errors on webapp reload Type: Session Abstract: What causes the error. Typical root causes. How to find
RE: Headstart on Resolving OOM-PermGen errors on webapp reload
Hi Chris, I'll follow up later tonight. Hopefully I'll have less typos then, but don't be surprised if I just go triple-negative instead :) ~Mark -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Tuesday, April 21, 2009 3:47 PM To: Tomcat Users List Subject: Re: Headstart on Resolving OOM-PermGen errors on webapp reload -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Mark, On 4/21/2009 6:17 PM, mark_desp...@mcafee.com wrote: None of the issues I've looked into have never been attributed to Tomcat. You mean ever attributed to Tomcat, right? Good. ;) * A webapp registering an object with another object that outlives the webapp and forgetting to unregister it webapp shutdown. As a result, that object cannot be garbage collected, which also prevents that webapp's classloader from being garbage collected. It's worse than that: even if most of the objects created from classes loaded by that ClassLoader have gone out of scope and can be GC'd, the java.lang.Class, java.lang.Package, and java.lang.reflect.* objects that are created from those loaded classes won't be released until the ClassLoader is released. So if you have a single object that is being held by an object loaded by another ClassLoader, you can have potentially hundreds or thousands of objects left around in memory just wasting space. * Instantiating a new Thread somehow (e.g. directly via its constructor, or indirectly via a thread pool, a Timer, or a ScheduledExecutorService) in response to a request to a webapp. This is because, by default, the new Thread inherits the parent thread's context classloader, which in Tomcat (5.5.x) is set to that webapp's classloader. This is a very good point to consider, because many folks aren't aware of the thread-classloader relationship. If you don't kill your threads, you will keep your whole app's ClassLoader around forever. The 'daemonness' of the thread is not relevant. I'm looking at you, Quartz! To help pro-actively detect webapp classloader garbage collection issues, I've leveraged the Insane library (a library I found from Netbeans while researching the topic) to write a utility that searches for webapp classloaders that should have been garbage collected. Using the utility in combination with automated tests has been definitely helped catch and diagnose issues. Could you post some more info on this? I'm sure a lot of folks would be interested in this kind of thing. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAknuTNYACgkQ9CaO5/Lv0PCN6wCgkDKwhzpRB7re4StuClVe1Rt/ 3K0Anj5eXjLiTql97dxbhrNFavPXGIvC =Iuos -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Headstart on Resolving OOM-PermGen errors on webapp reload
Ok, so my wife actually wrote a couple of month ago in Japanese about using strategy for leveraging the Insane library and a continuous integration server in order to prevent webapp classloader leakage issues from creeping in. If you can read Japanese, check out http://pdxwhitebox.blog118.fc2.com/blog-entry-33.html For those of us who can't read Japanese, today she posted the English text it was based upon here: http://pdxwhitebox.blog118.fc2.com/blog-entry-161.html The information there is a little light on the specifics, but it does a great job at describing the strategy, the rationale, where to find Insane, and which API to use. When you read it, treat plugins as interchangeable with webapps. Getting a little more specific, we leveraged Insane's ScannerUtils.scan() method to search for webapp classloaders that should not be reachable via a strong reference. Doing this involved (1) implementing a org.netbeans.insane.scanner.Visitor that gathers relevant classloaders and (2) implementing a org.netbeans.insane.scanner.Filter that causes weak/soft/phantom references to be skipped over by ScannerUtils.scan(). (1) To implement the Visitor class, just have visitObject(ObjectMap map, Object object) collect relevant classloaders passed as the second argument (probably instances of org.apache.catalina.loader.WebappClassLoader). All other methods can have empty implementations. (2) To implement the Filter class, have accept(Object obj, Object referredFrom, Field reference) return false if the second argument is an instance of SoftReference, WeakReference, or PhantomReference. Return true for everything else. With the Visitor and Filter implemented, the scanning code would look something like the following: SetObject roots = ScannerUtils.interestingRoots(); // add to this set anything else you'd like to make sure gets visits MyWebAppClassLoaderGatherer visitor = new MyWebAppClassLoaderGatherer(); MySkipWeakReferencesFilter filter = new MySkipWeakReferencesFilter(); ScannerUtils.scan( filter, visitor, roots, true ); // do whatever is needed with the classloaders gathered by the visitor Unfortunately, I don't remember why the last argument to ScannerUtils.scan() should be true, and I don't have the Insane source or javadoc handy to remind me. Sorry about that... With this in place, you can then setup your test environment to exercise a given webapp, shut it down, and then invoke your ScannerUtils code to see if that the webapp's classloader is still hanging around. This seems to work well with in combination with a continuous integration (CI) server whose job is to run tests against checkins all day. If your CI server tracks who checked it what, you'll be able to quickly narrow down what may have caused the leak, assuming that your web app wasn't leaking a classloader to begin with. A word of warning... this is a very heavy weight operation. If there are better ways this can be done, I'd love to hear about it! Best regards, Mark DeSpain -Original Message- From: Despain, Mark Sent: Tuesday, April 21, 2009 4:01 PM To: users@tomcat.apache.org Subject: RE: Headstart on Resolving OOM-PermGen errors on webapp reload Hi Chris, I'll follow up later tonight. Hopefully I'll have less typos then, but don't be surprised if I just go triple-negative instead :) ~Mark -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Tuesday, April 21, 2009 3:47 PM To: Tomcat Users List Subject: Re: Headstart on Resolving OOM-PermGen errors on webapp reload -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Mark, On 4/21/2009 6:17 PM, mark_desp...@mcafee.com wrote: None of the issues I've looked into have never been attributed to Tomcat. You mean ever attributed to Tomcat, right? Good. ;) * A webapp registering an object with another object that outlives the webapp and forgetting to unregister it webapp shutdown. As a result, that object cannot be garbage collected, which also prevents that webapp's classloader from being garbage collected. It's worse than that: even if most of the objects created from classes loaded by that ClassLoader have gone out of scope and can be GC'd, the java.lang.Class, java.lang.Package, and java.lang.reflect.* objects that are created from those loaded classes won't be released until the ClassLoader is released. So if you have a single object that is being held by an object loaded by another ClassLoader, you can have potentially hundreds or thousands of objects left around in memory just wasting space. * Instantiating a new Thread somehow (e.g. directly via its constructor, or indirectly via a thread pool, a Timer, or a ScheduledExecutorService) in response to a request to a webapp. This is because, by default, the new Thread inherits the parent thread's context classloader, which in Tomcat (5.5.x) is set to that webapp's
