Re: Large webapps in 7.0.29 and Annotation Scanning not respecting metadata-complete
Andreas Pålsson wpt2...@hotmail.com wrote: I have been thinking about this issue all day, and I think the specification needs more work on this area. There is certainly no need for a container to wade through hundreds maybe even thousands of classes in search of something that does not exist. It is an unnecessary task which consumes both time and resources. I am all for making it easier for a developer, and make the container scan everything by default. But there is also an obvious need for a solution to specifically define where a container should look for annotations when we have web applications with very large amount of classes. Even if it has to be an XML-file, such as in Spring where you can instruct the framework where to look for beans, entities, controllers and such. Mark, is there a way we can influence the EG to change this? Thanks. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org The bad news is that this is a 3.0 feature so we are way past being able to influence it. Personally I dislike most of the 3.0 pluggability featutes. If you have ideas how to improve things then the EG lists Rainer pointed you to is the place to take them. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Large webapps in 7.0.29 and Annotation Scanning not respecting metadata-complete
On 01/08/2012 02:53, andreas palsson wrote: Good morning. After upgrading to 7.0.29, one of my larger webapps could no longer be deployed due to OutOfMemoryError. As far as I know, this is related to the new feature of annotation scanning which was enabled in v7.0.29 (see the stack trace below). Have anyone else experienced the same issue? There are multiple issues going on here. There have been reported previously and are understood. I'll try and explain the full story. The servlet spec requires (and the expert group (EG) confirmed) that Tomcat always scans for ServletContainerInitializers (SCIs), even if metadata-complete is true. Tomcat was changed to do this in 7.0.29. SCIs can have @HandlesTypes annotations. If one is present, Tomcat must scan every class and its hierarchy looking for matches to the HandlesTypes. Tomcat caches the results of this scan to speed up processing so it doesn't repeatedly scan the same classes. If A B both inherit from C, when Tomcat scans A it also scans C and caches the result so it doesn't have to scan C again when checking B. Tomcat was caching more than it needed to making the cache excessively large. This is bug 53535 which has already been fixed and will be in 7.0.30 onwards. This then raises the question of how to turn of scanning for classes that match HandlesTypes. While the spec says that JARs excluded using an absolute-ordering are not scanned for matches to HandlesTypes it was unclear what is meant to happen when metadata-complete is true since ordering is ignored in that case. The EG has recently clarified this and stated that JARs excluded using an absolute-ordering are not scanned for matches to HandlesTypes regardless of the setting of metadata-complete. The discussion is not yet complete and we are waiting for it to conclude and before implementing the results of that discussion in 7.0.30. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Large webapps in 7.0.29 and Annotation Scanning not respecting metadata-complete
Thank you Mark for explaining this. I'll see if I can get my hands on a 7.0.30 and see how it works out. One thing though, would it be possible to allow Tomcat to _only_ scan for SCI's in WEB-INF/classes and exclude WEB-INF/lib (even for parent classes inherited in the former)? It would be great to have some control over the SCI's as some could be located in any of the dependencies I have added to the application. // andreas Date: Wed, 1 Aug 2012 08:51:04 +0100 From: ma...@apache.org To: users@tomcat.apache.org Subject: Re: Large webapps in 7.0.29 and Annotation Scanning not respecting metadata-complete There are multiple issues going on here. There have been reported previously and are understood. I'll try and explain the full story. [...] Mark
Re: Large webapps in 7.0.29 and Annotation Scanning not respecting metadata-complete
On 01/08/2012 09:35, andreas palsson wrote: Thank you Mark for explaining this. I'll see if I can get my hands on a 7.0.30 and see how it works out. You'll need to wait until all the issues have been fixed and then build from trunk. Keep an eye on the dev list. One thing though, would it be possible to allow Tomcat to _only_ scan for SCI's in WEB-INF/classes and exclude WEB-INF/lib (even for parent classes inherited in the former)? It would be great to have some control over the SCI's as some could be located in any of the dependencies I have added to the application. No. You can exclude SCIs in JARs through absolute ordering but you can't stop Tomcat scanning WEB-INF/classes for matching types. Mark // andreas Date: Wed, 1 Aug 2012 08:51:04 +0100 From: ma...@apache.org To: users@tomcat.apache.org Subject: Re: Large webapps in 7.0.29 and Annotation Scanning not respecting metadata-complete There are multiple issues going on here. There have been reported previously and are understood. I'll try and explain the full story. [...] Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Large webapps in 7.0.29 and Annotation Scanning not respecting metadata-complete
Mark Thomas wrote: On 01/08/2012 02:53, andreas palsson wrote: Good morning. After upgrading to 7.0.29, one of my larger webapps could no longer be deployed due to OutOfMemoryError. As far as I know, this is related to the new feature of annotation scanning which was enabled in v7.0.29 (see the stack trace below). Have anyone else experienced the same issue? There are multiple issues going on here. There have been reported previously and are understood. I'll try and explain the full story. The servlet spec requires (and the expert group (EG) confirmed) that Tomcat always scans for ServletContainerInitializers (SCIs), even if metadata-complete is true. Tomcat was changed to do this in 7.0.29. SCIs can have @HandlesTypes annotations. If one is present, Tomcat must scan every class and its hierarchy looking for matches to the HandlesTypes. Tomcat caches the results of this scan to speed up processing so it doesn't repeatedly scan the same classes. If A B both inherit from C, when Tomcat scans A it also scans C and caches the result so it doesn't have to scan C again when checking B. Tomcat was caching more than it needed to making the cache excessively large. This is bug 53535 which has already been fixed and will be in 7.0.30 onwards. This then raises the question of how to turn of scanning for classes that match HandlesTypes. While the spec says that JARs excluded using an absolute-ordering are not scanned for matches to HandlesTypes it was unclear what is meant to happen when metadata-complete is true since ordering is ignored in that case. The EG has recently clarified this and stated that JARs excluded using an absolute-ordering are not scanned for matches to HandlesTypes regardless of the setting of metadata-complete. The discussion is not yet complete and we are waiting for it to conclude and before implementing the results of that discussion in 7.0.30. Philosophical note : Without for a minute questioning the technical competence and intellectual brilliance of the actors involved, for me the above perfectly illustrates a phenomenon that is common in the computer software industry (and in lawmaking and government). Start with something that is reasonably simple and works reasonably well. It is a good bet that over time - and no doubt with the best intentions in the world - this thing will be the subject of amendments and additional features which will gradually transform it into a monster which no single human person is able to comprehend and dominate fully anymore. Simultaneously and perversely, the increasing complication of the thing will auto-generate a whole ecosystem of specialists, whose sole social purpose will be to comprehend parts of the monster and explain it to lesser humans. And since they derive their income of doing this, they will have all the right incentives to keep on adding amendments and features, indirectly allowing their own population to prosper and multiply. Logically but unfortunately, it rarely seems to happen that all this energy would be focused in the other direction, which could be described as simplify and optimise. Over the last 5 years, at constant price, the sheer speed of computer hardware and networks must have increased by a factor 10 or more. Yet, it still takes about the same elapsed time to process the average HTTP request and display its results on a workstation's desktop, and it still takes about the same elapsed time (and cost) to write a simple web application. In fact, if one really takes into account the time needed to absorb the technical material required to be able to write a webapp, and keep current with this material, it probably takes more people, time and money now than 5 years ago. Note that in the above, I am only using Tomcat as an example because this is a Tomcat list, and only using Mark's excellent explanation as a trigger for my rambling. I am aware that annotations are a Java feature, not a Tomcat feature. And I can imagine that Java annotations do bring some benefits. But when their implementation in Tomcat brings along the need of scanning hundreds of files and using significantly more memory and time to start Tomcat, and when avoiding this extra time and memory usage itself requires further changes and additional parameters and code, I believe that something is amiss. Not in Tomcat, but deeper. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Large webapps in 7.0.29 and Annotation Scanning not respecting metadata-complete
On 01.08.2012 09:51, Mark Thomas wrote: On 01/08/2012 02:53, andreas palsson wrote: Good morning. After upgrading to 7.0.29, one of my larger webapps could no longer be deployed due to OutOfMemoryError. As far as I know, this is related to the new feature of annotation scanning which was enabled in v7.0.29 (see the stack trace below). Have anyone else experienced the same issue? There are multiple issues going on here. There have been reported previously and are understood. I'll try and explain the full story. The servlet spec requires (and the expert group (EG) confirmed) that Tomcat always scans for ServletContainerInitializers (SCIs), even if metadata-complete is true. Tomcat was changed to do this in 7.0.29. SCIs can have @HandlesTypes annotations. If one is present, Tomcat must scan every class and its hierarchy looking for matches to the HandlesTypes. Tomcat caches the results of this scan to speed up processing so it doesn't repeatedly scan the same classes. If A B both inherit from C, when Tomcat scans A it also scans C and caches the result so it doesn't have to scan C again when checking B. Tomcat was caching more than it needed to making the cache excessively large. This is bug 53535 which has already been fixed and will be in 7.0.30 onwards. This then raises the question of how to turn of scanning for classes that match HandlesTypes. While the spec says that JARs excluded using an absolute-ordering are not scanned for matches to HandlesTypes it was unclear what is meant to happen when metadata-complete is true since ordering is ignored in that case. The EG has recently clarified this and stated that JARs excluded using an absolute-ordering are not scanned for matches to HandlesTypes regardless of the setting of metadata-complete. The discussion is not yet complete and we are waiting for it to conclude and before implementing the results of that discussion in 7.0.30. Would *.jar in jarsToSkip (catalina.properties) be a workaround, or is that skip list also ignored for SCI / @HandlesTypes processing? Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Large webapps in 7.0.29 and Annotation Scanning not respecting metadata-complete
I have been thinking about this issue all day, and I think the specification needs more work on this area. There is certainly no need for a container to wade through hundreds maybe even thousands of classes in search of something that does not exist. It is an unnecessary task which consumes both time and resources. I am all for making it easier for a developer, and make the container scan everything by default. But there is also an obvious need for a solution to specifically define where a container should look for annotations when we have web applications with very large amount of classes. Even if it has to be an XML-file, such as in Spring where you can instruct the framework where to look for beans, entities, controllers and such. Mark, is there a way we can influence the EG to change this? Thanks. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Large webapps in 7.0.29 and Annotation Scanning not respecting metadata-complete
On 01.08.2012 18:10, Andreas Pålsson wrote: I have been thinking about this issue all day, and I think the specification needs more work on this area. There is certainly no need for a container to wade through hundreds maybe even thousands of classes in search of something that does not exist. It is an unnecessary task which consumes both time and resources. I am all for making it easier for a developer, and make the container scan everything by default. But there is also an obvious need for a solution to specifically define where a container should look for annotations when we have web applications with very large amount of classes. Even if it has to be an XML-file, such as in Spring where you can instruct the framework where to look for beans, entities, controllers and such. Mark, is there a way we can influence the EG to change this? I'm not Mark and don't want to preempt any answer from him, but there is the following public discussion available: http://java.net/jira/browse/SERVLET_SPEC-36 and http://java.net/jira/browse/SERVLET_SPEC-36 Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Large webapps in 7.0.29 and Annotation Scanning not respecting metadata-complete
On 01.08.2012 18:52, Rainer Jung wrote: Mark, is there a way we can influence the EG to change this? I'm not Mark and don't want to preempt any answer from him, but there is the following public discussion available: http://java.net/jira/browse/SERVLET_SPEC-36 and http://java.net/jira/browse/SERVLET_SPEC-36 Grrr, the second link should have been: http://java.net/projects/servlet-spec/lists/jsr340-experts/archive/2012-07/message/28 Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Large webapps in 7.0.29 and Annotation Scanning not respecting metadata-complete
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 André, On 8/1/12 5:55 AM, André Warnier wrote: Without for a minute questioning the technical competence and intellectual brilliance of the actors involved, for me the above perfectly illustrates a phenomenon that is common in the computer software industry (and in lawmaking and government). Start with something that is reasonably simple and works reasonably well. It is a good bet that over time - and no doubt with the best intentions in the world - this thing will be the subject of amendments and additional features which will gradually transform it into a monster which no single human person is able to comprehend and dominate fully anymore. Tomcat can't read email, yet, so we're not done. http://en.wikipedia.org/wiki/Jamie_Zawinski#Zawinski.27s_law_of_software_envelopment - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlAZlqMACgkQ9CaO5/Lv0PCxaQCgo1ZYy3e+V9hAFYDjTRYYZeoE UYMAn31/kuV0hfHtR2iUCHaNz6qReK2m =vZGH -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Large webapps in 7.0.29 and Annotation Scanning not respecting metadata-complete
Good morning. After upgrading to 7.0.29, one of my larger webapps could no longer be deployed due to OutOfMemoryError. As far as I know, this is related to the new feature of annotation scanning which was enabled in v7.0.29 (see the stack trace below). Have anyone else experienced the same issue? This new behaviour can supposedly be turned off by setting 'metadata-complete=true' in 'web.xml', however that seems not be working as intended. Temporarily I can solve it by increasing memory for 7.0.29, but to me this is still a bug considering it works fine in 7.0.28. :) Following is the stack trace, system, java version, command line, the list of JAR-files from /lib and head of 'web.xml'. Thanks. SEVERE: Error waiting for multi-thread deployment of WAR files to complete java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) at java.util.concurrent.FutureTask.get(FutureTask.java:111) at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:752) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:472) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1413) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:313) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119) at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90) at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:401) at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:346) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:1140) at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:785) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1559) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1549) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: java.lang.OutOfMemoryError: Java heap space at java.io.DataInputStream.readUTF(DataInputStream.java:661) at java.io.DataInputStream.readUTF(DataInputStream.java:564) at org.apache.tomcat.util.bcel.classfile.ConstantUtf8.init(ConstantUtf8.java:48) at org.apache.tomcat.util.bcel.classfile.Constant.readConstant(Constant.java:129) at org.apache.tomcat.util.bcel.classfile.ConstantPool.init(ConstantPool.java:60) at org.apache.tomcat.util.bcel.classfile.ClassParser.readConstantPool(ClassParser.java:209) at org.apache.tomcat.util.bcel.classfile.ClassParser.parse(ClassParser.java:119) at org.apache.catalina.startup.ContextConfig.processAnnotationsStream(ContextConfig.java:2066) at org.apache.catalina.startup.ContextConfig.processAnnotationsJar(ContextConfig.java:1942) at org.apache.catalina.startup.ContextConfig.processAnnotationsUrl(ContextConfig.java:1908) at org.apache.catalina.startup.ContextConfig.processAnnotations(ContextConfig.java:1893) at org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1296) at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:855) at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:346) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119) at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5173) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:618) at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:963) at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1600) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ... 5 more Linux mybox 3.4.6-2.fc17.i686 #1 SMP Thu Jul 19 22:15:33 UTC 2012 i686 i686 i386 GNU/Linux java version 1.7.0_05 Java(TM) SE Runtime Environment (build
