Re: Jar scanning, SCI scanning, fragment scanning
On 14/06/2013 03:31, Christopher Schultz wrote: It might be nice if this were not a one-of-many decision, but if a client could choose more than one type. A bit-mask or a list of scan-types would be nice. I could see the same type of scanner being used for multiple different purposes. That is what ServletContainerIniiializers are for. They only handle class files but the other types (e.g. TLDs) are always in known locations and are inexpensive to find. The expensive bit is scanning all the classes and SCIs provide a mechanism to do that only once for all interested parties. Another option would be to have a scanner that would scan for everything and then notify registered listeners. If you only care about TLDs, you just register a TLD listener. For classes, you register a class listener. A listener-based scanner would even open the doors for a webapp to be able to register its own listeners to sniff scanner events during context startup. That is exactly what SCIs do. I know this all sounds way over-engineered, but there are quite a few webapps that, when they first launch, go ahead and scan the entire reachable class loader hierarchy for classes with annotations, etc. so you get this environment where the same classes are being scanner over and over and over again just because they have to be scanned in different places for different reasons. Then the libraries doing the scanning need to be updated to use the container provided scanning mechanism that is already in place. I spoke with David Blevins about this way back in Vancouver: applications that endlessly perform JAR scanning just because the components can't seem to coordinate with each other. If there were a scanner component that could allow these unrelated components to share infrastructure, we could all save a lot of time. Such a scanning component could, much like the Digester, graduate from Tomcat and become more widely useful. I really don't see the need for this at all given that SCIs are available. If there are use cases that SCIs don't meet then extending SCIs is the way to go so the same feature is available in any container. Mark - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Jar scanning, SCI scanning, fragment scanning
i think all containers actually hack this part since it doesn't match the need when you are just a little bit bigger than tomcat. today almost all apps scans (even spring ;) so would be great to get a nicer scanner. *Romain Manni-Bucau* *Twitter: @rmannibucau https://twitter.com/rmannibucau* *Blog: **http://rmannibucau.wordpress.com/*http://rmannibucau.wordpress.com/ *LinkedIn: **http://fr.linkedin.com/in/rmannibucau* *Github: https://github.com/rmannibucau* 2013/6/14 Mark Thomas ma...@apache.org On 14/06/2013 03:31, Christopher Schultz wrote: It might be nice if this were not a one-of-many decision, but if a client could choose more than one type. A bit-mask or a list of scan-types would be nice. I could see the same type of scanner being used for multiple different purposes. That is what ServletContainerIniiializers are for. They only handle class files but the other types (e.g. TLDs) are always in known locations and are inexpensive to find. The expensive bit is scanning all the classes and SCIs provide a mechanism to do that only once for all interested parties. Another option would be to have a scanner that would scan for everything and then notify registered listeners. If you only care about TLDs, you just register a TLD listener. For classes, you register a class listener. A listener-based scanner would even open the doors for a webapp to be able to register its own listeners to sniff scanner events during context startup. That is exactly what SCIs do. I know this all sounds way over-engineered, but there are quite a few webapps that, when they first launch, go ahead and scan the entire reachable class loader hierarchy for classes with annotations, etc. so you get this environment where the same classes are being scanner over and over and over again just because they have to be scanned in different places for different reasons. Then the libraries doing the scanning need to be updated to use the container provided scanning mechanism that is already in place. I spoke with David Blevins about this way back in Vancouver: applications that endlessly perform JAR scanning just because the components can't seem to coordinate with each other. If there were a scanner component that could allow these unrelated components to share infrastructure, we could all save a lot of time. Such a scanning component could, much like the Digester, graduate from Tomcat and become more widely useful. I really don't see the need for this at all given that SCIs are available. If there are use cases that SCIs don't meet then extending SCIs is the way to go so the same feature is available in any container. Mark --**--**- To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.**orgdev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Jar scanning, SCI scanning, fragment scanning
On 14/06/2013 08:18, Romain Manni-Bucau wrote: i think all containers actually hack this part since it doesn't match the need when you are just a little bit bigger than tomcat. What requirements can't be met with an SCI? Mark - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Jar scanning, SCI scanning, fragment scanning
On 14/06/2013 04:05, Nick Williams wrote: On Jun 13, 2013, at 11:23 AM, Mark Thomas wrote: As I make my way through the Servlet 3.1 spec reviewing all the changes to make sure they have all been implemented, I have reached section 8.2. There are a number of related issues here. 1. Three known issues with the current SCI / fragment handling [1] - SCI scan does not obey class loader ordering - fragments in container JARs are processed - Container SCIs may be exluded 2. A wish for per context jarsToSkip [2] 3. A wish for greater control over jarsToSkip [3] These issues are all inter-related. Fixing them is going to require either some very ugly hacks or changes to the JarScanner API. I have started down the route of the JAR scanner API changes. The overall idea is: - jarsToSkip is replaced by a JarScanFilter with the default implementation doing what jarsToSkip currently does I would request doing what jarsToSkip currently does plus what is wanted with jarsToScan in 3. If it wasn't clear from my original e-mail, that is exactly what these changes are intended to do. It shouldn't be extra complicated or require re- or duplicate-configuration of anything to whitelist one JAR. It is in no-ones interest to make anything more complicated than it needs to be. If a user wants to whitelist one JAR they are going to have to reconfigure something. Tomcat can't read minds. I agree duplicating configuration is something to be avoided where practical. Also, it's very important that log4j-core*.jar and log4j-taglib*.jar be whitelisted by default. I agree we don't want to exclude those JARs from scanning by default. Whitelisting isn't the only way of doing that. A less broad entry in the skip list would achieve the same result. That said, using whitelisting for these would be a nice example to users of how it works. Mark - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Jar scanning, SCI scanning, fragment scanning
wait SCIs stands for ServletContainerInitializer right? if so just think all apps are not only webapps (ear typically) + the issue is really tomcat scanning shouldn't occur if already done (or the opposite, myfaces should reuse tomcat scanned info is here, cxf, spring, cdi container...). *Romain Manni-Bucau* *Twitter: @rmannibucau https://twitter.com/rmannibucau* *Blog: **http://rmannibucau.wordpress.com/*http://rmannibucau.wordpress.com/ *LinkedIn: **http://fr.linkedin.com/in/rmannibucau* *Github: https://github.com/rmannibucau* 2013/6/14 Mark Thomas ma...@apache.org On 14/06/2013 08:18, Romain Manni-Bucau wrote: i think all containers actually hack this part since it doesn't match the need when you are just a little bit bigger than tomcat. What requirements can't be met with an SCI? Mark --**--**- To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.**orgdev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Jar scanning, SCI scanning, fragment scanning
On 14/06/2013 08:35, Romain Manni-Bucau wrote: wait SCIs stands for ServletContainerInitializer right? if so just think all apps are not only webapps (ear typically) I'd disagree. I'd say WARs are far more popular than EARs. For EARs if there is a case for application wide scanning (not sure there is but I don't know the rest of the J2EE spec well enough to be certain) then such scanning needs to be provided by the container in the same way SCIs provide scanning in Servlet containers. In a full J2EE container it should be possible for the results of the application wide scan to be fed to the SCI so there is no need for multiple scans. + the issue is really tomcat scanning shouldn't occur if already done (or the opposite, myfaces should reuse tomcat scanned info is here, cxf, spring, cdi container...). Then MyFaces should use an SCI. As should any other library that needs to scan the WAR for classes and/or annotations in a Servlet container. If there is something one of these libraries needs that can't be done with an SCI then that needs to be raised with the Servlet EG. The downside of using an SCI is that it requires Servlet 3. Some library developers may not like the idea of introducing a dependency on Servlet 3.0. I'd suggest the way forward is to use an SCI if available and if not, then fall-back to whatever internal scanning is currently used. Mark - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Jar scanning, SCI scanning, fragment scanning
that's what we do in TomEE but i don't get how a SCI (order is not deterministic IIRC) can solve the issue. *Romain Manni-Bucau* *Twitter: @rmannibucau https://twitter.com/rmannibucau* *Blog: **http://rmannibucau.wordpress.com/*http://rmannibucau.wordpress.com/ *LinkedIn: **http://fr.linkedin.com/in/rmannibucau* *Github: https://github.com/rmannibucau* 2013/6/14 Mark Thomas ma...@apache.org On 14/06/2013 08:35, Romain Manni-Bucau wrote: wait SCIs stands for ServletContainerInitializer right? if so just think all apps are not only webapps (ear typically) I'd disagree. I'd say WARs are far more popular than EARs. For EARs if there is a case for application wide scanning (not sure there is but I don't know the rest of the J2EE spec well enough to be certain) then such scanning needs to be provided by the container in the same way SCIs provide scanning in Servlet containers. In a full J2EE container it should be possible for the results of the application wide scan to be fed to the SCI so there is no need for multiple scans. + the issue is really tomcat scanning shouldn't occur if already done (or the opposite, myfaces should reuse tomcat scanned info is here, cxf, spring, cdi container...). Then MyFaces should use an SCI. As should any other library that needs to scan the WAR for classes and/or annotations in a Servlet container. If there is something one of these libraries needs that can't be done with an SCI then that needs to be raised with the Servlet EG. The downside of using an SCI is that it requires Servlet 3. Some library developers may not like the idea of introducing a dependency on Servlet 3.0. I'd suggest the way forward is to use an SCI if available and if not, then fall-back to whatever internal scanning is currently used. Mark --**--**- To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.**orgdev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Jar scanning, SCI scanning, fragment scanning
On 14/06/2013 08:54, Romain Manni-Bucau wrote: that's what we do in TomEE but i don't get how a SCI (order is not deterministic IIRC) can solve the issue. You have yet to articulate exactly what the issue is. You are correct that SCI processing order is not fully deterministic. Container SCIs are under the control of the container so can be processed in any order the container sees fit so that part can be deterministic. Application SCIs do not have an order defined. If that is a problem, raise it with the EG. I'd suggest the simple solution (which Tomcat already implements) is to treat JARs without a fragment as if it contains an empty fragment and then process SCIs in the same order as fragments are processed. Mark - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Jar scanning, SCI scanning, fragment scanning
well i think it will be handled for next spec as it was for interceptors (but it is not that important for initializer in *applications*). I don't get what's the issue using a scanner registry with get or create semantic. Is it only tomcat doesn't need it by itself? If so my point is today tomcat is almost nothing without libs and most of libs scan so IMO it would be a nice have to get it (even if not mandatory). *Romain Manni-Bucau* *Twitter: @rmannibucau https://twitter.com/rmannibucau* *Blog: **http://rmannibucau.wordpress.com/*http://rmannibucau.wordpress.com/ *LinkedIn: **http://fr.linkedin.com/in/rmannibucau* *Github: https://github.com/rmannibucau* 2013/6/14 Mark Thomas ma...@apache.org On 14/06/2013 08:54, Romain Manni-Bucau wrote: that's what we do in TomEE but i don't get how a SCI (order is not deterministic IIRC) can solve the issue. You have yet to articulate exactly what the issue is. You are correct that SCI processing order is not fully deterministic. Container SCIs are under the control of the container so can be processed in any order the container sees fit so that part can be deterministic. Application SCIs do not have an order defined. If that is a problem, raise it with the EG. I'd suggest the simple solution (which Tomcat already implements) is to treat JARs without a fragment as if it contains an empty fragment and then process SCIs in the same order as fragments are processed. Mark --**--**- To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.**orgdev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Jar scanning, SCI scanning, fragment scanning
On 13/06/2013 17:23, Mark Thomas wrote: As I make my way through the Servlet 3.1 spec reviewing all the changes to make sure they have all been implemented, I have reached section 8.2. There are a number of related issues here. 1. Three known issues with the current SCI / fragment handling [1] - SCI scan does not obey class loader ordering - fragments in container JARs are processed - Container SCIs may be exluded 2. A wish for per context jarsToSkip [2] 3. A wish for greater control over jarsToSkip [3] snip/ I currently have the start of this in a local git repo. Any objections to me committing what I have to trunk and continuing there? No objections and some good discussion that I think will be helped if there is something concrete to discuss so I have committed the work so far. Issues 2 3 have been implemented and the information (container vs application JAR) is now available to implement 1 but there is still work required to resolve the issues identified in 1. Mark [1] http://markmail.org/message/22e3sjmsy2gt4tkw [2] https://issues.apache.org/bugzilla/show_bug.cgi?id=54083 [3] https://issues.apache.org/bugzilla/show_bug.cgi?id=54770 - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Jar scanning, SCI scanning, fragment scanning
On 14/06/2013 09:14, Romain Manni-Bucau wrote: well i think it will be handled for next spec as it was for interceptors (but it is not that important for initializer in *applications*). I don't get what's the issue using a scanner registry with get or create semantic. Is it only tomcat doesn't need it by itself? If so my point is today tomcat is almost nothing without libs and most of libs scan so IMO it would be a nice have to get it (even if not mandatory). The issue is that you are advocating a solution (a scanner registry although you have not defined what one of those is) without describing a problem. Both container and application components that need to scan JARs for classes and/or annotations can use an SCI which will result in a single scan being performed. You have yet to articulate a requirement that cannot be met with an SCI. If the problem you are trying to solve is that lots of libraries perform scans then I see no point in implementing a proprietary solution (a scanner registry) when there is a standard solution (SCIs) available. Libraries are going to have to change their code for either solution so using the specification defined standard solution rather than a single proprietary solution used by only one container is clearly the way to go. Mark - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Jar scanning, SCI scanning, fragment scanning
SCIs are too webapp linked to be usable by others, that's the point so it needs to be proprietary ATM + SCI doesn't scan eveything (thinking to cdi you cannot guess a bean is a cdi one before having analyzed it). *Romain Manni-Bucau* *Twitter: @rmannibucau https://twitter.com/rmannibucau* *Blog: **http://rmannibucau.wordpress.com/*http://rmannibucau.wordpress.com/ *LinkedIn: **http://fr.linkedin.com/in/rmannibucau* *Github: https://github.com/rmannibucau* 2013/6/14 Mark Thomas ma...@apache.org On 14/06/2013 09:14, Romain Manni-Bucau wrote: well i think it will be handled for next spec as it was for interceptors (but it is not that important for initializer in *applications*). I don't get what's the issue using a scanner registry with get or create semantic. Is it only tomcat doesn't need it by itself? If so my point is today tomcat is almost nothing without libs and most of libs scan so IMO it would be a nice have to get it (even if not mandatory). The issue is that you are advocating a solution (a scanner registry although you have not defined what one of those is) without describing a problem. Both container and application components that need to scan JARs for classes and/or annotations can use an SCI which will result in a single scan being performed. You have yet to articulate a requirement that cannot be met with an SCI. If the problem you are trying to solve is that lots of libraries perform scans then I see no point in implementing a proprietary solution (a scanner registry) when there is a standard solution (SCIs) available. Libraries are going to have to change their code for either solution so using the specification defined standard solution rather than a single proprietary solution used by only one container is clearly the way to go. Mark --**--**- To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.**orgdev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Jar scanning, SCI scanning, fragment scanning
On 14/06/2013 09:28, Romain Manni-Bucau wrote: SCIs are too webapp linked to be usable by others, that's the point so it needs to be proprietary ATM If the problem you are trying to solve is JAR scanning outside of the Servlet container then that - frankly - isn't a Tomcat problem. Tomcat is concerned about scanning for classes and annotations within a web application and the standard API for that is an SCI. SCIs are just an interface so if a broader scanning solution was available (either in Java EE or Java SE) then that broader solution could be taken advantage of either directly in Tomcat or by whatever product was embedding Tomcat. If TomEE has a broader solution and it could be easier to plug the results of that into Tomcat so make them available through the SCI interface then present some concrete proposals on changes to make the scanning more pluggable and they'll be considered. + SCI doesn't scan eveything (thinking to cdi you cannot guess a bean is a cdi one before having analyzed it). SCIs don't scan anything. SCIs are merely the interface through which clients declare their interests and the servlet container passes back the results. It is certainly the case that every class has to be scanned if a @HandlesType is present on any SCI. If you are saying that CDI implementations need information that isn't available though the SCI interface then that is something that needs to be raised with the Servlet EG. Mark - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Jar scanning, SCI scanning, fragment scanning
can work this way, today that's not as easy as it could to replace tomat scanning(s) (tld, classes) by tomee one. About CDI: you basically need to pass all classes on jars with a beans.xml to the CDI container so @HandlesType doesn't match this need really well. *Romain Manni-Bucau* *Twitter: @rmannibucau https://twitter.com/rmannibucau* *Blog: **http://rmannibucau.wordpress.com/*http://rmannibucau.wordpress.com/ *LinkedIn: **http://fr.linkedin.com/in/rmannibucau* *Github: https://github.com/rmannibucau* 2013/6/14 Mark Thomas ma...@apache.org On 14/06/2013 09:28, Romain Manni-Bucau wrote: SCIs are too webapp linked to be usable by others, that's the point so it needs to be proprietary ATM If the problem you are trying to solve is JAR scanning outside of the Servlet container then that - frankly - isn't a Tomcat problem. Tomcat is concerned about scanning for classes and annotations within a web application and the standard API for that is an SCI. SCIs are just an interface so if a broader scanning solution was available (either in Java EE or Java SE) then that broader solution could be taken advantage of either directly in Tomcat or by whatever product was embedding Tomcat. If TomEE has a broader solution and it could be easier to plug the results of that into Tomcat so make them available through the SCI interface then present some concrete proposals on changes to make the scanning more pluggable and they'll be considered. + SCI doesn't scan eveything (thinking to cdi you cannot guess a bean is a cdi one before having analyzed it). SCIs don't scan anything. SCIs are merely the interface through which clients declare their interests and the servlet container passes back the results. It is certainly the case that every class has to be scanned if a @HandlesType is present on any SCI. If you are saying that CDI implementations need information that isn't available though the SCI interface then that is something that needs to be raised with the Servlet EG. Mark --**--**- To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.**orgdev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Jar scanning, SCI scanning, fragment scanning
Mark, On 6/14/13 3:16 AM, Mark Thomas wrote: On 14/06/2013 03:31, Christopher Schultz wrote: It might be nice if this were not a one-of-many decision, but if a client could choose more than one type. A bit-mask or a list of scan-types would be nice. I could see the same type of scanner being used for multiple different purposes. That is what ServletContainerIniiializers are for. Well, crap. I had never seen those before. I'm curious, though. Thinking about this the other day with a colleague who was complaining about some kind of Spring configuration that evidently loaded every class available on the classpath and kept them in memory (thus leading to heap and PermGen issues), I concluded that the only sane way to do this kind of probing would be to probe everything in a ClassLoader that was intended to be discarded after the probing as to avoid loading classes unnecessarily. If an SCI retains a reference to any of the Class objects in the SetClass parameter, that hypothetical throw-away ClassLoader is no longer thrown-away. They only handle class files but the other types (e.g. TLDs) are always in known locations and are inexpensive to find. The expensive bit is scanning all the classes and SCIs provide a mechanism to do that only once for all interested parties. Thanks for pointing that out. I actually had no idea. I really don't see the need for this at all given that SCIs are available. If there are use cases that SCIs don't meet then extending SCIs is the way to go so the same feature is available in any container. I think you are absolutely right. My idea was born before I knew that SCIs existed. Time to re-read the API now that I'm actually starting to update all my webapps towards 3.0. Thanks, -chris signature.asc Description: OpenPGP digital signature
Re: Jar scanning, SCI scanning, fragment scanning
Romain,
On 6/14/13 5:38 AM, Romain Manni-Bucau wrote:
can work this way, today that's not as easy as it could to replace tomat
scanning(s) (tld, classes) by tomee one.
About CDI: you basically need to pass all classes on jars with a beans.xml
to the CDI container so @HandlesType doesn't match this need really well.
It's a pathological case, but there's nothing stopping you from doing this:
@HandlesType('java.lang.Object')
That way, you'll get everything. Then you can scan the Class objects to
your heart's content looking for whatever you need. CDI can be
implemented this way: just replace the library JAR scanner with an SCI
that responds as if it had scanned all the classes itself.
I have to imaging any JAR scanner would be written with a callback
interface, so just loop-over all the Classes in the Set you get and
fire-off those call-backs: you get the scanning for free already done by
Tomcat. Set a global-ish flag in the library saying that scanning has
been done and stop.
Then your fall-back scanner runs just in case SCIs aren't available
(pre-3.0 container for example -- the Servlet EG recently clarified that
even a webapp with a pre-3.0 deployment descriptor still needs to have
all its 3.0-type features fired which is a decision which seems
astoundingly foolish to me) -- check the aforementioned flag and skip
traditional scanning if you already got it for free.
The only thing missing is a standard scanner -- which was what I was
suggesting. If you implement a standard scanner and it's good, maybe
Tomcat will adopt it and wire it through SCIs because that's what
web containers do.
-chris
signature.asc
Description: OpenPGP digital signature
Re: Jar scanning, SCI scanning, fragment scanning
Cdi handle interfaces too which are not Object.
About throwing away classloader that's what we do in tomee.
Le 14 juin 2013 18:09, Christopher Schultz ch...@christopherschultz.net
a écrit :
Romain,
On 6/14/13 5:38 AM, Romain Manni-Bucau wrote:
can work this way, today that's not as easy as it could to replace tomat
scanning(s) (tld, classes) by tomee one.
About CDI: you basically need to pass all classes on jars with a
beans.xml
to the CDI container so @HandlesType doesn't match this need really well.
It's a pathological case, but there's nothing stopping you from doing this:
@HandlesType('java.lang.Object')
That way, you'll get everything. Then you can scan the Class objects to
your heart's content looking for whatever you need. CDI can be
implemented this way: just replace the library JAR scanner with an SCI
that responds as if it had scanned all the classes itself.
I have to imaging any JAR scanner would be written with a callback
interface, so just loop-over all the Classes in the Set you get and
fire-off those call-backs: you get the scanning for free already done by
Tomcat. Set a global-ish flag in the library saying that scanning has
been done and stop.
Then your fall-back scanner runs just in case SCIs aren't available
(pre-3.0 container for example -- the Servlet EG recently clarified that
even a webapp with a pre-3.0 deployment descriptor still needs to have
all its 3.0-type features fired which is a decision which seems
astoundingly foolish to me) -- check the aforementioned flag and skip
traditional scanning if you already got it for free.
The only thing missing is a standard scanner -- which was what I was
suggesting. If you implement a standard scanner and it's good, maybe
Tomcat will adopt it and wire it through SCIs because that's what
web containers do.
-chris
Re: Jar scanning, SCI scanning, fragment scanning
Romain,
On 6/14/13 12:13 PM, Romain Manni-Bucau wrote:
Cdi handle interfaces too which are not Object.
There are interfaces which are do not extend Object?
There is essentially a base interface that declares every method in
java.lang.Object. It's simply not possible to have an instance of an
interface that does not have a toString method, for example.
java.lang.Object.class.isAssignableFrom([any class or interface]) will
always return true.
So I would expect that @HandlesType('java.lang.Object') will work.
I have not actually tested it, though.
-chris
signature.asc
Description: OpenPGP digital signature
Re: Jar scanning, SCI scanning, fragment scanning
On 14/06/2013 16:57, Christopher Schultz wrote: Mark, On 6/14/13 3:16 AM, Mark Thomas wrote: On 14/06/2013 03:31, Christopher Schultz wrote: It might be nice if this were not a one-of-many decision, but if a client could choose more than one type. A bit-mask or a list of scan-types would be nice. I could see the same type of scanner being used for multiple different purposes. That is what ServletContainerIniiializers are for. Well, crap. I had never seen those before. I'm curious, though. Thinking about this the other day with a colleague who was complaining about some kind of Spring configuration that evidently loaded every class available on the classpath and kept them in memory (thus leading to heap and PermGen issues), I concluded that the only sane way to do this kind of probing would be to probe everything in a ClassLoader that was intended to be discarded after the probing as to avoid loading classes unnecessarily. If an SCI retains a reference to any of the Class objects in the SetClass parameter, that hypothetical throw-away ClassLoader is no longer thrown-away. The early Tomcat 7 implementations did it like this - loaded every class and then examined the class object. More recent implementations use BCEL to look at the byte code. It is faster and uses less memory. We also use caching to ensure each class is only processed once. Mark - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Jar scanning, SCI scanning, fragment scanning
Mark,
On 6/14/13 12:21 PM, Mark Thomas wrote:
On 14/06/2013 16:57, Christopher Schultz wrote:
Mark,
On 6/14/13 3:16 AM, Mark Thomas wrote:
On 14/06/2013 03:31, Christopher Schultz wrote:
It might be nice if this were not a one-of-many decision, but if a
client could choose more than one type. A bit-mask or a list of
scan-types would be nice. I could see the same type of scanner being
used for multiple different purposes.
That is what ServletContainerIniiializers are for.
Well, crap. I had never seen those before.
I'm curious, though. Thinking about this the other day with a colleague
who was complaining about some kind of Spring configuration that
evidently loaded every class available on the classpath and kept them in
memory (thus leading to heap and PermGen issues), I concluded that the
only sane way to do this kind of probing would be to probe everything in
a ClassLoader that was intended to be discarded after the probing as to
avoid loading classes unnecessarily.
If an SCI retains a reference to any of the Class objects in the
SetClass parameter, that hypothetical throw-away ClassLoader is no
longer thrown-away.
The early Tomcat 7 implementations did it like this - loaded every class
and then examined the class object. More recent implementations use BCEL
to look at the byte code. It is faster and uses less memory. We also use
caching to ensure each class is only processed once.
Cool. What Classloader gets used to actually load the Class objects,
though? In the pathological case of @HandlesType('java.lang.Object')
that means everything gets loaded into .. the WebappClassLoader?
One could imagine a case where an SCI wants to look at everything, but
then only ends up caring about 2% of what's been loaded. Is that just
the price you have to pay for inspecting everything -- that you
seriously waste PermGen? (at least for current Oracle JVMs)
-chris
signature.asc
Description: OpenPGP digital signature
Re: Jar scanning, SCI scanning, fragment scanning
On 14 June 2013 17:35, Christopher Schultz ch...@christopherschultz.net wrote:
Mark,
On 6/14/13 12:21 PM, Mark Thomas wrote:
On 14/06/2013 16:57, Christopher Schultz wrote:
Mark,
On 6/14/13 3:16 AM, Mark Thomas wrote:
On 14/06/2013 03:31, Christopher Schultz wrote:
It might be nice if this were not a one-of-many decision, but if a
client could choose more than one type. A bit-mask or a list of
scan-types would be nice. I could see the same type of scanner being
used for multiple different purposes.
That is what ServletContainerIniiializers are for.
Well, crap. I had never seen those before.
I'm curious, though. Thinking about this the other day with a colleague
who was complaining about some kind of Spring configuration that
evidently loaded every class available on the classpath and kept them in
memory (thus leading to heap and PermGen issues), I concluded that the
only sane way to do this kind of probing would be to probe everything in
a ClassLoader that was intended to be discarded after the probing as to
avoid loading classes unnecessarily.
If an SCI retains a reference to any of the Class objects in the
SetClass parameter, that hypothetical throw-away ClassLoader is no
longer thrown-away.
The early Tomcat 7 implementations did it like this - loaded every class
and then examined the class object. More recent implementations use BCEL
to look at the byte code. It is faster and uses less memory. We also use
caching to ensure each class is only processed once.
That functionality sounds like it might be useful as a general purpose
library item, possibly as part of a utility jar for Commons BCEL.
For example JMeter has to scan classes for certain interfaces on
startup. It's current implementation is a bit wasteful.
Cool. What Classloader gets used to actually load the Class objects,
though?
BCEL reads class files as files.
http://commons.apache.org/proper/commons-bcel/
In the pathological case of @HandlesType('java.lang.Object')
that means everything gets loaded into .. the WebappClassLoader?
One could imagine a case where an SCI wants to look at everything, but
then only ends up caring about 2% of what's been loaded. Is that just
the price you have to pay for inspecting everything -- that you
seriously waste PermGen? (at least for current Oracle JVMs)
-chris
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Jar scanning, SCI scanning, fragment scanning
FYI xbean has runtime constraints bcel skips (valid read and loaded info
are consistent)
Le 14 juin 2013 18:56, sebb seb...@gmail.com a écrit :
On 14 June 2013 17:35, Christopher Schultz ch...@christopherschultz.net
wrote:
Mark,
On 6/14/13 12:21 PM, Mark Thomas wrote:
On 14/06/2013 16:57, Christopher Schultz wrote:
Mark,
On 6/14/13 3:16 AM, Mark Thomas wrote:
On 14/06/2013 03:31, Christopher Schultz wrote:
It might be nice if this were not a one-of-many decision, but if a
client could choose more than one type. A bit-mask or a list of
scan-types would be nice. I could see the same type of scanner being
used for multiple different purposes.
That is what ServletContainerIniiializers are for.
Well, crap. I had never seen those before.
I'm curious, though. Thinking about this the other day with a colleague
who was complaining about some kind of Spring configuration that
evidently loaded every class available on the classpath and kept them
in
memory (thus leading to heap and PermGen issues), I concluded that the
only sane way to do this kind of probing would be to probe everything
in
a ClassLoader that was intended to be discarded after the probing as to
avoid loading classes unnecessarily.
If an SCI retains a reference to any of the Class objects in the
SetClass parameter, that hypothetical throw-away ClassLoader is no
longer thrown-away.
The early Tomcat 7 implementations did it like this - loaded every class
and then examined the class object. More recent implementations use BCEL
to look at the byte code. It is faster and uses less memory. We also use
caching to ensure each class is only processed once.
That functionality sounds like it might be useful as a general purpose
library item, possibly as part of a utility jar for Commons BCEL.
For example JMeter has to scan classes for certain interfaces on
startup. It's current implementation is a bit wasteful.
Cool. What Classloader gets used to actually load the Class objects,
though?
BCEL reads class files as files.
http://commons.apache.org/proper/commons-bcel/
In the pathological case of @HandlesType('java.lang.Object')
that means everything gets loaded into .. the WebappClassLoader?
One could imagine a case where an SCI wants to look at everything, but
then only ends up caring about 2% of what's been loaded. Is that just
the price you have to pay for inspecting everything -- that you
seriously waste PermGen? (at least for current Oracle JVMs)
-chris
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Jar scanning, SCI scanning, fragment scanning
Sebb, On 6/14/13 12:55 PM, sebb wrote: On 14 June 2013 17:35, Christopher Schultz ch...@christopherschultz.net wrote: Mark, On 6/14/13 12:21 PM, Mark Thomas wrote: On 14/06/2013 16:57, Christopher Schultz wrote: Mark, On 6/14/13 3:16 AM, Mark Thomas wrote: On 14/06/2013 03:31, Christopher Schultz wrote: It might be nice if this were not a one-of-many decision, but if a client could choose more than one type. A bit-mask or a list of scan-types would be nice. I could see the same type of scanner being used for multiple different purposes. That is what ServletContainerIniiializers are for. Well, crap. I had never seen those before. I'm curious, though. Thinking about this the other day with a colleague who was complaining about some kind of Spring configuration that evidently loaded every class available on the classpath and kept them in memory (thus leading to heap and PermGen issues), I concluded that the only sane way to do this kind of probing would be to probe everything in a ClassLoader that was intended to be discarded after the probing as to avoid loading classes unnecessarily. If an SCI retains a reference to any of the Class objects in the SetClass parameter, that hypothetical throw-away ClassLoader is no longer thrown-away. The early Tomcat 7 implementations did it like this - loaded every class and then examined the class object. More recent implementations use BCEL to look at the byte code. It is faster and uses less memory. We also use caching to ensure each class is only processed once. That functionality sounds like it might be useful as a general purpose library item, possibly as part of a utility jar for Commons BCEL. For example JMeter has to scan classes for certain interfaces on startup. It's current implementation is a bit wasteful. Cool. What Classloader gets used to actually load the Class objects, though? BCEL reads class files as files. http://commons.apache.org/proper/commons-bcel/ No, I get that, but the scanner ultimately has to use a ClassLoader to load Class objects so they can be passed-off to the SCIs. Which ClassLoader does that? -chris signature.asc Description: OpenPGP digital signature
Re: Jar scanning, SCI scanning, fragment scanning
Geronimo's xbean-finder also supplies this functionality using asm.
david jencks
On Jun 14, 2013, at 9:55 AM, sebb seb...@gmail.com wrote:
On 14 June 2013 17:35, Christopher Schultz ch...@christopherschultz.net
wrote:
Mark,
On 6/14/13 12:21 PM, Mark Thomas wrote:
On 14/06/2013 16:57, Christopher Schultz wrote:
Mark,
On 6/14/13 3:16 AM, Mark Thomas wrote:
On 14/06/2013 03:31, Christopher Schultz wrote:
It might be nice if this were not a one-of-many decision, but if a
client could choose more than one type. A bit-mask or a list of
scan-types would be nice. I could see the same type of scanner being
used for multiple different purposes.
That is what ServletContainerIniiializers are for.
Well, crap. I had never seen those before.
I'm curious, though. Thinking about this the other day with a colleague
who was complaining about some kind of Spring configuration that
evidently loaded every class available on the classpath and kept them in
memory (thus leading to heap and PermGen issues), I concluded that the
only sane way to do this kind of probing would be to probe everything in
a ClassLoader that was intended to be discarded after the probing as to
avoid loading classes unnecessarily.
If an SCI retains a reference to any of the Class objects in the
SetClass parameter, that hypothetical throw-away ClassLoader is no
longer thrown-away.
The early Tomcat 7 implementations did it like this - loaded every class
and then examined the class object. More recent implementations use BCEL
to look at the byte code. It is faster and uses less memory. We also use
caching to ensure each class is only processed once.
That functionality sounds like it might be useful as a general purpose
library item, possibly as part of a utility jar for Commons BCEL.
For example JMeter has to scan classes for certain interfaces on
startup. It's current implementation is a bit wasteful.
Cool. What Classloader gets used to actually load the Class objects,
though?
BCEL reads class files as files.
http://commons.apache.org/proper/commons-bcel/
In the pathological case of @HandlesType('java.lang.Object')
that means everything gets loaded into .. the WebappClassLoader?
One could imagine a case where an SCI wants to look at everything, but
then only ends up caring about 2% of what's been loaded. Is that just
the price you have to pay for inspecting everything -- that you
seriously waste PermGen? (at least for current Oracle JVMs)
-chris
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Jar scanning, SCI scanning, fragment scanning
On 14/06/2013 18:53, Christopher Schultz wrote: BCEL reads class files as files. http://commons.apache.org/proper/commons-bcel/ No, I get that, but the scanner ultimately has to use a ClassLoader to load Class objects so they can be passed-off to the SCIs. Which ClassLoader does that? The web application class loader. Mark - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Jar scanning, SCI scanning, fragment scanning
As I make my way through the Servlet 3.1 spec reviewing all the changes to make sure they have all been implemented, I have reached section 8.2. There are a number of related issues here. 1. Three known issues with the current SCI / fragment handling [1] - SCI scan does not obey class loader ordering - fragments in container JARs are processed - Container SCIs may be exluded 2. A wish for per context jarsToSkip [2] 3. A wish for greater control over jarsToSkip [3] These issues are all inter-related. Fixing them is going to require either some very ugly hacks or changes to the JarScanner API. I have started down the route of the JAR scanner API changes. The overall idea is: - jarsToSkip is replaced by a JarScanFilter with the default implementation doing what jarsToSkip currently does - The client passes the type of scan (TLD, Pluggability, etc) to the JarScanner - The JarScanner indicates whether or not any JAR found is an application or container JAR - The JacScanFilter filters the JARs returned based on scan type and on configuration This addresses the above issues in the following way: 1 - Knowing if the JAR is a container JAR or not allows the ContextConfig class to do the right thing. 2 - The StandardJarScanFilter can be configured per context in the same way the StandardJarScanner can. 3 - Same as 2. The StandardJarScanFilter will have enough options to implement the requirements of 2 3. I currently have the start of this in a local git repo. Any objections to me committing what I have to trunk and continuing there? [1] http://markmail.org/message/22e3sjmsy2gt4tkw [2] https://issues.apache.org/bugzilla/show_bug.cgi?id=54083 [3] https://issues.apache.org/bugzilla/show_bug.cgi?id=54770 - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Jar scanning, SCI scanning, fragment scanning
Mark, On 6/13/13 12:23 PM, Mark Thomas wrote: As I make my way through the Servlet 3.1 spec reviewing all the changes to make sure they have all been implemented, I have reached section 8.2. There are a number of related issues here. 1. Three known issues with the current SCI / fragment handling [1] - SCI scan does not obey class loader ordering - fragments in container JARs are processed - Container SCIs may be exluded 2. A wish for per context jarsToSkip [2] 3. A wish for greater control over jarsToSkip [3] These issues are all inter-related. Fixing them is going to require either some very ugly hacks or changes to the JarScanner API. I have started down the route of the JAR scanner API changes. The overall idea is: - jarsToSkip is replaced by a JarScanFilter with the default implementation doing what jarsToSkip currently does - The client passes the type of scan (TLD, Pluggability, etc) to the JarScanner It might be nice if this were not a one-of-many decision, but if a client could choose more than one type. A bit-mask or a list of scan-types would be nice. I could see the same type of scanner being used for multiple different purposes. Another option would be to have a scanner that would scan for everything and then notify registered listeners. If you only care about TLDs, you just register a TLD listener. For classes, you register a class listener. A listener-based scanner would even open the doors for a webapp to be able to register its own listeners to sniff scanner events during context startup. I know this all sounds way over-engineered, but there are quite a few webapps that, when they first launch, go ahead and scan the entire reachable class loader hierarchy for classes with annotations, etc. so you get this environment where the same classes are being scanner over and over and over again just because they have to be scanned in different places for different reasons. I spoke with David Blevins about this way back in Vancouver: applications that endlessly perform JAR scanning just because the components can't seem to coordinate with each other. If there were a scanner component that could allow these unrelated components to share infrastructure, we could all save a lot of time. Such a scanning component could, much like the Digester, graduate from Tomcat and become more widely useful. -chris signature.asc Description: OpenPGP digital signature
Re: Jar scanning, SCI scanning, fragment scanning
On Jun 13, 2013, at 11:23 AM, Mark Thomas wrote: As I make my way through the Servlet 3.1 spec reviewing all the changes to make sure they have all been implemented, I have reached section 8.2. There are a number of related issues here. 1. Three known issues with the current SCI / fragment handling [1] - SCI scan does not obey class loader ordering - fragments in container JARs are processed - Container SCIs may be exluded 2. A wish for per context jarsToSkip [2] 3. A wish for greater control over jarsToSkip [3] These issues are all inter-related. Fixing them is going to require either some very ugly hacks or changes to the JarScanner API. I have started down the route of the JAR scanner API changes. The overall idea is: - jarsToSkip is replaced by a JarScanFilter with the default implementation doing what jarsToSkip currently does I would request doing what jarsToSkip currently does plus what is wanted with jarsToScan in 3. It shouldn't be extra complicated or require re- or duplicate-configuration of anything to whitelist one JAR. Also, it's very important that log4j-core*.jar and log4j-taglib*.jar be whitelisted by default. These are 2.0-specific JARs and they must be scanned. log4j-core contains listeners/filters/fragment to initialize and deinitialize in a Servlet container properly, and log4j-taglib contains a TLD. Nick - The client passes the type of scan (TLD, Pluggability, etc) to the JarScanner - The JarScanner indicates whether or not any JAR found is an application or container JAR - The JacScanFilter filters the JARs returned based on scan type and on configuration This addresses the above issues in the following way: 1 - Knowing if the JAR is a container JAR or not allows the ContextConfig class to do the right thing. 2 - The StandardJarScanFilter can be configured per context in the same way the StandardJarScanner can. 3 - Same as 2. The StandardJarScanFilter will have enough options to implement the requirements of 2 3. I currently have the start of this in a local git repo. Any objections to me committing what I have to trunk and continuing there? [1] http://markmail.org/message/22e3sjmsy2gt4tkw [2] https://issues.apache.org/bugzilla/show_bug.cgi?id=54083 [3] https://issues.apache.org/bugzilla/show_bug.cgi?id=54770 - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Jar scanning, SCI scanning, fragment scanning
Hi Exactly, same for tld would be possible! The idea is more or less to start from xbean then create a kind of commons-scanner. Would be great to get it in Tomcat itself. Le 14 juin 2013 05:06, Nick Williams nicho...@nicholaswilliams.net a écrit : On Jun 13, 2013, at 11:23 AM, Mark Thomas wrote: As I make my way through the Servlet 3.1 spec reviewing all the changes to make sure they have all been implemented, I have reached section 8.2. There are a number of related issues here. 1. Three known issues with the current SCI / fragment handling [1] - SCI scan does not obey class loader ordering - fragments in container JARs are processed - Container SCIs may be exluded 2. A wish for per context jarsToSkip [2] 3. A wish for greater control over jarsToSkip [3] These issues are all inter-related. Fixing them is going to require either some very ugly hacks or changes to the JarScanner API. I have started down the route of the JAR scanner API changes. The overall idea is: - jarsToSkip is replaced by a JarScanFilter with the default implementation doing what jarsToSkip currently does I would request doing what jarsToSkip currently does plus what is wanted with jarsToScan in 3. It shouldn't be extra complicated or require re- or duplicate-configuration of anything to whitelist one JAR. Also, it's very important that log4j-core*.jar and log4j-taglib*.jar be whitelisted by default. These are 2.0-specific JARs and they must be scanned. log4j-core contains listeners/filters/fragment to initialize and deinitialize in a Servlet container properly, and log4j-taglib contains a TLD. Nick - The client passes the type of scan (TLD, Pluggability, etc) to the JarScanner - The JarScanner indicates whether or not any JAR found is an application or container JAR - The JacScanFilter filters the JARs returned based on scan type and on configuration This addresses the above issues in the following way: 1 - Knowing if the JAR is a container JAR or not allows the ContextConfig class to do the right thing. 2 - The StandardJarScanFilter can be configured per context in the same way the StandardJarScanner can. 3 - Same as 2. The StandardJarScanFilter will have enough options to implement the requirements of 2 3. I currently have the start of this in a local git repo. Any objections to me committing what I have to trunk and continuing there? [1] http://markmail.org/message/22e3sjmsy2gt4tkw [2] https://issues.apache.org/bugzilla/show_bug.cgi?id=54083 [3] https://issues.apache.org/bugzilla/show_bug.cgi?id=54770 - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
