Re: Bug #57505 - Integration tests for JspC

[Jeremy Boynes]

On Sep 21, 2015, at 11:23 AM, Dinesh Joshi  wrote:
> 
> Hi all,
> 
> I'm looking to contribute to Tomcat development. I am planning to take a 
> crack at this bug - https://bz.apache.org/bugzilla/show_bug.cgi?id=57505 
> Anybody working on it? Can someone give me some pointers on how I should 
> proceed with it?

IIRC, I opened this after we had a couple of regressions during refactoring of 
Jasper that worked within the container but did not work when run for 
pre-compilation. I don’t think anyone is looking at it and all contributions 
would be welcome.

Pointers … this would be a different type of test case as most of our existing 
integration tests spin up an embedded server which is the opposite of what is 
needed here. JspC is a JavaBean designed for use in an Ant task so executing it 
from a test should not be difficult. Most of the work would be in verifying the 
results. Ideas here might be to pre-process one or more webapps and inspect the 
generated Java or compile it, deploy it to an embedded server and execute the 
pre-compiled pages; the latter might be easier.

Cheers
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [Tomcat Wiki] Update of Development/NestedFilesystem by jboynes

[Jeremy Boynes]

 On Apr 5, 2015, at 2:25 PM, Apache Wiki wikidi...@apache.org wrote:
 
 Dear Wiki user,
 
 You have subscribed to a wiki page or wiki category on Tomcat Wiki for 
 change notification.
 
 The Development/NestedFilesystem page has been changed by jboynes:
 https://wiki.apache.org/tomcat/Development/NestedFilesystem

I started capturing thoughts on the NIO FS in this page but still have a bit 
more to add.


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [VOTE] Release Apache Tomcat 8.0.21

[Jeremy Boynes]

 The proposed 8.0.21 release is:
 [ ] Broken - do not release
 [X] Stable - go ahead and release as 8.0.21

Used to test Standard Taglib on OS X, Java 1.8 and to test examples.
—
Jeremy


-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Re: svn commit: r1668021 - /tomcat/site/trunk/docs/taglibs/CHANGES.txt

[Jeremy Boynes]

Sorry about that, bad svn config. Shouldn’t happen again.
Thanks for fixing.

 On Mar 20, 2015, at 7:04 AM, kkoli...@apache.org wrote:
 
 Author: kkolinko
 Date: Fri Mar 20 14:04:50 2015
 New Revision: 1668021
 
 URL: http://svn.apache.org/r1668021
 Log:
 svn:eol-style = native
 
 Modified:
tomcat/site/trunk/docs/taglibs/CHANGES.txt   (props changed)
 
 Propchange: tomcat/site/trunk/docs/taglibs/CHANGES.txt
 --
svn:eol-style = native
 
 
 
 -
 To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
 For additional commands, e-mail: dev-h...@tomcat.apache.org
 



signature.asc
Description: Message signed with OpenPGP using GPGMail

[ANN] Apache Standard Taglib 1.2.5 released

[Jeremy Boynes]

The Apache Tomcat team announces the immediate availability of Apache Standard 
Taglib 1.2.5.

Apache Standard Taglib is an open source software implementation of the JSP 
Standard Tag Library (JSTL) technology.

This release supports JSTL version 1.2 and includes minor bug-fixes to the 
previous 1.2.3 release. The notable changes include:
- Reversion of a change released in 1.2.1 where c:import set the HTTP method 
to GET when performing a local include
- Avoid an AccessControlException if permission has not been granted to read 
the accessExternalEntity property

Please refer to the project website for additional information.
  http://tomcat.apache.org/taglibs.html

Enjoy!

- The Apache Tomcat team



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: WAR FileSystem for fast nested JAR access?

[Jeremy Boynes]

On Mar 17, 2015, at 9:01 AM, Christopher Schultz ch...@christopherschultz.net 
wrote:
 
 Jeremy,
 
 On 3/17/15 2:39 AM, Jeremy Boynes wrote:
 On Mar 7, 2015, at 10:13 AM, Jeremy Boynes jboy...@apache.org wrote:
 
 On Mar 6, 2015, at 7:43 AM, Mark Thomas ma...@apache.org wrote:
 Interesting. The deciding factor for me will be performance. Keep in
 mind that we might not need all the API. As long as there is enough to
 implement WebResourceSet and WebResource, we probably have all we need.
 
 I ran a micro-benchmark using the greenhouse WAR associated with the 
 original bug. I instrumented JarWarResource to log all resources opened 
 during startup and record the time. On my system it took ~21,000ms to start 
 the application of which ~16,000ms was spent in getJarInputStreamWrapper(). 
 2935 resources were opened, primarily class files.
 
 I then replayed the log against the sandbox FS. With the current 
 implementation it took ~300ms to open the war, ~350ms to open all the jars, 
 and ~5ms to open all the entries with newInputStream().
 
 I interpret that to mean that there is pretty constant time taken to 
 inflate 15MB of data - the 300ms to scan the archive and the ~350ms to scan 
 each of the jars within (each one that was used at least). The speed up 
 here comes because we only scan each archive once, the downside is the 
 extra memory used to store the inflated data.
 
 This is promising enough to me that I’m going to keep exploring.
 
 Konstantin’s patch, AIUI, creates an index for each jar which eliminates 
 the need to scan jars on the classpath that don’t contain the class being 
 requested. However, once the classloader has determined the jar file to use 
 we still need to stream through that jar until we reach the desired entry.
 
 I think we can avoid that here by digging into the zip file’s internal 
 metadata. Where I am currently  streaming the jar to build the directory, 
 with random access I can build an index just by reading the central 
 directory structure. An index entry would contain the name, metadata, and 
 the offset in the archive of the entry’s data. When an entry is opened 
 would we inflate the data so that it could be used to underpin the channel. 
 When the channel is closed the memory would be released.
 
 In general, I don’t think there’s a need for the FileSystem to retain 
 inflated data after the channel is closed. This would be particularly true 
 for the leaf resources which are not likely to be reused; for example, once 
 a ClassLoader has used the .class file to define the Class or once a 
 framework has processed a .xml config file then neither will need it again.
 
 However, I think the WAR ClassLoader would benefit from keeping the JAR 
 files on the classpath open to avoid re-inflating them. The pattern though 
 would be bursty e.g. lots of class loads during startup followed by 
 quiescence. I can think of two ways to handle that:
 1) FileSystem has maintains a cache of inflated entries much like a disk 
 filesystem has buffers
  The FileSystem would be responsible for evictions, perhaps on a LRU or 
 timed basis.
 2) Having the classloader keep the JARs opened/mounted after loading a 
 resource until such time as it thinks quiescence is reached. It would then 
 unmount JARs to free the memory.
 We could do both as they don’t conflict.
 
 Next step will be to look into building the index directly from the 
 archive’s central directory rather than by streaming it.
 
 Next step was actually just to verify that we could make a URLClassLoader 
 work with this API. I got this to work by turning the path URIs into 
 collection URLs (ending in ‘/‘) which prevented the classloader from trying 
 to open them as JarFiles.
 
 The classloader works but the classpath search is pretty inefficient relying 
 on UrlConnection#getInputStream throwing an Exception to detect if a 
 resource exists. Using it to load the 2935 resources from before took 
 ~1900ms even after the jars had been indexed. getInputStream() was called 
 ~120,000 times as the classpath was scanned, i.e. 15us per check with an 
 average of ~40 checks per resource which seems about right for a classpath 
 that contains 73 jars.
 
 An obvious solution to avoid the repeated search would be to union the jars’ 
 directories into a single index. I may try this with a PathClassLoader that 
 operates using a list of Paths rather than URLs.
 
 I just wanted to let you know that I'm reading these with interest. I'm
 anxious to find out if this is going to pan-out.

Thanks. Real-life is a bit busy at the moment so progress will be sporadic. If 
you or anyone would like to jump in there are a few of areas which still have 
unknowns:
* a way to read the zip’s central directory
* a way to seek into a deflated zip entry without inflating the entire thing
* is a ClassLoader from a list of Path helpful?
* how to deal with the locking model on Windows platform
* how to work with Paths that are directories - do we get

Re: Taglibs sub-site

[Jeremy Boynes]

 On Feb 27, 2015, at 7:57 AM, Jeremy Boynes jboy...@apache.org wrote:
 
 On Feb 27, 2015, at 3:38 AM, kkoli...@apache.org wrote:
 
 Author: kkolinko
 Date: Fri Feb 27 11:38:13 2015
 New Revision: 1662662
 
 URL: http://svn.apache.org/r1662662
 Log:
 Announcement for Standard Taglib 1.2.3.
 
 I do not like the CVE link (goes to announce@a.o mail archive) and CHANGES 
 link (goes to SVN), as I noted in a FIXME comment in index.xml.  Any better 
 ideas?
 
 We could add a security-taglibs page to the main site and link from the 
 security.html page.
 
 I find the frankensite, as Henri called it, a pain in general. I’m thinking 
 about merging it in with the main Tomcat site source and give it an overhaul 
 (including moving away from using Maven to build it).
 
 It integrates Taglibs more with the main Tomcat project structure. Does 
 anyone have any concern about that?
 

In r1667455 I updated the main site to integrate Taglibs more closely. Primary 
changes are:
* a menu link to the download page
* replace Maven sub-site with a taglibs page
* added a page for security issues (linked off security.html)

Cheers
Jeremy




signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: WAR FileSystem for fast nested JAR access?

[Jeremy Boynes]

On Mar 7, 2015, at 10:13 AM, Jeremy Boynes jboy...@apache.org wrote:
 
 On Mar 6, 2015, at 7:43 AM, Mark Thomas ma...@apache.org wrote:
 Interesting. The deciding factor for me will be performance. Keep in
 mind that we might not need all the API. As long as there is enough to
 implement WebResourceSet and WebResource, we probably have all we need.
 
 I ran a micro-benchmark using the greenhouse WAR associated with the original 
 bug. I instrumented JarWarResource to log all resources opened during startup 
 and record the time. On my system it took ~21,000ms to start the application 
 of which ~16,000ms was spent in getJarInputStreamWrapper(). 2935 resources 
 were opened, primarily class files.
 
 I then replayed the log against the sandbox FS. With the current 
 implementation it took ~300ms to open the war, ~350ms to open all the jars, 
 and ~5ms to open all the entries with newInputStream().
 
 I interpret that to mean that there is pretty constant time taken to inflate 
 15MB of data - the 300ms to scan the archive and the ~350ms to scan each of 
 the jars within (each one that was used at least). The speed up here comes 
 because we only scan each archive once, the downside is the extra memory used 
 to store the inflated data.
 
 This is promising enough to me that I’m going to keep exploring.
 
 Konstantin’s patch, AIUI, creates an index for each jar which eliminates the 
 need to scan jars on the classpath that don’t contain the class being 
 requested. However, once the classloader has determined the jar file to use 
 we still need to stream through that jar until we reach the desired entry.
 
 I think we can avoid that here by digging into the zip file’s internal 
 metadata. Where I am currently  streaming the jar to build the directory, 
 with random access I can build an index just by reading the central directory 
 structure. An index entry would contain the name, metadata, and the offset in 
 the archive of the entry’s data. When an entry is opened would we inflate the 
 data so that it could be used to underpin the channel. When the channel is 
 closed the memory would be released.
 
 In general, I don’t think there’s a need for the FileSystem to retain 
 inflated data after the channel is closed. This would be particularly true 
 for the leaf resources which are not likely to be reused; for example, once a 
 ClassLoader has used the .class file to define the Class or once a framework 
 has processed a .xml config file then neither will need it again.
 
 However, I think the WAR ClassLoader would benefit from keeping the JAR files 
 on the classpath open to avoid re-inflating them. The pattern though would be 
 bursty e.g. lots of class loads during startup followed by quiescence. I can 
 think of two ways to handle that:
 1) FileSystem has maintains a cache of inflated entries much like a disk 
 filesystem has buffers
   The FileSystem would be responsible for evictions, perhaps on a LRU or 
 timed basis.
 2) Having the classloader keep the JARs opened/mounted after loading a 
 resource until such time as it thinks quiescence is reached. It would then 
 unmount JARs to free the memory.
 We could do both as they don’t conflict.
 
 Next step will be to look into building the index directly from the archive’s 
 central directory rather than by streaming it.

Next step was actually just to verify that we could make a URLClassLoader work 
with this API. I got this to work by turning the path URIs into collection URLs 
(ending in ‘/‘) which prevented the classloader from trying to open them as 
JarFiles.

The classloader works but the classpath search is pretty inefficient relying on 
UrlConnection#getInputStream throwing an Exception to detect if a resource 
exists. Using it to load the 2935 resources from before took ~1900ms even after 
the jars had been indexed. getInputStream() was called ~120,000 times as the 
classpath was scanned, i.e. 15us per check with an average of ~40 checks per 
resource which seems about right for a classpath that contains 73 jars.

An obvious solution to avoid the repeated search would be to union the jars’ 
directories into a single index. I may try this with a PathClassLoader that 
operates using a list of Paths rather than URLs.

Cheers
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

[RESULT] [VOTE] Release Apache Standard Taglib 1.2.5

[Jeremy Boynes]

On Mar 10, 2015, at 9:22 AM, Jeremy Boynes jboy...@apache.org wrote:
 
 The proposed Apache Standard Taglib 1.2.5 release is now available for voting.
 
 This release contains two minor bug fixes:
 57673 AccessControlException accessing 
 org.apache.taglibs.standard.xml.accessExternalEntity
 37466 Reverted changes that overrode HTTP method when importing local 
 resources.
 
 It can be obtained from:
 https://dist.apache.org/repos/dist/dev/tomcat/taglibs/taglibs-standard-1.2.5/
 
 The Maven staging repo is:
 https://repository.apache.org/content/repositories/orgapachetomcat-1038/
 
 The SVN tag is:
 http://svn.apache.org/repos/asf/tomcat/taglibs/standard/tags/taglibs-standard-1.2.5/
 
 The proposed 1.2.5 release is:
 [ ] Broken - do not release
 [ ] Stable - go ahead and release as 1.2.5 Stable

The following votes were cast:

Binding:
+1: jboynes, kkolinko, violetagg

The vote thereby passes.

Thank you,
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [VOTE] Release Apache Standard Taglib 1.2.5

[Jeremy Boynes]

On Mar 10, 2015, at 9:22 AM, Jeremy Boynes jboy...@apache.org wrote:
 
 The proposed 1.2.5 release is:
 [ ] Broken - do not release
 [X] Stable - go ahead and release as 1.2.5 Stable

TCK passed on Mac OS X 10.10.2, Java 7 “1.7.0_75” and Tomcat 7.0.57 with 
-security set. Run with
$ CATALINA_OPTS=-Dorg.apache.taglibs.standard.xml.accessExternalEntity=http 
-Djavax.xml.accessExternalDTD=http ./tomcat/bin/catalina.sh run -security

and with the following additional permissions granted to all webapps:
permission java.net.SocketPermission localhost:8080, connect;
permission java.net.SocketPermission localhost:1527, connect;
permission java.util.PropertyPermission 
org.apache.taglibs.standard.xml.accessExternalEntity, read;

Tested in following configurations:
Using -jstlel in webapp
Using -jstlel in container
Using -compat in webapp
Using -compat in container

Thanks
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

[VOTE] Release Apache Standard Taglib 1.2.5

[Jeremy Boynes]

The proposed Apache Standard Taglib 1.2.5 release is now available for voting.

This release contains two minor bug fixes:
57673 AccessControlException accessing 
org.apache.taglibs.standard.xml.accessExternalEntity
37466 Reverted changes that overrode HTTP method when importing local resources.

It can be obtained from:
https://dist.apache.org/repos/dist/dev/tomcat/taglibs/taglibs-standard-1.2.5/

The Maven staging repo is:
https://repository.apache.org/content/repositories/orgapachetomcat-1038/

The SVN tag is:
http://svn.apache.org/repos/asf/tomcat/taglibs/standard/tags/taglibs-standard-1.2.5/

The proposed 1.2.5 release is:
[ ] Broken - do not release
[ ] Stable - go ahead and release as 1.2.5 Stable

Thanks
Jeremy


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: WAR FileSystem for fast nested JAR access?

[Jeremy Boynes]

On Mar 8, 2015, at 9:53 AM, Mark Thomas ma...@apache.org wrote:
 
 On 07/03/2015 18:13, Jeremy Boynes wrote:
 I interpret that to mean that there is pretty constant time taken to
 inflate 15MB of data - the 300ms to scan the archive and the ~350ms
 to scan each of the jars within (each one that was used at least).
 The speed up here comes because we only scan each archive once, the
 downside is the extra memory used to store the inflated data.
 
 Do you mean the entire entire deflated WAR is in memory? If we wanted to
 go that way it is fairly easy to do with the existing WebResource
 implementation.

It does in this quick and dirty version as I wanted to get a sense of what was 
possible i.e. does the API support the access patterns needed. The rest of the 
email is exploring ways to avoid that.

 
 In general, I don’t think there’s a need for the FileSystem to retain
 inflated data after the channel is closed.
 
 Agreed. When to cache or not should be left to WebResources.
 
 However, I think the WAR ClassLoader would benefit from keeping the
 JAR files on the classpath open to avoid re-inflating them.
 
 That isn't an option unless you can solve the locked file problem on
 Windows. It always has to be possible to delete the WAR on the file
 system to trigger an undeployment.

Remember, it would be opening files on the virtual filesystem not the 
underlying OS so we can control the locking behaviour. IIRC, part of the 
problem is that ZipFile uses mmap to access the file which requires a lock on 
Windows; we may not want to do that (or it could an OS-specific behavioral 
options).
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Re: WAR FileSystem for fast nested JAR access?

[Jeremy Boynes]

On Mar 8, 2015, at 5:28 AM, Christopher Schultz ch...@christopherschultz.net 
wrote:
 
 Jeremy,
 
 On 3/7/15 1:13 PM, Jeremy Boynes wrote:
 On Mar 6, 2015, at 7:43 AM, Mark Thomas ma...@apache.org wrote:
 Interesting. The deciding factor for me will be performance. Keep
 in mind that we might not need all the API. As long as there is
 enough to implement WebResourceSet and WebResource, we probably
 have all we need.
 
 I ran a micro-benchmark using the greenhouse WAR associated with the
 original bug. I instrumented JarWarResource to log all resources
 opened during startup and record the time. On my system it took
 ~21,000ms to start the application of which ~16,000ms was spent in
 getJarInputStreamWrapper(). 2935 resources were opened, primarily
 class files.
 
 I then replayed the log against the sandbox FS. With the current
 implementation it took ~300ms to open the war, ~350ms to open all the
 jars, and ~5ms to open all the entries with newInputStream().
 
 I interpret that to mean that there is pretty constant time taken to
 inflate 15MB of data - the 300ms to scan the archive and the ~350ms
 to scan each of the jars within (each one that was used at least).
 The speed up here comes because we only scan each archive once, the
 downside is the extra memory used to store the inflated data.
 
 This is promising enough to me that I’m going to keep exploring.
 
 Konstantin’s patch, AIUI, creates an index for each jar which
 eliminates the need to scan jars on the classpath that don’t contain
 the class being requested. However, once the classloader has
 determined the jar file to use we still need to stream through that
 jar until we reach the desired entry.
 
 I have a dumb question about this: why does the JAR file have to be
 /searched/ for a particular entry? Opening the JAR file should seek to
 the end of the file to read the TOC, and then the file offsets should be
 immediately available. Need file #27? Look in entries[27] and the offset
 into the file should be available.
 
 Is the problem that, because the JAR is inside a WAR file, the offset
 into the JAR file is meaningless because there isn't an easy way to
 determine the mapping from uncompressed-offset to compressed-offset?

It’s limitation of the classical API. We can open the WAR as a JarFile which 
allows this type of random access using just this mechanism. You can access any 
entry and retrieve an InputStream which we wrap as a JarInputStream,. That’s 
not seekable so the only way to locate an entry in that JarInputStream (e.g. a 
resource or class) is to search through that stream.

JarFile though can only open a File (i.e. something on the default filesystem). 
This is why it is much faster when the JAR is extracted to the file system 
where it can then be opened with JarFile to give random access to its contents.

There are many code paths in our code and in the JDK (e.g. 
URLClassLoader/URLClassPath) to detect whether something is on disk and can be 
optimized (i.e. is it a directory which allows path manipulation, or is it 
file:// URL that could be opened using JarFile).

 
 I think we can avoid that here by digging into the zip file’s
 internal metadata. Where I am currently  streaming the jar to build
 the directory, with random access I can build an index just by
 reading the central directory structure. An index entry would contain
 the name, metadata, and the offset in the archive of the entry’s
 data.
 
 Which archive do you mean, here? The inner JAR or the outer WAR?

Either, basically whatever archive is being mounted. During the mount, it 
builds an index of the entries in the jar. As a quick hack, it currently does 
that by scanning all the content using a JarInputStream. What I’m planning on 
doing next is building that index by reading the archive’s structure directly. 
Doing that requires seeking backwards through the zip data structures which is 
what the current APIs don’t support.

 
 When an entry is opened would we inflate the data so that it
 could be used to underpin the channel. When the channel is closed the
 memory would be released.
 
 In general, I don’t think there’s a need for the FileSystem to retain
 inflated data after the channel is closed. This would be particularly
 true for the leaf resources which are not likely to be reused; for
 example, once a ClassLoader has used the .class file to define the
 Class or once a framework has processed a .xml config file then
 neither will need it again.
 
 You could use a small LRU cache or something if you wanted to get fancy.
 Once the majority of class loading is done, it might help for other
 resources that are requested with some regularity.
 
 However, I think the WAR ClassLoader would benefit from keeping the
 JAR files on the classpath open to avoid re-inflating them. The
 pattern though would be bursty e.g. lots of class loads during
 startup followed by quiescence. I can think of two ways to handle
 that: 1) FileSystem has maintains a cache

Re: WAR FileSystem for fast nested JAR access?

[Jeremy Boynes]

On Mar 6, 2015, at 7:43 AM, Mark Thomas ma...@apache.org wrote:
 Interesting. The deciding factor for me will be performance. Keep in
 mind that we might not need all the API. As long as there is enough to
 implement WebResourceSet and WebResource, we probably have all we need.

I ran a micro-benchmark using the greenhouse WAR associated with the original 
bug. I instrumented JarWarResource to log all resources opened during startup 
and record the time. On my system it took ~21,000ms to start the application of 
which ~16,000ms was spent in getJarInputStreamWrapper(). 2935 resources were 
opened, primarily class files.

I then replayed the log against the sandbox FS. With the current implementation 
it took ~300ms to open the war, ~350ms to open all the jars, and ~5ms to open 
all the entries with newInputStream().

I interpret that to mean that there is pretty constant time taken to inflate 
15MB of data - the 300ms to scan the archive and the ~350ms to scan each of the 
jars within (each one that was used at least). The speed up here comes because 
we only scan each archive once, the downside is the extra memory used to store 
the inflated data.

This is promising enough to me that I’m going to keep exploring.

Konstantin’s patch, AIUI, creates an index for each jar which eliminates the 
need to scan jars on the classpath that don’t contain the class being 
requested. However, once the classloader has determined the jar file to use we 
still need to stream through that jar until we reach the desired entry.

I think we can avoid that here by digging into the zip file’s internal 
metadata. Where I am currently  streaming the jar to build the directory, with 
random access I can build an index just by reading the central directory 
structure. An index entry would contain the name, metadata, and the offset in 
the archive of the entry’s data. When an entry is opened would we inflate the 
data so that it could be used to underpin the channel. When the channel is 
closed the memory would be released.

In general, I don’t think there’s a need for the FileSystem to retain inflated 
data after the channel is closed. This would be particularly true for the leaf 
resources which are not likely to be reused; for example, once a ClassLoader 
has used the .class file to define the Class or once a framework has processed 
a .xml config file then neither will need it again.

However, I think the WAR ClassLoader would benefit from keeping the JAR files 
on the classpath open to avoid re-inflating them. The pattern though would be 
bursty e.g. lots of class loads during startup followed by quiescence. I can 
think of two ways to handle that:
1) FileSystem has maintains a cache of inflated entries much like a disk 
filesystem has buffers
   The FileSystem would be responsible for evictions, perhaps on a LRU or timed 
basis.
2) Having the classloader keep the JARs opened/mounted after loading a resource 
until such time as it thinks quiescence is reached. It would then unmount JARs 
to free the memory.
We could do both as they don’t conflict.

Next step will be to look into building the index directly from the archive’s 
central directory rather than by streaming it.

Cheers
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Time for 7.0.60

[Jeremy Boynes]

On Mar 6, 2015, at 9:19 AM, Konstantin Kolinko knst.koli...@gmail.com wrote:
 
 2015-03-05 22:39 GMT+03:00 Violeta Georgieva violet...@apache.org:
 Hi,
 
 I plan to start preparing 7.0.60 for voting.
 
 If you want to add something to this release please reply here.
 
 
 Testing the examples app with Security Manager enabled
 
 1) JSPs using JSTL fail, in the same way as I mentioned for Tomcat 6
 in its STATUS.txt,
 http://svn.apache.org/r1664031
 
 A read permission for JSTL configuration property is needed.

I’ve updated taglibs so that falls back to the default if permission has not 
been granted to read the property. I’ve verified that fixes the problem in tc7 
trunk. Should we do quick release of 1.2.4 first rather than add a 
webapp-specific permission grant?

—
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [VOTE][RESULT] Release Apache Taglibs 1.2.3

[Jeremy Boynes]

On Mar 6, 2015, at 1:06 AM, Mark Thomas ma...@apache.org wrote:
 
 On 20/02/2015 16:01, Konstantin Kolinko wrote:
 2015-02-20 18:39 GMT+03:00 Jeremy Boynes jboy...@apache.org:
 The following vote were cast:
 
 Binding:
 +1: jboynes, kkolinko, kfujino
 
 The vote therefore passes.
 
 Actually, it doesn't. At the time of this vote jboynes was not a member
 of the Tomcat PMC. This vote therefore failed.
 
 Since Jeremy has just been voted to the PMC a quick +1 in this thread
 from Jeremy post his addition to the PMC should fix this.
 
 I've been back through the archives to check the other taglibs votes:
 
 1.2.2 - vote failed
 1.2.1 - vote passed (rjung, olamy, markt)
 1.2.0 - vote failed
 
 Therefore there are no other issues.
 
 Mark
 
 P.S. Welcome to the PMC Jeremy :)

Apologies for that, my formal +1 to correct.
Thank for catching that, and the welcome.

—
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: WAR FileSystem for fast nested JAR access?

[Jeremy Boynes]

On Mar 4, 2015, at 9:09 AM, Jeremy Boynes jboy...@apache.org wrote:
 
 My suggestion for using an NIO2 FileSystem is because its API provides for 
 nesting and for random access to the entries in the filesystem. Something 
 like:
 
   Path war = FileSystems.getDefault().getPath(“real/path/of/application.war”);
   FileSystem warFS = FileSystems.newFileSystem(“war:” + war.toURI());
   Path nestedJar = warFS.getPath(“/WEB-INF/lib/some.jar”);
   FileSystem jarFS = FileSystems.newFileSystem(“jar:” + nestedJar.toURI());
   Path resource = jarFS.getPath(“some/resource.txt”);
   return Files.newInputStream(resource); // or newFileChannel(resource) etc.
 
 There are two requirements on the archive FileSystem implementation for this 
 to work:
 * Support for nesting in the URI
 * Functioning implementation of newByteChannel or newFileChannel
 
 Unfortunately the jar: provider that comes with the JRE won’t do that. It has 
 ye olde jar: URL nesting issues and requires the archive Path be provided by 
 the default FileSystem. Its newByteChannel() returns a SeekableByteChannel 
 that is not seekable (doh!) and newFileChannel() works by extracting the 
 entry to a temp file.
 
 The former problem seems easy to work around. To support a seekable channel 
 without extraction would be trickier as you would need to convert channel 
 positions to the actual position in the compressed data which would mean 
 digging into the compression block structure. However, I think the naive 
 approach of scanning the entry data and then caching the block offsets would 
 still be quicker than inflating to a temp file.

I started exploring this in http://svn.apache.org/r1664650

This has a (very) crude implementation of a FileSystem that allows nesting on 
top of JAR-style archives using the newByteChannel() API. It enables you to 
“mount” an archive and than access resources in it using the standard Files API.

Still to investigate is how Paths can represented as URIs that can be converted 
to URLs to be returned from e.g. getResource(). I’m exploring URL-encoding the 
authority component to allow the base Path’s URI to be stored in a regular 
hierarchical URI with the nesting simply triggering multiple levels of 
encoding. This is different to the jar scheme’s approach of using 
non-hierarchical URIs with custom paths and the “!/“ separator.

Hopefully we would be able to create a URLClassLoader from those URLs and have 
it operate normally. The URLStreamHandler would handle archive: URLs by 
locating a mounted filesystem and opening a stream to the Path within. We would 
still need to install a custom URLStreamHandlerFactory.

Cheers
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: WAR FileSystem for fast nested JAR access?

[Jeremy Boynes]

On Mar 4, 2015, at 3:49 AM, Konstantin Kolinko knst.koli...@gmail.com wrote:
 
 2015-03-04 8:20 GMT+03:00 Jeremy Boynes jboy...@apache.org:
 In https://bz.apache.org/bugzilla/show_bug.cgi?id=57251, Mark Thomas wrote:
 
 The fix for bug 57472 might shave a few seconds of the deployment time but
 it doesn't appear to make a significant difference.
 
 The fundamental problem when running from a packed WAR is that to access any
 resource in a JAR, Tomcat has to do the following:
 - open the WAR
 - get the entry for the JAR
 - get the InputStream for the JAR entry
 - Create a JarInputStream
 - Read the JarInputStream until it finds the entry it wants
 
 This is always going to be slow.
 
 The reason that it is fast in Tomcat 7 and earlier took some digging. In
 unpackWARs is false in Tomcat 7, it unpacks the JARs anyway into the work
 directory and uses them from there. Performance is therefore comparable with
 unpackWARs=true.
 
 Has anyone looked into using a NIO2 FileSystem for this? It may offer a way 
 to avoid having to stream the entry in order to be able to locate a 
 resource. ZipFile is fast, I believe, because it has random access to the 
 archive and can seek directly to an entry's location based on the zip index; 
 the jar: FileSystem seems to be able to do the same.
 
 However, neither can cope with nested entries: ZipFile because its 
 constructor takes a File rather than a Path and uses native code, and ZipFS 
 because it relies on URIs and can't cope with a jar: URI based on another 
 jar: URI (ye olde problem with jar: URL syntax).
 
 What a FileSystem can do differently is return a FileChannel which supports 
 seek operations over the archive's content. IOW, if ZipFS can work given a 
 random access channel to bytes on disk, the same approach could be adopted 
 with a random access channel to bytes on a virtual FileSystem.
 
 I imagine that would get pretty hairy for write operations but fortunately 
 we would not need to deal with that.
 
 If no-one’s looked at it yet I'll take a shot.
 Cheers
 Jeremy
 
 FWIW, this could also be exposed to web applications e.g.
  FileSystem webappFS = servletContext.getFileSystem();
  Path resource = webappFS.getPath(request.getPathInfo());
  Files.copy(resource, response.getOutputStream());
 
 
 The fundamental issue is how the data of JAR file (as a whole) is
 available via API.
 
 To be able to use random access with the JAR you technically have to
 
 1) Jump to the end of the JAR file and read the ZIP index (Central
 directory) that is located there. See the image at:
 http://en.wikipedia.org/wiki/Zip_%28file_format%29
 
 2) Jump to the specific file.
 
 As JAR itself is compressed, there is no real API to jump to a
 position in it, besides maybe InputStream.skip(). This skip() will
 involve the same overhead as the current implementation that scans the
 jar, unless the war has zero compression.
 
 
 Also
 1. Reading the zip index takes time and would better be cached. That
 is the issue behind
 https://bz.apache.org/bugzilla/show_bug.cgi?id=52448
 
 2. It makes sense to cache the list of directories (packages) in the
 zip file. Scanning the whole jar for a class that is not present there
 is the worst case.  A bonus is that it can improve handling of JARs
 that do not have explicit entries for directories.

I agree caching would help but I’m not convinced the lack thereof is the main 
cause of the speed issue here. From Mark’s description above, Read the 
JarInputStream until it finds the entry it wants” sounds more problematic.

“Open the WAR” and “get the entry for the JAR” can use ZipFile which uses 
random access to locate the bytes for the nested JAR. However, ZipFile only 
provides access to those bytes as an InputStream so we need to stream to locate 
the resource entry.

As an aside, there’s also the issue that zip archives can have zombie entries 
left in the stream but removed from the central directory, so the only way to 
know if an entry should actually be returned is to read to the directory which 
happens to be at the end. AIUI, ZipInputStream will return those zombies as it 
proceeds. This is seldom an issue for JARs as they typically don’t have zombies.

My suggestion for using an NIO2 FileSystem is because its API provides for 
nesting and for random access to the entries in the filesystem. Something like:

   Path war = FileSystems.getDefault().getPath(“real/path/of/application.war”);
   FileSystem warFS = FileSystems.newFileSystem(“war:” + war.toURI());
   Path nestedJar = warFS.getPath(“/WEB-INF/lib/some.jar”);
   FileSystem jarFS = FileSystems.newFileSystem(“jar:” + nestedJar.toURI());
   Path resource = jarFS.getPath(“some/resource.txt”);
   return Files.newInputStream(resource); // or newFileChannel(resource) etc.

There are two requirements on the archive FileSystem implementation for this to 
work:
* Support for nesting in the URI
* Functioning implementation of newByteChannel or newFileChannel

Unfortunately the jar: provider

WAR FileSystem for fast nested JAR access?

[Jeremy Boynes]

In https://bz.apache.org/bugzilla/show_bug.cgi?id=57251, Mark Thomas wrote:

 The fix for bug 57472 might shave a few seconds of the deployment time but
 it doesn't appear to make a significant difference.

 The fundamental problem when running from a packed WAR is that to access any
 resource in a JAR, Tomcat has to do the following:
 - open the WAR
 - get the entry for the JAR
 - get the InputStream for the JAR entry
 - Create a JarInputStream
 - Read the JarInputStream until it finds the entry it wants

 This is always going to be slow.

 The reason that it is fast in Tomcat 7 and earlier took some digging. In
 unpackWARs is false in Tomcat 7, it unpacks the JARs anyway into the work
 directory and uses them from there. Performance is therefore comparable with
 unpackWARs=true.

Has anyone looked into using a NIO2 FileSystem for this? It may offer a way to 
avoid having to stream the entry in order to be able to locate a resource. 
ZipFile is fast, I believe, because it has random access to the archive and can 
seek directly to an entry's location based on the zip index; the jar: 
FileSystem seems to be able to do the same.

However, neither can cope with nested entries: ZipFile because its constructor 
takes a File rather than a Path and uses native code, and ZipFS because it 
relies on URIs and can't cope with a jar: URI based on another jar: URI (ye 
olde problem with jar: URL syntax).

What a FileSystem can do differently is return a FileChannel which supports 
seek operations over the archive's content. IOW, if ZipFS can work given a 
random access channel to bytes on disk, the same approach could be adopted with 
a random access channel to bytes on a virtual FileSystem.

I imagine that would get pretty hairy for write operations but fortunately we 
would not need to deal with that.

If no-one’s looked at it yet I'll take a shot.
Cheers
Jeremy

FWIW, this could also be exposed to web applications e.g.
  FileSystem webappFS = servletContext.getFileSystem();
  Path resource = webappFS.getPath(request.getPathInfo());
  Files.copy(resource, response.getOutputStream());



signature.asc
Description: Message signed with OpenPGP using GPGMail

Taglibs/JSP future work, was: Time for Taglibs to be sent to the archive?

[Jeremy Boynes]

On Mar 2, 2015, at 5:21 AM, Konstantin Kolinko knst.koli...@gmail.com wrote:
 
 2015-03-01 1:04 GMT+03:00 Jeremy Boynes jboy...@apache.org:
...
 - Put Standard 1.2.x in maintenance mode for bug fixes only
 - Start a tree/branch for new work without the limitations of the 
 now-ancient 1.2 spec
 - Plan to release new work early  often
 
 I do not see a reason for branching. The library implements
 specification.   In what direction is trunk supposed to go?
 
 Possible directions:
 a) Improve it for newer versions of Java, while still maintaining the same 
 API?

We had talked previously about dropping Java5 support and decided not to do 
that in 1.2.x release. I was thinking of a 2.x trunk that would allow us to 
baseline on a newer version, specifically Java8 given the public version of 
Java7 reaches EOL next month (April 2015). It was thinking about the lambda 
support that led me to the straw man I posted earlier today. Which leads to ...

 b) Extend the library outside of specification?
 c) Update it to newer versions of specification? I think there is no
 new specification version now, but maybe there is some
 development/plans?

JSP is included in the list of technologies “expected to be updated” in Java EE 
8 but as the JSR is still closed we won’t see anything until the PFD phase. 
Given recent history I would not expect to see any major changes.

Since JSTL 1.0 came out we’ve see many changes in its space:
* EL became a standalone thing
* EL can call Java functions directly eliminating the need for function 
definitions in TLDs
* Script-free JSP pages became best-practice
* Pooling became less important leading to the SimpleTag model
* Annotation, introspection and injection became commonplace
* MVC meant JSPs became primarily a View component rather than playing the 
Controller role

With the introduction of pruning abilities in EE6 some things that could go 
might include:
* EL 1.0 and the legacy JSP APIs that support it
* JSTL’s XML and SQL tags

Rather than wait for the spec we could start to simplify the the model for 
people working at the JSP level (i.e. above Servlets but lower than JSF, 
perhaps in conjunction with the new MVC spec).

 d) Work on built-in implementation of JSTL tags in Jasper? (Jasper Tag 
 plugins)

Jasper has plugin for JSTL core but there are some edge cases it handles 
incorrectly. I should open bugs for them and we could fix those :) I’d actually 
like to take a second look at the plugin mechanism anyway - for example, why 
does it key off the tag’s implementation class rather than its QName.

 I mean: If there is no new development planned, we would better stay
 on the current trunk without separate maintenance branch. It is a bit
 more risky, but it has less overhead.
 
 I would like to fix some of many generics warnings shown by Eclipse
 IDE (either implement generics or to add @SupressWarning). No definite
 time slot for that though as I am not sure whether it is worth to
 spend time on that.

Things I was thinking of in the current implementation include:
* Making Java 8 the baseline JRE and updating to it (separate from any 
functional work)
* Making Java EE 7 the baseline container and dropping -jstlel in favour of 
-compat
  - This would get rid of a many of those code-health warnings in one swoop :)
  - -compat would merge back into -impl
* Separating the 4 libraries into separate modules (core, fmt, xml, sql) so:
  - Users would only need the libraries they actually used
  - We could release each module separately as needed
  - Xalan would only be needed when using the XML library
  - We could do an alternate fmt using ICU4J which typically updates quicker 
than the JDK

I probably wasn’t clear originally - I meant this type of work would be on 
trunk and we’d create a stable branch 1.2.x to support bug-fixes.

—
Jeremy




signature.asc
Description: Message signed with OpenPGP using GPGMail

Using functional interfaces for tags and JSP

[Jeremy Boynes]

After the recent thread around taglibs future, I wanted to explore the 
possibility of using Java8’s functional interfaces to implement tags. 
http://svn.apache.org/r1663366 is a straw man I used.

To explain the approach here, I started with the thought that the SimpleTag 
model in JSP2.0 was a kind of functional implementation. Rather than having to 
define a class for each tag and a XML description of it, could we represent it 
as a single method and gain all the needed information through annotation and 
introspection. Basically I wanted to be able write a tag as something like

   void tagName(String attribute1, int attribute2, JspFragment body)

TagLibrary describes the semantics I came up with, and BasicTagLibrary attempts 
to demonstrate how the various patterns used by today’s tags could be 
implemented with the new model.

I think this can be supported on JSP2.x engines by using an annotation 
processor to generate SimpleTag implementations and the XML descriptor for the 
library. We could also use a ServletContainerInitializer to locate TagLibrary 
classes during webapp startup and pass the information to Jasper.

However, I started to wonder if there was a more efficient model that could 
also be used for the JSPPage itself, one that might avoid generation for all 
cases not involving scriptlets. The thought was that a page can be represented 
by a series of fragments that are either:
  * text
  * text produced by EL ValueExpressions
  * tag invocations
  * scriptlets (but current best practices tend discourage that)

The servlet samples contain various alternatives for representing such a page.

I tried to make these work with the existing JSP APIs as much as possible but 
ran into a couple of impedance mismatches:
  * the JSP interfaces ofter have additional methods which prevent them being 
FunctionalInterfaces
for example, JSPFragment is a abstract class and exposes two operations
  * JSP generally uses checked exceptions (JspException) whereas the 
j.u.function APIs do not. I introduced AbortException as a way to tunnel the 
checked exceptions to the higher levels
  * JSP historically has relied on life-cycle methods e.g. JspFactory’s 
get/releasePageContext. I explored using try-with-resources to control 
life-cycle instead

The interfaces in o.a.jsp.functional try to address these issues.

The code builds but I’ve not verified that it actually works as intended - 
that’s the next step :)

Cheers
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Time for Taglibs to be sent to the archive?

[Jeremy Boynes]

Resurrecting this old thread[1] as the topic came up related to the site 
changes. First message included for context below.

TL;DR: Konstantin and I had interest in new work on Standard, Henri has helped 
with logistics. No-one had any interest in RDC. A concern at the time was lack 
of progress toward a release but since then we released 1.2.1 in 2014/01 and 
recently 1.2.3. The 1.2 version has been picked up by some downstream projects; 
to my knowledge, it is the only implementation available under an Apache-style 
license.

Rather than retire Taglibs now I would propose we attempt to reignite community 
interest which will require steps beyond simply maintaining 1.2.x. I’d propose 
the following:

- Retire RDC to the Attic
- Put Standard 1.2.x in maintenance mode for bug fixes only
- Fix the sub-site so it’s easier to maintain
- Start a tree/branch for new work without the limitations of the now-ancient 
1.2 spec
- Plan to release new work early  often

Thoughts?
Jeremy

[1] 
http://mail-archives.apache.org/mod_mbox/tomcat-dev/201210.mbox/%3C50694B65.9040904%40apache.org%3E

 On Oct 1, 2012, at 12:51 AM, Mark Thomas ma...@apache.org wrote:
 
 In the two+ years since Taglibs moved to the Tomcat project there have
 been a few short bursts of activity totalling just over 200 commits but
 no releases. There has also been no progress towards a migration to
 svnpubsub for the taglibs part of the web site.
 
 Given the lack of progress I would propose that Taglibs is moved to our
 archive. This would mean:
 
 - removing the Taglibs link from tomcat.a.o
 - removing the Taglibs web pages
 - moving the tomcat/tagslibs svn tree to tomcat/archive/taglibs
 - making the Taglibs BZ project read-only
 - moving the Taglibs committers to emeritus status
 - removing the Taglibs from Gump
 
 There is nothing to remove from dist.a.o since there have been no releases
 
 Note: This is intended as a discussion topic - not a formal proposal/vote.
 
 Thoughts?
 
 Mark
 
 -
 To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
 For additional commands, e-mail: dev-h...@tomcat.apache.org
 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: svn commit: r1662913 - /tomcat/tc7.0.x/trunk/test/org/apache/catalina/core/TestStandardContextAliases.java

[Jeremy Boynes]

On Feb 28, 2015, at 1:18 AM, r...@apache.org wrote:
 
 Author: remm
 Date: Sat Feb 28 09:18:57 2015
 New Revision: 1662913
 
 URL: http://svn.apache.org/r1662913
 Log:
 Fix JSTL related test (already in skip list).

Sorry about that, I didn’t think about running the tests for a change affecting 
the examples (I did run the server and checked they worked).

—
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: svn commit: r1662736 - in /tomcat/trunk/webapps/examples/WEB-INF/lib: jstl.jar standard.jar taglibs-standard-impl-1.2.3.jar taglibs-standard-spec-1.2.3.jar

[Jeremy Boynes]

On Feb 27, 2015, at 11:19 AM, Konstantin Kolinko knst.koli...@gmail.com wrote:
 
 2015-02-27 19:26 GMT+03:00  jboy...@apache.org:
 Author: jboynes
 Date: Fri Feb 27 16:26:21 2015
 New Revision: 1662736
 
 URL: http://svn.apache.org/r1662736
 Log:
 Replace old taglibs library with Apache Standard Taglib 1.2.3
 
 Added:
tomcat/trunk/webapps/examples/WEB-INF/lib/taglibs-standard-impl-1.2.3.jar
tomcat/trunk/webapps/examples/WEB-INF/lib/taglibs-standard-spec-1.2.3.jar
 Removed:
tomcat/trunk/webapps/examples/WEB-INF/lib/jstl.jar
tomcat/trunk/webapps/examples/WEB-INF/lib/standard.jar
 
 
 
 I guess that taglibs-standard-impl-*.jar has to be added to
 tomcat.util.scan.StandardJarScanFilter.jarsToSkip setting
 in conf/catalina.properties  to exclude it during TLD scanning, as the
 jar contains no TLDs.
 
 That setting already has jstl.jar but not the new file name.
 
 (The examples app should log a warning when it compiles the first JSP).

This confused me this morning and I was going to look in to it tonight. I did 
not see that warning when running on trunk or tc8 but did on tc7. I expected 
to. Thanks for updating those two branches; I’ll figure out what was going on 
assuming it’s not just that I was under-caffinated.

Thanks
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: svn commit: r1662736 - in /tomcat/trunk/webapps/examples/WEB-INF/lib: jstl.jar standard.jar taglibs-standard-impl-1.2.3.jar taglibs-standard-spec-1.2.3.jar

[Jeremy Boynes]

On Feb 27, 2015, at 6:43 PM, Jeremy Boynes jboy...@apache.org wrote:
 
 On Feb 27, 2015, at 11:19 AM, Konstantin Kolinko knst.koli...@gmail.com 
 wrote:
 
 2015-02-27 19:26 GMT+03:00  jboy...@apache.org:
 Author: jboynes
 Date: Fri Feb 27 16:26:21 2015
 New Revision: 1662736
 
 URL: http://svn.apache.org/r1662736
 Log:
 Replace old taglibs library with Apache Standard Taglib 1.2.3
 
 Added:
   tomcat/trunk/webapps/examples/WEB-INF/lib/taglibs-standard-impl-1.2.3.jar
   tomcat/trunk/webapps/examples/WEB-INF/lib/taglibs-standard-spec-1.2.3.jar
 Removed:
   tomcat/trunk/webapps/examples/WEB-INF/lib/jstl.jar
   tomcat/trunk/webapps/examples/WEB-INF/lib/standard.jar
 
 
 
 I guess that taglibs-standard-impl-*.jar has to be added to
 tomcat.util.scan.StandardJarScanFilter.jarsToSkip setting
 in conf/catalina.properties  to exclude it during TLD scanning, as the
 jar contains no TLDs.
 
 That setting already has jstl.jar but not the new file name.
 
 (The examples app should log a warning when it compiles the first JSP).
 
 This confused me this morning and I was going to look in to it tonight. I did 
 not see that warning when running on trunk or tc8 but did on tc7. I expected 
 to. Thanks for updating those two branches; I’ll figure out what was going on 
 assuming it’s not just that I was under-caffinated.

Nope, the caffine level was adequate. The message is now only emitted if no 
jars contain a TLD whereas it should if at least one does not.

Opened https://bz.apache.org/bugzilla/show_bug.cgi?id=5764
I’ll try to get to it this weekend.

Cheers
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Taglibs sub-site

[Jeremy Boynes]

On Feb 27, 2015, at 3:38 AM, kkoli...@apache.org wrote:
 
 Author: kkolinko
 Date: Fri Feb 27 11:38:13 2015
 New Revision: 1662662
 
 URL: http://svn.apache.org/r1662662
 Log:
 Announcement for Standard Taglib 1.2.3.
 
 I do not like the CVE link (goes to announce@a.o mail archive) and CHANGES 
 link (goes to SVN), as I noted in a FIXME comment in index.xml.  Any better 
 ideas?

We could add a security-taglibs page to the main site and link from the 
security.html page.

I find the frankensite, as Henri called it, a pain in general. I’m thinking 
about merging it in with the main Tomcat site source and give it an overhaul 
(including moving away from using Maven to build it).

It integrates Taglibs more with the main Tomcat project structure. Does anyone 
have any concern about that?

Cheers
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

[ANN} Apache Standard Taglib 1.2.3 released

[Jeremy Boynes]

The Apache Tomcat team announces the immediate availability of Apache Standard 
Taglib 1.2.3.

Apache Standard Taglib is an open source software implementation of the JSP 
Standard Tag Library (JSTL) technology.

This release supports JSTL version 1.2 and includes bug-fixes and improvements 
on the previous 1.2.1 release. The notable changes include:
- Updated -compat library to use container’s EL implementation for JSTL 1.0
- Better classloading support when deployed in-container vs. in-webapp
- Improvements to XML processing, see binary README for additional flags
  
https://www.apache.org/dist/tomcat/taglibs/taglibs-standard-1.2.3/README_bin.txt

Please refer to the change log included in the distribution for a complete list 
of the issues addressed.

Downloads:
http://tomcat.apache.org/download-taglibs.cgi


signature.asc
Description: Message signed with OpenPGP using GPGMail

[SECURITY] CVE-2015-0254 XXE and RCE via XSL extension in JSTL XML tags

[Jeremy Boynes]

CVE-2015-0254 XXE and RCE via XSL extension in JSTL XML tags

Severity: Important

Vendor:
The Apache Software Foundation

Versions Affected:
Standard Taglibs 1.2.1
The unsupported 1.0.x and 1.1.x versions may also be affected.

Description:
When an application uses x:parse or x:transform tags to process untrusted 
XML documents, a request may utilize external entity references to access 
resources on the host system or utilize XSLT extensions that may allow remote 
execution.

Mitigation:
Users should upgrade to Apache Standard Taglibs 1.2.3 or later.

This version uses JAXP’s FEATURE_SECURE_PROCESSING to restrict XML processing. 
Depending on the Java runtime version in use, additional configuration may be 
required:
Java8: External entity access is automatically disabled if a SecurityManager is 
active.
Java7: JAXP properties may need to be used to disable external access. See 
http://docs.oracle.com/javase/tutorial/jaxp/properties/properties.html
Java6 and earlier: A new system property 
org.apache.taglibs.standard.xml.accessExternalEntity may be used to specify the 
protocols that can be used to access external entities. This defaults to “all” 
if no SecurityManager is present and to “” (thereby disabling access) if a 
SecurityManager is detected.

Credit:
David Jorm of IIX


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: TestPojoEndpointBase failing on trunk

[Jeremy Boynes]

On Feb 22, 2015, at 10:57 AM, Mark Thomas ma...@apache.org wrote:
 
 On 19/02/2015 18:27, Mark Thomas wrote:
 On 19/02/2015 15:17, Jeremy Boynes wrote:
 Mark
 
 I am seing regular failures in TestPojoEndpointBase in trunk where
 the socket is not being closed (see attached log). I’ve not dug in
 yet but this has started since r1660358 and may be related to recent
 NIO changes (maybe r1660582?).
 
 I'm fairly sure I have a fix for that wrapped up in my current changes.
 I'll get those committed as soon as I can but I am still working through
 intermittent unit test failures.
 
 To add to the 'fun' my email provider has decided the most helpful thing
 they could do is add a random 20 min to 12+ hour delay to all my e-mail.
 That is causing its own problems - not least of which is me not seeing
 dev@ e-mails in a timely manner. Unsurprisingly DynDNS will be losing a
 customer just as quickly as I can find an alternative to migrate to.
 
 Goodbye to the incompetence of DynDNS / DuoCircle.
 
 Hello to DNSExit/GhettoSMTP and a system that works (and costs 50% to
 75% less).
 
 Having exchanged emails and instant messages with representatives of
 both companies I already have a much warmer feeling about them than I
 did with DynDNS and DuoCircle. Obviously time will tell in terms of
 service but so far so good.
 
 Now I have e-mail arriving with seconds of it being sent, back to what
 I'd much rather be doing...
 
 I think I have a fix for this that I have just committed. The unit tests
 pass consistently (so far) on OSX. Now I am back at home, I'll run them
 on Windows and Linux as well as BuildBot and Gump.
 
 There are still some strange failures on Gump that I don't understand at
 all. Any help investigating those would be much appreciated.

Thanks, this one is passing now. However, I’m seeing some other tests failing:

test:
   [concat] Testsuites with skipped tests:
   [concat] TEST-org.apache.catalina.connector.TestRequest.NIO.txt
   [concat] TEST-org.apache.el.parser.TestELParser.NIO.txt
   [concat] TEST-org.apache.tomcat.util.net.TestClientCert.NIO.txt
   [concat] TEST-org.apache.tomcat.util.net.TestCustomSsl.NIO.txt
   [concat] TEST-org.apache.tomcat.util.net.TestSsl.NIO.txt
   [concat] TEST-org.apache.tomcat.util.net.jsse.openssl.TestCipher.NIO.txt
   [concat] 
TEST-org.apache.tomcat.util.net.jsse.openssl.TestOpenSSLCipherConfigurationParser.NIO.txt
   [concat] TEST-org.apache.tomcat.websocket.TestConnectionLimit.NIO.txt
   [concat] TEST-org.apache.tomcat.websocket.pojo.TestEncodingDecoding.NIO.txt
   [concat] Testsuites with failed tests:
   [concat] TEST-org.apache.tomcat.websocket.TestWebSocketFrameClientSSL.NIO.txt
   [concat] TEST-org.apache.tomcat.websocket.TestWsWebSocketContainer.NIO.txt

This is on OS X 10.10.2 with Java 1.8.0_31
Logs attached.

Testsuite: org.apache.tomcat.websocket.TestWsWebSocketContainer
Tests run: 26, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 70.925 sec
- Standard Error -
22-Feb-2015 14:23:16.747 INFO [main] 
org.apache.catalina.startup.LoggingBaseTest.setUp Starting test case 
[testMaxMessageSize01]
22-Feb-2015 14:23:17.007 INFO [main] org.apache.coyote.AbstractProtocol.init 
Initializing ProtocolHandler [http-nio-127.0.0.1-auto-1]
22-Feb-2015 14:23:17.028 INFO [main] 
org.apache.catalina.core.StandardService.startInternal Starting service Tomcat
22-Feb-2015 14:23:17.028 INFO [main] 
org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: 
Apache Tomcat/9.0.0-dev
22-Feb-2015 14:23:17.151 INFO [main] org.apache.coyote.AbstractProtocol.start 
Starting ProtocolHandler [http-nio-127.0.0.1-auto-1-51485]
22-Feb-2015 14:23:17.446 INFO [main] org.apache.coyote.AbstractProtocol.pause 
Pausing ProtocolHandler [http-nio-127.0.0.1-auto-1-51485]
22-Feb-2015 14:23:17.502 INFO [main] 
org.apache.catalina.core.StandardService.stopInternal Stopping service Tomcat
22-Feb-2015 14:23:17.509 INFO [main] org.apache.coyote.AbstractProtocol.stop 
Stopping ProtocolHandler [http-nio-127.0.0.1-auto-1-51485]
22-Feb-2015 14:23:17.511 INFO [main] org.apache.coyote.AbstractProtocol.destroy 
Destroying ProtocolHandler [http-nio-127.0.0.1-auto-1-51485]
22-Feb-2015 14:23:17.518 INFO [main] 
org.apache.catalina.startup.LoggingBaseTest.setUp Starting test case 
[testMaxMessageSize02]
22-Feb-2015 14:23:17.519 INFO [main] org.apache.coyote.AbstractProtocol.init 
Initializing ProtocolHandler [http-nio-127.0.0.1-auto-2]
22-Feb-2015 14:23:17.520 INFO [main] 
org.apache.catalina.core.StandardService.startInternal Starting service Tomcat
22-Feb-2015 14:23:17.520 INFO [main] 
org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: 
Apache Tomcat/9.0.0-dev
22-Feb-2015 14:23:17.525 INFO [main] org.apache.coyote.AbstractProtocol.start 
Starting ProtocolHandler [http-nio-127.0.0.1-auto-2-51488]
22-Feb-2015 14:23:17.633 INFO [main] org.apache.coyote.AbstractProtocol.pause 
Pausing ProtocolHandler [http-nio-127.0.0.1-auto-2-51488]
22-Feb

Re: [VOTE][RESULT] Release Apache Taglibs 1.2.3

[Jeremy Boynes]

The following vote were cast:

Binding:
+1: jboynes, kkolinko, kfujino

The vote therefore passes.

I will release the staging repository. Could a PMC member please update 
dist.a.o. I will announce after that.

Thanks
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

TestPojoEndpointBase failing on trunk

[Jeremy Boynes]

Mark

I am seing regular failures in TestPojoEndpointBase in trunk where the socket 
is not being closed (see attached log). I’ve not dug in yet but this has 
started since r1660358 and may be related to recent NIO changes (maybe 
r1660582?).

Cheers
Jeremy

Testsuite: org.apache.tomcat.websocket.pojo.TestPojoEndpointBase
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 10.992 sec
- Standard Error -
19-Feb-2015 07:12:20.251 INFO [main] 
org.apache.catalina.startup.LoggingBaseTest.setUp Starting test case 
[testBug54716]
19-Feb-2015 07:12:20.680 INFO [main] org.apache.coyote.AbstractProtocol.init 
Initializing ProtocolHandler [http-nio-127.0.0.1-auto-1]
19-Feb-2015 07:12:20.703 INFO [main] 
org.apache.catalina.core.StandardService.startInternal Starting service Tomcat
19-Feb-2015 07:12:20.703 INFO [main] 
org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: 
Apache Tomcat/9.0.0-dev
19-Feb-2015 07:12:20.810 INFO [main] org.apache.coyote.AbstractProtocol.start 
Starting ProtocolHandler [http-nio-127.0.0.1-auto-1-49282]
19-Feb-2015 07:12:20.993 SEVERE [http-nio-127.0.0.1-auto-1-exec-1] 
org.apache.tomcat.websocket.pojo.PojoEndpointBase.onError No error handling 
configured for [org.apache.tomcat.websocket.pojo.TestPojoEndpointBase$Bug54716] 
and the following error occurred
 java.lang.RuntimeException
at 
org.apache.tomcat.websocket.pojo.TestPojoEndpointBase$Bug54716.onOpen(TestPojoEndpointBase.java:131)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at 
org.apache.tomcat.websocket.pojo.PojoEndpointBase.doOnOpen(PojoEndpointBase.java:66)
at 
org.apache.tomcat.websocket.pojo.PojoEndpointServer.onOpen(PojoEndpointServer.java:70)
at 
org.apache.tomcat.websocket.server.WsHttpUpgradeHandler.init(WsHttpUpgradeHandler.java:138)
at 
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:691)
at 
org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:181)
at 
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741)
at 
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1698)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at 
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)

19-Feb-2015 07:12:25.997 INFO [main] org.apache.coyote.AbstractProtocol.pause 
Pausing ProtocolHandler [http-nio-127.0.0.1-auto-1-49282]
19-Feb-2015 07:12:26.055 INFO [main] 
org.apache.catalina.core.StandardService.stopInternal Stopping service Tomcat
19-Feb-2015 07:12:26.059 WARNING [localhost-startStop-1] 
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web 
application [ROOT] appears to have started a thread named 
[http-nio-127.0.0.1-auto-1-exec-1] but has failed to stop it. This is very 
likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
 
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
 java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
 
org.apache.tomcat.websocket.FutureToSendHandler.get(FutureToSendHandler.java:93)
 
org.apache.tomcat.websocket.WsRemoteEndpointImplBase.startMessageBlock(WsRemoteEndpointImplBase.java:275)
 org.apache.tomcat.websocket.WsSession.sendCloseMessage(WsSession.java:570)
 org.apache.tomcat.websocket.WsSession.doClose(WsSession.java:478)
 org.apache.tomcat.websocket.WsSession.close(WsSession.java:445)
 org.apache.tomcat.websocket.WsSession.close(WsSession.java:439)
 
org.apache.tomcat.websocket.pojo.PojoEndpointBase.handleOnOpenOrCloseError(PojoEndpointBase.java:96)
 
org.apache.tomcat.websocket.pojo.PojoEndpointBase.doOnOpen(PojoEndpointBase.java:79)
 
org.apache.tomcat.websocket.pojo.PojoEndpointServer.onOpen(PojoEndpointServer.java:70)
 
org.apache.tomcat.websocket.server.WsHttpUpgradeHandler.init(WsHttpUpgradeHandler.java:138)
 
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:691)
 
org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:181)
 
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741)
 

Re: Question about taglibs. Issue 37466

[Jeremy Boynes]

On Feb 17, 2015, at 8:23 AM, Jeremy Boynes jboy...@apache.org wrote:
 
 On Feb 17, 2015, at 4:34 AM, Konstantin Kolinko knst.koli...@gmail.com 
 wrote:
 
 2015-02-14 20:04 GMT+03:00 Stephan van Loendersloot (LIST)
 step...@republika.nl:
 Hi everyone,
 
 I have a question about this issue:
 https://bz.apache.org/bugzilla/show_bug.cgi?id=37466
 
 I tried to switch to the latest Tomcat TagLibs implementation, but due to
 this fixed bug, it seems that posted form parameters aren't visible anymore
 in relative pages imported by c:import tags.
 
 The JSTl specicication 1.2, Section 7.2 (page 57) states that relative URLs
 should be processed in the exact same way as the include action of the JSP
 specification (jsp:include) when used in the same application context.
 
 The older, Jakarta TagLibs don't have this problem, but the Tomcat TagLibs
 do...
 
 Can anyone tell me if the fix breaks the specification guidelines, or did I
 misread the whole thing?
 
 
 
 BZ 37466 fix (r495005) was committed 8 years ago.
 
 You need to provide an example that reproduces your issue.  Wrapping a
 request with overwritten getMethod() should not break parsing of POST
 parameters.  The parameters parsing is done by tomcat internals and
 operates on the original request, not on the wrapper.
 
 I think there may be an issue here but would like an example to confirm. This 
 change was introduced between 1.1 and 1.2 and after GlassFish forked away.
 
 The original thread that led to BZ 37466 related to HEAD requests where the 
 original HttpServletRequest was being passed to a RequestDispatcher. What I 
 took from the thread is that this would result an empty response from the 
 imported resource which the application was not expecting - it tried to parse 
 empty content and failed. The r495005 patch attempts to fix this by forcing 
 the method to GET, mirroring the semantic of an external absolute URL where 
 the JSTL spec mandates a GET request is made when using HTTP.
 
 I’m not convinced this change was correct. For relative URLs it specifies the 
 “whole environment is available … as well as request parameters” and by 
 forcing the method to GET we are indicating that there is no request body to 
 parse and so parameters in the original POST body should be skipped. As 
 Konstantin said, Tomcat does not take account of wrappers so will still 
 extract parameters from the request body but that may not be true for other 
 containers or if the application code is parsing the request content.
 
 Perhaps the issue here is actually with Tomcat’s implementation of 
 HttpServlet and/or DefaultServlet where it appears doHead() never returns 
 content even for dispatched includes. If these only suppressed content for 
 the original response but did return content for included responses then BZ 
 37466 would not be an issue. This would also seem to be necessary to return 
 the same headers for HEAD as GET e.g. ensuring that the Content-Length 
 returned included bytes from the included resource.
 
 Stephan, please can you submit an example showing what’s failing for you and 
 include info on which container you are using and what version of taglibs you 
 are upgrading from.
 

In the original issue, a JSP page was using c:import to read a XML resource 
for parsing:
  c:import url=/org/apache/taglibs/standard/tag/el/core/Test37466.xml 
varReader=xmlSource”

This is dispatched and handled by DefaultServlet:
  service() (from HttpServlet) uses getMethod() to determine that doHead() 
should be called
  doHead() calls serveResource() with content = false
  if content is false, serveResource() does not write any data to the response
  the value captured by the HttpResponseWrapper will be empty

If instead, a JSP is used to serve the included resource its _jspService method 
does not check the HTTP method and always emits content.

This can be verified with a simple JSP that does a jsp:include of a .txt or 
.jsp resource where the Content-Length header is different depending on whether 
a GET or HEAD request is made.

Similarly, HttpServlet’s doHead() method is implemented by wrapping the 
response in a NoBodyResponse and then calling doGet().

The patch for 37466 attempts to fix this by overriding getMethod() to always 
return GET. This avoids the issue with DefaultServlet but breaks other servlets 
that actually use the method to dispatch e.g. anything that expects to handle a 
POST request in doPost().

I think instead the fix should be in Tomcat’s HttpServlet and DefaultServlet 
which should not suppress the output if they being called as an include. I’ll 
open an issue against Tomcat and propose a patch for this.

—
Jeremy


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Question about taglibs. Issue 37466

[Jeremy Boynes]

 On Feb 18, 2015, at 3:10 AM, Stephan van Loendersloot (LIST) 
 step...@republika.nl wrote:
 
 Hi Konstantin, Jeremy,
 
 Yes I can provide a simple example shortly...
 
 Apologies for breaking the first rule of questioning: provide all details 
 about operating system, container, libraries, etc.
 
 Would it be best if I continue this discussion here or at Jira?

I reopened 37466[1]. Please add your example as an attachment there.
Thanks
Jeremy

[1] https://bz.apache.org/bugzilla/show_bug.cgi?id=37466


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Question about taglibs. Issue 37466

[Jeremy Boynes]

On Feb 17, 2015, at 4:34 AM, Konstantin Kolinko knst.koli...@gmail.com wrote:
 
 2015-02-14 20:04 GMT+03:00 Stephan van Loendersloot (LIST)
 step...@republika.nl:
 Hi everyone,
 
 I have a question about this issue:
 https://bz.apache.org/bugzilla/show_bug.cgi?id=37466
 
 I tried to switch to the latest Tomcat TagLibs implementation, but due to
 this fixed bug, it seems that posted form parameters aren't visible anymore
 in relative pages imported by c:import tags.
 
 The JSTl specicication 1.2, Section 7.2 (page 57) states that relative URLs
 should be processed in the exact same way as the include action of the JSP
 specification (jsp:include) when used in the same application context.
 
 The older, Jakarta TagLibs don't have this problem, but the Tomcat TagLibs
 do...
 
 Can anyone tell me if the fix breaks the specification guidelines, or did I
 misread the whole thing?
 
 
 
 BZ 37466 fix (r495005) was committed 8 years ago.
 
 You need to provide an example that reproduces your issue.  Wrapping a
 request with overwritten getMethod() should not break parsing of POST
 parameters.  The parameters parsing is done by tomcat internals and
 operates on the original request, not on the wrapper.

I think there may be an issue here but would like an example to confirm. This 
change was introduced between 1.1 and 1.2 and after GlassFish forked away.

The original thread that led to BZ 37466 related to HEAD requests where the 
original HttpServletRequest was being passed to a RequestDispatcher. What I 
took from the thread is that this would result an empty response from the 
imported resource which the application was not expecting - it tried to parse 
empty content and failed. The r495005 patch attempts to fix this by forcing the 
method to GET, mirroring the semantic of an external absolute URL where the 
JSTL spec mandates a GET request is made when using HTTP.

I’m not convinced this change was correct. For relative URLs it specifies the 
“whole environment is available … as well as request parameters” and by forcing 
the method to GET we are indicating that there is no request body to parse and 
so parameters in the original POST body should be skipped. As Konstantin said, 
Tomcat does not take account of wrappers so will still extract parameters from 
the request body but that may not be true for other containers or if the 
application code is parsing the request content.

Perhaps the issue here is actually with Tomcat’s implementation of HttpServlet 
and/or DefaultServlet where it appears doHead() never returns content even for 
dispatched includes. If these only suppressed content for the original response 
but did return content for included responses then BZ 37466 would not be an 
issue. This would also seem to be necessary to return the same headers for HEAD 
as GET e.g. ensuring that the Content-Length returned included bytes from the 
included resource.

Stephan, please can you submit an example showing what’s failing for you and 
include info on which container you are using and what version of taglibs you 
are upgrading from.

Thanks
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [VOTE] Release Apache Taglibs 1.2.3

[Jeremy Boynes]

Ping?

 On Feb 13, 2015, at 7:46 AM, Jeremy Boynes jboy...@apache.org wrote:
 
 Please could we have a third set of eyes on this release.
 
 Thanks
 Jeremy
 
 On Feb 10, 2015, at 8:40 PM, Jeremy Boynes jboy...@apache.org wrote:
 
 The proposed Apache Taglibs 1.2.3 release is now available for voting.
 
 It can be obtained from:
 https://dist.apache.org/repos/dist/dev/tomcat/taglibs/taglibs-standard-1.2.3/
 
 The Maven staging repo is:
 https://repository.apache.org/content/repositories/orgapachetomcat-1034/
 
 The SVN tag is:
 http://svn.apache.org/repos/asf/tomcat/taglibs/standard/tags/taglibs-standard-1.2.3/
 
 The proposed 1.2.3 release is:
 [ ] Broken - do not release
 [ ] Stable - go ahead and release as 1.2.3 Stable
 
 Thanks
 Jeremy
 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [VOTE] Release Apache Taglibs 1.2.3

[Jeremy Boynes]

Please could we have a third set of eyes on this release.

Thanks
Jeremy

 On Feb 10, 2015, at 8:40 PM, Jeremy Boynes jboy...@apache.org wrote:
 
 The proposed Apache Taglibs 1.2.3 release is now available for voting.
 
 It can be obtained from:
 https://dist.apache.org/repos/dist/dev/tomcat/taglibs/taglibs-standard-1.2.3/
 
 The Maven staging repo is:
 https://repository.apache.org/content/repositories/orgapachetomcat-1034/
 
 The SVN tag is:
 http://svn.apache.org/repos/asf/tomcat/taglibs/standard/tags/taglibs-standard-1.2.3/
 
 The proposed 1.2.3 release is:
 [ ] Broken - do not release
 [ ] Stable - go ahead and release as 1.2.3 Stable
 
 Thanks
 Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [VOTE] Release Apache Taglibs 1.2.3

[Jeremy Boynes]

 On Feb 10, 2015, at 8:40 PM, Jeremy Boynes jboy...@apache.org wrote:
 
 The proposed Apache Taglibs 1.2.3 release is now available for voting.
 
 It can be obtained from:
 https://dist.apache.org/repos/dist/dev/tomcat/taglibs/taglibs-standard-1.2.3/
 
 The Maven staging repo is:
 https://repository.apache.org/content/repositories/orgapachetomcat-1034/
 
 The SVN tag is:
 http://svn.apache.org/repos/asf/tomcat/taglibs/standard/tags/taglibs-standard-1.2.3/
 
 The proposed 1.2.3 release is:
 [ ] Broken - do not release
 [X] Stable - go ahead and release as 1.2.3 Stable

JSTL TCK passed in the following environment:
Mac OS X 10.10.2
Apache Tomcat 7.0.57
Apache Xalan 2.7.1
CATALINA_OPTS=-Dorg.apache.taglibs.standard.xml.accessExternalEntity=http 
-Djavax.xml.accessExternalDTD=http

with the following configurations:
Java 1.8.0_31, -jstlel jar, in container, with security manager
Java 1.8.0_31, -compat jar, in container, with security manager
Java 1.7.0_75, -jstlel jar, in container, with security manager
Java 1.6.0_65, -jstlel jar, in container, with security manager

Thanks
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

[VOTE] Release Apache Taglibs 1.2.3

[Jeremy Boynes]

The proposed Apache Taglibs 1.2.3 release is now available for voting.

It can be obtained from:
 https://dist.apache.org/repos/dist/dev/tomcat/taglibs/taglibs-standard-1.2.3/

The Maven staging repo is:
 https://repository.apache.org/content/repositories/orgapachetomcat-1034/

The SVN tag is:
 
http://svn.apache.org/repos/asf/tomcat/taglibs/standard/tags/taglibs-standard-1.2.3/

The proposed 1.2.3 release is:
[ ] Broken - do not release
[ ] Stable - go ahead and release as 1.2.3 Stable

Thanks
Jeremy


signature.asc
Description: Message signed with OpenPGP using GPGMail

[RESULT] [VOTE] Release Apache Taglibs 1.2.2

[Jeremy Boynes]

On Feb 5, 2015, at 10:01 AM, Konstantin Kolinko knst.koli...@gmail.com wrote:
 
 2015-02-02 14:05 GMT+03:00 Konstantin Kolinko knst.koli...@gmail.com:
 2015-01-27 19:08 GMT+03:00 Jeremy Boynes jer...@boynes.com:
 
 On Jan 27, 2015, at 7:59 AM, Konstantin Kolinko knst.koli...@gmail.com 
 wrote:
 
 Here comes my vote:
 
 The proposed 1.2.2 release is:
 [ ] Broken - do not release
 [x] Stable - go ahead and release as 1.2.2 Stable
 
 
 Tested (taglibs-standard-spec-1.2.2.jar +
 taglibs-standard-impl-1.2.2.jar) on my web applications.
 
 
 Unfortunately I have to change the above to
 [x] Broken - do not release
 
 
 My tests were on Java 7 and later.
 
 The library breaks when running on Java 5.
 E.g. Tomcat 6 + JDK 5u20 + Tomcat examples webapp.
 
 I am looking for a fix.
 For starters, the project has a compilation failure under Java 5 in a
 test class.
 
 XPathFactory.newInstance(String, String, ClassLoader) method used in
 ForEachTagTest is @since Java 1.6.

I’m canceling this vote due to regressions seen on older JREs.

Cheers
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: svn commit: r1657653 - in /tomcat/taglibs/standard/trunk: impl/src/test/java/org/apache/taglibs/standard/tag/common/xml/ForEachTagTest.java pom.xml

[Jeremy Boynes]

 On Feb 5, 2015, at 3:32 PM, Konstantin Kolinko kkoli...@apache.org wrote:
 
 2015-02-05 21:12 GMT+03:00  kkoli...@apache.org:
 Author: kkolinko
 Date: Thu Feb  5 18:12:31 2015
 New Revision: 1657653
 
 URL: http://svn.apache.org/r1657653
 Log:
 Fix compilation failure in a test class when using JDK 1.5.
 The XPathFactory.newInstance(String, String, ClassLoader) method is @since 
 Java 1.6.
 
…

 For a record,
 if I enable the xpathPerformance test (by removing @Ignore annotation)
 and run mvn install with JDK 1.6,
 
 using 1-argument version of XPathFactory.newInstance() results in printing
 [[[
 Running org.apache.taglibs.standard.tag.common.xml.ForEachTagTest
 factory.getClass() = class org.apache.xpath.jaxp.XPathFactoryImpl
 ]]]
 
 using 3-arguments version of XPathFactory.newInstance() results in printing
 [[[
 Running org.apache.taglibs.standard.tag.common.xml.ForEachTagTest
 factory.getClass() = class 
 com.sun.org.apache.xpath.internal.jaxp.XPathFactoryIm
 pl
 ]]]
 
 So there is a difference.
 At the same time, in both cases the test does not complete after
 running for 1 hour. I had to abort the tests and kill java process
 forked by maven/junit. So here is no difference.
 
 A comment for @Ignore is Takes  20s to run, so I was prepared to
 wait, but an hour is more than I expected from that comment.
 
 Maybe reducing the size of a sample document
 (newBenchmarkDocument(20)) will help.
 
 This is not my priority for now. I just run it as I wanted to verify
 whether my code change mattered.

This test was added when I was evaluating different options for addressing the 
XPath performance problems from bug #27717. I was trying to find a solution 
that used the standard JDK XPath APIs rather than having to rely on Xalan’s 
internal DTM interface. This test was used to distinguish between the 
JAXP-conformant xpath implementation built into the JDK and the one supplied as 
part of Xalan. As it turned out, both suffered from the same issue - they are 
O(N^2) compared to the xalanPerformance test which is O(N).

The test is @Ignored as it is just for performance measurement rather than 
functional testing. I left it in in case anyone wanted to re-run the tests if 
an improved JDK XPath implementation came out.

The 3-arg form here is used to ensure the JRE’s implementation is used rather 
than Xalan’s. This change breaks that as illustrated above where you can it is 
using the o.a.xpath version not the com.sun version from the JDK.

An alternative might be to instantiate the class using refection as we should 
not reference a com.sun class directly. Alternatively, we could just revert the 
change as this just blocks building with JDK5 which is EOL’d (although we still 
generate Java5 bytecode for the production classes.)

—
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [VOTE] Release Apache Taglibs 1.2.2

[Jeremy Boynes]

Ping ?

 On Feb 2, 2015, at 8:29 PM, Jeremy Boynes jboy...@apache.org wrote:
 
 
 On Feb 2, 2015, at 3:05 AM, Konstantin Kolinko knst.koli...@gmail.com 
 wrote:
 
 2015-01-27 19:08 GMT+03:00 Jeremy Boynes jer...@boynes.com:
 
 On Jan 27, 2015, at 7:59 AM, Konstantin Kolinko knst.koli...@gmail.com 
 wrote:
 
 2015-01-26 20:17 GMT+03:00 Jeremy Boynes jboy...@apache.org:
 The proposed Apache Taglibs 1.2.2 release is now available for voting.
 
 It can be obtained from:
 https://dist.apache.org/repos/dist/dev/tomcat/taglibs/taglibs-standard-1.2.2/
 
 The Maven staging repo is:
 https://repository.apache.org/content/repositories/orgapachetomcat-1032/
 
 The SVN tag is:
 http://svn.apache.org/repos/asf/tomcat/taglibs/standard/tags/taglibs-standard-1.2.2/
 
 
 The directory at dist.apache.org contains only 2 files,
 taglibs-standard-1.2.2-source-release.zip
 taglibs-standard-1.2.2-source-release.zip.asc
 
 I went on and added NOTICE, READMEs and checksums there.
 I plan to add the other files from Maven, like it was done for 1.2.1 here:
 https://dist.apache.org/repos/dist/release/tomcat/taglibs/taglibs-standard-1.2.1/
 
 But here is a question the taglibs-standard-1.2.2-source-release.zip
 on dist.a.o differs from one in Maven at
 https://repository.apache.org/content/repositories/orgapachetomcat-1032/org/apache/taglibs/taglibs-standard/1.2.2/
 
 The only difference is timestamps. The contents of the files in the
 archives is the same, but in the Maven staing repository zip all files
 have the same timestamp, while in the dist.a.o zip the timestamps for
 files in the archive are different.
 
 I think that it would be better to replace the zip file on dist.a.o
 with the one from Maven.  Any objections or concerns here?
 
 +1
 
 Thanks for catching that.
 Jeremy
 
 Done.
 I replaced the sources zip and added binary jars - all taken from the
 Maven staging repo.  We now have the same set of files like we had for
 1.2.1.
 
 Here comes my vote:
 
 The proposed 1.2.2 release is:
 [ ] Broken - do not release
 [x] Stable - go ahead and release as 1.2.2 Stable
 
 
 Tested (taglibs-standard-spec-1.2.2.jar +
 taglibs-standard-impl-1.2.2.jar) on my web applications.
 
 Thank you.
 
 I have 2 +1 votes so far (jboynes + kkolinko), would appreciate if others 
 could take a look.
 Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [VOTE] Release Apache Taglibs 1.2.2

[Jeremy Boynes]

 On Feb 2, 2015, at 3:05 AM, Konstantin Kolinko knst.koli...@gmail.com wrote:
 
 2015-01-27 19:08 GMT+03:00 Jeremy Boynes jer...@boynes.com:
 
 On Jan 27, 2015, at 7:59 AM, Konstantin Kolinko knst.koli...@gmail.com 
 wrote:
 
 2015-01-26 20:17 GMT+03:00 Jeremy Boynes jboy...@apache.org:
 The proposed Apache Taglibs 1.2.2 release is now available for voting.
 
 It can be obtained from:
 https://dist.apache.org/repos/dist/dev/tomcat/taglibs/taglibs-standard-1.2.2/
 
 The Maven staging repo is:
 https://repository.apache.org/content/repositories/orgapachetomcat-1032/
 
 The SVN tag is:
 http://svn.apache.org/repos/asf/tomcat/taglibs/standard/tags/taglibs-standard-1.2.2/
 
 
 The directory at dist.apache.org contains only 2 files,
 taglibs-standard-1.2.2-source-release.zip
 taglibs-standard-1.2.2-source-release.zip.asc
 
 I went on and added NOTICE, READMEs and checksums there.
 I plan to add the other files from Maven, like it was done for 1.2.1 here:
 https://dist.apache.org/repos/dist/release/tomcat/taglibs/taglibs-standard-1.2.1/
 
 But here is a question the taglibs-standard-1.2.2-source-release.zip
 on dist.a.o differs from one in Maven at
 https://repository.apache.org/content/repositories/orgapachetomcat-1032/org/apache/taglibs/taglibs-standard/1.2.2/
 
 The only difference is timestamps. The contents of the files in the
 archives is the same, but in the Maven staing repository zip all files
 have the same timestamp, while in the dist.a.o zip the timestamps for
 files in the archive are different.
 
 I think that it would be better to replace the zip file on dist.a.o
 with the one from Maven.  Any objections or concerns here?
 
 +1
 
 Thanks for catching that.
 Jeremy
 
 Done.
 I replaced the sources zip and added binary jars - all taken from the
 Maven staging repo.  We now have the same set of files like we had for
 1.2.1.
 
 Here comes my vote:
 
 The proposed 1.2.2 release is:
 [ ] Broken - do not release
 [x] Stable - go ahead and release as 1.2.2 Stable
 
 
 Tested (taglibs-standard-spec-1.2.2.jar +
 taglibs-standard-impl-1.2.2.jar) on my web applications.

Thank you.

I have 2 +1 votes so far (jboynes + kkolinko), would appreciate if others could 
take a look.
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [VOTE] Release Apache Tomcat 7.0.58

[Jeremy Boynes]

On Jan 26, 2015, at 6:54 AM, Violeta Georgieva violet...@apache.org wrote:
 
 The proposed Apache Tomcat 7.0.58 release is now available for voting.
 
 It can be obtained from:
 https://dist.apache.org/repos/dist/dev/tomcat/tomcat-7/v7.0.58/
 The Maven staging repo is:
 https://repository.apache.org/content/repositories/orgapachetomcat-1031/
 The svn tag is:
 http://svn.apache.org/repos/asf/tomcat/tc7.0.x/tags/TOMCAT_7_0_58/
 
 The proposed 7.0.58 release is:
 [X] Broken - do not release
 [ ] Stable - go ahead and release as 7.0.58 Stable

https://issues.apache.org/bugzilla/show_bug.cgi?id=57504
breaks JspC with webapps that use tags


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [VOTE] Release Apache Taglibs 1.2.2

[Jeremy Boynes]

 On Jan 27, 2015, at 7:59 AM, Konstantin Kolinko knst.koli...@gmail.com 
 wrote:
 
 2015-01-26 20:17 GMT+03:00 Jeremy Boynes jboy...@apache.org:
 The proposed Apache Taglibs 1.2.2 release is now available for voting.
 
 It can be obtained from:
  
 https://dist.apache.org/repos/dist/dev/tomcat/taglibs/taglibs-standard-1.2.2/
 
 The Maven staging repo is:
  https://repository.apache.org/content/repositories/orgapachetomcat-1032/
 
 The SVN tag is:
  
 http://svn.apache.org/repos/asf/tomcat/taglibs/standard/tags/taglibs-standard-1.2.2/
 
 The proposed 1.2.2 release is:
 [ ] Broken - do not release
 [ ] Stable - go ahead and release as 1.2.2 Stable
 
 The directory at dist.apache.org contains only 2 files,
 taglibs-standard-1.2.2-source-release.zip
 taglibs-standard-1.2.2-source-release.zip.asc
 
 I went on and added NOTICE, READMEs and checksums there.
 I plan to add the other files from Maven, like it was done for 1.2.1 here:
 https://dist.apache.org/repos/dist/release/tomcat/taglibs/taglibs-standard-1.2.1/
 
 But here is a question the taglibs-standard-1.2.2-source-release.zip
 on dist.a.o differs from one in Maven at
 https://repository.apache.org/content/repositories/orgapachetomcat-1032/org/apache/taglibs/taglibs-standard/1.2.2/
 
 The only difference is timestamps. The contents of the files in the
 archives is the same, but in the Maven staing repository zip all files
 have the same timestamp, while in the dist.a.o zip the timestamps for
 files in the archive are different.
 
 I think that it would be better to replace the zip file on dist.a.o
 with the one from Maven.  Any objections or concerns here?

+1

Thanks for catching that.
Jeremy


-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Re: Release of Apache Taglibs

[Jeremy Boynes]

On Jan 8, 2015, at 9:26 PM, Jeremy Boynes jboy...@apache.org wrote:
 
 On Jan 8, 2015, at 1:42 PM, Konstantin Kolinko knst.koli...@gmail.com wrote:
 
 2015-01-08 9:42 GMT+03:00 Jeremy Boynes jboy...@apache.org:
 I made a couple of minor improvements to Apache Taglibs last year related 
 to XML parsing and to how we load libraries (it now uses the TCCL which 
 means the jars can be shared between webapps as well as be included in a 
 webapp as before). I would like to perform a minor release (1.2.2) to 
 include these. Before I do, are there any other issues that should be 
 addressed?
 
 One change in 1.2 I think was not useful was the split of the 1.0 EL 
 evaluator into its own jar. I propose to merge that back so we end up with 
 two jars: one with the javax classes and one with our implementation.
 
 I do not see a point in merging them back. Personally, I do not use
 JSTL 1.0 EL implementation and do not plan to use it. (IIRC, the only
 occurrence when I used them was when I made a typo in tag library url,
 like copy-pasting from a wrong tld file. Hilarity ensued).
 Unexpectedly using a wrong implementation may be confusing.
 
 That was the kind of thing that prompted me to split it off in the first 
 place.
 
 The other was to the allow users to use a newer EL implementation (such as 
 the one from the container) which could have performance or efficiency 
 improvements over the original one, especially now EL has been decoupled from 
 JSTL and JSP. However, I found a problem with the -compat packaging that 
 would prevent that jar being used; see #57427. As no-one else had reported 
 that my assumption was that this was not that useful.
 
 Another point is that I do not like changing release packaging in a
 point release.  Is this driven by a real users' demand? Have there
 been any public complaints from real users (e.g. an issue in
 Bugzilla)? What do we do with Maven artifacts? Abandon one of them?
 
 As I’d not seen any feedback on the split (public or private) I was thinking 
 of simply reverting to the 1.1.x model. Sounds better though to fix the 
 problems with the -compat version and defer merging until a potential 1.3 
 release.

I’ve fixed the issues with the -compat version and verified that that 
configuration also passes the TCK. I am going to update the release notes and 
start the release process.

Cheers
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

[VOTE] Release Apache Taglibs 1.2.2

[Jeremy Boynes]

The proposed Apache Taglibs 1.2.2 release is now available for voting.

It can be obtained from:
  https://dist.apache.org/repos/dist/dev/tomcat/taglibs/taglibs-standard-1.2.2/

The Maven staging repo is:
  https://repository.apache.org/content/repositories/orgapachetomcat-1032/

The SVN tag is:
  
http://svn.apache.org/repos/asf/tomcat/taglibs/standard/tags/taglibs-standard-1.2.2/

The proposed 1.2.2 release is:
[ ] Broken - do not release
[ ] Stable - go ahead and release as 1.2.2 Stable

Thanks
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [VOTE] Release Apache Taglibs 1.2.2

[Jeremy Boynes]

On Jan 26, 2015, at 9:17 AM, Jeremy Boynes jboy...@apache.org wrote:
 
 The proposed 1.2.2 release is:
 [ ] Broken - do not release
 [X] Stable - go ahead and release as 1.2.2 Stable

JSTL TCK passed in the following environment:
Mac OSX 10.10.1
Java 1.7.0_75
Apache Tomcat 7.0.57
Apache Xalan 2.7.1

with the following configurations:
-jstlel jar, in webapp, no security manager
-jstlel jar, in webapp, with security manager
-jstlel jar, in container, with security manager
-compat jar, in container, with security manager



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Release of Apache Taglibs

[Jeremy Boynes]

On Jan 8, 2015, at 1:42 PM, Konstantin Kolinko knst.koli...@gmail.com wrote:
 
 2015-01-08 9:42 GMT+03:00 Jeremy Boynes jboy...@apache.org:
 I made a couple of minor improvements to Apache Taglibs last year related to 
 XML parsing and to how we load libraries (it now uses the TCCL which means 
 the jars can be shared between webapps as well as be included in a webapp as 
 before). I would like to perform a minor release (1.2.2) to include these. 
 Before I do, are there any other issues that should be addressed?
 
 One change in 1.2 I think was not useful was the split of the 1.0 EL 
 evaluator into its own jar. I propose to merge that back so we end up with 
 two jars: one with the javax classes and one with our implementation.
 
 I do not see a point in merging them back. Personally, I do not use
 JSTL 1.0 EL implementation and do not plan to use it. (IIRC, the only
 occurrence when I used them was when I made a typo in tag library url,
 like copy-pasting from a wrong tld file. Hilarity ensued).
 Unexpectedly using a wrong implementation may be confusing.

That was the kind of thing that prompted me to split it off in the first place.

The other was to the allow users to use a newer EL implementation (such as the 
one from the container) which could have performance or efficiency improvements 
over the original one, especially now EL has been decoupled from JSTL and JSP. 
However, I found a problem with the -compat packaging that would prevent that 
jar being used; see #57427. As no-one else had reported that my assumption was 
that this was not that useful.

 Another point is that I do not like changing release packaging in a
 point release.  Is this driven by a real users' demand? Have there
 been any public complaints from real users (e.g. an issue in
 Bugzilla)? What do we do with Maven artifacts? Abandon one of them?

As I’d not seen any feedback on the split (public or private) I was thinking of 
simply reverting to the 1.1.x model. Sounds better though to fix the problems 
with the -compat version and defer merging until a potential 1.3 release.

—
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Release of Apache Taglibs

[Jeremy Boynes]

I made a couple of minor improvements to Apache Taglibs last year related to 
XML parsing and to how we load libraries (it now uses the TCCL which means the 
jars can be shared between webapps as well as be included in a webapp as 
before). I would like to perform a minor release (1.2.2) to include these. 
Before I do, are there any other issues that should be addressed?

One change in 1.2 I think was not useful was the split of the 1.0 EL evaluator 
into its own jar. I propose to merge that back so we end up with two jars: one 
with the javax classes and one with our implementation.

Thanks
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: TldScanner + Config, was: Jasper Improvements

[Jeremy Boynes]

On Feb 27, 2014, at 4:09 AM, Konstantin Kolinko knst.koli...@gmail.com wrote:

 2014-02-27 10:17 GMT+04:00 Jeremy Boynes jboy...@apache.org:
 
 I was thinking of supporting compile-on-start as a option where the 
 initializer would scan the war for JSP(x) files and initiate the compilation 
 during startup. This was intended for production use where use of JspC was 
 not practical (e.g. because the actual version of runtime is not known at 
 pre-compilation time) but where taking a hit on access would be undesirable.
 
 http://tomcat.apache.org/tomcat-7.0-doc/jasper-howto.html
 - Background JSP compilation
 
 Compiling can take notable time, so I think it wouldn't be good to
 delay application startup for that time.

I had in mind a production configuration where the JSPs are not expected to 
change between deployments. IIRC, background compilation only works with JSPs 
that have already been hit so there is still a compilation delay on first 
touch. This adds jitter to the latency timings for an application. The idea was 
to avoid that by compiling everything during startup before the application 
starts taking live traffic.

As you say, this would slow startup down but (assuming multiple hosts or 
parallel deployment) would not impact user requests. A variation would be to 
kick off a background task so that startup is not delayed.

 
 I was thinking there to combine the TLD and JSP scans into one.
 
 Those are different scans.
 For TLDs you have to scan jar files.
 For JSPs you have to scan application resources.

TldScanner#scanResourcePaths has to scan all application resources under 
/WEB-INF as well. For a MVC-style application where the JSPs are under /WEB-INF 
there can be substantial overlap.

—
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

TldScanner + Config, was: Jasper Improvements

[Jeremy Boynes]

On Feb 26, 2014, at 4:55 PM, Greg Wilkins gr...@intalio.com wrote:

 Just in case it was missed, here is the pull request I generated:
 
 https://github.com/apache/tomcat/pull/4
 
 Happy to talk about individual changes either in the comments there or here.

Related to the TldScanner changes, I’m assuming you have some integration code 
in Jetty that is sub-classing the initializer and overriding newTldScanner?

I was thinking of supporting compile-on-start as a option where the initializer 
would scan the war for JSP(x) files and initiate the compilation during 
startup. This was intended for production use where use of JspC was not 
practical (e.g. because the actual version of runtime is not known at 
pre-compilation time) but where taking a hit on access would be undesirable. I 
was thinking there to combine the TLD and JSP scans into one.

There may be other reasons to scan e.g. looking for tag-plugins packaged in 
jars (no XML config needed a bit like a Tld).

We also have an issue where Jasper is very reliant on system properties and the 
Options built from the init-params of JspServlet.

In short, I’m wondering if instead of the subclassing we define a 
JasperConfiguration interface and have the initializer look for an instance of 
it in a context attribute. The configuration there would allow the container to 
replace parts of Jasper’s initialization (e.g. by supplying a CollectionURL 
of pre-scanned TLDs, by identifying TLDs provided by the container” etc.) and 
other configuration on a per-context basis.

Would something like that work for you?
Cheers
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Jasper Improvements

[Jeremy Boynes]

On Feb 23, 2014, at 5:55 PM, Greg Wilkins gr...@apache.org wrote:

 On Jun 26, 2013, at 05:44:23 GMT Jeremy Boynes jboy...@apache.org wrote:
 
 I have been thinking about improvements to Jasper as well around better 
 support for Servlet
 3.0 concepts. One area would be decoupling it from Tomcat, bootstrapping 
 using an SCI as hinted
 in ContextConfig. I'd also be interested in improving the Ant task as well, 
 such as support
 for pre-compiling a separate package that would be treated as a web fragment 
 (including web.xml-less
 pre-compilation, generating a web-fragment.xml rather than a web.xml snippet 
 or potentially
 eliminating the XML entirely if the generated code can be annotated with 
 @WebServlet).
 
 
 Jeremy et al,

Hi Greg,

 
 The Jetty project has been considering switching to directly consume the
 apache version of Jasper JSP.
 
 However, there are some tomcat dependencies in jasper that means we cannot
 directly consume the jar produced by the apache project.We had assumed
 that there would be little interest here to make jasper separable so we had
 begun our own effort at github.  However, when I gave a heads up to
 priv...@tomcat.apache.org, I was pointed at Jeremy's email.  So if
 there is some interest here, then it would definitely be better to do this
 within apache rather than via an intermediate repository.

My original interest was partly academic in that Servlet 3.0 implies the JSP 
engine should be pluggable and I was wondering if it actually could. I was also 
interested in where we could take the JSP engine in light of other developments 
in server-side template engines especially the JavaScript-based ones (e.g. 
Handlebars, Dust). I hadn’t originally envisioned Jetty picking this up but 
would be happy to help.

The work I’ve done to date has all been in trunk (Tomcat 8.x) and IMO is 
disruptive enough that it would not be wise to attempt a back port into Tomcat 
7.x. It focused on decoupling Catalina from Jasper i.e. removing any need for 
Jasper-specific initialization in Catalina. I think that is mostly complete, 
with Jasper’s SCI now handling bootstrap of the JSP engine. There is still some 
work to do setting up the JSP servlet (the *.jsp[x] mapping) as in Tomcat that 
is defined by the default web.xml. It would be easy enough for the SCI to 
register a servlet and mapping but Jasper would ideally have a configuration 
mechanism for it that does not require modification to container or application 
web.xml files.

 What we have done so far is to create a github project at github:
 https://github.com/jetty-project/jasper-jsp. This project mounts the
 tomcat github repo as a subtree and removes everything that is not el,
 jasper or a used utility.   We've then changed a few hard tomcat
 dependencies to produce a container neutral version of jasper, which we
 then consume and specialise within a build of the jetty project (not yet at
 eclipse).  We have this working at the moment, but have not yet done any
 releases, so there are no indelible maven artefacts yet.  Nor have we done
 the IP clearance work to officially consume such an artefact within the
 eclipse project.

I’ve not had a chance to take a look yet but will this evening.

 
 The changes that we made to make jasper neutral were:
 
   - Modified JuliLog LogFactory to use a ServiceLoader to find an
   implementation of Log.  Within the jetty project we add an impl of the Juli
   Log that logs to the jetty log and we set that as a service in our own jars
   META-INF.   Note that judging by some of the comments in the classes, there
   is not much desire to make logging discoverable?

   - We have replaced the InstanceManager with a pretty simple
   non-container version.   It does not do post construct or pre destroy, but
   don't think these are needed for Jasper.  I guess rather than replace the
   top level InstanceManager, it would be better to have a Jasper Instance
   manager that could be instantiated as the container specific version,
   perhaps also with a ServiceLoader?

I thought pre- and post- construct were now necessary parts of tag lifecycle?

This seemed to me to be one of the areas where the Servlet API was lacking. It 
provides bean factories for the Servlet components but does not make a 
general-purpose bean factory available to frameworks like JSP. In a JavaEE 
world with CDI available, I think Jasper could use the CDI BeanFactory to 
create JSP-servlet and tag instances but that is not something available in a 
standalone Servlet environment.

   - We changed the jasper ServletContainerInitialiser to allow some
   pluggability of the scanner and we turned off ServiceLoader for it so that
   we can provide our own extension of it.
   - We made the Tld scanning check for a prescanned list so that we can do
   the TLD discovery in our own scanning (rather than do another scan) and
   push those into jasper.   A  neutral version of Jasper could make scanning
   container specific

Re: Git

[Jeremy Boynes]

Setting aside Maven and Gradle, did we reach any consensus on Git?

On Jan 26, 2014, at 5:34 PM, Costin Manolache cos...@gmail.com wrote:

 Gradle is becoming the standard build tool for android.
 I didn't like it at first - I don't really see the point of groovy - but
 it's still better than 'programming'
 in ant, and you don't have to touch any XML. You can use any ant task from
 gradle - best to think of it as a non-xml ant file, with reasonable
 defaults and less typing for the most common operations.
 
 Git is best.
 
 Costin
 
 
 On Tue, Jan 21, 2014 at 7:49 AM, Rémy Maucherat r...@apache.org wrote:
 
 2014/1/21 Yoav Shapira yo...@apache.org
 
 Also +1 to separating the git vs svn discussion from the build system
 discussion (Maven, Ant, whatever.) (Side note: been using gradle
 recently, fun.)
 
 
 +1
 
 Never used that Gradle yet though.
 
 Rémy
 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Git

[Jeremy Boynes]

On Jan 21, 2014, at 1:58 AM, Mark Thomas ma...@apache.org wrote:

 I've been using Git more and more for Tomcat development and was
 wondering what folks thought about moving Tomcat to git. This isn't a
 formal proposal or a vote, I'm just trying to gather some views.
 
 On the plus side:
 - it is much easier to have multiple issues in progress at the
  same time and switch between them
 - being able to work off-line but still commit is a huge benefit
  when working on a complex issue and you don't have internet
  access
 - merging between branches (assuming all supported branches were in a
  single repo) is simpler
 
 Neutral
 - we would need to agree some simple guidelines for how we used git
 - tooling seems equivalent to that available for svn (at for what I
  use anyway)
 
 On the down side:
 - there is much more potential to mess things up
 - cleaning up is potentially more complex
 - the disruption of the move - particularly if we want to move to a
  single git repo - could be significant
 
 
 Thoughts?

Generally +0

I think the ability to work offline on multiple issues can be a double edged 
sword. I just did that this week and having Git would have been helpful. The 
downside is that it makes it easier to work more in isolation rather than 
collaborating with other developers. In a quick experiment, I tried Git with 
some cookie changes and now have a series of changes to merge in one go rather 
than incrementally and I think that will be harder to review.

Having said that, this may just be an artifact of using git with svn as I think 
if the primary repo was natively Git then I would have done that work in a 
feature branch in the shared repo.

I’ve not followed this on infra@ but from what I understand the web tooling at 
Apache lags behind SVN's. There’s a richer interface at GitHub but that’s 
moving yet more away from the Apache community. I’ve found the web view very 
useful when working with people who are not engaged in the project on a regular 
basis.

—
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Using HttpParser for Cookie header?

[Jeremy Boynes]

On Jan 20, 2014, at 1:57 PM, Mark Thomas ma...@apache.org wrote:

 Signed PGP part
 On 20/01/2014 21:38, Jeremy Boynes wrote:
  I started to look at using HttpParser for the Cookie header but
  there are some differences in the way it works compared to the
  existing parser in Cookies that I wanted to check direction before
  getting too far in.
 
  The area I’m concerned about is the need to copy the bytes in
  order to parse the header. The Cookies parser relies heavily on
  MessageBytes and avoids copying to a String as far as possible.
  HttpParser, however, operates on a StringReader which requires
  converting to a String before parsing.
 
  After digging into the usage of Cookies I think there are only two
  places that read them: 1) Request#getCookies(), which needs to
  copy to Strings anyway in order to create the Cookie instances it
  returns 2) CoyoteAdapter#parseSessionCookiesId(), which parses the
  header and compares names as MessageBytes, only allocating a String
  for the value if the session cookie is found
 
  It’s this second one that has me concerned about switching to
  HttpParser as this gets called for every request. If we switch
  then there is going to be allocation and copying of the header that
  we currently don’t do.
 
 I share your concern. Worst case, we'll need to do a specific
 MessageBytes based parse just for the session cookie. Assuming that
 the session cookie name and value will remain US-ASCII (see no reason
 why this should not be the case), we could get away with this as long
 as we are mindful that there might be some UTF-8 we need to skip over.

I started implementing this and a patch can be found here:
  
http://people.apache.org/~jboynes/patches/0013-Start-of-a-more-lenient-RFC6265-cookie-parser.patch

The best illustration of how it is parsing is probably the test cases. The main 
difference is that it does not apply RFC2109 rules unless $Version=1 is 
specified and because of that I did not end up using as much of HttpParser as 
I’d hoped.

I believe the implementation handles correctly formatted headers according to 
the rules from Netscape, RFC2109 and RFC6265. 

It is lenient with respect to malformed input, breaking the header down into 
pairs separated by semi-colons (allowing for RFC2109 quoting/escaping where 
applicable) and attempting to accept each pair in isolation. If an individual 
pair is invalid (e.g. contains disallowed characters), it defaults to accepting 
that rather than dropping the cookie. The only cookies that actually get 
dropped are ones that do not specify a name or ones where Cookie’s constructor 
throws an IAE on the name. What this means is that an application will see 
every cookie Cookie will allow, making it symmetrical, but it will not see 
cookies whose names the Servlet spec configuration (i.e. STRICT_NAMING) would 
disallow.

I kept the de-quoting behavior with one difference: for V0 cookies, RFC6265 
quoting rules are applied which means any escapes are not undone. IOW for V0 
«a\bc» will be returned as «a\bc» whereas for V1 it would be returned as 
«abc». I’m not convinced this is useful and it may be more predictable for 
users if we simply returned the actual value, quotes and all, by default with a 
config option to enable unquoting for V1 values only (perhaps the same option 
as would enable alternative G1a).

Feedback welcome, ideally in the form of a test case showing what should be 
returned for a header text input.

Cheers
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: svn commit: r1559707 - in /tomcat/trunk: java/org/apache/jasper/compiler/ELParser.java java/org/apache/jasper/resources/LocalStrings.properties test/org/apache/el/TestELInJsp.java test/webapp/bug5

[Jeremy Boynes]

On Jan 20, 2014, at 6:22 AM, ma...@apache.org wrote:

 Author: markt
 Date: Mon Jan 20 14:22:49 2014
 New Revision: 1559707
 
 URL: http://svn.apache.org/r1559707
 Log:
 Additional fix for bug 56029. Now whitespace is not skipped, make sure it is 
 trimmed before the function name is used.
 Includes a test case based on a jspx file by kkolinko.

I think this caused TestELParser to start failing. I committed a change 
http://svn.apache.org/r1559798 that I think fixes that but would appreciate a 
second set of eyes before back-porting to 7.0.x.

Thanks
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Using HttpParser for Cookie header?

[Jeremy Boynes]

I started to look at using HttpParser for the Cookie header but there are some 
differences in the way it works compared to the existing parser in Cookies that 
I wanted to check direction before getting too far in.

The area I’m concerned about is the need to copy the bytes in order to parse 
the header. The Cookies parser relies heavily on MessageBytes and avoids 
copying to a String as far as possible. HttpParser, however, operates on a 
StringReader which requires converting to a String before parsing.

After digging into the usage of Cookies I think there are only two places that 
read them:
1) Request#getCookies(), which needs to copy to Strings anyway in order to 
create the Cookie instances it returns
2) CoyoteAdapter#parseSessionCookiesId(), which parses the header and compares 
names as MessageBytes, only allocating a String for the value if the session 
cookie is found

It’s this second one that has me concerned about switching to HttpParser as 
this gets called for every request. If we switch then there is going to be 
allocation and copying of the header that we currently don’t do. 

Having said that, the current parse relies heavily on the assumption that the 
header is US-ASCII encoded and that it is only dealing with 7-bit characters 
(it freely casts bytes to chars). The cookie change proposal has us supporting 
UTF-8 as specified by HTML5 which means a more robust decoder will be needed 
and the copy may not be avoidable.

My plan here is to KISS and implement a parser similar to the others in 
HttpParser assuming the header has already been decoded so it can just deal 
with the chars. Then if we notice any performance degradation we can focus on 
improving HttpParser which will have the benefit of working for the other 
header parsers as well. I’ll implement this alongside the existing code 
(actually, in the parser package) to make it easier to do an A-B comparison.

There would likely be some follow-on changes from such a change. Cookies and 
ServerCookie are recyclable objects associated with the request. By moving away 
from MessageBytes these could be replaced by basic String values and may not be 
needed e.g. Request already caches the array of Cookie values returned from 
getCookies() and that could be now populated directly from the parse. These 
classes may end up going away.

Thanks
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [Taglibs] Site updated; ready for announce.

[Jeremy Boynes]

On Jan 18, 2014, at 8:12 PM, Henri Yandell flame...@gmail.com wrote:

 I've updated the site to point to the 1.2.1 release. It's a bit of a kludge
 right now. Once RDC heads in the direction of the Attic, I'd like to move
 the site fully into the tomcat-site directory, avoiding any of the
 generated Maven noise. Then the only oddity is copying the javadoc in.
 
 Jeremy - would you like to do the honours and send the announcement email
 to users@tomcat and to annou...@apache.org?

I found another broken link. +1 on integrating with the main site to avoid 
frankenstein and to get the look and feel to match; I think mvn is more work 
than benefit here.

I’ll send out the announcement.
Thanks
Jeremy




signature.asc
Description: Message signed with OpenPGP using GPGMail

[ANN] Apache Standard Taglib 1.2.1 released

[Jeremy Boynes]

The Apache Tomcat team announces the immediate availability of Apache Standard 
Taglib 1.2.1.

Apache Standard Taglib is an open source software implementation of the JSP 
Standard Tag Library (JSTL) technology.

This release supports JSTL version 1.2 along with numerous bug-fixes and 
improvements on the previous 1.1.2 release. The notable changes include:
- Transitioned project from Apache Jakarta to Apache Tomcat
- Implementation of new functionality for JSTL 1.2; requires Java 1.5
- Switched to Maven-based build
- Renamed JARs and modified packaging structure; OSGI metadata added
- XML processing performance improved by direct use of Xalan

Please refer to the change log included in the distribution for a complete list 
of the issues addressed.

Downloads:
http://tomcat.apache.org/download-taglibs.cgi




signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Status of cookies proposals

[Jeremy Boynes]

On Jan 18, 2014, at 2:50 AM, Mark Thomas ma...@apache.org wrote:

 Signed PGP part
 On 18/01/2014 04:34, Jeremy Boynes wrote:
  On Jan 15, 2014, at 2:38 AM, Mark Thomas ma...@apache.org wrote:
 
  Does any one have anything else to add to
  http://wiki.apache.org/tomcat/Cookies ? It feels like we should
  be looking to make some decisions on where to go with this.
 
  My $0.02 to start this discussion is that we should adopt the all
  of the proposed changes with the following notes: - C1 rather
  C1a
 
  I have uploaded a patch for the Cxx changes here:
  http://people.apache.org/~jboynes/patches/cookie.patch It’s also
  linked off the wiki page.
 
 Generally +1.
 
 The build.xml change does not belong in the patch.

Sorry about that, bad changelist foo.

 Formatting changes can be committed now, removing them from the patch
 and making it easier to read.

I will commit two changes now, one with formatting changes and another to 
convert from indexOf to BitSet without changing the existing functionality, and 
then post another candidate with the changes in behaviour.

 I had to stop and think when looking at TOKEN. It is right but it
 isn't 100% clear what is happening. I'd suggest using
 TOKEN.set(32,127) and a comment on the separators list that \t is
 already excluded but is included in the list for completeness.
 
 Use String.length() == 0 or String.isEmpty() consistently.
 
 I'm a little concerned about anonymous cookies. Is the browser support
 consistent? Is anyone using them?

I’ll double check but IIRC it was consistent across the ones I tried. I don’t 
have access at the moment to older browsers to check their behaviour. I’m 
presuming they are the same as I can’t believe anyone would have added this 
“feature recently.

If they are using them with Servlets they would need to jump through hoops as I 
don’t think other implementations allow null or empty names either. Our 
implementation maps these to name-only cookies but that results in an 
inconsistency with the browsers that will happily store two separate cookies 
for the two pairs «X=» and «X» and send both back.

We know people actually use name-only cookies so I think we should prioritize 
support for the «X=» form (i.e. it gets be the Cookie with name == “X” and 
value == “). We then have two options for the «X» form:
1) drop it
2) return it as the anonymous cookie

For rfc2109 and rfc6265 modes, it is not a valid token so we would normally 
drop it as Netscape mode needs to be explicitly enabled. We could just do the 
same for Netscape mode which would be consistent but means there’s a cookie 
browsers will store that we can’t handle. Fidelity with the browser was why I 
was leaning toward allowing it but either option resolves the ambiguity with 
the actual name-only cookie.

—
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Test failure for TestGroupChannelMemberArrival

[Jeremy Boynes]

On Jan 18, 2014, at 1:23 AM, Mark Thomas ma...@apache.org wrote:

 Signed PGP part
 On 18/01/2014 02:20, Jeremy Boynes wrote:
  I have started seeing a test failure on one of the tribes tests on
  trunk (see log below). OSX 10.9.1 with 1.7.0_51 or 1.7.0_45
  r1559303
 
  Is anyone else seeing similar, or could you provide me with clues
  on where to start digging?
 
 I've seen the occasional failure like this but have never been able to
 repeat it. Neither can I see anything in the code that would suggest
 why one of the listeners doesn't see any of the other members.
 
 As a starting point, I think I'd adjust the test so it reports what
 all of the listeners have heard before failing if any of them have not
 seen all the other members. That would give a better idea of what was
 going wrong and might help steer the next step.

Stepping through it turned out no messages were being received on the multicast 
socket.

I noticed I had VMWare Fusion started and it had created two local networks for 
its virts (vmnet1 and vmnet8) which were running even though no guest VMs were 
active. When I stopped Fusion completely so just en0 (wired ethernet) was 
active then the test passed.

I’ve not dug in to whether this is an issue with Fusion or with the setup of 
the test case just assuming there's a single network interface.

Cheers
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Test failure for TestGroupChannelMemberArrival

[Jeremy Boynes]

I have started seeing a test failure on one of the tribes tests on trunk (see 
log below).
OSX 10.9.1 with 1.7.0_51 or 1.7.0_45
r1559303

Is anyone else seeing similar, or could you provide me with clues on where to 
start digging?

Thanks
Jeremy

-

Testsuite: org.apache.catalina.tribes.group.TestGroupChannelMemberArrival
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 8.41 sec
- Standard Output ---
1390010749589 All channels started.
-  ---
- Standard Error -
Jan 17, 2014 6:05:42 PM org.apache.catalina.tribes.util.UUIDGenerator clinit
INFO: Creation of SecureRandom instance for UUID generation using [SHA1PRNG] 
took [298] milliseconds.
Jan 17, 2014 6:05:42 PM org.apache.catalina.tribes.transport.ReceiverBase bind
INFO: Receiver Server Socket bound to:/10.0.1.22:4000
Jan 17, 2014 6:05:42 PM org.apache.catalina.tribes.transport.ReceiverBase bind
INFO: Receiver Server Socket bound to:/10.0.1.22:4001
Jan 17, 2014 6:05:42 PM org.apache.catalina.tribes.transport.ReceiverBase bind
INFO: Receiver Server Socket bound to:/10.0.1.22:4002
Jan 17, 2014 6:05:42 PM org.apache.catalina.tribes.transport.ReceiverBase bind
INFO: Receiver Server Socket bound to:/10.0.1.22:4003
Jan 17, 2014 6:05:42 PM org.apache.catalina.tribes.transport.ReceiverBase bind
INFO: Receiver Server Socket bound to:/10.0.1.22:4004
Jan 17, 2014 6:05:42 PM org.apache.catalina.tribes.transport.ReceiverBase bind
INFO: Receiver Server Socket bound to:/10.0.1.22:4005
Jan 17, 2014 6:05:42 PM org.apache.catalina.tribes.transport.ReceiverBase bind
INFO: Receiver Server Socket bound to:/10.0.1.22:4006
Jan 17, 2014 6:05:42 PM org.apache.catalina.tribes.transport.ReceiverBase bind
INFO: Receiver Server Socket bound to:/10.0.1.22:4007
Jan 17, 2014 6:05:42 PM org.apache.catalina.tribes.transport.ReceiverBase bind
INFO: Receiver Server Socket bound to:/10.0.1.22:4008
Jan 17, 2014 6:05:42 PM org.apache.catalina.tribes.transport.ReceiverBase bind
INFO: Receiver Server Socket bound to:/10.0.1.22:4009
Jan 17, 2014 6:05:42 PM org.apache.catalina.tribes.membership.McastServiceImpl 
setupSocket
INFO: Setting cluster mcast soTimeout to 500
Jan 17, 2014 6:05:42 PM org.apache.catalina.tribes.membership.McastServiceImpl 
setupSocket
INFO: Setting cluster mcast soTimeout to 500
Jan 17, 2014 6:05:42 PM org.apache.catalina.tribes.membership.McastServiceImpl 
setupSocket
INFO: Setting cluster mcast soTimeout to 500
Jan 17, 2014 6:05:42 PM org.apache.catalina.tribes.membership.McastServiceImpl 
setupSocket
INFO: Setting cluster mcast soTimeout to 500
Jan 17, 2014 6:05:42 PM org.apache.catalina.tribes.membership.McastServiceImpl 
setupSocket
INFO: Setting cluster mcast soTimeout to 500
Jan 17, 2014 6:05:42 PM org.apache.catalina.tribes.membership.McastServiceImpl 
waitForMembers
INFO: Sleeping for 1000 milliseconds to establish cluster membership, start 
level:4
Jan 17, 2014 6:05:42 PM org.apache.catalina.tribes.membership.McastServiceImpl 
setupSocket
INFO: Setting cluster mcast soTimeout to 500
Jan 17, 2014 6:05:42 PM org.apache.catalina.tribes.membership.McastServiceImpl 
setupSocket
INFO: Setting cluster mcast soTimeout to 500
Jan 17, 2014 6:05:42 PM org.apache.catalina.tribes.membership.McastServiceImpl 
setupSocket
INFO: Setting cluster mcast soTimeout to 500
Jan 17, 2014 6:05:42 PM org.apache.catalina.tribes.membership.McastServiceImpl 
setupSocket
INFO: Setting cluster mcast soTimeout to 500
Jan 17, 2014 6:05:42 PM org.apache.catalina.tribes.membership.McastServiceImpl 
waitForMembers
INFO: Sleeping for 1000 milliseconds to establish cluster membership, start 
level:4
Jan 17, 2014 6:05:42 PM org.apache.catalina.tribes.membership.McastServiceImpl 
waitForMembers
INFO: Sleeping for 1000 milliseconds to establish cluster membership, start 
level:4
Jan 17, 2014 6:05:42 PM org.apache.catalina.tribes.membership.McastServiceImpl 
waitForMembers
INFO: Sleeping for 1000 milliseconds to establish cluster membership, start 
level:4
Jan 17, 2014 6:05:42 PM org.apache.catalina.tribes.membership.McastServiceImpl 
waitForMembers
INFO: Sleeping for 1000 milliseconds to establish cluster membership, start 
level:4
Jan 17, 2014 6:05:42 PM org.apache.catalina.tribes.membership.McastServiceImpl 
waitForMembers
INFO: Sleeping for 1000 milliseconds to establish cluster membership, start 
level:4
Jan 17, 2014 6:05:42 PM org.apache.catalina.tribes.membership.McastServiceImpl 
waitForMembers
INFO: Sleeping for 1000 milliseconds to establish cluster membership, start 
level:4
Jan 17, 2014 6:05:42 PM org.apache.catalina.tribes.membership.McastServiceImpl 
waitForMembers
INFO: Sleeping for 1000 milliseconds to establish cluster membership, start 
level:4
Jan 17, 2014 6:05:42 PM org.apache.catalina.tribes.membership.McastServiceImpl 
setupSocket
INFO: Setting cluster mcast soTimeout to 500
Jan 17, 2014 6:05:42 PM org.apache.catalina.tribes.membership.McastServiceImpl 
waitForMembers

Re: Status of cookies proposals

[Jeremy Boynes]

On Jan 15, 2014, at 2:38 AM, Mark Thomas ma...@apache.org wrote:

 Does any one have anything else to add to
 http://wiki.apache.org/tomcat/Cookies ? It feels like we should be
 looking to make some decisions on where to go with this.
 
 My $0.02 to start this discussion is that we should adopt the all of the
 proposed changes with the following notes:
 - C1 rather C1a

I have uploaded a patch for the Cxx changes here:
  http://people.apache.org/~jboynes/patches/cookie.patch
It’s also linked off the wiki page.

 - G1a rather than G1
  (we need to keep an option to switch for backwards compatibility)
 - G3a rather than G3
  (Tomcat currently handles quoting / escaping for RFC2109 cookies so
   it needs to continue to do so for backwards compatibility)

I will take a look at the Gxx and Pxx changes this weekend.

Cheers
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Status of cookies proposals

[Jeremy Boynes]

On Jan 15, 2014, at 2:38 AM, Mark Thomas ma...@apache.org wrote:

 Does any one have anything else to add to
 http://wiki.apache.org/tomcat/Cookies ? It feels like we should be
 looking to make some decisions on where to go with this.

There are two things on my mind I’ve not had a chance to write up yet:

1) Clearer distinction between name-only and value-only cookies. Existing code 
treats a name-value pair of «X» as being name-only but the browsers treat that 
as being the value of a cookie-with-no-name.

2) How to handle cookies whose name might be valid RF6265 but won’t be allowed 
by Cookie’s constructor e.g. «Expires», «$Foo» or the «» name from above. If an 
application sub-classes Cookie so getName() returns such a value, will we still 
generate the header as we currently do? If the browser sends such a header 
should we just drop it (meaning the app would need to parse the Cookie header 
itself and it’s not round-trippable), or should we allow it (which would mean 
we would need a sub-class, see C1a)?

 My $0.02 to start this discussion is that we should adopt the all of the
 proposed changes with the following notes:
 - C1 rather C1a
 - G1a rather than G1
  (we need to keep an option to switch for backwards compatibility)
 - G3a rather than G3
  (Tomcat currently handles quoting / escaping for RFC2109 cookies so
   it needs to continue to do so for backwards compatibility)

The concern I have with G1a and G3a is that the browsers are not round-tripping 
the cookies in RFC2109 format. So although we can add quotes to make it a valid 
RFC2109 value when we send it, when we get it back we will see a RFC6265 format 
header with no indicator on whether the quotes were there originally or if we 
added them.

The best I’ve come up with, which seems lame, is to have individual flags 
controlling this behaviour:
* escape-on-write, when enabled would escape any non-cookie-octet values, 
implies quote-on-write
* quote-on-write, when enabled would add dquotes to anything with a 
non-cookie-octet value
* unquote-on-read, when enabled would remove leading and trailing quotes
* unescape-on-read, when enabled would also replace any ‘\’ escaped characters, 
implies unquote-on-read

This would allow an application owner to run in one of three modes:
* new, where client and server views of the value are the same
* compatible, where client sees quoted and server sees unquoted
* transitional, where quotes are stripped on read and no longer added on write

In “transitional mode, the browser would send a quoted value previously set, 
we would strip the quotes when returning the value to the application, the 
application would re-set the cookie, we would send that without quotes, and any 
client-side code working with that response would always see the new unquoted 
value.

An application could transition by switching directly to “new” mode where we 
would pass it the quoted value and it would handle removing the quotes. That 
would give it more control but would require code changes.


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: name only cookies

[Jeremy Boynes]

On Jan 15, 2014, at 10:37 AM, Mark Thomas ma...@apache.org wrote:

 On 15/01/2014 16:58, Jeremy Boynes wrote:
 
  1) Clearer distinction between name-only and value-only cookies.
  Existing code treats a name-value pair of «X» as being name-only
  but the browsers treat that as being the value of a
  cookie-with-no-name.
 
 This could be problematic. We have had explicit requests for name only
 cookie support.

Sorry, I was rushing so perhaps wasn’t clear. Based on where things end up 
being stored client-side, browsers seem to treat:
  «X» as unnamed with just a value, same as «=X» although they do not actually 
send that
  «X=» as named with no value
  «X=“”» as named with a value comprising 2 dquotes

On write, although Cookie will not allow the first form, it can be sub-classed 
so getName() could return null or an empty String. As an application actually 
had to create and use a sub-class to do that, there’s an argument we should 
respect their intent by writing the anonymous cookie (i.e. «value»), at least 
if version == 0; I’m not sure I buy that argument, just calling it out as we 
allow it now (addCookie() does not recheck the name).

If getValue() returns null or an empty String, then the application is setting 
a name-only cookie and:
* for V0, we emit «name=»
* for V1, we either emit «name=“”» if auto-quoting is enabled or throw an IAE 
if not (as it is not a valid VALUE).

On read, if there is a ‘=‘ present then everything before it is the name and 
everything after is the value. The change would be that «value» would now be 
treated as a value-only cookie rather than a name-only one. However, Cookie’s 
constructor won’t allow an empty name so if we are going to return it to the 
application we would need to use a sub-class to bypass the checks. My proposal 
would be to drop the cookie by default, potentially with a property to enable a 
value-only cookie.


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: allowed cookie names

[Jeremy Boynes]

On Jan 15, 2014, at 10:37 AM, Mark Thomas ma...@apache.org wrote:

 On 15/01/2014 16:58, Jeremy Boynes wrote:
  2) How to handle cookies whose name might be valid RF6265 but won’t
  be allowed by Cookie’s constructor e.g. «Expires», «$Foo» or the «»
  name from above. If an application sub-classes Cookie so getName()
  returns such a value, will we still generate the header as we
  currently do? If the browser sends such a header should we just
  drop it (meaning the app would need to parse the Cookie header
  itself and it’s not round-trippable), or should we allow it (which
  would mean we would need a sub-class, see C1a)?
 
 Remind me why aren't those allowable. Looking at the Javadoc, the
 default is RFC2109 but we can optionally support Netscape.


RFC2109 syntax allows names starting with “$” but there’s a semantic constraint 
in 4.2.2 that says they are reserved so rejecting those seems right. The 
JavaDoc refers to matches a token reserved for use by the cookie protocol” 
which we’ve taken to include the attributes names but strictly they are a 
different part of the protocol and there is no ambiguity. I think we are being 
over restrictive here in excluding them; IMO it may be confusing but it is 
valid. 

Strictly Netscape does not preclude an empty name either so perhaps turning 
that mode on should also allow that by suppressing the null or empty” clause 
in the JavaDoc. This would also address the value-only cookie problem and allow 
them to be round-tripped.

We could make STRICT_NAMING multi-valued:
- if it’s “false” or “netscape” allow anything per Netscape’s spec including 
empty
- if it’s “true” or “rfc2109” allow anything per RFC2109 (i.e. token and not 
starting with ‘$’)
- if it’s unset or “rfc6265” allow anything per RFC6265 (i.e. token and may 
start with ‘$’)
with STRICT_SERVLET_COMPLIANCE implying rfc2109


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [Taglibs] Extended?

[Jeremy Boynes]

On Jan 13, 2014, at 9:51 PM, Henri Yandell flame...@gmail.com wrote:

 Any thoughts Jeremy on our containing tags outside of the Standard
 implementation?
 
 I was pondering folding the Extended one (which contains two very tiny
 tags) into the Standard taglib, or if you don't see any likelihood for
 adding new ones, just removing it.

If anything, I think I would rather go in the other direction, breaking 
Standard down into individual taglibs. I think there are a number of users who 
primarily rely only on core  fn in their applications, using other libraries 
for the functionality in fmt, sql, and xml. It would be nice to be able to 
consume them that way. Splitting them up would also allow specific libraries to 
be optimized through tag plugins or by Jasper itself.

Those other libraries have also not really kept up with the times. For example, 
fmt is heavily coupled to native Java L10N which I think still lags behind 
icu4j and hasn’t added basics like named placeholders, sql has been superseded 
by frameworks like JPA but even the basic JDBC support could take advantage of 
“new things like @Resource injection, and we’ve added a hard dependency on 
Xalan to address xml performance and the spec still hasn’t touched new features 
like XPath 2 or XQuery.

“Extended” is a vague name so I would be in favor of just dropping it and 
replacing it with more specific libraries e.g. localization, xpath, json or 
whatever we decide to work on.

Cheers
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Taglib Site Update

[Jeremy Boynes]

On Jan 12, 2014, at 10:56 PM, Henri Yandell flame...@gmail.com wrote:

 
 
 On Jan 11, 2014, at 10:58 PM, Henri Yandell flame...@gmail.com wrote:
 
 Remaining tasks:
 
 * Create a page to show source location.
 * Create a .cgi page for download mirrors *I hope that's gotten easier*
 * Change urls to point to mirror rather than archive
 * Push content into tomcat-site/taglibs manually
 * Change the news item on Tomcat itself to announce the release
 
 
 *http://tomcat.apache.org/download-taglibs.cgi
 http://tomcat.apache.org/download-taglibs.cgi* now exists.
 
 Jeremy - could you upload your public sig to a KEYS file at:
 
 https://www.apache.org/dist/tomcat/taglibs/KEYS
 
 It seems there's a trunk/KEYS for all Tomcat RMs, but also specific KEYS
 files per downloaded product. I'm not sure of the right way to put files in
 dist.apache.org nowadays, but I figure you would have hit it during release
 :)

I don’t think I can - access to the release tree is limited to PMC members. I 
have added my key here:
  https://svn.apache.org/repos/asf/tomcat/trunk/KEYS
if someone can copy it.

Thanks
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Taglib Site Update

[Jeremy Boynes]

On Jan 11, 2014, at 10:58 PM, Henri Yandell flame...@gmail.com wrote:

 Thanks. Wish I'd looked here before digging myself. I assumed it was
 something weird I did when moving over from Jakarta and spent far too much
 time digging into the old code there :)
 
 ---
 
 I've gone ahead and put that file back in, but in the overall Taglibs site.
 I'm not convinced we need a separate Maven generated site. A single file,
 and copy the apidocs over from the taglib itself.
 
 I'm also going to dump the link to CI (not working), test javadoc (dull)
 and the link to the wiki (not there). The main loss in ditching the
 separate Maven site are any items of value in the project reports at
 http://tomcat.apache.org/taglibs/standard/project-info.html. Most are noise
 or duplicates. Primarily we lack a page showing where to get the source.
 
 Remaining tasks:
 
 * Create a page to show source location.
 * Create a .cgi page for download mirrors *I hope that's gotten easier*
 * Change urls to point to mirror rather than archive
 * Push content into tomcat-site/taglibs manually
 * Change the news item on Tomcat itself to announce the release

Is this all in taglibs/site?
I’d like to add a page with docs on how to use the libraries as that has 
changed in this version (basically a web copy of the README for the binary).

Thanks
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Permissions to create release directory for taglibs?

[Jeremy Boynes]

On Jan 2, 2014, at 2:55 PM, Olivier Lamy ol...@apache.org wrote:

 Just did that:
 
 mb-olamy:sirona olamy$ svn mkdir
 https://dist.apache.org/repos/dist/release/tomcat/taglibs
 
 
 Committed revision 4030.
 
 mb-olamy:sirona olamy$ svn mv
 https://dist.apache.org/repos/dist/dev/tomcat/taglibs/taglibs-standard-1.2.1
 https://dist.apache.org/repos/dist/release/tomcat/taglibs
 
 
 Committed revision 4031.
 
 Should be ok now.
 

Thank you.



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: 8-bit text in cookie values

[Jeremy Boynes]

Adding more confusion to the pile, HTML5[1] now specifies that JavaScript can 
set Unicode characters through document.cookie and that they must be encoded as 
UTF-8 in the header. Quick testing with Chrome shows it does just that (i.e. 
U+00E1 is sent as 0xC3 0xA1). If client and server-side application code is 
going to interoperate then we would need to accept them in a Cookie header and 
allow them to be sent in a Set-Cookie header. However, this is ambiguous when 
compared to Netscape and its implicit assumption of ISO-8859-1.

[1] http://www.w3.org/html/wg/drafts/html/master/single-page.html#cookie

On Jan 1, 2014, at 10:18 AM, Jeremy Boynes jboy...@apache.org wrote:

 On Jan 1, 2014, at 8:59 AM, Mark Thomas ma...@apache.org wrote:
 
 Signed PGP part
 On 26/12/2013 19:23, Jeremy Boynes wrote:
 On Dec 26, 2013, at 2:47 AM, Mark Thomas ma...@apache.org wrote:
 
 Focusing on the 8-bit issue address by the patch, leaving the other
 RFC6265 thread for broader discussion ...
 
 The change only allows these characters in values if version ==
 0 where Netscape’s rather than RFC2109’s syntax applies (per
 the Servlet spec). The Netscape spec is vague in that it does
 not define “OPAQUE_STRING at all and defines “VALUE” as
 containing equally undefined “characters” although
 historically[1] those have been taken to be OCTETs as permitted
 by RFC2616’s “*TEXT” variant of “field-content.” The change
 will continue to reject these characters in names and in
 unquoted values when version != 0 (RFC2109’s “word rule)
 
 [1] based on comments by Fielding et al. on http-state and
 what I’ve seen in the wild
 
 Can you provide references for [1]?
 
 This is the mail in the run up to RFC6265 that triggered the
 discussion:
 http://www.ietf.org/mail-archive/web/http-state/current/msg01232.html
 
 Thanks
 
 for that reference. What a complete mess. RFC6265 really
 dropped the ball on this. The grammar for cookie-value is a disaster.
 So far the issues include:
 - no support for 0x80 to 0xFF
 - no support for \ sequences
 - no support for using whitespace, comma, semi-colon, backslash
 
 I was beginning to think that factoring out the cookie generation /
 parsing and then providing different implementations (one for Netscape
 + RFC2109 - roughly what we have now with a few fixes, one for RFC6265
 and maybe one very relaxed) would be the way to go. Having looked at
 the first issue that plan already looks like it needs a re-think.
 
 I'm still hoping that by documenting all the various issues in one
 place we will be able to come up with a solution that both addresses
 all the issues you have raised and is better than the handful of
 system properties we have currently.
 
 I think they did a reasonable job given the mess cookies are in the wild 
 today. They summarize this in the preamble:
 The recommendations for cookie generation provided in Section 4 represent a 
 preferred subset of current server behavior, and even the more liberal 
 cookie processing algorithm provided in Section 5 does not recommend all of 
 the syntactic and semantic variations in use today.
 
 Section 4 recommends guidelines for servers generating cookies. I interpret 
 that as being “if you follow these guidelines, you have a good chance of 
 actually getting back the value you tried to set.” The rules above (no 8-bit, 
 no escaping, no Netscape delimiters) reflect that principle. A server 
 application can step outside those guidelines but thar ther be dragons.
 
 —
 Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

[VOTE][RESULT] Release Apache Standard Taglib 1.2.1

[Jeremy Boynes]

binding +1: rjung, olamy, markt
non-binding +1: jboynes, hyandell

No other votes have been cast so I am calling this as passed. I will publish 
the release artifacts though the Maven and diet repositories and then update 
the website before sending a general announcement.

I would like to thank everyone for their help in making possible the first 
release of Apache’s taglibs in many years.

Thanks
Jeremy


signature.asc
Description: Message signed with OpenPGP using GPGMail

Permissions to create release directory for taglibs?

[Jeremy Boynes]

On Dec 29, 2013, at 3:10 PM, Mark Thomas ma...@apache.org wrote:
 
  I’ve uploaded a copy of the source release here:
  https://dist.apache.org/repos/dist/dev/tomcat/taglibs/taglibs-standard-1.2.1/
 
   Is that what you had in mind? Do I just svn mv that to “release”
  when done?
 
 Yes and yes.

I am trying to create the parent directory for this but do not have permission:
$ svn --username jboynes mkdir 
https://dist.apache.org/repos/dist/release/tomcat/taglibs -mcreate release 
directory for taglibs
svn: E175013: Access to '/repos/dist/!svn/ver/3994/release/tomcat’ forbidden

Am I going about this the wrong way or do I just not have enough karma?
Thanks
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Support RFC6265 cookie processing

[Jeremy Boynes]

-Cookie: =X
 Set-Cookie: Y===test
 Set-Cookie: Z
Chrome, Aurora and IE generate:
 Cookie: 
 X!#$%'()*+-./0123456789:?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}~X=test;
  Y===test; Z

and Safari:
 Cookie: 
 X!#$%'()*+-./0123456789:?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}~X=test;
  Y===test

The long name contains every cookie-octet value except equals; note these are 
valid per Netscape. 

Safari is missing the cookie from the “=X” and “Z” lines, the others treat it 
as a cookie with an empty name. IOW, if just “=X” is set, then the Cookie 
header would contain “X” whereas if just “Z” is set then the header would 
contain “Z” but because both are set the “Z” overwrites “=X” as the value of 
the cookie-with-no-name. However, this is not quite the correct model as there 
is no ‘=‘ character added to the Cookie header for this cookie (i.e. “=X” is 
returned as “X” not “=X”).

On Dec 27, 2013, at 10:47 AM, Jeremy Boynes jboy...@apache.org wrote:

 I think this is actually the principle that is the crux of the matter:
 With respect to the
 requirements of section 4.2, the Tomcat developers have always taken
 the view that cookie headers should be treated as combinations of
 token, separators, and quoted-string rather than *TEXT for the
 purposes of compliance with RFC2616. This view is rather fundamental
 to how cookies have been handled to date. 
 
 
 Whilst I agree with the principle I think there are sufficient edge-cases in 
 the ways user-agents and other servers handle cookies that trying to handle 
 values by using RFC2109’s rules to avoid the need for handling “*TEXT” have 
 resulted in some of the issues seen in the past. Simply put, a Netscape 
 format Cookie header is not a combination of token, separators, and 
 quoted-string.
 
 Part of this is that I have not seen widespread adoption of RFC2109 by 
 user-agents. Instead, in the way they handle values they appear to accept 
 RFC2109 format Set-Cookie headers from servers, process them in a 
 Netscape-format manner, and then generate a Cookie header conforming to 
 Netscape’s rules. Their RFC2109 support is primarily limited to attribute 
 handing (e.g. allowing Max-Age or DQUOTE characters in Path (except IE)). 
 They specifically don’t handle “Version” and don’t include the “$Version” 
 attribute required by RFC2109 in the Cookie header.
 
 Reviewing the “fun,” much of it can be attributed to presuming RFC2109 can be 
 applied. For example, the fun around “=“ or “/“ characters, or the use of 
 separator characters in unquoted values caused by trying to apply “token” 
 rules to free text. Upgrading to RFC2109 to use quoted values is also an 
 issue as user-agents are not parsing them as such; for example, Chrome-31 
 does not handle a quoted-string value per RFC2109:
  Set-Cookie:Customer=WILE_;_COYOTE; Version=1; Path=/
 instead treating it, loosely, as a Netscape-style value:
  Cookie:Customer=WILE_
 Note, that value is not valid per RFC6265’s set-cookie-header rule but is per 
 Netscape’s.
 
 I think we can avoid this “fun” if we only apply RFC2109 rules to cookies 
 that explicitly use version 1 (i.e. if setVersion(1) is called when setting a 
 cookie or if $Version=“1” is present when parsing). For all others we would 
 use RFC6265’s rules treating them as a less ambiguous version of Netscape’s 
 syntax.
 
 Whether that is actually true or not will depend on the behaviour we actually 
 see from user-agents. With that in mind, I plan to start extending the tests 
 to capture that behaviour so we have a clearer model of what is actually 
 happening. That would then define what we need to do to ensure symmetry for 
 the application developer and avoid the type of problem I mentioned above 
 with Chrome. If that can be an incremental change then that would be ideal 
 but that may not be possible if the change needed is to handle V0 and V1 
 cookies separately.
 
 Cheers
 Jeremy
 
 On Dec 24, 2013, at 2:24 AM, Mark Thomas ma...@apache.org wrote:
 
 Signed PGP part
 On 24/12/2013 01:21, Jeremy Boynes wrote:
 In comments on issue #55917, there was suggestion for refactoring
 cookie support along the lines described in RFC6265.
 
 No, that isn't what I said. What I said was that now might be a good
 time to refactor the cookie parsing to use the HttpParser and that if
 we did that, that we should keep RFC6265 in mind. My intention was to
 suggest that if there were places where RFC6265 suggested relaxing the
 parsing rules and those rules could be relaxed without creating
 ambiguities or security concerns then we should consider such a change.
 
 I also hinted at the 'fun' we have had with cookies in the past. I
 strongly recommend that you go and read all the various discussions
 and bug reports relating to cookie parsing.
 
 In particular, the requirements of RFC2616 for HTTP headers were often
 overlooked.
 
 Reading this RFC, it appears to be more of an effort to standardize
 the actual behaviour seen

Re: 8-bit text in cookie values

[Jeremy Boynes]

On Jan 1, 2014, at 8:59 AM, Mark Thomas ma...@apache.org wrote:

 Signed PGP part
 On 26/12/2013 19:23, Jeremy Boynes wrote:
  On Dec 26, 2013, at 2:47 AM, Mark Thomas ma...@apache.org wrote:
 
  Focusing on the 8-bit issue address by the patch, leaving the other
  RFC6265 thread for broader discussion ...
 
  The change only allows these characters in values if version ==
  0 where Netscape’s rather than RFC2109’s syntax applies (per
  the Servlet spec). The Netscape spec is vague in that it does
  not define “OPAQUE_STRING at all and defines “VALUE” as
  containing equally undefined “characters” although
  historically[1] those have been taken to be OCTETs as permitted
  by RFC2616’s “*TEXT” variant of “field-content.” The change
  will continue to reject these characters in names and in
  unquoted values when version != 0 (RFC2109’s “word rule)
 
  [1] based on comments by Fielding et al. on http-state and
  what I’ve seen in the wild
 
  Can you provide references for [1]?
 
  This is the mail in the run up to RFC6265 that triggered the
  discussion:
  http://www.ietf.org/mail-archive/web/http-state/current/msg01232.html
 
 Thanks
 
 for that reference. What a complete mess. RFC6265 really
 dropped the ball on this. The grammar for cookie-value is a disaster.
 So far the issues include:
 - no support for 0x80 to 0xFF
 - no support for \ sequences
 - no support for using whitespace, comma, semi-colon, backslash
 
 I was beginning to think that factoring out the cookie generation /
 parsing and then providing different implementations (one for Netscape
 + RFC2109 - roughly what we have now with a few fixes, one for RFC6265
 and maybe one very relaxed) would be the way to go. Having looked at
 the first issue that plan already looks like it needs a re-think.
 
 I'm still hoping that by documenting all the various issues in one
 place we will be able to come up with a solution that both addresses
 all the issues you have raised and is better than the handful of
 system properties we have currently.

I think they did a reasonable job given the mess cookies are in the wild today. 
They summarize this in the preamble:
 The recommendations for cookie generation provided in Section 4 represent a 
 preferred subset of current server behavior, and even the more liberal cookie 
 processing algorithm provided in Section 5 does not recommend all of the 
 syntactic and semantic variations in use today.

Section 4 recommends guidelines for servers generating cookies. I interpret 
that as being “if you follow these guidelines, you have a good chance of 
actually getting back the value you tried to set.” The rules above (no 8-bit, 
no escaping, no Netscape delimiters) reflect that principle. A server 
application can step outside those guidelines but thar ther be dragons.

—
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [Tomcat Wiki] Update of Cookies by markt

[Jeremy Boynes]

On Dec 31, 2013, at 12:46 PM, Mark Thomas ma...@apache.org wrote:

 Signed PGP part
 On 31/12/2013 20:29, Mark Thomas wrote:
  On 31/12/2013 17:03, Jeremy Boynes wrote:
  On Dec 31, 2013, at 3:55 AM, Mark Thomas ma...@apache.org
  wrote:
 
  On 31/12/2013 11:39, Apache Wiki wrote:
  Dear Wiki user,
 
  You have subscribed to a wiki page or wiki category on
  Tomcat Wiki for change notification.
 
  The Cookies page has been changed by markt:
  https://wiki.apache.org/tomcat/Cookies
 
  New page: #acl AdminGroup:read,write All:read ##language:en
 
  = Cookies =
 
  I’m not able to edit that page - is the acl right?
 
  No, it isn't. It was copied from another page. I'll go through the
  wiki and check all of the pages.
 
 Try now. I just got locked out for requesting too many pages too fast
 but I think the cookie page should be editable by anyone in the
 contributors group now. If you aren't in that group reply with your
 wiki ID and someone will add you.

I still don’t have an “Edit” action - my wiki id is jboynes.
Thanks
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [Tomcat Wiki] Update of Cookies by markt

[Jeremy Boynes]

On Dec 31, 2013, at 3:55 AM, Mark Thomas ma...@apache.org wrote:

 On 31/12/2013 11:39, Apache Wiki wrote:
 Dear Wiki user,
 
 You have subscribed to a wiki page or wiki category on Tomcat Wiki for 
 change notification.
 
 The Cookies page has been changed by markt:
 https://wiki.apache.org/tomcat/Cookies
 
 New page:
 #acl AdminGroup:read,write All:read
 ##language:en
 
 = Cookies =

I’m not able to edit that page - is the acl right?
Thanks
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [VOTE] Release Apache Standard Taglib 1.2.1

[Jeremy Boynes]

Konstantin added the binary artifacts to the dist folder as well. Thank you.

I’ll leave the vote open for 72hrs in case anyone would like a final review. As 
a reminder, the locations are:
Dist:  
https://dist.apache.org/repos/dist/dev/tomcat/taglibs/taglibs-standard-1.2.1/
Maven: https://repository.apache.org/content/repositories/orgapachetomcat-132/
SVN:   
https://svn.apache.org/repos/asf/tomcat/taglibs/standard/tags/taglibs-standard-1.2.1

Current votes are:
  binding +1: rjung, olamy, markt
  non-binding +1: jboynes, hyandell

Thanks
Jeremy

On Dec 29, 2013, at 3:10 PM, Mark Thomas ma...@apache.org wrote:

 Signed PGP part
 On 29/12/2013 15:50, Jeremy Boynes wrote:
  On Dec 29, 2013, at 6:04 AM, Mark Thomas ma...@apache.org wrote:
 
  On 14/11/2013 02:58, Jeremy Boynes wrote:
  I'd like to release Apache Standard Taglib 1.2.1. This is an
  update to the withdrawn 1.2.0, built with JDK 1.7.0_45 to
  address JavaDoc issues and incorporating feedback on the
  documentation.
 
  Maven Staging Repository:
  https://repository.apache.org/content/repositories/orgapachetomcat-132/
 
 
 
 Source Distribution:
  https://repository.apache.org/content/repositories/orgapachetomcat-132/org/apache/taglibs/taglibs-standard/1.2.1/
 
 
 
 SVN tag:
  https://svn.apache.org/repos/asf/tomcat/taglibs/standard/tags/taglibs-standard-1.2.1
 
 
 @ r1541786 https://svn.apache.org/r1541786
 
  KEYS: https://svn.apache.org/repos/asf/tomcat/trunk/KEYS
 
  The proposed 1.2.1 release is: [ ] Broken - do not release [X]
  OK - release as 1.2.1
 
  My vote is conditional on the src tarball being made available
  via the Apache Mirror system. I'd would have expected a link to
  https://dist.apache.org/repos/dist/dev/tomcat/... included in the
  vote thread.
 
  With the build being Maven based, I followed the doc here:
  http://www.apache.org/dev/publishing-maven-artifacts.html and I did
  not see anything about publishing through both channels.
 
 Some folks are of the view having the source in the ASF Nexus repo is
 sufficient to meet the ASF release requirements. I do not share that
 view. I might be in the minority.
 
 My view is that every source release should be present on the mirrors
 and available from archive.apache.org. That will happen automatically
 if the source distributions are made available via dist.a.o
 
  I’ve uploaded a copy of the source release here:
  https://dist.apache.org/repos/dist/dev/tomcat/taglibs/taglibs-standard-1.2.1/
 
   Is that what you had in mind? Do I just svn mv that to “release”
  when done?
 
 Yes and yes.
 
 Mark
 
 
 -
 To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
 For additional commands, e-mail: dev-h...@tomcat.apache.org
 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [VOTE] Release Apache Standard Taglib 1.2.1

[Jeremy Boynes]

On Dec 29, 2013, at 6:04 AM, Mark Thomas ma...@apache.org wrote:

 On 14/11/2013 02:58, Jeremy Boynes wrote:
 I'd like to release Apache Standard Taglib 1.2.1. This is an update
 to the withdrawn 1.2.0, built with JDK 1.7.0_45 to address JavaDoc
 issues and incorporating feedback on the documentation.
 
 Maven Staging Repository: 
 https://repository.apache.org/content/repositories/orgapachetomcat-132/
 
 Source Distribution: 
 https://repository.apache.org/content/repositories/orgapachetomcat-132/org/apache/taglibs/taglibs-standard/1.2.1/
 
 SVN tag: 
 https://svn.apache.org/repos/asf/tomcat/taglibs/standard/tags/taglibs-standard-1.2.1
 @ r1541786 https://svn.apache.org/r1541786
 
 KEYS: https://svn.apache.org/repos/asf/tomcat/trunk/KEYS
 
 The proposed 1.2.1 release is: [ ] Broken - do not release [X] OK -
 release as 1.2.1
 
 My vote is conditional on the src tarball being made available via the
 Apache Mirror system. I'd would have expected a link to
 https://dist.apache.org/repos/dist/dev/tomcat/... included in the vote
 thread.

With the build being Maven based, I followed the doc here: 
  http://www.apache.org/dev/publishing-maven-artifacts.html
and I did not see anything about publishing through both channels.

I’ve uploaded a copy of the source release here:
  https://dist.apache.org/repos/dist/dev/tomcat/taglibs/taglibs-standard-1.2.1/

Is that what you had in mind? Do I just svn mv that to “release” when done?

Thanks
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Support RFC6265 cookie processing

[Jeremy Boynes]

I think this is actually the principle that is the crux of the matter:
 With respect to the
 requirements of section 4.2, the Tomcat developers have always taken
 the view that cookie headers should be treated as combinations of
 token, separators, and quoted-string rather than *TEXT for the
 purposes of compliance with RFC2616. This view is rather fundamental
 to how cookies have been handled to date. 


Whilst I agree with the principle I think there are sufficient edge-cases in 
the ways user-agents and other servers handle cookies that trying to handle 
values by using RFC2109’s rules to avoid the need for handling “*TEXT” have 
resulted in some of the issues seen in the past. Simply put, a Netscape format 
Cookie header is not a combination of token, separators, and quoted-string.

Part of this is that I have not seen widespread adoption of RFC2109 by 
user-agents. Instead, in the way they handle values they appear to accept 
RFC2109 format Set-Cookie headers from servers, process them in a 
Netscape-format manner, and then generate a Cookie header conforming to 
Netscape’s rules. Their RFC2109 support is primarily limited to attribute 
handing (e.g. allowing Max-Age or DQUOTE characters in Path (except IE)). They 
specifically don’t handle “Version” and don’t include the “$Version” attribute 
required by RFC2109 in the Cookie header.

Reviewing the “fun,” much of it can be attributed to presuming RFC2109 can be 
applied. For example, the fun around “=“ or “/“ characters, or the use of 
separator characters in unquoted values caused by trying to apply “token” rules 
to free text. Upgrading to RFC2109 to use quoted values is also an issue as 
user-agents are not parsing them as such; for example, Chrome-31 does not 
handle a quoted-string value per RFC2109:
  Set-Cookie:Customer=WILE_;_COYOTE; Version=1; Path=/
instead treating it, loosely, as a Netscape-style value:
  Cookie:Customer=WILE_
Note, that value is not valid per RFC6265’s set-cookie-header rule but is per 
Netscape’s.

I think we can avoid this “fun” if we only apply RFC2109 rules to cookies that 
explicitly use version 1 (i.e. if setVersion(1) is called when setting a cookie 
or if $Version=“1” is present when parsing). For all others we would use 
RFC6265’s rules treating them as a less ambiguous version of Netscape’s syntax.

Whether that is actually true or not will depend on the behaviour we actually 
see from user-agents. With that in mind, I plan to start extending the tests to 
capture that behaviour so we have a clearer model of what is actually 
happening. That would then define what we need to do to ensure symmetry for the 
application developer and avoid the type of problem I mentioned above with 
Chrome. If that can be an incremental change then that would be ideal but that 
may not be possible if the change needed is to handle V0 and V1 cookies 
separately.

Cheers
Jeremy

On Dec 24, 2013, at 2:24 AM, Mark Thomas ma...@apache.org wrote:

 Signed PGP part
 On 24/12/2013 01:21, Jeremy Boynes wrote:
  In comments on issue #55917, there was suggestion for refactoring
  cookie support along the lines described in RFC6265.
 
 No, that isn't what I said. What I said was that now might be a good
 time to refactor the cookie parsing to use the HttpParser and that if
 we did that, that we should keep RFC6265 in mind. My intention was to
 suggest that if there were places where RFC6265 suggested relaxing the
 parsing rules and those rules could be relaxed without creating
 ambiguities or security concerns then we should consider such a change.
 
 I also hinted at the 'fun' we have had with cookies in the past. I
 strongly recommend that you go and read all the various discussions
 and bug reports relating to cookie parsing.
 
 In particular, the requirements of RFC2616 for HTTP headers were often
 overlooked.
 
  Reading this RFC, it appears to be more of an effort to standardize
  the actual behaviour seen on the Internet for different browser and
  server implementations. The observation is the RFC2109 has received
  limited adoption and RFC2965 virtually none at all, with most
  implementations falling back to the original specification released
  by Netscape that contains certain ambiguities.
 
 While I agree that RFC2965 has received little adoption, I disagree
 with the remainder. With the exception of internet explorer (and this
 might have changed in the few years since I last looked) the browsers
 all had pretty good support for RFC2109.
 
  The Servlet spec’s JavaDoc for Cookie refers to RFC2109 behaviour
  with caveats around interoperability. It defines version 0 as
  complying with Netscape’s original specification and version 1 as
  complying RFC2109 (with the note “Since RFC 2109 is still somewhat
  new, consider version 1 as experimental; do not use it yet on
  production sites”).
 
 That text is unchanged since at least Servlet 2.3 (12 years) so I
 wouldn't give it too much weight. The important part

Re: 8-bit text in cookie values

[Jeremy Boynes]

On Dec 26, 2013, at 2:47 AM, Mark Thomas ma...@apache.org wrote:

Focusing on the 8-bit issue address by the patch, leaving the other RFC6265 
thread for broader discussion ...

 The change only allows these characters in values if version == 0
 where Netscape’s rather than RFC2109’s syntax applies (per the
 Servlet spec). The Netscape spec is vague in that it does not
 define “OPAQUE_STRING at all and defines “VALUE” as containing
 equally undefined “characters” although historically[1] those have
 been taken to be OCTETs as permitted by RFC2616’s “*TEXT” variant
 of “field-content.” The change will continue to reject these
 characters in names and in unquoted values when version != 0
 (RFC2109’s “word rule)
 
 [1] based on comments by Fielding et al. on http-state and what
 I’ve seen in the wild
 
 Can you provide references for [1]?

This is the mail in the run up to RFC6265 that triggered the discussion:
http://www.ietf.org/mail-archive/web/http-state/current/msg01232.html

The relevant bit was:
 Changing the ABNF
 to include base64 does not do that -- it is just another
 fantasy production that differs from all prior specs of
 the cookie algorithm.  Changing it to
 
  cookie-value  = %x21-2B / %x2D-3A / %x3C-7E / %x80-FF
 
 or just the minimum
 
  cookie-value  = %x21-2B / %x2D-3A / %x3C-7E
 
 returns the definition to the original Netscape spec (at
 least in the first case), reflects how they are implemented
 on the Internet, and eliminates this artificial distinction
 between the server and user agent requirements.

with the observation that the rule including %x80-ff was the one matching the 
Netscape spec. The RFC6265 editor actually chose the latter production which 
led to the following exchange
http://www.ietf.org/mail-archive/web/http-state/current/msg01234.html
http://www.ietf.org/mail-archive/web/http-state/current/msg01236.html
asserting that the support for 8-bit characters implied by *TEXT was implicit 
in the original Netscape spec.

In this message:
http://www.ietf.org/mail-archive/web/http-state/current/msg01207.html
Roy asserts that the
  cookie-value  = %x21-2B / %x2D-3A / %x3C-7E / %x80-FF
production would be needed to support cookies currently in the wild, including 
the issue with the __utmz cookie that I’ve seen.

Further discussion resulted in the final production:
 cookie-value  = *cookie-octet / ( DQUOTE *cookie-octet DQUOTE )
 cookie-octet  = %x21 / %x23-2B / %x2D-3A / %x3C-5B / %x5D-7E
   ; US-ASCII characters excluding CTLs,
   ; whitespace DQUOTE, comma, semicolon,
   ; and backslash

on the basis that setting headers with the top bit set was deemed a bad idea by 
httpbis (I don’t have a reference for that). It was noted though that 
conformance to this was qualified by Servers SHOULD NOT send Set-Cookie 
headers that fail to conform to the following grammar” which discourages 8-bit 
values but still allows them to be sent and means that parsers receiving a 
cookie value need to be prepared to handle them.

Given cookies with these values may be set by other servers in the domain and 
are sent by user agents, failing hard as we do now prevents the application 
handling the request at all. The patch tolerates those characters and lets them 
through to the application. I don’t know of any security issue there given they 
are being decoded as ISO-8859-1 rather than UTF-8. I believe it’s backwards 
compatible in that the consequence to the application is that it will now see 
the request with a cookie that it either expects or that it would be ignoring 
anyway (on the basis that the cookie would be present if it didn’t have an 
8-bit character).

The patch does not change the generation behaviour so any attempt to set a V0 
cookie value containing one of these characters will still cause an IAE from 
HttpServletResponse#addCookie().

Cheers
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

8-bit text in cookie values, was: svn commit: r1553187

[Jeremy Boynes]

On Dec 24, 2013, at 2:29 AM, Mark Thomas ma...@apache.org wrote:

 On 23/12/2013 19:15, jboy...@apache.org wrote:
 Author: jboynes
 Date: Mon Dec 23 19:15:35 2013
 New Revision: 1553187
 
 URL: http://svn.apache.org/r1553187
 Log:
 fix #55917 by allowing 8-bit ISO-8859-1 characters in V0 cookie values
 
 -1 (veto)
 
 The Tomcat community has (to date) always taken the view that cookie
 headers should be restricted (as per RFC2616 section 4.2) to
 combinations of token, separators, and quoted-string.
 
 That does not permit 0x80 to 0xFF in tokens.

I thought the V2 patch had addressed your concern about limiting cookie names 
to tokens so went ahead and applied it. 

RFC2616 4.2 defines “message-header” as
message-header = field-name : [ field-value ]
   field-name = token
   field-value= *( field-content | LWS )
   field-content  = the OCTETs making up the field-value
and consisting of either *TEXT or combinations
of token, separators, and quoted-string

where TEXT is defined in 2.2 as
   TEXT   = any OCTET except CTLs,
but including LWS

The change only allows these characters in values if version == 0 where 
Netscape’s rather than RFC2109’s syntax applies (per the Servlet spec). The 
Netscape spec is vague in that it does not define “OPAQUE_STRING at all and 
defines “VALUE” as containing equally undefined “characters” although 
historically[1] those have been taken to be OCTETs as permitted by RFC2616’s 
“*TEXT” variant of “field-content.” The change will continue to reject these 
characters in names and in unquoted values when version != 0 (RFC2109’s “word 
rule)

I don’t want to proliferate them but would it help to add another system 
property gating this behaviour? Perhaps with 3 values: “reject” (the default, 
causing an IAE as now), “skip” (logged but not returned to the application), 
and “allow” (returned to the application).

Thanks
Jeremy

[1] based on comments by Fielding et al. on http-state and what I’ve seen in 
the wild


signature.asc
Description: Message signed with OpenPGP using GPGMail

Support RFC6265 cookie processing

[Jeremy Boynes]

In comments on issue #55917, there was suggestion for refactoring cookie 
support along the lines described in RFC6265. Reading this RFC, it appears to 
be more of an effort to standardize the actual behaviour seen on the Internet 
for different browser and server implementations. The observation is the 
RFC2109 has received limited adoption and RFC2965 virtually none at all, with 
most implementations falling back to the original specification released by 
Netscape that contains certain ambiguities. 

The Servlet spec’s JavaDoc for Cookie refers to RFC2109 behaviour with caveats 
around interoperability. It defines version 0 as complying with Netscape’s 
original specification and version 1 as complying RFC2109 (with the note “Since 
RFC 2109 is still somewhat new, consider version 1 as experimental; do not use 
it yet on production sites”).

The current implementation uses a number of system properties to control how 
cookies are validated. In implementing RFC6265 I hope that some of these can be 
eliminated. If not, I would propose to add configuration options on the 
Connector or Host objects to allow the configuration to be set separately for 
different host domains.

RFC6265 has separate sections in respect for generating and parsing cookie 
headers. It follows the practice that generation be strict but parsing be more 
tolerant of invalid input. Our current implementation generally follows that 
trend by suppressing invalid input data (after logging). However, for some 
input data, primary CTLs, it throws an IllegalArgumentException from the 
connector which does not allow the application to recover. In refactoring, I 
would propose to simply ignore that input thereby allowing the application to 
handle it, for example by parsing the header field manually. Cookie parsing in 
particular needs to be tolerant of cookies set by other sources, including 
different servers handling other parts of the domain and JavaScript or other 
client-side code setting values in the browser.

In light of this, I propose separating the “Set-Cookie” generation side from 
the “Cookie” parsing side.

Generation
==
The general principle here would be to use the version property of Cookie to 
determine the level of verification to perform: if 0 follow RFC6265, if 1 use 
RFC2109. The primary verification point would be in 
HttpServletRequest#addCookie() which would use the version in the Cookie 
instance. Characters will always be converted to octets using the ISO-8859-1 
charset; unmappable values will result in an IAE.

The Servlet spec requires an IAE be thrown in Cookie’s constructor if the name 
is not valid pre RFC2109. Both RFC6265 and RFC2109 define the name to be a 
“token” (per RFC2616 HTTP/1.1) so I would propose to always validate by those 
rules; this would allow US-ASCII characters except CTLs and separators. This 
will different from the current implementation that slash “/“ would be treated 
as a separator which would not be allowed in a name by default; this is 
consistent with the RFC’s and Glassfish’s implementation and I’m assuming that 
allowing it in our current implementation is a hangover from where we enabled 
use of “/“ in values. 

The spec allows vendors to provide a configuration option that allows cookie 
names conforming to the original Netscape Cookie Specification to be accepted” 
and I propose to retain the system property 
“org.apache.tomcat.util.http.ServerCookie.STRICT_NAMING” for that. If 
explicitly set to false, it will verify names using Netscape’s rules and allow 
a sequence of characters excluding semi-colon, comma and white space” but also 
excluding “=“ and CTLs per RFC2616; note this *would* allow 8-bit ISO-8859-1 
characters in the name and relax the RFC2109 constraint that NAMEs that begin 
with $ are reserved for other uses and must not be used by applications.” 

The value would not be checked until addCookie() was called and the cookie 
version is known. This would in principle use RFC6265’s “cookie-value” rule if 
version == 0 or RFC2109’s “value” rules if version == 1; values that do not 
conform would result in an IAE from addCookie(). Unlike the current 
implementation, this would not automatically upgrade the version or add quotes 
around RFC2109 “values” that did not match the “token” rule.

If STRICT_SERVLET_COMPLIANCE is set, the rule for version 0 values would be 
relaxed to allow any value conforming to the Netscape specification except 
CTLs; this would effectively add DQUOTE, backslash, and 0x80-0xFF. For more 
granular control, I propose adding the system property 
“org.apache.tomcat.util.http.ServerCookie.ALLOW_IN_VALUE” which would take one 
of the following enum values to determine what octets were allowed:
* Netscape
* RFC2616_token
* RFC2109_value
* RFC6265_cookie_octet
* Netscape_restricted (limits the permitted characters as recommended in the 
Servlet spec)
* RFC6265_ISO-8859-1 (adds 0x80-0xff to cookie_octet)

RFC6265 does allow value to be 

Re: [VOTE] Release Apache Standard Taglib 1.2.1

[Jeremy Boynes]

On Dec 1, 2013, at 3:00 PM, Olivier Lamy ol...@apache.org wrote:

 Build the source release: ok
 Rat check give those files
 
  CHANGES.txt
 
  DEPENDENCIES
 
  README_bin.txt
 
  README_src.txt
 
  
 standard-test/src/test/resources/org/apache/taglibs/standard/tag/el/core/TestOutTag.txt
 
 Not a big problem (at least for me)

Those should have been excluded by the plugin configuration in the root module.

 +1 (binding)
 
 (BTW I don't understand why you build those manually and not used the
 maven release plugin?)

I did build using the release plugin, are you seeing a different result?

Thanks
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [VOTE] Release Apache Standard Taglib 1.2.1

[Jeremy Boynes]

On Nov 20, 2013, at 12:35 AM, Konstantin Kolinko knst.koli...@gmail.com wrote:

 2013/11/14 Jeremy Boynes jboy...@apache.org:
 I'd like to release Apache Standard Taglib 1.2.1. This is an update to the 
 withdrawn 1.2.0, built with JDK 1.7.0_45 to address JavaDoc issues and 
 incorporating feedback on the documentation.
 
 Maven Staging Repository:
 https://repository.apache.org/content/repositories/orgapachetomcat-132/
 
 Source Distribution:
 https://repository.apache.org/content/repositories/orgapachetomcat-132/org/apache/taglibs/taglibs-standard/1.2.1/
 
 SVN tag:
 https://svn.apache.org/repos/asf/tomcat/taglibs/standard/tags/taglibs-standard-1.2.1
  @ r1541786
 https://svn.apache.org/r1541786
 
 KEYS: https://svn.apache.org/repos/asf/tomcat/trunk/KEYS
 
 
 
 Several questions on packaging.
 
 1. taglibs-build-tools component.
 
 This name is too generic. Do you intend to share this component among
 all org.apache.taglibs ? Will it carry the same version number (1.2.1)
 as the standard taglib?
 
 Actually it is a jar that contains nothing besides checkstyle.xml file
 and files in META-INF.  Do we actually need to release this component?

It is referenced as part of the build for each of the modules. Having it in the 
repo allows each of the individual sub-modules to be built on its own.

 
 2. It is odd that there is no downloadable zip file with binaries.
 Will we instruct the users to download those from Maven?
 
 It was OK with Tomcat Maven Plugin, because the plugin is not usable
 without Maven. It is odd to do so with a regular library.  E.g. Apache
 Commons do release binary zips/targzs for their libraries.
 
 It is not a show-stopper though. It is better to have the binaries
 somewhere than none at all.
 
 I have not tested the code yet.
 
 Best regards,
 Konstantin Kolinko
 
 -
 To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
 For additional commands, e-mail: dev-h...@tomcat.apache.org
 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [VOTE] Release Apache Standard Taglib 1.2.1

[Jeremy Boynes]

On Nov 19, 2013, at 9:10 PM, Jeremy Boynes jboy...@apache.org wrote:

 On Nov 16, 2013, at 8:29 AM, Jeremy Boynes jboy...@apache.org wrote:
 
 On Nov 13, 2013, at 6:58 PM, Jeremy Boynes jboy...@apache.org wrote:
 
 Nudge - please could a couple of PMC members vote on this release (there are 
 no PMC votes at the moment).
 
 Another gentle nudge - we currently have one binding +1 from rjung@ and one 
 non-binding +1 from jboynes@

Nudge.

Thanks
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: svn commit: r1544165 - XML validation

[Jeremy Boynes]

On Nov 21, 2013, at 7:16 PM, Jeremy Boynes jboy...@apache.org wrote:

 On Nov 21, 2013, at 4:59 AM, ma...@apache.org wrote:
 
 Author: markt
 Date: Thu Nov 21 12:59:36 2013
 New Revision: 1544165
 
 URL: http://svn.apache.org/r1544165
 Log:
 Extend the reach of the Context attribute xmlValidation to cover XML parsing 
 by Jasper the JSP engine.
 Fix a TODO and enable validation (if required) for implicit TLD files.
 
 JSP 6.2.4 raises issues around attempting to validate JSP Documents (.jspx 
 and .tagx) due to the mixin's from the jsp: namespace. This could perhaps be 
 enabled using a separate option from validation of the descriptors so users 
 can validate the standard files but not need to add anything to the 
 DOCTYPE/XSD for the application content.

This may not be that simple, and I think I've some other issues in this area, 
see http://svn.apache.org/r1545537

The test there fails because the schema cannot be found. JspDocumentParser is 
not setting the baseURI for the document being parser and its implementation of 
URI resolver always returns null. This means it cannot resolve the systemId 
valid.xsd relative to the document, tries using the current working 
directory, does not find it there and hence fails because it cannot find the 
schema to validate against.

If I replace the relative URI with a absolute location then it attempts to 
validate but fails because the jsp:text element is not valid in the string 
simple type of root.

If I do not enable validation in JspDocumentParser, the .jspx evaluates 
correctly and the client is able to parse and validate the XML document 
response (showing the XSD and XSI are valid).

If I disable XSD validation but leave DTD validation on then it fails because 
there is no DOCTYPE in the .jspx

It seems to me the original behaviour from JSP6.2.4 is the best we can do:
* parse with no validation
* if a DOCTYPE is found, restart with DTD validation on
* never attempt XSD validation

This would eliminate the need for the org.apache.jasper.XML_VALIDATE_DOC 
option and related context property so I propose we back that change out.

I'd been meaning to update JspDP to be a DefaultHandler2 to match the XML 
refactoring in Digester so I'll fix the baseURI issue as part of that.

Cheers
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: svn commit: r1544165 - XML validation

[Jeremy Boynes]

On Nov 22, 2013, at 1:28 AM, Mark Thomas ma...@apache.org wrote:

Still catching up on the changes but this jumped out:
 
  Looks like JasperInitializer now reads to read an attribute not an
   init-param. In trunk, I'd been trying to decouple Jasper from
  Catalina so having the latter set a jasper-specific attribute
  reverts that - in TC7 this is controlled for each JspServlet using
  a validating init-param (i.e. it does not inherit from the
  Context).
 
 There are multiple aspects to this.
 
 Jasper needs to be usable in other containers so it can't depend on
 Catalina. Both attributes and init-params allow this.
 
 There are some things - like XML validation - that apply to both
 Catalina and Jasper and users shouldn't have to configure this in
 multiple places. attributes vs. init-params are really just the
 mechanism used to pass config from Catalina to Jasper in this case. I
 think it is easier to do this with attributes than init-params.

The issue with magic attributes though is that that container needs to set 
them, whereas init-params can be set in a descriptor supplied with the 
application. I think attributes are easier as well but think there's benefit in 
the other container case of having Jasper's configuration be standalone not 
need container support.

Ideally this configuration would be exposed to an SCI in way defined by the 
spec (e.g. a javax. attribute or method on ServletContext).

Thanks
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: XML parsing config clean-up

[Jeremy Boynes]

On Nov 22, 2013, at 5:40 AM, Mark Thomas ma...@apache.org wrote:

 On 22/11/2013 13:04, Konstantin Kolinko wrote:
 2013/11/22 Mark Thomas ma...@apache.org:
...
 
 It does not mention anything else.
 
 There might be 3rd-party tag libraries that do not validate. I think
 it is good to have a separate option.
 
 OK. That is enough to nudge me off the fence. I'll get that code
 re-instated. Another advantage is that 7.0.x and 8.0.x will the same code.

Is the change backward compatible in 7.0.x? E.g. does validating init-param 
for JspServlet still work?


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: XML parsing config clean-up

[Jeremy Boynes]

On Nov 22, 2013, at 8:24 AM, Mark Thomas ma...@apache.org wrote:

 Signed PGP part
 On 22/11/2013 16:18, Jeremy Boynes wrote:
  On Nov 22, 2013, at 5:40 AM, Mark Thomas ma...@apache.org wrote:
 
  On 22/11/2013 13:04, Konstantin Kolinko wrote:
  2013/11/22 Mark Thomas ma...@apache.org:
  ...
 
  It does not mention anything else.
 
  There might be 3rd-party tag libraries that do not validate. I
  think it is good to have a separate option.
 
  OK. That is enough to nudge me off the fence. I'll get that code
  re-instated. Another advantage is that 7.0.x and 8.0.x will the
  same code.
 
  Is the change backward compatible in 7.0.x? E.g. does validating
  init-param for JspServlet still work?
 
 Where is (was?) that in the 7.0.x code base? I can;t see it now and I
 don't recall removing it from 7.0.x.

This was the change:
http://svn.apache.org/viewvc/tomcat/tc7.0.x/trunk/java/org/apache/jasper/EmbeddedServletOptions.java?r1=1155369r2=1544198diff_format=h

Looking at the code, it looks the option wouldn't work in the first place as 
the default value in ParserUtils is false and ESO never sets the value true.

Cheers
Jeremy




signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: svn commit: r1544165 - XML validation

[Jeremy Boynes]

On Nov 21, 2013, at 4:59 AM, ma...@apache.org wrote:

 Author: markt
 Date: Thu Nov 21 12:59:36 2013
 New Revision: 1544165
 
 URL: http://svn.apache.org/r1544165
 Log:
 Extend the reach of the Context attribute xmlValidation to cover XML parsing 
 by Jasper the JSP engine.
 Fix a TODO and enable validation (if required) for implicit TLD files.

JSP 6.2.4 raises issues around attempting to validate JSP Documents (.jspx and 
.tagx) due to the mixin's from the jsp: namespace. This could perhaps be 
enabled using a separate option from validation of the descriptors so users can 
validate the standard files but not need to add anything to the DOCTYPE/XSD for 
the application content.

Looks like JasperInitializer now reads to read an attribute not an init-param. 
In trunk, I'd been trying to decouple Jasper from Catalina so having the latter 
set a jasper-specific attribute reverts that - in TC7 this is controlled for 
each JspServlet using a validating init-param (i.e. it does not inherit from 
the Context).

I've been thinking of adding a descriptor file for Jasper (e.g. 
META-INF/jasper.xml to go alongside context.xml) which would contain 
configuration of the JSP engine that is currently split between context 
init-params, per-servlet init-params, and system properties. How about we 
configure validation there?

Cheers
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [VOTE] Release Apache Standard Taglib 1.2.1

[Jeremy Boynes]

On Nov 16, 2013, at 8:29 AM, Jeremy Boynes jboy...@apache.org wrote:

 On Nov 13, 2013, at 6:58 PM, Jeremy Boynes jboy...@apache.org wrote:
 
 Nudge - please could a couple of PMC members vote on this release (there are 
 no PMC votes at the moment).

Another gentle nudge - we currently have one binding +1 from rjung@ and one 
non-binding +1 from jboynes@

Thanks
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: svn commit: r1542836 - /tomcat/trunk/java/org/apache/jasper/compiler/TagLibraryInfoImpl.java

[Jeremy Boynes]

On Nov 17, 2013, at 1:42 PM, ma...@apache.org wrote:

 Author: markt
 Date: Sun Nov 17 21:42:26 2013
 New Revision: 1542836
 
 URL: http://svn.apache.org/r1542836
 Log:
 Revert part of r1542565 which added rather than removed an IDE warning
 
 Modified:
tomcat/trunk/java/org/apache/jasper/compiler/TagLibraryInfoImpl.java
 
 Modified: tomcat/trunk/java/org/apache/jasper/compiler/TagLibraryInfoImpl.java
 URL: 
 http://svn.apache.org/viewvc/tomcat/trunk/java/org/apache/jasper/compiler/TagLibraryInfoImpl.java?rev=1542836r1=1542835r2=1542836view=diff
 ==
 --- tomcat/trunk/java/org/apache/jasper/compiler/TagLibraryInfoImpl.java 
 (original)
 +++ tomcat/trunk/java/org/apache/jasper/compiler/TagLibraryInfoImpl.java Sun 
 Nov 17 21:42:26 2013
 @@ -138,7 +138,8 @@ class TagLibraryInfoImpl extends TagLibr
 // Add TLD within the JAR to the dependency list
 String entryName = tldResourcePath.getEntryName();
 try {
 -pageInfo.addDependant(jar.getURL(entryName), 
 jar.getLastModified(entryName));
 +pageInfo.addDependant(jar.getURL(entryName),
 +Long.valueOf(jar.getLastModified(entryName)));
 } catch (IOException ioe) {
 throw new JasperException(ioe);
 }

Hmm, differences in IDE's opinion I assume - with this IDEA gives me a warning 
about unnecessary boxing and nothing without. What are you seeing?


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: svn commit: r1542339 - in /tomcat/trunk: java/org/apache/catalina/realm/JNDIRealm.java webapps/docs/config/realm.xml

[Jeremy Boynes]

On Nov 17, 2013, at 2:09 PM, Mark Thomas ma...@apache.org wrote:

 On 15/11/2013 20:09, Konstantin Kolinko wrote:
 Wrong spelling of the property name.
 s/spengo/spnego/
 
 Grr. I keep doing that. No idea why.

I believe it has a reality distortion field. At least, I always have to remind 
myself it's not spegno. 


signature.asc
Description: Message signed with OpenPGP using GPGMail

Coding styles...

[Jeremy Boynes]

On Nov 17, 2013, at 2:33 PM, Konstantin Kolinko knst.koli...@gmail.com wrote:

 2013/11/18 Mark Thomas ma...@apache.org:
...
 
 The coding standard used in Tomcat is that all boxing and unboxing must
 be explicit. The second parameter in the addDependant() method is a Long
 so the long must be explicitly converted.
 
 
 +1
 
 By the way,
 the settings for Eclipse IDE are documented here:
 
 http://svn.apache.org/viewvc/tomcat/trunk/res/ide-support/eclipse/java-compiler-errors-warnings.txt?view=markup

Not all of us use Eclipse and relying on the IDE's defaults is ambiguous. I 
added an export from IDE for the settings I'm using but don't know what else is 
needed.

For things that aren't checked in the build, can we please document them 
somewhere.
Thanks
Jeremy


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [VOTE] Release Apache Standard Taglib 1.2.1

[Jeremy Boynes]

On Nov 13, 2013, at 6:58 PM, Jeremy Boynes jboy...@apache.org wrote:

 I'd like to release Apache Standard Taglib 1.2.1. This is an update to the 
 withdrawn 1.2.0, built with JDK 1.7.0_45 to address JavaDoc issues and 
 incorporating feedback on the documentation.
 
 Maven Staging Repository:
 https://repository.apache.org/content/repositories/orgapachetomcat-132/
 
 Source Distribution:
 https://repository.apache.org/content/repositories/orgapachetomcat-132/org/apache/taglibs/taglibs-standard/1.2.1/
 
 SVN tag:
 https://svn.apache.org/repos/asf/tomcat/taglibs/standard/tags/taglibs-standard-1.2.1
  @ r1541786
 https://svn.apache.org/r1541786
 
 KEYS: https://svn.apache.org/repos/asf/tomcat/trunk/KEYS
 
 The proposed 1.2.1 release is:
 [ ] Broken - do not release
 [ ] OK - release as 1.2.1

Nudge - please could a couple of PMC members vote on this release (there are no 
PMC votes at the moment).

Thanks
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: [VOTE] Release Apache Standard Taglib 1.2.1

[Jeremy Boynes]

On Nov 16, 2013, at 10:33 AM, Rainer Jung rainer.j...@kippdata.de wrote:
 
 Build warnings:
 
 6 times:
 
 WARNING:  'org.apache.xerces.jaxp.SAXParserImpl: Property
 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not
 recognized.'
 Warning:  org.apache.xerces.parsers.SAXParser: Feature
 'http://javax.xml.XMLConstants/feature/secure-processing' is not recognized.
 Warning:  org.apache.xerces.parsers.SAXParser: Property
 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not
 recognized.
 Warning:  org.apache.xerces.parsers.SAXParser: Property
 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not
 recognized.
 Warning:  org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser: Property
 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not
 recognized.

I started seeing those as well once I switched to JDK 1.7.0_45. They come from 
the RAT plugin used to verify the file licenses so I think are OK for this 
release (to be looked at in trunk, may just need a different version of the 
plugin).

Thanks
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

MBeanDeregistration test failure

[Jeremy Boynes]

On Nov 15, 2013, at 11:29 PM, Mark Thomas ma...@apache.org wrote:
 
 On a related note, when I ran the unit tests last night I saw a couple of 
 intermittent failures that I'd like to try and get to the bottom of before 
 tagging the next RC.

Is one of those org.apache.catalina.mbeans.TestRegistration? It has recently 
started failing for me in BIO mode and looks related to r1539452.

Thanks
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

TldLocation

[Jeremy Boynes]

On Nov 12, 2013, at 8:53 AM, Mark Thomas ma...@apache.org wrote:

 On 12/11/2013 15:27, Jeremy Boynes wrote:
 On Nov 12, 2013, at 5:00 AM, ma...@apache.org wrote:
 
 Author: markt Date: Tue Nov 12 13:00:03 2013 New Revision:
 1541041
 
 URL: http://svn.apache.org/r1541041 Log: Replace
 TldLocationsCache with the new TldCache that also caches the
 contents of the TLDs. This is the next step in the refactoring of
 TLD handling.
 
 How far do you intend to go with these changes? I was just starting
 to get back to them and don't want to conflict.
 
 I was planning on completing the removal of TldLocation. I'm currently
 working my way through understanding what that means.

I'd started by trying to remove the parsing code from TagLibraryInfoImpl, 
making a thin decorator for a TaglibXml that could be shared (given the 
TLIImpls are page/compilation specific due to both prefix and the reference to 
other TLIs). IIRC, that worked for tags but there was a coupling somewhere in 
the TagFileProcessor call used to populate the TagInfo for a tag file. I was 
trying to figure out why
ctxt.setTagFileJarResource(path, jar);
was needed in before calling it and how #46471 had been fixed.

In investigating, I was also tempted to try and separate the directive and load 
processing (visitors) in TagFileProcessor and see if there was a way to only do 
that once.

I should have more time to contribute over the next couple of weeks and could 
pick this up again if you've not got too far.

Cheers
Jeremy



signature.asc
Description: Message signed with OpenPGP using GPGMail

[VOTE] Release Apache Standard Taglib 1.2.1

[Jeremy Boynes]

I'd like to release Apache Standard Taglib 1.2.1. This is an update to the 
withdrawn 1.2.0, built with JDK 1.7.0_45 to address JavaDoc issues and 
incorporating feedback on the documentation.

Maven Staging Repository:
https://repository.apache.org/content/repositories/orgapachetomcat-132/

Source Distribution:
https://repository.apache.org/content/repositories/orgapachetomcat-132/org/apache/taglibs/taglibs-standard/1.2.1/

SVN tag:
https://svn.apache.org/repos/asf/tomcat/taglibs/standard/tags/taglibs-standard-1.2.1
 @ r1541786
https://svn.apache.org/r1541786

KEYS: https://svn.apache.org/repos/asf/tomcat/trunk/KEYS

The proposed 1.2.1 release is:
[ ] Broken - do not release
[ ] OK - release as 1.2.1

Thanks
Jeremy


signature.asc
Description: Message signed with OpenPGP using GPGMail