[ANN] Apache Tomcat 9.0.56 available

2021-12-08 Thread Rémy Maucherat
The Apache Tomcat team announces the immediate availability of Apache
Tomcat 9.0.56.

Apache Tomcat 9 is an open source software implementation of the Java
Servlet, JavaServer Pages, Java Unified Expression Language, Java
WebSocket and JASPIC technologies.

Apache Tomcat 9.0.56 is a bugfix and feature release. The notable
changes compared to 9.0.55 include:

- Provide protection against a known OS bug that causes the acceptor to
   report an incoming connection more than once.

- Implement a workaround for a JVM bug that can trigger a file
   descriptor leak when using multi-part upload and the application does
   not explicitly close an input stream for an uploaded file that was
   cached on disk.

- Fix exceptions when the security manager is enabled and the first
   request received after starting is an HTTP request to a TLS enabled
   NIO2 connector.

Along with lots of other bug fixes and improvements.

Please refer to the change log for the complete list of changes:
https://tomcat.apache.org/tomcat-9.0-doc/changelog.html


Downloads:
https://tomcat.apache.org/download-90.cgi

Migration guides from Apache Tomcat 7.x and 8.x:
https://tomcat.apache.org/migration.html

Enjoy!

- The Apache Tomcat team

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



[ANN] Apache Tomcat 9.0.55 available

2021-11-15 Thread Rémy Maucherat
The Apache Tomcat team announces the immediate availability of Apache
Tomcat 9.0.55.

Apache Tomcat 9 is an open source software implementation of the Java
Servlet, JavaServer Pages, Java Unified Expression Language, Java
WebSocket and JASPIC technologies.

Apache Tomcat 9.0.55 is a bugfix and feature release. The notable
changes compared to 9.0.54 include:

- Experimental OpenSSL support through the Panama API incubating in Java
17, with support for OpenSSL 1.1+.

- Add support for custom caching strategies for web application
resources. This initial implementation allows control over whether or
not a resource is cached.

- Improve robustness of JNDIRealm for exceptions occurring when getting
the connection.

Along with lots of other bug fixes and improvements.

Please refer to the change log for the complete list of changes:
https://tomcat.apache.org/tomcat-9.0-doc/changelog.html


Downloads:
https://tomcat.apache.org/download-90.cgi

Migration guides from Apache Tomcat 7.x and 8.x:
https://tomcat.apache.org/migration.html

Enjoy!

- The Apache Tomcat team

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Not able to enable SSL on AJP connector

2021-11-01 Thread Rémy Maucherat
On Mon, Nov 1, 2021 at 3:32 PM Jasvant Singh
 wrote:
> The SSLHostConfig works fine with HTTP/1.1 but not with AJP/1.3 Connector.

The AJP protocol does not support use of TLS. If you want TLS, you can
use HTTP as the proxy protocol, or set up tunnels that use TLS.

Rémy

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: JNDIRealm thread blocking issue

2021-10-29 Thread Rémy Maucherat
On Fri, Oct 29, 2021 at 9:28 AM Suvendu Sekhar Mondal  wrote:
>
> Hello Chris,
>
> On Fri, Oct 29, 2021 at 2:46 AM Christopher Schultz
>  wrote:
> >
> > Suvendu,
> >
> > On 10/28/21 12:55, Suvendu Sekhar Mondal wrote:
> > > Hello Everyone,
> > >
> > > I was investigating one thread pool exhaustion issue. Thread dump
> > > analysis showed that all HTTP threads were waiting for a ReentrantLock
> > > object. Object address 0x00066d727f28 were same for all of the
> > > waiting threads:
> > >
> > > "http-nio-18100-exec-86" #32808 daemon prio=5 os_prio=0
> > > tid=0x51835800 nid=0x29bc waiting on condition
> > > [0x7a5be000]
> > > java.lang.Thread.State: WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00066d727f28> (a
> > > java.util.concurrent.locks.ReentrantLock$NonfairSync)
> > > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> > > at 
> > > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> > > at 
> > > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
> > > at 
> > > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
> > > at 
> > > java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
> > > at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
> > > at org.apache.catalina.realm.JNDIRealm.get(JNDIRealm.java:2385)
> > > at org.apache.catalina.realm.JNDIRealm.authenticate(JNDIRealm.java:1274)
> > >
> > > There was no hint in the thread dump about which thread was owning the
> > > lock. Luckily, one heap dump was taken before generating thread dump.
> > > When I queried the heap dump for that ReentrantLock object, I saw that
> > > another thread(http-nio-18100-exec-4) was holding the
> > > lock(exclusiveOwnerThread). There was NO trace of
> > > http-nio-18100-exec-4 thread in any of the thread dumps! So it was a
> > > "lock without an owner" case.
> >
> > I think you are looking at several pieces of evidence that may or may
> > not correlate to each other at all. The fact that the thread wasn't in
> > the thread dump indicates that the thread (or even the whole JVM) had
> > terminated between the time you took the heap-dump and the thread dump.
> > Most likely, the monitor was owned by another thread when you took your
> > thread-dump. Try using other tools which *do* reveal the lock-holders
> > identity.
> >
>
> This issue has happened a few times. "Busy Thread Count" was high
> during the problem period. JVM was up and running when I collected
> heap and thread dumps - pid was not changed in-between. I used jstack,
> visualvm, jcmd - nothing revealed owing thread details. Only heap
> dumps had some information on that object and which thread was holding
> onto it. Here is a snap: https://pasteboard.co/D7dV3jej6zId.jpg
>
> I can simulate similar blocking without Tomcat with dummy code. There
> also nothing reveals the owner's identity except the heap dump. Here
> is sample: https://gist.github.com/suv3ndu/2ec9fe660d2b833996817ed62186eac2
>
> > > After glancing through the Tomcat’s JNDIRealm.get() code and
> > > beyond[1], I can see lock is being acquired on singleConnectionLock.
> > > That lock is getting released either in the close() or release()
> > > method. So, if something bad happens to the thread which is trying to
> > > establish a connection, then lock will be held without a proper owner
> > > and a thread blocking situation will be created. Am I interpreting the
> > > code correctly? Should we not handle any failure inside get()?
> > >
> > > Also, I still have not got the reason why the thread got terminated.
> > > Any suggestions on how I can enable any specific logging?
> > >
> > > My setup is:
> > > Tomcat version: 9.0.39
> > > Connector: NIO
> > > JDK: AdoptOpenJDK: 1.8.192
> > > OS: Windows 2016
> >
> > Looks like you need a whole bunch of upgrades. Search the Tomcat 9.x
> > changelog for "JNDIRealm" and you'll see there have been changes since
> > 9.0.39 that may have already resolved this issue. Are you able to
> > re-test with Tomcat 9.0.54?
> >
>
> It will not be easy for me to upgrade it and test it. Lots of approval
> is required to get that done. :(
>
> >  > [1]
> > https://github.com/apache/tomcat/blob/57a6a40fc9f995e4d449358bbde047aab6d9f39a/java/org/apache/catalina/realm/JNDIRealm.java#L2553
> >
> > Note that you are looking at the current version of JNDIRealm.java. The
> > version you are running is 17 commits behind that.
> >
> > The line of code calling ReentrantLock.lock in your code would be
> > https://github.com/apache/tomcat/blob/57a6a40fc9f995e4d449358bbde047aab6d9f39a/java/org/apache/catalina/realm/JNDIRealm.java#L2385
> > which is "return null" indicating that there is a version mismatch
> > between the code you are running and the code you are reading.
> >
>
> 

[ANN] Apache Tomcat 9.0.54 available

2021-10-04 Thread Rémy Maucherat
The Apache Tomcat team announces the immediate availability of Apache
Tomcat 9.0.54.

Apache Tomcat 9 is an open source software implementation of the Java
Servlet, JavaServer Pages, Java Unified Expression Language, Java
WebSocket and JASPIC technologies.

Apache Tomcat 9.0.54 is a bugfix and feature release. The notable
changes compared to 9.0.53 include:

- Further robustness improvements to HTTP/2 flow control window
   management.

- Improvements to the DataSourceUserDatabase.

- Fix an issue that caused some Servlet non-blocking API reads of the
   HTTP request body to incorrectly use blocking IO.

Along with lots of other bug fixes and improvements.

Please refer to the change log for the complete list of changes:
https://tomcat.apache.org/tomcat-9.0-doc/changelog.html


Downloads:
https://tomcat.apache.org/download-90.cgi

Migration guides from Apache Tomcat 7.x and 8.x:
https://tomcat.apache.org/migration.html

Enjoy!

- The Apache Tomcat team

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



[ANN] Apache Tomcat 9.0.53 available

2021-09-13 Thread Rémy Maucherat
The Apache Tomcat team announces the immediate availability of Apache
Tomcat 9.0.53.

Apache Tomcat 9 is an open source software implementation of the Java
Servlet, JavaServer Pages, Java Unified Expression Language, Java
WebSocket and JASPIC technologies.

Apache Tomcat 9.0.53 is a bugfix and feature release. The notable
changes compared to 9.0.52 include:

- Add a UserDatabase implementation as a superset of the DataSourceRealm
   functionality.

- Update the internal fork of Apache Commons DBCP to 2.9.0 and Apache
   Commons Pool to 2.11.1

- Update the packaged version of the Tomcat Native Library to 1.2.31 to
   pick up Windows binaries built with OpenSSL 1.1.1l.

Along with lots of other bug fixes and improvements.

Please refer to the change log for the complete list of changes:
http://tomcat.apache.org/tomcat-9.0-doc/changelog.html


Downloads:
http://tomcat.apache.org/download-90.cgi

Migration guides from Apache Tomcat 7.x and 8.x:
http://tomcat.apache.org/migration.html

Enjoy!

- The Apache Tomcat team

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: BasicDataSource restart()

2021-09-03 Thread Rémy Maucherat
On Fri, Sep 3, 2021 at 2:46 AM Jerry Malcolm  wrote:
>
> I have a requirement to start a new log database on the first of every
> month.  I still need to have access to older monthly log databases.   I
> do not want to create a bunch of hardcoded manually configured
> individual datasources, one for each month.  I have a dynamic datasource
> solution that is completely implemented and working except for one
> little thing.
>
> I access the BasicDataSource implementation class for the datasource.  I
> have an algorithm that substitutes _MM at the appropriate spot in
> the configured URL and then updates the url in the datasource.   All of
> this works great.  I can live with the fact that the datasource can only
> point to one database at a time.  My concern is that once I transition
> to another database, there are existing connections in the pool that are
> already attached to the old database.  I need to clear those out and
> start over.  But I don't have the luxury of bouncing tomcat to clean it up.
>
> The apache commons BasicDataSource has a restart() method.  But
> unfortunately that method is omitted from the Tomcat version. There is a
> close() method on the BasicDataSource.  But I don't see anything that
> will re-open it after closing.  I thought about changing maxActive to 0,
> and waiting for it to drain, then setting it back to the original
> value.  None of these sound like an ideal solution.  Without a restart()
> method, is there any other way to force all existing connections to
> close and start clean?

The code is kept in sync with DBCP (with a bit of lag maybe), so these
lifecycle methods were also added to Tomcat one year ago (9.0.38+ and
8.5.58+).

Rémy

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



[ANN] Apache Tomcat 9.0.52 available

2021-08-07 Thread Rémy Maucherat
The Apache Tomcat team announces the immediate availability of Apache
Tomcat 9.0.52.

Apache Tomcat 9 is an open source software implementation of the Java
Servlet, JavaServer Pages, Java Unified Expression Language, Java
WebSocket and JASPIC technologies.

Apache Tomcat 9.0.52 is a bugfix and feature release. The notable
changes compared to 9.0.50 include:

- Correct a regression in the previous release in the HTTP/2 flow
   control window management

- Correct a regression the could cause some TLS connections to hang when
   using NIO

- Use of GraalVM native images no longer automatically disables JMX
   support.

Along with lots of other bug fixes and improvements.

Please refer to the change log for the complete list of changes:
http://tomcat.apache.org/tomcat-9.0-doc/changelog.html


Downloads:
http://tomcat.apache.org/download-90.cgi

Migration guides from Apache Tomcat 7.x and 8.x:
http://tomcat.apache.org/migration.html

Enjoy!

- The Apache Tomcat team

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Possible Http11NioProtocol regression since 9.0.48?

2021-07-06 Thread Rémy Maucherat
On Tue, Jul 6, 2021 at 10:41 AM Mark Thomas  wrote:
>
> On 05/07/2021 20:19, André van der Lugt wrote:
> >
> > Mark, thank you for your efforts so far.
>
> No problem. Happy to help.
>
> At this point it is hard to tell where the root cause is. It is possible
> that the Tomcat changes introduced a bug in Tomcat. It is also possible
> that the Tomcat changes triggered a set of circumstances that exposed a
> pre-existing bug elsewhere (Java, OS, network).

I can revert the changes again and we can figure out what's wrong in
10.x eventually.

Rémy

>
> As the next step, I'd suggest collecting a network trace from the Tomcat
> VM at the same you collect a trace from the client. The aim of this
> would be to determine if Tomcat / Java / OS/ VM is delaying those final
> packets or if the delay is occurring somewhere in the network. By
> comparing the two traces you should be able to narrow down where the
> delay is happening.
>
> Mark
>
>
> >
> >>
> >> Other questions that come to mind:
> >>
> >> 1. Was the network trace created from a client connected directly to
> >> Tomcat or was there a proxy / firewall / something else between the
> >> client and Tomcat?
> >
> > Tomcat is running on the company network to which I'm connected
> > through company mandated VPN and who knows what else
> > equipment there is between me and Tomcat's VM.
> >
> > Your question made me think: what if I try from a browser installed
> > on another VM on the company network? So I did from two different
> > machines and bingo ... the problem does not manifest itself.
> > Maybe I need to have a word with our IT?
> >
> >>
> >> 2. Where was the trace collected from? On the client? On the server?
> >> Somewhere else?
> >
> > Trace was collected on the client.
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



[ANN] Apache Tomcat 9.0.50 available

2021-07-05 Thread Rémy Maucherat
The Apache Tomcat team announces the immediate availability of Apache
Tomcat 9.0.50.

Apache Tomcat 9 is an open source software implementation of the Java
Servlet, JavaServer Pages, Java Unified Expression Language, Java
WebSocket and JASPIC technologies.

Apache Tomcat 9.0.50 is a bugfix and feature release. The notable
changes compared to 9.0.48 include:

- Re-work the HTTP/2 overhead protection to reduce the likelihood of
   false positives. Note that the default overheadCountFactor has changed
   from 1 to 10 and that the useful range is now 0 to ~20.

- Update to Eclipse JDT compiler 4.20.

- Fix regressions in JSP compilation in the previous release.

Along with lots of other bug fixes and improvements.

Please refer to the change log for the complete list of changes:
http://tomcat.apache.org/tomcat-9.0-doc/changelog.html


Downloads:
http://tomcat.apache.org/download-90.cgi

Migration guides from Apache Tomcat 7.x and 8.x:
http://tomcat.apache.org/migration.html

Enjoy!

- The Apache Tomcat team

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



[ANN] Apache Tomcat 9.0.48 available

2021-06-17 Thread Rémy Maucherat
The Apache Tomcat team announces the immediate availability of Apache
Tomcat 9.0.48.

Apache Tomcat 9 is an open source software implementation of the Java
Servlet, JavaServer Pages, Java Unified Expression Language, Java
WebSocket and JASPIC technologies.

Apache Tomcat 9.0.48 is a bugfix and feature release. The notable
changes compared to 9.0.46 include:

- Improve robustness of HTTP/2 HPACK decoding

- Improvements to the handling of the Transfer-Encoding header

- Review code used to generate Java source from JSPs and tags and remove
   code found to be unnecessary.

- Backport the updated blocking NIO code and optimizations from Tomcat 10.0.

Along with lots of other bug fixes and improvements.

Please refer to the change log for the complete list of changes:
http://tomcat.apache.org/tomcat-9.0-doc/changelog.html


Downloads:
http://tomcat.apache.org/download-90.cgi

Migration guides from Apache Tomcat 7.x and 8.x:
http://tomcat.apache.org/migration.html

Enjoy!

- The Apache Tomcat team

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



[ANN] Apache Tomcat 9.0.48 available

2021-06-17 Thread Rémy Maucherat
The Apache Tomcat team announces the immediate availability of Apache
Tomcat 9.0.48.

Apache Tomcat 9 is an open source software implementation of the Java
Servlet, JavaServer Pages, Java Unified Expression Language, Java
WebSocket and JASPIC technologies.

Apache Tomcat 9.0.48 is a bugfix and feature release. The notable
changes compared to 9.0.46 include:

- Improve robustness of HTTP/2 HPACK decoding

- Improvements to the handling of the Transfer-Encoding header

- Review code used to generate Java source from JSPs and tags and remove
   code found to be unnecessary.

- Backport the updated blocking NIO code and optimizations from Tomcat 10.0.

Along with lots of other bug fixes and improvements.

Please refer to the change log for the complete list of changes:
http://tomcat.apache.org/tomcat-9.0-doc/changelog.html


Downloads:
http://tomcat.apache.org/download-90.cgi

Migration guides from Apache Tomcat 7.x and 8.x:
http://tomcat.apache.org/migration.html

Enjoy!

- The Apache Tomcat team


Re: Trouble with HTTP/2 during concurrent bulk data transfer (server -> client)

2021-06-17 Thread Rémy Maucherat
On Thu, Jun 17, 2021 at 9:27 AM Mark Thomas  wrote:

> On 17/06/2021 07:56, Rémy Maucherat wrote:
>
> > The main benefit is that it removes some blocking IO which is a good
> idea.
> > NIO2 is worth testing with your new test, BTW.
>
> NIO2 works. The issue appears to be limited to the NIO connector.
>

Interesting. At least it's another workaround then. Are you testing NIO on
Java 11+ ?

Rémy


>
> Mark
>
> > Transferring large files with HTTP/2 is a bad idea though, it's always
> > going to be inefficient compared to HTTP/1.1. And if the idea is to
> > multiplex, then it will become terrible, that's a given and it should not
> > be done. HTTP/2 is very good only if you have tons of small entities to
> > transfer (that's what sites have these days, so that's good).
> >
> > Rémy
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>


Re: Trouble with HTTP/2 during concurrent bulk data transfer (server -> client)

2021-06-17 Thread Rémy Maucherat
On Wed, Jun 16, 2021 at 11:02 PM Mark Thomas  wrote:

> On 16/06/2021 19:42, Deshmukh, Kedar wrote:
> > Thanks Mark for the quick update.
> >
> > Can you please provide how useAsyncIO="false" makes impact in terms of
> performance, scalability (number of connections to the server) and
> reliability ?
>
> Well, if you set useAsyncIO="false" it works. If you set
> useAsyncIO="true" it fails 90% of the time.
>
> I'd suggest that, with that failure rate, discussion of performance and
> scalability are somewhat irrelevant.
>
> With test cases that do not trigger this issue the difference between
> the two is marginal. Remy is more familiar with the details than me but
> from memory useAsyncIO="true" is a little better on older JREs. On
> modern JREs there isn't much in it. It is also likely to depend on
> application usage patterns, OS, etc. In short, you'd need to test the
> difference on your application with your hardware etc. I'd expect the
> difference to be hard to measure.
>

The main benefit is that it removes some blocking IO which is a good idea.
NIO2 is worth testing with your new test, BTW.

Transferring large files with HTTP/2 is a bad idea though, it's always
going to be inefficient compared to HTTP/1.1. And if the idea is to
multiplex, then it will become terrible, that's a given and it should not
be done. HTTP/2 is very good only if you have tons of small entities to
transfer (that's what sites have these days, so that's good).

Rémy


>
> Mark
>
>
> >
> > Regards,
> > Kedar
> >
> >
> > -Original Message-
> > From: Mark Thomas 
> > Sent: Wednesday, June 16, 2021 11:41 PM
> > To: users@tomcat.apache.org
> > Subject: Re: Trouble with HTTP/2 during concurrent bulk data transfer
> (server -> client)
> >
> > On 16/06/2021 18:47, Rémy Maucherat wrote:
> >> On Wed, Jun 16, 2021 at 7:36 PM Mark Thomas  wrote:
> >>
> >>> On 16/06/2021 18:01, Deshmukh, Kedar wrote:
> >>>
> >>>> I have one additional question at this point. How easy is this issue
> >>>> to
> >>> reproduce? Does it happen every time? In 10% of requests? 1% ?
> >>>>
> >>>> [Kedar] It is reproducible 9/10 times in my environment. So 90% time
> >>>> it
> >>> is reproducible when concurrency is 5 or more and file sizes are
> >>> between 1GB-5GB.
> >>>
> >>> Thanks for the confirmation. I have converted your test classes into
> >>> a Tomcat unit test (easy for me to work with) and the issue looks to
> >>> be repeatable on Linux with the latest 10.1.x code.
> >>>
> >>> I'm starting to look at this now. I'll post again when I have more
> >>> information. I'm not expecting this to be quick.
> >
> > Kedar,
> >
> > If you set useAsyncIO="false" on the Connector that should work around
> the problem for now. The Servlet Async API will still be available.
> > Tomcat just uses a different code path to write to the network.
> >
> >> I did not expect it would be so easy to reproduce. Can you commit the
> test ?
> >
> > It is a bit of a hack at the moment. The code isn't particularly clean
> and I have hard-coded some file paths for my system (I have a bunch of 5GB
> Windows MSDN ISOs I am using for the large files. I also don't think we
> want test cases that using multi-GB files running on every test run.
> >
> > If I clean things up a bit, parameterise the hard-coded paths bits and
> disable the test by default it should be in a reasonable state to commit.
> >
> > It looks very much like the vectoredOperation and the associated
> semaphore is where things are going wrong.
> >
> > I'm aiming to work on this some more tomorrow.
> >
> > Mark
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
> >
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>


Re: Trouble with HTTP/2 during concurrent bulk data transfer (server -> client)

2021-06-16 Thread Rémy Maucherat
On Wed, Jun 16, 2021 at 7:36 PM Mark Thomas  wrote:

> On 16/06/2021 18:01, Deshmukh, Kedar wrote:
>
> > I have one additional question at this point. How easy is this issue to
> reproduce? Does it happen every time? In 10% of requests? 1% ?
> >
> > [Kedar] It is reproducible 9/10 times in my environment. So 90% time it
> is reproducible when concurrency is 5 or more and file sizes are between
> 1GB-5GB.
>
> Thanks for the confirmation. I have converted your test classes into a
> Tomcat unit test (easy for me to work with) and the issue looks to be
> repeatable on Linux with the latest 10.1.x code.
>
> I'm starting to look at this now. I'll post again when I have more
> information. I'm not expecting this to be quick.
>

I did not expect it would be so easy to reproduce. Can you commit the test ?

Rémy


Re: #tomcat on Freenode?

2021-05-20 Thread Rémy Maucherat
On Thu, May 20, 2021 at 9:57 AM Mark Thomas  wrote:

> On 19/05/2021 20:28, Coty Sutherland wrote:
> > Hi all,
> >
> > I was just notified about some mess going on with Freenode which has
> > seemingly resulted in a mass exodus of users from the freenode servers.
> > There are some updates available at
> > https://gist.github.com/joepie91/df80d8d36cd9d1bde46ba018af497409/ which
> > make it seem like we should no longer point users to #tomcat on freenode
> > (we point to it on https://tomcat.apache.org/irc.html).
> >
> > Should we take any action on that, like remove the page or update it to
> > point to https://libera.chat/ after we establish a channel there? I'm
> not
> > sure how much value there is/was in the freenode channel because
> questions
> > are so infrequent, so we may be able to safely drop the reference.
>
> +1 to removing the freenode link
>
> If we replace it, I'd be happy with https://libera.chat/ but like you I
> wonder if we need it at all. We should probably add the Slack channel
> somewhere too.
>

We should probably drop the IRC page and focus on the mailing lists for the
website (they are the only official channel). A wiki page can mention other
things like Slack.

Rémy


>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>


Re: TOMCAT 9 Upgrade :--java.lang.NoClassDefFoundError: org/apache/tomcat/util/net/ServerSocketFactory

2021-02-23 Thread Rémy Maucherat
On Tue, Feb 23, 2021 at 4:47 PM Christopher Schultz <
ch...@christopherschultz.net> wrote:

> Ravi,
>
> On 2/23/21 10:19, Ravi Kumar wrote:
> > Using this Interface , trying to obtain the object of type
> > ServerSocketFactory in my application.
> > Could you please point towards any other viable alternative ?
> >
> > @Override
> > 
> >  public ServerSocketFactory
> > <
> http://java.sun.com/j2se/1%2E5%2E0/docs/api/javax/net/ServerSocketFactory.html
> >
> > getServerSocketFactory(AbstractEndpoint abstractendpoint) {
> >  return new JSSESocketFactory(abstractendpoint);
> >  }
>
> No, that's "how you are doing it". Nobody just wants to get a reference
> to a ServerSocketFactory object because it looks pretty. What are you
> actually trying to accomplish?
>
> (e.g. log the TLS protocol, customize the cipher suites, etc.)
>

I know people who extended this to add features. The problem is that this
had to go following the removal of java.io in Tomcat, and now the SSLEngine
has to be used instead [compatible with NIO and NIO2]. But it's a huge
change.

Rémy


>
> -chris
>
> > On Tue, Feb 23, 2021 at 8:15 PM Christopher Schultz <
> > ch...@christopherschultz.net> wrote:
> >
> >> Ravi,
> >>
> >> Please don't email list members directly. I'm happy to offer paid
> >> support if you want to email me directly.
> >>
> >> On 2/23/21 05:11, Ravi Kumar wrote:
> >>> We are upgrading the tomcat web server getting used in our web
> >>> application. Currently we are using tomcat 7 and now migrating to
> TOMCAT
> >>> 9.0.43.
> >>> We have an existing  HTTPS based application created using tomcat 7.
> >>> Now after migration and upgrading to tomcat 9, while starting this same
> >>> https application, we are receiving the error message as mentioned
> below
> >>> *SEVERE: Failed to initialize component [Connector[HTTP/1.1-8443]]
> >>> java.lang.NoClassDefFoundError:
> >>> org/apache/tomcat/util/net/ServerSocketFactory*
> >>> *
> >>> *
> >>>It seems that org.apache.tomcat.util.net
> >>> .*ServerSocketFaactory *class has
> >>> been removed since tomcat 8.5.53 and in Tomcat 9 also.
> >>
> >> Correct.
> >>
> >>> Requesting for the helpful suggestion to make our existing application
> >>> working with Tomcat 9 and to resolve this error.
> >>> Please let me know if more information is required.
> >>
> >> What do you need this class for? Tomcat's socket handling has been
> >> refactored between 7.x and 8.x and that class was no longer necessary.
> >>
> >> If your code uses that class directly, you will have to find another way
> >> to accomplish what you are doing.
> >>
> >> So... what are you trying to do?
> >>
> >> -chris
> >>
> >> -
> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >> For additional commands, e-mail: users-h...@tomcat.apache.org
> >>
> >>
> >
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>


Re: Upgraded to 8.5.63, ssl stopped working...?

2021-02-12 Thread Rémy Maucherat
On Fri, Feb 12, 2021 at 8:17 AM Mark Thomas  wrote:

> On February 11, 2021 11:01:27 PM UTC, Jim Weill 
> wrote:
> >Yes, the file is there and readable.  The NTFS permissions have only
> >the
> >built-in SYSTEM, CREATOR OWNER, and domain administrators group as
> >having
> >any kind of access to the folder.  This was working before I started
> >upgrading.  The last modified date is 2017 on that file.
>
> Check the user the Tomcat service is running as. This changed from Local
> System (essentially an admin account) to the less privileged Local Service.
>

I think the problem which messes up everything is a supposed non absolute
URI. I don't remember why this is legitimate, but it probably is, and that
means the error messages are microsoftian. I improved them.

Rémy


> Mark
>
>
> >
> >jim
> >
> >On Thu, Feb 11, 2021 at 2:17 PM Rémy Maucherat  wrote:
> >
> >> On Thu, Feb 11, 2021 at 10:33 PM Jim Weill
> >
> >> wrote:
> >>
> >> > Sorry, I should have posted it yesterday.  This was the only thing
> >I
> >> could
> >> > find that had anything like an error in the stderr log:
> >> >
> >> > 10-Feb-2021 17:34:09.930 SEVERE [main]
> >> > org.apache.catalina.core.StandardService.initInternal Failed to
> >> initialize
> >> > connector [Connector[HTTP/1.1-8443]]
> >> > org.apache.catalina.LifecycleException: Protocol handler
> >initialization
> >> > failed
> >> > at
> >> >
> >org.apache.catalina.connector.Connector.initInternal(Connector.java:1077)
> >> > at
> >org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
> >> > at
> >> >
> >> >
> >>
>
> >org.apache.catalina.core.StandardService.initInternal(StandardService.java:552)
> >> > at
> >org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
> >> > at
> >> >
> >> >
> >>
>
> >org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:846)
> >> > at
> >org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
> >> > at org.apache.catalina.startup.Catalina.load(Catalina.java:639)
> >> > at org.apache.catalina.startup.Catalina.load(Catalina.java:662)
> >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> >> > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> >> > at java.lang.reflect.Method.invoke(Unknown Source)
> >> > at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:302)
> >> > at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:472)
> >> > Caused by: java.lang.IllegalArgumentException: Illegal character in
> >> opaque
> >> > part at index 2: D:\_ssh\_.ICSI.Berkeley.EDU.key
> >> > at
> >> > org.apache.tomcat.util.net
> >> >
> >.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:100)
> >> > at
> >> > org.apache.tomcat.util.net
> >> > .AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:72)
> >> > at
> >org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:247)
> >> > at
> >> > org.apache.tomcat.util.net
> >> > .AbstractEndpoint.init(AbstractEndpoint.java:1143)
> >> > at
> >> > org.apache.tomcat.util.net
> >> > .AbstractJsseEndpoint.init(AbstractJsseEndpoint.java:222)
> >> > at
> >org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:599)
> >> > at
> >> >
> >> >
> >>
>
> >org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:80)
> >> > at
> >> >
> >org.apache.catalina.connector.Connector.initInternal(Connector.java:1075)
> >> > ... 13 more
> >> > Caused by: java.lang.IllegalArgumentException: Illegal character in
> >> opaque
> >> > part at index 2: D:\_ssh\_.ICSI.Berkeley.EDU.key
> >> > at java.net.URI.create(Unknown Source)
> >> > at java.net.URI.resolve(Unknown Source)
> >> > at
> >> >
> >> >
> >>
>
> >org.apache.tomcat.util.file.ConfigFileLoader.getURI(ConfigFileLoader.java:105)
> >> > at
> >> >
> >> >
> >>
>
> >org.apache.tomcat.util.file.ConfigFileLoader.getInputStream(ConfigFileLoader.java:88)
> >> > 

Re: Upgraded to 8.5.63, ssl stopped working...?

2021-02-11 Thread Rémy Maucherat
On Thu, Feb 11, 2021 at 10:33 PM Jim Weill 
wrote:

> Sorry, I should have posted it yesterday.  This was the only thing I could
> find that had anything like an error in the stderr log:
>
> 10-Feb-2021 17:34:09.930 SEVERE [main]
> org.apache.catalina.core.StandardService.initInternal Failed to initialize
> connector [Connector[HTTP/1.1-8443]]
> org.apache.catalina.LifecycleException: Protocol handler initialization
> failed
> at
> org.apache.catalina.connector.Connector.initInternal(Connector.java:1077)
> at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
> at
>
> org.apache.catalina.core.StandardService.initInternal(StandardService.java:552)
> at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
> at
>
> org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:846)
> at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
> at org.apache.catalina.startup.Catalina.load(Catalina.java:639)
> at org.apache.catalina.startup.Catalina.load(Catalina.java:662)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:302)
> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:472)
> Caused by: java.lang.IllegalArgumentException: Illegal character in opaque
> part at index 2: D:\_ssh\_.ICSI.Berkeley.EDU.key
> at
> org.apache.tomcat.util.net
> .AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:100)
> at
> org.apache.tomcat.util.net
> .AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:72)
> at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:247)
> at
> org.apache.tomcat.util.net
> .AbstractEndpoint.init(AbstractEndpoint.java:1143)
> at
> org.apache.tomcat.util.net
> .AbstractJsseEndpoint.init(AbstractJsseEndpoint.java:222)
> at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:599)
> at
>
> org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:80)
> at
> org.apache.catalina.connector.Connector.initInternal(Connector.java:1075)
> ... 13 more
> Caused by: java.lang.IllegalArgumentException: Illegal character in opaque
> part at index 2: D:\_ssh\_.ICSI.Berkeley.EDU.key
> at java.net.URI.create(Unknown Source)
> at java.net.URI.resolve(Unknown Source)
> at
>
> org.apache.tomcat.util.file.ConfigFileLoader.getURI(ConfigFileLoader.java:105)
> at
>
> org.apache.tomcat.util.file.ConfigFileLoader.getInputStream(ConfigFileLoader.java:88)
> at org.apache.tomcat.util.net.jsse.PEMFile.(PEMFile.java:98)
> at org.apache.tomcat.util.net.jsse.PEMFile.(PEMFile.java:90)
> at
> org.apache.tomcat.util.net
> .SSLUtilBase.getKeyManagers(SSLUtilBase.java:313)
> at
> org.apache.tomcat.util.net
> .SSLUtilBase.createSSLContext(SSLUtilBase.java:245)
> at
> org.apache.tomcat.util.net
> .AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:98)
> ... 20 more
> Caused by: java.net.URISyntaxException: Illegal character in opaque part at
> index 2: D:\_ssh\_.ICSI.Berkeley.EDU.key
> at java.net.URI$Parser.fail(Unknown Source)
> at java.net.URI$Parser.checkChars(Unknown Source)
> at java.net.URI$Parser.parse(Unknown Source)
> at java.net.URI.(Unknown Source)
> ... 29 more
> 10-Feb-2021 17:34:09.930 INFO [main]
> org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
> ["ajp-nio-127.0.0.1-8009"]
> 10-Feb-2021 17:34:09.930 INFO [main]
> org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a
> shared
> selector for servlet write/read
> 10-Feb-2021 17:34:09.930 INFO [main]
> org.apache.catalina.startup.Catalina.load Initialization processed in 1522
> ms
>

This happens when D:\_ssh\_.ICSI.Berkeley.EDU.key is not a file (or not
there) and it then tries as a URL. Is your keystore still there after your
update ?
There were recent changes, but there's no difference that I can see with
that location as input.

Rémy


>
> On Thu, Feb 11, 2021 at 12:17 AM Mark Thomas  wrote:
>
> > On 11/02/2021 02:06, Jim Weill wrote:
> > > I had 8.5.41 working and decided to do the upgrade to 8.5.63 today on
> > > Windows Server 2012r2.  I've had success with stopping the service,
> > > renaming the tomcat directory, putting the unzipped files of the new
> > > version in its place, and dropping in the server.xml, and web.xml files
> > to
> > > replace the default files.  As well, I copy over the webapps folder for
> > the
> > > site, then restart the service.  This process has worked many times
> > before.
> > >
> > > When I did this today, the service would not start for some reason.
> So I
> > > renamed the folders back to their original names, and then ran the
> > > uninstall from the add/remove programs.  I ran the install executable
> on
> > > 8.5.63, dropped in the webapps folder, and the server.xml and web.xml
> > files
> > > and 

Re: Are we able to deploy the same WAR to Tomcat 9 and 10?

2021-02-05 Thread Rémy Maucherat
On Fri, Feb 5, 2021 at 4:03 PM Mark Thomas  wrote:

> On 05/02/2021 14:45, Johan Compagner wrote:
> > Hi,
> >
> > I already now get the first support request that our application doesn't
> > run under Tomcat 10.
> >
> > So just want to get straight how this is going to work in the future.
> >
> > i see there is a migration tool, but that is for now quite useless for us
> > because we also need to support Tomcat 9 or 8..
> >
> > We are a tool/framework vendor we don't control what our users do, they
> can
> > still be on what ever servlet container they are on. (we only require the
> > websocket implementation so that makes certain stuff the minimum)
> >
> > Is there or will there be a way that we can drop in a jar that is just a
> > "redirect/wrapper"?
>
> At the moment, no.
>
> > And then I don't care too much about if it goes from javax.servlet to
> > jakarta.servlet or the other way around.. I just want to support both
> > deployments. That our customers can dump in the generated war by our
> > tooling in any tomcat version (8,9 or 10) and it just works.
>
> There has been talk of integrating the migration tool into Tomcat 10 so
> if you drop in a Java EE 8 app it automagically converts it to Jakarta
> EE 9 before starting it.
>

Yes, I was thinking about having a "ee8appBase" attribute on the host
(defaults to ee8webapps, or something). Then any artefact placed in that
folder gets processed (by the HostConfig) to the regular "appBase" using
the migration tool with a direct API call, equivalent to using the CLI.
Then it gets deployed as before from "appBase".

The lack of interest about this was quite impressive ;) And it's rather
gimmicky of course, it simply adds a possible deployment failure and
inefficiency, when the user should instead process the artifact ahead of
time.


>
> I'm guessing you'd prefer this to having to provide separate Java EE and
> Jakarta EE versions (even if all you had to do to create the Jakarta EE
> version was run it through the migration tool).
>

I'm a bit skeptical that it will be possible to avoid having native Jakarta
versions forever.


> On a related topic, it would be helpful to know if the migration tool
> successfully converts your app.
>

+1

Rémy

>
> Mark
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>


Re: SSL trouble in embeddedLand

2021-01-20 Thread Rémy Maucherat
On Tue, Jan 19, 2021 at 5:02 AM Rob Sargent  wrote:

>
> Stuck in my basement with no real domain I'm having trouble setting up
> SSL/TLS on an embedded tomcat instance. And I'm very lost, having tried
> more dead ends than I can remember.
>
> I used this to generate cert and key
> openssl req -out localhost.crt -key localhost.key \
> -newkey rsa:2048 -nodes -sha256 \
> -subj '/CN=localhost' -extensions EXT -config <( \
> printf "[dn]\nCN=localhost\n[req]\ndistinguished_name =
> dn\n[EXT]\nsubjectAltName=DNS:localhost\nkeyUsage=digitalSignature,
> nonRepudiation, keyEncipherment,
> dataEncipherment\nextendedKeyUsage=serverAuth")
>
> and try to apply those with
>
> -Djavax.net.ssl.trustStore=/tmp/key/localhost.crt
> -Djavax.net.ssl.trustStorePassword=changeit
> -Djavax.net.ssl.trustStoreType=PKCS12
> -Djavax.net.ssl.keyStore=/tmp/key/localhost.key
> -Djavax.net.ssl.keyStorePassword=changeit
> -Dhttps.protocols=TLSv1.2,TLSv1.3
> -Djdk.tls.client.protocols=TLSv1.2,TLSv1.3
>
> I get "toDerInputStream rejects tag type 45".
>
> That error rears its ugly head in several places such when adding those
> same two localhost files to a pkcs12 rendition of the OS's cacerts file
> using below. The crt goes in quietly. (I'm couldn't convince my self of
> the openssl equivalent of this command)
> root@gitanmax:/tmp/key# keytool -importkeystore -srckeystore
> localhost.key -srckeypass changeit -srcstoretype pkcs12 -destkeystore
> cacerts.pkcs12 -deststoretype pkcs12 -srcalias localhost -destalias
> tomcatlocal
> Importing keystore localhost.key to cacerts.pkcs12...
> Enter destination keystore password: changeit
> Enter source keystore password: changeit
> keytool error: java.io.IOException: toDerInputStream rejects tag type 45
>
> My actual first attempt was to just use keytool to generate a cert
> interactively, filling in my own bogus values but that hit the domain
> name mismatch gotcha.
>
> Then, there's confusion on the java side as well.
>
> My plan was to add a connector for my webapp
>
> embeddedTomcat = new Tomcat();
> logger.debug("handing in CATALINA_HOME as " +
> System.getenv("CATALINA_HOME"));
> embeddedTomcat.setPort(tomcatPort-1);
> embeddedTomcat.enableNaming();
> Connector defaultConnector = embeddedTomcat.getConnector(); // an init,
> really
> defaultConnector.setSecure(true); // force https?
>
> Service service = embeddedTomcat.getService();
> service.addConnector(addTLSConnector(tomcatPort));
>
> logger.info("tomcatd listening on port " + tomcatPort);
> String contextRootPath = System.getenv("CATALINA_HOME");
> logger.debug("contextRootPath is {}", contextRootPath);
> Context contextTomcat = embeddedTomcat.addContext("", new
> File(contextRootPath + "/sgs").getAbsolutePath());
> logger.debug("host: {}, general context basename {}",
> embeddedTomcat.getHost(), contextTomcat.getBaseName());
> //TODO: cut from here ...
> java.time.LocalDateTime bootDT = java.time.LocalDateTime.now();
> Tomcat.addServlet(contextTomcat, "monitor", new HttpServlet() {
> @Override
> protected void doGet(HttpServletRequest req, HttpServletResponse resp)
> throws ServletException, IOException {
> ServletOutputStream out = resp.getOutputStream();
> out.write("SGS Agent monitor: SGS_OK\n".getBytes());
> out.write(("Up since " + bootDT.toString()).getBytes());
> out.write(("\nTime now " +
> java.time.LocalDateTime.now().toString()).getBytes());
> out.flush();
> out.close();
> }
> });
> contextTomcat.addServletMappingDecoded("/monitor", "monitor");
> // My webapp
> Context sgsContext = embeddedTomcat.addWebapp("/sgs", contextRootPath +
> "/sgs"); // /sgs.war
> embeddedTomcat.start();
> embeddedTomcat.getServer().await();
>
> private Connector addTLSConnector(Connector connector, int tcport) {
> File keyFile = new File (System.getProperty("SGSSRVR_keystoreFile"));
> if (! keyFile.exists()) throw new RuntimeException("where's the
> keystore?");
> connector.setPort(tcport);
> connector.setSecure(true);
> connector.setScheme("https");
> connector.setProperty("protocol", "HTTP/1.1");
> connector.setProperty("sslProtocol", "TLS");
> connector.setProperty("address","127.0.0.1");
> connector.setProperty("clientAuth", "false");
> connector.setProperty("maxThreads", "200");
> connector.setProperty("protocol",
> "org.apache.coyote.http11.Http11NioProtocol");
> connector.setProperty("SSLEnabled", "true");
> return connector;
> }
>
> private Connector addTLSConnector(int tcport) {
> Connector connector = new Connector();
> addTLSConnector(connector, tcport);
> return connector;
> }
>
> That "monitor" works, on a separate port, on either http or https as per
> defaultConnector.setSecure(isHttps);
> but possibly because I've diddle with chrome's security settings.
>
> I have a similar "monitor" as a servlet which I use as my
> is-anybody-home call to start my analyses. And of course can also use
> it via the browser at /seg/webmonitor.
>
> All these behave when SSL is not in play, but I'm forced, very much
> against my will, to use SSL so here 

Re: Examples failing

2020-12-04 Thread Rémy Maucherat
On Fri, Dec 4, 2020 at 2:05 PM Ibone Gonzalez Mauraza 
wrote:

> Hi,
>
> The examples of async of the newest versions are failing.
>

This is fixed already:
https://github.com/apache/tomcat/commit/0b5d5c91bafd3f61938917490ab531474dbb778a

Rémy


> When I try to run the test, I got the following error:
>
> tomcat_1  | 04-Dec-2020 12:36:55.183 WARNING [http-nio-8080-exec-6]
> org.apache.catalina.connector.Request.startAsync Unable to start async
> because the following classes in the processing chain do not support async
> [org.apache.catalina.filters.HttpHeaderSecurityFilter]
> tomcat_1  | java.lang.IllegalStateException: A filter or servlet of the
> current chain does not support asynchronous operations.
> tomcat_1  | at
> org.apache.catalina.connector.Request.startAsync(Request.java:1696)
> tomcat_1  | at
> org.apache.catalina.connector.Request.startAsync(Request.java:1689)
> tomcat_1  | at
> org.apache.catalina.connector.RequestFacade.startAsync(RequestFacade.java:1031)
> tomcat_1  | at async.Async0.service(Async0.java:48)
> tomcat_1  | at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
> tomcat_1  | at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
> tomcat_1  | at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> tomcat_1  | at
> org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
> tomcat_1  | at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> tomcat_1  | at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> tomcat_1  | at
> org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:126)
> tomcat_1  | at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> tomcat_1  | at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> tomcat_1  | at
> org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:109)
> tomcat_1  | at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> tomcat_1  | at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> tomcat_1  | at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
> tomcat_1  | at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
> tomcat_1  | at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:544)
> tomcat_1  | at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
> tomcat_1  | at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
> tomcat_1  | at
> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:690)
> tomcat_1  | at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
> tomcat_1  | at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
> tomcat_1  | at
> org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:616)
> tomcat_1  | at
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
> tomcat_1  | at
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:831)
> tomcat_1  | at org.apache.tomcat.util.net
> .NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1634)
> tomcat_1  | at org.apache.tomcat.util.net
> .SocketProcessorBase.run(SocketProcessorBase.java:49)
> tomcat_1  | at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> tomcat_1  | at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> tomcat_1  | at
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> tomcat_1  | at java.base/java.lang.Thread.run(Thread.java:834)
>
> Regards,
> Ibone.
>
>


Re: Custom protocol implementation not found

2020-10-14 Thread Rémy Maucherat
On Wed, Oct 14, 2020 at 11:38 AM Maarten van den Broek <
mbr...@messagedesign.nl> wrote:

> I use tomcat 9.0.33 with windows10 home and amazon corretto jdk1.8.0_212.
>
> Below a snapshot of two different Connector definitions in server.xml
>
>   maxThreads="150" SSLEnabled="true" scheme="https"
> secure="true"
> protocol="nl.messagedesign.tomcatlib.EncryptedPassword_Http11Nio2Protocol"
> clientAuth="false" sslEnabledProtocols="TLSv1.2"
> minSpareThreads="5"
> enableLookups="true" disableUploadTimeout="true"
> keystoreFile="C:/Users/Maarten/Certificaten/gm_messagedesign_nl2020.jks"
> keystorePass="ZURV/6aoh/mLRxJGFhnvEpVZ7PoL72h3"
> />
>
>   disableUploadTimeout="true" enableLookups="true" maxThreads="150"
> minSpareThreads="5" port="443"
> protocol="nl.messagedesign.tomcatlib.EncryptedPassword_Http11Nio2Protocol"
> SSLEnabled="true" scheme="https" secure="true">
>  
>   certificateKeystoreFile="C:/Users/Maarten/Certificaten/gm_messagedesign_nl2020.jks"
>
> certificateKeystorePassword="ZURV/6aoh/mLRxJGFhnvEpVZ7PoL72h3"
> certificateKeystoreType="JKS"/>
>  
>  
>
> Using the first Connector everything is working fine. Debugging the
> setKeystorePass method of the class
> nl.messagedesign.tomcatlib.EncryptedPassword_Http11Nio2Protocol in the
> protocol attribute shows that the encrypted password gets decrypted.
>
> Using the second connector with the SSLHostConfig element instead of the
> deprecated attributes debugging shows that the setKeystorePass method is
> not called and I get errors for the incorrect password of the keystore.
>
> What am I doing wrong in migrating to the configuration with the
> SSLHostConfig element?
>
> Sincerely yours,Maarten van den Broek
>

If you simply want to obfuscate server.xml attributes, you should look into
the digester property sources instead of engaging in this sort of stuff.
One such property source out there:
https://github.com/web-servers/tomcat-vault

Note: This capability is not included directly into Tomcat itself because
it provides no actual extra security.

Rémy


Re: A lot of EOFs when using NIO2 with HTTP2

2020-10-06 Thread Rémy Maucherat
On Tue, Oct 6, 2020 at 8:31 AM Martin Grigorov  wrote:

> Hi,
>
> I face an issue with the NIO2 protocol.
> When I increase the request rate to more than 500 requests per second it
> starts failing with:
>
> 06-Oct-2020 09:18:19.964 FINE [https-jsse-nio2-8080-exec-6]
> org.apache.coyote.http2.Http2UpgradeHandler.init Connection [1], State
> [NEW]
> 06-Oct-2020 09:18:19.978 FINE [https-jsse-nio2-8080-exec-6]
> org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.failed
> Connection [1], Stream [0], Frame type [null], Error
> java.io.EOFException
> at
> org.apache.tomcat.util.net
> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1005)
>

Also, it is a real EOF as the error says, caused by the client
disconnecting for whatever reason (good or bad).

Rémy


Re: ByteBuffer pooling

2020-09-28 Thread Rémy Maucherat
On Mon, Sep 28, 2020 at 5:31 PM Martin Grigorov 
wrote:

> On Mon, Sep 28, 2020 at 6:11 PM Rémy Maucherat  wrote:
>
> > On Mon, Sep 28, 2020 at 4:34 PM Martin Grigorov 
> > wrote:
> >
> > > Hi,
> > >
> > > I've profiled the memory allocation during load testing HTTP2:
> > > https://pasteboard.co/Jtblqfl.png
> > >
> > > As you can see there are a lot of ByteBuffer.allocate(int) calls.
> > > org.apache.catalina.connector.Response#Response(int)
> > > org.apache.catalina.connector.Request#inputBuffer
> > > org.apache.coyote.http2.Http2AsyncParser#readFrame
> > > org.apache.coyote.http2.Stream.StreamOutputBuffer#buffer
> > >
> > > Has it been discussed in the past to use a pool of ByteBuffer
> instances ?
> > > Netty provides such for its abstraction ByteBuf:
> > > https://netty.io/wiki/using-as-a-generic-library.html
> > >
> > > Here is a simple implementation:
> > >
> > >
> >
> https://github.com/jhg023/Pbbl/blob/39c749b9e65f4f8a840a07812559cf8830bd5eae/src/main/java/com/github/pbbl/AbstractBufferPool.java#L44
> > > It's give() method could be optimized to not return the buffer to the
> > pool
> > > if the pool size is bigger than N, so it doesn't use huge list of
> buffers
> > > which are used just once.
> > >
> > > The big unknown to me is: where to return the buffers to the pool ?
> > > For HTTP2 maybe this could be done after writing the buffer to the
> > socket.
> > > If the Request/Response is recycled then their Input/OutputBuffer are
> > > reused and everything is OK. But if recycling is not in use then new
> > > allocations are done for each new request.
> > >
> > > I see that org.apache.tomcat.util.net.WriteBuffer does some pooling
> > > already.
> > >
> > > What do you think?
> > >
> >
> > I removed a lot of pooling in the connector with zero impact, so I don't
> > see the point.
> >
>
> ConcurrentDateFormat pools SimpleDateFormat instances and it does make a
> difference:
>
> https://medium.com/@martin.grigorov/compare-performance-of-javas-simpledateformat-against-datetimeformatter-31be58cadf1d
>  !
> I'd give it a try for the HTTP2 Stream related allocations and let you know
> how it goes!
>

This is not a simple object, this involves parsing and computations. A BB
however is a very simple object, even if this somehow shows up in a
microbenchmark the actual impact will be negative (more memory is retained,
the life of the GC is harder, and most importantly contention on the
cache). I mean these were the optimizations I was doing in the 2000s, but
now it seems to work as the JDK people say.
There's an exception for direct BBs in theory, but in practice retaining
the memory is an even bigger issue.

Rémy


Re: ByteBuffer pooling

2020-09-28 Thread Rémy Maucherat
On Mon, Sep 28, 2020 at 4:34 PM Martin Grigorov 
wrote:

> Hi,
>
> I've profiled the memory allocation during load testing HTTP2:
> https://pasteboard.co/Jtblqfl.png
>
> As you can see there are a lot of ByteBuffer.allocate(int) calls.
> org.apache.catalina.connector.Response#Response(int)
> org.apache.catalina.connector.Request#inputBuffer
> org.apache.coyote.http2.Http2AsyncParser#readFrame
> org.apache.coyote.http2.Stream.StreamOutputBuffer#buffer
>
> Has it been discussed in the past to use a pool of ByteBuffer instances ?
> Netty provides such for its abstraction ByteBuf:
> https://netty.io/wiki/using-as-a-generic-library.html
>
> Here is a simple implementation:
>
> https://github.com/jhg023/Pbbl/blob/39c749b9e65f4f8a840a07812559cf8830bd5eae/src/main/java/com/github/pbbl/AbstractBufferPool.java#L44
> It's give() method could be optimized to not return the buffer to the pool
> if the pool size is bigger than N, so it doesn't use huge list of buffers
> which are used just once.
>
> The big unknown to me is: where to return the buffers to the pool ?
> For HTTP2 maybe this could be done after writing the buffer to the socket.
> If the Request/Response is recycled then their Input/OutputBuffer are
> reused and everything is OK. But if recycling is not in use then new
> allocations are done for each new request.
>
> I see that org.apache.tomcat.util.net.WriteBuffer does some pooling
> already.
>
> What do you think?
>

I removed a lot of pooling in the connector with zero impact, so I don't
see the point.

Rémy


>
> Martin
>


Re: Low throughput with HTTP2

2020-09-21 Thread Rémy Maucherat
On Mon, Sep 21, 2020 at 2:49 PM Martin Grigorov 
wrote:

> Hi Remy,
>
> On Mon, Sep 21, 2020 at 2:56 PM Rémy Maucherat  wrote:
>
> 
>
>
> > > 2020-09-21 14:25:04.850 DEBUG 232086 --- [https-jsse-nio-18080-exec-8]
> > > o.a.coyote.http11.Http11NioProtocol  : Found processor [null] for
> > > socket [org.apache.tomcat.util.net.SecureNioChannel@2b435926
> > > :java.nio.channels.SocketChannel[connected
> > > local=/127.0.0.1:18080 remote=/127.0.0.1:42229]]
> > > 2020-09-21 14:25:04.850 DEBUG 232086 --- [https-jsse-nio-18080-exec-6]
> > > o.a.coyote.http2.Http2UpgradeHandler : Connection [2]
> > >
> > > java.io.IOException: Unable to unwrap data, invalid status [CLOSED]
> > > at org.apache.tomcat.util.net
> > > .SecureNioChannel.read(SecureNioChannel.java:766)
> > > at org.apache.tomcat.util.net
> > >
> >
> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468)
> > >
> >
> > When I tested HTTP/2, I used h2c (unencrypted) to get a better idea at
> > what's going on. Otherwise you don't really know what proportion of TLS
> or
> > HTTP/2 impacts performance more [and then you have to test more things
> like
> > OpenSSL, be careful that the ciphers used are the same and equally well
> > optimized in each impl, etc].
> >
> > Then once you feel happy about h2c, you can see how things go with TLS.
> >
>
> Chris also suggested that TLS might be the problem for the low throughput
> but I get good throughput for both HTTP (15-16 K) and HTTPS (12-13 K
> reqs/s). Only when HTTP2 is enabled it drops. The reason is more clear now
> - it is due to the extra RST packets being sent.
>
> Vegeta does support h2c! So I can give it a try!
>
> How one configures Tomcat to support h2c ? Just remove  ?
> I've just tried this but I am not sure how to verify that it works.
> The browsers do not support h2c. curl and httpie neither.
> Quick search suggested me https://github.com/fstab/h2c "h2c connect
> localhost:18080" failed with "Failed to connect to localhost:18080: tls:
> oversized record received with length 20527"
>

Yes, it depends on the client. h2load works just fine without TLS for
example, and it's actually the reason why I added h2c support in Tomcat.

As for the configuration, it's a normal non TLS HTTP/1.1 config with the
UpgradeProtocol element added.

Rémy


>
>
>
> >
> > Rémy
> >
> >
> >
> >
>


Re: Low throughput with HTTP2

2020-09-21 Thread Rémy Maucherat
On Mon, Sep 21, 2020 at 1:36 PM Martin Grigorov 
wrote:

> On Mon, Sep 21, 2020 at 12:08 PM Martin Grigorov 
> wrote:
>
> >
> >
> > On Mon, Sep 21, 2020 at 11:23 AM Mark Thomas  wrote:
> >
> >> On 21/09/2020 08:18, Martin Grigorov wrote:
> >> > On Fri, Sep 18, 2020 at 6:16 PM Mark Thomas  wrote:
> >> >
> >> >> On 18/09/2020 14:07, Martin Grigorov wrote:
> >> >>
> >> >> 
> >> >>
> >> >>> What is the difference
> >> >>> between org.apache.coyote.http2.StreamStateMachine.State#CLOSED_RX
> >> >>> and org.apache.coyote.http2.StreamStateMachine.State#CLOSED_TX ?
> >> >>
> >> >> Compare the parameters used to construct the enums.
> >> >>
> >> >>> I read some parts of https://tools.ietf.org/html/rfc7540 but I
> don't
> >> see
> >> >>> anything related to two types of CLOSED state.
> >> >>
> >> >> Section 5.1. Definition of the closed state (page 20) explains the
> >> >> difference between the two.
> >> >>
> >> >
> >> > Still I do not understand what RX and TX stand for. But this is not
> >> > important.
> >>
> >> TX - transmit
> >>
> >> RX - receive
> >>
> >
> > Thanks!
> >
> >
> >>
> >>
> >> > The following patch fixes the issue for me/Vegeta:
> >> >
> >> > @@ -1570,12 +1571,15 @@ class Http2UpgradeHandler extends
> AbstractStream
> >> > implements InternalHttpUpgradeH
> >> >
> >> >  @Override
> >> >  public void reset(int streamId, long errorCode) throws
> >> Http2Exception
> >> >  {
> >> > -Stream stream = getStream(streamId, true);
> >> > -boolean active = stream.isActive();
> >> > -stream.checkState(FrameType.RST);
> >> > -stream.receiveReset(errorCode);
> >> > -if (active) {
> >> > -activeRemoteStreamCount.decrementAndGet();
> >> > +boolean unknownIsError = Http2Error.CANCEL.getCode() !=
> >> errorCode;
> >> > +Stream stream = getStream(streamId, unknownIsError);
> >> > +if (stream != null) {
> >> > +boolean active = stream.isActive();
> >> > +stream.checkState(FrameType.RST);
> >> > +stream.receiveReset(errorCode);
> >> > +if (active) {
> >> > +activeRemoteStreamCount.decrementAndGet();
> >> > +}
> >> >  }
> >> >  }
> >> >
> >> > I.e. do not throw an error if the remote peer is trying to cancel an
> >> > already closed stream.
> >>
> >> RFC 7540 allows the connection to be closed with a protocol error if the
> >> reset is received "a significant time after sending END_STREAM".
> >>
> >> Tomcat retains state for maxConcurrentStreams + 10% with closed streams
> >> being pruned (oldest first) every 10 new streams.
> >>
> >
> > The spec talks about time but Tomcat uses the number of streams.
> > I understand that under normal usage this could be enough "time" but
> under
> > heavy load this number is reached quickly and the time is short.
> >
> > I've tried with these values for
> > maxConcurrentStreams/maxConcurrentStreamExecution: 8 (number of CPU
> cores),
> > 16, 100 (the default), 1024 and 8196. The throughput is around 10K only
> > until 100. After that it drops (as you said earlier).
> >
> >
> >>
> >> It isn't clear at this point why the client is sending the RST_STREAM.
> >>
> >
> > I don't know either. But Vegeta is using the Golang standard libraries,
> so
> > any Golang application should behave the same way.
> > Also as I investigated last week - Golang/Node.js/Rust/Netty HTTP2
> servers
> > do not complain about it.
> >
> >
> >> This normally indicates an error condition. From the description of the
> >> load test I don't see why the client would be sending a RST_STREAM.
> >>
> >> If the RST_STREAM is valid and it isn't received a significant time
> >> after the END_STREAM then Tomcat needs to handle this.
> >>
> >> If the RST_STREAM is valid but received a significant amount of time
> >> after the END_STREAM that would be a client error would be a client
> error.
> >>
> >> Of course, significant isn't defined in the specification.
> >>
> >> Any special handling for RST_STREAM also needs to be applied to
> >> WINDOW_UPDATE. It also needs to differentiate between a frame received
> >> for a past closed stream and a frame received for an stream that never
> >> existed.
> >>
> >> I think the key question here is why is the client sending the
> >> RST_STREAM in the first place. Is Tomcat doing something it shouldn't /
> >> not doing something it should to cause this? If so, we need to fix the
> >> root cause rather than tackle the symptom.
> >>
> >> There are a couple of options for debugging this to see why the client
> >> is sending the RST_STREAM:
> >> - Enable debug logging for HTTP/2 in Tomcat. This is very verbose and
> >>   if the root cause is threading related typically changes the timing
> >>   enough to mask the issue.
> >>
> >
> > Here are the logs of 20 seconds load:
> >
> >
> https://gist.githubusercontent.com/martin-g/37dacc12149d5cecbfd4e16dc2248cfd/raw/76693cd1dd1f37ba126edb84ebd7631802aa91a6/tomcat-http2.log
> >
>

Re: Low throughput with HTTP2

2020-09-15 Thread Rémy Maucherat
On Tue, Sep 15, 2020 at 5:08 PM Martin Grigorov 
wrote:

> Hi Mark,
>
> On Tue, Sep 15, 2020 at 3:34 PM Mark Thomas  wrote:
>
> > On 15/09/2020 12:46, Martin Grigorov wrote:
> > > On Tue, Sep 15, 2020 at 2:37 PM Martin Grigorov 
> > > wrote:
> > >
> > >> Hi,
> > >>
> > >> I am running some load tests on Tomcat and I've noticed that when
> HTTP2
> > is
> > >> enabled the throughput drops considerably.
> > >>
> > >> Here are the steps to reproduce:
> > >>
> > >> 1) Enable HTTP2, e.g. by commenting out this connector:
> > >>
> > >>
> >
> https://github.com/apache/tomcat/blob/d381d87005fa89d1f19d9091c0954f317c135d9d/conf/server.xml#L103-L112
> > >>
> > >> 2) Download Vegeta load tool from:
> > >> https://github.com/tsenart/vegeta/releases/
> > >>
> > >> 3) Run the load tests:
> > >>
> > >> 3.1) HTTP/1.1
> > >> echo -e '{"method": "GET", "url": "http://localhost:8080/examples/"}'
> |
> > >> vegeta attack -format=json  -rate=0 -max-workers=1000 -duration=10s |
> > >> vegeta encode > /tmp/http1.json; and vegeta report -type=json
> > >> /tmp/http1.json | jq .
> > >>
> > >> 3.2) HTTP2
> > >> echo -e '{"method": "GET", "url": "https://localhost:8443/examples/
> "}'
> > |
> > >> vegeta attack -format=json -http2 -rate=0 -max-workers=1000 -insecure
> > >> -duration=10s | vegeta encode > /tmp/http2.json; and vegeta report
> > >> -type=json /tmp/http2.json | jq .
> > >>
> > >> As explained at https://github.com/tsenart/vegeta#-rate -rate=0 means
> > >> that Vegeta will try to send as many requests as possible with the
> > >> configured number of workers.
> > >> I use '-insecure' because I use self-signed certificate.
> > >>
> > >> On my machine I get around 14-15K reqs/sec for HTTP1.1 with only
> > responses
> > >> with code=200 .
> > >> But for HTTP2 Tomcat starts returning such kind of errors:
> > >>
> > >>  "errors": [
> > >> "Get \"https://localhost:8443/examples/\": http2: server sent
> > GOAWAY
> > >> and closed the connection; LastStreamID=9259, ErrCode=PROTOCOL_ERROR,
> > >> debug=\"Stream [9,151] has been closed for some time\"",
> > >> "http2: server sent GOAWAY and closed the connection;
> > >> LastStreamID=9259, ErrCode=PROTOCOL_ERROR, debug=\"Stream [9,151] has
> > been
> > >> closed for some time\"",
> > >> "Get \"https://localhost:8443/examples/\": http2: server sent
> > GOAWAY
> > >> and closed the connection; LastStreamID=239, ErrCode=PROTOCOL_ERROR,
> > >> debug=\"Stream [49] has been closed for some time\""
> > >>   ]
> > >>
> > >> when I ask for more than 2000 reqs/sec, i.e. -rate=2000/1s
> >
> > That indicates that the client has sent a frame associated with a stream
> > that the server closed previously and that that stream has been removed
> > from the Map of known streams to make room for new ones. See
> > Http2UpgardeHandler.pruneClosedStreams()
> >
> > It looks like the client is making assumptions about server behaviour
> > that go beyond the requirements of RFC 7540, section 5.3.4.
> >
>
> This is possible!
> I've just tested with two more HTTP2 impls:
>
> 1) Node.js
>
> http2-server.js
> ===
> const http2 = require('http2');
> const fs = require('fs');
>
> const server = http2.createSecureServer({
> key: fs.readFileSync('/path/to/server.key'),
> cert: fs.readFileSync('/path/to/server.crt')
> });
> server.on('error', (err) => console.error(err));
>
> server.on('stream', (stream, headers) => {
> // stream is a Duplex
> stream.respond({
> 'content-type': 'text/plain; charset=utf-8',
> ':status': 200
> });
> stream.end('Hello world!');
> });
>
> server.listen(18080);
> ===
>
> run with: node http2-server.js
>
> Runs fine with -rate=0 and gives around 8K reqs/sec
>
> 2) Rust
>
> Cargo.toml
> ===
> [package]
> name = "my-http2-server"
> version = "0.0.1"
> publish = false
> authors = ["Martin Grigorov "]
> license = "MIT/Apache-2.0"
> description = "Load test HTTP/2 "
> repository = "https://github.com/martin-g/http2-server-rust;
> keywords = ["http2"]
> edition = "2018"
>
> [dependencies]
> actix-web = { version = "3", features = ["openssl"] }
> openssl = { version = "0.10", features = ["v110"] }
> ===
>
> src/main.rs
> ===
> use actix_web::{web, App, HttpRequest, HttpServer, Responder};
> use openssl::ssl::{SslAcceptor, SslFiletype, SslMethod};
>
> async fn index(_req: HttpRequest) -> impl Responder {
> "Hello world!"
> }
>
> #[actix_web::main]
> async fn main() -> std::io::Result<()> {
> let mut builder =
> SslAcceptor::mozilla_intermediate(SslMethod::tls()).unwrap();
> builder
> .set_private_key_file("/path/to/server.key", SslFiletype::PEM)
> .unwrap();
> builder.set_certificate_chain_file("/path/to/server.crt").unwrap();
>
> HttpServer::new(|| App::new().route("/", web::get().to(index)))
> 

Re: rewrite stopped working

2020-07-21 Thread Rémy Maucherat
On Tue, Jul 21, 2020 at 10:30 PM Kenneth Noel  wrote:

> I recently upgrade from 9.0.12 to 9.0.35.  I had a working rewrite.config
> file and all the rewrite rules in it were working.  After the upgrade only
> the first rewrite rule worked.
>
> I tried several things to get them working and nother helps.  Anyone have
> any idea??
>

Yes, upgrade to 9.0.36, this is a known regression in the 9.0.35 release.

Rémy


>
> i.e. rewrite.conf only the first cmsadmin rewrite works.  If I make one of
> the others first then that works.
>
> RewriteRule  /cmsadmin /dev/f?p=100
> RewriteRule /orgchart2 /dev/f?p=450
> RewriteRule /phone /dev/f?p=772
>
> Ken
>
>
> --
> Kenneth Noel
> Boston College
> DBA
> (617) 552-8511
>


Re: How to encrypt db password in tomcat context.xml

2020-06-29 Thread Rémy Maucherat
On Mon, Jun 29, 2020 at 8:03 AM Carsten Klein  wrote:

> Hi Jürgen and Olaf,
>
> I can really understand Jürgen's intentions. The core problem is not
> security but administrators and so called security panels in
> "professional" (non-open source related) companies. I really know this
> from my own experiences. Maybe that's a German problem, since Germans
> are said to be over-correct? Sometimes, that turns into paranoia...
> (I'm from Germany, so I know this circumstances quite well, sounds
> like Jürgen is German as well...)
>
> True is, that there are administrators, which have very little
> knowledge of software in general and security. Those tend to stick to
> their personal categorical rules, which often are far off from what is
> considered sensible by real IT and software professionals.
> Furthermore, there are "security" panels, working out policies for a
> company. These often only consist of people with very little *real* IT
> an security knowledge.
>
> The (sad) point is, that the policies passed by such a council are
> actually valid and no one ever again asks whether these make sense or
> are *correct* from a security professional's point of view. Changing
> user passwords on a regular basis (e.g. after 90 days) still today is
> one prominent example of that.
>
> So, in order to make Tomcat fit into such "professional" company
> environments easily (w/o requiring people to implement their own data
> source class), it may be a good idea to add such a "encrypted
> password" feature to Tomcat. Consider this as pure "syntactic sugar"
> and keep in mind, that it's NOT a security feature (need to emphasis
> that in the docs).
>
> My idea is, that *all* passwords read by Tomcat MAY be
> encrypted/obfuscated with a small number of algorithms. The algorithm
> applied to the password could be prefixed like Jürgen suggested:
>
> password="+duk6<7v@LD#"(plain, no encryption)
> password="base64:K2R1azY8N3ZATEQj" (base64 obfuscation)
> password="3des:hkph5ewjEwv70CvTB16v/w=="   (3DES with hard-coded key,
> expressed as base64 string)
>
> The decoding algorithm could be implemented in a common util method
> String decodePassword(String password) in Tomcat, so it could easily
> be called from all those places at which Tomcat actually reads a
> password.
>
> Also, a small separate tool should be provided, which encodes such
> passwords (like htpasswd does for httpd). However, it should be
> sufficient to just print the encoded password to standard out. Then,
> the user is responsible for copy and pasting it into the config file.
>
> I offer my help for writing such an enhancement, since I believe that
> it's a way to make Tomcat more out-of-the-box usable in such
> "professional" company's environments (for some people it may be the
> only way).
>
> Again, I know this is NOT a security feature as it adds no extra
> security to Tomcat. However, I may make some administrators and CEOs
> happy, that are solely guided by questionable policies they don't
> understand.
>
> Some ideas on that?
>

The Tomcat committers' decision has always been to block inclusion of such
a feature, for the reasons explained in the wiki page here
https://cwiki.apache.org/confluence/display/TOMCAT/Password
As a result, your proposal will not be considered.

If you want a ready to use tool, go here:
https://github.com/web-servers/tomcat-vault

Rémy


>
> Carsten
>
>
> On 28.06.2020 23:49, Jürgen Weber wrote:
> > I'd just put some nice password as byte[] into Tomcat's source code
> > and provide a way to have passwords in the configs encrypted with that
> > nice password.
> >
> >> Use properties replacement so that in the xml config you have
> >> ${db.password} and in conf/catalina.properties you put the password
> >> there.
> >
> > So one could have samething like db.pass=3des: in
> > catalina.properties
> >
> > Greetings, Juergen
> >
> > Am So., 28. Juni 2020 um 21:19 Uhr schrieb Olaf Kock  >:
> >>
> >>
> >> On 28.06.20 19:50, Jürgen Weber wrote:
> >>  I would like to know how to encrypt and decrypt the database
> >> password in
> >>  context.xml when the application is running which also allow
> >> me to change
> >>  the db password for the purpose of security.
> >> >> https://cwiki.apache.org/confluence/display/TOMCAT/Password
> >> > Well, I know a chief open source app server that has the password to
> >> > decrypt all passwords buried in its open source, and I know auditors
> >> > who are good if root cannot read passwords at first sight. The
> >> > reasoning behind that is that running java -jar someappserverlib.jar
> >> > -decrypt is a deliberate act that a god guy root does not do. So a
> >> > hidden password is a step better, even if not in the cryptographic
> >> > sense.
> >>
> >> Hi Jürgen,
> >>
> >> I don't get your point here. Are you arguing that the linked wiki
> >> article is incorrect, insufficient or invalid?
> >>
> >> Because I believe that the article documents how to 

Re: Should Tomcat 10 enable response compression by default?

2020-06-10 Thread Rémy Maucherat
On Tue, Jun 9, 2020 at 10:20 PM Mark Thomas  wrote:

> Hi all,
>
> An enhancement has been opened to enable response compression by default:
> https://bz.apache.org/bugzilla/show_bug.cgi?id=64431
>
> In short, the proposal is to change the default for the Connector's
> compression attribute from "off" to "on".
>
> This would be for Tomcat 10 onwards only.
>
> The following would be unchanged:
> - compressibleMimeType
> - compressionMinSize
> - noCompressionStrongETag
>
> It would be helpful to know what the range of views of the user
> community are on this proposal.
>
> So, thoughts?
>

I would prefer to leave it as it is now, but if you want to publicize it
more, maybe compression="off" could be added to the default server.xml so
that people know it's there.

Rémy


Re: HTTP2 keepAliveTimeout changed from Infinite to 20 seconds (default)

2020-06-03 Thread Rémy Maucherat
Hi,

On Wed, Jun 3, 2020 at 4:35 PM Arshiya Shariff
 wrote:

> Hi Mark,
> The keepAliveTimeout for HTTP2 , in the later tomcat versions is set to a
> default value of 20 seconds and can be overridden .
> Is there a reason that the keepAliveTimeout has been changed from infinite
> to 20 seconds (default) . Were there any issues around this ?
>
> This is just for an information.
>

Please don't email developers directly.

Are you referring to this commit ?
https://github.com/apache/tomcat/commit/c16d9d810a1f64cd768ff33058936cf8907e3117

If so, then this is in 9.0.16+. HTTP/2 is a newer part of Tomcat and a
large component, so you'll have to be prepared to update relatively often
to pick up necessary fixes.

Rémy


>
> Thanks and Regards
> Arshiya Shariff
>


Re: Tomcat SSL Connector - Http11NioProtocol - javax.crypto.ShortBufferException on second request

2020-04-13 Thread Rémy Maucherat
On Mon, Apr 13, 2020 at 7:07 PM Mark Thomas  wrote:

> On 13/04/2020 11:39, Parigino Andrea Aiello wrote:
> > Hello!
> > i'm having a problem with Tomcat 8.5.51 hosting my Spring Boot 2
> > application (with 2-way SSL);
>
> The first thing to do is to update to 8.5.54 and re-test.
>

Also test OpenSSL and Java 11 [if Java 8 was used here], to see what
happens.

Rémy


>
> Mark
>
> > In short is an application with both server and client SOAP interfaces
> > (first called as server, then it act as client).
> > The problem:
> > on first request (sent by SoapUI or other external client) everything
> works
> > fine, no exception;
> > on the second one i got this exception:
> >
> >1. 13-Apr-2020 11:45:09.757 INFO [https-jsse-nio-234-exec-1]
> >org.apache.coyote.http11.Http11Processor.service Error parsing HTTP
> request
> >header
> > Note: further occurrences of HTTP request parsing errors will be
> logged
> >at DEBUG level.
> >java.lang.ArrayIndexOutOfBoundsException:
> >javax.crypto.ShortBufferException: Need at least 336 bytes of space in
> >output buffer
> >at
> sun.security.ssl.CipherBox.decrypt(CipherBox.java:591)
> >at
> >sun.security.ssl.EngineInputRecord.decrypt(EngineInputRecord.java:200)
> >at
> >sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:963)
> >at
> >sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:896)
> >at
> >sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766)
> >at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
> >at
> >org.apache.tomcat.util.net
> .SecureNioChannel.read(SecureNioChannel.java:607)
> >at
> >org.apache.tomcat.util.net
> .NioEndpoint$NioSocketWrapper.fillReadBuffer(NioEndpoint.java:1289)
> >at
> >org.apache.tomcat.util.net
> .NioEndpoint$NioSocketWrapper.read(NioEndpoint.java:1225)
> >at
> >
> org.apache.coyote.http11.Http11InputBuffer.fill(Http11InputBuffer.java:737)
> >at
> >
> org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:368)
> >at
> >
> org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:502)
> >at
> >
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
> >at
> >
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:818)
> >at
> >org.apache.tomcat.util.net
> .NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1623)
> >at
> >org.apache.tomcat.util.net
> .SocketProcessorBase.run(SocketProcessorBase.java:49)
> >at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> >at
> >
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> >at java.lang.Thread.run(Thread.java:748)
> >
> > To be noted that on the second request i do not get even a single line of
> > log from my application, looks like the request doesn't even reach my
> code.
> > here is the Connector config:
> >
> >  >
> sslImplementationName="org.apache.tomcat.util.net.jsse.JSSEImplementation"
> > port="234" maxThreads="200" scheme="https" secure="true"
> > SSLEnabled="true" clientAuth="true" sslProtocol="TLS"
> > keyAlias="agweb2ca"
> > keystoreFile="conf\cert\keystore_s.jks" keystorePass="*"
> > truststoreFile="conf\cert\truststore_s.jks" truststorePass="**"
> > />
> >
> > i've also tried all the buffer parameter for the connector (
> >
> https://tomcat.apache.org/tomcat-8.5-doc/config/ajp.html#NIO_specific_configuration
> > --> setting them to -1/illimited) but seem to not work.
> >
> > Another thing to say is that between the acting as SOAP Server and acting
> > SOAP Client there are some http (not https) calls to another system.
> >
> > Any help would be really appreciated.
> > Thanks a lot!
> >
> > Andrea
> >
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>


Re: [ANN] Apache Tomcat 9.0.34 available

2020-04-09 Thread Rémy Maucherat
On Thu, Apr 9, 2020 at 6:36 PM Mark Thomas  wrote:

> On 09/04/2020 17:27, Mark Thomas wrote:
> > On 09/04/2020 17:16, Bill Stewart wrote:
> >> On Thu, Apr 9, 2020 at 9:08 AM Mark Thomas  wrote:
> >>
> >>> The Apache Tomcat team announces the immediate availability of Apache
> >>> Tomcat 9.0.34.
> >>
> >> Thank you.
> >>
> >> FYI: The file tcnative-1.dll is missing from the
> >> 'apache-tomcat-9.0.34-windows-x64.zip' distribution.
> >
> > That is strange. The release build process is scripted so that sort of
> > thing shouldn't happen. Let me take a look.
>
> Bill,
>
> I'm not seeing this.
>
> The file "tcnative-1.dll" is present in the "bin" directory in the zip
> file generated by the build process (this happens on a dedicated VM and
> I keep the previous build directory until I start the next point release
> for that version).
>

It looks fine for me as well.

Rémy


>
> The file "tcnative-1.dll" is present in the "bin" directory in the zip
> file I downloaded from tomcat.apache.org
>
> I also confirmed that the SHA512 hashes were the same for both zip files
> confirming that nothing changed between the version I built and the file
> that is available on the download page.
>
> Where did you get that zip file from?
> What is the SHA512 hash of the zip file you downloaded?
> Did you download over HTTPS?
>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>


Re: javax.servlet.ServletContainerInitializer defined in jar not loading on Tomcat 7.0.100

2020-02-26 Thread Rémy Maucherat
On Wed, Feb 26, 2020 at 12:36 PM Rémy Maucherat  wrote:

> On Wed, Feb 26, 2020 at 9:57 AM Mark Thomas  wrote:
>
>> On 26/02/2020 01:35, SS jong wrote:
>>
>> 
>>
>> > Has anyone encountered this issue on Tomcat 7.0.100 and if this
>> behavior change is intentional?
>>
>> Possibly the same root cause as
>>
>> https://bz.apache.org/bugzilla/show_bug.cgi?id=64153
>>
>
> I can confirm the issue, it is limited to 7.0 as 8.5 is fine.
>
> Indeed removing the leading "/" in the findResources call works around the
> problem.
>

The fix for the issue will be in 7.0.101.

Rémy


Re: javax.servlet.ServletContainerInitializer defined in jar not loading on Tomcat 7.0.100

2020-02-26 Thread Rémy Maucherat
On Wed, Feb 26, 2020 at 9:57 AM Mark Thomas  wrote:

> On 26/02/2020 01:35, SS jong wrote:
>
> 
>
> > Has anyone encountered this issue on Tomcat 7.0.100 and if this behavior
> change is intentional?
>
> Possibly the same root cause as
>
> https://bz.apache.org/bugzilla/show_bug.cgi?id=64153
>

I can confirm the issue, it is limited to 7.0 as 8.5 is fine.

Indeed removing the leading "/" in the findResources call works around the
problem.

Rémy


Re: Enhancement: New option 'persistAuthentication' for session manager

2020-02-18 Thread Rémy Maucherat
On Tue, Feb 18, 2020 at 9:19 AM Carsten Klein  wrote:

> Rémy,
>
>
> > Can you describe an actual use case for this ? Without clustering, I
> don't
> > understand why the auth persistence is useful at all [when using
> > clustering, the delta manager persists that auth information]. To be
> > honest, that's also the case for session persistence itself, which does
> not
> > provide a QoS level that is still relevant today. But the feature is
> > already there [maybe it could be considered for removal in 10 actually
> ...].
>
> Why is session persistence across restarts not still relevant today?
> Tomcat, as well as likely any other servlet container, stops/starts a
> context when actually a reload (aka reconfigure) action would be
> required (I guess, that's specified servlet specs, right?).
>
> A context needs to be completely restarted for any configuration change
> (oh, we need to add a new allowed host to the CORS filter, etc.).
> Restarting, and so loosing all runtime data (the session), makes this
> simple stop/start approach (from a programmer's point of view) difficult
> to use in a production environment. I often find myself doing such
> configuration changes in the evening in order not to drop logged on
> users (that's when I'd better enjoy some freetime *g*).
>
> That's why I believe, that session persistence across restarts is
> required still today. Of course, there may be frameworks, including
> clustering, that can do a lot for you. However, I'm talking about
> Tomcat, not other frameworks. Why shouldn't Tomcat on its own provide a
> production-system-ready "reconfigure" action? It did in the past, why
> dropping it? That feature seems not too buggy...
>
> Also, for me, setting up a Tomcat cluster with at least two machines,
> only for getting session persistence during reconfiguring the nodes
> seems a bit overdrawn. In our setups we typically have one server
> machine, serving between 10 and 50 users at a time. I just can't tell my
> customer to double the hardware just in order to keep users logged on
> while we are working on the configuration.
>

Maybe you think it's "overdrawn", but right now you have very poor QoS.
What happens while the webapp/server is away ? Do you think the user will
not see anything bad ? [aka: please don't do anything until the restart is
done ...] Plus what happens if you have 1s of sessions ? It gets slow
and some have complained about it in the past. Etc etc.
That's not how things should happen today.


>
> It may be the case, that today, many setups use WAR files, including
> code and configuration, which are deployed at regular intervals or at
> scheduled deploy times. However, that's not how we work. We like to be
> able to react contemporary when a problem in a production system comes
> up (our customers like that, too).
>
> So, that's my plea for session persistence across restarts, as
> implemented in recent versions. Nearly the same use case applies to
> persisting authentication information as well.
>

The feature is there right now, so you're good ...


>
> In our setups, we use the session mainly for authentication. The problem
> that comes with not persisting authentication may not be obvious with
> simple (browser driven) BASIC authentication, since the Browser sends
> those Base64 encoded credentials with each request. Using session based
> authentication (send credentials once and rely that your session remains
> authenticated) is more modern and likely a bit more secure, right?
>
> However, in that scenario, it's really bad, if your authentication has
> gone after the context has been restarted. In that case, you may be
> forwarded to the login page (when using FORM authentication). In our
> case, our AJAX-style JavaScript client just logs you off and all your
> work is gone (that's like if your browser crashes occasionally and has
> no "restore previously opened tabs" feature...). In our case,
> authentication information is the only valuable data in the session (we
> don't save any attributes). So, isn't persisting authentication
> information (for the sake of completeness) a feature of its own?
>
> In other words, why not persisting authentication information? That data
> is the only data, that currently is NOT persisted. Why not? Comments in
> the code only mention, that authType and principal are transient. But
> why? If there is a session persistence feature in Tomcat, shouldn't it
> persist the session as a whole (at least, as much as possible)? Since
> recent versions of Tomcat's default GenericPrincipal do no longer save
> passwords, persisting authType and principal seems not being harmful to me.
>
> Again, session persistence across restarts is just a "workaround" for
> the missing "reconfigure" action. For me, it's the tribute for using the
> much simpler stop/start schema in servlet containers (in contrast to a
> real reconfigure action, which often tends to get quite complex).
> Typically, persisted sessions do not remain on 

Re: Enhancement: New option 'persistAuthentication' for session manager

2020-02-17 Thread Rémy Maucherat
On Mon, Feb 17, 2020 at 6:31 PM Carsten Klein  wrote:

> Hi there,
>
> finally, I got my first Tomcat enhancement ready. You can view its code
> at my Tomcat fork on GitHub:
>
>
> https://github.com/cklein05/tomcat/tree/session-manager-persist-authentication
>
> Before I'm opening an enhancement in Tomcat's Bugzilla, maybe, Mark and
> Christopher (or whoever else is interested), could you please have a
> quick look at the code?
>
> BTW, recent versions of Tomcat do no longer save and store passwords in
> GenericPrincipal. That makes using this enhancement even less critical.
>

Can you describe an actual use case for this ? Without clustering, I don't
understand why the auth persistence is useful at all [when using
clustering, the delta manager persists that auth information]. To be
honest, that's also the case for session persistence itself, which does not
provide a QoS level that is still relevant today. But the feature is
already there [maybe it could be considered for removal in 10 actually ...].

Rémy


>
>
> Some brief description of what I've done:
>
> 1. The new boolean option 'persistAuthentication' is implemented in
> ManagerBase and so can be configured for StandardManager or
> PersistentManager (defaults to false). Also added this to
> mbeans-descriptor.xml, so it's writable through JMX.
>
> 2. That option is set for any new StandardSession upon creation (in
> method ManagerBase.createSession(String)). Once a session got this
> option, it's not being changed during the session's lifetime.
>
> 3. In StandardSession, method doWriteObject writes authentication
> information (authType and principal) only, if 'persistAuthentication' is
> true, the session is authenticated (authType or principal != null) and
> principal is serializable.
>
> 4. The "is principal serializable" test is, by default, a deep
> (expensive?) check, which actually serializes the whole principal object
> to an ObjectOutputStream backed by a new NullInputStream, which just
> discards all bytes written. This check can be skipped by setting system
> property
> org.apache.catalina.session.StandardSession.PRINCIPAL_SERIALIZABLE_CHECK
> to false (defaults to true, however). If skipped, the principal is
> considered serializable if (principal instanceof Serializable). That's
> how the session's attribute values are checked for being serializable or
> not.
>
> However, that is odd, if such a serialized object has a child object
> which is not serializable. In that case, the ObjectOutputStream is left
> in an inconsistent state and no so called fatal exception code is
> written to the stream (that is, when reading such a stream, no
> WriteAbortedException is not thrown for such an error).
>
> 5. A Boolean object is used as a tag/marker that determines, whether
> authentication information is present id the stream or not. If none of
> the above conditions are met, both authType and principal are not
> serialized (than, only the initial Boolean false marker has been emitted
> to the stream).
>
> BTW, the Boolean false marker is not even required (if there is no
> authentication information in the stream) since the reading code works
> fine without any Boolean in the stream. So emitting Boolean false for
> signalling "no auth info" is actually optional (we could consider
> omitting it).
>
> 6. When sessions are loaded, ManagerBase provides a
> org.apache.catalina.util.CustomObjectInputStream instance to read
> sessions from. That instance is configured with the session's
> sessionAttributeValueClassNamePattern property. This essentially defines
> the classes, session attribute values may consist of. This pattern
> defaults to "java\\.lang\\.(?:Boolean|Integer|Long|Number|String)", so
> only attributes with these simple types can be loaded from a session.
>
> That filter pattern is only in effect, if a security manager is active,
> or if a pattern has been configured for the manager (e.g. in context.xml).
>
> Currently, however, all session data (including its base properties like
> creationTime, isNew, isValid etc) is loaded with that filter mechanism
> in place. Since those so called 'scalar instance properties' actually
> only consist of those simple types, that was not a problem.
>
> However, loading the serialized principal from the object stream is now
> subject to that filter mechanism (BTW, HA's DeltaManager and
> DeltaSession just do not utilize the CustomObjectInputStream).
>
> Since, as the name implies, the sessionAttributeValueClassNamePattern
> applies to attribute values only, I decided to give the
> CustomObjectInputStream a boolean 'filterActive' property, which is set
> to true, just before StandardSession starts deserializing attributes.
> The initial value of 'filterActive' can be specified though the
> constructor, to which both StandardManager and StoreBase pass false
> (actually, only StandardManager and PersistenManager (through StoreBase)
> do use the CustomObjectInputStream class).
>
> 7. Conclusion
>
> - Tomcat 

Re: [ANN] Apache Tomcat 9.0.31 available

2020-02-13 Thread Rémy Maucherat
On Thu, Feb 13, 2020 at 1:04 PM Olivier Jaquemet <
olivier.jaque...@jalios.com> wrote:

>
> On 13/02/2020 12:41, Mark Thomas wrote:
> > On 13/02/2020 09:57, Olivier Jaquemet wrote:
> >> I understand the need to introduce a "secured by default" AJP
> >> configuration.
> >> However, I question one choice that was made for this change : the
> >> default behavior of the AJP connector to listen only on the loopback
> >> address.
> >>
> >> [...]
> > You can specify "0.0.0.0" (IPv4) or "::" (IPv6) to restore the behaviour
> > of listening on any address.
> >
> > Mark
>
> Thank you Mark. This should address this use case.
>
> Would you consider adding this binding information to the documentation
> (in the documentation of the address attribute) ?
>

I would say no. These special address values are not specific to Tomcat,
it's like documenting 127.0.0.1 is the IPv4 loopback address [I guess it's
more well known ? hopefully ?].
The address specified on the Connector element goes through
InetAddress.getByName so it's very flexible, it allows more than simple IP
addresses.

Rémy


>
>
> To sum up :
> When migrating to Tomcat 8.5.51, 9.0.31 (and probably the next 7.0.x as
> I saw you had also backported this change to the 7.0.x branch),
> if your server architecture is to expose tomcat with an AJP connector,
> to a remote distant front server, you can :
>
> either, *secure your installation* (obviously the recommended way on
> untrusted network) :
>
> - by specifying the valid IP address on which the connector must bind;
> This is done with address attribute of the AJP connector.
>
> - by specifying a shared secret between the front server and the
> connector ; This is done with the secret attribute of the AJP connector
>
>
> or else, if you want your server.xml to be agnostic to the running host
> and remote front server, change your configuration back to the previous
> behavior, *BUT ONLY IF AND ONLY IF you are on trusted network* :
>
> - by removing explicit bind of AJP connector, by specifying "0.0.0.0"
> (IPv4) or "::" (IPv6) in the address attribute of the AJP connector
>
> - by removing need for shared secret between front and tomcat ; This is
> done with the secretRequired="false" attribute of the AJP connector.
>
> Olivier
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>


Re: [ANN] Apache Tomcat 9.0.31 available

2020-02-13 Thread Rémy Maucherat
On Thu, Feb 13, 2020 at 9:33 AM Olivier Jaquemet <
olivier.jaque...@jalios.com> wrote:

> On 13/02/2020 01:02, Stefan Mayr wrote:
> > Hi,
> >
> >> - AJP defaults changed to listen the loopback address, require a secret
> >>and to be disabled in the sample server.xml
> > What was the motivation behind this breaking change to require a secret
> > or to explitly disable it? What makes an open AJP connector more unsafe
> > than an open HTTP connector?
> >
> > We have hundreds of Tomcats behind Apache httpd with mod_jk. My
> > interpretation is that upgrading Tomcat 8.5 or 9.0 will break that setup
> > until we disable the secret in all of them (or add a secret in mod_jk
> > and Tomcat).
> > I would understand that for a new major version 10.x but not in the
> > lifecycle of an existing major version.
>
> Hi,
>
> I second those questions.
>
> We also have many tomcat instances behind Apache HTTPD, most of them are
> not the same server.
> It is my understanding that the new default listening behavior on the
> loopback address would break our installation, as the AJP connector
> would no longer be reachable to our remote Apache HTTP server. It would
> requires that we update all our tomcat's server.xml configuration to
> explicitely listen to an additional address by specifying the "address"
> attribute of the AJP connector.
> Am I correct ? Why such a change ? Why no bugzilla issue for proper
> tracking and context ?
> What are your recommendations regarding AJP connector configuration ?
>

It is obviously best to keep default configurations as stable as possible.
But sometimes things have to change ... As a result, you'll indeed need to
adjust your server.xml according to your deployment and AJP usage.

The documentation for the new attributes and updated defaults is here:
http://tomcat.apache.org/tomcat-9.0-doc/config/ajp.html#Standard_Implementations

Rémy


Re: Tomcat 8.5.51 fails

2020-02-13 Thread Rémy Maucherat
On Thu, Feb 13, 2020 at 10:13 AM kohmoto  wrote:

> Hi,
>
> I have install Tomcat 8.5.51 today and found something wrong.
>
> I have been using tomcat for last 5 years and never met this
> kind of problem.
>
> It would be appreciated if I could be advised.
>

Ok, so ...


>  Caused by: java.lang.IllegalArgumentException: The
> AJP Connector is configured with secretRequired="true" but
> the secret attribute is either null or "". This combination
> is not valid.
>
> The error message gives the explanation. The AJP defaults changed so you
need to adjust your server.xml to that.

Rémy


Re: Some attribute lost after calling storeConfig

2020-02-12 Thread Rémy Maucherat
On Wed, Feb 12, 2020 at 5:17 PM Arnaud Yahoo 
wrote:

> Thanks for you reply, shall I fill a bug for this ?
>

No need. The fix will be in 9.0.32 and 8.5.52.

Rémy


>
> On 12/02/2020 16:43, Rémy Maucherat wrote:
> > On Wed, Feb 12, 2020 at 3:05 PM Arnaud Yahoo 
> > wrote:
> >
> >> Hello,
> >>
> >> Recently storeConfig has been fixed thanks to
> >>
> >>
> https://github.com/apache/tomcat/commit/010fdb7e458d9d8755e2b67203ac4757d78c2f64
> >>
> >> It is very interesting, because it allows to persist across restart
> >> configurations made to tomcat itself and deployed webapp through JMX.
> >>
> >> In my case after saving a configuration to my webapp I lost my
> >> cacheMaxSize customization.
> >>
> >> I had in my context xml file this:
> >>
> >> 
> >>
> >> and after calling storeConfig It has been changed to this
> >>
> >>  so
> >> cacheMaxSize has been reverted to default value.
> >>
> >> I understand it is because in
> >> /org/apache/catalina/storeconfig/server-registry.xml /
> >> /
> >>
> >> / >> //tag="Resources"//
> >> //standard="false"//
> >> //default="false"//
> >> //children="true"//
> >> //tagClass="org.apache.catalina.WebResourceRoot"//
> >>
> //storeFactoryClass="org.apache.catalina.storeconfig.WebResourceRootSF">//
> >> //allowLinking//
> >> //cachingAllowed//
> >> //cacheTtl//
> >> //cacheMaxSize//
> >> //cacheObjectMaxSize//
> >> //cached//
> >> //caseSensitive//
> >> //domain//
> >> // /
> >>
> >> cacheMaxSize is declared as Transient so not saved in updated context
> >> xml file.
> >>
> >> Is it something expected ? or can it be considered as a bug ?
> >>
> > The old resources were configured on the Context so that's where this
> > transient attribute list comes from. However, that doesn't work like this
> > anymore with the new resources so the list must be empty except domain.
> >
> > Rémy
> >
> >> Thanks,
> >>
> >> Arnaud
> >>
> >>
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>


Re: Tomcat doesn't propogate Security Credentials during session failover

2020-02-12 Thread Rémy Maucherat
On Wed, Feb 12, 2020 at 4:55 PM Klein, Carsten  wrote:

> Hi there,
>
> actually, Tomcat just does not serialize authentication information,
> that is AuthType (BASIC, DIGEST etc.) and the Principal, during session
> serialization. That affects session persistence across restarts (no
> matter what manager is used) as well as session transfer between cluster
> nodes. As a result, persisted and reloaded sessions as well as sessions
> transferred between cluster nodes are no longer authenticated.
>
> I was pointing that out in this mailing list in late 2019 (Topic "Tomcat
> 7.x.x, 8.x.x, 8.5.x and 9.x.x: Session serialization w/o authentication
> related information").
>
> To overcome this, I have a simple solution in mind. It's only about 10
> new lines of code. Basically, it's a new boolean property
> 'persistAuthentication' for the manager. Mark Thomas agreed with
> optionally saving authentication information along with the persisted
> session (only, if the new option is true). He encouraged me to write an
> enhancement/patch and provide it as a Pull Request.
>
> The only problem for me is lack of time. Although the code itself is
> quite simple, the things making me holding back are the Git stuff,
> making Tomcat build in my Eclipse etc...
>

Please look at DeltaSession serialization.

The use case for persistence of the principal across Tomcat restarts is
useless however, so -1.

Rémy


>
> Carsten
>
> > -BEGIN PGP SIGNED MESSAGE-
> > Hash: SHA256
> >
> > Jon,
> >
> > On 2/11/20 9:33 PM, Jonathan S. Fisher wrote:
> >> Apologies, I'm not seeing how this helps, I don't see where
> >> authentication information is transmitted
> >
> > No, seriously, what session manager are you using?
> >
> > - -chris
> >
> >> On Tue, Feb 11, 2020 at 5:39 PM Christopher Schultz <
> >> ch...@christopherschultz.net> wrote:
> >>
> >> Jon,
> >>
> >> On 2/11/20 5:36 PM, Jonathan S. Fisher wrote:
> >> What do you mean by logged out If it's one from Redisson,
> >> then you should look at their code and not
> > Tomcat's code.
> >
> > So you have two tomcat nodes: A & B, clustered in any
> > fashion (forget I mentioned redisson) of your choosing; let's
> > say they're clustered using the built in tcp point-to-point
> > replication.
> >> The choice of session manager is ... pretty critical, here. So
> >> which session manager are you using/
> >>
> > Have 5 people logged into an application on the first server
> > using standard JavaEE APIs (HttpServletrequest.login) Now
> > turn off server A. Your load balancer starts sending traffic
> > to server B. Their sessions will be there, BUT they will be
> > logged out; one has to call HttpServletRequest.login() again.
> > Upon login, Tomcat destroys the previous session (as it
> > should), nullifying any benefit for clustering the
> > application in the first place.
> >> Tomcat does not destroy sessions when authenticating.
> >>
> > In the two links I provided, the StandardSession object goes
> > to great length to ensure that the security principal is not
> > serialized into the session
> >>
> >> True.
> >>
> > and therefore [not] replicated in the cluster.
> >>
> >> False.
> >>
> > Why is that? Why not serialize the security credential so the
> > user can bounce between servers?
> >>
> >> Authentication information is transmitted in a different way.
> >>
> >> I would really encourage you to look at the code for DeltaManager,
> >> which is the session manager typically used for clustering in
> >> Tomcat. If you are not using the DeltaManager, then you need to
> >> look at the code being used for your actual SessionManager.
> >>
> >> -chris
> >>
> > On Tue, Feb 11, 2020 at 4:27 PM Christopher Schultz
> >  > > wrote:
> >
> > Jon,
> >
> > On 2/11/20 2:35 PM, exabrial wrote:
> >> https://stackoverflow.com/questions/59833043/tomcat-logs-user-out-
> > dur
> >>
> >>
> > i
> >
> >>
> >> ng-session-failover-event-and-restarts
> >  > dur
> >>
> >
> > ing-session-failover-event-and-restarts
> >>  > ing-session-failover-event-and-restarts>
> >
> >
> >
> >
> >
> >
> >
> >>
> > We've implemented session replication using Redisson, but we
> > noticed
> >> that if we intentionally fail a node, the user's sessions
> >> do get replicated, but they're logged out when they're
> >> restored on the new server.
> >
> > What exactly do you mean when you say "logged-out"?
> >
> >> Is there a way to make this work properly so the user
> >> doesn't get logged out during a failover event?
> >
> >> Most /More importantly, is there a technical or security
> >> reason for this?
> >
> > FYI the servlet specification 

Re: Some attribute lost after calling storeConfig

2020-02-12 Thread Rémy Maucherat
On Wed, Feb 12, 2020 at 3:05 PM Arnaud Yahoo 
wrote:

> Hello,
>
> Recently storeConfig has been fixed thanks to
>
> https://github.com/apache/tomcat/commit/010fdb7e458d9d8755e2b67203ac4757d78c2f64
>
> It is very interesting, because it allows to persist across restart
> configurations made to tomcat itself and deployed webapp through JMX.
>
> In my case after saving a configuration to my webapp I lost my
> cacheMaxSize customization.
>
> I had in my context xml file this:
>
> 
>
> and after calling storeConfig It has been changed to this
>
>  so
> cacheMaxSize has been reverted to default value.
>
> I understand it is because in
> /org/apache/catalina/storeconfig/server-registry.xml /
> /
>
> / //tag="Resources"//
> //standard="false"//
> //default="false"//
> //children="true"//
> //tagClass="org.apache.catalina.WebResourceRoot"//
> //storeFactoryClass="org.apache.catalina.storeconfig.WebResourceRootSF">//
> //allowLinking//
> //cachingAllowed//
> //cacheTtl//
> //cacheMaxSize//
> //cacheObjectMaxSize//
> //cached//
> //caseSensitive//
> //domain//
> // /
>
> cacheMaxSize is declared as Transient so not saved in updated context
> xml file.
>
> Is it something expected ? or can it be considered as a bug ?
>

The old resources were configured on the Context so that's where this
transient attribute list comes from. However, that doesn't work like this
anymore with the new resources so the list must be empty except domain.

Rémy

>
> Thanks,
>
> Arnaud
>
>


Re: RewriteValve does not work on HTTPS

2020-02-06 Thread Rémy Maucherat
On Thu, Feb 6, 2020 at 2:56 PM Hua Zhang  wrote:

> Thank you for the response. I am finally able to confirm the issue.
>
> When I put the following line in comment, everything works fine. I mean as
> expected.
>
> **
>
>
> If the above line UpgradeProtocol is activated, I observe now at least two
> weird situations.
>
> 1) As mentioned before, RewriteValve does not work as expected.
> 2) Besides it, I observed that serviet behaviors weird. According to the
> log file it seems that a servlet is sometimes called *twice by one
> request*.
>
> This is a snapshot of my log files. You can see that two https-443-exec are
> called almost at the same time.
>
> 06-Feb-2020 13:38:04.676 SEVERE *[https-openssl-apr-443-exec-9]*
> org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service()
> for servlet [uploadServlet] in context with path [] threw exception
>
> org.apache.commons.fileupload.MultipartStream$MalformedStreamException:
> Stream ended unexpectedly
> 06-Feb-2020 13:38:04.999 SEVERE *[https-openssl-apr-443-exec-2]*
> org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service()
> for servlet [uploadServlet] in context with path [] threw exception
>
> org.apache.commons.fileupload.MultipartStream$MalformedStreamException:
> Stream ended unexpectedly
>

Ok, so your problem seems to be about HTTP/2 rather than the rewrite valve:
HTTP/2 is usually not used without TLS and ALPN. You should continue to
post more details. Something you can try is use the NIO connector (with
OpenSSL or JSSE) rather than APR, it would be a more common configuration.

Rémy

>
>
> Best regards,
>
> Hua
>
>
> On Wed, Feb 5, 2020 at 2:15 PM Felix Schumacher <
> felix.schumac...@internetallee.de> wrote:
>
> > Am 04.02.2020 22:16, schrieb Hua Zhang:
> > > What I mean with word 'works' is: the RewriteRule has been executed.
> > >
> > > That is not the case by HTTPS. The rule has not been executed while the
> > > RewriteCond is fulfilled.
> >
> > Can you give us more information on your setup? Is there any
> > Proxy/Loadbalancer in front of your tomcat? If so, can you show us
> > details on that setup?
> > What is the value of the host request header in both cases?
> >
> > Felix
> > >
> > > Olaf Kock  于 2020年2月4日周二 下午9:06写道:
> > >
> > >>
> > >> On 04.02.20 20:31, Hua Zhang wrote:
> > >> > Best tomcat team,
> > >> >
> > >> > Hereby I have a question about an issue I found by using
> RewriteValve
> > >> > on tomcat 9.30
> > >> >
> > >> > The rewrite.config is very simple:
> > >> >
> > >> > /RewriteCond %{HTTP_HOST} =youkoop.com 
> > >> > RewriteRule ^.*$ https://www.youkoop.com [R=301,L]
> > >> > /
> > >> >
> > >> > All I want is just redirect a naked root domain to a www domain with
> > >> > HTTPS.
> > >> >
> > >> > The redirection works on HTTP but not HTTPS.
> > >> >
> > >> > http://youkoop.com => https://www.youkoop.com *works*
> > >> >
> > >> Note: Images don't get through in this mailing list. I can imagine
> > >> what
> > >> "works" means, but for your next example: Please elaborate what "does
> > >> not work" means.
> > >> >
> > >> > *https*://youkoop.com  =>
> > >> > https://www.youkoop.com *does not work*
> > >>
> > >> First thing to test: Does https://youkoop.com work without the
> > >> redirect,
> > >> then with the "wrong" host name? Otherwise it might be as simple as a
> > >> misconfigured TLS host that's never invoked because of a certificate
> > >> mismatch.
> > >>
> > >> Olaf
> > >>
> > >>
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
> >
> >
>


Re: lower/uppercase rewrite maps

2020-01-09 Thread Rémy Maucherat
On Thu, Jan 9, 2020 at 5:16 PM Felix Schumacher <
felix.schumac...@internetallee.de> wrote:

>
> Am 09.01.20 um 17:01 schrieb Chris Cheshire:
> > Looking through the documentation for the rewrite valve [1], I see
> > there is an example of how to write and use a rewrite map to convert a
> > value to upper case. This is the inverse of what I want (lowercase),
> > so great, easy enough to implement. This seems like something that
> > could be included by default but I couldn't see anything in
> > catalina.jar.
> >
> > Is this something that would be included if I create a patch for it,
> > and how would I go about it?
>
> I have opened a PR a bit ago (https://github.com/apache/tomcat/pull/221)
> but hadn't had time to investigate any further. Remy thought it would be
> a bit overengineered. Romain liked the idea of ServiceLoader but wanted
> to have it a bit more optimized (see
>
> https://lists.apache.org/thread.html/472e875a46e811370f7df8b7d4fae37170a31d73c3d814a48e4d565c%40%3Cdev.tomcat.apache.org%3E
> ).
>
> Would this be something you like to have?
>
> I think of committing the first part of the PR in any case, as I believe
> that the parsing of the parameters should be more in line with that of
> httpd.
>

I was planning to pull the non service loader parts of the PR as they are
likely useful utility classes, but I didn't (procrastination ... and bad
colds ...). I think I'm really against ServiceLoader configuration for
Tomcat at the moment, it seems even worse than system properties overall.

Rémy


>
> Felix
>
> >
> > Chris
> >
> > [1] http://tomcat.apache.org/tomcat-9.0-doc/rewrite.html
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
> >
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>


Re: performance issue with Tomcat 8.5.35 in org.apache.tomcat.util.net.NioBlockingSelector.write API

2020-01-07 Thread Rémy Maucherat
On Tue, Jan 7, 2020 at 6:33 AM Rathore, Rajendra  wrote:

> Hi Rémy/ Christopher,
>
> It will stuck there for 10-15 minutes, so it will take time to load simple
> Web UI, there is no WebSocket call. I am giving you one of the sample where
> it will take 90% time in write operation, sometime it will reach to 100%.
>
>
>  ||
>
> O-org.apache.coyote.ajp.AjpProcessor.writeData(AjpProcessor.java:1331)
> count=1669(%92.877)
>  ||
>   
> O-org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:385)
> count=1669(%92.877)
>  ||
> 
> O-org.apache.tomcat.util.net.SocketWrapperBase.writeBlocking(SocketWrapperBase.java:462)
> count=1669(%92.877)
>  ||
>   
> O-org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:726)
> count=1669(%92.877)
>  ||
> 
> O-org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1316)
> count=1669(%92.877)
>  ||
>   
> O-org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:157)
> count=1669(%92.877)
>  ||
> 
> O-org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:114)
> count=1667(%92.766)
>  ||
> |  
> O-org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.awaitWriteLatch(NioEndpoint.java:1160)
> count=1667(%92.766)
>  ||
> |
> O-org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.awaitLatch(NioEndpoint.java:1157)
> count=1667(%92.766)
>  ||
> |
> O-java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
> count=1667(%92.766)
>
>
It's a normal blocking write, and the await does not consume CPU (it sits
there however and a profiler will count that but it doesn't matter).
There's a problem only if things are blocked improperly, for example if the
client is correctly reading the data and/or there's no network backlog.
Also the timeout configured on the connector must be respected by the
operation.

Rémy


Re: performance issue with Tomcat 8.5.35 in org.apache.tomcat.util.net.NioBlockingSelector.write API

2020-01-06 Thread Rémy Maucherat
On Mon, Jan 6, 2020 at 1:27 PM Rathore, Rajendra  wrote:

> Hi Team,
>
>
>
> We are facing performance issue during 
> *org.apache.tomcat.util.net.NioBlockingSelector.write
> API call, *,most of our thread stuck and spending more time in that API,
> you can check below screenshot for more details.
>
>
>
>
>
>
>
>
>
> We debug the code and found that NioChannel.write method return 0(Zero)
> value, in that case our threads are stuck, Please  let us know why this
> happen.
>

As NIO is non blocking IO, 0 bytes written means there's a backlog and
output will block in NioBlockingSelector since it emulates blocking IO
there. It should not actually consume CPU, just wait until there's a poller
event indicating writing data may continue.

Rémy


>
>
> Please let me know if you need more details.
>
>
>
>
>
> Thanks and Regards,
>
> Rajendra Rathore
>
> 9922701491
>
>
>


Re: ServletRequest Obj Randomly not Processing x-www-form-urlencoded parms

2019-12-10 Thread Rémy Maucherat
On Tue, Dec 10, 2019 at 4:36 PM Mark Thomas  wrote:

> On 10/12/2019 14:27, Christopher Schultz wrote:
>
> 
>
> > Would using org.apache.catalina.connector.RECYCLE_FACADES=true have
> > made this problem go away? Or would the behavior have been the same,
> > just less dangerous?
>
> I think it would have triggered some NPEs in the background thread.
>
> > I'm wondering if Tomcat could or should have another safety feature to
> > help catch this sort of thing in development. In all my development
> > environments, I have the JDBC connection pool size set to a fixed
> > maximum of 1 connection. This means that any potential deadlocks in
> > the application due to sloppy connection-management will cause pretty
> > early because we'll get pool-fetch timeouts, missing-return-connection
> > errors, etc.
> >
> > Request object reuse has a measurable positive effect on performance
> > in production,
>
> It would be worth confirming that is still the case for the Request and
> Response objects. I suspect it is but it would be good to get some
> recent, hard data.
>

It would generate more GC per request and it is measurably slower on a best
case scenario with a static file (the worst would be starting to use
charsets and char input/output, or "large" Servlet buffers - the default is
8KB but people often increase it).

Quick test: add request = null; line 305 of CoyoteAdapter. -10% with ab -k
on tomcat.gif


>
> > but in development probably doesn't matter quite so
> > much. In the same way that WebappClassLoader becomes inert when the
> > application has been stopped, perhaps we could "shut-down" request /
> > response / session objects that have been loaned to a
> > request-processor thread.
> >
> > Something like this:
>
> I'm not sure why we need this over and above the RequestFacade object.
>

Rémy


Re: Error after upgrading to Tomcat 9.0.29

2019-11-25 Thread Rémy Maucherat
gt;  at
>
> org.apache.myfaces.view.facelets.tag.ui.CompositionHandler.apply(CompositionHandler.java:150)
>
>  [From here ... ]
>  at
>
> org.apache.myfaces.view.facelets.compiler.NamespaceHandler.apply(NamespaceHandler.java:59)
>  at
>
> org.apache.myfaces.view.facelets.compiler.EncodingHandler.apply(EncodingHandler.java:48)
>  at
>
> org.apache.myfaces.view.facelets.impl.DefaultFacelet.include(DefaultFacelet.java:519)
>  at
>
> org.apache.myfaces.view.facelets.impl.DefaultFacelet.include(DefaultFacelet.java:573)
>  at
>
> org.apache.myfaces.view.facelets.impl.DefaultFacelet.include(DefaultFacelet.java:551)
>  at
>
> org.apache.myfaces.view.facelets.impl.DefaultFaceletContext.includeFacelet(DefaultFaceletContext.java:240)
>  at
>
> org.apache.myfaces.view.facelets.tag.ui.CompositionHandler.apply(CompositionHandler.java:150)
>  [... to here repeats 140 times]
>
>  at
>
> org.apache.myfaces.view.facelets.compiler.NamespaceHandler.apply(NamespaceHandler.java:59)
>
> So it does not seem to be an endless recursion, as it seems to escape,
> but it seems to be to late and to deep in the stack. I'm really not sure
> what to make of this.
>

This (mistakenly it seems) got sent only to me.

Not sure about the issue at all or how it could be a regression caused by
Tomcat. A URL stream handler was (re)introduced but I fail to see the real
relevance.

Rémy


>
> Cheers,
>
> Juri
>
> On 11/25/19 2:52 PM, Rémy Maucherat wrote:
> > On Mon, Nov 25, 2019 at 2:15 PM Juri Berlanda <
> juri.berla...@tuwien.ac.at>
> > wrote:
> >
> >> Hi all,
> >>
> >> I just tried to deploy my WebApplication (OpenWebBeans, MyFaces) to
> >> Tomcat 9.0.29. While everything works fine in 9.0.27, on 9.0.29 as soon
> >> as I access any page I get:
> >>
> >> 25-Nov-2019 14:01:34.842 SEVERE [http-nio-8080-exec-4]
> >> org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service()
> >> for servlet [Faces Servlet] in context with path [/censored] threw
> >> exception [null] with root cause
> >>   java.lang.StackOverflowError
> >>
> >> Since it is a StackOverflow, I'm not posting the Stacktrace here.
> >>
> > Well ;) Please post some parts of the stack trace so that we know where
> and
> > what occurs.
> >
> >
> >> Has anybody had a different error? Is there a known fix or a workaround
> >> for this?
> >>
> >> I'm happy to help debugging and fixing the issue, if there is one in
> >> Tomcat. Just let me know how I can help.
> >>
> > Rémy
> >
>


Re: Error after upgrading to Tomcat 9.0.29

2019-11-25 Thread Rémy Maucherat
On Mon, Nov 25, 2019 at 2:15 PM Juri Berlanda 
wrote:

> Hi all,
>
> I just tried to deploy my WebApplication (OpenWebBeans, MyFaces) to
> Tomcat 9.0.29. While everything works fine in 9.0.27, on 9.0.29 as soon
> as I access any page I get:
>
> 25-Nov-2019 14:01:34.842 SEVERE [http-nio-8080-exec-4]
> org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service()
> for servlet [Faces Servlet] in context with path [/censored] threw
> exception [null] with root cause
>  java.lang.StackOverflowError
>
> Since it is a StackOverflow, I'm not posting the Stacktrace here.
>

Well ;) Please post some parts of the stack trace so that we know where and
what occurs.


>
> Has anybody had a different error? Is there a known fix or a workaround
> for this?
>
> I'm happy to help debugging and fixing the issue, if there is one in
> Tomcat. Just let me know how I can help.
>

Rémy


Re: SameSite cookies

2019-11-08 Thread Rémy Maucherat
On Fri, Nov 8, 2019 at 4:04 PM Christopher Schultz <
ch...@christopherschultz.net> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> All,
>
> I'm looking at using "samesite" cookies within my application. It
> looks as simple as setting the "sameSite" attribute appropriately on
> the CookieProcessor for the , which isn't there in a default
> configuration. So you just have to add it:
>
> 
>
>
>
> 
>
> Cool, now my JSESSIONID cookies are coming back with the SameSite=Lax
> parameter.
>
> But it also applies to all the other cookies my application creates.
> It looks like there is no way to set/reset this parameter on an
> individual-cookie basis. That would require a change to the Servlet
> API, right?
>
> I'm okay with SameSite being applied to ALL my cookies, but maybe not
> everybody is. Are there any workarounds for this?
>

The Servlet API has no remove cookie API. If you use a Valve, you can
remove cookies using Response.getCookies and then remove from the list.
But this is not really the problem here since the same site thing is added
when the cookie header is generated. You can extend the CookieGenerator to
add more flexibility for your use case maybe ?

Rémy


Re: Roadmap for Tomcat and Jakarta EE

2019-11-05 Thread Rémy Maucherat
On Tue, Nov 5, 2019 at 8:19 PM Campbell, Lance  wrote:

> Is there an article describing a general roadmap for Tomcat and Jakarta EE
> that I could read?
>

http://tomcat.apache.org/presentations.html
The latest "State of the Cat" presentation from ApacheCon EU looks into the
Jakarta stuff.

Rémy


>
>
> Thanks,
>
>
>
> *LANCE CAMPBELL *
>
> *Software Architect*
>
>
>
> Web Services 
>
> Public Affairs 
>
> Contact the Webtools Team 
>
> 217.333.0382
>
> la...@illinois.edu
>
>
>
>
>
> [image:
> /var/folders/wp/1f6l7hw95y718z976kgnl5f9kr5rtc/T/com.microsoft.Outlook/WebArchiveCopyPasteTempFiles/signature_logo.png]
> 
>
>
>
> *Under the Illinois Freedom of Information Act any written communication
> to or from university employees regarding university business is a public
> record and may be subject to public disclosure.*
>
>
>


Re: Jakarta EE 9

2019-10-28 Thread Rémy Maucherat
On Mon, Oct 28, 2019 at 3:53 PM i...@flyingfischer.ch 
wrote:

> Am 28.10.19 um 15:39 schrieb Mark Thomas
> >> If this is going to be disruptive and we cannot maintain compat, why
> >> not
> >> go the extra step and explicitly move Tomcat code to
> >> org.apache.tomcat.*
> >> for Tomcat 10? Git renames will work flawlessly for backports.
> > It will break things for users that code against those APIs. It is a
> small number but some do.
> >
> > Mark
> Do you have any evidence based number on this matter? May be in turns
> oout that this "small number" isn't as small as assumed...
>
> Of course, it's actually rather huge:
- Anyone using Tomcat embedded
- Anyone having written a custom Catalina component
- Anyone using a library with a Tomcat integration
- Anyone using a framework that embeds Tomcat

The rationale is that the jakarta change is already breaking compatibility,
and thus to go all in. But I don't share that opinion, it's simply easier
to explain to users that there's a single change: rename javax.servlet (and
the others) to jakarta.servlet. Plus there might be tools or deployment
tricks to do that magically in most cases, while there would likely be
nothing for Tomcat package changes.

Rémy


Re: Jakarta EE 9

2019-10-28 Thread Rémy Maucherat
On Mon, Oct 28, 2019 at 3:11 PM Michael Osipov  wrote:

> Am 2019-10-28 um 14:59 schrieb Mark Thomas:
> > On October 28, 2019 12:37:14 PM UTC, Johan Compagner <
> jcompag...@servoy.com> wrote:
> >> Hi
> >>
> >>
> >>
> >> On Mon, 28 Oct 2019 at 13:15, Mark Thomas  wrote:
> >>
> >>> Hi all,
> >>>
> >>> A frequent topic of discussion at ApacheCon EU was Jakarta EE 9. For
> >> those
> >>> of you who aren't familiar with Jakarta EE the key points are:
> >>>
> >>> - Oracle have donated Java EE to Eclipse
> >>> - Eclipse have released Jakarta EE 8 which is essentially identical
> >> to
> >>> Java EE 8
> >>> - Oracle have refused to allow changes to the APIs in the javax
> >> namespace
> >>> - The Jakarta EE community seem to be reaching consensus on releasing
> >>> Jakarta EE 9 which will rename all the Java packages from javax.* to
> >>> jakarta.*
> >>>
> >>>
> >>
> >> what does this rename really mean?
> >>
> >> import javax.servlet.http.HttpSession;
> >> import javax.websocket.Session;
> >>
> >> those are renamed?
> >> If that is yes that would mean pretty much everything will break?
> >
> > Correct. Hence the question on options for how we consider maintaining
> compatibility.
>
> If this is going to be disruptive and we cannot maintain compat, why not
> go the extra step and explicitly move Tomcat code to org.apache.tomcat.*
> for Tomcat 10? Git renames will work flawlessly for backports.
>
> I assume that most users never knew or don't understand where "catalina"
> comes from.
>

There's no reason to break APIs just for the sake of it (and the current
package structure looks relatively reasonable to me, catalina is the actual
container, coyote is the connector, jasper is Jasper). This is really like
"oh, this is going to be a pain for users, so while we're at it let's
destroy the rest".

At this point I think the differences with the current 9 should be as
limited as possible to avoid user problems. If more significant items get
included such as, let's say, APR removal or some similar feature that
wouldn't be backported to 9, then it should be limited to these items and
it justifies the branch being called "10" rather than "9.something".

Rémy


Re: Jakarta EE 9

2019-10-28 Thread Rémy Maucherat
On Mon, Oct 28, 2019 at 1:15 PM Mark Thomas  wrote:

> Hi all,
>
> A frequent topic of discussion at ApacheCon EU was Jakarta EE 9. For those
> of you who aren't familiar with Jakarta EE the key points are:
>
> - Oracle have donated Java EE to Eclipse
> - Eclipse have released Jakarta EE 8 which is essentially identical to
> Java EE 8
> - Oracle have refused to allow changes to the APIs in the javax namespace
> - The Jakarta EE community seem to be reaching consensus on releasing
> Jakarta EE 9 which will rename all the Java packages from javax.* to
> jakarta.*
> - Jakarta EE 9 will not contain any other API changes
> - It will be a requirement to maintain backwards comparability (maybe not
> 100% compatibility) with Jakarta EE 8
> - Jakarta EE 9 will be released in the next ~11 months
> - Jakarta EE 10 will then follow which is where new features will be
> introduced
>
> This raises various questions for the Tomcat community including:
>
> 1. Do we implement Jakarta EE 9?
>
> Assuming yes to q1.
>

+0.1

>
> 2. As Tomcat 10 or some other version?
>

I'm not convinced yet by using 10 for that since EE 9 is functionally
equivalent.


>
> 3. How do we manage development (in a branch, as master, ...)?
>

I would say master.

>
> 4. How do we implement backwards compatability? Runtime, deploy time,
> compilation time?
>

I have no idea how this would realistically be achieved at the moment. To
run on jakarta, the most problematic are libraries and frameworks and it
seems difficult to me to think it is really possible to hack our way in at
runtime. Deployment, I don't know, so you seem to believe BCEL is a
possibility.

>
> 5. At what point do we EOL Tomcat 7? (since Jakarta EE 9 is just a package
> renaming?)
>

We should mandate that, except for the jakarta package rename changes
and/or functionality, all commits to that master should be backported to
9.0 (no exceptions). Keeping 7 is possible, but it means more work.

>
> 6. Do we commit to supporting Tomcat 9 for an extended period? (to provide
> a version that supports java.* directly)
>

I would be in favor of that as a mirror of that new branch, since it would
be hard to guarantee zero issues for "legacy" javax.* webapps, which will
basically continue to exist forever.

>
> I'm posting this to users@ as how Tomcat users want to use and adopt
> Jakarta EE 9 is likely to be the biggest driving factor in choosing answers
> to these questions.
>
> We don't have to have all of the answers right now. I think we do need
> answers for 1 & 3 so we can start. A few options and 'wait and see' is
> likely good enough for the rest for now.
>
> I have my own views on the answers but I have tried not to express them
> here so we can get as wide a discussion as possible.
>
> Thoughts?
>

Rémy


Re: Jakarta EE 9

2019-10-28 Thread Rémy Maucherat
On Mon, Oct 28, 2019 at 2:22 PM Michael Osipov  wrote:

> Am 2019-10-28 um 13:15 schrieb Mark Thomas:
> > Hi all,
> >
> > A frequent topic of discussion at ApacheCon EU was Jakarta EE 9. For
> those of you who aren't familiar with Jakarta EE the key points are:
> >
> > - Oracle have donated Java EE to Eclipse
> > - Eclipse have released Jakarta EE 8 which is essentially identical to
> Java EE 8
> > - Oracle have refused to allow changes to the APIs in the javax namespace
> > - The Jakarta EE community seem to be reaching consensus on releasing
> Jakarta EE 9 which will rename all the Java packages from javax.* to
> jakarta.*
> > - Jakarta EE 9 will not contain any other API changes
> > - It will be a requirement to maintain backwards comparability (maybe
> not 100% compatibility) with Jakarta EE 8
> > - Jakarta EE 9 will be released in the next ~11 months
> > - Jakarta EE 10 will then follow which is where new features will be
> introduced
>
> W/o going into the questions. You may want to answer two simple
> questions to our users who solely use a servlet container like me:
>
> * Will my stuff break if I have written it against javax.servlet...?
> * Since there is no canonical body (? is it Eclipse Foundation now) for
> the specs, is there a possiblity that someone will fork the specs and
> create their own, e.g., commercial vendors or Undertow, Netty, Jetty,
> etc.?! (Mainly portability within the API)
>

Yes, the spec body for EE (starting with EE 9, which is functionally
equivalent to EE 8 except for the package rename to jakarta.*) is now the
Jakarta project at Eclipse. As with any standard, what matters is if people
implement it.

Rémy


>
> Michael
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>


Re: Jakarta EE 9

2019-10-28 Thread Rémy Maucherat
On Mon, Oct 28, 2019 at 1:46 PM Johan Compagner 
wrote:

> Hi
>
>
>
> On Mon, 28 Oct 2019 at 13:15, Mark Thomas  wrote:
>
> > Hi all,
> >
> > A frequent topic of discussion at ApacheCon EU was Jakarta EE 9. For
> those
> > of you who aren't familiar with Jakarta EE the key points are:
> >
> > - Oracle have donated Java EE to Eclipse
> > - Eclipse have released Jakarta EE 8 which is essentially identical to
> > Java EE 8
> > - Oracle have refused to allow changes to the APIs in the javax namespace
> > - The Jakarta EE community seem to be reaching consensus on releasing
> > Jakarta EE 9 which will rename all the Java packages from javax.* to
> > jakarta.*
> >
> >
>
> what does this rename really mean?
>
> import javax.servlet.http.HttpSession;
> import javax.websocket.Session;
>
> those are renamed?
> If that is yes that would mean pretty much everything will break?
>

https://eclipse-foundation.blog/2019/05/03/jakarta-ee-java-trademarks/
I thought everyone knew about this. We were supposed to have a session on
this rename at ApacheCon EU, but unfortunately it didn't happen.

Rémy


Re: Tomcat 8 epoll spinning issue (100% CPU)

2019-10-07 Thread Rémy Maucherat
On Mon, Oct 7, 2019 at 11:15 AM Emmanuel Lecharny 
wrote:

>
>
> On 2019/10/05 11:12:46, Rémy Maucherat  wrote:
> > On Fri, Oct 4, 2019 at 10:38 PM Emmanuel Lecharny 
> > wrote:
> >
> > > Hi remy,
> > >
> > > On 2019/10/04 15:37:36, Rémy Maucherat  wrote:
> > > > On Fri, Oct 4, 2019 at 3:40 PM Emmanuel Lecharny <
> elecha...@apache.org>
> > > > wrote:
> > > >
> > > > > Hi !
> > > > >
> > > > > I filled a ticket yesterday about a pb we face with many NIO
> framework,
> > > > > which I think could hit Tomcat too (see
> > > > > https://bz.apache.org/bugzilla/show_bug.cgi?id=63802). Actually, I
> > > think
> > > > > I'm facing this problem on a project I'm working on atm.
> > > > >
> > > > > Remy suggested we discuss it on this mailing list.
> > > > >
> > > > > Bottom line, what happens is that under some circumstances not well
> > > > > defined, the call to select() might end to an infinite loop eating
> all
> > > the
> > > > > CPU (select() returns 0, so select is immediately called again,
> and we
> > > > > loop).
> > > > >
> > > > > In various NIO framworks - and being a MINA committer, I have
> > > implemented
> > > > > the discussed workaround -, we are controlling this situation by
> > > breaking
> > > > > this infinite loop this way :
> > > > > - if the select() call returns 0
> > > > > - then if we have called select() more than N times in less than M
> ms
> > > > > (N=10, M=100 in MINA)
> > > > > - then we create a new Selector, register all the selectionKey that
> > > were
> > > > > registered on the broken selector, and ditch the old selector.
> > > > >
> > > > > This workaround does not cost a lot when the selector works as
> > > designed,
> > > > > as a select() call should never return 0.
> > > > >
> > > >
> > > > There's actually a very similar hack for APR that has been placed by
> > > myself
> > > > a long time ago [
> > > >
> > >
> https://github.com/apache/tomcat/blob/master/java/org/apache/tomcat/util/net/AprEndpoint.java#L1410
> > > > ], I don't even know if it's actually useful and it's certainly not
> > > > testable. Overall what it does is pretty terrible :(
> > > >
> > > > Personally I would like to know more about this "long lived bug
> either in
> > > > the JDK or even in Linux epoll implementation" like actual platform
> > > details
> > > > and JVM versions used since I've never heard about it in the first
> place.
> > >
> > > for the record, I had a discussion yesterday with one of my close
> friend
> > > and co-worker back in the 90's. He remember clearly, while working on
> the
> > > SUN TCP stack,  that such a problem occorded back then. Yes, 25 years
> > > ago... Ok, that was just for the fun, it's likely be perfectly
> unrelated ;-)
> > >
> > > At MINA, we were hit by this bug in 2009 (see
> > > https://issues.apache.org/jira/browse/DIRMINA-678), and it was linked
> to
> > > a bug reported on Jetty (
> > >
> http://jetty.4.x6.nabble.com/jira-Created-JETTY-937-SelectChannelConnector-100-CPU-usage-on-Linux-td36385.html
> ),
> > > itself related to some JDK bugs, supposedly fixed since then.
> > >
> > > I had a long conversation with Jean-François Arcand somewhere around
> this
> > > date, and he suggested we adopt the same workaround he applied to
> Grizzly.
> > > We also had a convo with Alan Bateman during a Java One in SF, but
> nothing
> > > specific resulted from this convo, except that AFAICR, he aknowledge
> there
> > > is an issue.
> > >
> > > So this problem started with JDK 6, but I can't guarantee it wasn't
> > > already present in JDK 5 or 4, on linux, and not on any other OS like
> > > windows or Mac OSX. It's not exactly fresh in my mind, because it was
> > > already 10 years ago.
> > >
> >
> > NIO support was added in Tomcat 6.0, supporting Java 5+, it wasn't very
> > good then. It's only with Java 6 that NIO started getting epoll support
> ant
> > I'm pretty sure the original issue did not actually survive. Despite the
> > popularity of the NIO connector this was not reported for Tomc

Re: Tomcat 8 epoll spinning issue (100% CPU)

2019-10-05 Thread Rémy Maucherat
On Fri, Oct 4, 2019 at 10:38 PM Emmanuel Lecharny 
wrote:

> Hi remy,
>
> On 2019/10/04 15:37:36, Rémy Maucherat  wrote:
> > On Fri, Oct 4, 2019 at 3:40 PM Emmanuel Lecharny 
> > wrote:
> >
> > > Hi !
> > >
> > > I filled a ticket yesterday about a pb we face with many NIO framework,
> > > which I think could hit Tomcat too (see
> > > https://bz.apache.org/bugzilla/show_bug.cgi?id=63802). Actually, I
> think
> > > I'm facing this problem on a project I'm working on atm.
> > >
> > > Remy suggested we discuss it on this mailing list.
> > >
> > > Bottom line, what happens is that under some circumstances not well
> > > defined, the call to select() might end to an infinite loop eating all
> the
> > > CPU (select() returns 0, so select is immediately called again, and we
> > > loop).
> > >
> > > In various NIO framworks - and being a MINA committer, I have
> implemented
> > > the discussed workaround -, we are controlling this situation by
> breaking
> > > this infinite loop this way :
> > > - if the select() call returns 0
> > > - then if we have called select() more than N times in less than M ms
> > > (N=10, M=100 in MINA)
> > > - then we create a new Selector, register all the selectionKey that
> were
> > > registered on the broken selector, and ditch the old selector.
> > >
> > > This workaround does not cost a lot when the selector works as
> designed,
> > > as a select() call should never return 0.
> > >
> >
> > There's actually a very similar hack for APR that has been placed by
> myself
> > a long time ago [
> >
> https://github.com/apache/tomcat/blob/master/java/org/apache/tomcat/util/net/AprEndpoint.java#L1410
> > ], I don't even know if it's actually useful and it's certainly not
> > testable. Overall what it does is pretty terrible :(
> >
> > Personally I would like to know more about this "long lived bug either in
> > the JDK or even in Linux epoll implementation" like actual platform
> details
> > and JVM versions used since I've never heard about it in the first place.
>
> for the record, I had a discussion yesterday with one of my close friend
> and co-worker back in the 90's. He remember clearly, while working on the
> SUN TCP stack,  that such a problem occorded back then. Yes, 25 years
> ago... Ok, that was just for the fun, it's likely be perfectly unrelated ;-)
>
> At MINA, we were hit by this bug in 2009 (see
> https://issues.apache.org/jira/browse/DIRMINA-678), and it was linked to
> a bug reported on Jetty (
> http://jetty.4.x6.nabble.com/jira-Created-JETTY-937-SelectChannelConnector-100-CPU-usage-on-Linux-td36385.html),
> itself related to some JDK bugs, supposedly fixed since then.
>
> I had a long conversation with Jean-François Arcand somewhere around this
> date, and he suggested we adopt the same workaround he applied to Grizzly.
> We also had a convo with Alan Bateman during a Java One in SF, but nothing
> specific resulted from this convo, except that AFAICR, he aknowledge there
> is an issue.
>
> So this problem started with JDK 6, but I can't guarantee it wasn't
> already present in JDK 5 or 4, on linux, and not on any other OS like
> windows or Mac OSX. It's not exactly fresh in my mind, because it was
> already 10 years ago.
>

NIO support was added in Tomcat 6.0, supporting Java 5+, it wasn't very
good then. It's only with Java 6 that NIO started getting epoll support ant
I'm pretty sure the original issue did not actually survive. Despite the
popularity of the NIO connector this was not reported for Tomcat, if we got
the report at the same time as the others it would be more logical so
something is different here.
https://github.com/netty/netty/issues/327 has details but I'm still not
very convinced. You should give details on your platform and everything
else since it's obvious at this point this is far less common with Tomcat.


>
> > Also I'd like to know since NIO2 doesn't expose its poller and almost
> > certainly doesn't have such a platform specific mysterious thing inside
> it
> > [we can check I guess].
>
> No idea, but I think NIO.2 has just added some coating over what was NIO.1
> (guts feeling here...).
>

It's a new codebase with very clean code.


>
> In the context of NIO, do you have evidence the
> > hack has been tested to work (besides avoiding the CPU loop) and allowed
> > the server to continue its regular operation without any impact ?
>
> Absolutely. We do log in MINA when a new selector is created, and we have
> had some issue related to a case where this piece of 

Re: Tomcat 8 epoll spinning issue (100% CPU)

2019-10-04 Thread Rémy Maucherat
On Fri, Oct 4, 2019 at 3:40 PM Emmanuel Lecharny 
wrote:

> Hi !
>
> I filled a ticket yesterday about a pb we face with many NIO framework,
> which I think could hit Tomcat too (see
> https://bz.apache.org/bugzilla/show_bug.cgi?id=63802). Actually, I think
> I'm facing this problem on a project I'm working on atm.
>
> Remy suggested we discuss it on this mailing list.
>
> Bottom line, what happens is that under some circumstances not well
> defined, the call to select() might end to an infinite loop eating all the
> CPU (select() returns 0, so select is immediately called again, and we
> loop).
>
> In various NIO framworks - and being a MINA committer, I have implemented
> the discussed workaround -, we are controlling this situation by breaking
> this infinite loop this way :
> - if the select() call returns 0
> - then if we have called select() more than N times in less than M ms
> (N=10, M=100 in MINA)
> - then we create a new Selector, register all the selectionKey that were
> registered on the broken selector, and ditch the old selector.
>
> This workaround does not cost a lot when the selector works as designed,
> as a select() call should never return 0.
>

There's actually a very similar hack for APR that has been placed by myself
a long time ago [
https://github.com/apache/tomcat/blob/master/java/org/apache/tomcat/util/net/AprEndpoint.java#L1410
], I don't even know if it's actually useful and it's certainly not
testable. Overall what it does is pretty terrible :(

Personally I would like to know more about this "long lived bug either in
the JDK or even in Linux epoll implementation" like actual platform details
and JVM versions used since I've never heard about it in the first place.
Also I'd like to know since NIO2 doesn't expose its poller and almost
certainly doesn't have such a platform specific mysterious thing inside it
[we can check I guess]. In the context of NIO, do you have evidence the
hack has been tested to work (besides avoiding the CPU loop) and allowed
the server to continue its regular operation without any impact ?

Rémy


>
> I suggest Tomcat add such a workaround in the various versions (8 and 9 at
> least).
>
> Emmanuel Lécharny
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>


Re: Tomcat 9.0.24/9.0.26 suspected memory leak

2019-09-30 Thread Rémy Maucherat
On Sat, Sep 28, 2019 at 9:05 PM Mark Thomas  wrote:

> On 27/09/2019 22:39, Chen Levy wrote:
> > -Original Message-
> > From: Mark Thomas 
> > Sent: Friday, September 27, 2019 15:34
> > To: users@tomcat.apache.org
> > Subject: Re: Tomcat 9.0.24/9.0.26 suspected memory leak
> >
> > On 27/09/2019 16:34, Chen Levy wrote:
> >> On 26/09/2019 18:22, Chen Levy wrote:
> >
> > 
> >
> >>> The HashMap referenced in the report appears to be "waitingProcessors"
> inside AbstractProtocol which contain 262K entries.
> >>
> >> OK. Those are asynchronous Servlets that are still in async mode.
> >
> > 
> >
> >> * I do not employ async servlets in my application
> >
> > OK. Do you use WebSocket? There is a code path to add Processors to the
> waitingProcessors Map for WebSocket as well.
> >
> > Mark
> >
> >
> > No, no WebSocket either; just plain old Servlets, Filters and the
> occasional JSP
>
> OK. That narrows down where/how this might be happening.
>
> What about if you disable HTTP/2. Do you still see the issue then?
>

I added debug code in
AbstractProtocol.ConnectionHandler.release(SocketWrapperBase) to check
if the processor considered was present in the waitingProcessors map. The
result is the following:
TEST-javax.servlet.http.TestHttpServletResponseSendError.NIO.txt:CHECK
PROCESSOR FAILED org.apache.coyote.http11.Http11Processor@77b16580
TEST-javax.servlet.http.TestHttpServletResponseSendError.NIO.txt:CHECK
PROCESSOR FAILED org.apache.coyote.http11.Http11Processor@1d902704
TEST-javax.servlet.http.TestHttpServletResponseSendError.NIO.txt:CHECK
PROCESSOR FAILED org.apache.coyote.http11.Http11Processor@610c4fc8
TEST-javax.servlet.http.TestHttpServletResponseSendError.NIO.txt:CHECK
PROCESSOR FAILED org.apache.coyote.http11.Http11Processor@1a3a3cb6
TEST-javax.servlet.http.TestHttpServletResponseSendError.NIO.txt:CHECK
PROCESSOR FAILED org.apache.coyote.http11.Http11Processor@336f552d
TEST-javax.servlet.http.TestHttpServletResponseSendError.NIO.txt:CHECK
PROCESSOR FAILED org.apache.coyote.http11.Http11Processor@3cd94f25
TEST-javax.servlet.http.TestHttpServletResponseSendError.NIO.txt:CHECK
PROCESSOR FAILED org.apache.coyote.http11.Http11Processor@66e24762
TEST-javax.servlet.http.TestHttpServletResponseSendError.NIO.txt:CHECK
PROCESSOR FAILED org.apache.coyote.http11.Http11Processor@7c7a1c3c
TEST-org.apache.coyote.http11.TestHttp11Processor.NIO.txt:CHECK PROCESSOR
FAILED org.apache.coyote.http11.Http11Processor@55a44822
TEST-org.apache.coyote.http11.upgrade.TestUpgradeInternalHandler.NIO.txt:CHECK
PROCESSOR FAILED
org.apache.coyote.http11.upgrade.UpgradeProcessorInternal@6e55ff60
TEST-org.apache.coyote.http11.upgrade.TestUpgrade.NIO.txt:CHECK PROCESSOR
FAILED org.apache.coyote.http11.upgrade.UpgradeProcessorExternal@37d98b7f
TEST-org.apache.tomcat.websocket.server.TestShutdown.NIO.txt:CHECK
PROCESSOR FAILED
org.apache.coyote.http11.upgrade.UpgradeProcessorInternal@6be9bd85
TEST-org.apache.tomcat.websocket.TestWsRemoteEndpoint.NIO.txt:CHECK
PROCESSOR FAILED
org.apache.coyote.http11.upgrade.UpgradeProcessorInternal@3bd4e02f
TEST-org.apache.tomcat.websocket.TestWsRemoteEndpoint.NIO.txt:CHECK
PROCESSOR FAILED
org.apache.coyote.http11.upgrade.UpgradeProcessorInternal@4bb23a77
TEST-org.apache.tomcat.websocket.TestWsRemoteEndpoint.NIO.txt:CHECK
PROCESSOR FAILED
org.apache.coyote.http11.upgrade.UpgradeProcessorInternal@32e20d65
TEST-org.apache.tomcat.websocket.TestWsRemoteEndpoint.NIO.txt:CHECK
PROCESSOR FAILED
org.apache.coyote.http11.upgrade.UpgradeProcessorInternal@16abf52f

All instances of not removed processors are either from async or upgraded
processors (the internal kind), as expected. I have verified the processor
instances above are never removed so it might be more robust to simply call
proto.removeWaitingProcessor(processor); in
AbstractProtocol.ConnectionHandler.release(SocketWrapperBase) (after all
the socket is closed and done after that point). There could be a more fine
grained solution of course.

However, this does not match the leak scenario described by the user, this
doesn't happen without async or websockets being used.

Rémy


Re: SSL issue : java.security.KeyStoreException: Cannot store non-PrivateKeys

2019-09-27 Thread Rémy Maucherat
On Fri, Sep 27, 2019 at 9:40 AM Mark Thomas  wrote:

> >  >   certificateFile="key_store/ssl_certificate.p7b"
> >   certificateAlias="bla"
> >   keystoreFile="/key_store/blabla.jks" type="RSA"
> >   keystoreType="JKS"
> >   keyChainFile="key_store/linux_apex_inter_x509.cer"
> >   keystorePassword="
>
> We need to exactly how each of the following files were created and/or
> exactly what is in each file:
>
> - ssl_certificate.p7b
> - blabla.jks
> - linux_apex_inter_x509.cer
>
> It might be as simple as you need to import the p7b file into the
> keystore and remove the certificateFile setting. But that is just a wild
> guess without knowing what is in those files.
>

I'm a bit lost here.

Normally certificateFile and keystoreFile should be "exclusive" (if
keystoreFile is set, then certificateFile will be ignored - it seems it
could be nice to add a warning ...), and I don't know about a keyChainFile
attribute either (I verified I get a proper "WARNING [main]
org.apache.catalina.startup.SetAllPropertiesRule.begin
[SetAllPropertiesRule]{Server/Service/Connector/SSLHostConfig/Certificate}
Setting property 'keyChainFile' to 'foobar' did not find a matching
property." in my logs).

So the config should be looked at again first, I think only keystoreFile
will be used and it will be the cause of the error.

Since you made plenty of special cases fixes since 8.5.32 for this, Venkat
should probably first test again with 8.5.46 (IMO).

Rémy


> Mark
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>


Re: ApacheCon NA Tomcat track videos

2019-09-19 Thread Rémy Maucherat
On Thu, Sep 19, 2019 at 10:03 AM Woonsan Ko  wrote:

> Hi Rémy,
>
> I just watched the first one: ApacheCon NA 2019: State of the Cat, New !
> The sound quality is not bad! Feel like being there.
> Thank you so much for uploading all!
>

Cool, thanks :) As I said this was improvised once I saw there was no
equipment in the room (the only item besides the projector was a voice
recorder left on the table - I was seriously wondering what it would pick
up) and I had a good phone on hand (and a "tripod").

Rémy


>
> Cheers,
>
> Woonsan
>
> On Thu, Sep 19, 2019 at 5:12 AM Rémy Maucherat  wrote:
> >
> > Hi,
> >
> > I finished uploading the videos of the twelve sessions from the Apache
> > Tomcat track at  ApacheCon NA 2019 in Las Vegas to the Apache Tomcat
> > YouTube channel:
> > https://www.youtube.com/playlist?list=PLI91YOalv6p9bivF8eb_MR9bVQcVfLkH9
> >
> > The sound recording quality is rather bad, apologies for that. I'll buy
> > basic movie-recording-with-smartphone equipment in case I have to do it
> > again, but the whole thing was improvised this time.
> >
> > Enjoy ! [hopefully ;) ]
> >
> > Rémy
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>


ApacheCon NA Tomcat track videos

2019-09-18 Thread Rémy Maucherat
Hi,

I finished uploading the videos of the twelve sessions from the Apache
Tomcat track at  ApacheCon NA 2019 in Las Vegas to the Apache Tomcat
YouTube channel:
https://www.youtube.com/playlist?list=PLI91YOalv6p9bivF8eb_MR9bVQcVfLkH9

The sound recording quality is rather bad, apologies for that. I'll buy
basic movie-recording-with-smartphone equipment in case I have to do it
again, but the whole thing was improvised this time.

Enjoy ! [hopefully ;) ]

Rémy


Re: Sending http request to https endpoint logs SEVERE in tomcat 9.0.24

2019-08-29 Thread Rémy Maucherat
On Thu, Aug 29, 2019 at 11:02 AM Mark Thomas  wrote:

> On August 29, 2019 8:21:05 AM UTC, Senthalan Kanagalingam
>  wrote:
> >Hi all,
> >
> >When sending an http request to https endpoint logs the following
> >SEVERE in
> >tomcat 9.0.24,
> >
> >29-Aug-2019 13:34:40.088 SEVERE [https-jsse-nio-8443-exec-10]
> >org.apache.coyote.AbstractProtocol$ConnectionHandler.process Error
> >reading
> >request, ignored
> >java.lang.NullPointerException
> >at
> >org.apache.tomcat.util.net
> .NioEndpoint$NioSocketWrapper.getSslSupport(NioEndpoint.java:1392)
> >at
>
> >org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:853)
> >at
> >org.apache.tomcat.util.net
> .NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1593)
> >at
> >org.apache.tomcat.util.net
> .SocketProcessorBase.run(SocketProcessorBase.java:49)
> >at
>
> >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >at
>
> >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> >at org.apache.tomcat.util.threads.TaskThre
> >
> >Here is my connector configuration,
> > >protocol="org.apache.coyote.http11.Http11NioProtocol"
> >   maxThreads="150" SSLEnabled="true">
> >
> > >certificateKeystorePassword="wso2carbon"
> > type="RSA" />
> >
> >
> >
> >When analysing the tomcats code, I have figured out this occurred after
> >an
> >improvement done to "Include failed TLS handshakes in the access
> >log"[1].
> >But I could able to find out which causes the SSL engine to be null.
> >
> >Is something wrong with my setup? if not can we fix this SEVERE level
> >log
> >as someone can flood our logs by sending the HTTP request to the
> >endpoints?
> >
> >[1] -
> >
> https://github.com/apache/tomcat/commit/acf6076d7118571ebc881984b96792f861b72bb2
> >
> >Thanks and regards,
> >Senthalan
>
> That will be a bug. Almost certainly my fault. Sorry. Please open a
> bugzilla issue and it will get fixed for the next release.
>

I can confirm the issue (also for NIO2). APR works as intended.
Conceptually I find it weird to process a socket that it concurrently
closed:
 } else if (handshake == -1 ) {
+getHandler().process(socketWrapper,
SocketEvent.CONNECT_FAIL);
 socketWrapper.close();
 }

Rémy


Re: Memmory leak java.lang.OutOfMemoryError: GC overhead limit exceeded

2019-07-17 Thread Rémy Maucherat
On Wed, Jul 17, 2019 at 3:19 PM Mozgo Andrei  wrote:

> Hi all.
>
> Working on Update Tomcat 8.0.30 to Tomcat 9.0.14 I faced with a memory
> leak issue. The application failed after some hours with the next
> exception:
>
>
>
> *Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded*
>
>
> The memory was completely consumed and the garbage collector was
> struggling all the time trying to free it:
> [image: 1.png]
>
>
> Heap Dump contained 120 instances of
> org.apache.catalina.connector.Response each consumed 10Mb of memory:
> [image: image.png]
>
> I found out that the next Tomcat version 9.0.16 contains a fix for a bunch
> of memory leak issues. Upgrade to 9.0.16 just doubled hour application
> worked then it fails with the same issue. However, it could just be a
> temporary load decrease.
>
>
>
> I tried to use Tomcat 8.5.16 that works fine with another module. Another
> module heap dump didn't contain any instances of
> org.apache.catalina.connector.Response even after more than 116 hours.
> However, Tomcat 8.5.16 didn’t solve the problem. The application failed
> with the same error and heap dump looked the same.
>
>
> Then I looked into the heap dump of the application working on Tomcat
> 8.0.30. It contained instances of org.apache.catalina.connector.Response as
> well. But their size was much smaller:
>
>
> [image: image.png]
>
> It looks like there is a memory leak issue with
> org.apache.catalina.connector.Response in all Tomcat versions (8, 8.5, 9)
> with the only difference in the size of OutputBuffer which is a part of
> Response:
> [image: image.png]
> Here are GC roots of Response instance:
> [image: image.png]
>
> Could you please give a piece of advice on how to manage with this?
>

First, having Response and OutputBuffer instances is normal. The
OutputBuffer implementation was changed from using our own Byte/CharChunk
impl to using Byte/CharBuffer. This is "ok" but has some side effects on
memory use. In particular if response.setBufferSize is used. If it is,
while previously it set an upper limit to which the buffer would gradually
grow as needed, now it simply has to allocate the whole buffer. You might
want to look into that.
Also, the previous code used to discard the allocated char array, while now
it is not possible.

So now you have to be careful with use of setBufferSize (which is ok since
using it is a very bad idea, the default is fine as it is and will perform
well).

Rémy


>
> Thanks,
> Andrei Mozgo
>


Re: Http11NioProtocol deadlocks with useAsyncIO="true" and HTTPS/2.0

2019-07-17 Thread Rémy Maucherat
On Tue, Jul 16, 2019 at 6:47 PM Manuel Dominguez Sarmiento 
wrote:

> Hi, we had been running Tomcat 9.0.17 for quite some time on our high-load
> production servers, using the attached server.xml configuration.
>
> Upon upgrading to 9.0.21 we started experiencing many random deadlocks. We
> run performance advertising campaigns, and our conversion rates dropped to
> below half of what they usually are, which was an obvious consequence of
> our servers randomly locking up. Plus, it was very easy to reproduce the
> deadlocks, which seemed to "magically unlock" when opening a second
> tab/window and opening the same URL that was locked on the other
> window/tab. Doing this unlocked both windows/tabs at once, immediately.
>
> We found this was only happening on HTTPS, but NOT on HTTP. Furthermore,
> we found this was only happening when the browser negotiated an upgrade to
> HTTPS/2.0
> Once we found this, we temporarily removed the  className="org.apache.coyote.http2.Http2Protocol" /> configuration, and all
> was back to normal.
> However, we need HTTP/2, so we continued to look for a proper solution.
>
> Looking at the Tomcat changelog, we found there have been many changes
> since 9.0.17 related to useAsyncIO and HTTP/2. One particular change for
> 9.0.22 caught our attention:
>
>
> *"Remove a source of potential deadlocks when using HTTP/2 when the
> Connector is configured with useAsyncIO as true. (markt)" *We also found
> the following discussion thread, which describes issues similar to what we
> were experiencing:
>
> http://mail-archives.apache.org/mod_mbox/tomcat-dev/201906.mbox/%3c20190606204631.bab6c8a...@gitbox.apache.org%3e
>
> So we upgraded to 9.0.22 thinking that the deadlock would be gone. But
> alas, it was not. The deadlocks remained.
> We found that 9.0.20 changed the default for useAsyncIO from "false" to
> "true". So we changed useAsyncIO back to what it was when we were running
> 9.0.17 (false) and all is back to normal on 9.0.22
>
> So the conclusion is: there are still deadlock bugs on the NIO connector
> with useAsyncIO="true" and upgrades to HTTP/2.0
> Besides fixing them, we believe that the useAsyncIO default should be
> reverted to "false".
>
> We could find no Java deadlocked threads at all by inspecting jconsole
> (not with the automatic "find deadlocks" functionality, nor by inspection
> of a thread dump). We performed several thread dumps WHILE the deadlock was
> clearly visible on screen (this was very easily reproduceable).
>
> The deadlock is definitely there though and goes away as soon as we turn
> off "useAsyncIO".
> Since we could not find Java-level deadlocks, we believe the problem
> probably lies in the interaction with native code.
> We are using org.apache.catalina.core.AprLifecycleListener as well as
> Tomcat Native 1.2.23 on Linux.
> We could not find any pointers in the Tomcat Native changelog dealing with
> similar issues.
>
> Any ideas? Thanks,
>

To investigate your problem, we would need to have an idea on how to
reproduce it.

If you want to debug it yourself, you can turn on the debug logging in
conf/logging.properties:
org.apache.coyote.http2.level = FINE
It's very verbose though.

Rémy


Re: CONNECTION_CLOSED 200 Error with HTTP/2 Enabled

2019-05-28 Thread Rémy Maucherat
On Tue, May 28, 2019 at 1:18 PM Tom Coudyzer  wrote:

> Hi Mark,
>
> The same goes for you,  so thank you for helping to solve this and
> providing the test build.
> I downloaded and installed this version and it works now.
>
> Following tests were executed:
>
> - without useAsyncIO attribute
> - with useAsyncIO="true"
> - with useAsyncIO="false"
>
> In all 3 scenarios it worked correctly and no errors were reported in the
> browser console.
>

Nice, thanks for the testing !

Rémy


Re: CONNECTION_CLOSED 200 Error with HTTP/2 Enabled

2019-05-24 Thread Rémy Maucherat
On Thu, May 23, 2019 at 10:59 PM Tom Coudyzer  wrote:

>
> Should we see this as an issue in our code which requires (for some reason)
> that scripts are loaded in some sequence or is this something that needs to
> be fixed?
>

No. But it would be nice if you would test another test build later.

Rémy


Re: CONNECTION_CLOSED 200 Error with HTTP/2 Enabled

2019-05-23 Thread Rémy Maucherat
On Thu, May 23, 2019 at 8:44 PM Tom Coudyzer  wrote:

> Hi,
>
> I tested with the 9.0.21-dev version and the issue is still there.
> I attached two screenshots on what is happening in Chrome (same behavior
> is present in Firefox). In Internet Explorer 11 there are no errors.
>
> While I wait on some feedback will try to find out how I can enable the
> logging (with more info). As I told earlier Tomcat is embedded in an
> application and it looks something has changed to the standard logging
> mechanism. Since I was only asked recently to have a look at this product I
> need to find this out.Think we'll need this so that we can see what's going
> on.
>
> When I need to try something or you have some ideas, just let me know.
> Will try them out.
>

Ok, so the new async IO might be the cause of your problem, you should set
useAsyncIO="false" on the connector.

I don't know what your page is doing, but I pulled the HTTP/2 test page we
did demo with at conferences and if I insist I can see problems.
This is caused by the check line 1530 in Http2UpgradeHandler, the state
being CLOSED_RX (so no window updating normally). If I comment it out, I
see no error or problems anymore in the logs, and things work reliably. So
maybe it's a processing ordering problem caused by async. I added some sync
in the parser and it works too, not sure why right now.

Rémy


>
> Thanks already!
>
> /Tom
>
>
> On Thu, May 23, 2019 at 3:42 PM Mark Thomas  wrote:
>
>> On 23/05/2019 11:52, Tom Coudyzer wrote:
>> > Hi Mark,
>> >
>> > Thanks for the info.
>> >
>> > If you say where we can download the test build we are happy to do that
>> and
>> > see if it makes any difference.
>>
>> http://people.apache.org/~markt/dev/v9.0.21-dev-d6d3b31/
>>
>> > No problem if any (serious) issues can occur, we'll have the fire
>> > extinguishers ready :-)
>>
>> Great. I just wanted you to be aware that this only a test build and is
>> in no way an official ASF release.
>>
>> Let us know how you get on.
>>
>> Thanks,
>>
>> Mark
>>
>>
>> >
>> > Once again many thanks for your help, much appreciated!
>> >
>> > Regards,
>> > Tom
>> >
>> > On Thu, May 23, 2019 at 11:09 AM Mark Thomas  wrote:
>> >
>> >> On 23/05/2019 09:04, Tom Coudyzer wrote:
>> >>> Hi,
>> >>>
>> >>> We analyzed the problem further and we have set up another version
>> which
>> >> is
>> >>> using Tomcat 9.0.19. We don't have the issue anymore. So it looks like
>> >> it's
>> >>> caused by something that has changed between the two versions. Based
>> on
>> >> the
>> >>> info I have it looks like Chrome is stalling and getting too many
>> >> responses
>> >>> at the same time. In the meantime we activated already the access log
>> and
>> >>> all files in that file are sent with status 200. So it looks like a
>> >> client
>> >>> / browser side issue.
>> >>>
>> >>> Does this information give more information on what could be causing
>> the
>> >>> problem or which parameter we should set up / change or do you still
>> need
>> >>> more information?
>> >>
>> >> Given the issue appears between 9.0.19 and 9.0.20 a Tomcat issue looks
>> >> more likely than a browser issue.
>> >>
>> >> If I point you towards a test build (no guarantees, use at your own
>> >> risk, if your server catches fire don't blame me etc.) for 9.0.21 could
>> >> you test that as there are some HTTP/2 changes in 9.0.20 that made a
>> >> long standing bug more likely that have been fixed in 9.0.21-dev.
>> >>
>> >> If you still see the issue then we can look at specific debug logging
>> >> and/or isolated test cases.
>> >>
>> >> Mark
>> >>
>> >>
>> >>>
>> >>> On Wed, May 22, 2019 at 5:01 PM Mark Thomas  wrote:
>> >>>
>>  On 22/05/2019 15:47, Tom Coudyzer wrote:
>> > Hi,
>> >
>> > We wanted to upgrade our application to start using HTTP/2. We added
>> >> the
>> > necessary and we see that the browser is using HTTP/2 in the
>> browsers'
>> > development tools.
>> >
>> > However since we activated it we get random CONNECTION_CLOSED 200
>> >> Errors
>>  in
>> > Chrome. It's not always on the same files and sometimes there are
>> more,
>> > sometimes there are less, sometimes it works. When we disable HTTP/2
>> > (remove the upgradeprotocol tag) everything works fine.
>> >
>> > We are running Tomcat 9.0.20 (x64) on a Windows Server 2008 R2
>> server.
>> >
>> > Did we configure something incorrectly, is this an HTTP/2 issue or
>> >> should
>> > we look at network issues on our end?
>> >
>> > If you need more information or we need to run something to be able
>> to
>> > troubleshoot it better please let me know.
>> >
>> > Help is much appreciated !
>> 
>>  You can try enabling debug logging if the issue is fairly easy to
>>  reproduce. That might shed some light on what Tomcat is doing and
>> why.
>> 
>>  There are also some HTTP/2 fixes due in the next set of releases that
>>  might help.
>> 
>>  Mark
>> 
>>  

Re: CONNECTION_CLOSED 200 Error with HTTP/2 Enabled

2019-05-23 Thread Rémy Maucherat
On Thu, May 23, 2019 at 12:53 PM Tom Coudyzer  wrote:

> Hi Mark,
>
> Thanks for the info.
>
> If you say where we can download the test build we are happy to do that and
> see if it makes any difference.
> No problem if any (serious) issues can occur, we'll have the fire
> extinguishers ready :-)
>
> Once again many thanks for your help, much appreciated!
>

Tomcat 9.0.20 has known HTTP/2 issues:
- The new async NIO code still had some random bugs that caused connection
errors
- The timeout issue which caused the connection to be often closed once it
hit its window size

The current fixed Tomcat looks ok to me with chrome, but if the problem is
rare it would be more difficult to reproduce.

Rémy


>
> Regards,
> Tom
>
> On Thu, May 23, 2019 at 11:09 AM Mark Thomas  wrote:
>
> > On 23/05/2019 09:04, Tom Coudyzer wrote:
> > > Hi,
> > >
> > > We analyzed the problem further and we have set up another version
> which
> > is
> > > using Tomcat 9.0.19. We don't have the issue anymore. So it looks like
> > it's
> > > caused by something that has changed between the two versions. Based on
> > the
> > > info I have it looks like Chrome is stalling and getting too many
> > responses
> > > at the same time. In the meantime we activated already the access log
> and
> > > all files in that file are sent with status 200. So it looks like a
> > client
> > > / browser side issue.
> > >
> > > Does this information give more information on what could be causing
> the
> > > problem or which parameter we should set up / change or do you still
> need
> > > more information?
> >
> > Given the issue appears between 9.0.19 and 9.0.20 a Tomcat issue looks
> > more likely than a browser issue.
> >
> > If I point you towards a test build (no guarantees, use at your own
> > risk, if your server catches fire don't blame me etc.) for 9.0.21 could
> > you test that as there are some HTTP/2 changes in 9.0.20 that made a
> > long standing bug more likely that have been fixed in 9.0.21-dev.
> >
> > If you still see the issue then we can look at specific debug logging
> > and/or isolated test cases.
> >
> > Mark
> >
> >
> > >
> > > On Wed, May 22, 2019 at 5:01 PM Mark Thomas  wrote:
> > >
> > >> On 22/05/2019 15:47, Tom Coudyzer wrote:
> > >>> Hi,
> > >>>
> > >>> We wanted to upgrade our application to start using HTTP/2. We added
> > the
> > >>> necessary and we see that the browser is using HTTP/2 in the
> browsers'
> > >>> development tools.
> > >>>
> > >>> However since we activated it we get random CONNECTION_CLOSED 200
> > Errors
> > >> in
> > >>> Chrome. It's not always on the same files and sometimes there are
> more,
> > >>> sometimes there are less, sometimes it works. When we disable HTTP/2
> > >>> (remove the upgradeprotocol tag) everything works fine.
> > >>>
> > >>> We are running Tomcat 9.0.20 (x64) on a Windows Server 2008 R2
> server.
> > >>>
> > >>> Did we configure something incorrectly, is this an HTTP/2 issue or
> > should
> > >>> we look at network issues on our end?
> > >>>
> > >>> If you need more information or we need to run something to be able
> to
> > >>> troubleshoot it better please let me know.
> > >>>
> > >>> Help is much appreciated !
> > >>
> > >> You can try enabling debug logging if the issue is fairly easy to
> > >> reproduce. That might shed some light on what Tomcat is doing and why.
> > >>
> > >> There are also some HTTP/2 fixes due in the next set of releases that
> > >> might help.
> > >>
> > >> Mark
> > >>
> > >> -
> > >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > >> For additional commands, e-mail: users-h...@tomcat.apache.org
> > >>
> > >>
> > >
> >
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
> >
> >
>


Re: unable to serve static files with embedded Tomcat

2019-04-19 Thread Rémy Maucherat
On Fri, Apr 19, 2019 at 11:14 PM Garret Wilson 
wrote:

> On 4/19/2019 3:24 PM, Rémy Maucherat wrote:
> > On Fri, Apr 19, 2019 at 7:46 PM Woonsan Ko  wrote:
> >
> >> I found this before from
> >>
> https://stackoverflow.com/questions/48998387/code-works-with-embedded-apache-tomcat-8-but-not-with-9-whats-changed/49011424
> >>
> >>  // Note: make sure that tomcat creates the default connector...
> >>  tomcat.getConnector();
> >>
> >> Worth trying...
>
>
> Yes, that fixed it, Woonsan! Thank you so much!
>
>
> > That looks correct, and it's easy to spot with the logging (the connector
> > logs that it's starting if it's there). I removed the magic creation of
> the
> > connector because it was causing far more hacks (where the magic
> connector
> > had to be removed after its startup). It seemed more normal to have to
> > explicitly create a connector.
> >
> > Rémy
>
> I don't know the history of this, but I have a couple of doubts on the
> face of it.
>
> First, the impression I get from reading _Apache Tomcat 7_ (Apress) is
> that I can create a server piecing together the components (Server,
> Service, Connector, Engine, Host, Context) if I want, but the whole
> point of the `Tomcat` class is to be a helper for doing all this
> automatically. So it seems a bit countertuitive that the `Tomcat` class
> is now making me do some of this wiring automatically. Why then would I
> use the `Tomcat` class? Why don't I just wire it all together from
> scratch? And why did we arbitrarily pick `Connector` for this "magic"
> creation? Why don't we have to call `Tomcat.getHost()` to make sure the
> host gets built, for instance?
>
> Secondly, calling a getter method in order to invoke some secret magic
> creation sauce behind the scenes reeks of a kludge. (Nothing personal;
> I'm just commenting on the design.) This is not "normal". It's not at
> all obvious that there is this magic step required before the thing will
> work. For example, when you see the following code, without some blatant
> comment it's easy to think that this is a no-op call that should be
> removed:
>
>  tomcat.setPort(8080);
>  tomcat.getConnector();  //"It doesn't look like this does anything;
> maybe it we should remove it… "
>  tomcat.setBaseDir("/foo/bar");
>
> If I'm forced to create a connector, I would at least like the option to
> do something normal and expected, such as:
>
> tomcat.getService().addConnector(tomcat.createDefaultConnector());
>
> I _could_ create a default connector manually, but I'd have to be
> copying more of the source code (along with "magic" identifiers such as
> "HTTP/1.1"). So returning to the first doubt above: why am I using the
> `Tomcat` class if it doesn't do the default things for me automatically?
>

Sorry, but the SpringBoot use case is more important [they want to be able
to create a Tomcat without a connector], so that's the way it is now.

tomcat.getService().addConnector(new Connector()); works very well, etc,
just look at what getConnector() does, it's very simple.

The Tomcat class is not supposed to help much in this area, but rather on
setting up the Tomcat instance right, creating contexts right, etc. You can
look at the code and you'll see what the more complex parts are. Adding a
connector isn't one of them.

You can also use a server.xml (and some other important config files like
web.xml) for your embedded now, you can look at Tomcat.main(...) (it is
used by the Tomcat container image example). If your configuration becomes
complex, it could be better than using Java to configure Tomcat.

Rémy


Re: unable to serve static files with embedded Tomcat

2019-04-19 Thread Rémy Maucherat
On Fri, Apr 19, 2019 at 7:46 PM Woonsan Ko  wrote:

> On Fri, Apr 19, 2019 at 1:24 PM Garret Wilson 
> wrote:
> >
> > Embedding Tomcat 9 (with OpenJDK 11 on Windows 10) I want to serve
> > static files from `/foo/bar`. From scouring the web (primarily
> > https://stackoverflow.com/a/15235711/421049 ), the documentation, and
> > some books (primarily _Apache Tomcat 7_), I have this:
> >
> >  Tomcat tomcat = new Tomcat();
> >  tomcat.setPort(8080);
> >  tomcat.setBaseDir("/foo");
>
> I found this before from
>
> https://stackoverflow.com/questions/48998387/code-works-with-embedded-apache-tomcat-8-but-not-with-9-whats-changed/49011424
>
> // Note: make sure that tomcat creates the default connector...
> tomcat.getConnector();
>
> Worth trying...
>

That looks correct, and it's easy to spot with the logging (the connector
logs that it's starting if it's there). I removed the magic creation of the
connector because it was causing far more hacks (where the magic connector
had to be removed after its startup). It seemed more normal to have to
explicitly create a connector.

Rémy


>
> Woonsan
>
> >  Context ctx = tomcat.addContext("", "/foo/bar");
> >  final Wrapper defaultServlet = ctx.createWrapper();
> >  defaultServlet.setName("default");
> >
> defaultServlet.setServletClass("org.apache.catalina.servlets.DefaultServlet");
> >  defaultServlet.addInitParameter("debug", "1");
> >  defaultServlet.addInitParameter("listings", "false");
> >  defaultServlet.setLoadOnStartup(1);
> >  ctx.addChild(defaultServlet);
> >  ctx.addServletMappingDecoded("/", "default");
> >  ctx.addWelcomeFile("index.html");
> >  tomcat.start();
> >  tomcat.getServer().await();
> >
> > Everything looks like it starts up:
> >
> >  Apr 19, 2019 2:18:09 PM org.apache.catalina.core.StandardService
> > startInternal
> >  INFO: Starting service [Tomcat]
> >  Apr 19, 2019 2:18:09 PM org.apache.catalina.core.StandardEngine
> > startInternal
> >  INFO: Starting Servlet engine: [Apache Tomcat/9.0.19]
> >  Apr 19, 2019 2:18:10 PM
> > org.apache.catalina.util.SessionIdGeneratorBase createSecureRandom
> >  WARNING: Creation of SecureRandom instance for session ID
> > generation using [SHA1PRNG] took [281] milliseconds.
> >  Apr 19, 2019 2:18:10 PM org.apache.catalina.core.ApplicationContext
> log
> >  INFO: default: DefaultServlet.init:  input buffer size=2048, output
> > buffer size=2048
> >
> > But connections to http://localhost:8080/ time out. (I'm pretty sure but
> > not positive that I don't have anything blocking this in the firewall.)
> >
> > Is there some simple thing I'm missing to connect things together?
> >
> > Thanks,
> >
> > Garret
> >
> > P.S. The documentation on the web for the details of this is
> > surprisingly sparse.
> >
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
> >
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>


Re: HTTP/1.x broken with Tomcat 9, Java 11 and Http11NioProtocol

2019-03-27 Thread Rémy Maucherat
On Wed, Mar 27, 2019 at 4:21 PM Mark Thomas  wrote:

> On 27/03/2019 15:13, Mark Thomas wrote:
> > On 27/03/2019 04:23, Jason Rivard wrote:
> >> I'm trying to get the following configuration working:
> >>
> >> - Tomcat 9.0.17 (also tested with 9.0.16)
> >> - AdoptOpenJDK Java 11.0.2 on Linux (also tested on Windows)
> >> - Http11NioProtocol Connector
> >> - Http2Protocol ProtocolUpgrade
> >>
> >> I'm using the following connector config:
> >>
> >>  protocol="org.apache.coyote.http11.Http11NioProtocol"
> >>
> sslImplementationName="org.apache.tomcat.util.net.jsse.JSSEImplementation"
> >>  SSLEnabled="true" scheme="https" secure="true"
> >>  keystoreFile="/appData/jks-keystore" keystorePass="password">
> >>   
> >> 
> >>
> >> With the UpgradeProtocol in place, HTTP/2.0 works fine, but when I
> >> attempt a pure HTTP/1.0 or HTTP/1.1 request the server closes the
> >> connection before responding.  This breaks any non-HTTP2.0 client.
> >>
> >> My first question is: Should this configuration work?
> >
> > Yes.
> >
> >> My second is: Why is it broken?  :)
> >
> > Don't know yet. Just started looking.
>
> That didn't take long.
>
> When ALPN is available but no protocol is negotiated, OpenSSL returns
> null whereas JSSE returns "".
>
> Tomcat was doing:
> if (negotiatedProtocol != null) { ...
>
> rather than
>
> if (negotiatedProtocol != null && negotiatedProtocol.length() > 0) {
>
> and trying to create a Processor object for the "" protocol, failing and
> then closing the connection.
>
> The fix will be in the next set of releases (expected towards the end of
> next week).
>

This is a rather serious bug IMO, so that release cannot be delayed.

Rémy


Re: Tomcat 9 Nio2+OpenSSL problem (very likely a bug)

2019-03-19 Thread Rémy Maucherat
On Mon, Mar 18, 2019 at 4:44 PM Igor T  wrote:

> success: 1, read 73 bytes for: 109ms
> success: 2, read 73 bytes for: 218ms
> success: 3, read 73 bytes for: 203ms
> success: 4, read 73 bytes for: 203ms
>

You are right there is something wrong here as well, especially since the
time gets worse on the second attempt. What is the actual roundtrip time
with your server ?

Rémy


Re: Tomcat 9 Nio2+OpenSSL problem (very likely a bug)

2019-03-19 Thread Rémy Maucherat
On Mon, Mar 18, 2019 at 4:44 PM Igor T  wrote:

> > Since 9.0.12 and 16 do the same, I wouldn't look at that at all.
> Something
> > simple like this works in the general case, there must be something
> > specific here. So it's Windows, which some unspecified OpenSSL version.
> >
> > Rémy
>
> That's not right. After many tests I've found out that 9.0.12 build
> comes with [OpenSSL 1.0.2o  27 Mar 2018], while 9.0.16 comes with
> [OpenSSL 1.1.1a  20 Nov 2018].
> The problem was localized to OpenSSL 1.1.1a on Nio2.
> Also it became clear that establishing of connection takes more time
> with OpenSSL 1.1.1a on Nio.
> So it looks like OpenSSL 1.1.1a build is much less optimized and buggy.
>
> So the question is: how to change OpenSSL version that is shipped with
> the latest tomcat build back to 1.0.2?
> Any feedback appreciated.
>

Ok, thanks for the information. The code has been updated for TLS 1.3 when
using OpenSSL 1.1.1, so there are significant changes in all components. We
will investigate.

Rémy


>
>
>
> Detailed test results:
>
> The problem exist:
> Apache Tomcat 9.0.16/Http11Nio2Protocol/OpenSSL 1.1.1a
> 18-Mar-2019 14:34:54.103 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded
> APR based Apache Tomcat Native library [1.2.21] using APR version
> [1.6.5].
> 18-Mar-2019 14:34:54.103 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR
> capabilities: IPv6 [true], sendfile [true], accept filters [false],
> random [true].
> 18-Mar-2019 14:34:54.103 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent
> APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
> 18-Mar-2019 14:34:54.103 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL
> successfully initialized [OpenSSL 1.1.1a  20 Nov 2018]
> 18-Mar-2019 14:34:54.306 INFO [main]
> org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
> ["http-nio2-80"]
> 18-Mar-2019 14:34:54.353 INFO [main]
> org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
> ["https-openssl-nio2-443"]
> 18-Mar-2019 14:34:54.947 INFO [main]
> org.apache.catalina.startup.Catalina.load Server initialization in
> [1,516] milliseconds
> 18-Mar-2019 14:34:54.994 INFO [main]
> org.apache.catalina.core.StandardService.startInternal Starting
> service [Catalina]
> 18-Mar-2019 14:34:54.994 INFO [main]
> org.apache.catalina.core.StandardEngine.startInternal Starting Servlet
> engine: [Apache Tomcat/9.0.16]
> success: 1, read 73 bytes for: 125ms
> denial: 1, Connection reset
> success: 2, read 73 bytes for: 94ms
> denial: 2, Connection reset
> success: 3, read 73 bytes for: 93ms
> denial: 3, Connection reset
> success: 4, read 73 bytes for: 78ms
> denial: 4, Connection reset
> success: 5, read 73 bytes for: 94ms
> denial: 5, Connection reset
>
> Apache Tomcat 9.0.17/Http11Nio2Protocol/OpenSSL 1.1.1a
> 18-Mar-2019 14:41:46.708 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded
> APR based Apache Tomcat Native library [1.2.21] using APR version
> [1.6.5].
> 18-Mar-2019 14:41:46.708 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR
> capabilities: IPv6 [true], sendfile [true], accept filters [false],
> random [true].
> 18-Mar-2019 14:41:46.708 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent
> APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
> 18-Mar-2019 14:41:46.724 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL
> successfully initialized [OpenSSL 1.1.1a  20 Nov 2018]
> 18-Mar-2019 14:41:46.896 INFO [main]
> org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
> ["http-nio2-80"]
> 18-Mar-2019 14:41:46.912 INFO [main]
> org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
> ["https-openssl-nio2-443"]
> 18-Mar-2019 14:41:47.443 INFO [main]
> org.apache.catalina.startup.Catalina.load Server initialization in
> [1,335] milliseconds
> 18-Mar-2019 14:41:47.474 INFO [main]
> org.apache.catalina.core.StandardService.startInternal Starting
> service [Catalina]
> 18-Mar-2019 14:41:47.474 INFO [main]
> org.apache.catalina.core.StandardEngine.startInternal Starting Servlet
> engine: [Apache Tomcat/9.0.17]
> success: 1, read 73 bytes for: 78ms
> denial: 1, Connection reset
> success: 2, read 73 bytes for: 93ms
> denial: 2, Connection reset
> success: 3, read 73 bytes for: 78ms
> denial: 3, Connection reset
> success: 4, read 73 bytes for: 94ms
> denial: 4, Connection reset
> success: 5, read 73 bytes for: 78ms
> denial: 5, Connection reset
>
>
> The problem does not exist:
> Apache Tomcat 9.0.12/Http11Nio2Protocol/OpenSSL 1.0.2o
> 18-Mar-2019 14:30:21.917 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent 

Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload

2019-03-14 Thread Rémy Maucherat
On Thu, Mar 14, 2019 at 2:58 PM Santhosh Kumar 
wrote:

> I have tried the same set of testcases using h2load with tomcat 9.0.17
> from github releases
> (https://github.com/apache/tomcat/releases/tag/9.0.17) .
>
> h2load -n 100 -d /tmp/1k https://localhost:8443/
> requests: 100 total, 64 started, 63 done, 63 succeeded, 37 failed, 37
> errored, 0 timeout
>
> h2load -n 200 -d /tmp/512b https://localhost:8443/
> requests: 200 total, 128 started, 127 done, 127 succeeded, 73 failed,
> 73 errored, 0 timeout
>
> The issue still occurs in latest version too.
>

This is not related to the issue that you reported (you gave a stacktrace
for it, and it was helpful). Please use a real test with a real servlet
consuming the data,

Rémy


Re: Tomcat 9 Nio2+OpenSSL problem (very likely a bug)

2019-03-13 Thread Rémy Maucherat
On Wed, Mar 13, 2019 at 10:21 PM Mark Thomas  wrote:

> On 13/03/2019 20:30, Igor T wrote:
> > Prerequisites:
> > OS: Windows Server 2012 R2
> > Java: checked on both jdk1.8.0_162 jdk1.8.0_181
> > Tomcat: windows x64 builds checked on 9.0.12, 9.0.16, 9.0.17-dev
>
> 9.0.17-dev at which point in time?
>

Since 9.0.12 and 16 do the same, I wouldn't look at that at all. Something
simple like this works in the general case, there must be something
specific here. So it's Windows, which some unspecified OpenSSL version.

Rémy


>
> Have you tested the current 9.0.17 release candidate (see dev@ for
> details)
>
> Mark
>
>
>
> > Valid SSL certificates
> > Content of file located at webapp/ROOT/1.txt: []
> > Tomcat's connector settings:
> >  >
>  protocol="org.apache.coyote.http11.Http11Nio2Protocol"
> >
> > sslImplementationName="org.apache.tomcat.util.net
> .openssl.OpenSSLImplementation"
> > connectionTimeout="5000"
> > SSLEnabled="true"
> > scheme="https"
> > secure="true"
> > >
> >
> > This configuration leads to 50% of the traffic to be rejected with
> > Connection resets. First socket connects and receives the service, but
> > every second is resetted.
> >
> > Exactly this combination leads to connection resets:
> > protocol="org.apache.coyote.http11.Http11Nio2Protocol"
> > sslImplementationName="org.apache.tomcat.util.net
> .openssl.OpenSSLImplementation"
> >
> > Configurations that work well without connection resets:
> > protocol="org.apache.coyote.http11.Http11Nio2Protocol"
> >
>  sslImplementationName="org.apache.tomcat.util.net.jsse.JSSEImplementation"
> > or
> > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > sslImplementationName="org.apache.tomcat.util.net
> .openssl.OpenSSLImplementation"
> >
> > Java code to reproduce the connection resets (works well with any
> > other secure server):
> > (there is no resets if a variable named FIX__gotoSleepAfterHandshake =
> true)
> >
> > public class CheckConnectionResets{
> > static String host = "your-test-host";
> >
> > public static void main( String[] args ) throws
> > InterruptedException, IOException{
> >
> > SSLSocketFactory factory =
> > (SSLSocketFactory)SSLSocketFactory.getDefault();
> > int nRuns = 4;
> > int success = 0;
> > int denial = 0;
> >
> > boolean FIX__gotoSleepAfterHandshake = false;
> >
> > for( int i = 0; i < nRuns; i++ ){
> > try ( SSLSocket socket = (SSLSocket)factory.createSocket(
> > host, 443 ) ){
> >
> > if( FIX__gotoSleepAfterHandshake ){
> > socket.startHandshake();
> > Thread.sleep( 500 );
> > }
> > try ( PrintWriter out = new PrintWriter( new
> > BufferedWriter( new OutputStreamWriter( socket.getOutputStream() ) )
> > );
> > InputStream is = socket.getInputStream(); ){
> >
> > out.println( "GET /1.txt HTTP/1.1" );
> > out.println( "Host: " + host );
> > out.println( "Accept: */*" );
> > out.println();
> > out.flush();
> >
> > if( out.checkError() ){
> > System.out.println( "SSLSocketClient:
> > java.io.PrintWriter error" );
> > }
> >
> > Instant start = Instant.now();
> > /* read full response */
> > byte[] buff = new byte[ 1024 ];
> > int read = is.read( buff );
> > success++;
> > System.out.println( "success: " + success + ",
> > read " + read + " bytes for: " + start.until( Instant.now(),
> > ChronoUnit.MILLIS ) + "ms" );
> >
> > } catch ( IOException e ) {
> > denial++;
> > System.err.println( "denial: " + denial + ", " +
> > e.getMessage() );
> > }
> >
> > Sample output:
> > success: 1, read 73 bytes for: 78ms
> > denial: 1, Connection reset
> > success: 2, read 73 bytes for: 78ms
> > denial: 2, Connection reset
> >
> > The bug is stable, and always reproducible.
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>


Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload

2019-03-07 Thread Rémy Maucherat
On Thu, Mar 7, 2019 at 5:22 PM Rémy Maucherat  wrote:

> On Thu, Mar 7, 2019 at 2:54 PM Rémy Maucherat  wrote:
>
>> On Thu, Mar 7, 2019 at 1:47 PM Mark Thomas  wrote:
>>
>>> On 07/03/2019 07:40, Santhosh Kumar wrote:
>>> > From some of the test cases I can safely say that tomcat is hitting
>>> some
>>> > limits, I have two test cases ran with two diff size of payload and
>>> without
>>> > any queryParams. The servlet is a empty servlet just returns after
>>> > receiving without doing any business side logic
>>>
>>> Can you repeat those tests with the NIO connector? It would be helpful
>>> to know if we should be looking at the HTTP/2 code or the low-level
>>> connector I/O code.
>>>
>>
>> I was planning to investigate since I'm hunting NIO2 additional issues
>> after the fix for BZ63182. This one looks simpler to reproduce at least
>> [assuming there's an issue].
>>
>
> Ok, so it's a buffer size issue with vectored IO and SSL, the sizes used
> are too optimized.
>

The "fix" will be in Tomcat 9.0.17, the read buffer used simply needs to be
larger with SSL. Both the JSSE and OpenSSL engines exhibit the same
behavior, so no possible workaround elsewhere.

Rémy


Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload

2019-03-07 Thread Rémy Maucherat
On Thu, Mar 7, 2019 at 2:54 PM Rémy Maucherat  wrote:

> On Thu, Mar 7, 2019 at 1:47 PM Mark Thomas  wrote:
>
>> On 07/03/2019 07:40, Santhosh Kumar wrote:
>> > From some of the test cases I can safely say that tomcat is hitting some
>> > limits, I have two test cases ran with two diff size of payload and
>> without
>> > any queryParams. The servlet is a empty servlet just returns after
>> > receiving without doing any business side logic
>>
>> Can you repeat those tests with the NIO connector? It would be helpful
>> to know if we should be looking at the HTTP/2 code or the low-level
>> connector I/O code.
>>
>
> I was planning to investigate since I'm hunting NIO2 additional issues
> after the fix for BZ63182. This one looks simpler to reproduce at least
> [assuming there's an issue].
>

Ok, so it's a buffer size issue with vectored IO and SSL, the sizes used
are too optimized.

Rémy


>
> Rémy
>
>
>>
>> Thanks,
>>
>> Mark
>>
>>
>> >
>> > h2load -n100 -c1 -m1 --header="Content-Type:application/json" -d
>> > /home/local/santhosh/A-Test/nghttp2/agentRequest.txt
>> > https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet
>> > starting benchmark...
>> > spawning thread #0: 1 total client(s). 100 total requests
>> > TLS Protocol: TLSv1.3
>> > Cipher: TLS_AES_256_GCM_SHA384
>> > Server Temp Key: X25519 253 bits
>> > Application protocol: h2
>> > progress: 10% done
>> > progress: 20% done
>> > progress: 30% done
>> > progress: 40% done
>> > progress: 50% done
>> >
>> > finished in 5.16s, 10.48 req/s, 552B/s
>> > requests: 100 total, 55 started, 54 done, 54 succeeded, 46 failed, 46
>> > errored, 0 timeout
>> > status codes: 55 2xx, 0 3xx, 0 4xx, 0 5xx
>> > traffic: 2.78KB (2846) total, 1.77KB (1815) headers (space savings
>> 43.10%),
>> > 0B (0) data
>> >  min max mean sd+/-
>> sd
>> > time for request: 1.57ms  9.43ms  2.24ms  1.17ms
>> 94.44%
>> > time for connect: 4.69ms  4.69ms  4.69ms 0us
>>  100.00%
>> > time to 1st byte:0us 0us 0us 0us
>>  0.00%
>> > req/s   :  10.48   10.48   10.480.00
>>  100.00%
>> >
>> > This above configuration always returns 54 succeeded, payload size is
>> 1200B
>> > (1200x54=64800)
>> >
>> --
>> > Now reduce the payload and trying the same test,
>> >
>> > h2load -n100 -c1 -m1 --header="Content-Type:application/json" -d
>> > /home/local/santhosh/A-Test/nghttp2/agentRequest2.txt
>> > https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet
>> > starting benchmark...
>> > spawning thread #0: 1 total client(s). 100 total requests
>> > TLS Protocol: TLSv1.3
>> > Cipher: TLS_AES_256_GCM_SHA384
>> > Server Temp Key: X25519 253 bits
>> > Application protocol: h2
>> > progress: 10% done
>> > progress: 20% done
>> > progress: 30% done
>> > progress: 40% done
>> > progress: 50% done
>> > progress: 60% done
>> > progress: 70% done
>> > progress: 80% done
>> >
>> > finished in 5.21s, 16.11 req/s, 839B/s
>> > requests: 100 total, 85 started, 84 done, 84 succeeded, 16 failed, 16
>> > errored, 0 timeout
>> > status codes: 85 2xx, 0 3xx, 0 4xx, 0 5xx
>> > traffic: 4.27KB (4376) total, 2.74KB (2805) headers (space savings
>> 43.10%),
>> > 0B (0) data
>> >  min max mean sd+/-
>> sd
>> > time for request: 1.43ms  5.80ms  2.04ms   760us
>> 89.29%
>> > time for connect: 5.02ms  5.02ms  5.02ms 0us
>>  100.00%
>> > time to 1st byte:0us 0us 0us 0us
>>  0.00%
>> > req/s   :  16.11   16.11   16.110.00
>>  100.00%
>> >
>> > This above configuration always returns 84 succeeded, payload size is
>> 775B
>> > (775x84=65100)
>> >
>> --
>> > Reducing the payload much smaller,
>> >
>> > h2load -n200 -c1 -m1 --header="

Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload

2019-03-07 Thread Rémy Maucherat
On Thu, Mar 7, 2019 at 1:47 PM Mark Thomas  wrote:

> On 07/03/2019 07:40, Santhosh Kumar wrote:
> > From some of the test cases I can safely say that tomcat is hitting some
> > limits, I have two test cases ran with two diff size of payload and
> without
> > any queryParams. The servlet is a empty servlet just returns after
> > receiving without doing any business side logic
>
> Can you repeat those tests with the NIO connector? It would be helpful
> to know if we should be looking at the HTTP/2 code or the low-level
> connector I/O code.
>

I was planning to investigate since I'm hunting NIO2 additional issues
after the fix for BZ63182. This one looks simpler to reproduce at least
[assuming there's an issue].

Rémy


>
> Thanks,
>
> Mark
>
>
> >
> > h2load -n100 -c1 -m1 --header="Content-Type:application/json" -d
> > /home/local/santhosh/A-Test/nghttp2/agentRequest.txt
> > https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet
> > starting benchmark...
> > spawning thread #0: 1 total client(s). 100 total requests
> > TLS Protocol: TLSv1.3
> > Cipher: TLS_AES_256_GCM_SHA384
> > Server Temp Key: X25519 253 bits
> > Application protocol: h2
> > progress: 10% done
> > progress: 20% done
> > progress: 30% done
> > progress: 40% done
> > progress: 50% done
> >
> > finished in 5.16s, 10.48 req/s, 552B/s
> > requests: 100 total, 55 started, 54 done, 54 succeeded, 46 failed, 46
> > errored, 0 timeout
> > status codes: 55 2xx, 0 3xx, 0 4xx, 0 5xx
> > traffic: 2.78KB (2846) total, 1.77KB (1815) headers (space savings
> 43.10%),
> > 0B (0) data
> >  min max mean sd+/-
> sd
> > time for request: 1.57ms  9.43ms  2.24ms  1.17ms
> 94.44%
> > time for connect: 4.69ms  4.69ms  4.69ms 0us
>  100.00%
> > time to 1st byte:0us 0us 0us 0us
>  0.00%
> > req/s   :  10.48   10.48   10.480.00
>  100.00%
> >
> > This above configuration always returns 54 succeeded, payload size is
> 1200B
> > (1200x54=64800)
> >
> --
> > Now reduce the payload and trying the same test,
> >
> > h2load -n100 -c1 -m1 --header="Content-Type:application/json" -d
> > /home/local/santhosh/A-Test/nghttp2/agentRequest2.txt
> > https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet
> > starting benchmark...
> > spawning thread #0: 1 total client(s). 100 total requests
> > TLS Protocol: TLSv1.3
> > Cipher: TLS_AES_256_GCM_SHA384
> > Server Temp Key: X25519 253 bits
> > Application protocol: h2
> > progress: 10% done
> > progress: 20% done
> > progress: 30% done
> > progress: 40% done
> > progress: 50% done
> > progress: 60% done
> > progress: 70% done
> > progress: 80% done
> >
> > finished in 5.21s, 16.11 req/s, 839B/s
> > requests: 100 total, 85 started, 84 done, 84 succeeded, 16 failed, 16
> > errored, 0 timeout
> > status codes: 85 2xx, 0 3xx, 0 4xx, 0 5xx
> > traffic: 4.27KB (4376) total, 2.74KB (2805) headers (space savings
> 43.10%),
> > 0B (0) data
> >  min max mean sd+/-
> sd
> > time for request: 1.43ms  5.80ms  2.04ms   760us
> 89.29%
> > time for connect: 5.02ms  5.02ms  5.02ms 0us
>  100.00%
> > time to 1st byte:0us 0us 0us 0us
>  0.00%
> > req/s   :  16.11   16.11   16.110.00
>  100.00%
> >
> > This above configuration always returns 84 succeeded, payload size is
> 775B
> > (775x84=65100)
> >
> --
> > Reducing the payload much smaller,
> >
> > h2load -n200 -c1 -m1 --header="Content-Type:application/json" -d
> > /home/local/santhosh/A-Test/nghttp2/agentRequest3.txt
> > https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet
> > starting benchmark...
> > spawning thread #0: 1 total client(s). 200 total requests
> > TLS Protocol: TLSv1.3
> > Cipher: TLS_AES_256_GCM_SHA384
> > Server Temp Key: X25519 253 bits
> > Application protocol: h2
> > progress: 10% done
> > progress: 20% done
> > progress: 30% done
> > progress: 40% done
> > progress: 50% done
> > progress: 60% done
> > progress: 70% done
> > progress: 80% done
> > progress: 90% done
> >
> > finished in 5.41s, 34.40 req/s, 1.73KB/s
> > requests: 200 total, 187 started, 186 done, 186 succeeded, 14 failed, 14
> > errored, 0 timeout
> > status codes: 187 2xx, 0 3xx, 0 4xx, 0 5xx
> > traffic: 9.35KB (9578) total, 6.03KB (6171) headers (space savings
> 43.10%),
> > 0B (0) data
> >  min max mean sd+/-
> sd
> > time for request: 1.18ms 13.49ms  1.91ms  1.13ms
> 95.16%
> > time for connect: 5.93ms  5.93ms  5.93ms 0us
>  100.00%
> > time to 1st byte:0us 0us 0us   

Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload

2019-03-04 Thread Rémy Maucherat
On Mon, Mar 4, 2019 at 10:40 AM Santhosh Kumar 
wrote:

> Hi,
>
> We have a tomcat instance which is http2 enabled and it needs to serve
> large number of requests using multiplexing, so we have configured our
> instance as follows,
>
>  sslImplementationName="org.apache.tomcat.util.net
> .openssl.OpenSSLImplementation"
> protocol="org.apache.coyote.http11.Http11Nio2Protocol"
>  maxThreads="5" SSLEnabled="true"
>
> compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json,application/xml"
>  compression="on" minSpareThreads="25"
> noCompressionUserAgents="gozilla, traviata" scheme="https" secure="true"
> keystoreFile="conf/myfile.keystore" keystorePass="password"
>  socket.appReadBufSize="81920"
> socket.appWriteBufSize="81920" socket.rxBufSize="251880"
> socket.txBufSize="438000">
> maxConcurrentStreamExecution="200"
> maxConcurrentStreams="200"
> className="org.apache.coyote.http2.Http2Protocol"/>
>   
>

Do you actually need all those values and where do they come from in the
first place ? Do you understand what they do ? The buffer sizes seem way
too large, and maxConcurrentStreamExecution is as well (multiplexing 200
threads output over a single socket is not a very good idea usually, which
is why the setting was added).

Rémy


>
> This instance mainly serves concurrent POST request which will have payload
> of size, approx 1000-1500, which can be verified by tomcat logs
>
> org.apache.coyote.http2.Http2Parser.validateFrame Connection [0], Stream
> [19], Frame type [DATA], Flags [1], Payload size [*1195*]
>
> We tested our server with the help of h2load as follows,
>
> h2load -n100 -c1 -m100 https://localhost:9191/ -d '/agentRequest.txt'
>
> We are getting this error as follows,
>
>
> org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch Connection [0]
>  java.io.IOException: Unable to unwrap data, invalid status
> [BUFFER_OVERFLOW]
> at
> org.apache.tomcat.util.net
> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1041)
> at
> org.apache.tomcat.util.net
> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000)
> at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
> at java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158)
> at
> java.base/sun.nio.ch
> .UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:552)
> at
> java.base/sun.nio.ch
> .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
> at
> java.base/sun.nio.ch
> .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
> at
> org.apache.tomcat.util.net
> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1027)
> at
> org.apache.tomcat.util.net
> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000)
> at
> org.apache.tomcat.util.net
> .SecureNio2Channel.read(SecureNio2Channel.java:1067)
> at
> org.apache.tomcat.util.net
> .Nio2Endpoint$Nio2SocketWrapper$VectoredIOCompletionHandler.completed(Nio2Endpoint.java:1153)
> at
> org.apache.tomcat.util.net
> .Nio2Endpoint$Nio2SocketWrapper.read(Nio2Endpoint.java:1026)
> at
> org.apache.tomcat.util.net
> .SocketWrapperBase.read(SocketWrapperBase.java:1012)
> at
>
> org.apache.coyote.http2.Http2AsyncParser.readFrame(Http2AsyncParser.java:61)
> at
> org.apache.coyote.http2.Http2Parser.readFrame(Http2Parser.java:69)
> at
>
> org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch(Http2UpgradeHandler.java:322)
> at
>
> org.apache.coyote.http2.Http2AsyncUpgradeHandler.upgradeDispatch(Http2AsyncUpgradeHandler.java:37)
> at
>
> org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54)
> at
>
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:53)
> at
>
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)
> at
> org.apache.tomcat.util.net
> .Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1769)
> at
> org.apache.tomcat.util.net
> .SocketProcessorBase.run(SocketProcessorBase.java:49)
> at
> org.apache.tomcat.util.net
> .AbstractEndpoint.processSocket(AbstractEndpoint.java:1048)
> at
> org.apache.tomcat.util.net
> .SecureNio2Channel$HandshakeWriteCompletionHandler.completed(SecureNio2Channel.java:116)
> at
> org.apache.tomcat.util.net
> .SecureNio2Channel$HandshakeWriteCompletionHandler.completed(SecureNio2Channel.java:109)
> at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
> at java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158)
>
> Why is this error is thrown? How can I configure tomcat to handle
> concurrent POST requests which have a decent payload?
>
>

Re: Hanging requests with nio2/openssl connector

2019-01-16 Thread Rémy Maucherat
On Wed, Jan 16, 2019 at 1:53 PM Roger Brechbühl 
wrote:

> Hello,
>
> When using Nio2 connector with OpenSSL we sometimes had hanging requests
> which finally ended in a timeout. This behaviour was not reproducable with
> single requests (e.g. via Browser, with curl etc) but while doing some
> research we were able to find a pattern and could reproduce this with some
> tools/software [5]:
>
> Pattern: first request ok, all other were hanging until timeout
>
> Gatling Performance-Tool with activated http client sharing (which is
> default) [3]
> => with using disableClientSharing we could get around it.
>
> When using Nginx as a proxy between client and Tomcat, with activated
> ssl_session_reuse [4].
> => setting proxy_ssl_session_reuse to off, no hanging requests.
>
> When using HAProxy (we were not able to configure to make it work)
>
> As it always works with other connector configurations (NIO/OpenSSL and
> APR/OpenSSL) we assume that the problem must be in Tomcat/Nio2/OpenSSL
> implementation related to ssl session reuse.


There are relevant changes in newer Tomcat releases regarding some harder
to reproduce OpenSSL problems.
Also, although the proxies probably use HTTP/1.1 and it may not be
relevant, APR and NIO2 have HTTP/2 ALPN enabled while NIO does not.

Rémy


Re: Unable to determine Catalina base to load file

2019-01-08 Thread Rémy Maucherat
On Tue, Jan 8, 2019 at 8:03 PM Mark Thomas  wrote:

> This looks like an environmental issue. Debugging is probably your best
> option to get to the bottom of this.
>

Maybe I broke it with my refactorings, but I'm not very sure at the moment
since you couldn't reproduce it. I can also see that
Container.getCatalinaHome/Base seem quite useless and complex.

Rémy


Re: Tomcat 9.0.14 Windows service slow to stop

2019-01-08 Thread Rémy Maucherat
On Tue, Jan 8, 2019 at 4:25 PM Jean-Pascal Houde  wrote:

> "Catalina-utility-1" prio=1 tid=15 WAITING
>

Ok, so the new default for the utility thread is non daemon so you should
have your fix there. Is nothing calling Server.destroy then ? Normally it
should (Catalina.stop does it) and the utility executor will get shutdown
then.

Rémy


Re: Wrong content-type for CSS files since 8.5.37 / 9.0.14

2018-12-27 Thread Rémy Maucherat
On Thu, Dec 27, 2018 at 9:30 PM Mark Thomas  wrote:

> On December 26, 2018 9:49:00 PM UTC, "i...@flyingfischer.ch" <
> i...@flyingfischer.ch> wrote:
> >Tomcat versions 8.5.37 and 9.0.14 seem to serve CSS files embedded in a
> >webapp as
> >
> >content-type: text/html;charset=UTF-8
> >
> >instead of
> >
> >content-type: text/css;charset=UTF-8
> >
> >This causes the browser (FF) not to interpret the CSS.
> >
> >I suspect the listed change in changelog of 8.5.36: "The default
> >Servlet
> >should not override a previously set content-type. (remm)"
>
> Sounds likely.
>
> Whatever part of the app is setting the incorrect content type (probably a
> filter) needs to be fixed.
>

Yes, I also prefer keeping the new behavior, which was suggested by
Christopher. Worst case scenario there should be a new init-param.

Rémy


Re: tomcat 8.5.35 warning using NIO 2 (or NIO) connector w APR: An unknown setting with identifier [2147483647] and value [2] was ignored

2018-12-20 Thread Rémy Maucherat
On Thu, Dec 20, 2018 at 9:37 PM John Palmer  wrote:

> I'm working with tomcat 8.5.35 to configure SSL
> (current system is tomcat 7.5 using JKS keystore and truststore)..
>
> I finally have the certificate parts working with the default (commented
> out) APR connector..
> it bothers me (doesn't seem intuitive) that the logging shows
> "useAprConnector [false]" as in:
> INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent
> APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
>
> so I'm experimenting with changing the protocol section from:
>  protocol="org.apache.coyote.http11.Http11AprProtocol"
> to
>  protocol="org.apache.coyote.http11.Http11NioProtocol"
> (or
>  protocol="org.apache.coyote.http11.Http11Nio2Protocol"
> )
> since those are what are actually being used.
> but with either of those selected, each time a browser accesses the system,
> the catalina log shows:
> org.apache.coyote.http2.ConnectionSettingsBase.set Connection [0], An
> unknown setting with identifier [2147483647] and value [2] was ignored
> (this doesn't happen when the APR protocol is selected).
> Any idea what setting this refers to?
>

It means there was an unknown setting in the settings frame, and it's a non
fatal warning. The logging is not good because it logs the constant rather
than the unknown id, but then it's only an int.

Rémy


Re: [ANN] New committer: Woonsan Ko

2018-12-20 Thread Rémy Maucherat
On Thu, Dec 20, 2018 at 3:05 AM Woonsan Ko  wrote:

> Thank you very much for the warm welcomes!
> I have watched many volunteering hearts here, helping each other in
> this community, for years. This community has been my biggest input
> source from which I have learned how to grow ourselves together, in
> positive, collaborative, collective, inclusive, sustainable ways.
> I will do my best where I can help to keep the health of our heartbeats.
>

Thanks for the work so far :)

Rémy


Re: How to use server.xml with embedded Tomcat 9?

2018-12-18 Thread Rémy Maucherat
On Tue, Dec 18, 2018 at 7:33 AM Ryan Palmer  wrote:

> Hello,
>
> I'm using the Tomcat class to embed the container in my application. I
> have configured the CATLINA_HOME and _BASE properties, and I know those are
> working because the 'work' folder gets generated there as expected. However
> if I put a server.xml file in a 'conf' folder in the same directory, it
> does not seem to be loaded when calling Tomcat.init().
>
> Documentation is very sparse on the Tomcat class so I am unsure how to
> configure it the "standard" way.
>

I very recently refactored Tomcat embedded (in Tomcat 9.0.14) to be more
flexible, with the main assumption that trying to replicate a complex
server.xml with Java code while keeping good configurability was a waste of
resources.

As for using it, there's "big jar" packaging (but with the important note
that it remains EE webapp based, rather than proprietary):
https://github.com/apache/tomcat/tree/trunk/res/tomcat-maven
Or you can do it yourself, you need to use
Tomcat.init(ConfigurationSource). You can use
startup.CatalinaBaseConfigurationSource to replicate the default Tomcat
behavior, or implement ConfigurationSource and have more flexibility. A few
components don't use that abstraction because they persist configs (JASPIC,
storeconfig).

Rémy


Re: Translations update

2018-11-27 Thread Rémy Maucherat
On Tue, Nov 27, 2018 at 12:01 PM André Warnier (tomcat) 
wrote:

> I must say that, although I tried to participate as much I could, I have
> some reservations
> about this whole translation project.  And that is because most of the
> original messages
> which I have seen, are really "technical" and not at all oriented to a
> general public
> which may be using applications built on tomcat, but rather to a public
> having to deal
> specifically with tomcat Java code and tomcat configuration files.
> This public is going to need messages which they can later connect to that
> code and/or to
> the configuration files language and/or to the available documentation.
> And let's face it : in terms of anything computer-related,
> non-native-English-speakers
> (such as myself) lost out a long time ago, and have had, and will have, to
> learn a modicum
> of English technical computer language anyway, just to understand the
> basics of their
> field of expertise.
> That is not what most of us would culturally prefer, but it is a fact of
> life.
>

Yes, I agree: it's not possible for non english speakers to use Tomcat, so
it did seem pointless. Mark still wanted to do the experiment, and since
the tool was easy and I had some time I did that French stuff. Anyway it's
done now, and people are sometimes happy to get i18n, so...

Rémy


Re: Translations update

2018-11-27 Thread Rémy Maucherat
On Mon, Nov 26, 2018 at 5:31 PM André Warnier (tomcat) 
wrote:

> On 26.11.2018 15:01, Ludovic Pénet wrote:
> >>> - "endpoint" (for websockets, and for the Tomcat connectors, so
> >> >possibly
> >>> >>two different terms): "point d'entrée" ?
> >> >
> >> >That sounds like exactly the opposite of "endpoint" to me.
> >> >Although I must say that even in English, the vocabulary used in some
> >> >reference documents
> >> >(in particular everything to do with XML-based protocols, such as SOAP,
> >> >SAML, OASIS and
> >> >the like) is sometimes mysterious and counter-intuitive.
> >> >What about "cible" here ?
> >> >Or more literally, "point final" ?
> > Terminaison ?
> >
> >
> >> >
> +1 for "terminaison", else untranslated "endpoint".
>

Ok for "terminaison" in the end, updated.

Kinda meh for "répartiteur", which sounds ok in theory but not so great in
practice.

Example:
The dispatcher returned from the ServletContext does not support
asynchronous dispatching
->
Le répartiteur de Servlets retourné par le ServletContext ne supporte pas
de répartition (???) asynchrone
Looks like a good candidate for untranslated now ...

And for Tribes membership, I'm updating to "registre de membres" rather
than "liste", it sounds more service-ish like it actually does something
rather than being dumb and static.

Rémy


Re: Translations update

2018-11-26 Thread Rémy Maucherat
On Mon, Nov 26, 2018 at 3:46 PM Christopher Schultz <
ch...@christopherschultz.net> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> André,
>
> On 11/26/18 08:35, André Warnier (tomcat) wrote:
> > On 26.11.2018 13:29, Rémy Maucherat wrote:
> >> On Sat, Nov 24, 2018 at 9:48 AM Ludovic Pénet 
> >> wrote:
> >>
> >>> Le vendredi 23 novembre 2018 à 23:51 +0100, Rémy Maucherat a
> >>> écrit :
> >>>> On Wed, Nov 21, 2018 at 10:58 AM Mark Thomas
> >>>>  wrote:
> >>>>
> >>>>> - French has increased from 18% to 64% coverage
> >>>>>
> >>>>
> >>>> Done (well, close enough, a few tribes/ha remain) !
> >>> A single translation remains to be performed.
> >>>
> >>> Jump to https://poeditor.com/join/project/NUTIjDWzrl and be the
> >>> one to complete the French translation. ;-)
> >>>
> >>
> >> Ok, you could have finished it, I was busy.
> >>
> >> Now we can try to harmonize terms, fixes are then easy to do with
> >> the search feature
> >>
> >> Common ones we have right now: - "socket" (usually untranslated
> >> or cleverly omitted): ? - "endpoint" (for websockets, and for the
> >> Tomcat connectors, so possibly two different terms): "point
> >> d'entrée" ?
> >
> > That sounds like exactly the opposite of "endpoint" to me. Although
> > I must say that even in English, the vocabulary used in some
> > reference documents (in particular everything to do with XML-based
> > protocols, such as SOAP, SAML, OASIS and the like) is sometimes
> > mysterious and counter-intuitive. What about "cible" here ? Or more
> > literally, "point final" ?
>
> I disagree.
>
> An "endpoint" is a thing to which clients connect... an "entry point",
> as Rémy suggests.
>

French and English constructs are the opposite in a lot of cases so that's
why I though that "point d'entrée" was pretty good, as you stay the
endpoint for the client is the "startingpoint" for the server (but there it
sounds really bad).


>
> > For "socket", "soquet" (like the piece in which you insert a plug,
> > or a lightbulb) sounds ok to me.
>
> This sounds okay to me, thought I don't know French at all. :)
>
> >> - "thread" (often it is untranslated elsewhere): "fil
> >> d'exécution" ? - "membership" (that's the clustering object):
> >> "gestionnaire de membres" ?
> >
> > "Membership" refers to "le fait d'être membre", no ? "adhésion" ?
> > (like "cluster members" -> "adhérents au cluster" (with the
> > appropriate French pronounciation for "cleustère") :-)
>
> What would you call a list of people who belong to a certain fancy
> club or society? That's the word that should be used, here.
>

So ... In that case it would simply be "liste de membres". Which after a
quick check actually looks quite good in the context of the Tribes strings.

I have another difficult one for Tribes: that "replicated map" which should
be ?? "structure répliquée" ?
I used various terms for that annoying one ...


>
> >> - "dispatch"/"dispatcher" (for the Servlet request dispatcher):
> >> ?
> >>
> >
> > dépêcher / dépêcheur ?
> >
> >> And I just saw it is really "connexion" and not "connection".
> >> Oooops, I thought both were ok. I guess it's the same kind of
> >> mistake with English-UK vs English-US, where I usually hate the
> >> UK style (except in HarryP and Discworld, it's part of the charm
> >> I suppose).
> >>
> >
> > Maybe a note : the target audience of most of these messages is not
> > the members of the Académie or the jury of the Prix Goncourt. Its
> > is programmers, sysadmins and qualified tomcat/webservers users.
> > The translations should be helpful to them, to get a first idea of
> > the issue and be able to search later in the on-line documentation.
> > Which happens to be only available/up-to-date/searchable in
> > English, no ?
> >
> > So I believe that a translation such as "La requête PTHT recue sur
> > le soquet du connecteur de toile a été dépêchée au conducteur du
> > groupe d'adhérents" may be stylistically correct, but ultimately
> > quite counter-productive.
> >
> > (Sorry for the missing c cédille, can't type it here) (PTHT =
> > Protocol de Transport Hyper-Texte)
>
> HTTP should always be spelled HTTP and never PTHT, just like UTC is
> always spelled UTC, even in English (where the acronym makes no sense
> to Englist speakers).
>
> I think maybe you were kidding, but ... just in case :)
>

We were super serious, like for Apache Matou :)

Rémy


Re: Translations update

2018-11-26 Thread Rémy Maucherat
On Mon, Nov 26, 2018 at 2:35 PM André Warnier (tomcat) 
wrote:

> On 26.11.2018 13:29, Rémy Maucherat wrote:
> > On Sat, Nov 24, 2018 at 9:48 AM Ludovic Pénet  wrote:
> >
> >> Le vendredi 23 novembre 2018 à 23:51 +0100, Rémy Maucherat a écrit :
> >>> On Wed, Nov 21, 2018 at 10:58 AM Mark Thomas 
> >>> wrote:
> >>>
> >>>> - French has increased from 18% to 64% coverage
> >>>>
> >>>
> >>> Done (well, close enough, a few tribes/ha remain) !
> >> A single translation remains to be performed.
> >>
> >> Jump to https://poeditor.com/join/project/NUTIjDWzrl and be the one to
> >> complete the French translation. ;-)
> >>
> >
> > Ok, you could have finished it, I was busy.
> >
> > Now we can try to harmonize terms, fixes are then easy to do with the
> > search feature
> >
> > Common ones we have right now:
> > - "socket" (usually untranslated or cleverly omitted): ?
> > - "endpoint" (for websockets, and for the Tomcat connectors, so possibly
> > two different terms): "point d'entrée" ?
>
> That sounds like exactly the opposite of "endpoint" to me.
> Although I must say that even in English, the vocabulary used in some
> reference documents
> (in particular everything to do with XML-based protocols, such as SOAP,
> SAML, OASIS and
> the like) is sometimes mysterious and counter-intuitive.
> What about "cible" here ?
> Or more literally, "point final" ?
>

There are two contexts for it:
- The "NIO Endpoint" (or APR, etc) that is the backend of the Tomcat
connector, it accepts the sockets and deals with the low level stuff from
there
- The WebSocket endpoint javax.websocket.RemoteEndpoint
They can have a different word, actually.


>
> For "socket", "soquet" (like the piece in which you insert a plug, or a
> lightbulb) sounds
> ok to me.
>

Hum, ok, let's forget about this one.


>
> > - "thread" (often it is untranslated elsewhere): "fil d'exécution" ?
> > - "membership" (that's the clustering object): "gestionnaire de membres"
> ?
>
> "Membership" refers to "le fait d'être membre", no ? "adhésion" ?
> (like "cluster members" -> "adhérents au cluster" (with the appropriate
> French
> pronounciation for "cleustère") :-)
>

"adhérents" sounds good good and fits some most likely, "appartenance"
likely fits some others. I'll need to look in context.


>
> > - "dispatch"/"dispatcher" (for the Servlet request dispatcher): ?
> >
>
> dépêcher / dépêcheur ?
>

That "répartiteur" from Emmanuel sounds better in theory, will have to see
in context.


>
> > And I just saw it is really "connexion" and not "connection". Oooops, I
> > thought both were ok. I guess it's the same kind of mistake with
> English-UK
> > vs English-US, where I usually hate the UK style (except in HarryP and
> > Discworld, it's part of the charm I suppose).
> >
>
> Maybe a note : the target audience of most of these messages is not the
> members of the
> Académie or the jury of the Prix Goncourt. Its is programmers, sysadmins
> and qualified
> tomcat/webservers users.  The translations should be helpful to them, to
> get a first idea
> of the issue and be able to search later in the on-line documentation.
> Which happens to
> be only available/up-to-date/searchable in English, no ?
>
> So I believe that a translation such as "La requête PTHT recue sur le
> soquet du connecteur
> de toile a été dépêchée au conducteur du groupe d'adhérents" may be
> stylistically correct,
> but ultimately quite counter-productive.
>
> (Sorry for the missing c cédille, can't type it here)
> (PTHT = Protocol de Transport Hyper-Texte)
>

PTHT :D

So I was fine with "fil d'exécution" but it's more complex too than
untranslated. So "thread/socket" it is.


>
> This being said, all these translations leave out what is really the main
> theme here :
> tomcat. So what about a new name too ? what about "matou" ?
> Or does this require a fourchette ?
>

+1 for "Apache Matou" since it's so funny :) We need to apply for a
trademark asap.

Rémy


Re: Translations update

2018-11-26 Thread Rémy Maucherat
On Sat, Nov 24, 2018 at 9:48 AM Ludovic Pénet  wrote:

> Le vendredi 23 novembre 2018 à 23:51 +0100, Rémy Maucherat a écrit :
> > On Wed, Nov 21, 2018 at 10:58 AM Mark Thomas 
> > wrote:
> >
> > > - French has increased from 18% to 64% coverage
> > >
> >
> > Done (well, close enough, a few tribes/ha remain) !
> A single translation remains to be performed.
>
> Jump to https://poeditor.com/join/project/NUTIjDWzrl and be the one to
> complete the French translation. ;-)
>

Ok, you could have finished it, I was busy.

Now we can try to harmonize terms, fixes are then easy to do with the
search feature

Common ones we have right now:
- "socket" (usually untranslated or cleverly omitted): ?
- "endpoint" (for websockets, and for the Tomcat connectors, so possibly
two different terms): "point d'entrée" ?
- "thread" (often it is untranslated elsewhere): "fil d'exécution" ?
- "membership" (that's the clustering object): "gestionnaire de membres" ?
- "dispatch"/"dispatcher" (for the Servlet request dispatcher): ?

And I just saw it is really "connexion" and not "connection". Oooops, I
thought both were ok. I guess it's the same kind of mistake with English-UK
vs English-US, where I usually hate the UK style (except in HarryP and
Discworld, it's part of the charm I suppose).

Rémy


Re: Translations update

2018-11-24 Thread Rémy Maucherat
On Sat, Nov 24, 2018 at 12:01 AM Mark Thomas  wrote:

> On 23/11/2018 22:51, Rémy Maucherat wrote:
> > On Wed, Nov 21, 2018 at 10:58 AM Mark Thomas  wrote:
> >
> >> - French has increased from 18% to 64% coverage
> >>
> >
> > Done (well, close enough, a few tribes/ha remain) !
> >
> > Or rather, the initial work is done, gradual proofreading, rewording and
> > harmonizing would be needed. But it's not too bad as search and replace
> is
> > easier than initial translation (IMO).
> >
> > And the source English strings are not exempt from that either, for
> example:
> > - many strings have big WARNING at first: it should be removed as it
> > duplicates the log level of the logger
> > - similar: message location info, like the thing occurred in FooBarClass,
> > which well, is probably going to be the log category
>

Explanation for Jasper: it uses the ServletContext "logger" (I had
forgotten), and it only has a generic log method which logs either as info
or error. However the "WARN" should probably not be i18n-ed to be able to
be further processed if needed and it shouldn't be part of the String, it
should be "WARN: " + i18n.


> > - debug with lots of random variables like in Tribes/HA, this shouldn't
> > have i18n
> > - some random digressions, IMO it should be kept to the point and avoid
> > many sentences
> >
> > Is it possible to edit English directly in POEditor, or should it be done
> > in svn/git ?
>
> We can edit the English directly. We just need to be careful about
> keeping POEditor and svn in sync.
>
> Let me check how in sync they are at the moment for English...
>
> Look to be 100% in sync so edit away in POEditor if that is easier.
> Worth mentioning your plans on dev@ in case anyone is thinking of
> updating the terms and/or English values.
>

Ok, nice.

Rémy


Re: Translations update

2018-11-23 Thread Rémy Maucherat
On Wed, Nov 21, 2018 at 10:58 AM Mark Thomas  wrote:

> - French has increased from 18% to 64% coverage
>

Done (well, close enough, a few tribes/ha remain) !

Or rather, the initial work is done, gradual proofreading, rewording and
harmonizing would be needed. But it's not too bad as search and replace is
easier than initial translation (IMO).

And the source English strings are not exempt from that either, for example:
- many strings have big WARNING at first: it should be removed as it
duplicates the log level of the logger
- similar: message location info, like the thing occurred in FooBarClass,
which well, is probably going to be the log category
- debug with lots of random variables like in Tribes/HA, this shouldn't
have i18n
- some random digressions, IMO it should be kept to the point and avoid
many sentences

Is it possible to edit English directly in POEditor, or should it be done
in svn/git ?

Rémy


Re: Translations update

2018-11-22 Thread Rémy Maucherat
On Wed, Nov 21, 2018 at 10:58 AM Mark Thomas  wrote:

> - Simplified Chinese has been added and has already reached 32% coverage
>

There's actually a problem with the Chinese translation, it's been deleted
for some reason.

Rémy


Re: Translation help wanted

2018-11-13 Thread Rémy Maucherat
On Tue, Nov 13, 2018 at 4:39 PM André Warnier (tomcat) 
wrote:

> On 13.11.2018 13:32, Rémy Maucherat wrote:
> > On Mon, Nov 12, 2018 at 12:49 PM Mark Thomas  wrote:
> >
> >> I'm aiming to export the translations on a regular basis to the Tomcat
> >> source code. How regularly will depend on the rate of new/updated
> >> translations but as a minimum, I'm aiming to get any updates into the
> >> next Tomcat 9 release.
> >>
> >
> > Ok. Could you remove "French (MC)" ? No idea where it comes from but in
> > Monaco they do standard French, and it's too small anyway :D
> >
>
> You could add "French (B)" though.  There's more of us than Monégasques,
> and we have some
> words all our own, for instance in the numbers category. I'm sure the
> Swiss would agree.
>
> Just kidding..
>

Users can add languages, so no problem :)

Rémy


  1   2   3   >