Re: No thread name in AccessLogValve printed

2020-10-20 Thread Mark Thomas
On 20/10/2020 10:10, Michael Osipov wrote:
> Folks,
> 
> I have seen recently entried like this on our access logs:
>> 2020-10-19T20:00:05.591 [null] xyz - "-" 400 - 0
>> 2020-10-19T20:00:05.591 [null] abc- "-" 400 - 0
>> 2020-10-19T20:00:05.592 [null] abc - "-" 400 - 0
>> 2020-10-19T20:00:05.593 [null] abc - "-" 400 - 0
>> 2020-10-19T20:00:05.616 [null] abc - "-" 400 - 0
> 
> with pattern:
>> %{-MM-dd'T'HH:mm:ss.SSS}t [%I] %h %u %r %s %b %D
> 
> While I am quite certain that these ary "security" scans at work
> I wonder why RequestInfo#getWorkerThreadName() is null.
> Is the request rejected because it is malformed because before it is
> handled to a worker?

Have you tried looking at the source code? Callers of
RequestInfo.setWorkerThreadName() are likely to be enlightening.

> I am not sure whether "info != null &&
> info.getgetWorkerThreadName() != null" would be the right change here.

That would address the symptom. There may be a better fix that addresses
(or at least gets closer) to the cause.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Weirdest Tomcat Behavior Ever?

2020-10-16 Thread Mark Thomas
On 16/10/2020 12:37, Eric Robinson wrote:
>> From: Mark Thomas 



>> I'd like to see those screen shots please. Better still would be access to 
>> the
>> captures themselves (just the relevant connections not the whole thing). I
>> believe what you are telling us but long experience tells me it is best to
>> double check the original data as well.
>>
> 
> I'll send you a link to the screen shot first, then I'll package up the 
> captures and send a link to that in a bit. As the files may contain somewhat 
> sensitive information, I'll send a secure mail direct to your inbox.

Thanks. The screenshots didn't shed any light on this so far.

>> I have observed something similar ish in the CI systems. In that case it is 
>> the
>> requests that disappear. Client side logging shows the request was made but
>> there is no sign of it ever being received by Tomcat. I don't have network
>> traces for that (yet) so I'm not sure where the data is going missing.
>>
>> I am beginning to suspect there is a hard to trigger Tomcat or JVM bug here. 
>> I
>> think a Tomcat bug is more likely although I have been over the code several
>> times and I don't see anything.
>>
> 
> I'm thinking a bug of some kind, too, but I've been hosting about 1800 
> instances of tomcat for 15 years and I have never seen this behavior before.
> 
>> A few more questions:
>>
> 
> This is where I will begin to struggle bit.
> 
>> Which HTTP connector are you using? BIO, NIO or APR/Native?
>>
> 
> I believe BIO is the default? server.xml just says...
> 
> connectionTimeout="2"
>redirectPort="8443" />

That will be BIO or APR/Native depending on whether you have Tomcat
Native installed. If you look at the logs for when Tomcat starts you
should see something like:

INFO: Initializing ProtocolHandler ["http-bio-3016"]
or
INFO: Initializing ProtocolHandler ["http-apr-3016"]

What do you see between the square brackets?

>> Is the issue reproducible if you switch to a different connector?
>>
> 
> In 15 years of using tomcat in production, we've never tried switching the 
> connector type. (Probably because the app vendor never suggested it.) I did a 
> little research and I'm beginning to think about the pros/cons.

If you wanted to try this, I'd recommend:

protocol="org.apache.coyote.http11.Http11NioProtocol"

>> How easy is it for you to reproduce this issue?
>>
> 
> It's not reproducible at will but it happens frequently enough that we don't 
> have to wait long for it to happen. I have wireshark capturing to disk 
> continuously and rotating the files at 10 minute intervals to keep them 
> smallish. Then I just tail the logs and wait.

Ack.

>> How are you linking the request you see in the access log with the request
>> you see in Wireshark?
> 
> Aside from the timestamp of the packets and the timestamp of the tomcat log 
> messages, each HTTP request also contains a high-resolution timestamp and a 
> unique random number. That way, even if the same request occurs multiple 
> times in rapid succession, we can still isolate the exact one that failed.

Excellent.

>> How comfortable are you running a patched version of Tomcat (drop class
>> files provided by me into $CATALINA_BASE/lib in the right directory structure
>> and restart Tomcat)? Just thinking ahead about collecting additional debug
>> information.
> 
> That would be a tricky in our production environment, but the users are 
> getting desperate enough that we'd be willing to explore that approach.

Understood.

Some other questions that have come to mind:

- Has this app always had this problem?

- If not, when did it start and what changed at that point (JVM version,
Tomcat version etc)

- I notice the the requests in the screenshots aren't using HTTP
keep-alive. Do you know why?

- How heavily loaded is the system (typical requests per second etc)?

- I'm assuming there is nothing in the log files when the failed request
happens. Is this correct?

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Weirdest Tomcat Behavior Ever?

2020-10-16 Thread Mark Thomas
On 16/10/2020 10:05, Eric Robinson wrote:
> Hi Mark --
> 
> Those are great questions. See answers below.
> 
> 
>> -Original Message-
>> From: Mark Thomas 
>> Sent: Friday, October 16, 2020 2:20 AM
>> To: users@tomcat.apache.org
>> Subject: Re: Weirdest Tomcat Behavior Ever?
>>
>> On 16/10/2020 00:27, Eric Robinson wrote:
>>
>> 
>>
>>> The localhost_access log shows a request received and an HTTP 200
>> response sent, as follows...
>>>
>>> 10.51.14.133 [15/Oct/2020:12:52:45 -0400] 57 GET
>>> /app/code.jsp?gizmoid=64438=5=2020-10-
>> 15
>>>
>> lterId=0=0=71340=321072
>> e
>>> ssid=40696=0.0715816=15102020125245.789063 HTTP/1.0
>>> ?gizmoid=64438=5=2020-10-
>> 15=0
>>>
>> ionDID=0=71340=321072=40696&
>> rn
>>> d2=0.0715816=15102020125245.789063 200
>>>
>>> But WireShark shows what really happened. The server received the GET
>> request, and then it sent a FIN to terminate the connection. So if tomcat 
>> sent
>> an HTTP response, it did not make it out the Ethernet card.
>>>
>>> Is this the weirdest thing or what? Ideas would sure be appreciated!
>>
>> I am assuming there is a typo in your Java version and you are using Java 8.
>>
> 
> Yes, Java 8.
> 
>> That Tomcat version is over 3.5 years old (and Tomcat 7 is EOL in less than 6
>> months). If you aren't already planning to upgrade (I'd suggest to 9.0.x) 
>> then
>> you might want to start thinking about it.
>>
> 
> Vendor constraint. It's a canned application published by a national software 
> company, and they have not officially approved tomcat 8 for use on Linux yet.
> 
>> I have a few ideas about what might be going on but rather than fire out
>> random theories I have some questions that might help narrow things down.
>>
>> 1. If this request was successful, how big is the response?
>>
> 
> 1035 bytes.
> 
>> 2. If this request was successful, how long would it typically take to
>> complete?
>>
> 
> Under 60 ms.
> 
>> 3. Looking at the Wireshark trace for a failed request, how long after the 
>> last
>> byte of the request is sent by the client does Tomcat send the FIN?
>>
> 
> Maybe 100 microseconds.
> 
>> 4. Looking at the Wireshark trace for a failed request, is the request fully 
>> sent
>> (including terminating CRLF etc)?
>>
> 
> Yes, the request as seen by the tomcat server is complete and is terminated 
> by 0D 0A.
> 
>> 5. Are there any proxies, firewalls etc between the user agent and Tomcat?
>>
> 
> User agent -> firewall -> nginx plus -> upstream tomcat servers
> 
>> 6. What timeouts are configured for the Connector?
>>
> 
> Sorry, which connector are you referring to?
> 
>> 7. Is this HTTP/1.1, HTTP/2, AJP, with or without TLS?
>>
> 
> HTTP/1.1
> 
>> 8. Where are you running Wireshark? User agent? Tomcat? Somewhere
>> else?
> 
> On the nginx proxy and both upstream tomcat servers. (On the user agent, too, 
> but that doesn't help us in this case.)
> 
> If you would like to see a screen shot showing all 4 captures side-by-size, I 
> can send you a secure link. It will verify my answers above. It shows 4 
> separate WireShark captures taken simultaneously:
> 
> (a) the request going from the nginx proxy to tomcat 1
> (b) tomcat 1 receiving the request and terminating the connection
> (c) nginx sending the request to tomcat 2
> (d) tomcat 2 replying to the request (but the reply does not help the user 
> because the tomcat server does not recognize the user agent's JSESSIONID 
> cookie, so it responds "invalid session."

Hmm.

That rules out most of my ideas.

I'd like to see those screen shots please. Better still would be access
to the captures themselves (just the relevant connections not the whole
thing). I believe what you are telling us but long experience tells me
it is best to double check the original data as well.

I have observed something similar ish in the CI systems. In that case it
is the requests that disappear. Client side logging shows the request
was made but there is no sign of it ever being received by Tomcat. I
don't have network traces for that (yet) so I'm not sure where the data
is going missing.

I am beginning to suspect there is a hard to trigger Tomcat or JVM bug
here. I think a Tomcat bug is more likely although I have been over the
code several times and I don't see anything.

A few more questions:

Which HTTP connector are you using? BIO, NIO or APR/Native?

Is the issue reproducible if you switch to

Re: Weirdest Tomcat Behavior Ever?

2020-10-16 Thread Mark Thomas
On 16/10/2020 00:27, Eric Robinson wrote:



> The localhost_access log shows a request received and an HTTP 200 response 
> sent, as follows...
> 
> 10.51.14.133 [15/Oct/2020:12:52:45 -0400] 57 GET 
> /app/code.jsp?gizmoid=64438=5=2020-10-15=0=0=71340=321072=40696=0.0715816=15102020125245.789063
>  HTTP/1.0 
> ?gizmoid=64438=5=2020-10-15=0=0=71340=321072=40696=0.0715816=15102020125245.789063
>  200
> 
> But WireShark shows what really happened. The server received the GET 
> request, and then it sent a FIN to terminate the connection. So if tomcat 
> sent an HTTP response, it did not make it out the Ethernet card.
> 
> Is this the weirdest thing or what? Ideas would sure be appreciated!

I am assuming there is a typo in your Java version and you are using Java 8.

That Tomcat version is over 3.5 years old (and Tomcat 7 is EOL in less
than 6 months). If you aren't already planning to upgrade (I'd suggest
to 9.0.x) then you might want to start thinking about it.

I have a few ideas about what might be going on but rather than fire out
random theories I have some questions that might help narrow things down.

1. If this request was successful, how big is the response?

2. If this request was successful, how long would it typically take to
complete?

3. Looking at the Wireshark trace for a failed request, how long after
the last byte of the request is sent by the client does Tomcat send the FIN?

4. Looking at the Wireshark trace for a failed request, is the request
fully sent (including terminating CRLF etc)?

5. Are there any proxies, firewalls etc between the user agent and Tomcat?

6. What timeouts are configured for the Connector?

7. Is this HTTP/1.1, HTTP/2, AJP, with or without TLS?

8. Where are you running Wireshark? User agent? Tomcat? Somewhere else?

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Virtual event focussed on Tomcat Security

2020-10-15 Thread Mark Thomas
On 29/09/2020 12:25, Mark Thomas wrote:
> Hi all,
> 
> We (the Tomcat community) have some funding from Google to help us
> improve Tomcat security. Our original plan was to use the funding to
> support an in-person security focussed hackathon. As you would expect,
> those plans are on hold for now. We would, therefore, like to explore
> the possibility of doing something virtually.
> 
> The purpose of this email is to gather input from the community about
> what such an event should look like. With that input we can put together
> a plan for the event. So, over to you. What would your ideal virtual
> event focussed on Tomcat Security look like?

Summarising the suggestions so far:
- application security / OWASP
- making HTTP requests *from* Tomcat
 - SSO / SAML / OpenIDConnect

The first two are more application security focussed and would not have
to be Tomcat specific.

The third is more likely to Tomcat specific depending on the extent to
which the SSO mechanism ties into Tomcat's internals.

All the suggestions so far have been for conference like presentations
(if I am reading them correctly).

Other possibilities:
- hackathon to implement (with support from committers) new security
  features (no idea what these might be - suggestions welcome)

- hackathon to run $tool_of_choice against Tomcat code base, review the
  results and fix (with committer support) those that need fixing.
  Suggestions as to tools to use welcome*

Anything else you'd like to suggest that is related to Tomcat and security.

There hasn't been any thought given to timing yet.

Mark



* I'll note that over the years most if not all of the major static
analysis tools have been run against the Tomcat code base and the
results have been very heavy on the false positives. Most of the work is
likely to be separating the few useful results from a lot of noise.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Custom protocol implementation not found

2020-10-14 Thread Mark Thomas
On 14/10/2020 10:38, Maarten van den Broek wrote:
> I use tomcat 9.0.33 with windows10 home and amazon corretto jdk1.8.0_212.



> Using the first Connector everything is working fine. Debugging the
> setKeystorePass method of the class
> nl.messagedesign.tomcatlib.EncryptedPassword_Http11Nio2Protocol in the
> protocol attribute shows that the encrypted password gets decrypted.
> 
> Using the second connector with the SSLHostConfig element instead of the
> deprecated attributes debugging shows that the setKeystorePass method is
> not called and I get errors for the incorrect password of the keystore.
> 
> What am I doing wrong in migrating to the configuration with the
> SSLHostConfig element?

I do wonder a) what risk(s) you are attempting to mitigate with this and
b) where that custom connector obtains the necessary pass-phrase to
decrypt the supplied value.

I am assuming you have extended the existing Http11Nio2Protocol
implementation and over-ridden setKeystorePass() as that won't get
called when an SSLHostConfig element is explicitly configured.

Based on the assumptions above, the following approach should work:
- override init()
- iterate over the results of findSslHostConfigs()
- for each SSLHostConfig instance
  - call getCertificateKeystorePassword()
  - decrypt it
  - call setCertificateKeystorePassword()

If you have multiple certificates per host you'll need to iterate over
the nested SSLHostConfigCertificate instances rather than use the
short-cut methods above that work with the default certificate instance.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Deploying war, Negative Date exception

2020-10-12 Thread Mark Thomas
On 12/10/2020 18:48, Christopher Schultz wrote:



 > There is already a check for -1 for the last-modified time for the file
> in the ZIP archive. Also for 0 for some reason (sorry, Brian Kernighan,
> you can't store your first file in a ZIP file!).

I've tracked that change back to this:
https://bz.apache.org/bugzilla/show_bug.cgi?id=33636

I don't see why "0" is not allowed (but neither do I see not allowing it
being a significant problem).

> I think that any negative value should be ignored, so the expanded-file
> on the disk should get "now" as its last-updated time. So just change
> the comparison to:
> 
>   if(lastModified > 0) {
> expandedFile.setLastModified(lastModified);
>   }

Hmm. I do wonder about silently swallowing what is almost certainly some
some of build system error (and breaking caching as per BZ 33636) when
lastModified is negative.

Ignoring files that return -1 (not specified) seems reasonable.

I think an exception is better than silently swallowing here.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)

2020-10-12 Thread Mark Thomas
On 12/10/2020 08:02, Arshiya Shariff wrote:
> Hi Mark , 
> 
> The issue is reproduced with version 9.0.39 as well. Max threads in Tomcat is 
> 200.
> 
> Please find the case:
> Client:JMeter 5.2.1 (With http2 plugin)
> TPS: around 20
> No of users from JMeter : 700
> Message payload size: 6 KB to 34 KB
> Loop: Infinite 
> We let the loop run infinitely and see the java.lang.StackOverflowError trace 
> printed multiple times in the log within few minutes of starting the test.

POSTing to what URL?

What does that URL do with the POSTed content? Ignore it? Read it from
an InputStream? Read it via getParameter()?

Is JMeter run on the same machine as Tomcat?

Do you use the JMeter GUI or the command line?

What are the specs of the server(s) being used?

You need to provide the exact steps to recreate this issue on a clean
install of Tomcat 9.0.39 as provided by the ASF.

Mark


> Please help us with this . What is the impact of StackOverflowError ?
> 
> Thanks and Regards
> Arshiya Shariff
> 
> -Original Message-
> From: Mark Thomas  
> Sent: Friday, October 9, 2020 5:31 PM
> To: users@tomcat.apache.org
> Subject: Re: HTTP2: memory filled up fast on increasing the connections to 
> 1000/2000 (Embedded tomcat 9.0.38)
> 
> On 09/10/2020 12:32, Arshiya Shariff wrote:
>> Hi,
>>
>> Mark , with the test runs that I performed over clean 9.0.x branch I was not 
>> able to reproduce this.
> 
> Good. But I'd really like to understand why...
> 
>> But with 9.0.38 and the jars built from 9.0.x with hash: 
>> c8ec2d4cde3a31b0e9df9a30e7915d77ba725545  , with 700 or 1000 users 
>> (connections) and on sending 1000 Requests per second (or even lesser) , 
>> payload of 16K  from JMeter I can see that this Exception occurs within few 
>> minutes of starting the test . The maxThreads configured in tomcat is 200 .
>>
>> How often do you see these errors in your test run?
>> Randomly, at times 2 or 3 such traces.
> 
> OK. Definitely a timing issue then.
> 
>> Do you have the other end of that stack trace?
>> It is only the two lines that is recursively printed till the end about  
>> ~500 times in one trace  :
>> at 
>> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
>> at 
>> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandl
>> er.completed(SocketWrapperBase.java:1100)
> 
> Doesn't tell me much unfortunately.
> 
>> I see the trace starting with :
>> Exception in thread "http-nio-x.y.z-1090-exec-107" 
>> java.lang.StackOverflowError 
>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:446)
>> at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174)
>> at 
>> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468)
>> at 
>> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandl
>> er.completed(SocketWrapperBase.java:1100)
>>
>>  (OR)
>>
>> Exception in thread "http-nio-x.y.z-1090-exec-87" 
>> java.lang.StackOverflowError
>> at sun.nio.ch.IOVecWrapper.get(IOVecWrapper.java:96)
>> at sun.nio.ch.IOUtil.read(IOUtil.java:240)
>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:440)
>> at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174)
>> at 
>> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468)
>> at 
>> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
>> at 
>> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
>> at 
>> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
>> .
>> .
>> .
>> .
>> at 
>> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
>> at 
>> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
>> at 
>> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
>> at 
>> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandl
>> er.completed(SocketWrapperBase.java:1100)
>>
>> Is there anything that was fixed around this in latest 9.0.x branch ?
> 

Re: Deploying war, Negative Date exception

2020-10-12 Thread Mark Thomas
On 12/10/2020 13:53, Mark Thomas wrote:
> On 12/10/2020 12:49, Mark Thomas wrote:
>> On 12/10/2020 12:19, Peter Henderson wrote:
>>> Hello fellow tomcat users.
>>>
>>> My environment.
>>> Tomcat: 9.0.39
>>> Java: openjdk 11.0.8 2020-07-14
>>> OS: Ubuntu 18.04.5 LTS
>>>
>>> Source code [0]
>>>
>>> When deploying this war [1], by copying it into the webapps directory,
>>> I get this exception. [2]
>>> java.lang.IllegalArgumentException: Negative time
>>>
>>>
>>> I only started seeing this exception when I upgraded my projects build tool
>>> version
>>> from
>>> sbt.version=1.3.10
>>> to
>>> sbt.version=1.4.0
>>>
>>>
>>> Is this a tomcat bug, a build tool bug or most likely something I'm doing
>>> wrong?
>>
>> Looks like an issue with the dates of files in the WAR.
>>
>> If you look at the dates of the files in the WAR nearly all of them are
>> in the future (07 Feb 2106, 06:28).
>>
>> It looks like something is overflowing but a Java long shouldn't do that.
>>
>> Need to dig into exactly what is going on as this looks like it should
>> work - even if the WAR contains files created almost a century in the
>> future.
> 
> Hmm. I see the 2106 date on the file system and with Java 8 but with
> Java 11 I see 1969-dec-31 23:00 which gives -360 which triggers the
> exception.
> 
> The root cause appears to be in the JRE at this point. Whether it is in
> the JRE used to create the WAR or the JRE reading the WAR is TBD.
> 
> I think I am going to have to look at the raw bytes and the zip file
> spec to figure out where the root cause is.

That was fun.

Per the zip spec, the last modified time on those files is:

1969-Dec-31 23:00:00 UTC

i.e. 1 hour before the Epoch at 1970-Jan-01 00:00:00 UTC

It is stored as a signed 32-bit int (F1F0) which is -3600 (zip
timestamps are in seconds since the Epoch).

Java 8 reads this incorrectly as an unsigned int (4294963696) which,
when taken as seconds since Epoch, gives 2016-Feb-07 05:28:16 UTC.

(Incidently the archiver that ships with Ubuntu appears to make the same
error)

Java 11 reads this correctly but Java does not let you set times before
the Epoch so the exception is triggered.

The short version is that the modification times of the files in the WAR
are being set to "1969-Dec-31 23:00:00 UTC" which Java doesn't like.

Tomcat could handle this more gracefully but the end result will be the
same - the WAR isn't going to deploy. I'm not convinced it is worth
doing anything here.

It looks like the fix will be somewhere in the build system used to
create the WAR.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Deploying war, Negative Date exception

2020-10-12 Thread Mark Thomas
On 12/10/2020 12:49, Mark Thomas wrote:
> On 12/10/2020 12:19, Peter Henderson wrote:
>> Hello fellow tomcat users.
>>
>> My environment.
>> Tomcat: 9.0.39
>> Java: openjdk 11.0.8 2020-07-14
>> OS: Ubuntu 18.04.5 LTS
>>
>> Source code [0]
>>
>> When deploying this war [1], by copying it into the webapps directory,
>> I get this exception. [2]
>> java.lang.IllegalArgumentException: Negative time
>>
>>
>> I only started seeing this exception when I upgraded my projects build tool
>> version
>> from
>> sbt.version=1.3.10
>> to
>> sbt.version=1.4.0
>>
>>
>> Is this a tomcat bug, a build tool bug or most likely something I'm doing
>> wrong?
> 
> Looks like an issue with the dates of files in the WAR.
> 
> If you look at the dates of the files in the WAR nearly all of them are
> in the future (07 Feb 2106, 06:28).
> 
> It looks like something is overflowing but a Java long shouldn't do that.
> 
> Need to dig into exactly what is going on as this looks like it should
> work - even if the WAR contains files created almost a century in the
> future.

Hmm. I see the 2106 date on the file system and with Java 8 but with
Java 11 I see 1969-dec-31 23:00 which gives -360 which triggers the
exception.

The root cause appears to be in the JRE at this point. Whether it is in
the JRE used to create the WAR or the JRE reading the WAR is TBD.

I think I am going to have to look at the raw bytes and the zip file
spec to figure out where the root cause is.

Mark
=

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Deploying war, Negative Date exception

2020-10-12 Thread Mark Thomas
On 12/10/2020 12:19, Peter Henderson wrote:
> Hello fellow tomcat users.
> 
> My environment.
> Tomcat: 9.0.39
> Java: openjdk 11.0.8 2020-07-14
> OS: Ubuntu 18.04.5 LTS
> 
> Source code [0]
> 
> When deploying this war [1], by copying it into the webapps directory,
> I get this exception. [2]
> java.lang.IllegalArgumentException: Negative time
> 
> 
> I only started seeing this exception when I upgraded my projects build tool
> version
> from
> sbt.version=1.3.10
> to
> sbt.version=1.4.0
> 
> 
> Is this a tomcat bug, a build tool bug or most likely something I'm doing
> wrong?

Looks like an issue with the dates of files in the WAR.

If you look at the dates of the files in the WAR nearly all of them are
in the future (07 Feb 2106, 06:28).

It looks like something is overflowing but a Java long shouldn't do that.

Need to dig into exactly what is going on as this looks like it should
work - even if the WAR contains files created almost a century in the
future.

Mark




> 
> Thanks
> Peter.
> 
> 
> [0]
> https://github.com/bollinger/NegativeDate
> 
> [1]
> https://github.com/bollinger/NegativeDate/blob/master/Negative.war
> 
> [2]
> 2020-10-12 11:41:35.932 SEVERE oacs.HostConfig Error deploying web
> application archive [/home/peter/apache-tomcat-9.0.39/webapps/Negative.war]
> java.lang.IllegalStateException: Error starting child
> at
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:720)
> at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:690)
> at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:706)
> at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:978)
> at
> org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1848)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
> at
> java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
> at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:773)
> at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:427)
> at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1620)
> at
> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:305)
> at
> org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
> at
> org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1151)
> at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1353)
> at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1357)
> at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1335)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: org.apache.catalina.LifecycleException: Failed to start
> component
> [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/Negative]]
> at
> org.apache.catalina.util.LifecycleBase.handleSubClassException(LifecycleBase.java:440)
> at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:198)
> at
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:717)
> ... 24 more
> Caused by: java.lang.IllegalArgumentException: Negative time
> at java.base/java.io.File.setLastModified(File.java:1441)
> at org.apache.catalina.startup.ExpandWar.expand(ExpandWar.java:169)
> at
> org.apache.catalina.startup.ContextConfig.fixDocBase(ContextConfig.java:820)
> at
> org.apache.catalina.startup.ContextConfig.beforeStart(ContextConfig.java:958)
> at
> org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:305)
> at
> org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
> at
> org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423)
> at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:182)
> ... 25 more
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



[SECURITY] CVE-2020-13943 Apache Tomcat HTTP/2 Request mix-up

2020-10-12 Thread Mark Thomas
CVE-2020-13943 Apache Tomcat HTTP/2 Request mix-up

Severity: Moderate

Vendor: The Apache Software Foundation

Versions Affected:
Apache Tomcat 10.0.0-M1 to 10.0.0-M7
Apache Tomcat 9.0.0.M5 to 9.0.37
Apache Tomcat 8.5.1 to 8.5.57

Description:
If an HTTP/2 client exceeded the agreed maximum number of concurrent
streams for a connection (in violation of the HTTP/2 protocol), it was
possible that a subsequent request made on that connection could contain
HTTP headers - including HTTP/2 pseudo headers - from a previous request
rather than the intended headers. This could lead to users seeing
responses for unexpected resources.

Mitigation:
- Upgrade to Apache Tomcat 10.0.0-M8 or later
- Upgrade to Apache Tomcat 9.0.38 or later
- Upgrade to Apache Tomcat 8.5.58 or later

Credit:
This issue was identified by the Apache Tomcat Security Team.

References:
[1] http://tomcat.apache.org/security-10.html
[2] http://tomcat.apache.org/security-9.html
[3] http://tomcat.apache.org/security-8.html

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



[ANN] Apache Tomcat 8.5.59 available

2020-10-12 Thread Mark Thomas
The Apache Tomcat team announces the immediate availability of Apache
Tomcat 8.5.59.

Apache Tomcat 8 is an open source software implementation of the Java
Servlet, JavaServer Pages, Java Unified Expression Language, Java
WebSocket and Java Authentication Service Provider Interface for
Containers technologies.

Apache Tomcat 8.5.x replaces 8.0.x and includes new features pulled
forward from the 9.0.x branch. The notable changes since 8.5.58 include:

- Refactor the handling of closed HTTP/2 streams to reduce the heap
  usage associated with used streams and to retain information for more
  streams in the priority tree.

- Deprecate the JDBCRealm.

- Ensure that none of the methods on a ServletContext instance always
  fail when running under a SecurityManager. Pull request provided by
  Kyle Stiemann.

Please refer to the change log for the complete list of changes:
http://tomcat.apache.org/tomcat-8.5-doc/changelog.html


Downloads:
http://tomcat.apache.org/download-80.cgi

Migration guides from Apache Tomcat 7.x and 8.0.x:
http://tomcat.apache.org/migration.html

Enjoy!

- The Apache Tomcat team


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



[ANN] Apache Tomcat 9.0.39 available

2020-10-12 Thread Mark Thomas
The Apache Tomcat team announces the immediate availability of Apache
Tomcat 9.0.39.

Apache Tomcat 9 is an open source software implementation of the Java
Servlet, JavaServer Pages, Java Unified Expression Language, Java
WebSocket and JASPIC technologies.

Apache Tomcat 9.0.39 is a bugfix and feature release. The notable
changes compared to 9.0.38 include:

- Refactor the handling of closed HTTP/2 streams to reduce the heap
  usage associated with used streams and to retain information for more
  streams in the priority tree.

- Allow using the utility executor for annotation scanning. Patch
  provided by Jatin Kamnani.

- Add a bloom filter to speed up archive lookup and improve deployment
  speed of applications with a large number of JARs. Patch provided by
  Jatin Kamnani.

Please refer to the change log for the complete list of changes:
http://tomcat.apache.org/tomcat-9.0-doc/changelog.html


Downloads:
http://tomcat.apache.org/download-90.cgi

Migration guides from Apache Tomcat 7.x and 8.x:
http://tomcat.apache.org/migration.html

Enjoy!

- The Apache Tomcat team


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



[ANN] Apache Tomcat 10.0.0-M9 available

2020-10-12 Thread Mark Thomas
The Apache Tomcat team announces the immediate availability of Apache
Tomcat 10.0.0-M9.

Apache Tomcat 10 is an open source software implementation of the
Jakarta Servlet, Jakarta Server Pages, Jakarta Expression Language,
Jakarta WebSocket, Jakarta Authentication and Jakarta Annotations
specifications.

Users of Tomcat 10 onwards should be aware that, as a result of the move
from Java EE to Jakarta EE as part of the transfer of Java EE to the
Eclipse Foundation, the primary package for all implemented APIs has
changed from javax.* to jakarta.*. This will almost certainly require
code changes to enable applications to migrate from Tomcat 9 and earlier
to Tomcat 10 and later. A migration tool is under development to aid
this process.

Apache Tomcat 10.0.0-M9 is a milestone release of the 10.0.x
branch and has been made to provide users with early access to the new
features in Apache Tomcat 10.0.x so that they may provide feedback. The
notable changes compared to 10.0.0-M8 include:

- Refactor the handling of closed HTTP/2 streams to reduce the heap
  usage associated with used streams and to retain information for more
  streams in the priority tree.

- Allow using the utility executor for annotation scanning. Patch
  provided by Jatin Kamnani.

- Add a bloom filter to speed up archive lookup and improve deployment
  speed of applications with a large number of JARs. Patch provided by
  Jatin Kamnani.

Please refer to the change log for the complete list of changes:
http://tomcat.apache.org/tomcat-10.0-doc/changelog.html

Downloads:
http://tomcat.apache.org/download-10.cgi

Migration guides from Apache Tomcat 7.0.x, 8.5.x and 9.0.x:
http://tomcat.apache.org/migration.html

Enjoy!

- The Apache Tomcat team

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: java.lang.OutOfMemoryError: PermGen space when we redeploy same application multiple times

2020-10-12 Thread Mark Thomas
On 11/10/2020 02:39, Prabhu Gurunathan wrote:
> Hi All,
> 
> We have an setup where we are using OpenJDK 1.7 and Tomcat 7.0.100 ,
> in CentOs 7 Env . and we have many application deployed in
> Tomcat/webapps and the common lib's needed for those apps are kept in
> Tomcat/lib directory like log4j , commons-fileupload  ,xerces , Xalan
> .. etc
> 
> The problem here is When we try to undeploy and deploy same
> applications multiple time we are  getting into
> java.lang.OutOfMemoryError: PermGen space very quickly . Want to know
> is it very generic problem on this deployment model or is this can be
> fixed anyway ?

The memory leak could be in any of:
- the application code
- a library the application depends on
- the JVM
- Tomcat

In all cases, it should be possible to fix it. There might even be a
short-term workaround available.

First of all, make sure that the JreMemoryLeakPreventionListener is
enabled in your configuration.

Secondly, make sure that you don't get any warnings about possible
memory leaks in the logs when you reload an application. If you do, fix
the leaks identified.

If you still see issues, the short version is:
- user a profiler
- reload each app in turn until you see more strong references to
  org.apache.catalina.loader.[Parallel]WebappClassLoader instances in
  memory than you have web applications
- find the one where started = false
- trace its GC roots
- that will tell you where the memory leak is
- how it was created might be harder to track down

The long version is in a presentation linked from the Tomcat web site:
https://home.apache.org/~markt/presentations/2010-08-05-Memory-Leaks-JavaOne-60mins.pdf

If you have any questions, you can ask here.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: tc-embed-9.0.38 : module does not declare uses

2020-10-09 Thread Mark Thomas
On 09/10/2020 16:38, marc.davenp...@oracle.com wrote:
> Hello all,
> 
> I'm trying to upgrade from 9.0.35 to 9.0.38.  I know that explicit
> module definitions were added between .37 & .38.  I'm just trying to
> shake out the changes needed on our end to use it. It's my tentative
> grasp on proper use of modules, but I could use some help. Now when we
> are starting our embedded tomcat, I am getting an error as we
> instantiate and configure the AccessLogValve.
> 
> Exception in thread "main" java.util.ServiceConfigurationError:
> org.apache.juli.logging.Log: module org.apache.tomcat.embed.core does
> not declare `uses`

https://bz.apache.org/bugzilla/show_bug.cgi?id=64751

Fixed in 9.0.39 which looks as if it will be released very (hours) shortly.

Mark


>     at java.base/java.util.ServiceLoader.fail(Unknown Source)
>     at java.base/java.util.ServiceLoader.checkCaller(Unknown Source)
>     at java.base/java.util.ServiceLoader.(Unknown Source)
>     at java.base/java.util.ServiceLoader.load(Unknown Source)
>     at
> org.apache.tomcat.embed.core@9.0.38/org.apache.juli.logging.LogFactory.(LogFactory.java:89)
> 
>     at
> org.apache.tomcat.embed.core@9.0.38/org.apache.juli.logging.LogFactory.(LogFactory.java:66)
> 
>     at
> org.apache.tomcat.embed.core@9.0.38/org.apache.catalina.util.LifecycleBase.(LifecycleBase.java:39)
> 
>     at our/Server.start(OurServer.java:104) -> new AccessLogValve().
> 
> We are running against a custom jre made by jlink which includes the
> following mods which might be pertinent.
> 
> /mods/commons-logging-1.2.jar
> ...
> /mods/slf4j-api-1.8.0-beta4.jar
> /mods/slf4j-log4j-100.0.0.0.0-SNAPSHOT.jar (shaded jar)
> ...
> /mods/tomcat-annotations-api-9.0.38.jar
> /mods/tomcat-embed-core-9.0.38.jar
> /mods/tomcat-embed-el-9.0.38.jar
> /mods/tomcat-embed-jasper-9.0.38.jar
> 
> I'm not sure what the implication of this error message is and any help
> would be appreciated.
> 
> Thank you,
> 
> Marc
> 
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)

2020-10-09 Thread Mark Thomas
On 09/10/2020 12:32, Arshiya Shariff wrote:
> Hi, 
> 
> Mark , with the test runs that I performed over clean 9.0.x branch I was not 
> able to reproduce this.

Good. But I'd really like to understand why...

> But with 9.0.38 and the jars built from 9.0.x with hash: 
> c8ec2d4cde3a31b0e9df9a30e7915d77ba725545  , with 700 or 1000 users 
> (connections) and on sending 1000 Requests per second (or even lesser) , 
> payload of 16K  from JMeter I can see that this Exception occurs within few 
> minutes of starting the test . The maxThreads configured in tomcat is 200 .
> 
> How often do you see these errors in your test run?
> Randomly, at times 2 or 3 such traces.

OK. Definitely a timing issue then.

> Do you have the other end of that stack trace?
> It is only the two lines that is recursively printed till the end about  ~500 
> times in one trace  :
> at 
> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
> at 
> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)

Doesn't tell me much unfortunately.

> I see the trace starting with :
> Exception in thread "http-nio-x.y.z-1090-exec-107" 
> java.lang.StackOverflowError 
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:446)
> at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174)
> at 
> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468)
> at 
> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
> 
>   (OR)
> 
> Exception in thread "http-nio-x.y.z-1090-exec-87" java.lang.StackOverflowError
> at sun.nio.ch.IOVecWrapper.get(IOVecWrapper.java:96)
> at sun.nio.ch.IOUtil.read(IOUtil.java:240)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:440)
> at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174)
> at 
> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468)
> at 
> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
> at 
> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
> at 
> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
> .
> .
> .
> .
> at 
> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
> at 
> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
> at 
> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
> at 
> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
> 
> Is there anything that was fixed around this in latest 9.0.x branch ?

Not obviously. I've reviewed every commit since c8ec2d4c. There is
nothing that directly works with the I/O. There is 1e97ab2 which fixes a
relatively recent regression in the HTTP/2 code. I guess it is possible
(but it seems a bit of a stretch) that that bug is triggering an issue
in JMeter which in turn is sending invalid HTTP/2 packets.

I think at this point, given the relatively small number of commits
between c8ec2d4c and HEAD, the most useful thing you could do is run a
binary search to find out at which commit the issue is fixed. If we know
which commit to look at that should help track down the root cause.

Mark

> 
> Thanks and Regards
> Arshiya Shariff
> 
> -Original Message-
> From: Mark Thomas  
> Sent: Monday, October 5, 2020 9:52 PM
> To: users@tomcat.apache.org
> Subject: Re: HTTP2: memory filled up fast on increasing the connections to 
> 1000/2000 (Embedded tomcat 9.0.38)
> 
> On 05/10/2020 10:56, Arshiya Shariff wrote:
>> Hi All,
>>
>> Thank you so much Mark . 
>> We tested the jars built from latest 9.0.x  with 2000 / 5000 users 
>> (connections) from JMeter , We see a very good improvement with the 
>> heap usage
> 
> Good news. As is the fact that the other errors have been cleared up.
> 
>> But I see this exception printed multiple times , I am not sure why this 
>> occurs :
>> Exception in thread "http-nio-x.y.z-1234-exec-213" 
>> java.lang.StackOverflowError 
>> at sun.nio.ch.IOUtil.read(IOUtil.java:240)
>> at 

Re: Tomcat 9.0.37 Clustered DeltaManager Duplicates Session And Loses Session Attributes

2020-10-08 Thread Mark Thomas
On 08/10/2020 10:04, Tim N wrote:
> Hi,
> 
> I'm in the early stages of analysing this problem:
> 
>- Tomcat Embedded 9.0.37 with clustering enabled
>- SpringBoot application, 2.1.16
>- Login with no existing JSESSIONID fails
> 
> I can see the following code is executed twice within one request, the
> login attempt:
> 
> *~/.m2/repository/org/apache/tomcat/tomcat-catalina/9.0.37/tomcat-catalina-9.0.37-sources.jar!/org/apache/catalina/session/ManagerBase.java:677*
> 
> public void add(Session session) {
> sessions.put(session.getIdInternal(), session);
> 
> The first time it's executed, the session with the Spring context is added.
> The second time it's executed, a second session with the same ID, but
> without the Spring context, or any other session attribute I add for that
> matter, overwrites the existing session, and login fails. If I debug and
> prevent this by renaming the second session ID, login works because the
> original session is preserved.
> 
> The stack-trace for the first call is shown below:
> 
> add:678, ManagerBase (org.apache.catalina.session)
> setId:358, StandardSession (org.apache.catalina.session)
> setId:327, DeltaSession (org.apache.catalina.ha.session)
> setId:345, DeltaSession (org.apache.catalina.ha.session)
> createSession:719, ManagerBase (org.apache.catalina.session)
> createSession:422, DeltaManager (org.apache.catalina.ha.session)
> createSession:410, DeltaManager (org.apache.catalina.ha.session)
> doGetSession:3043, Request (org.apache.catalina.connector)
> getSession:2441, Request (org.apache.catalina.connector)



This is the app triggering the creation of the session.

> The stack-trace for the second call is shown below:
> 
> add:678, ManagerBase (org.apache.catalina.session)
> setId:358, StandardSession (org.apache.catalina.session)
> setId:327, DeltaSession (org.apache.catalina.ha.session)
> handleSESSION_CREATED:1322, DeltaManager (org.apache.catalina.ha.session)
> messageReceived:1192, DeltaManager (org.apache.catalina.ha.session)



This is the DeltaManager receiving notification that a new session has
been created.

> Any help would be appreciated. I can replicate this every time and spend
> some time investigating this.

The new session created message should be send to (and then processed
on) every node *except* the node on which the session was originally
created.

Can you show us how you configured this cluster please?

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: new module-info.class & tomcat-embed-programmatic.jar in 9.0.38 embed distribution.

2020-10-08 Thread Mark Thomas
On 07/10/2020 18:51, Mark Thomas wrote:
> 
> 
> On 07/10/2020 18:29, marc.davenp...@oracle.com wrote:
>> I'm doing a simple upgrade from 9.0.35 to 9.0.38 embed. I notice some
>> significant changes that I don't see listed in the changelog.
> 
> There are covered by the entry for bug 64540.
> 
>> Until 9.0.37 tomcat-embed-core.jar did not have a module-info. Now in
>> 9.0.38 it does.  This changes the modules I need to require when using
>> it. This may be the right way to be doing thing all along, but it does
>> feel like a significant change and not a simple upgrade with a version
>> bump.
>>
>> There is also a new tomcat-embed-programmatic.jar in the
>> apache-tomcat-9.0.38-embed.zip distribution but it does not seem to be
>> available alone in maven central:
>> https://mvnrepository.com/artifact/org.apache.tomcat.embed
> 
> I suspect the new JAR was added to the main build script but not the
> script that does the Maven upload.
> 
> That should be something we can fix for the November release round.

I should have spotted that you were using mvnrepository.com. That site
should not be considered reliable. It is often missing (recently?)
released JARs.

You should use the Maven Central repository at:
https://search.maven.org/

The tomcat-embed-programmatic.jar is available in that repo.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: new module-info.class & tomcat-embed-programmatic.jar in 9.0.38 embed distribution.

2020-10-08 Thread Mark Thomas
On 08/10/2020 09:20, Mark Thomas wrote:
> On 07/10/2020 20:55, marc.davenp...@oracle.com wrote:
>> On 10/7/20 2:50 PM, Mark Thomas wrote:
>>> On 07/10/2020 19:47, marc.davenp...@oracle.com wrote:
>>>> On 10/7/20 1:51 PM, Mark Thomas wrote:
>>>>> On 07/10/2020 18:29, marc.davenp...@oracle.com wrote:
>>>>>> I'm doing a simple upgrade from 9.0.35 to 9.0.38 embed. I notice some
>>>>>> significant changes that I don't see listed in the changelog.
>>>>> There are covered by the entry for bug 64540.
>>>> Ah, thank you for the clarification. I didn't understand the
>>>> implications of that note.
>>
>> The annotations-api.jar module-info is using the java namespace?
> 
> That is probably a typo. I'll take a look.

Nope. Just confirmed with the reference implementation release that the
JPMS name is "java.annotation"

Mark


> 
> Mark
> 
>>
>> open module java.annotation {
>>   requires java.base;
>>
>>   exports javax.annotation;
>>   exports javax.annotation.security;
>>   exports javax.annotation.sql;
>> }
>>
>>>>
>>>>>> Until 9.0.37 tomcat-embed-core.jar did not have a module-info. Now in
>>>>>> 9.0.38 it does.  This changes the modules I need to require when using
>>>>>> it. This may be the right way to be doing thing all along, but it does
>>>>>> feel like a significant change and not a simple upgrade with a version
>>>>>> bump.
>>>>>>
>>>>>> There is also a new tomcat-embed-programmatic.jar in the
>>>>>> apache-tomcat-9.0.38-embed.zip distribution but it does not seem to be
>>>>>> available alone in maven central:
>>>>>> https://urldefense.com/v3/__https://mvnrepository.com/artifact/org.apache.tomcat.embed__;!!GqivPVa7Brio!Mrj4iTWd6AiGf5VPU0EyhkGtFSZfsmaM_UC7lBc_eDHuZc2GpP9iUTE2VAucYNb_A5U$
>>>>>>
>>>>>>
>>>>> I suspect the new JAR was added to the main build script but not the
>>>>> script that does the Maven upload.
>>>>>
>>>>> That should be something we can fix for the November release round.
>>>> What is the intention for this new jar?
>>> I believe it is intended for use in environments (Graal) where you don't
>>> want to use reflection (i.e. standard XML configuration files) to
>>> configure Tomcat.
>>>
>>> Mark
>>>
>>> -
>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>>
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
> 
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: new module-info.class & tomcat-embed-programmatic.jar in 9.0.38 embed distribution.

2020-10-08 Thread Mark Thomas
On 07/10/2020 20:55, marc.davenp...@oracle.com wrote:
> On 10/7/20 2:50 PM, Mark Thomas wrote:
>> On 07/10/2020 19:47, marc.davenp...@oracle.com wrote:
>>> On 10/7/20 1:51 PM, Mark Thomas wrote:
>>>> On 07/10/2020 18:29, marc.davenp...@oracle.com wrote:
>>>>> I'm doing a simple upgrade from 9.0.35 to 9.0.38 embed. I notice some
>>>>> significant changes that I don't see listed in the changelog.
>>>> There are covered by the entry for bug 64540.
>>> Ah, thank you for the clarification. I didn't understand the
>>> implications of that note.
> 
> The annotations-api.jar module-info is using the java namespace?

That is probably a typo. I'll take a look.

Mark

> 
> open module java.annotation {
>   requires java.base;
> 
>   exports javax.annotation;
>   exports javax.annotation.security;
>   exports javax.annotation.sql;
> }
> 
>>>
>>>>> Until 9.0.37 tomcat-embed-core.jar did not have a module-info. Now in
>>>>> 9.0.38 it does.  This changes the modules I need to require when using
>>>>> it. This may be the right way to be doing thing all along, but it does
>>>>> feel like a significant change and not a simple upgrade with a version
>>>>> bump.
>>>>>
>>>>> There is also a new tomcat-embed-programmatic.jar in the
>>>>> apache-tomcat-9.0.38-embed.zip distribution but it does not seem to be
>>>>> available alone in maven central:
>>>>> https://urldefense.com/v3/__https://mvnrepository.com/artifact/org.apache.tomcat.embed__;!!GqivPVa7Brio!Mrj4iTWd6AiGf5VPU0EyhkGtFSZfsmaM_UC7lBc_eDHuZc2GpP9iUTE2VAucYNb_A5U$
>>>>>
>>>>>
>>>> I suspect the new JAR was added to the main build script but not the
>>>> script that does the Maven upload.
>>>>
>>>> That should be something we can fix for the November release round.
>>> What is the intention for this new jar?
>> I believe it is intended for use in environments (Graal) where you don't
>> want to use reflection (i.e. standard XML configuration files) to
>> configure Tomcat.
>>
>> Mark
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: new module-info.class & tomcat-embed-programmatic.jar in 9.0.38 embed distribution.

2020-10-07 Thread Mark Thomas
On 07/10/2020 19:47, marc.davenp...@oracle.com wrote:
> 
> On 10/7/20 1:51 PM, Mark Thomas wrote:
>>
>> On 07/10/2020 18:29, marc.davenp...@oracle.com wrote:
>>> I'm doing a simple upgrade from 9.0.35 to 9.0.38 embed. I notice some
>>> significant changes that I don't see listed in the changelog.
>> There are covered by the entry for bug 64540.
> 
> Ah, thank you for the clarification. I didn't understand the
> implications of that note.
> 
>>
>>> Until 9.0.37 tomcat-embed-core.jar did not have a module-info. Now in
>>> 9.0.38 it does.  This changes the modules I need to require when using
>>> it. This may be the right way to be doing thing all along, but it does
>>> feel like a significant change and not a simple upgrade with a version
>>> bump.
>>>
>>> There is also a new tomcat-embed-programmatic.jar in the
>>> apache-tomcat-9.0.38-embed.zip distribution but it does not seem to be
>>> available alone in maven central:
>>> https://urldefense.com/v3/__https://mvnrepository.com/artifact/org.apache.tomcat.embed__;!!GqivPVa7Brio!Mrj4iTWd6AiGf5VPU0EyhkGtFSZfsmaM_UC7lBc_eDHuZc2GpP9iUTE2VAucYNb_A5U$
>>>
>> I suspect the new JAR was added to the main build script but not the
>> script that does the Maven upload.
>>
>> That should be something we can fix for the November release round.
> 
> What is the intention for this new jar?

I believe it is intended for use in environments (Graal) where you don't
want to use reflection (i.e. standard XML configuration files) to
configure Tomcat.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: new module-info.class & tomcat-embed-programmatic.jar in 9.0.38 embed distribution.

2020-10-07 Thread Mark Thomas



On 07/10/2020 18:29, marc.davenp...@oracle.com wrote:
> I'm doing a simple upgrade from 9.0.35 to 9.0.38 embed. I notice some
> significant changes that I don't see listed in the changelog.

There are covered by the entry for bug 64540.

> Until 9.0.37 tomcat-embed-core.jar did not have a module-info. Now in
> 9.0.38 it does.  This changes the modules I need to require when using
> it. This may be the right way to be doing thing all along, but it does
> feel like a significant change and not a simple upgrade with a version
> bump.
> 
> There is also a new tomcat-embed-programmatic.jar in the
> apache-tomcat-9.0.38-embed.zip distribution but it does not seem to be
> available alone in maven central:
> https://mvnrepository.com/artifact/org.apache.tomcat.embed

I suspect the new JAR was added to the main build script but not the
script that does the Maven upload.

That should be something we can fix for the November release round.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: A lot of EOFs when using NIO2 with HTTP2

2020-10-06 Thread Mark Thomas
On 06/10/2020 10:35, Martin Grigorov wrote:
> On Tue, Oct 6, 2020 at 12:15 PM Martin Grigorov 
> wrote:
> 
>>
>>
>> On Tue, Oct 6, 2020 at 12:06 PM Mark Thomas  wrote:
>>
>>> On 06/10/2020 07:30, Martin Grigorov wrote:
>>>> Hi,
>>>>
>>>> I face an issue with the NIO2 protocol.
>>>> When I increase the request rate to more than 500 requests per second it
>>>> starts failing with:
>>>
>>> I'm unable to reproduce this with a 10.0.x build requesting a simple
>>> text file from the ROOT web app using NIO2 and Java 1.8.0_265.
>>>
>>> I see around 15k req/s and no failures.
>>>
>>
>> I use latest 9.0.x.
>> Let me try with non-embedded Tomcat!
>>
> 
> Same build of Tomcat started with ./bin/startup.sh has no such problems!
> It seems
> https://github.com/martin-g/http2-server-perf-tests/blob/master/java/tomcat/src/main/java/info/mgsolutions/tomcat/TomcatEmbedded.java
> either misses something or does something more than what conf/server.xml
> does.
> I've tried to minify server.xml by removing all listeners, valves and JNDI
> related stuff but it still works fine.

maxThreads="8" setting? That looks very low to me.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: A lot of EOFs when using NIO2 with HTTP2

2020-10-06 Thread Mark Thomas
On 06/10/2020 07:30, Martin Grigorov wrote:
> Hi,
> 
> I face an issue with the NIO2 protocol.
> When I increase the request rate to more than 500 requests per second it
> starts failing with:

I'm unable to reproduce this with a 10.0.x build requesting a simple
text file from the ROOT web app using NIO2 and Java 1.8.0_265.

I see around 15k req/s and no failures.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)

2020-10-05 Thread Mark Thomas
On 05/10/2020 10:56, Arshiya Shariff wrote:
> Hi All, 
> 
> Thank you so much Mark . 
> We tested the jars built from latest 9.0.x  with 2000 / 5000 users 
> (connections) from JMeter , We see a very good improvement with the heap 
> usage 

Good news. As is the fact that the other errors have been cleared up.

> But I see this exception printed multiple times , I am not sure why this 
> occurs :
> Exception in thread "http-nio-x.y.z-1234-exec-213" 
> java.lang.StackOverflowError 
> at sun.nio.ch.IOUtil.read(IOUtil.java:240)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:440)
> at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174)
> at 
> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468)
> at 
> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
> at 
> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
> at 
> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
> at 
> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
> at 
> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
> at 
> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)

That looks like an infinite loop reading an incoming frame.
New frames are read using a 9 byte buffer for the header and a 16k
buffer for the payload (since Tomcat sets this as the max frame size).

The loop is occurring because one of those buffers is simultaneously
both full and still has more data to read. That should not be possible
and I haven't yet been able to figure out how this is happening.

How easy is this to reproduce?

How often do you see these errors in your test run?

Do you have a reliable test case that reproduces this on a clean Tomcat
9.0.x build? If is, can you share the details?

Do you have the other end of that stack trace? I'm interested in how the
code enters the loop.

Thanks,

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)

2020-10-01 Thread Mark Thomas
On 30/09/2020 18:47, Martin Grigorov wrote:
> On Wed, Sep 30, 2020 at 7:47 PM Mark Thomas  wrote:
>> On 30/09/2020 16:17, Mark Thomas wrote:



>>> That is helpful. Looks like you have found a way to reproduce the buffer
>>> issues reported in https://bz.apache.org/bugzilla/show_bug.cgi?id=64710
>>
>> Can you share the command you used to trigger those errors please.
>>
> 
> The Vegeta command I used is:
> 
> jq -ncM '{"method": "POST", "url": "https://localhost:8080/testbed/plaintext;,
> "body":"payload=Some
> sdgggwwsdgssfshffheeepayload"
> | @base64, header: {"Content-Type":
> ["application/x-www-form-urlencoded"]}}' | vegeta attack -format=json
> -http2 -rate=1000 -max-workers=8 -insecure -duration=2m | vegeta encode >
> /tmp/http2.json; and vegeta report -type=json /tmp/http2.json | jq .
> 
> The app is at
> https://github.com/martin-g/http2-server-perf-tests/tree/master/java/tomcat.
> Just start EmbeddedTomcat#main() with -Dtomcat.http2=true

Definitely timing related as I am unable to reproduce the problem with
that command or some variations.

However, I think I have managed to track down the root cause. The good
news is that the BufferOverflowException is largely harmless. It is a
side-effect of the connection being closed due to an error. My guess is
that the error was a combination of vegeta sending an unexpected reset
frame and Tomcat maintaining state for a very small number of streams in
some circumstances.

If you could retest with the latest 9.0.x that would be very helpful.
The memory usage, stream state maintenance and this
BufferOverflowException should all be fixed.

Mark


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)

2020-09-30 Thread Mark Thomas
On 30/09/2020 16:17, Mark Thomas wrote:
> On 30/09/2020 13:53, Martin Grigorov wrote:
>> On Wed, Sep 30, 2020 at 12:50 PM Martin Grigorov 
> 
> 
> 
> 
>> When I load test HTTP2 with POST (with big bodies) there are many errors
>> like:
>>
>> 1)
>> Exception in thread "https-jsse-nio-8080-exec-5"
>> java.nio.BufferOverflowException
>> at java.base/java.nio.ByteBuffer.put(ByteBuffer.java:957)
>> at java.base/java.nio.HeapByteBuffer.put(HeapByteBuffer.java:247)
>> at
>> org.apache.tomcat.util.net.SocketBufferHandler.unReadReadBuffer(SocketBufferHandler.java:100)
>> at
>> org.apache.tomcat.util.net.SocketWrapperBase.unRead(SocketWrapperBase.java:401)
>> at
>> org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:307)
>> at
>> org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:164)
>> at
>> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1087)
>> at
>> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
>> at
>> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>> at java.base/java.lang.Thread.run(Thread.java:832)
>>
>> 2)
>> Sep 30, 2020 3:44:04 PM org.apache.tomcat.util.net.NioEndpoint$Poller events
>> SEVERE: Failed to register socket with selector from poller
>> java.nio.channels.ClosedChannelException
>> at
>> java.base/java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:222)
>> at
>> org.apache.tomcat.util.net.NioEndpoint$Poller.events(NioEndpoint.java:609)
>> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:703)
>> at java.base/java.lang.Thread.run(Thread.java:832)
> 
> That is helpful. Looks like you have found a way to reproduce the buffer
> issues reported in https://bz.apache.org/bugzilla/show_bug.cgi?id=64710

Can you share the command you used to trigger those errors please.

Thanks,

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)

2020-09-30 Thread Mark Thomas
On 30/09/2020 09:32, Arshiya Shariff wrote:
> Thank you for the response Mark ,
> 
>>> Are you able to test with a custom Tomcat build and/or build Tomcat 9 from 
>>> source for testing?
> Yes Mark , we will be able to test with the jars built from Tomcat 9 source 
> for testing .

The reduced memory footprint changes have now been back-ported to 9.0.x.
Please let us know how you get on with your testing.

Mark


> 
> Thanks and Regards
> Arshiya Shariff
> 
> -Original Message-
> From: Mark Thomas  
> Sent: Tuesday, September 29, 2020 12:25 AM
> To: users@tomcat.apache.org
> Subject: Re: HTTP2: memory filled up fast on increasing the connections to 
> 1000/2000 (Embedded tomcat 9.0.38)
> 
> On 28/09/2020 17:58, Arshiya Shariff wrote:
>> Hi All,
>> With 200 threads(users) , ramp up duration of 2 seconds , loop count 80 and 
>> by sending 1000 http2 requests/sec from JMeter Client to an embedded tomcat 
>> application we did not observe any memory issue , but on sending 1000 http2 
>> requests/sec with 2000 or 1000 users from JMeter , the application's heap 
>> space of 20 GB is occupied in 2 minutes and after 2 full GCs the memory 
>> clears and comes down to 4GB (expected) .
>>
>> Embedded tomcat Version:9.0.38
>> Max Threads : 200
>> All other properties are the tomcat defaults.
>>
>> Why is tomcat not able to process many connections ?
> 
> You haven't provided any evidence that Tomcat isn't able to process "many" 
> connections.
> 
>> Why is the memory filled when the connections are increased, are there any 
>> parameters to tune connections ?
> 
> It looks like users == HTTP/2 Connection. Connections are required to 
> maintain state for closed streams for both prioritisation and for error 
> handling. More connections == more state == more memory.
> 
> Given the number of connections increased by a factor of between 12.5 and 25, 
> that the memory usage only increased by a factor of 5 looks to be a positive 
> result rather than an issue.
> 
> There are significant improvements to memory usage in this area in Tomcat 
> 10.0.x that will get back-ported to 9.0.x but more testing is required.
> 
> Are you able to test with a custom Tomcat build and/or build Tomcat 9 from 
> source for testing?
> 
> Mark
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)

2020-09-30 Thread Mark Thomas
On 30/09/2020 13:53, Martin Grigorov wrote:
> On Wed, Sep 30, 2020 at 12:50 PM Martin Grigorov 




> When I load test HTTP2 with POST (with big bodies) there are many errors
> like:
> 
> 1)
> Exception in thread "https-jsse-nio-8080-exec-5"
> java.nio.BufferOverflowException
> at java.base/java.nio.ByteBuffer.put(ByteBuffer.java:957)
> at java.base/java.nio.HeapByteBuffer.put(HeapByteBuffer.java:247)
> at
> org.apache.tomcat.util.net.SocketBufferHandler.unReadReadBuffer(SocketBufferHandler.java:100)
> at
> org.apache.tomcat.util.net.SocketWrapperBase.unRead(SocketWrapperBase.java:401)
> at
> org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:307)
> at
> org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:164)
> at
> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1087)
> at
> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
> at
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> at java.base/java.lang.Thread.run(Thread.java:832)
> 
> 2)
> Sep 30, 2020 3:44:04 PM org.apache.tomcat.util.net.NioEndpoint$Poller events
> SEVERE: Failed to register socket with selector from poller
> java.nio.channels.ClosedChannelException
> at
> java.base/java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:222)
> at
> org.apache.tomcat.util.net.NioEndpoint$Poller.events(NioEndpoint.java:609)
> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:703)
> at java.base/java.lang.Thread.run(Thread.java:832)

That is helpful. Looks like you have found a way to reproduce the buffer
issues reported in https://bz.apache.org/bugzilla/show_bug.cgi?id=64710

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)

2020-09-30 Thread Mark Thomas
On 30/09/2020 06:42, Arshiya Shariff wrote:
> Hi Martin , 
> 
> Thank you for the response. 
> 
> With a payload of 200 bytes we were able to send 20K requests/sec with 200 
> users from Jmeter without any memory issue . On increasing the payload to 5Kb 
> and the number of users to 1000 in Jmeter and sending 1000 requests per 
> second , the heap of 20GB got filled in 2 minutes . With 200 users the memory 
> is cleared in the G1 mixed GC itself , but with 1000 users the memory is not 
> cleared in the mixed GC , it takes full GCs of 7 to 10 seconds to clear the 
> memory. These cases were executed with maxThreads 200 in tomcat , so we tried 
> increasing the maxThreads from 200 to 1000, but still GC was struggling .
> 
> When we tried with 10 instances of JMeter , each with 100 users , where each 
> instance was started with a delay of 1 minute we were able to see 1000 
> connections created in tomcat without any memory issues. But when 1000 users 
> are created using single instance of JMeter in 20 seconds , tomcat's memory 
> is filling fast- 20GB in 2 minutes. 
> We suspect that the burst of connections being opened has a problem . Please 
> help us with the same .
> 
> On analyzing the heap dump we see 
> org.apache.tomcat.util.collections.SynchronizedStack occupying around 93% of 
> 3GB live data ,the remaining 17GB is Garbage collected in the heap dump.

You can't have high throughput, low GC pauses and small heap sizes.
Broadly you can have any two of those three at the expense of the third.

The way Tomcat currently retains information about completed h2 streams
means you are likely to need a large heap under heavy load. There are
some changes already in 10.0.x that I plan to back-port to 9.0.x and
8.5.x later today that should significantly reduce the heap requirements.

Mark


> 
> Thanks and Regards
> Arshiya Shariff
> 
> -Original Message-
> From: Martin Grigorov  
> Sent: Monday, September 28, 2020 11:44 PM
> To: Tomcat Users List 
> Subject: Re: HTTP2: memory filled up fast on increasing the connections to 
> 1000/2000 (Embedded tomcat 9.0.38)
> 
> Hi Arshiya,
> 
> 
> On Mon, Sep 28, 2020 at 7:59 PM Arshiya Shariff 
>  wrote:
> 
>> Hi All,
>> With 200 threads(users) , ramp up duration of 2 seconds , loop count 
>> 80 and by sending 1000 http2 requests/sec from JMeter Client to an 
>> embedded tomcat application we did not observe any memory issue , but 
>> on sending
>> 1000 http2 requests/sec with 2000 or 1000 users from JMeter , the 
>> application's heap space of 20 GB is occupied in 2 minutes and after 2 
>> full GCs the memory clears and comes down to 4GB (expected) .
>>
> 
> I am not sure whether you follow the other discussions at users@.
> In another email thread we discuss load testing Tomcat HTTP2 and we are able 
> to make around 12K reqs/s with another load testing tool - 
> https://protect2.fireeye.com/v1/url?k=f8cfc13c-a66f0379-f8cf81a7-8692dc8284cb-2c0aae53194b790f=1=6a9c569d-7da1-4394-a9ac-bf72724992fa=https%3A%2F%2Fgithub.com%2Ftsenart%2Fvegeta
> For me JMeter itself failed with OOM when increasing the number of the 
> virtual users above 2K.
> There are several improvements in Tomcat master and 9.0.x in the HTTP2 area. 
> Some of the changes are not yet downported to 9.0.x. We still test them, 
> trying to avoid introducing regressions in 9.0.x.
> 
> 
>>
>> Embedded tomcat Version:9.0.38
>> Max Threads : 200
>>
> 
> The number of threads should be less if you do only CPU calculations without 
> IO/network. If your app blocks on IO/network calls then you need more spare 
> threads.
> With more threads there will be more context switches and less throughput.
> That's why there is no one golden rule that applies to all applications.
> 200 is a good default that works for most of the applications. But you need 
> to test with different values to see which one gives the best performance for 
> your scenaria.
> 
> 
>> All other properties are the tomcat defaults.
>>
>> Why is tomcat not able to process many connections ?
>>
> 
> You can tell us by enabling -XX:+HeapDumpOnOutOfMemoryError and 
> -XX:HeapDumpPath=. Once you have the .hprof file you can 
> examine it with Eclipse Memory Analyzer tool and see what is leaking.
> I will try to reproduce this issue tomorrow with Vegeta.
> 
> 
>> Why is the memory filled when the connections are increased, are there 
>> any parameters to tune connections ?
>> Please let us know.
>>
>> Thanks and Regards
>> Arshiya Shariff
>>
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Enabling FIPS for Tomcat

2020-09-29 Thread Mark Thomas
On 29/09/2020 16:25, Amit Pande wrote:
> Dear all,
> 
> The link below documents how to enable FIPS (using Bouncy Castle) for Tomcat.
> 
> https://github.com/amitlpande/tomcat-9-fips
> 
> Kindly let me know your inputs if this needs any corrections, enhancements.
> 
> Also, a request to Tomcat leads: It is possible for these steps to be part of 
> (extended) Tomcat documentation? Even if currently it uses Bouncy Castle as 
> FIPS JCA/JCE provider, with minor changes it would work for any other 
> provider too (e.g. CryptoComply for Java / CCJ from Safelogic)

You can always create a page in the wiki. Maybe "FAQ > Security > FIPS".

You'll need to create an account and then ask (on this list is fine) for
write access.

Mark


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Virtual event focussed on Tomcat Security

2020-09-29 Thread Mark Thomas
Hi all,

We (the Tomcat community) have some funding from Google to help us
improve Tomcat security. Our original plan was to use the funding to
support an in-person security focussed hackathon. As you would expect,
those plans are on hold for now. We would, therefore, like to explore
the possibility of doing something virtually.

The purpose of this email is to gather input from the community about
what such an event should look like. With that input we can put together
a plan for the event. So, over to you. What would your ideal virtual
event focussed on Tomcat Security look like?

Thanks,

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)

2020-09-28 Thread Mark Thomas
On 28/09/2020 17:58, Arshiya Shariff wrote:
> Hi All,
> With 200 threads(users) , ramp up duration of 2 seconds , loop count 80 and 
> by sending 1000 http2 requests/sec from JMeter Client to an embedded tomcat 
> application we did not observe any memory issue , but on sending 1000 http2 
> requests/sec with 2000 or 1000 users from JMeter , the application's heap 
> space of 20 GB is occupied in 2 minutes and after 2 full GCs the memory 
> clears and comes down to 4GB (expected) .
> 
> Embedded tomcat Version:9.0.38
> Max Threads : 200
> All other properties are the tomcat defaults.
> 
> Why is tomcat not able to process many connections ?

You haven't provided any evidence that Tomcat isn't able to process
"many" connections.

> Why is the memory filled when the connections are increased, are there any 
> parameters to tune connections ?

It looks like users == HTTP/2 Connection. Connections are required to
maintain state for closed streams for both prioritisation and for error
handling. More connections == more state == more memory.

Given the number of connections increased by a factor of between 12.5
and 25, that the memory usage only increased by a factor of 5 looks to
be a positive result rather than an issue.

There are significant improvements to memory usage in this area in
Tomcat 10.0.x that will get back-ported to 9.0.x but more testing is
required.

Are you able to test with a custom Tomcat build and/or build Tomcat 9
from source for testing?

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: RemoteIpValve doesn't maintain context in worker thread after startAsync

2020-09-28 Thread Mark Thomas
On 28/09/2020 16:15, Solas, Nathan wrote:
> I'm using RemoteIpValve to capture protocolHeader x-forwarded-proto and 
> upgrade the request to secure when SSL is terminated at the load balancer - 
> so far so good.
> 
> When using theServletRequest.startAsync() and then passing the work to a 
> threadpool executor, it seems the RemoteIpValve.invoke finally{} block 
> executes first and undoes all the changes to the Request object. The result 
> is by the time the worker picks up the request, it's demoted back to http and 
> nothing works.
> 
> This was noted in 2017 with no answers, but the internet is pretty quiet 
> about others facing this:
> https://marc.info/?l=tomcat-user=150971153905722=2
> 
> Maybe I'm not understanding how this should work? Seems right to me... Thanks 
> for any information,
> Nate

You'll probably have more luck with the Filter than the Valve. One of
the advantages the Filter has is that it can warp the request and/or
response.

Mark



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Connection header override

2020-09-28 Thread Mark Thomas
On 28/09/2020 08:33, Mark Thomas wrote:
> On 27/09/2020 00:07, Pawel Veselov wrote:
>> Hello!
>>
>> Tomcat 9.0.x
>>
>> I'd like to force connection closure on some endpoints.
> 
> Why? Generally, this is something that should not be an application concern.
> 
>> I'm trying this on a simple JSP page.
>> If I call response.setHeader("Connection","close"), I see that the
>> response has "Connection: close, keep-alive".
>> I assume Tomcat inserts the keep-alive part. It looks like the browsers
>> still close the connection based on this, but I was wondering if it's
>> possible to have Tomcat honor the header value set by the application.
> 
> The most recent discussion on this topic was whether or not Tomcat
> should block any attempt by an application to manipulate the Connection
> header. The consensus was leaning towards implementing a block but
> no-one has implemented it yet.
> 
> See https://github.com/apache/tomcat/pull/277
> 
> I did wonder if this was a regression introduced by some clean-up in the
> handling of the Connection header a little while ago but it appears not.
> Note that Tomcat will only add this header for HTTP/1.0 requests.
> Separately, you may also see a "Keep-Alive" header for HTTP/1.1 requests.

Testing shows that isn't right - I was mis-reading the code. You will
see this with HTTP/1.1 requests where the client explicitly sends a
"Connection: keep-alive header".

I'm currently working on expanding the unit tests to cover this
(although I'd still like to know why the app needs to close the connection).

Mark


> 
> I'm leaning towards a small fix that would prevent the keep-alive header
> being added in this case.
> 
>> I was also wondering what does it mean - when multiple connection tokens
>> are specified for a connection header. Breezing through the RFCs I can't
>> find a clear statement on that except that multiple connection tokens are
>> allowed in the header...
> 
> As per RFC 7230, section 6.3 if the close option is present it takes
> priority.
> 
> Mark
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Connection header override

2020-09-28 Thread Mark Thomas
On 27/09/2020 00:07, Pawel Veselov wrote:
> Hello!
> 
> Tomcat 9.0.x
> 
> I'd like to force connection closure on some endpoints.

Why? Generally, this is something that should not be an application concern.

> I'm trying this on a simple JSP page.
> If I call response.setHeader("Connection","close"), I see that the
> response has "Connection: close, keep-alive".
> I assume Tomcat inserts the keep-alive part. It looks like the browsers
> still close the connection based on this, but I was wondering if it's
> possible to have Tomcat honor the header value set by the application.

The most recent discussion on this topic was whether or not Tomcat
should block any attempt by an application to manipulate the Connection
header. The consensus was leaning towards implementing a block but
no-one has implemented it yet.

See https://github.com/apache/tomcat/pull/277

I did wonder if this was a regression introduced by some clean-up in the
handling of the Connection header a little while ago but it appears not.
Note that Tomcat will only add this header for HTTP/1.0 requests.
Separately, you may also see a "Keep-Alive" header for HTTP/1.1 requests.

I'm leaning towards a small fix that would prevent the keep-alive header
being added in this case.

> I was also wondering what does it mean - when multiple connection tokens
> are specified for a connection header. Breezing through the RFCs I can't
> find a clear statement on that except that multiple connection tokens are
> allowed in the header...

As per RFC 7230, section 6.3 if the close option is present it takes
priority.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: CurrentThreads not increasing in Apache Tomcat/9.0.37

2020-09-26 Thread Mark Thomas
On 25/09/2020 22:29, Vicente Perez wrote:



>  >Look in the logs for a line that contains:
> 24-Sep-2020 17:00:13.744 INFO [main] org.apache.coyote.AbstractProtocol.init 
> Initializing ProtocolHandler ["http-nio-8080"]
> 
> 24-Sep-2020 17:00:13.407 INFO [main] 
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The Apache 
> Tomcat Native library which allows using OpenSSL was not found on the 
> java.library.path: 
> [/usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib]

You are using the NIO HTTP connector. You are not using APR/native.

> 24-Sep-2020 17:00:13.404 INFO [main] 
> org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
> -Dcom.sun.management.jmxremote
> 24-Sep-2020 17:00:13.404 INFO [main] 
> org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
> -Dcom.sun.management.jmxremote.port=9010
> 24-Sep-2020 17:00:13.404 INFO [main] 
> org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
> -Dcom.sun.management.jmxremote.rmi.port=9010
> 24-Sep-2020 17:00:13.404 INFO [main] 
> org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
> -Dcom.sun.management.jmxremote.local.only=false
> 24-Sep-2020 17:00:13.404 INFO [main] 
> org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
> -Dcom.sun.management.jmxremote.ssl=false
> 24-Sep-2020 17:00:13.405 INFO [main] 
> org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
> -Dcom.sun.management.jmxremote.authenticate=false

Remote JMX enabled without TLS? I'd check that is what was intended.

>> To investigate this further you'll need to provide the steps to
>>   reproduce this from a clean install of the latest stable 9.0.x release
>>  (9.0.38 as I type this) and the latest stable Tomcat Native (1.2.25).
> 
> I will try to reproduce the error in that way.

Thanks.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: CurrentThreads not increasing in Apache Tomcat/9.0.37

2020-09-25 Thread Mark Thomas
On 25/09/2020 18:07, Vicente Perez wrote:



> Thank you for the answer.
> 
> Yes. It is  APR/Native Connector.

Are you sure? It isn't impossible that you are using the APR/Native
connector given your configuration but it would be unusual.

Look in the logs for a line that contains:
org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ...

What is the full text of that line?

Also, what version of the Tomcat Native Connector are you using?
Look in the logs for a line that contains:

org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded
Apache Tomcat Native library [???] using APR version [???]

What is the full text of that line?

> Please let me know if you need more information about the 
> configuration/environment.

I ran some simple tests with ab and I cannot repeat the problem you
describe.

As I increase the concurrency with ab, currentThreadCount increases.

I even reduced minSpareThreads to 0 and I still could not reproduce the
behaviour.

To investigate this further you'll need to provide the steps to
reproduce this from a clean install of the latest stable 9.0.x release
(9.0.38 as I type this) and the latest stable Tomcat Native (1.2.25).

Ideally the test case would involve nothing more than a clean Tomcat
install and ab to generate the load.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: CurrentThreads not increasing in Apache Tomcat/9.0.37

2020-09-25 Thread Mark Thomas
On 25/09/2020 14:02, Vicente Perez wrote:
> Hello,
> 
> We have a server with Apache Tomcat/9.0.37 which is dedicated for less than 
> 100 concurrent users. We are using the ARP Connector with the parameter 
> “maxThreads=400”.

I assume you mean the APR/Native connector here.

>  connectionTimeout="2"
>redirectPort="8443"  maxThreads="400" />
> 
> We have the next behavior:
> Tomcat does not process any new request with currentThreadCount = X and  
> currentThreadBusy = X, but with X=30 or 50. ConnectionCount is increasing.
> 
> A far as I understood, the number of threads should increase if  
> currentThreadBusy is increasing.

Correct.

> We prevent this behavior by setting minSpareThreads to 100.
> 
>   *   Is this behavior correct?

Doesn't look right.

>   *   How the Connector should be configured in order to increase the 
> currentThreadCount when currentThreadBusy is equal to currentThreadCount?

This looks like a potential bug that needs further investigation.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: HTTP2: Tomcat sends GOAWAY when trying to respond over a stream where the client has already sent RST_STREAM:NO ERROR

2020-09-25 Thread Mark Thomas
On 25/09/2020 15:47, Arshiya Shariff wrote:
> Thanks for the response Martin.
> 
> Below is the only exception we are getting in the logs while sending response 
> to a closed stream , few times with MUST_COMPLETE instead of MUST_ERROR :
> Exception occurred while sending response. ExceptionMessage:Calling 
> [asyncComplete()] is not valid for a request with Async state [MUST_ERROR]

That suggests issues with error handling. Either errors not being
handled or some sort of threading issue.

If you attempt to write to a closed stream Tomcat should simply swallow
the data and not send it to the client.

The only reason Tomcat will send a GOAWAY frame in normal usage if there
is a connection level error.

Mark

> 
> Thanks and Regards
> Arshiya Shariff
> 
> -Original Message-
> From: Martin Grigorov  
> Sent: Friday, September 25, 2020 4:37 PM
> To: Tomcat Users List 
> Subject: Re: HTTP2: Tomcat sends GOAWAY when trying to respond over a stream 
> where the client has already sent RST_STREAM:NO ERROR
> 
> On Fri, Sep 25, 2020 at 1:18 PM Arshiya Shariff 
>  wrote:
> 
>> Thanks Martin .
>>
>> Our expectation Is that , on receiving a RST_STREAM with CANCEL or 
>> NO_ERROR from the client after 1.5 secs for a particular stream , we 
>> don’t want the connection to be closed  with a GOAWAY:NO_ERROR  (while 
>> trying to send response after 2secs asynchronously over a stream that 
>> is closed) as the other streams during the same time are processing data 
>> fine .
>>
>> Additional info for GOAWAY: Connection [], Stream[],An error occurred 
>> during processing that was fatal to the connection .
>>
> 
> I guess some other error happened and that led to the GOAWAY. But I cannot be 
> sure.
> Any errors in the logs ?
> 
> 
>>
>> Thanks and Regards
>> Arshiya Shariff
>>
>> -Original Message-
>> From: Martin Grigorov 
>> Sent: Tuesday, September 22, 2020 7:31 PM
>> To: Tomcat Users List 
>> Subject: Re: HTTP2: Tomcat sends GOAWAY when trying to respond over a 
>> stream where the client has already sent RST_STREAM:NO ERROR
>>
>> Hi,
>>
>> On Tue, Sep 22, 2020 at 1:47 PM Arshiya Shariff < 
>> arshiya.shar...@ericsson.com.invalid> wrote:
>>
>>> Thank you so much Martin for the response.
>>>  Yes , 9.0.38 testing is on going .
>>>
>>> As we don’t get this clear with the RFC , please help us with the 
>>> below two cases :
>>
>>   * If a client sends RST_STREAM with NO_ERROR, then while sending 
>> async
>>> response is it expected behavior to close connection with GOAWAY ?
>>>   * If a client sends RST_STREAM with CANCEL , then while sending 
>>> async response will tomcat send RST_STREAM or GOAWAY , from http2 
>>> protocol perspective ?
>>>
>>
>> As per https://tools.ietf.org/html/rfc7540#section-6.4 and
>> https://tools.ietf.org/html/rfc7540#section-5.1 if "send R" or "recv R"
>> are processed then the stream is moved to CLOSED state.
>> GOAWAY should be sent to the client only if the connection will be 
>> closed ( https://tools.ietf.org/html/rfc7540#section-6.8). The server 
>> should close the connection only if some serious problem has happened, 
>> e.g. an IOException.
>>
>> Tell us more about your use case. What do you want to do when 1.5secs 
>> pass ? What do you expect to happen ?
>>
>>
>>>
>>> Thanks and Regards
>>> Arshiya Sharif
>>>
>>> -Original Message-
>>> From: Martin Grigorov 
>>> Sent: Tuesday, September 22, 2020 1:18 AM
>>> To: Tomcat Users List 
>>> Subject: Re: HTTP2: Tomcat sends GOAWAY when trying to respond over 
>>> a stream where the client has already sent RST_STREAM:NO ERROR
>>>
>>> Hi,
>>>
>>> On Mon, Sep 21, 2020 at 7:56 PM Arshiya Shariff < 
>>> arshiya.shar...@ericsson.com.invalid> wrote:
>>>
 Hi All,

 The client has configured a response timeout of 1.5 seconds. In a 
 case when our application tries to respond over a http2 stream 
 asynchronously after 2 seconds where the client has already sent 
 RST_STREAM with NO ERROR in 1.5 seconds
>>>
>>>
>>> Why does the client send NO_ERROR to the server ? I think it should 
>>> send a CANCEL instead.
>>> https://tools.ietf.org/html/rfc7540 mentions NO_ERROR only for 
>>> "Graceful shutdown of the server".
>>>
>>>
 (due to no response) , then tomcat sends GOAWAY and closes the 
 HTTP2 connection . Is this behavior of GOAWAY and connection 
 closure
>> expected ?
 We have planned to upgrade to Embedded tomcat version 9.0.38 . 
 These are the behaviors we see in production with version 9.0.22 ,  
 so we need some help with analyzing / validating  the existing 
 behavior before
>>> the upgrade .
 Please let us know.

>>>
>>> Friendly advice:
>>> Please setup 9.0.38 locally and test on it.
>>> 9.0.22 is way too old. It is up to you to use it for your production 
>>> but for reporting bugs it is recommended to use the latest available
>> version.
>>> I, personally, prefer to spend my spare time with my family and 
>>> friends than to debug old versions just because 

Re: Low throughput with HTTP2

2020-09-25 Thread Mark Thomas
On 25/09/2020 12:18, Berneburg, Cris J. - US wrote:
> Thanks again Mark  :-)
> 
> mt> how that Map is pruned (it is currently too aggressive)
> 
> mt> if Tomcat is processing 10k req/s just keeping track of
> mt> the last 30s is potentially 300k streams. How to do that
> mt> efficiently for all usage patterns is a problem that
> mt> needs some thought.
> 
> Sounds a bit like garbage collection.  Is aging part of the process - a 
> map/queue combo?

Yes, but only very simplisticly. Streams with lower IDs are removed first.

> cjb> How could the closed stream footprint be reduced?
> cjb> Could the structure holding a closed stream:
> cjb> a. Be replaced with a smaller one?
> cjb> c. Or did you already have something in mind?
> 
> mt> A form of a). I'm looking at this now.

I committed this earlier today. See dev@ for details.

> cjb> b. De-reference other objects no longer needed?
> cjb> Hmm... that might lead to NPE's and thus unnecessary
> cjb> null checking.
> 
> mt> Tried that. Lots of NPE regressions to the point that
> mt> I reverted the change to look for a better solution.
> 
> Hey great, I'm beginning to understand!  :-D
> 
> mt> we have all the plumbing to correctly determine
> mt> relative priority [...] we don't use it to prioritise
> mt> streams when flow control windows are not an issue
> 
> mt> I started to look at this a while ago but it gets very
> mt> complex quite quickly. It would be simpler if we were
> mt> just serving static content.
> 
> Ha ha, httpd!  Hang on, does httpd handle a similar situation too?

I don't know. I imagine so.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Tomcat's support for path parameters can expose resources despite reverse proxy access restrictions

2020-09-24 Thread Mark Thomas
On 24/09/2020 17:28, Christopher Schultz wrote:



> Tomcat will only use path parameters in the final segment of a URL e.g.
> https://www.example.com/app/servlet;jsessionid=ABCD1234?q=search

Not quite. Tomcat will only *add* the jsessionid at the end but it will
accept it on any segment.

Internally, Tomcat has an API to access path parameters but it only
tracks name and value (as that is all that is required to extract
jsesisonid). It would be trivial to extend it to include path
information as well.

> Assuming your application doesn't use path-parameters for anything else,
> you should be able to detect and block any non-terminal path-segment
> which contains a parameter and simply refuse the request with 400 or
> something similar.

That is probably the simplest option in this case.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Tomcat's support for path parameters can expose resources despite reverse proxy access restrictions

2020-09-24 Thread Mark Thomas
On 24/09/2020 11:02, Nils Breunese wrote:



> - Envoy allows the request based on the /v1/* rule, because it does not 
> support path parameters, because they are not part of any recent standard 
> (RFC 2396 dropped them in 1998 [1])

Envoy does support path parameters and is correctly doing so in this
case as an HTTP reverse proxy by passing them through to the back-end.
>From an HTTP perspective "/bar" and "/foo/..;a=1/bar" are completely
different URIs. Envoy's behaviour as an HTTP reverse proxy is correct.

That the back-end may map URIs to resources in such a way that both
those URIs result in the same resource being served is solely a concern
for the back-end and entirely transparent to the HTTP reverse proxy.

The current RFC for URIs (3986) continues to make reference to
parameters within segments and using ';' as a delimiter for them. See
the final paragraph of section 3.3.
The reference in section D.2 to obsolete rules is a simplification of
the grammar, not a removal of features.

> - Tomcat parses ;color=red as a path parameter, resolves the rest of the path 
> to /v1/../internal/secret and consequently serves /internal/secret
> - I’m pretty sure that whoever configured that RBAC policy did not want this 
> to be possible

Then whoever configured the role-based access control shouldn't have
assumed that the reverse proxy used the same rules for URI to proxy
target mapping as the back-end used for mapping URIs to resources. They
either needed to ensure that Envoy was configured to map proxied
requests using the same mapping rules as the Servlet spec (I'd be
surprised if such a configuration setting was available) or to configure
the access control in the back-end.

This is a common error.

Since mod_jk and the the ISAPI redirector are provided by the Apache
Tomcat project it is reasonable for users to assume that they map URIs
to proxy targets using the same rules as the Servlet spec does for URIs
to resources. Where differences have been found over the years, CVEs
have been allocated.

If a generic HTTP reverse proxy (e.g. mod_proxy_http) is used then the
mapping of URIs to proxy targets is going to be HTTP based, not Servlet
based. That said, I understand that the httpd team is looking at
implementation a configuration option for mod_proxy that would cause it
to use Servlet mapping rules to map URIs to proxy targets.



> Would it be reasonable to create a ticket with a feature request for a flag 
> to disable path parameter support in Tomcat?

No. Support for path parameters is an explicit requirement of the
Servlet specification.

(Aside:
1. While the Servlet spec language isn't aligned with RFC 3986 the
expert group has made it clear that the intention is that references to
"path parameter" should be read as "parameter associated with any
segment of the URI".
2. Whether the Servlet spec should still be supporting URI re-writing as
a method of session tracking is debatable.
https://github.com/eclipse-ee4j/servlet-api/issues is the place to raise
the issue of removing it.
)

> Or an even more secure suggestion: make path parameter support an opt-in 
> feature, because it was dropped from the URI standard in 1998 and can make 
> setups like I described vulnerable?

No, because it hasn't been dropped from the URI standard and it is a
supporting path parameters is a mandatory requirement of the Servlet spec.

Clearly there is scope for more education and documentation on this
issue. The (very) short version is when using a reverse proxy in front
of Tomcat:
1. Ideally don't rely on the reverse proxy to limit access to resources
on Tomcat. Secure Tomcat as if everything was accessible.
2. If you can't do 1, use mod_jk or the ISAPI redirector as they should
map URIs to proxy targets using the same mapping rules as Tomcat (and if
they don't we'll almost certainly treat that as a security issue).
3. If you can't do 1 or 2 you can try and block potentially malicious
URIs in the reverse proxy but this is hard (need to think about "/../"
and "/./" sequences, path parameters, %nn encoding and combinations of
all three.

If you get as far as option 3, you really need to go back and
re-consider option 1 or 2 as the chances of getting 3 100% right are slim.

Finally, it is probably worth setting out what are the mapping rules
defined by the Servlet spec as the definition is very much implicit
rather than explicit.

1. Remove all path parameters
2. Collapse all "//" to "/"
3. Remove "/./" segments
4. Remove "/xxx/../" segments
   (If a leading "/../" remains that is an error)

Then Tomcat will map the resulting URI to a resource as per chapter 12
of the Servlet 4.0 spec.
mod_jk (and ISAPI) will map the resulting URI to the proxy targets as
per the docs
http://tomcat.apache.org/connectors-doc/reference/uriworkermap.html

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: 

Re: Low throughput with HTTP2

2020-09-23 Thread Mark Thomas
On 23/09/2020 15:50, Berneburg, Cris J. - US wrote:
> Hi Mark
> 
> Thanks for taking the time to explain that to me.  :-)
> 
> A few more questions, if you don't mind.
> 
> cjb> TC thinks the stream should be closed when the client
> cjb> thinks the stream is still open?  Basically RST_STREAM
> cjb> is a keep-alive?
> 
> mt> No. The stream closed cleanly. The client is sending
> mt> RST_STREAM due to what is suspected to be a client bug.
> mt> RFC 7540 says the server must ignore such frames and can,
> mt> if a frame is received a significant period after the
> mt> stream closed, treat it as a protocol error (and close
> mt> the connection).
> 
> mt> Separately, the server should (as per the RFC) retain
> mt> state for closed streams to support prioritisation.
> 
> mt> Currently Tomcat uses a single Map to track the state of
> mt> closed streams for priority and to identify streams have
> mt> been closed for an *in*significant amount of time.
> 
> mt> The issues immediately at hand are:
> mt> - how that Map is pruned (it is currently too aggressive)
> 
> What would you consider "less aggressive"?

Not reducing the number of streams below the maximum number of
concurrent streams.

>  Would aggressiveness depend on system load?

Possibly. The less it does so, the better. But, for example, if Tomcat
is processing 10k req/s just keeping track of the last 30s is
potentially 300k streams. How to do that efficiently for all usage
patterns is a problem that needs some thought.

> mt> - that under high load a "significant period" becomes a
> mt>   few milliseconds
> 
> Sounds like "significant period" varies depending on system load.

At the moment there is a fairly direct inverse relationship. Ideally
there should be no relationship. Not sure how achievable that is at the
moment.

> mt> currently memory footprint of a closed stream is much
> mt> larger than it needs to be
> 
> How could the closed stream footprint be reduced?  Could the structure 
> holding a closed stream:
> 
> a. Be replaced with a smaller one?

Potentially.

> b. De-reference other objects no longer needed?  Hmm... that might lead to 
> NPE's and thus unnecessary null checking.

Tried that. Lots of NPE regressions to the point that I reverted the
change to look for a better solution.

> c. Or did you already have something in mind?

A form of a). I'm looking at this now.

> mt> while we have all the plumbing to correctly determine
> mt> relative priority and use it when allocating window
> mt> updates in the case where the connection flow control
> mt> window is smaller than the total data the streams want
> mt> to send - we don't use it to prioritise streams when
> mt> flow control windows are not an issue
> 
> Is that an FYI or a to-do?

Both. I started to look at this a while ago but it gets very complex
quite quickly. It would be simpler if we were just serving static content.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Low throughput with HTTP2

2020-09-22 Thread Mark Thomas
On 22/09/2020 13:47, Berneburg, Cris J. - US wrote:
> Hi Mark
> 
> As with most topics here, I struggle to understand what is being discussed.  
> :-)  So please bear with me.
> 
>> improving how Tomcat handles traffic like this.
>>
>> Looks like Tomcat could prune the closed streams
>> less aggressively.
>>
>> At the moment it waits until there are
>> maxConcurrentStreams + 10% in the map and then:
>> - removes all closed streams without children
>> - [snip] with children [snip]
>> - [snip] closed final streams [snip]
>>
>> [snip] the size of the map increases to ~110 and
>> then drops to ~5, increases to ~110 and repeats.
>>
>> I'm currently thinking about different pruning
>> strategies. The associated memory footprint is
>> also part of my thinking.
> 
> TC thinks the stream should be closed when the client thinks the stream is 
> still open?  Basically RST_STREAM is a keep-alive?

No. The stream closed cleanly. The client is sending RST_STREAM due to
what is suspected to be a client bug. RFC 7540 says the server must
ignore such frames and can, if a frame is received a significant period
after the stream closed, treat it as a protocol error (and close the
connection).

Separately, the server should (as per the RFC) retain state for closed
streams to support prioritisation.

Currently Tomcat uses a single Map to track the state of closed streams
for priority and to identify streams have been closed for an
*in*significant amount of time.

The issues immediately at hand are:
- how that Map is pruned (it is currently too aggressive)
- that under high load a "significant period" becomes a few milliseconds

Also in the mix:
- currently memory footprint of a closed stream is much larger than it
  needs to be
- while we have all the plumbing to correctly determine relative
  priority and use it when allocating window updates in the case
  where the connection flow control window is smaller than the total
  data the streams want to send - we don't use it to prioritise streams
  when flow control windows are not an issue

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Low throughput with HTTP2

2020-09-21 Thread Mark Thomas
On 21/09/2020 15:52, Mark Thomas wrote:



> That doesn't
> exclude, of course, the possibility of improving how Tomcat handles
> traffic like this.

Looks like Tomcat could prune the closed streams less aggressively.

At the moment it waits until there are maxConcurrentStreams + 10% in the
map and then:
- removes all closed streams without children
- removes all closed streams with children (re-prioritising as
  appropriate)
- if there are still more than maxConcurrentStreams + 10% streams in the
  map, removes closed final streams (that are likely part of the
  priority tree) until the number of streams in the map is less than
  maxConcurrentStreams + 10% or there are no more closed streams to
  remove

This means, with defaults and the load profile in this test, that the
size of the map increases to ~110 and then drops to ~5, increases to
~110 and repeats.

I'm currently thinking about different pruning strategies. The
associated memory footprint is also part of my thinking.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Low throughput with HTTP2

2020-09-21 Thread Mark Thomas
On 21/09/2020 13:48, Martin Grigorov wrote:
> Hi Remy,
> 
> On Mon, Sep 21, 2020 at 2:56 PM Rémy Maucherat  wrote:
> 
> 
> 
> 
>>> 2020-09-21 14:25:04.850 DEBUG 232086 --- [https-jsse-nio-18080-exec-8]
>>> o.a.coyote.http11.Http11NioProtocol  : Found processor [null] for
>>> socket [org.apache.tomcat.util.net.SecureNioChannel@2b435926
>>> :java.nio.channels.SocketChannel[connected
>>> local=/127.0.0.1:18080 remote=/127.0.0.1:42229]]
>>> 2020-09-21 14:25:04.850 DEBUG 232086 --- [https-jsse-nio-18080-exec-6]
>>> o.a.coyote.http2.Http2UpgradeHandler : Connection [2]
>>>
>>> java.io.IOException: Unable to unwrap data, invalid status [CLOSED]
>>> at org.apache.tomcat.util.net
>>> .SecureNioChannel.read(SecureNioChannel.java:766)
>>> at org.apache.tomcat.util.net
>>>
>> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468)
>>>
>>
>> When I tested HTTP/2, I used h2c (unencrypted) to get a better idea at
>> what's going on. Otherwise you don't really know what proportion of TLS or
>> HTTP/2 impacts performance more [and then you have to test more things like
>> OpenSSL, be careful that the ciphers used are the same and equally well
>> optimized in each impl, etc].
>>
>> Then once you feel happy about h2c, you can see how things go with TLS.
>>
> 
> Chris also suggested that TLS might be the problem for the low throughput
> but I get good throughput for both HTTP (15-16 K) and HTTPS (12-13 K
> reqs/s). Only when HTTP2 is enabled it drops. The reason is more clear now
> - it is due to the extra RST packets being sent.
> 
> Vegeta does support h2c! So I can give it a try!

I've just done that. I am seeing the client sending RST_STREAM frames
AFTER the request/response has been processed cleanly.

The RST_STREAM frames are being sent with the CANCEL error. i.e. the
client is no longer interested in reading the response body.

This doesn't make much sense as there is little more than two or three
milliseconds between the start of the request and the RST_STREAM. It
isn't happening due to a timeout. It is also always sent AFTER the
server has sent the complete response (the end of stream flag is set)
and sometimes after the next request has been sent.

However you look at it, this looks like a bug in Vegeta. That doesn't
exclude, of course, the possibility of improving how Tomcat handles
traffic like this.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Low throughput with HTTP2

2020-09-21 Thread Mark Thomas
On 21/09/2020 10:44, Martin Grigorov wrote:
> On Mon, Sep 21, 2020 at 12:08 PM Martin Grigorov 
> wrote:
>> On Mon, Sep 21, 2020 at 11:23 AM Mark Thomas  wrote:



>>> RFC 7540 allows the connection to be closed with a protocol error if the
>>> reset is received "a significant time after sending END_STREAM".
>>>
>>> Tomcat retains state for maxConcurrentStreams + 10% with closed streams
>>> being pruned (oldest first) every 10 new streams.
>>
>> The spec talks about time but Tomcat uses the number of streams.
>> I understand that under normal usage this could be enough "time" but under
>> heavy load this number is reached quickly and the time is short.

Agreed. The closed stream is being pruned within a fraction of a second.
 However, the window where the client could legitimately send RST_STREAM
is also very small. Looking at the timings, I don't think the client is
sending the RST_STREAM within that window.

You get back to the (lack of) definition of significant. I think I could
make a case for significant being anything from a few milliseconds to
several minutes.

>>> It isn't clear at this point why the client is sending the RST_STREAM.
>>>
>>
>> I don't know either. But Vegeta is using the Golang standard libraries, so
>> any Golang application should behave the same way.
>> Also as I investigated last week - Golang/Node.js/Rust/Netty HTTP2 servers
>> do not complain about it.
>>
>>
>>> This normally indicates an error condition. From the description of the
>>> load test I don't see why the client would be sending a RST_STREAM.
>>>
>>> If the RST_STREAM is valid and it isn't received a significant time
>>> after the END_STREAM then Tomcat needs to handle this.
>>>
>>> If the RST_STREAM is valid but received a significant amount of time
>>> after the END_STREAM that would be a client error would be a client error.
>>>
>>> Of course, significant isn't defined in the specification.
>>>
>>> Any special handling for RST_STREAM also needs to be applied to
>>> WINDOW_UPDATE. It also needs to differentiate between a frame received
>>> for a past closed stream and a frame received for an stream that never
>>> existed.
>>>
>>> I think the key question here is why is the client sending the
>>> RST_STREAM in the first place. Is Tomcat doing something it shouldn't /
>>> not doing something it should to cause this? If so, we need to fix the
>>> root cause rather than tackle the symptom.
>>>
>>> There are a couple of options for debugging this to see why the client
>>> is sending the RST_STREAM:
>>> - Enable debug logging for HTTP/2 in Tomcat. This is very verbose and
>>>   if the root cause is threading related typically changes the timing
>>>   enough to mask the issue.
>>>
>>
>> Here are the logs of 20 seconds load:
>>
>> https://gist.githubusercontent.com/martin-g/37dacc12149d5cecbfd4e16dc2248cfd/raw/76693cd1dd1f37ba126edb84ebd7631802aa91a6/tomcat-http2.log

Thanks.

My first observation is that we aren't helping ourselves by having some
messages display stream IDs in 0,000 format, and others without the
commas. That should be a fairly simple fix.

I picked a request somewhere in the middle and I see the expected
sequence of frames for a normal request. Total processing time is about
30ms. Then about 50ms later, the client sends a RST_STREAM. I'd really
like to see what reason code was sent with that reset. That is another
simple fix.

>> This is with my patch in #reset(). Without it there are also errors that
>> stream XYZ is already closed

ACK.



>> According to https://tools.ietf.org/html/rfc7540#section-5.1 when the
>> stream is in HALF_CLOSED_REMOTE state it should send either END_STREAM or
>> RST_STREAM to the remote peer.
>> In the logs I see such sequence of events:
>>
>> ...
>> o.a.coyote.http2.StreamStateMachine  : Connection [124], Stream
>> [5,279], State changed from [OPEN] to [HALF_CLOSED_REMOTE]
>> o.a.coyote.http2.Http2UpgradeHandler : Connection [124], Stream
>> [5,279], Writing the headers
>> org.apache.coyote.http2.Stream   : Connection [124], Stream
>> [5,279], flushing output with buffer at position [12], writeInProgress
>> [false] and closed [true]
>> org.apache.coyote.http2.AbstractStream   : Connection [124], Stream
>> [5,279], reduce flow control window by [12] to [4194292]
>> o.a.coyote.http2.Http2UpgradeHandler : Connection [124], Stream
>> [5,279], Data length [12]
>> o.a.coyote.http2.StreamStateMachine  : Connection [1

Re: Low throughput with HTTP2

2020-09-21 Thread Mark Thomas
On 21/09/2020 08:18, Martin Grigorov wrote:
> On Fri, Sep 18, 2020 at 6:16 PM Mark Thomas  wrote:
> 
>> On 18/09/2020 14:07, Martin Grigorov wrote:
>>
>> 
>>
>>> What is the difference
>>> between org.apache.coyote.http2.StreamStateMachine.State#CLOSED_RX
>>> and org.apache.coyote.http2.StreamStateMachine.State#CLOSED_TX ?
>>
>> Compare the parameters used to construct the enums.
>>
>>> I read some parts of https://tools.ietf.org/html/rfc7540 but I don't see
>>> anything related to two types of CLOSED state.
>>
>> Section 5.1. Definition of the closed state (page 20) explains the
>> difference between the two.
>>
> 
> Still I do not understand what RX and TX stand for. But this is not
> important.

TX - transmit

RX - receive


> The following patch fixes the issue for me/Vegeta:
> 
> @@ -1570,12 +1571,15 @@ class Http2UpgradeHandler extends AbstractStream
> implements InternalHttpUpgradeH
> 
>  @Override
>  public void reset(int streamId, long errorCode) throws Http2Exception
>  {
> -Stream stream = getStream(streamId, true);
> -boolean active = stream.isActive();
> -stream.checkState(FrameType.RST);
> -stream.receiveReset(errorCode);
> -if (active) {
> -activeRemoteStreamCount.decrementAndGet();
> +boolean unknownIsError = Http2Error.CANCEL.getCode() != errorCode;
> +Stream stream = getStream(streamId, unknownIsError);
> +if (stream != null) {
> +boolean active = stream.isActive();
> +stream.checkState(FrameType.RST);
> +stream.receiveReset(errorCode);
> +if (active) {
> +activeRemoteStreamCount.decrementAndGet();
> +}
>  }
>  }
> 
> I.e. do not throw an error if the remote peer is trying to cancel an
> already closed stream.

RFC 7540 allows the connection to be closed with a protocol error if the
reset is received "a significant time after sending END_STREAM".

Tomcat retains state for maxConcurrentStreams + 10% with closed streams
being pruned (oldest first) every 10 new streams.

It isn't clear at this point why the client is sending the RST_STREAM.
This normally indicates an error condition. From the description of the
load test I don't see why the client would be sending a RST_STREAM.

If the RST_STREAM is valid and it isn't received a significant time
after the END_STREAM then Tomcat needs to handle this.

If the RST_STREAM is valid but received a significant amount of time
after the END_STREAM that would be a client error would be a client error.

Of course, significant isn't defined in the specification.

Any special handling for RST_STREAM also needs to be applied to
WINDOW_UPDATE. It also needs to differentiate between a frame received
for a past closed stream and a frame received for an stream that never
existed.

I think the key question here is why is the client sending the
RST_STREAM in the first place. Is Tomcat doing something it shouldn't /
not doing something it should to cause this? If so, we need to fix the
root cause rather than tackle the symptom.

There are a couple of options for debugging this to see why the client
is sending the RST_STREAM:
- Enable debug logging for HTTP/2 in Tomcat. This is very verbose and
  if the root cause is threading related typically changes the timing
  enough to mask the issue.
- Client side logging (I don't know what the capabilities are here).
- JVM level TLS debug logging (even more verbose than Tomcat's HTTP/2
  debug and harder to work with)
- Wireshark (or similar)

I'd recommend Wireshark. I don't think JSSE supports SSLKEYLOGFILE so
I'd use the APR/Native connector with SSLKEYLOGFILE to get the cleartext
in Wireshark.

> With this change and Vegeta's -max-workers=100 I can get 12 K reqs/sec.

That is odd. If the client is sending RST_STREAM messages then I'd
expect to see requests failing as RST_STREAM indicates something has
gone wrong or the client has aborted the request.

> With more workers it starts failing with:
> 
> "status_codes": {
> "0": 1000,
> "200": 1
>   },
>   "errors": [
> "Get \"https://localhost:18080/testbed/plaintext\": net/http: request
> canceled while waiting for connection (Client.Timeout exceeded while
> awaiting headers)",
> "Get \"https://localhost:18080/testbed/plaintext\": context deadline
> exceeded",
> "Get \"https://localhost:18080/testbed/plaintext\": context deadline
> exceeded (Client.Timeout exceeded while awaiting headers)"
>   ]
> i.e. out of 1001 requests only one succeeds and the others fail with
> timeout. I will try to debug this on

Re: Low throughput with HTTP2

2020-09-18 Thread Mark Thomas
On 18/09/2020 14:07, Martin Grigorov wrote:



> What is the difference
> between org.apache.coyote.http2.StreamStateMachine.State#CLOSED_RX
> and org.apache.coyote.http2.StreamStateMachine.State#CLOSED_TX ?

Compare the parameters used to construct the enums.

> I read some parts of https://tools.ietf.org/html/rfc7540 but I don't see
> anything related to two types of CLOSED state.

Section 5.1. Definition of the closed state (page 20) explains the
difference between the two.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: tomcat warnings. [EXTERNAL]

2020-09-17 Thread Mark Thomas
On 17/09/2020 14:06, Beard, Shawn wrote:
> Yes its 9.0.31.0
> 
> [mwuser@usilg01-tcd003 ~]$ ./version.sh
> Using CATALINA_BASE:   /path/to/catalina_base
> Using CATALINA_HOME:   /path/to/catalina_home
> Using CATALINA_TMPDIR: /path/to/catalina_base/temp
> Using JRE_HOME:/
> Using CLASSPATH:   /path/to/catalina_base 
> /bin/bootstrap.jar:/usr/apache/tomcat/tomcat-current/bin/tomcat-juli.jar
> Server version: Apache Tomcat/9.0.31
> Server built:   Feb 5 2020 19:32:12 UTC
> Server number:  9.0.31.0
> OS Name:Linux
> OS Version: 3.10.0-957.21.3.el7.x86_64
> Architecture:   amd64
> JVM Version:1.8.0_212-b04
> JVM Vendor: Oracle Corporation
> 
> I do have 3 connectors, one of them is the AJP connector, could the warning 
> be coming from them?

Yes. The compression settings are not valid for AJP connectors.

Mark


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: tomcat warnings. [EXTERNAL]

2020-09-17 Thread Mark Thomas
On 16/09/2020 20:45, Beard, Shawn wrote:
> protocol="HTTP/1.1"
>connectionTimeout="2"
>Server=" "
>maxHttpHeaderSize="8192"
>maxThreads="500"
>minSpareThreads="30"
>enableLookups="false"
>disableUploadTimeout="true"
>acceptCount="150"
>redirectPort="9444"
>
> compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/xml,application/x-javascript,application/javascript,application/x-font-woff,application/font-woff,application/pdf"
>compression="on"
>compressionMinSize="2048"
>noCompressionUserAgents="gozilla,traviata" />

I've added that exact configuration (copy and paste) to clean builds of
10.0.x, 9.0.x and 9.0.31 and I don't see the errors you are seeing.

Are you sure you are running 9.0.31?

Have you tested this with a clean 9.0.31 install downloaded from
tomcat.apache.org?

Mark


> 
> 
> 
> Shawn Beard
> Sr. Systems Engineer
> BTS
> +1-515-564-2528
> 
> -Original Message-
> From: Mark Thomas 
> Sent: Wednesday, September 16, 2020 2:40 PM
> To: users@tomcat.apache.org
> Subject: Re: tomcat warnings. [EXTERNAL]
> 
> ** CAUTION: External message
> 
> 
> On 16/09/2020 19:46, Beard, Shawn wrote:
>> I’m getting these in the log:
>>
>>
>>
>> 16-Sep-2020 14:39:42.909 WARNING [main]
>> org.apache.catalina.startup.SetAllPropertiesRule.begin
>> [SetAllPropertiesRule]{Server/Service/Connector} Setting property
>> 'compressibleMimeType' to
>> 'text/html,text/xml,text/plain,text/css,text/javascript,application/xml,application/x-javascript,application/javascript,application/x-font-woff,application/font-woff,application/pdf'
>> did not find a matching property.
>>
>> 16-Sep-2020 14:39:42.909 WARNING [main]
>> org.apache.catalina.startup.SetAllPropertiesRule.begin
>> [SetAllPropertiesRule]{Server/Service/Connector} Setting property
>> 'compression' to 'on' did not find a matching property.
>>
>> 16-Sep-2020 14:39:42.909 WARNING [main]
>> org.apache.catalina.startup.SetAllPropertiesRule.begin
>> [SetAllPropertiesRule]{Server/Service/Connector} Setting property
>> 'compressionMinSize' to '2048' did not find a matching property.
>>
>> 16-Sep-2020 14:39:42.909 WARNING [main]
>> org.apache.catalina.startup.SetAllPropertiesRule.begin
>> [SetAllPropertiesRule]{Server/Service/Connector} Setting property
>> 'noCompressionUserAgents' to 'gozilla,traviata' did not find a
>> matching property.
>>
>>
>>
>> I’m running Tomcat 9.0.31.
>>
>>
>>
>> According to documentation these are valid connector attributes/
>>
>> https://urldefense.com/v3/__https://tomcat.apache.org/tomcat-9.0-doc/c
>> onfig/http.html__;!!Li8W9_Um1Taa!tE5GafpOhFxZJTxhrvKgtQvRPdfMY04jnCLVE
>> GdwcPdOT4zoevjuCwYb1Yrbu8i-$
>>
>>
>>
>> Here is what is in the connector. Any ideas?
> 
> Full connector configuration please.
> 
> Mark
> 
> 
>>
>>
>>
>> compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/xml,application/x-javascript,application/javascript,application/x-font-woff,application/font-woff,application/pdf"
>>
>>compression="on"
>>
>>compressionMinSize="2048"
>>
>>noCompressionUserAgents="gozilla,traviata"
>>
>>
>>
>>
>>
>> CONFIDENTIALITY NOTICE: This e-mail and the transmitted documents
>> contain private, privileged and confidential information belonging to
>> the sender. The information therein is solely for the use of the
>> addressee. If your receipt of this transmission has occurred as the
>> result of an error, please immediately notify us so we can arrange for
>> the return of the documents. In such circumstances, you are advised
>> that you may not disclose, copy, distribute or take any other action
>> in reliance on the information transmitted.
>>
>> **
>>
>> *Shawn Beard • Sr. Systems Engineer
>> Middleware Engineering*
>>
>> **
>>
>>
>>
>> *
>>  3840 109th Street Urbandale, IA 50322
>>  Phone: +1-515-564-2528
>>  Email: sbe...@wrberkley.com
>>  Website: berkleytechnologyservices.com
>> <https://www.berkleytechnologyse

Re: tomcat warnings.

2020-09-16 Thread Mark Thomas
On 16/09/2020 19:46, Beard, Shawn wrote:
> I’m getting these in the log:
> 
>  
> 
> 16-Sep-2020 14:39:42.909 WARNING [main]
> org.apache.catalina.startup.SetAllPropertiesRule.begin
> [SetAllPropertiesRule]{Server/Service/Connector} Setting property
> 'compressibleMimeType' to
> 'text/html,text/xml,text/plain,text/css,text/javascript,application/xml,application/x-javascript,application/javascript,application/x-font-woff,application/font-woff,application/pdf'
> did not find a matching property.
> 
> 16-Sep-2020 14:39:42.909 WARNING [main]
> org.apache.catalina.startup.SetAllPropertiesRule.begin
> [SetAllPropertiesRule]{Server/Service/Connector} Setting property
> 'compression' to 'on' did not find a matching property.
> 
> 16-Sep-2020 14:39:42.909 WARNING [main]
> org.apache.catalina.startup.SetAllPropertiesRule.begin
> [SetAllPropertiesRule]{Server/Service/Connector} Setting property
> 'compressionMinSize' to '2048' did not find a matching property.
> 
> 16-Sep-2020 14:39:42.909 WARNING [main]
> org.apache.catalina.startup.SetAllPropertiesRule.begin
> [SetAllPropertiesRule]{Server/Service/Connector} Setting property
> 'noCompressionUserAgents' to 'gozilla,traviata' did not find a matching
> property.
> 
>  
> 
> I’m running Tomcat 9.0.31.
> 
>  
> 
> According to documentation these are valid connector attributes/
> 
> https://tomcat.apache.org/tomcat-9.0-doc/config/http.html
> 
>  
> 
> Here is what is in the connector. Any ideas?

Full connector configuration please.

Mark


> 
>  
> 
> compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/xml,application/x-javascript,application/javascript,application/x-font-woff,application/font-woff,application/pdf"
> 
>    compression="on"
> 
>    compressionMinSize="2048"
> 
>    noCompressionUserAgents="gozilla,traviata"
> 
>  
> 
>  
> 
> CONFIDENTIALITY NOTICE: This e-mail and the transmitted documents
> contain private, privileged and confidential information belonging to
> the sender. The information therein is solely for the use of the
> addressee. If your receipt of this transmission has occurred as the
> result of an error, please immediately notify us so we can arrange for
> the return of the documents. In such circumstances, you are advised that
> you may not disclose, copy, distribute or take any other action in
> reliance on the information transmitted.
> 
> ** 
> 
> *Shawn Beard • Sr. Systems Engineer
> Middleware Engineering*
> 
> **
> 
>   
> 
> *
>  3840 109th Street Urbandale, IA 50322
>  Phone: +1-515-564-2528
>  Email: sbe...@wrberkley.com
>  Website: berkleytechnologyservices.com
> *
> 
> */Technology Leadership Unleashing Business Potential/*
> 
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



[ANN] Apache Tomcat 8.5.58 available

2020-09-16 Thread Mark Thomas
The Apache Tomcat team announces the immediate availability of Apache
Tomcat 8.5.58.

Apache Tomcat 8 is an open source software implementation of the Java
Servlet, JavaServer Pages, Java Unified Expression Language, Java
WebSocket and Java Authentication Service Provider Interface for
Containers technologies.

Apache Tomcat 8.5.x replaces 8.0.x and includes new features pulled
forward from the 9.0.x branch. The notable changes since 8.5.57 include:

- For requests containing the Expect: 100-continue header, optional
  support has been added to delay sending an intermediate 100 status
  response until the servlet reads the request body, allowing the
  servlet the opportunity to respond without asking for the request
  body. Based on a pull request by malaysf.

- Add support for a read idle timeout and a write idle timeout to the
  WebSocket session via custom properties in the user properties
  instance associated with the session. Based on a pull request by
  sakshamverma.

- Update the packaged version of the Tomcat Native Library to 1.2.25

Please refer to the change log for the complete list of changes:
http://tomcat.apache.org/tomcat-8.5-doc/changelog.html


Downloads:
http://tomcat.apache.org/download-80.cgi

Migration guides from Apache Tomcat 7.x and 8.0.x:
http://tomcat.apache.org/migration.html

Enjoy!

- The Apache Tomcat team


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



[ANN] Apache Tomcat 9.0.38 available

2020-09-16 Thread Mark Thomas
The Apache Tomcat team announces the immediate availability of Apache
Tomcat 9.0.38.

Apache Tomcat 9 is an open source software implementation of the Java
Servlet, JavaServer Pages, Java Unified Expression Language, Java
WebSocket and JASPIC technologies.

Apache Tomcat 9.0.38 is a bugfix and feature release. The notable
changes compared to 9.0.37 include:

- For requests containing the Expect: 100-continue header, optional
  support has been added to delay sending an intermediate 100 status
  response until the servlet reads the request body, allowing the
  servlet the opportunity to respond without asking for the request
  body. Based on a pull request by malaysf.

- Add support for a read idle timeout and a write idle timeout to the
  WebSocket session via custom properties in the user properties
  instance associated with the session. Based on a pull request by
  sakshamverma.

- Update the packaged version of the Tomcat Native Library to 1.2.25

Please refer to the change log for the complete list of changes:
http://tomcat.apache.org/tomcat-9.0-doc/changelog.html


Downloads:
http://tomcat.apache.org/download-90.cgi

Migration guides from Apache Tomcat 7.x and 8.x:
http://tomcat.apache.org/migration.html

Enjoy!

- The Apache Tomcat team


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Low throughput with HTTP2

2020-09-15 Thread Mark Thomas
On 15/09/2020 12:46, Martin Grigorov wrote:
> On Tue, Sep 15, 2020 at 2:37 PM Martin Grigorov 
> wrote:
> 
>> Hi,
>>
>> I am running some load tests on Tomcat and I've noticed that when HTTP2 is
>> enabled the throughput drops considerably.
>>
>> Here are the steps to reproduce:
>>
>> 1) Enable HTTP2, e.g. by commenting out this connector:
>>
>> https://github.com/apache/tomcat/blob/d381d87005fa89d1f19d9091c0954f317c135d9d/conf/server.xml#L103-L112
>>
>> 2) Download Vegeta load tool from:
>> https://github.com/tsenart/vegeta/releases/
>>
>> 3) Run the load tests:
>>
>> 3.1) HTTP/1.1
>> echo -e '{"method": "GET", "url": "http://localhost:8080/examples/"}' |
>> vegeta attack -format=json  -rate=0 -max-workers=1000 -duration=10s |
>> vegeta encode > /tmp/http1.json; and vegeta report -type=json
>> /tmp/http1.json | jq .
>>
>> 3.2) HTTP2
>> echo -e '{"method": "GET", "url": "https://localhost:8443/examples/"}' |
>> vegeta attack -format=json -http2 -rate=0 -max-workers=1000 -insecure
>> -duration=10s | vegeta encode > /tmp/http2.json; and vegeta report
>> -type=json /tmp/http2.json | jq .
>>
>> As explained at https://github.com/tsenart/vegeta#-rate -rate=0 means
>> that Vegeta will try to send as many requests as possible with the
>> configured number of workers.
>> I use '-insecure' because I use self-signed certificate.
>>
>> On my machine I get around 14-15K reqs/sec for HTTP1.1 with only responses
>> with code=200 .
>> But for HTTP2 Tomcat starts returning such kind of errors:
>>
>>  "errors": [
>> "Get \"https://localhost:8443/examples/\": http2: server sent GOAWAY
>> and closed the connection; LastStreamID=9259, ErrCode=PROTOCOL_ERROR,
>> debug=\"Stream [9,151] has been closed for some time\"",
>> "http2: server sent GOAWAY and closed the connection;
>> LastStreamID=9259, ErrCode=PROTOCOL_ERROR, debug=\"Stream [9,151] has been
>> closed for some time\"",
>> "Get \"https://localhost:8443/examples/\": http2: server sent GOAWAY
>> and closed the connection; LastStreamID=239, ErrCode=PROTOCOL_ERROR,
>> debug=\"Stream [49] has been closed for some time\""
>>   ]
>>
>> when I ask for more than 2000 reqs/sec, i.e. -rate=2000/1s

That indicates that the client has sent a frame associated with a stream
that the server closed previously and that that stream has been removed
from the Map of known streams to make room for new ones. See
Http2UpgardeHandler.pruneClosedStreams()

It looks like the client is making assumptions about server behaviour
that go beyond the requirements of RFC 7540, section 5.3.4.

>> All the access logs look like:
>>
>> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200
>> 1126
>> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200
>> 1126
>> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200
>> 1126
>> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200
>> 1126
>> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200
>> 1126
>> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200
>> 1126
>>
>> i.e. there are no error codes, just 200.
>> Vegeta reports the error with status code = 0. I think this just means
>> that it didn't get a proper HTTP response but just TCP error.
>> There are no errors in catalina.out.
>>
>> Are there any settings I can tune to get better throughput with HTTP2 ?
>>
>> Tomcat 10.0.0-M8.

If you really want to maximise throughput then you need to reduce the
number of concurrent requests to (or a little above) the number of cores
available on the server. Go higher and you'll start to see throughput
tail off due to context switching.

If you want to demonstrate throughput with a large number of clients
you'll probably need to experiment with both maxThreads,
maxConcurrentStreams and maxConcurrentStreamExecution.

If I had to guess, I'd expect maxConcurrentStreams ==
maxConcurrentStreamExecution and low numbers for all of them to give the
best results.

Mark


> Forgot to mention that I've also tested with JMeter +
> https://github.com/Blazemeter/jmeter-http2-plugin but there it fails with
> OOM if I use more than 2000 virtual users. Increasing the memory still does
> not give such good results as Vegeta for HTTP/1.1. Also JMeter uses
> sequential model.
> 
> For comparison, I've also tested with a simple Golang based HTTP2 server:
> 
> http2-server.go:
> ==
> package main
> 
> import (
> "fmt"
> "log"
> "net/http"
> "os"
> )
> 
> func main() {
> 
> port := 8080
> if port == "" {
>   log.Fatal("Please specify the HTTP port as environment variable, e.g.
> env PORT=8081 go run http-server.go")
> }
> 
> tls_root := "/path/to/certs/"
> srv := {Addr: ":" + port, Handler: http.HandlerFunc(handle)}
> log.Fatal(srv.ListenAndServeTLS(tls_root + "server.crt", tls_root +
> "server.key"))
> }
> 
> func handle(w http.ResponseWriter, r *http.Request) 

[ANN] Apache Tomcat 10.0.0-M8 available

2020-09-15 Thread Mark Thomas
The Apache Tomcat team announces the immediate availability of Apache
Tomcat 10.0.0-M8.

Apache Tomcat 10 is an open source software implementation of the
Jakarta Servlet, Jakarta Server Pages, Jakarta Expression Language,
Jakarta WebSocket, Jakarta Authentication and Jakarta Annotations
specifications.

Users of Tomcat 10 onwards should be aware that, as a result of the move
from Java EE to Jakarta EE as part of the transfer of Java EE to the
Eclipse Foundation, the primary package for all implemented APIs has
changed from javax.* to jakarta.*. This will almost certainly require
code changes to enable applications to migrate from Tomcat 9 and earlier
to Tomcat 10 and later. A migration tool is under development to aid
this process.

Apache Tomcat 10.0.0-M8 is a milestone release of the 10.0.x
branch and has been made to provide users with early access to the new
features in Apache Tomcat 10.0.x so that they may provide feedback. The
notable changes compared to 10.0.0-M7 include:

- For requests containing the Expect: 100-continue header, optional
  support has been added to delay sending an intermediate 100 status
  response until the servlet reads the request body, allowing the
  servlet the opportunity to respond without asking for the request
  body. Based on a pull request by malaysf.

- Add support for a read idle timeout and a write idle timeout to the
  WebSocket session via custom properties in the user properties
  instance associated with the session. Based on a pull request by
  sakshamverma.

- Update the packaged version of the Tomcat Native Library to 1.2.25

Please refer to the change log for the complete list of changes:
http://tomcat.apache.org/tomcat-10.0-doc/changelog.html

Downloads:
http://tomcat.apache.org/download-10.cgi

Migration guides from Apache Tomcat 7.0.x, 8.5.x and 9.0.x:
http://tomcat.apache.org/migration.html

Enjoy!

- The Apache Tomcat team

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Unable to get http redirect to https on Apache Tomcat 8.5.57

2020-09-14 Thread Mark Thomas
On 14/09/2020 20:22, Lee Jarvis wrote:
> Hi all,
>  
> I’m trying to implement SSL and have defined a connector on ports 8080 and 
> 8443. I can connect to either port, but I want any incoming HTTP on 8080 to 
> be redirected to the HTTPS port on 8443, but that’s not happening as I have 
> things configured below. What
> am I missing?
>  
>      connectionTimeout="6"
>     redirectPort="8443"
>     relaxedQueryChars='^{}[]|' />
>  
>  
>      protocol="org.apache.coyote.http11.Http11NioProtocol"
>     connectionTimeout="6"
>     port="8443"
>     maxThreads="200"
>     scheme="https"
>     secure="true"
>     SSLEnabled="true"
>     
> keystoreFile="///C:/apache-tomcat-8.5.57/webapps/cmms/WEB-INF/classes/keystore.jks"
>     keystorePass=""
>     clientAuth="false"
>     sslProtocol="TLSv1.2"
>     relaxedQueryChars='^{}[]|' />
>  
> Thanks & regards,Lee Jarvis

In the configuration above, there is nothing to configure a redirect
from http to https. You'd normally do this with a transport guarantee in
web.xml (other solutions are available).

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Handling Upgrades

2020-09-14 Thread Mark Thomas
On 14/09/2020 17:44, Darryl Philip Baker wrote:
> Until recently most of our Tomcat installations were using the Red Hat 
> distributed version. A version of Tomcat7 with Red Hat backporting security 
> and important break fixes. Red Hat has moved their redistribution of Tomcat 
> to another package other than the OS. A package that it has been decided not 
> to buy. This means I have to support a number of applications that have 
> upgraded to Tomcat9 and are using the Apache-Tomcat binary distribution. This 
> presents me with a problem of how to remain current for security and 
> break-fix updates without spending a lot of people’s time, mine included, 
> regression testing applications.
> 
> Is there a long-term support version (LTS) where only break-fix and security 
> changes are made, and feature enhancements go into the next LTS?

Sorry, no.

Mark


> 
> Darryl Baker, GSEC  (he/him/his)
> Sr. System Administrator
> Distributed Application Platform Services
> Northwestern University
> 1800 Sherman Ave.
> Suite 6-600 – Box #39
> Evanston, IL  60201-3715
> darryl.ba...@northwestern.edu
> (847) 467-6674
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Any update on 9.0.38 release plan

2020-09-14 Thread Mark Thomas
On 14/09/2020 16:57, Christopher Schultz wrote:
> Arshiya,
> 
> On 9/14/20 10:54, Arshiya Shariff wrote:
>> Can we please get a tentative release date for 9.0.38 .
> 
> The vote was started on 2020-09-11 and usually stays open for at least
> 3 days. There are enough votes for the release-vote to pass and there
> have been no reports of any problems, and no "NO" votes.
> 
> So I suspect you'll have an official 9.0.38 sometime this week.
> 
> If you'd like to follow-along, you can subscribe to the dev@ mailing
> list where all the votes are publicly-held. You can look at the
> archives if you want to see the vote(s) but don't want to subscribe.

And I'll add that if, as it appears, that the 9.0.38 is important to you
then the most useful thing you can do is test the release candidate
*before* the release vote completes and, ideally, report back to the
community.

Problems reported before the release vote ends typically result in the
release voting being cancelled, the issue being fixed and a new release
being made. i.e. you get a release with the fix in a few days.

Problems reported after the release votes ends typically result in the
issue being fixed for the next release. i.e. you get a release with a
fix in around a month.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: HTTP2 : PING and GOAWAY sent in the same millisecond

2020-09-14 Thread Mark Thomas
On 13/09/2020 06:19, Arshiya Shariff wrote:
> Hi All,
> 
> The KeepAliveTimeout value is configured to the default value of 20 seconds. 
> So when the connection is idle for 20 seconds, tomcat server(Port:1090) is 
> sending PING followed by GOAWAY within the same millisecond. The client also 
> responds to the  PING in the same millisecond, but before that tomcat has 
> closed the connection.
> Embedded tomcat version-9.0.33
> 
> Please find the link to the pcap for reference:
> https://drive.google.com/file/d/1y9vt9LDw0CeR894TVYSxYSdgCR69wxbQ/view?usp=sharing
> 
> 
>   1.  Is the tomcat behavior correct in sending PING and GOAWAY in the same 
> millisecond ?

Yes. Well, sort of. Tomcat is working as intended but there is room for
improvement. There is no point sending the PING frame in this case
(connection timeout).

>   2.  Should tomcat wait for at least few seconds for PING 
> acknowledgement/response ?

No. Tomcat is not sending the PING frame to keep the connection open.
Tomcat is sending the PING frame to measure the round-tip time (which it
needs to send GOAWAY messages correctly when the Connector is paused.

>   3.  Ripple effect will be frequent connection closures as it finally ends 
> up in a memory leak due to connection open and close ( we know this memory 
> leak issue is fixed in 9.0.37)

No, this PING behaviour has no effect on connection closure. The
connections will close based on the configured timeouts. If you want
longer lived connections, change the timeouts.

Note that there is no API within Servlet to request that an HTTP/2 PING
frame is sent.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Replacing the standard JspWriter

2020-09-12 Thread Mark Thomas
On 12/09/2020 00:30, Adam Rauch wrote:
> I have implemented a custom JspWriter and registered it for use by our
> JSPs using the approach described here:
> https://stackoverflow.com/questions/29508245/jsp-using-a-delegate-for-out-jspwriter-with-jsp-includes-to-change-the-beh
> 
> 
> I created a custom JspFactory that returns a custom JspContext that
> returns my custom JspWriter. I then replaced the standard JspFactory by
> calling JspFactory.setDefaultFactory(). This works, though it results in
> some undesired behavior. I also note that the setDefaultFactory()
> JavaDoc seems to claim that my approach is "illegal".
> 
> So, is there a preferred way for my web application to provide a custom
> JspWriter for my JSPs to use?

How about using an include-prelude mapped to all JSPs to wrap the
default JspWriter with the custom writer?

Mark


> (If you're curious, our JspWriter HTML encodes all strings that aren't
> designated as safe-to-render, like React and other modern JavaScript
> frameworks do. The usual JSP approach is too susceptible to XSS
> vulnerabilities, IMO.)
> 
> Thanks,
> Adam
> 
> 
> 
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Microsoft Edge (Chromium based) not prompting for logons

2020-09-12 Thread Mark Thomas
On 11/09/2020 21:29, Dave Ford wrote:



> I can't find any useful information in the tomcat logs - is it possible
> to turn up the logging for the manager app to see exactly what
> credentials (well, username) is being passed by Edge to it?

If the user isn't authenticated, the request doesn't get as far as the
app. Authentication happens in the Realm (assuming you are using Tomcat
provided authentication.)

Looking at the Realm level logging, you'd need to enable TRACE logging
and even then, there isn't anything to tie the log messages to a
specific request.

Given the description, it sounds like you are using BASIC
authentication. Logging the Authorization header in the access log is
probably the simplest option in that case.

If possible, I'd visit one of the affected users and take a look at the
HTTP headers in the request and response.

HTH,

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: 400 error when upgrading tomcat

2020-09-10 Thread Mark Thomas
On 10/09/2020 21:23, Brian Harris wrote:
> Thanks Christopher.  You just nailed it buddy.  I changed them all to \r\n
> and it got a 200.  I was completely overlooking that as it had never
caused
> a problem before.  Something in 8.5.51 would not allow that anymore.

That is the fix for CVE-2020-1935

See http://tomcat.apache.org/security-8.html#Fixed_in_Apache_Tomcat_8.5.51

It isn't explicit in the changelog because it is security related and
the change log is public before the release is available.

Mark


>
> On Thu, Sep 10, 2020 at 4:07 PM Christopher Schultz <
> ch...@christopherschultz.net> wrote:
>
> Brian,
> 
> On 9/10/20 13:13, Brian Harris wrote:
 We’re having an issue when upgrading Tomcat from 8.5.50 to 8.5.51.
 Since moving to this version, requests sent to the http port are
 failing with a 400 error code(bad request).  The server.xml is
 configured to redirect the http port to the https port.  This has
 worked for years and did not start failing until the upgrade.
 Below is the connector config and the java class used to send a
 test transaction to the server.

 I’ve searched the change log and the only change I can see that
 might cause this is the Bug fix for bug 63966 – Charset of TLS
 message is hard coded to ISO-8859-1.  This bug fix was introduced
 into 8.5.51.  The reason I believe this might be the reason is when
 we would send this request to tomcat 8.5.50 the reply Content-Type
 would look like this:



 Content-Type: text/plain;charset=ISO-8859-1



 With tomcat 8.5.51, I get this:

 Content-Type: text/html;charset=utf-8



 Any ideas why I’m getting the 400 error when upgrading to 8.5.51
 and beyond ?



 Connector config:



 >>>
 connectionTimeout="2"

 redirectPort=""

 />



 >>>
 scheme="https" secure="true"
 ciphers="TLS_DHE_DSS_WITH_AES_128_GCM_SHA256,TLS_DHE_DSS_WITH_AES_256_
> GCM_SHA384,TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_RSA_WITH_AES_256_
> GCM_SHA384,TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDH_ECDSA_WITH_AE
> S_256_GCM_SHA384,TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDH_RSA_WITH_
> AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECD
> SA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECD
> HE_RSA_WITH_AES_256_GCM_SHA384"

  clientAuth="false" sslProtocol="TLS"
 sslEnabledProtocols="TLSv1.2"

 keyAlias="myKey"

 keystore="NONE"

 keystorePass="password"

 keystoreType="PKCS11"

 keystoreProvider="myprovider"

 enableLookups="false"

 server="server"

 "/>





 Java class used to send the test transaction:



 package com.testing;



 import java.io.*;

 import java.net.*;

 import java.util.Date;

 import java.text.DateFormat;

 import java.text.SimpleDateFormat;



 public class RunTestTran{



 public  RunTestTran() {

 }



 public static void main(String [] args){

 RunTestTran recordProcessorTest = new RunTestTran();

 recordProcessorTest.runTran("localhost", ,
 "/requestProcessor/rp");

 }



 private void runTran(String ip, int port, String appName){

 Socket socket = null;

 PrintWriter out = null;

 BufferedReader in = null;

 String dataToSend = "";



 //Create socket connection

 try {

 socket = new Socket(ip, port);

 out = new PrintWriter(socket.getOutputStream(), true);

 in = new BufferedReader(new
 InputStreamReader(socket.getInputStream()));

 } catch  (Exception e) {

 System.out.println("Exception:" + e.toString() );

 System.exit(1);

 }



 DateFormat dateFormat = new SimpleDateFormat("MMddHHmmsss");

 //get current date time with Date() to create a 11 digit tran id

 Date date = new Date();

 String tranId = dateFormat.format(date);

 String PRIMER_TRAN = " V " + tranId + "990JANE
 DOE 100 Redwood Shores Pkwy Redwood City
 CA94065000  PRIMER TRAN";





 try{

 dataToSend = URLEncoder.encode("inputRecord", "UTF-8") + "=" +
 URLEncoder.encode(PRIMER_TRAN, "UTF-8");



 }catch(Exception e){

 System.out.println("Exception caught!" + e.toString());

 }

 // send message

 StringBuffer sb = new StringBuffer();

 sb.append("POST /" + appName + "/wrp HTTP/1.1\r\n");

 // Try connection close-- see if it does close

 sb.append("Connection: close\r\n");

 sb.append("Accept: image/gif, image/x-xbitmap, 

Re: Tomcat Processing Timer Question

2020-09-08 Thread Mark Thomas
On 08/09/2020 21:46, Eric Robinson wrote:
> Hi Mark --
> 
> "If the request is split across multiple packets the timer starts when Tomcat 
> reads the first byte of the request from the first packet.
> Tomcat stops the timer on a request after the last byte of the response has 
> been accepted by the network stack."
> 
> Now we're getting somewhere. If tomcat starts its timer when it reads the 
> first byte of the client's request, and the request is split into multiple 
> packets, doesn't it stand to reason that the timer would run longer when 
> there are TCP retransmits?

For the request, it depends. If the retransmit is for part of the
request body and Tomcat hasn't read that far yet (or starting reading at
all) then it probably won't impact the processing time. If Tomcat is
performing a read and waiting for that packet then it will.

For the response, not unless the response is sfficiently big and the
retransmit sufficiently earlier in the response that the TCP buffers
fill and Tomcat is blocked from further writes.

Mark


> 
> --Eric
> 
>> -Original Message-
>> From: Mark Thomas 
>> Sent: Tuesday, September 8, 2020 3:34 PM
>> To: users@tomcat.apache.org
>> Subject: Re: Tomcat Processing Timer Question
>>
>> On 08/09/2020 21:19, Eric Robinson wrote:
>>> Hi Mark and Christopher,
>>>
>>> For clarification, suppose a client sends and HTTP POST request which
>> is bigger than the PMTU and has to be broken into multiple packets. It
>> sounds like you're saying that the request is buffered by the network stack,
>> and the stack does not send it up to tomcat until the full request is 
>> received.
>> That would make sense if every HTTP request is encapsulated in its own
>> separate TCP connection. Most of the time, that is not the case. A single
>> connection is held open and used for multiple HTTP requests. The network
>> stack has no understanding of anything above TCP, so it does not know when
>> an HTTP request complete. It must therefore deliver whatever it has, and it
>> would be up to tomcat to decide when the HTTP request is complete,
>> wouldn't it?
>>>
>>> If that is the case, tomcat could receive a partial HTTP request and
>> would have to wait for the rest before processing it. So when does tomcat
>> start its processing timer?
>>
>> Tomcat starts the processing timer as soon as Tomcat processes the first
>> bytes of the request. In practice, this means the network stack has to 
>> deliver
>> the data to Tomcat, the Poller fires a read event, a thread is allocated to
>> process that read event, any TLS handshake has completed and Tomcat has
>> read the first real byte of the request.
>>
>> If the request is split across multiple packets the timer starts when Tomcat
>> reads the first byte of the request from the first packet.
>>
>> Tomcat stops the timer on a request after the last byte of the response has
>> been accepted by the network stack.
>>
>> HTH,
>>
>> Mark
>>
>>>
>>>
>>>> -Original Message-
>>>> From: Christopher Schultz 
>>>> Sent: Tuesday, September 8, 2020 1:19 PM
>>>> To: users@tomcat.apache.org
>>>> Subject: Re: Tomcat Processing Timer Question
>>>>
>>> Eric,
>>>
>>> On 9/8/20 13:46, Eric Robinson wrote:
>>>>>> It is my understanding that the AccessLogValve %D field records the
>>>>>> time from when the last byte of the client's request is received to
>>>>>> when the last byte of the server's response is placed on the wire.
>>>>>> Is that correct? If so, would TCP retransmissions impact the timer?
>>>
>>> I'm not positive, but I believe Tomcat has zero visibility into that
>>> level of detail.
>>>
>>>>>> If there are connectivity issues between the client and server,
>>>>>> resulting in TCP retransmits, could that appear as higher response
>>>>>> times in the localhost_access logs?
>>>
>>> This would only happen if the re-transmissions were to cause network
>>> buffering in the OS such that the stream writes (at the Java level)
>>> were to block (and therefore "take time" instead of being essentially
>> instantaneous).
>>>
>>> -chris
>>>>
>>>> -
>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>>>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>>

Re: Tomcat Processing Timer Question

2020-09-08 Thread Mark Thomas
On 08/09/2020 21:19, Eric Robinson wrote:
> Hi Mark and Christopher,
>
> For clarification, suppose a client sends and HTTP POST request which
is bigger than the PMTU and has to be broken into multiple packets. It
sounds like you're saying that the request is buffered by the network
stack, and the stack does not send it up to tomcat until the full
request is received. That would make sense if every HTTP request is
encapsulated in its own separate TCP connection. Most of the time, that
is not the case. A single connection is held open and used for multiple
HTTP requests. The network stack has no understanding of anything above
TCP, so it does not know when an HTTP request complete. It must
therefore deliver whatever it has, and it would be up to tomcat to
decide when the HTTP request is complete, wouldn't it?
>
> If that is the case, tomcat could receive a partial HTTP request and
would have to wait for the rest before processing it. So when does
tomcat start its processing timer?

Tomcat starts the processing timer as soon as Tomcat processes the first
bytes of the request. In practice, this means the network stack has to
deliver the data to Tomcat, the Poller fires a read event, a thread is
allocated to process that read event, any TLS handshake has completed
and Tomcat has read the first real byte of the request.

If the request is split across multiple packets the timer starts when
Tomcat reads the first byte of the request from the first packet.

Tomcat stops the timer on a request after the last byte of the response
has been accepted by the network stack.

HTH,

Mark

>
>
>> -Original Message-
>> From: Christopher Schultz 
>> Sent: Tuesday, September 8, 2020 1:19 PM
>> To: users@tomcat.apache.org
>> Subject: Re: Tomcat Processing Timer Question
>>
> Eric,
> 
> On 9/8/20 13:46, Eric Robinson wrote:
 It is my understanding that the AccessLogValve %D field records the
 time from when the last byte of the client's request is received to
 when the last byte of the server's response is placed on the wire. Is
 that correct? If so, would TCP retransmissions impact the timer?
> 
> I'm not positive, but I believe Tomcat has zero visibility into that level of
> detail.
> 
 If there are connectivity issues between the client and server,
 resulting in TCP retransmits, could that appear as higher response
 times in the localhost_access logs?
> 
> This would only happen if the re-transmissions were to cause network
> buffering in the OS such that the stream writes (at the Java level) were to
> block (and therefore "take time" instead of being essentially instantaneous).
> 
> -chris
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>
> Disclaimer : This email and any files transmitted with it are
confidential and intended solely for intended recipients. If you are not
the named addressee you should not disseminate, distribute, copy or
alter this email. Any views or opinions presented in this email are
solely those of the author and might not represent those of Physician
Select Management. Warning: Although Physician Select Management has
taken reasonable precautions to ensure no viruses are present in this
email, the company cannot accept responsibility for any loss or damage
arising from the use of this email or attachments.
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Tomcat Processing Timer Question

2020-09-08 Thread Mark Thomas
On 08/09/2020 18:46, Eric Robinson wrote:
> It is my understanding that the AccessLogValve %D field records the time from 
> when the last byte of the client's request is received to when the last byte 
> of the server's response is placed on the wire. Is that correct? If so, would 
> TCP retransmissions impact the timer? If there are connectivity issues 
> between the client and server, resulting in TCP retransmits, could that 
> appear as higher response times in the localhost_access logs?

No. Tomcat has no visibility once the network stack has accepted the data.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: HTTP2: Connections abruptly closed by sending GOAWAY

2020-09-07 Thread Mark Thomas
On 07/09/2020 09:29, Arshiya Shariff wrote:
> Hi All,
> Tomcat is closing connections abruptly by sending GOAWAY with reason
> Connection [5309], Stream [57,359], An error occurred during processing that 
> was fatal to the connection .
> 
> Just trying to understand in what scenarios this happens. Can you please help 
> us.

An unhandled InterruptedIOException during I/O.

I/O exception writing an intermediate 100 response for a request with an
expectation.

I/O exception trying to initiate a push request.

I/O exception committing, flushing or closing a stream.

So, generally, any I/O exception that indicates the connection between
the client and the server is broken.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



[ANN] Apache Tomcat Native 1.2.25 released

2020-09-07 Thread Mark Thomas
The Apache Tomcat team announces the immediate availability of Apache
Tomcat Native 1.2.25 stable.

The key features of this release are:
- Improvements to the build system
- Add an option to allow the OCSP check to be bypassed

Please refer to the change log for the complete list of changes:
http://tomcat.apache.org/native-doc/miscellaneous/changelog.html

Downloads:
http://tomcat.apache.org/download-native.cgi

The Apache Tomcat Native Library provides portable API for features
not found in contemporary JDK's. It uses Apache Portable Runtime as
operating system abstraction layer and OpenSSL for SSL networking and
allows optimal performance in production environments.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: regarding CVE-2020-8022 applicable to tomcat 8.5.57

2020-09-02 Thread Mark Thomas
On 02/09/2020 09:28, Olaf Kock wrote:
> 
> On 02.09.20 10:16, Rathore, Rajendra wrote:
>> Please let me know whether CVE-2020-8022 applicable to tomcat 8.5.57 or not, 
>> if yes please let me know which release we fixing it.
> 
> 
> The CVE states:
> 
> "A Incorrect Default Permissions vulnerability in the *packaging of
> tomcat* on SUSE Enterprise Storage 5"
> 
> i.e. it's rather SUSE's packaging than tomcat itself. Correct me if I'm
> wrong.
> 
> If you're running any SUSE system, here are the releases that *they*
> fixed it: https://www.suse.com/de-de/security/cve/CVE-2020-8022/
> 
> I don't expect any update from the generic Apache distribution of Tomcat
> for this CVE, unless I've missed some information that was well hidden
> in the multitude of mentioned SUSE products in that report.

Correct. This is a SUSE issue, not a Tomcat issue.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Release date of latest Tomcat version - 9.0.38

2020-09-01 Thread Mark Thomas
On 01/09/2020 14:01, Christopher Schultz wrote:
> Arshiya,
> 
> On 9/1/20 08:13, Arshiya Shariff wrote:
>> Hi all,
> 
>> The following reported issue - "HTTP/2 Stream.receivedData method
>> throwing continuous NullPointerException in the logs" has been
>> fixed in the latest tomcat.
>> https://bz.apache.org/bugzilla/show_bug.cgi?id=64671
> 
>> Can you please share us the release date of tomcat version 9.0.38
>> . We are waiting for the release dates so we can plan accordingly.
> There are no promises about release schedule.
> 
> Mark, however, has fairly consistently been rolling releases around
> the beginning of each month. If you read the developers list, you'll
> see there was a conversation started last week about the next batch of
> releases.

I'm expecting to tag the next round of releases in a few days. I am
currently waiting for the Commons Daemon 1.2.3 release (it was delayed
by an issue with the code signing service) and the Tomcat Native 1.2.25
release (vote should hopefully complete shortly).

If the Commons Daemon release slips then the Tomcat release will
probably slip.

> Once a release candidate has been announced (look for [VOTE] threads),
> you can help by testing the release:

Big +1

>   1. Run the test suite (ant test)
>   2. Run the release-candidate on your own application
> 
> Your vote is not binding, but if you find a problem, we will likely
> stop the release to fix it.

Indeed. We've done that a few times. Much better to test the release
during the vote (when we can fix it and re-roll the release) rather than
once it is released (when you have to wait for the next release in ~1
month). The number of times people have downloaded a new release and
found an issue that they could have caught if they tested during the vote...

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Implications of setting createDirs attribute on host declarations to false in Tomcat

2020-09-01 Thread Mark Thomas
On 01/09/2020 08:42, Paul wrote:
> Hi Chris,
>
> First of all tnx for your response.
>
> For my own purpose it was about the conf/[engine]/[host] folder and I'm
> now creating that in my dockerfile and thus I got rid of the error.
>
> However, this question is not so much to solve it for just myself, but
> more to be able to give proper directions to anyone dockerizing Tomcat
> or providing 'generic' Tomcat base images.
>
> I'm guessing whether or not to turn of createDirs really depends on how
> Tomcat is used to deploy webapps/war files and whether or not turning
> off createDirs will cause issues on startup/at runtime might depends on
> how hosts are defined (unpackWAR/autoDeploy/deployOnStartup settings),
> but the exact correlation is unclear to me and I cannot find any
> documentation on it.

createDirs was added so that directory creation was triggered early and
any associated errors occurred during start-up rather than at some point
later during an automatic deployment.

The directories are only required if:
a) Tomcat is configured to extract the context.xml file from a WAR
   and copy it to the defined xmlBase for the Host. This will only
   happen if copyXML is true (defaults to false) and deployXML is
   true.
or
b) You want to define Contexts (web applications) via external
   context.xml files

See http://tomcat.apache.org/tomcat-10.0-doc/config/host.html

> Which folders does Tomcat depend on for the deployment of WAR files
> then? The standard Tomcat download comes with the following:
>
> 
>unpackWARs="true" autoDeploy="true">
>
> And already has the webapps folder in the distribution.
>
> Am I correct in saying that createDirs="true" would make sure the
> /webapps

No.

> and /conf/Catalina/localhost directories exist on startup?

Yes.

> Or any other directory of you'd configure other engines/hosts?

No.

> And when does Tomcat utilize the conf/[engine]/[host] folder? As I
> mentioned before I resolved the error in the logs for my own purpose by
> just creating the missing directory while dockerizing my app, but afaics
> Tomcat never writes something to it, which makes me think that the
> folder is only used in certain circumstances.

See above. You can also place appropriately named context.xml files
there yourself.

See
http://tomcat.apache.org/tomcat-10.0-doc/config/host.html#Automatic_Application_Deployment

and

http://tomcat.apache.org/tomcat-10.0-doc/config/automatic-deployment.html

> Again: my goal is to get clear what Tomcat needs in different scenarios,
> thus things can be done correctly in generic Tomcat docker base images
> and/or provide proper instructions with those base images as to how
> (re)configure/adjust the base images based on the different usages.
>
> Maybe it'd make sense to add such clarifications to the Tomcat docs as
> well (I'd be willing to create a PR)
>
> TIA,
>
> Paul
>
>
> On 8/31/2020 5:24 PM, Christopher Schultz wrote:
> Paul,
> 
> On 8/31/20 05:36, Paul wrote:
 Hello,

 When running Tomcat in a Docker container as non-root, I'm getting
 an error entry in the logs:

 Unable to create directory for deployment:
 [/usr/local/tomcat/conf/Catalina/localhost] I traced this down to
 the aforementioned directory not being there when Tomcat gets
 launched and the user running Tomcat does not having the
 permissions to create the directory.

 I also traced where this error message is coming from and its due
 to the createDirs attribute on the host declaration in server.xml
 not being set to false. The docs for the createDirs property states
 that if its set to true (the default) it'll attempt to create some
 directories, log an error when it fails, but the failure wont stop
 the startup of Tomcat.

 But what I cannot find anywhere is why those directories are
 created in the first place and what the ramifications are of them
 not being created when createDirs is set to false.
> You are deploying a WAR file, right?
> 
> If you don't set createDirs="true" then Tomcat will not be able to
> unpack WAR files for you.
> 
 This question is for Tomcat docker images in general (in order to
 determine whether generic Tomcat Docker images should do something
 different to prevent these Errors in the logs), in my specific
 scenario however I'm running one webapp: an exploded WAR in the
 webapps/ROOT directory, which comes with its on context.xml in
 /webapps/ROOT/META-INF/context.xml
> If webapps/ already exists, what directory is Tomcat trying to create?
> conf/[engine]/[host]?
> 
> Can you arrange for conf/[engine]/[host] (usually
> conf/Catalina/localhost) to exist so that the setting for createDirs
> doesn't matter?
> 
> -chris
>
>


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Probelm with shutdown script

2020-08-29 Thread Mark Thomas
On 28/08/2020 20:54, Christopher Schultz wrote:
> Calder,
> 
> On 8/27/20 18:23, calder wrote:
>> On Thu, Aug 27, 2020, 16:16 Christopher Schultz <
>> ch...@christopherschultz.net> wrote:
> 
>> [ snip ]
> 
>> If you want to *kill* the application and it won't shut down on
>> its
>>> own, SIGKILL is the answer. But that's not a great way to shut
>>> down an application /in general/ because the application might
>>> want/need to do something convenient on shutdown (flush caches,
>>> save state, etc.).
> 
> 
> 
>> SIGTERM is the polite way to ask an application to shut down. A JVM
>> will respond to this signal. (SIGHUP will also initiate a shut
>> down, but TERM is preferred).  Using TERM will cause the shutdown
>> hooks to initiate.
> 
>> As Chris states, SIGKILL is a last resort (hooks are not called).
> 
> If you send a SIGTERM to a Tomcat process, your application's
> ServletContextListeners and stuff like that will not run.

Yes it will. Tomcat will shutdown cleanly (after cleanly shutting down
any running web applications).

> Sure, you
> can register shutdown-hook, but ... eew. That's like, AWFUL, man.

You mean like Tomcat already does? Applications don't/shouldn't do this
because Tomcat has already done so.

Mark


> 
> -chris
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Problem class loaders dont find classes

2020-08-29 Thread Mark Thomas
On 29/08/2020 22:19, Carles Franquesa wrote:
> IS NOT ALLOWED TO STORE JSPS IN A HIERARCHY OF FOLDERS. ALL JSP FILES MUST
> GO ON THE ROOT WEB FOLDER.

That is not correct. JSPs can be placed anywhere in the web application.

If you provide the simplest possible set of steps to recreate the
problem from a clean install of Apache Tomcat someone should be able to
investigate what the problem is.

Mark


> 
> Three days to solved it. I feel quite a bit embarrassed for it.
> 
> Anyway, problem solved!
> 
> Thanks to whom's been interested on.
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Probelm with shutdown script

2020-08-27 Thread Mark Thomas
On 27/08/2020 19:43, Roger Marquis wrote:
> Mark Thomas wrote:
>> Those are all application issues. The application should shut itself
>> down cleanly. Tomcat is complaining because it hasn't.
> 
> I don't know Mark, most Java/Tomcat engineers expect an application to
> shutdown when it's os/container/shell/parent shuts-down.  Can you help
> us understand why Tomcat is different in this respect than say Apache
> httpd?

Java doesn't provide an API a cleanly stop another thread. If an
application starts a non-daemon thread but doesn't stop it there is
nothing Tomcat can do to stop that thread and the JVM won't shutdown
until the thread stops. Using OS level tools to stop the process is the
only option.

Hence, applications are responsible to cleaning up the resources they
create. For example, if an app starts a thread then the app is also
responsible for stopping it. Generally, apps use one or more
ServletContextListeners to do this.

Along similar lines, failing to clean up resources can also trigger
memory leaks. Tomcat will clean those up where it can and log the
problems it finds so the application can address them. While such issues
are unimportant when shutting Tomcat down entirely, they are much more
serious if the web application is being redeployed.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Tomcat 9.0.29 - HTTPS threads age, max connections reached, Tomcat not responding on 8443

2020-08-27 Thread Mark Thomas
On 27/08/2020 18:57, David wrote:
> On Thu, Aug 27, 2020 at 12:35 PM Christopher Schultz
>  wrote:
 Is there a graceful way to script the termination of threads in
 case Tomcat isn't able to for whatever reason?
> 
> Not really.

What you can do is take a thread dump when this happens so you can see
what the threads are doing. That should provide some insight to where
the problem is.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Tomcat v9 - Insecure transport vulnerability reported by Qualys

2020-08-27 Thread Mark Thomas
On 27/08/2020 11:32, Phoenix, Merka wrote:



> The error message returned by the Tomcat service, while certainly helpful to 
> the remote client, is returning more information than it should (from a 
> security-viewpoint).

What, exactly, are the security concerns here? Your comment suggests
there is some form of information disclosure vulnerability. What
information?

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Tomcat v9 - Insecure transport vulnerability reported by Qualys

2020-08-27 Thread Mark Thomas
On 27/08/2020 06:31, Terence M. Bandoian wrote:
> On 8/26/2020 11:27 PM, Pratik Shrestha wrote:



>> For me, there are two options for the fix which I am not able  to make
>> them
>> work.
>>
>> 1. Either show 'ERR_EMPTY_RESP' like old Tomcat version 7 used to
>> show. As
>> far as I know, with Tomcat 7 giving that error, Qualys did not use to
>> show
>> this vulnerability.
>> 2. *Best is to do a redirect* when Tomcat sees error 400 to https URL.
>> Like
>> in Apache, we can add below.
>> 'ErrorDocument 400 "*https*://xxx.xxx.xxx.xxx"
>> But as understood, redirect only works with error 3XX and ErrorDocument
>> feature is not there in Tomcat yet.



> With HTTPD rewrite, whether or not the request is encrypted or sent to
> the correct port can be detected and the request redirected as
> appropriate. Maybe the same can be done with the rewrite valve used with
> Tomcat.

This isn't currently possible with Tomcat because of detection of plain
text HTTP when TLS should be used (and the generation of the associated
response) is much, much earlier in the processing chain than the rewrite
valve.



>>>> On 8/26/20 13:59, Mark Thomas wrote:
>>>>> On 26/08/2020 17:50, Christopher Schultz wrote:



>>>>>> I'm interested in having Tomcat be able to pass these (admittedly
>>>>>> stupid) security requirements,
>>>>> I have no interest in adding bloat to Tomcat so it can pass so called
>>>>> security requirements that have no relevance to actual security. Those
>>>>> sort of changes are the sort that get me starting to think about using
>>>>> a veto.
> Understood. But what does the OP have in terms of options at this point?
> 
> 1. Ignore the complaint (probably not possible) 2. Request a waiver for
> this issue (probably not possible, or at least would require 10 years of
> red tape) 3. Front the server with httpd + "ErrorDocument 400" (which
> ... I
> think will *also* reply with a plaintext response, right?) 4. Switch to
> Jetty>
> I'm trying to avoid "the easiest thing" which is probably to switch to
> Jetty. I know our "customers" don't pay for Tomcat, but losing a
> "customer"
> sucks.

One of the things I love about working Tomcat is when this sort of
security nonsense comes along, I can a) call it out and b) veto (if I
have to) the implementation without someone higher up the organisational
hierarchy able to play the "I don't care if it is nonsense, our
customers want it so you have to implement it" card.

My objection to implementing or changing features in response to
"security nonsense" is that it perpetuates the problem. If people who
know this is "security nonsense" just accept it rather than arguing
against it, that nonsense eventually becomes "security fact". I think
the world could do with a little more security fact and a little less
security nonsense.

That said, I'm not against changing this feature where that change
offers real benefits to users.

> How about being able to specify the response text, possibly blank?

While I remember, there was the issue raised that the response wasn't
UTF-8 and we changed hard-coded response to UTF-8 rather than provide an
option.

My concern with anything along the lines of making it configurable is
that because this response is generated outside of the normal HTTP
processing infrastructure you can quickly get into the situation where
you end up replicating functionality we already have elsewhere.

> I think "ErrorDocument 400" with nothing else might mean the same
> thing as
> [[ErrorDocument 400 ""]] meaning that the response will include NO
> CONTENT.
> Maybe that's what Qualys is looking for.

My reading of the thread so far is that the security scanner expects
either a TLS error or a redirect to https.

The redirect option is interesting. I can see user benefit in http
requests to an https port getting redirected to https. The tricky part
is implementing it. Redirection to a fixed URL is quite simple. As soon
as you get into redirecting the actual user's request you open up the
HTTP request parsing can of worms.

I'm wondering if there is a way to utilise the standard request
processing infrastructure for these http over https requests (so we can
use the existing http processing infrastructure to parse the request and
issue a redirect to https) without creating the risk that we process the
http request without the redirect as that would be very bad.

Given the number of times redirect has been mentioned I think I'll spend
some time looking into how possible it is to redirect the user's http
request to https. My instinct is that it isn't going to be doable but it
is worth a look.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Tomcat v9 - Insecure transport vulnerability reported by Qualys

2020-08-26 Thread Mark Thomas
On 26/08/2020 17:50, Christopher Schultz wrote:
> On 8/26/20 05:27, Mark Thomas wrote:
>> On 26/08/2020 08:14, Martin Grigorov wrote:
>>> Hi,
>>>
>>> On Wed, Aug 26, 2020 at 7:53 AM Pratik Shrestha
>>>  wrote:
>>>
>>>> Thanks for reply,
>>>>
>>>> Hi Peter - it complains on port 8443 which belongs to Tomcat.
>>>>
>>>> Hi Mark - Yes. making HTTP request on HTTPS is wrong. But this
>>>> security vulnerability is given to us by Qualys scan. It tries
>>>> to post plain HTTP request on HTTPS port and then gets error
>>>> message "Bad Request. This combination of host and port
>>>> requires TLS." which is security loop hole for Qualys.
> 
>> On what basis?
> 
>> I fail to see any security issue here other than "Qualys says so"
>> which is not a valid description of a security vulnerability.
> 
> My guess is that this is some form of "server fingerprinting" that
> they are claiming, like "Zomg! It says Server: Apache-Coyote/1.1! You
> are $uper vulnerable to 0days, now!".

The entire response, including headers is,

=
HTTP/1.1 400
Content-Type: text/plain;charset=UTF-8
Connection: close

Bad Request
This combination of host and port requires TLS.
=

> Pratik, can you please be very clear about what the actual complaint
> is? Are they objecting to one or more of the following:
> 
> 0. Any legible response at all (meaning they just want a
> connection-drop response)
> 1. Server: Apache-Coyote/1.1 response header
> 2. Predictable / stock text (e.g. "Bad Request. This
> combination of host and port requires TLS." identifies the server as
> Tomcat v.x.y or later)
> 3. Actual Tomcat version number in response
> 
>> Absent a description of how this can be exploited (and I'll be
>> very surprised if this can be exploited), there is no security
>> issue here and Tomcat will not be making any changes.
> 
> It seems reasonable to (configurably) strip-out version information if
> there is anything in there... which there probably is not.

Correct, there isn't.

> I'm interested in having Tomcat be able to pass these (admittedly
> stupid) security requirements,

I have no interest in adding bloat to Tomcat so it can pass so called
security requirements that have no relevance to actual security. Those
sort of changes are the sort that get me starting to think about using a
veto.

> so maybe we could have a setting on the
>  that can allow ERR_EMPTY_RESP to be sent if the handshake
> fails due to probably-not-encrypted just like older versions of Tomcat> did.

That sounds suspiciously like bloat to me.

I've never been particularly convinced by the fingerprinting argument.
Either you are running a version without any published security
vulnerabilities that affect you (in which case fingerprinting doesn't
help the attacker) or you are running a version with published security
vulnerabilities that do affect you and you are relying on security by
obscurity - which is no security at all.

> IMO, being able to reply in plaintext like this is a *feature* (one
> that I personally and specifically lobbied to have added to Tomcat)
> and shouldn't be removed. If it's not the end of the world to add an
> option to disable it, though, I think we ought to do it.

I'm not (yet) convinced of the benefits of such an option.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Tomcat v9 - Insecure transport vulnerability reported by Qualys

2020-08-26 Thread Mark Thomas
On 26/08/2020 08:14, Martin Grigorov wrote:
> Hi,
> 
> On Wed, Aug 26, 2020 at 7:53 AM Pratik Shrestha  wrote:
> 
>> Thanks for reply,
>>
>> Hi Peter - it complains on port 8443 which belongs to Tomcat.
>>
>> Hi Mark - Yes. making HTTP request on HTTPS is wrong. But this security
>> vulnerability is given to us by Qualys scan. It tries to post plain HTTP
>> request on HTTPS port and then gets error message "Bad Request. This 
>> combination
>> of host and port requires TLS." which is security loop hole for Qualys.

On what basis?

I fail to see any security issue here other than "Qualys says so" which
is not a valid description of a security vulnerability.

Absent a description of how this can be exploited (and I'll be very
surprised if this can be exploited), there is no security issue here and
Tomcat will not be making any changes.

>> This is behaviour of Apache HTTP server also. But in Apache though, we can
>> get rid of this by using "ErrorDocument 400" directive. Do we have similar
>> in Tomcat? I have already tried using
>>
>> 
>>400
>>/error.jsp
>>  
>>
> 
> This won't work because Tomcat stops the request earlier and doesn't pass
> it to your application.
> I haven't tried it but it may work with a custom Valve, extending
> ErrorReportValve.

That won't work. The error message is generated in the low level I/O
code as part of the TLS handshake.

>> Not sure, but my idea was to add redirect code on error.jsp page. But
>> above never works. It never reaches error.jsp page. Just sticks in default
>> error message page mentioned above.
>>
>> Btw..you can see the result from Qualys attached.
>>
> 
> What is the desired behavior expected by Qualys ?
> Because at the moment Tomcat returns a text/html error page and you try to
> "fix" it by returning a custom text/html error page. I don't see how this
> will change the Qualys report.

Indeed.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Probelm with shutdown script

2020-08-25 Thread Mark Thomas
On 25/08/2020 16:40, ratatouille wrote:
> Mark Thomas  schrieb am 25.08.20 um 11:31:59 Uhr:
> 
>> On 25/08/2020 11:07, ratatouille wrote:
> 
>>> I am running openmeetings on a CentOS 8 server and start it with startup.sh
>>> in the bin-folder.
>>>
>>> The problem is when I execute shutdown.sh the process still exists after.
>>> I have to kill it manually.
> 
>>> I was told these scripts are un-modified scripts of Apache Tomcat
>>> 9.0.37, that's why I am asking here for help.
>>>
>>> Any hint on this?  
>>
>> Take a Java thread dump of the still running process. That should point
>> you towards what hasn't shutdown properly.
> 
> I see some warnings and stack traces when I execute shutdown.sh.
> For example:

Those are all application issues. The application should shut itself
down cleanly. Tomcat is complaining because it hasn't.

Mark


> 
> 25-Aug-2020 17:31:49.246 WARNUNG [main] 
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The 
> web application [openmeetings
> ] appears to have started a thread named [nioEventLoopGroup-3-4] but has 
> failed to stop it. This is very likely to create a memory leak. Stack tr
> ace of thread:
>  java.base@11.0.8/java.lang.Thread.sleep(Native Method)
>  
> io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
>  io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
>  
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
>  io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>  
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>  java.base@11.0.8/java.lang.Thread.run(Thread.java:834)
> 25-Aug-2020 17:31:49.280 SCHWERWIEGEND [main] 
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks 
> The web application [o
> penmeetings] created a ThreadLocal with key of type [java.lang.ThreadLocal] 
> (value [java.lang.ThreadLocal@14a07c8d]) and a value of type [io.nett
> y.util.internal.InternalThreadLocalMap] (value 
> [io.netty.util.internal.InternalThreadLocalMap@6096444e]) but failed to 
> remove it when the web app
> lication was stopped. Threads are going to be renewed over time to try and 
> avoid a probable memory leak.
> 25-Aug-2020 17:31:49.289 INFORMATION [main] 
> org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler 
> ["http-nio-5080"]
> 25-Aug-2020 17:31:49.296 INFORMATION [main] 
> org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler 
> ["https-jsse-nio-5443"]
> 25-Aug-2020 17:31:49.302 INFORMATION [main] 
> org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler 
> ["http-nio-5080"]
> 25-Aug-2020 17:31:49.305 INFORMATION [main] 
> org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler 
> ["https-jsse-nio-5443"]
> 25-Aug-2020 17:31:50.921 INFORMATION [mysql-cj-abandoned-connection-cleanup] 
> org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResour
> ceLoading Illegal access: this web application instance has been stopped 
> already. Could not load []. The following stack trace is thrown for debu
> gging purposes as well as to attempt to terminate the thread which caused the 
> illegal access.
> java.lang.IllegalStateException: Illegal access: this web application 
> instance has been stopped already. Could not load []. The following
>  stack trace is thrown for debugging purposes as well as to attempt to 
> terminate the thread which caused the illegal access.
> at 
> org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1385)
> at 
> org.apache.catalina.loader.WebappClassLoaderBase.getResource(WebappClassLoaderBase.java:1038)
> at 
> com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.checkThreadContextClassLoader(AbandonedConnectionCleanupThread.java:117)
> at 
> com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:84)
> at 
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at 
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 
> The output of the shutdown is a lot longer than the above.
> 
> I have no clue about Java and tomcat so I can't do anything about this.
> 
>   Andreas
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Upgrade: tomcat8w.exe //ES//example - dump Java Options and other information to tomcat9

2020-08-25 Thread Mark Thomas
On 04/08/2020 14:47, Christopher Schultz wrote:



>> Enhancement requests for this should go to Commons Daemon. Should
>> be simple enough just to dump current config.
> 
> Done.
> 
> https://issues.apache.org/jira/browse/DAEMON-422

Done.

It outputs the command to (re-)create the current config to stderr (in a
similar manner to version and usage). Users are free to pipe that to a
file or whatever else they want to do with it.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Tomcat v9 - Insecure transport vulnerability reported by Qualys

2020-08-25 Thread Mark Thomas
On 25/08/2020 11:14, Pratik Shrestha wrote:
> Hi all,
> 
> Tomcat version: 9.0.37
> 
> Our website is running on Tomcat. We did Qualys vulnerability scan on our
> site. Scan shows below vulnerability.
> 
> Insecure transport
> Group: Information Disclosure
> CWE CWE-319
> OWASP A3 Sensitive Data Exposure
> WASC WASC-4 INSUFFICIENT TRANSPORT LAYER PROTECTION
> 
> Please note
> 1. HTTP port is not enabled.
> 2. We have only opened HTTPS port 8443. But when we connect this HTTPS port
> with HTTP (http://www.oursite.com:8443/), we get an error "Bad Request. This
> combination of host and port requires TLS."
> 3. Due to the above error message, we get this vulnerability error from
> Qualys.
> 4. We have already enabled HSTS.
> 5. We have enabled Rewrite Valve also to direct all HTTP to HTTPS. But it
> never works. It is like, Tomcat doesn't care about Rewrite or HSTS. It just
> finds someone is accessing HTTPS port with HTTP protocol and then just
> throws error 400 'Bad Request'
> 6. Note that Tomcat version 7 used to send the error 'ERR_EMPTY_RESP' which
> should still be okay.
> 
> We already tried to find the fix for this issue on the web but in vain.
> 
> Kindly help if anyone has found a way to fix it.

Fix what?

If you make an HTTP request to an HTTPS port, Tomcat provides a helpful
error message.

I don't see any security issues here.

(And before anyone claims the request sent in the clear is insecure I'll
point out that the request is sent in the clear irrespective of whether
Tomcat responds with an HTTP/1.1 clear text error message or a cryptic
TLS failure).

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Probelm with shutdown script

2020-08-25 Thread Mark Thomas
On 25/08/2020 11:07, ratatouille wrote:
> Hello!
> 
> I am running openmeetings on a CentOS 8 server and start it with startup.sh
> in the bin-folder.
> 
> The problem is when I execute shutdown.sh the process still exists after.
> I have to kill it manually.
> 
> # 
> -
> # Stop script for the CATALINA Server
> # 
> -
> 
> # Better OS/400 detection: see Bugzilla 31132
> os400=false
> case "`uname`" in
> OS400*) os400=true;;
> esac
> 
> # resolve links - $0 may be a softlink
> PRG="$0"
> 
> while [ -h "$PRG" ] ; do
>   ls=`ls -ld "$PRG"`
>   link=`expr "$ls" : '.*-> \(.*\)$'`
>   if expr "$link" : '/.*' > /dev/null; then
> PRG="$link"
>   else
> PRG=`dirname "$PRG"`/"$link"
>   fi
> done
> 
> PRGDIR=`dirname "$PRG"`
> EXECUTABLE=catalina.sh
> 
> # Check that target executable exists
> if $os400; then
>   # -x will Only work on the os400 if the files are:
>   # 1. owned by the user
>   # 2. owned by the PRIMARY group of the user
>   # this will not work if the user belongs in secondary groups
>   eval
> else
>   if [ ! -x "$PRGDIR"/"$EXECUTABLE" ]; then
> echo "Cannot find $PRGDIR/$EXECUTABLE"
> echo "The file is absent or does not have execute permission"
> echo "This file is needed to run this program"
> exit 1
>   fi
> fi
> 
> exec "$PRGDIR"/"$EXECUTABLE" stop "$@"
> 
> I was told these scripts are un-modified scripts of Apache Tomcat
> 9.0.37, that's why I am asking here for help.
> 
> Any hint on this?

Take a Java thread dump of the still running process. That should point
you towards what hasn't shutdown properly.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Allowing dir listing of root (/) dir of the machine

2020-08-25 Thread Mark Thomas
On 25/08/2020 09:19, Mark Thomas wrote:
> On 24/08/2020 15:41, Aryeh Friedman wrote:
> 
> 
> 
>> Tried and it gives me /usr/local/apache-tomcat-9.0/webapps as the effective
>> dir.   This is *NOT* what I meant by the root dir I meant the one that is
>> the highest point in the file system hierarchy (i.e. the one you get when
>> at a shell prompt when you type "cd /") [this is for a Unix machine of
>> course since Windows has no concept of such a directory/folder]
> 
> Sorry, got my roots mixed up.
> 
> 
> 
> gives me an empty directory listing as well - and it isn't a file
> permissions issue.
> 
> I need to do some debugging to figure out what is going on...

Edge case bug in path validation. Will be fixed in the next round of
releases (expected early next month).

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Allowing dir listing of root (/) dir of the machine

2020-08-25 Thread Mark Thomas
On 24/08/2020 15:41, Aryeh Friedman wrote:



> Tried and it gives me /usr/local/apache-tomcat-9.0/webapps as the effective
> dir.   This is *NOT* what I meant by the root dir I meant the one that is
> the highest point in the file system hierarchy (i.e. the one you get when
> at a shell prompt when you type "cd /") [this is for a Unix machine of
> course since Windows has no concept of such a directory/folder]

Sorry, got my roots mixed up.



gives me an empty directory listing as well - and it isn't a file
permissions issue.

I need to do some debugging to figure out what is going on...

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Tomcat 9 : Unable to specify wildcard care name in Host

2020-08-24 Thread Mark Thomas
On 24/08/2020 13:14, Tom Chiverton wrote:



> Am I mis-reading the docs ?

Yes.

The relevant part is:

"Aliases may also use the wildcard form"

Alias is a sub-element of Host. The name element of Host needs to use a
valid host name.

Mark


> 
> Tom Chiverton --
> *Tom Chiverton*
> Lead Developer
> e: t...@extravision.com 
> p: 0161‌ 817‌ 2929
> t: @extravision
> w: extravision‌.com‌
> 
> Extravision 
> 
> Registered in the UK at: Tomorrow‌, MediaCityUK‌, Salford‌ M50‌ 2AB.
> Company Reg No: 05017214‌ VAT: GB 824‌ 5386‌ 19
> 
> This e-mail is intended solely for the person to whom it is addressed
> and may contain confidential or privileged information. Any views or
> opinions presented in this e-mail are solely of the author and do not
> necessarily represent those of Extravision Ltd.
> 
> 
> 
> __
> This email has been scanned by the Symantec Email Security.cloud service.
> For more information please visit http://www.symanteccloud.com
> __

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Allowing dir listing of root (/) dir of the machine

2020-08-24 Thread Mark Thomas
On 23/08/2020 22:05, Aryeh Friedman wrote:
> In order to allow my developers to quickly access any temporarily produced
> html files created/stored outside of webapps (such as those created by the
> jacoco test coverage tool) I want to allow read only access to the root
> directory of the development server (firewalled and all access outside of
> the LAN is disabled) via tomcat.   I can get it to do any directory
> *EXCEPT* / as the docBase but a docBase of "/" returns an empty dir listing
> (which is obviously wrong):
> 
> In config/web.xml:
> 
> default
> 
> org.apache.catalina.servlets.DefaultServlet
> 
> debug
> 0
> 
> 
> listings
> true
> 
> 1
> 

That should be sufficient to enable directory listings for all web
applications.

> In server.xml (this works):
>  unpackWARs="true" autoDeploy="true">
> 
> 
> 
> 
> 
>  directory="logs"
>prefix="localhost_access_log" suffix=".txt"
>pattern="%h %l %u %t %r %s %b" />
> 
> 

I'd do this with a ROOT.xml file in
$CATALINA_BASE/conf/Catalina/localhost but the above should work.

> But this does not work:
> 

The docBase is not correct (it should be "") but Tomcat probably will
let you get away with that.

I tested this locally and it works as expected.

Maybe a file permissions issue?

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Embedded and Standalone Tomcat

2020-08-21 Thread Mark Thomas
On 21/08/2020 11:27, S Abirami wrote:
> Hi All,
> 
> In our application, we used to create embedded tomcat instance by taking a 
> copy of lib jars from the Deployable Tomcat.
> It's working properly. I have noticed that there is some jars in Embed package
> 
> https://mirrors.estointernet.in/apache/tomcat/tomcat-9/v9.0.37/bin/embed/apache-tomcat-9.0.37-embed.tar.gz
> 
> Please let me know the difference between the jars in Core archive and Embed 
> archive.

The classes are the same. The embedded option just packages them in
fewer JARs.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: How to upload Files larger than 2GB

2020-08-19 Thread Mark Thomas
On 19/08/2020 10:58, Martin Knoblauch wrote:
> Hi,
> 
>  our customer has the following setup:
> 
> Apache/HTTPD(2.4.43)->mod_jk(1.2.48)->Tomcat(9.0.12).
> 
>  The application hosted by Tomcat has a REST interface that allows file
> upload using POST requests. The problem now is that we get a 500 response
> when we try to upload files larger than 2 GB. But this happens only when
> using the full path from Apache to Tomcat. When we try the upload directly
> via the Tomcat HTTP port all is good
> 
> This leaves me with the question where the problem is. Is it the Tomcat AJP
> part, is it mod_jk or is it HTTPD? From the logs I have the impression the
> file never appears on the Tomcat side.

What logs?

If the request reached Tomcat it should appear in the access log and a
Tomcat generated 500 response should result in a stack trace in the logs.

The mod_jk log will also confirm what is and is not sent to Tomcat.

> So I found the LimitRequestBody directive for HTTPD. But the description
> leaves me wondering. It says "This directive specifies the number of bytes
> from 0 (meaning unlimited) to 2147483647 (2GB) that are allowed in a
> request body." Means "unlimited" really no limit, or is it 2GB. Anyway, I
> set it to 0 with no success.
> 
> Then looking at the Tomcat configuration. The HTTP connector (working)
> looks like this:
> 
> connectionTimeout="2"
>maxPostSize="209715200"
>redirectPort="8443" />
> 
> Which makes me wonder why it works. It should bail out at 200 MB.

That limit only applies to the automated processing of request bodies as
per section 3.1.1 of the Servlet 4.0 specification.

If the application (or a library it uses) reads the request body
directly, there is no limit. The application is meant to provide
whatever limits it considers appropriate.

> The AJP
> connector looks like:
> 
> connectionTimeout="60"
> maxPostSize="2097152"
> maxThreads="300"
> minSpareThreads="10"
> redirectPort="8443" />
> 
> I upped the maxPostSize to 20GB, but it did not help.

Same as for HTTP above.

> Any advice is highly welcome. Just not
> 
> - Don't use HTTP for uploading large files. It is the mechanism the
> application offers
> - Don't allow upload of large files. Unfortunately it is a valid use-case.

Confirm with the mod_jk log whether the request is sent to Tomcat.

I wonder if httpd/mod_jk is trying cache the entire request body before
forwarding. How is the request sent? With chunked-encoding or with a
content-length? Does switching to the other one help?

You might need to take a look at the mod_jk (or httpd) source to see how
request bodies are handled.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Login appears only once

2020-08-18 Thread Mark Thomas
On 18/08/2020 19:45, Anwar AliKhan wrote:
> I rebooted the machine , then the login box appeared .
> Obviously this is not an ideal solution!

Did you close the browser between tests?

Mark


> On Tue, 18 Aug 2020, 19:07 Anwar AliKhan,  wrote:
> 
>> Hi,
>> I deployed an app called tomee using the tomcat manager app.
>>
>> The first time I selected the app in the tomcat manager to run it.
>> a login appeared asking for username and password.
>>
>> I had not set it up. So it took me to the 403  page .
>>
>> Now I have set  up tomee-admin user.
>>
>> I stopped restarted tomcat for it to register the contents of
>> tomcat-users.xml
>> I no longer get the login Box. It goes straight to the 403 page.
>>
>> *what is the problem ? Thanks for your help*
>>
>>
>>
>> 
>> 
>> 
>> 
>> 
>> 
>> HTTP Status 403 – Forbidden
>> --
>>
>> *Type* Status Report
>>
>> *Message* Access to the requested resource has been denied
>>
>> *Description* The server understood the request but refuses to authorize
>> it.
>> --
>> Apache Tomcat/9.0.37
>>
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Tomcat behind httpd, with Let's Encrypt and Certbot

2020-08-17 Thread Mark Thomas
On 16/08/2020 18:00, James H. H. Lampert wrote:
> Permit me to clarify:
> 
> 1. The existing httpd server on this box, and its certbot setup may be
> extended/expanded, but not otherwise disturbed.
> 
> 2. Running Tomcat independently of httpd on this box is not an option,
> because *both* are to be visible to the outside world on port 443 of the
> same IP address. Doing so was not merely "an option," but *mandatory* on
> the other box, which has Tomcat and httpd on separate ports.
> 
> 3. At this point, the concern is making certain that the httpd virtual
> host for the new subdomain provides for the needs of both Certbot and
> Tomcat. Then, I can worry about adding the new subdomain to Certbot.

First of all, to confirm I am reading the config correctly:

- httpd redirects all http requests to https
- anything proxied to Tomcat MUST have been received by httpd over https

Given you don't mind whether proxying to Tomcat is over http or https, I
recommend http and an http connector in Tomcat with the following settings:

SSLEnabled="false", secure="true", scheme="https"

I'd be wary of directory traversal issues with the IP controls on
Manager and Host Manager access in httpd. There are some edge cases
where the Servlet spec's view on matching URIs to targets and the HTTP
spec's view are not entirely consistent. This has been known to expose
directory traversal issues. I'd recommend using the RemoteIpValve to
expose the original IP to Tomcat and then perform the IP filtering in
Tomcat. Whether you keep the filtering in httpd (pro of early rejection
vs con of having to keep configs in sync) is up to you.

HTH,

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: CVE reporting discrepencies

2020-08-14 Thread Mark Thomas
On 14/08/2020 12:24, Nic P wrote:
> Mark - per NIST this CVEis listed as impact to tomcat
> https://nvd.nist.gov/vuln/detail/CVE-2016-5388 which is how we came to find
> evidence for audit on the version where this was remediated.

As per that description:

...this is not a CVE ID for a vulnerability.

Mark


> 
>  On Fri, Aug 14, 2020 at 4:15 AM Mark Thomas  wrote:
> 
>> On 13/08/2020 20:52, Nic P wrote:
>>> Hi
>>>
>>> Can anyone help me understand why some CVE's show in the changelog but
>> not
>>> on the security report?
>>>
>>> Example is  CVE-2016-5388 which shows as fixed in 8.0.37 changelog but
>>> missing on the security report.
>>>
>>> This has come up in a audit and hard to explain which is the System of
>>> Record information for security fixes.
>>>
>>>
>> https://tomcat.apache.org/security-8.html#Fixed_in_Apache_Tomcat_8.5.5_and_8.0.37
>>>
>>> https://tomcat.apache.org/tomcat-8.0-doc/changelog.html
>>
>> Because CVE-2016-5388 is not an Apache Tomcat vulnerability. The
>> changelog refers to the mitigation applied to Apache Tomcat to protect
>> users if they happen to be using vulnerable CGI executables.
>>
>> Mark
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>>
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: CVE reporting discrepencies

2020-08-14 Thread Mark Thomas
On 13/08/2020 20:52, Nic P wrote:
> Hi
> 
> Can anyone help me understand why some CVE's show in the changelog but not
> on the security report?
> 
> Example is  CVE-2016-5388 which shows as fixed in 8.0.37 changelog but
> missing on the security report.
> 
> This has come up in a audit and hard to explain which is the System of
> Record information for security fixes.
> 
> https://tomcat.apache.org/security-8.html#Fixed_in_Apache_Tomcat_8.5.5_and_8.0.37
> 
> https://tomcat.apache.org/tomcat-8.0-doc/changelog.html

Because CVE-2016-5388 is not an Apache Tomcat vulnerability. The
changelog refers to the mitigation applied to Apache Tomcat to protect
users if they happen to be using vulnerable CGI executables.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: request and response body logging for async servlets

2020-08-12 Thread Mark Thomas
On 12/08/2020 17:50, Suraj Puvvada wrote:
> I'm trying to capture the request and response body for async servlets.
> Currently I'm using a filter to wrap the request and response via the
> HttpServletRequestWrapper and HttpServletResponseWrapper and wrap the
> InputStream and OutputStream objects and copy the data whenever read or
> write calls are invoked.
> 
> This works fine for regular synchronous servlet calls. However for async
> servlets I noticed that the wrapper classes are unwrapped and associated
> with the AsyncContext so I no longer am able to capture the data.
> 
> This is in Tomcat 8.x.
> 
> The API to start async processing used is request.startAsync().

ServletRequest.startAsync() uses the original, unwrapped request and
response. You need to use
ServletRequest.startAsync(ServletRequest,ServletResponse)

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: SSL debug?

2020-08-12 Thread Mark Thomas
On 12/08/2020 16:29, James H. H. Lampert wrote:
> Question:
> 
> We are once again having SSL difficulties with our webapp connecting
> with an outside web service: the java.security override that had solved
> the problem in the past (specifically, removing "DESede" from the
> "jdk.tls.disabledAlgorithms" in an override file) is now failing with
> certain Java 8 JVMs on AS/400s.
> 
> Somebody on the Midrange Java List suggested using
> 
>> -Djavax.net.debug=ssl:handshake
> 
> in catalina.sh, to gather more information on the problem.
> 
> Would that actually help with our present situation? Or would it only
> cover problems with users connecting to Tomcat from their browsers?

That should show TLS handshakes from both user agent to Tomcat and
application to external service.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Tomcat 8.5.(x > 5) & SSL Connections (sun.security.provider.certpath.SunCertPathBuilderException)

2020-08-10 Thread Mark Thomas
P address on which the connection was received to select
the virtual host. It will then send the cert relevant to that virtual host.

Given you had client issues as well, I suspect what you were seeing was
the result of the client sending a different host header and/or the
client connection via localhost vs the public IP or some combination of
the that. And if IPv6 is enabled then that adds another potential dimension.

Generally, I'd recommend Wireshark or similar in this type of scenario.
Especially if you configure things so it can decrypt the TLS, it can be
very helpful to spot differences between different setups.

Mark


> 
>> On Aug 9, 2020, at 3:19 PM, Mark Thomas  wrote:
>>
>> On August 8, 2020 6:59:23 PM UTC, David Filip  wrote:
>>> Hello Everyone!
>>>
>>> I spent a large part of yesterday and this morning trying to debug an
>>> SSL problem on Tomcat 8.5.57 to no avail.  I've seen some discussion on
>>> either this problem or something related back in 2016, but wanted to
>>> confirm what the "correct" solution might be, because I got lost in the
>>> threads.
>>>
>>> I never had this problem with Tomcat 7.0.x, but it started once I
>>> upgraded to 8.5.57 (same application code), and it is related to making
>>> outgoing SSL connections to web services.  And this is NOT related to a
>>> self-signed, but to a commercial (GoDaddy) SSL certificate, albeit on a
>>> server that I also run in the cloud.
>>>
>>> The exception is being thrown when trying to connect to an SSL
>>> protected web service is:
>>>
>>> sun.security.validator.ValidatorException: PKIX path building failed:
>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to
>>> find valid certification path to requested target
>>>
>>> although the exact same code worked (and still works on other servers)
>>> reliably under Tomcat 7.0.x for several years.
>>>
>>> Now, here is the weird part: after Google'ing around, I thought the
>>> problem might be that Tomcat 8.5.5 and later -- at least this is the
>>> gist that I got -- no longer finds the 'default' Java certificate store
>>> (cacerts), so I added the following to /bin/catalina.sh (running on a
>>> Mac 10.14 / Mojave):
>>>
>>> export
>>> -Djavax.net.ssl.trustStore=/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/security/cacerts
>>>
>>> The weird part is that this appeared to fix the problem, so I thought I
>>> was done.  Then, I rebooted, and the problem re-appeared!
>>>
>>> I stopped and started Tomcat, and the problem was resolved again.  I
>>> rebooted again, and the problem re-appeared.
>>>
>>> Previously, when it worked, I refreshed the page several times, and it
>>> kept working.  When it doesn't work, if I keep refreshing the page, it
>>> continues to throw the exception.
>>>
>>> Does this mean that some "worker threads" can find the certificate
>>> store, and others can't?  Or am I going down the wrong rabbit hole?
>>>
>>> So, any idea?  The intermittent nature is driving me crazy!
>>>
>>> And I have can reproduce the problem on two separate servers (both Mac
>>> 10.14 / Mojave, both Java 1.8.0), one (new server) running 8.5.57 and
>>> one (slightly older server) running 8.5.35.  But again, I have several
>>> 7.0.x instances where I've never seen this problem before.
>>>
>>> Also, the generic 'SSLPoke' always connects to the service, and it
>>> appears that if I run (mostly) the same code from the command line
>>> outside of Tomcat (javac / java) it always works.  And if I paste the
>>> web service URL into Safari or Chrome, it always works.  And if I use
>>> the web service URL with curl (just for good measure), it always works.
>>> So it only seems to fall under Tomcat 8.5.x.
>>>
>>> Thanks in advance for any guidance, as I'm running out of things to
>>> Google and try.
>>>
>>> Regards,
>>>
>>> Dave.
>>
>> Tomcat has zero involvement in outgoing TLS connections.
>>
>> If the code works in a standalone Java app, it will work in a Servlet 
>> assuming the code is thread safe (I don't see why it wouldn't be but worth 
>> double checking any library being used) and configuration information is 
>> accessible.
>>
>> Mark
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
> 
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Tomcat 8.5.(x > 5) & SSL Connections (sun.security.provider.certpath.SunCertPathBuilderException)

2020-08-09 Thread Mark Thomas
On August 8, 2020 6:59:23 PM UTC, David Filip  wrote:
>Hello Everyone!
>
>I spent a large part of yesterday and this morning trying to debug an
>SSL problem on Tomcat 8.5.57 to no avail.  I've seen some discussion on
>either this problem or something related back in 2016, but wanted to
>confirm what the "correct" solution might be, because I got lost in the
>threads.
>
>I never had this problem with Tomcat 7.0.x, but it started once I
>upgraded to 8.5.57 (same application code), and it is related to making
>outgoing SSL connections to web services.  And this is NOT related to a
>self-signed, but to a commercial (GoDaddy) SSL certificate, albeit on a
>server that I also run in the cloud.
>
>The exception is being thrown when trying to connect to an SSL
>protected web service is:
>
>sun.security.validator.ValidatorException: PKIX path building failed:
>sun.security.provider.certpath.SunCertPathBuilderException: unable to
>find valid certification path to requested target
>
>although the exact same code worked (and still works on other servers)
>reliably under Tomcat 7.0.x for several years.
>
>Now, here is the weird part: after Google'ing around, I thought the
>problem might be that Tomcat 8.5.5 and later -- at least this is the
>gist that I got -- no longer finds the 'default' Java certificate store
>(cacerts), so I added the following to /bin/catalina.sh (running on a
>Mac 10.14 / Mojave):
>
>export
>-Djavax.net.ssl.trustStore=/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/security/cacerts
>
>The weird part is that this appeared to fix the problem, so I thought I
>was done.  Then, I rebooted, and the problem re-appeared!
>
>I stopped and started Tomcat, and the problem was resolved again.  I
>rebooted again, and the problem re-appeared.
>
>Previously, when it worked, I refreshed the page several times, and it
>kept working.  When it doesn't work, if I keep refreshing the page, it
>continues to throw the exception.
>
>Does this mean that some "worker threads" can find the certificate
>store, and others can't?  Or am I going down the wrong rabbit hole?
>
>So, any idea?  The intermittent nature is driving me crazy!
>
>And I have can reproduce the problem on two separate servers (both Mac
>10.14 / Mojave, both Java 1.8.0), one (new server) running 8.5.57 and
>one (slightly older server) running 8.5.35.  But again, I have several
>7.0.x instances where I've never seen this problem before.
>
>Also, the generic 'SSLPoke' always connects to the service, and it
>appears that if I run (mostly) the same code from the command line
>outside of Tomcat (javac / java) it always works.  And if I paste the
>web service URL into Safari or Chrome, it always works.  And if I use
>the web service URL with curl (just for good measure), it always works.
> So it only seems to fall under Tomcat 8.5.x.
>
>Thanks in advance for any guidance, as I'm running out of things to
>Google and try.
>
>Regards,
>
>Dave.

Tomcat has zero involvement in outgoing TLS connections.

If the code works in a standalone Java app, it will work in a Servlet assuming 
the code is thread safe (I don't see why it wouldn't be but worth double 
checking any library being used) and configuration information is accessible.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



  1   2   3   4   5   6   7   8   9   10   >