Re: Tomcat 9.0.36 - JDK 13/14

[Kiran Badi]

we fixed the issue Mark.

Actually tomcat was running on JDK 1.8 and applications were built using
JDK 13/14.So when they were deployed to tomcat running with 1.8, they were
giving 404.

Now plan is to explore and upgrade tomcat to at least jdk 13.

It would have been nice really to have at least some error saying major.minor
version exception or something like that in some logs somewhere which java
often throws in these cases.

Any ways we are good to go now. Thanks for your reply.



On Fri, Jun 26, 2020 at 6:34 AM Mark Thomas  wrote:

> On 26/06/2020 05:45, Kiran Badi wrote:
> > Hi All,
> >
> > I wanted to check if tomcat 9.0.36 supports open jdk 13/14.
>
> Supported Java versions are listed at:
> http://tomcat.apache.org/whichversion.html
>
> "Java 8 and later" includes Java 13 and Java 14.
>
> > I created a simple spring boot war file and compiled/built it with
> openjdk
> > 13/14. After running maven install , I deployed the war file from the
> > target directory to tomcat webapps using tomcat manager. It did not work
> > and gave me 404 messages with both 13/14. No error or any exception
> > anywhere in logs.Catalina log just says a war file is deployed.
>
> That is normally indicative of a configuration error.
>
> > Then i compiled the same spring boot app with jdk 8 and deployed it with
> > tomcat and it works fine. I am able to call my endpoints with no issues.
> >
> > I am having a hard time deploying angular/spring boot and building war
> file
> > and deploying it on tomcat 9.0x with openjdk 13. So I thought this might
> be
> > a good place to start with.
>
> If you can provide the code you use to create the sample, e.g. as a
> GitHub project somebody may be able to take a look.
>
> Mark
>
> >
> > I used the below pom file.
> >
> > 
> > http://maven.apache.org/POM/4.0.0; xmlns:xsi="
> > http://www.w3.org/2001/XMLSchema-instance;
> > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> > https://maven.apache.org/xsd/maven-4.0.0.xsd;>
> > 4.0.0
> > 
> > org.springframework.boot
> > spring-boot-starter-parent
> > 2.3.1.RELEASE
> >  
> > 
> > com.kiran
> > springwar
> > 1.0.2-SNAPSHOT
> > war
> > springwar
> > Sample project to deploy war to tomcat
> >
> > 
> > 14
> > 
> >
> > 
> > 
> > org.springframework.boot
> > spring-boot-starter-web
> > 
> >
> > 
> > org.springframework.boot
> > spring-boot-starter-tomcat
> > provided
> > 
> > 
> > org.springframework.boot
> > spring-boot-starter-test
> > test
> > 
> > 
> > org.junit.vintage
> > junit-vintage-engine
> > 
> > 
> > 
> > 
> >
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: jsvc - non root - log as root

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Mark and Jürgen,

On 6/26/20 06:23, Mark Thomas wrote:
> On 26/06/2020 08:21, Jürgen Weber wrote:
>> Hi,
>>
>> when you run tomcat with jsvc and have jsvc drop privileges to a
>> different user, stdout and stderr log files are still created
>> with root as owner. Can you make jsvc create them as the -user ?
>
> I'm no C expert but my reading of
> https://github.com/apache/commons-daemon/blob/master/src/native/unix/n
ative/jsvc-unix.c#L1039
>
>
is no.

To be fair, jsvc *could* (be made to) do this, but that is not what
the current code looks like. Since the euig of the process when the
files are created is root (or elevated in some way), the ownership and
permissions of the file should be able to be set at that time before
privileges are dropped.

If these lines were to be added after 1071 (for stdout):

  if(chown(outfile, uid, gid)) {
perror("chown");
exit(1);
  }

Than the file could be owned by the unprivileged user/group. The uid
and gid are not currently available in the set_output function.

Hmm. If doreopen is true, then when trying to reopen the log files
(after dropping privileges), I think we'll get ENOACCESS. I don't use
jsvc so I haven't played around with it at all. I might be completely
wrong :)

- -chris
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl72U0oACgkQHPApP6U8
pFhyPg//TXOriocgBl0cQW9a7TV7lW9K4FF3D/IIhvxcaed4H/Ugb8UYtJ7uKfLh
6xqko+pSkirUlfTBErSJ/Rnc9Mk2m43gL0onPOKOP1tKFq5VZG+Bu5IcHRKbVBcA
+pE2R7owUM3m8dnqeyvnqbAE2GH8AkxVj+A2ye1V+3R7b9iQd9C814rvucFN2p+s
UaQdlSXMPs5xqTbNQO1tHp0Tz91zyarc/MfViqBuAHfYuYCBOZevSwvlaLJ5I30Q
ef7htZM8eEUknGTbndQj/Rt/63xYcclKWT7cqBtRcMVqTiZfF90Q7ApBET7SuqjQ
nupUW9TdtocE7dLOIX48MH1VcC3xtVOYQXrCfDezV2Sk/INxr6Mubb4W8jdejGNI
Anc48N3FbJ58zzdYHonM976dfG2vFlolmITntb3k1YG6YrtL/pL9XEyXuFZuPaAm
Os4sB+7nhTw0ckVL3ZASvLSFg4JQmMObOmXdxxLk3VlUS1ZJgmPxb6HEh8SEXxgd
UeRw6C0ptkQfFBTqHLCT3ZFJnJGeBYlhLd6/K40o6OjDCJsce2W72xpNsiseO27L
fE/KI80/Jy+rtZNcFqJjWeVxmGdSvJuCWDEqwzvHeirexK/GDyaNaG7tNeCL93Nj
S6uG3ML1XKJjb0aNPWtR45DXkr1HU52qlPW4XqczdSNHWvRRy44=
=iOl+
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: SSL error [EXTERNAL]

[Beard, Shawn M.]

I was able to resolve this. I used keytool to create a new keystore/trust 
store, then imported the previous truststore that had all the CA certs in it. 
That seemed to work. So even though the previous truststore had the certs in it 
and was not empty, it must have had some kind of linking problem maybe?



Shawn Beard
Sr. Systems Engineer
BTS
+1-515-564-2528

-Original Message-
From: john.e.gr...@wellsfargo.com.INVALID 
Sent: Friday, June 26, 2020 1:32 PM
To: users@tomcat.apache.org
Subject: RE: SSL error [EXTERNAL]

** CAUTION: External message


Shawn,


-Original Message-
From: Beard, Shawn M. 
Sent: Friday, June 26, 2020 11:57 AM
To: Tomcat Users List 
Subject: RE: SSL error [EXTERNAL]

The code is calling a new webservice. It has godaddy as its ca signer. It was 
getting the error before I added those java options. Those java options were my 
attempt to resolve it. Ive also tried adding the godaddy ca certs to java's 
cacert file without those java options. Same result.



Shawn Beard
Sr. Systems Engineer
BTS
+1-515-564-2528

-Original Message-
From: calder 
Sent: Friday, June 26, 2020 11:45 AM
To: Tomcat Users List 
Subject: Re: SSL error [EXTERNAL]

** CAUTION: External message


In Fri, Jun 26, 2020, 10:37 Beard, Shawn M. 
wrote:

> We are running tomcat-7.0.52(old I know) and java 1.7.0_80.
>

yea, BOTH are very old.

When the app makes calls to an external webservice. It keeps throwing this
> error:
>
> javax.net.ssl.SSLException : javax.net.ssl.SSLException:
> java.lang.RuntimeException: Unexpected error:
> java.security.InvalidAlgorithmParameterException: the trustAnchors
> parameter must be non-empty
>
[1]

> I have this in the java options and have confirmed the proper CA certs
> for this webservice is in the truststore. Any ideas?
>
-Djavax.net.ssl.trustStore=/path/to/truststore/tomcatTrustStore.jks
> -Djavax.net.ssl.trustStorePassword=
> -Djavax.net.ssl.trustStoreType=jks
>

Did this runtime EVER work?

If yes, "what" changed?



[1]
https://urldefense.com/v3/__https://stackoverflow.com/questions/6784463/error-trustanchors-parameter-must-be-non-empty__;!!Li8W9_Um1Taa!uk48yx6ZQNHjmcqPmjBlJDFCcCWu6HMZu3OI_Yau1oJ4CBGoaFzI0pfKTaIrqOGk$
CONFIDENTIALITY NOTICE: This e-mail and the transmitted documents contain 
private, privileged and confidential information belonging to the sender. The 
information therein is solely for the use of the addressee. If your receipt of 
this transmission has occurred as the result of an error, please immediately 
notify us so we can arrange for the return of the documents. In such 
circumstances, you are advised that you may not disclose, copy, distribute or 
take any other action in reliance on the information transmitted.
B CB  [  
X  ܚX KK[XZ[  \ \  ][  X  ܚX P X ]  \X K ܙ B  ܈Y][ۘ[  [X[  
K[XZ[  \ \  Z[ X ]  \X K ܙ B

That error message comes from PKIXParameters.setTrustAnchors().  I was able to 
reproduce the problem with an empty trust store.  I also tried a trust store 
with the wrong certs but got a different error.

With -Djavax.net.debug=ssl, you should see output like this:

trustStore is: /path/to/trust.jks
trustStore type is: jks
trustStore provider is:
the last modified time is: Fri Jun 26 13:27:52 CDT 2020 Reload the trust store 
Reload trust certs Reloaded 1 trust certs adding as trusted cert:

Followed by a list of certs found in the store.

Is that what's happening in your case?

John

Т ХF  V 
7V'67&  R   â W6W'2 V 7V'67&  F  6B 6 R  Фf "FF F    6    G2 
R   â W6W'2ֆV  F  6B 6 R  Р

RE: SSL error [EXTERNAL]

[John.E.Gregg]

Shawn,


-Original Message-
From: Beard, Shawn M.  
Sent: Friday, June 26, 2020 11:57 AM
To: Tomcat Users List 
Subject: RE: SSL error [EXTERNAL]

The code is calling a new webservice. It has godaddy as its ca signer. It was 
getting the error before I added those java options. Those java options were my 
attempt to resolve it. Ive also tried adding the godaddy ca certs to java's 
cacert file without those java options. Same result.



Shawn Beard
Sr. Systems Engineer
BTS
+1-515-564-2528

-Original Message-
From: calder 
Sent: Friday, June 26, 2020 11:45 AM
To: Tomcat Users List 
Subject: Re: SSL error [EXTERNAL]

** CAUTION: External message


In Fri, Jun 26, 2020, 10:37 Beard, Shawn M. 
wrote:

> We are running tomcat-7.0.52(old I know) and java 1.7.0_80.
>

yea, BOTH are very old.

When the app makes calls to an external webservice. It keeps throwing this
> error:
>
> javax.net.ssl.SSLException : javax.net.ssl.SSLException:
> java.lang.RuntimeException: Unexpected error:
> java.security.InvalidAlgorithmParameterException: the trustAnchors 
> parameter must be non-empty
>
[1]

> I have this in the java options and have confirmed the proper CA certs 
> for this webservice is in the truststore. Any ideas?
>
-Djavax.net.ssl.trustStore=/path/to/truststore/tomcatTrustStore.jks
> -Djavax.net.ssl.trustStorePassword=
> -Djavax.net.ssl.trustStoreType=jks
>

Did this runtime EVER work?

If yes, "what" changed?



[1]
https://urldefense.com/v3/__https://stackoverflow.com/questions/6784463/error-trustanchors-parameter-must-be-non-empty__;!!Li8W9_Um1Taa!uk48yx6ZQNHjmcqPmjBlJDFCcCWu6HMZu3OI_Yau1oJ4CBGoaFzI0pfKTaIrqOGk$
CONFIDENTIALITY NOTICE: This e-mail and the transmitted documents contain 
private, privileged and confidential information belonging to the sender. The 
information therein is solely for the use of the addressee. If your receipt of 
this transmission has occurred as the result of an error, please immediately 
notify us so we can arrange for the return of the documents. In such 
circumstances, you are advised that you may not disclose, copy, distribute or 
take any other action in reliance on the information transmitted.
B CB  [  
X  ܚX KK[XZ[
 \ \  ][  X  ܚX P X ]
 \X K ܙ B  ܈Y][ۘ[  [X[  K[XZ[
 \ \  Z[ X ]
 \X K ܙ B 

That error message comes from PKIXParameters.setTrustAnchors().  I was able to 
reproduce the problem with an empty trust store.  I also tried a trust store 
with the wrong certs but got a different error.

With -Djavax.net.debug=ssl, you should see output like this:

trustStore is: /path/to/trust.jks
trustStore type is: jks
trustStore provider is: 
the last modified time is: Fri Jun 26 13:27:52 CDT 2020
Reload the trust store
Reload trust certs
Reloaded 1 trust certs
adding as trusted cert:

Followed by a list of certs found in the store.

Is that what's happening in your case?

John

RE: SSL error [EXTERNAL]

[Beard, Shawn M.]

The code is calling a new webservice. It has godaddy as its ca signer. It was 
getting the error before I added those java options. Those java options were my 
attempt to resolve it. Ive also tried adding the godaddy ca certs to java's 
cacert file without those java options. Same result.



Shawn Beard
Sr. Systems Engineer
BTS
+1-515-564-2528

-Original Message-
From: calder 
Sent: Friday, June 26, 2020 11:45 AM
To: Tomcat Users List 
Subject: Re: SSL error [EXTERNAL]

** CAUTION: External message


In Fri, Jun 26, 2020, 10:37 Beard, Shawn M. 
wrote:

> We are running tomcat-7.0.52(old I know) and java 1.7.0_80.
>

yea, BOTH are very old.

When the app makes calls to an external webservice. It keeps throwing this
> error:
>
> javax.net.ssl.SSLException : javax.net.ssl.SSLException:
> java.lang.RuntimeException: Unexpected error:
> java.security.InvalidAlgorithmParameterException: the trustAnchors
> parameter must be non-empty
>
[1]

> I have this in the java options and have confirmed the proper CA certs
> for this webservice is in the truststore. Any ideas?
>
-Djavax.net.ssl.trustStore=/path/to/truststore/tomcatTrustStore.jks
> -Djavax.net.ssl.trustStorePassword=
> -Djavax.net.ssl.trustStoreType=jks
>

Did this runtime EVER work?

If yes, "what" changed?



[1]
https://urldefense.com/v3/__https://stackoverflow.com/questions/6784463/error-trustanchors-parameter-must-be-non-empty__;!!Li8W9_Um1Taa!uk48yx6ZQNHjmcqPmjBlJDFCcCWu6HMZu3OI_Yau1oJ4CBGoaFzI0pfKTaIrqOGk$
CONFIDENTIALITY NOTICE: This e-mail and the transmitted documents contain 
private, privileged and confidential information belonging to the sender. The 
information therein is solely for the use of the addressee. If your receipt of 
this transmission has occurred as the result of an error, please immediately 
notify us so we can arrange for the return of the documents. In such 
circumstances, you are advised that you may not disclose, copy, distribute or 
take any other action in reliance on the information transmitted.

Re: SSL error

[calder]

In Fri, Jun 26, 2020, 10:37 Beard, Shawn M. 
wrote:

> We are running tomcat-7.0.52(old I know) and java 1.7.0_80.
>

yea, BOTH are very old.

When the app makes calls to an external webservice. It keeps throwing this
> error:
>
> javax.net.ssl.SSLException : javax.net.ssl.SSLException:
> java.lang.RuntimeException: Unexpected error:
> java.security.InvalidAlgorithmParameterException: the trustAnchors
> parameter must be non-empty
>
[1]

> I have this in the java options and have confirmed the proper CA certs for
> this webservice is in the truststore. Any ideas?
>
-Djavax.net.ssl.trustStore=/path/to/truststore/tomcatTrustStore.jks
> -Djavax.net.ssl.trustStorePassword=
> -Djavax.net.ssl.trustStoreType=jks
>

Did this runtime EVER work?

If yes, "what" changed?



[1]
https://stackoverflow.com/questions/6784463/error-trustanchors-parameter-must-be-non-empty

SSL error

[Beard, Shawn M.]

We are running tomcat-7.0.52(old I know) and java 1.7.0_80.  When the app makes 
calls to an external webservice. It keeps throwing this error:

javax.net.ssl.SSLException : javax.net.ssl.SSLException: 
java.lang.RuntimeException: Unexpected error: 
java.security.InvalidAlgorithmParameterException: the trustAnchors parameter 
must be non-empty

I have this in the java options and have confirmed the proper CA certs for this 
webservice is in the truststore. Any ideas?

-Djavax.net.ssl.trustStore=/path/to/truststore/tomcatTrustStore.jks 
-Djavax.net.ssl.trustStorePassword= -Djavax.net.ssl.trustStoreType=jks





Shawn Beard • Sr. Systems Engineer
Middleware Engineering

[cid:image75dd5a.PNG@4f1b1b38.44a2aecd]


 3840 109th Street Urbandale, IA 50322
 Phone: +1-515-564-2528
 Email: sbe...@wrberkley.com
 Website: 
berkleytechnologyservices.com

Technology Leadership Unleashing Business Potential



CONFIDENTIALITY NOTICE: This e-mail and the transmitted documents contain 
private, privileged and confidential information belonging to the sender. The 
information therein is solely for the use of the addressee. If your receipt of 
this transmission has occurred as the result of an error, please immediately 
notify us so we can arrange for the return of the documents. In such 
circumstances, you are advised that you may not disclose, copy, distribute or 
take any other action in reliance on the information transmitted.

RE: Question around catalina.policy change back with 9.0.33, etc.

[jonmcalexander]

Thank you so much Mark!!!


Dream * Excel * Explore * Inspire
Jon McAlexander
Asst Vice President

Middleware Product Engineering
Enterprise CIO | Platform Services | Middleware | Infrastructure Solutions

8080 Cobblestone Rd | Urbandale, IA 50322
MAC: F4469-010
Tel 515-988-2508 | Cell 515-988-2508

jonmcalexan...@wellsfargo.com


This message may contain confidential and/or privileged information. If you are 
not the addressee or authorized to receive this for the addressee, you must not 
use, copy, disclose, or take any action based on this message or any 
information herein. If you have received this message in error, please advise 
the sender immediately by reply e-mail and delete this message. Thank you for 
your cooperation.


-Original Message-
From: Mark Thomas  
Sent: Friday, June 26, 2020 5:14 AM
To: users@tomcat.apache.org
Subject: Re: Question around catalina.policy change back with 9.0.33, etc.

On 26/06/2020 00:15, jonmcalexan...@wellsfargo.com.INVALID wrote:
> I have a developer that is asking WHY the following policies were set to read 
> only. The Change Log doesn't illuminate why.
> 
> // The cookie code needs these.
> permission java.util.PropertyPermission
>  "org.apache.catalina.STRICT_SERVLET_COMPLIANCE", "read";
> permission java.util.PropertyPermission
>  "org.apache.tomcat.util.http.ServerCookie.STRICT_NAMING", "read";
> permission java.util.PropertyPermission
>  
> "org.apache.tomcat.util.http.ServerCookie.FWD_SLASH_IS_SEPARATOR", 
> "read";
> 
> Any information I can share with her?

Those permissions were removed, not set to read only, for 9.0.9 onwards.

It was the result of a refactoring:
https://github.com/apache/tomcat/commit/6ceb931e1aac0355e0980d09814559f24406a14a

I made the change but it was 2 years ago. I don't recall the motivation 
off-hand. /me heads off to look at the archives...

..and that is why we have the archives.

It appears to stem from this issue:
https://bz.apache.org/bugzilla/show_bug.cgi?id=43925

The fix for that issue led to this:
https://tomcat.markmail.org/thread/mab6jbyb57phslwk

Rather than add a permission, the code was refactored so the additional 
permission (and some of the existing permissions) were no longer required.

It isn't documented, but I strongly suspect that got me looking at other 
permissions which led to the refactoring that allowed the removal of the cookie 
permissions.

The general principle behind all of this being the fewer explicit permissions 
you need to give to applications the better.

HTH,

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: How to encrypt db password in tomcat context.xml

[Olaf Kock]

On 26.06.20 15:05, FANG YAP wrote:
> Hi Tomcat,
>
> I would like to know how to encrypt and decrypt the database password in
> context.xml when the application is running which also allow me to change
> the db password for the purpose of security.


https://cwiki.apache.org/confluence/display/TOMCAT/Password




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

How to encrypt db password in tomcat context.xml

[FANG YAP]

Hi Tomcat,

I would like to know how to encrypt and decrypt the database password in
context.xml when the application is running which also allow me to change
the db password for the purpose of security.

Database driver is Oracle
JDK: 1.8.0_251
Tomcat Version: 8.5.55


Appreciate ya help.

Rgs,
FanggDev.

Re: Connection Closure due to Fatal Stream with HTTP2

[Mark Thomas]

On 26/06/2020 12:48, Mark Thomas wrote:
> On 26/06/2020 12:45, Chirag Dewan wrote:
>> Absolutely Mark. Shouldn't take long.
> 
> Great. I think I have found a potential root cause. If I am right, NIO
> will show the same issues NIO2 did.
> 
> I should have a test build for you shortly.

Try this:
https://people.apache.org/~markt/dev/v9.0.37-dev/

Please note:

This is NOT an official Apache Release.

This build is only intended to be used to test this issue.

Using the test build is it your own risk.

Thanks,

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: CVE-2020-11996 Apache Tomcat HTTP/2 Denial of Service

[Mark Thomas]

On 26/06/2020 13:35, Kasteleijn, Wilco wrote:
> Hello, we would like to know if this vulnerability is only applicable for 
> usage of the coyote http connector?

It only applies when using the HTTP/2 protocol. That is only available
with an HTTP connector.

> We are using Tomcat 8.5.55 in combination with a apache HTTPD proxy setup 
> that is connected via the AJP connector. Are we also affected in that case?

No. AJP is not affected.

Mark


> Regards, Wilco.
> 
> 
> This message contains confidential or privileged information and is intended 
> only for the individual named. If you have received it by mistake, please let 
> us know by e-mail reply and delete it from your system; you may not copy this 
> message or disclose its contents to anyone. Please note that any views or 
> opinions presented in this email are solely those of the author and do not 
> necessarily represent those of the company. Nothing in this email is intended 
> to bind Elemica, Inc., which only operates under the terms of written 
> agreements signed by an authorized officer. E-mail transmission cannot be 
> guaranteed to be secure or error-free as information could be intercepted, 
> corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. 
> The sender therefore does not accept liability for any errors or omissions in 
> the contents of this message, which arise as a result of e-mail transmission.
> 
> Disclaimer
> 
> The information contained in this communication from the sender is 
> confidential. It is intended solely for use by the recipient and others 
> authorized to receive it. If you are not the recipient, you are hereby 
> notified that any disclosure, copying, distribution or taking action in 
> relation of the contents of this information is strictly prohibited and may 
> be unlawful.
> 
> This email has been scanned for viruses and malware, and may have been 
> automatically archived by Mimecast Ltd, an innovator in Software as a Service 
> (SaaS) for business. Providing a safer and more useful place for your human 
> generated data. Specializing in; Security, archiving and compliance. To find 
> out more visit the Mimecast website.
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

CVE-2020-11996 Apache Tomcat HTTP/2 Denial of Service

[Kasteleijn, Wilco]

Hello, we would like to know if this vulnerability is only applicable for usage 
of the coyote http connector?
We are using Tomcat 8.5.55 in combination with a apache HTTPD proxy setup that 
is connected via the AJP connector. Are we also affected in that case?
Regards, Wilco.


This message contains confidential or privileged information and is intended 
only for the individual named. If you have received it by mistake, please let 
us know by e-mail reply and delete it from your system; you may not copy this 
message or disclose its contents to anyone. Please note that any views or 
opinions presented in this email are solely those of the author and do not 
necessarily represent those of the company. Nothing in this email is intended 
to bind Elemica, Inc., which only operates under the terms of written 
agreements signed by an authorized officer. E-mail transmission cannot be 
guaranteed to be secure or error-free as information could be intercepted, 
corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The 
sender therefore does not accept liability for any errors or omissions in the 
contents of this message, which arise as a result of e-mail transmission.

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: Connection Closure due to Fatal Stream with HTTP2

[Mark Thomas]

On 26/06/2020 12:45, Chirag Dewan wrote:
> Absolutely Mark. Shouldn't take long.

Great. I think I have found a potential root cause. If I am right, NIO
will show the same issues NIO2 did.

I should have a test build for you shortly.

Mark


> 
> On Fri, 26 Jun, 2020, 4:16 pm Mark Thomas,  wrote:
> 
>> Aha!
>>
>> h2c could be the significant factor here. Let me take a look.
>>
>> Are you in a position to test against a dev build if the need arises?
>>
>> Mark
>>
>>
>> On 26/06/2020 11:30, Chirag Dewan wrote:
>>> Hey Mark,
>>>
>>> *Are you using h2c or h2 in your test?*
>>> We are using h2c
>>>
>>>
>>> *Do you see the same issue if you switch the the NIO connector? Note
>>> performance differences between NIO and NIO2 are very small.*
>>>
>>> I havent tried with NIO honestly. Can quickly execute and check. Will
>>> update the results.
>>>
>>> *How long does a single request take to process?*
>>> In normal scenarios, less than 3ms.
>>>
>>> Thanks,
>>> Chirag
>>>
>>> On Fri, Jun 26, 2020 at 3:26 PM Mark Thomas  wrote:
>>>
 Hi,

 Thanks for the additional information. The GC roots were particularly
 informative.

 Those RequestInfo objects are associated with HTTP/1.1 requests, not
 HTTP/2 requests.

 Some further questions to try and track down what is going on:

 - Are you using h2c or h2 in your test?

 - Do you see the same issue if you switch the the NIO connector? Note
   performance differences between NIO and NIO2 are very small.

 - How long does a single request take to process?

 Thanks,

 Mark

 On 26/06/2020 09:24, Chirag Dewan wrote:
> Thanks Mark.
>
> *What is the typical response size for one of these requests? *
> It' basically a dummy response JSON of ~300bytes. I expect 2300bytes of
> response in my production use case, but the purpose here was to isolate
 as
> many things as possible. Hence a dummy response.
>
> *How long does a typical test take to process? *
> I see Tomcat's memory reaching 28G in about 2hours at 19K TPS. The
>> number
> of streams on my client was 500 - which means about 40 connections per
> second.
>
> * What are the GC roots for those RequestInfo objects?*
> https://ibb.co/fMRmCXZ
>
> I hope I was able to answer everything as expected. Thanks.
>
> On Thu, Jun 25, 2020 at 8:30 PM Mark Thomas  wrote:
>
>> Thanks.
>>
>> I've looked at the code and I have tried various tests but I am unable
>> to re-create a memory leak.
>>
>> The code used to (before I made a few changes this afternoon) retain a
>> lot more memory per Stream and it is possible that what you are seeing
>> is a system that doesn't have enough memory to achieve steady state.
>>
>> If you are able to build the latest 9.0.x and test that, that could be
>> helpful. Alternatively, I could provide a test build for you to
>> experiment with.
>>
>> Some additional questions that might aid understanding:
>>
>> - What is the typical response size for one of these requests?
>> - How long does a typical test take to process?
>> - What are the GC roots for those RequestInfo objects?
>>
>> Thanks again,
>>
>> Mark
>>
>>
>>
>>
>> On 25/06/2020 15:10, Chirag Dewan wrote:
>>> Hi Mark,
>>>
>>> Its the default APR connector with 150 Threads.
>>>
>>> Chirag
>>>
>>> On Thu, 25 Jun, 2020, 7:30 pm Mark Thomas,  wrote:
>>>
 On 25/06/2020 11:00, Chirag Dewan wrote:
> Thanks for the quick check Mark.
>
> These are the images I tried referring to:
>
> https://ibb.co/LzKtRgh
>
> https://ibb.co/2s7hqRL
>
> https://ibb.co/KmKj590
>
>
> The last one is the MAT screenshot showing many RequestInfo
>> objects.

 Thanks. That certainly looks like a memory leak. I'll take a closer
 look. Out of interest, how many threads is the Connector configured
>> to
>> use?

 Mark


>
>
> Thanks,
>
> Chirag
>
> On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas 
 wrote:
>
>> On 24/06/2020 12:17, Mark Thomas wrote:
>>> On 22/06/2020 11:06, Chirag Dewan wrote:
 Hi,

 Update: We found that Tomcat goes OOM when a client closes and
 opens
 new
 connections every second. In the memory dump, we see a lot of
 RequestInfo objects that are causing the memory spike.

 After a while, Tomcat goes OOM and start rejecting request(I
>> get a
 request timed out on my client). This seems like a bug to me.

 For better understanding, let me explain my use case again:

 I have a 

Re: Connection Closure due to Fatal Stream with HTTP2

[Chirag Dewan]

Absolutely Mark. Shouldn't take long.

On Fri, 26 Jun, 2020, 4:16 pm Mark Thomas,  wrote:

> Aha!
>
> h2c could be the significant factor here. Let me take a look.
>
> Are you in a position to test against a dev build if the need arises?
>
> Mark
>
>
> On 26/06/2020 11:30, Chirag Dewan wrote:
> > Hey Mark,
> >
> > *Are you using h2c or h2 in your test?*
> > We are using h2c
> >
> >
> > *Do you see the same issue if you switch the the NIO connector? Note
> > performance differences between NIO and NIO2 are very small.*
> >
> > I havent tried with NIO honestly. Can quickly execute and check. Will
> > update the results.
> >
> > *How long does a single request take to process?*
> > In normal scenarios, less than 3ms.
> >
> > Thanks,
> > Chirag
> >
> > On Fri, Jun 26, 2020 at 3:26 PM Mark Thomas  wrote:
> >
> >> Hi,
> >>
> >> Thanks for the additional information. The GC roots were particularly
> >> informative.
> >>
> >> Those RequestInfo objects are associated with HTTP/1.1 requests, not
> >> HTTP/2 requests.
> >>
> >> Some further questions to try and track down what is going on:
> >>
> >> - Are you using h2c or h2 in your test?
> >>
> >> - Do you see the same issue if you switch the the NIO connector? Note
> >>   performance differences between NIO and NIO2 are very small.
> >>
> >> - How long does a single request take to process?
> >>
> >> Thanks,
> >>
> >> Mark
> >>
> >> On 26/06/2020 09:24, Chirag Dewan wrote:
> >>> Thanks Mark.
> >>>
> >>> *What is the typical response size for one of these requests? *
> >>> It' basically a dummy response JSON of ~300bytes. I expect 2300bytes of
> >>> response in my production use case, but the purpose here was to isolate
> >> as
> >>> many things as possible. Hence a dummy response.
> >>>
> >>> *How long does a typical test take to process? *
> >>> I see Tomcat's memory reaching 28G in about 2hours at 19K TPS. The
> number
> >>> of streams on my client was 500 - which means about 40 connections per
> >>> second.
> >>>
> >>> * What are the GC roots for those RequestInfo objects?*
> >>> https://ibb.co/fMRmCXZ
> >>>
> >>> I hope I was able to answer everything as expected. Thanks.
> >>>
> >>> On Thu, Jun 25, 2020 at 8:30 PM Mark Thomas  wrote:
> >>>
>  Thanks.
> 
>  I've looked at the code and I have tried various tests but I am unable
>  to re-create a memory leak.
> 
>  The code used to (before I made a few changes this afternoon) retain a
>  lot more memory per Stream and it is possible that what you are seeing
>  is a system that doesn't have enough memory to achieve steady state.
> 
>  If you are able to build the latest 9.0.x and test that, that could be
>  helpful. Alternatively, I could provide a test build for you to
>  experiment with.
> 
>  Some additional questions that might aid understanding:
> 
>  - What is the typical response size for one of these requests?
>  - How long does a typical test take to process?
>  - What are the GC roots for those RequestInfo objects?
> 
>  Thanks again,
> 
>  Mark
> 
> 
> 
> 
>  On 25/06/2020 15:10, Chirag Dewan wrote:
> > Hi Mark,
> >
> > Its the default APR connector with 150 Threads.
> >
> > Chirag
> >
> > On Thu, 25 Jun, 2020, 7:30 pm Mark Thomas,  wrote:
> >
> >> On 25/06/2020 11:00, Chirag Dewan wrote:
> >>> Thanks for the quick check Mark.
> >>>
> >>> These are the images I tried referring to:
> >>>
> >>> https://ibb.co/LzKtRgh
> >>>
> >>> https://ibb.co/2s7hqRL
> >>>
> >>> https://ibb.co/KmKj590
> >>>
> >>>
> >>> The last one is the MAT screenshot showing many RequestInfo
> objects.
> >>
> >> Thanks. That certainly looks like a memory leak. I'll take a closer
> >> look. Out of interest, how many threads is the Connector configured
> to
>  use?
> >>
> >> Mark
> >>
> >>
> >>>
> >>>
> >>> Thanks,
> >>>
> >>> Chirag
> >>>
> >>> On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas 
> >> wrote:
> >>>
>  On 24/06/2020 12:17, Mark Thomas wrote:
> > On 22/06/2020 11:06, Chirag Dewan wrote:
> >> Hi,
> >>
> >> Update: We found that Tomcat goes OOM when a client closes and
> >> opens
> >> new
> >> connections every second. In the memory dump, we see a lot of
> >> RequestInfo objects that are causing the memory spike.
> >>
> >> After a while, Tomcat goes OOM and start rejecting request(I
> get a
> >> request timed out on my client). This seems like a bug to me.
> >>
> >> For better understanding, let me explain my use case again:
> >>
> >> I have a jetty client that sends HTTP2 requests to Tomcat. My
> >> requirement is to close a connection after a configurable(say
> >> 5000)
> >> number of requests/streams and open a new connection that
> >> continues
> 

Re: Connection Closure due to Fatal Stream with HTTP2

[Mark Thomas]

Aha!

h2c could be the significant factor here. Let me take a look.

Are you in a position to test against a dev build if the need arises?

Mark


On 26/06/2020 11:30, Chirag Dewan wrote:
> Hey Mark,
> 
> *Are you using h2c or h2 in your test?*
> We are using h2c
> 
> 
> *Do you see the same issue if you switch the the NIO connector? Note
> performance differences between NIO and NIO2 are very small.*
> 
> I havent tried with NIO honestly. Can quickly execute and check. Will
> update the results.
> 
> *How long does a single request take to process?*
> In normal scenarios, less than 3ms.
> 
> Thanks,
> Chirag
> 
> On Fri, Jun 26, 2020 at 3:26 PM Mark Thomas  wrote:
> 
>> Hi,
>>
>> Thanks for the additional information. The GC roots were particularly
>> informative.
>>
>> Those RequestInfo objects are associated with HTTP/1.1 requests, not
>> HTTP/2 requests.
>>
>> Some further questions to try and track down what is going on:
>>
>> - Are you using h2c or h2 in your test?
>>
>> - Do you see the same issue if you switch the the NIO connector? Note
>>   performance differences between NIO and NIO2 are very small.
>>
>> - How long does a single request take to process?
>>
>> Thanks,
>>
>> Mark
>>
>> On 26/06/2020 09:24, Chirag Dewan wrote:
>>> Thanks Mark.
>>>
>>> *What is the typical response size for one of these requests? *
>>> It' basically a dummy response JSON of ~300bytes. I expect 2300bytes of
>>> response in my production use case, but the purpose here was to isolate
>> as
>>> many things as possible. Hence a dummy response.
>>>
>>> *How long does a typical test take to process? *
>>> I see Tomcat's memory reaching 28G in about 2hours at 19K TPS. The number
>>> of streams on my client was 500 - which means about 40 connections per
>>> second.
>>>
>>> * What are the GC roots for those RequestInfo objects?*
>>> https://ibb.co/fMRmCXZ
>>>
>>> I hope I was able to answer everything as expected. Thanks.
>>>
>>> On Thu, Jun 25, 2020 at 8:30 PM Mark Thomas  wrote:
>>>
 Thanks.

 I've looked at the code and I have tried various tests but I am unable
 to re-create a memory leak.

 The code used to (before I made a few changes this afternoon) retain a
 lot more memory per Stream and it is possible that what you are seeing
 is a system that doesn't have enough memory to achieve steady state.

 If you are able to build the latest 9.0.x and test that, that could be
 helpful. Alternatively, I could provide a test build for you to
 experiment with.

 Some additional questions that might aid understanding:

 - What is the typical response size for one of these requests?
 - How long does a typical test take to process?
 - What are the GC roots for those RequestInfo objects?

 Thanks again,

 Mark




 On 25/06/2020 15:10, Chirag Dewan wrote:
> Hi Mark,
>
> Its the default APR connector with 150 Threads.
>
> Chirag
>
> On Thu, 25 Jun, 2020, 7:30 pm Mark Thomas,  wrote:
>
>> On 25/06/2020 11:00, Chirag Dewan wrote:
>>> Thanks for the quick check Mark.
>>>
>>> These are the images I tried referring to:
>>>
>>> https://ibb.co/LzKtRgh
>>>
>>> https://ibb.co/2s7hqRL
>>>
>>> https://ibb.co/KmKj590
>>>
>>>
>>> The last one is the MAT screenshot showing many RequestInfo objects.
>>
>> Thanks. That certainly looks like a memory leak. I'll take a closer
>> look. Out of interest, how many threads is the Connector configured to
 use?
>>
>> Mark
>>
>>
>>>
>>>
>>> Thanks,
>>>
>>> Chirag
>>>
>>> On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas 
>> wrote:
>>>
 On 24/06/2020 12:17, Mark Thomas wrote:
> On 22/06/2020 11:06, Chirag Dewan wrote:
>> Hi,
>>
>> Update: We found that Tomcat goes OOM when a client closes and
>> opens
>> new
>> connections every second. In the memory dump, we see a lot of
>> RequestInfo objects that are causing the memory spike.
>>
>> After a while, Tomcat goes OOM and start rejecting request(I get a
>> request timed out on my client). This seems like a bug to me.
>>
>> For better understanding, let me explain my use case again:
>>
>> I have a jetty client that sends HTTP2 requests to Tomcat. My
>> requirement is to close a connection after a configurable(say
>> 5000)
>> number of requests/streams and open a new connection that
>> continues
 to
>> send requests. I close a connection by sending a GoAway frame.
>>
>> When I execute this use case under load, I see that after ~2hours
>> my
>> requests fail and I get a series of errors like request
>> timeouts(5seconds), invalid window update frame, and connection
 close
>> exception on my client.
>> On further debugging, I 

Re: Tomcat 9.0.36 - JDK 13/14

[Mark Thomas]

On 26/06/2020 05:45, Kiran Badi wrote:
> Hi All,
> 
> I wanted to check if tomcat 9.0.36 supports open jdk 13/14.

Supported Java versions are listed at:
http://tomcat.apache.org/whichversion.html

"Java 8 and later" includes Java 13 and Java 14.

> I created a simple spring boot war file and compiled/built it with openjdk
> 13/14. After running maven install , I deployed the war file from the
> target directory to tomcat webapps using tomcat manager. It did not work
> and gave me 404 messages with both 13/14. No error or any exception
> anywhere in logs.Catalina log just says a war file is deployed.

That is normally indicative of a configuration error.

> Then i compiled the same spring boot app with jdk 8 and deployed it with
> tomcat and it works fine. I am able to call my endpoints with no issues.
> 
> I am having a hard time deploying angular/spring boot and building war file
> and deploying it on tomcat 9.0x with openjdk 13. So I thought this might be
> a good place to start with.

If you can provide the code you use to create the sample, e.g. as a
GitHub project somebody may be able to take a look.

Mark

> 
> I used the below pom file.
> 
> 
> http://maven.apache.org/POM/4.0.0; xmlns:xsi="
> http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> https://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> 
> org.springframework.boot
> spring-boot-starter-parent
> 2.3.1.RELEASE
>  
> 
> com.kiran
> springwar
> 1.0.2-SNAPSHOT
> war
> springwar
> Sample project to deploy war to tomcat
> 
> 
> 14
> 
> 
> 
> 
> org.springframework.boot
> spring-boot-starter-web
> 
> 
> 
> org.springframework.boot
> spring-boot-starter-tomcat
> provided
> 
> 
> org.springframework.boot
> spring-boot-starter-test
> test
> 
> 
> org.junit.vintage
> junit-vintage-engine
> 
> 
> 
> 
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Connection Closure due to Fatal Stream with HTTP2

[Chirag Dewan]

Hey Mark,

*Are you using h2c or h2 in your test?*
We are using h2c


*Do you see the same issue if you switch the the NIO connector? Note
performance differences between NIO and NIO2 are very small.*

I havent tried with NIO honestly. Can quickly execute and check. Will
update the results.

*How long does a single request take to process?*
In normal scenarios, less than 3ms.

Thanks,
Chirag

On Fri, Jun 26, 2020 at 3:26 PM Mark Thomas  wrote:

> Hi,
>
> Thanks for the additional information. The GC roots were particularly
> informative.
>
> Those RequestInfo objects are associated with HTTP/1.1 requests, not
> HTTP/2 requests.
>
> Some further questions to try and track down what is going on:
>
> - Are you using h2c or h2 in your test?
>
> - Do you see the same issue if you switch the the NIO connector? Note
>   performance differences between NIO and NIO2 are very small.
>
> - How long does a single request take to process?
>
> Thanks,
>
> Mark
>
> On 26/06/2020 09:24, Chirag Dewan wrote:
> > Thanks Mark.
> >
> > *What is the typical response size for one of these requests? *
> > It' basically a dummy response JSON of ~300bytes. I expect 2300bytes of
> > response in my production use case, but the purpose here was to isolate
> as
> > many things as possible. Hence a dummy response.
> >
> > *How long does a typical test take to process? *
> > I see Tomcat's memory reaching 28G in about 2hours at 19K TPS. The number
> > of streams on my client was 500 - which means about 40 connections per
> > second.
> >
> > * What are the GC roots for those RequestInfo objects?*
> > https://ibb.co/fMRmCXZ
> >
> > I hope I was able to answer everything as expected. Thanks.
> >
> > On Thu, Jun 25, 2020 at 8:30 PM Mark Thomas  wrote:
> >
> >> Thanks.
> >>
> >> I've looked at the code and I have tried various tests but I am unable
> >> to re-create a memory leak.
> >>
> >> The code used to (before I made a few changes this afternoon) retain a
> >> lot more memory per Stream and it is possible that what you are seeing
> >> is a system that doesn't have enough memory to achieve steady state.
> >>
> >> If you are able to build the latest 9.0.x and test that, that could be
> >> helpful. Alternatively, I could provide a test build for you to
> >> experiment with.
> >>
> >> Some additional questions that might aid understanding:
> >>
> >> - What is the typical response size for one of these requests?
> >> - How long does a typical test take to process?
> >> - What are the GC roots for those RequestInfo objects?
> >>
> >> Thanks again,
> >>
> >> Mark
> >>
> >>
> >>
> >>
> >> On 25/06/2020 15:10, Chirag Dewan wrote:
> >>> Hi Mark,
> >>>
> >>> Its the default APR connector with 150 Threads.
> >>>
> >>> Chirag
> >>>
> >>> On Thu, 25 Jun, 2020, 7:30 pm Mark Thomas,  wrote:
> >>>
>  On 25/06/2020 11:00, Chirag Dewan wrote:
> > Thanks for the quick check Mark.
> >
> > These are the images I tried referring to:
> >
> > https://ibb.co/LzKtRgh
> >
> > https://ibb.co/2s7hqRL
> >
> > https://ibb.co/KmKj590
> >
> >
> > The last one is the MAT screenshot showing many RequestInfo objects.
> 
>  Thanks. That certainly looks like a memory leak. I'll take a closer
>  look. Out of interest, how many threads is the Connector configured to
> >> use?
> 
>  Mark
> 
> 
> >
> >
> > Thanks,
> >
> > Chirag
> >
> > On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas 
> wrote:
> >
> >> On 24/06/2020 12:17, Mark Thomas wrote:
> >>> On 22/06/2020 11:06, Chirag Dewan wrote:
>  Hi,
> 
>  Update: We found that Tomcat goes OOM when a client closes and
> opens
>  new
>  connections every second. In the memory dump, we see a lot of
>  RequestInfo objects that are causing the memory spike.
> 
>  After a while, Tomcat goes OOM and start rejecting request(I get a
>  request timed out on my client). This seems like a bug to me.
> 
>  For better understanding, let me explain my use case again:
> 
>  I have a jetty client that sends HTTP2 requests to Tomcat. My
>  requirement is to close a connection after a configurable(say
> 5000)
>  number of requests/streams and open a new connection that
> continues
> >> to
>  send requests. I close a connection by sending a GoAway frame.
> 
>  When I execute this use case under load, I see that after ~2hours
> my
>  requests fail and I get a series of errors like request
>  timeouts(5seconds), invalid window update frame, and connection
> >> close
>  exception on my client.
>  On further debugging, I found that it's a Tomcat memory problem
> and
> >> it
>  goes OOM after sometime under heavy load with multiple connections
>  being
>  re-established by the clients.
> 
>  image.png
> 
>  image.png
> 

Re: jsvc - non root - log as root

[Mark Thomas]

On 26/06/2020 08:21, Jürgen Weber wrote:
> Hi,
> 
> when you run tomcat with jsvc and have jsvc drop privileges to a
> different user, stdout and stderr log files are still created with
> root as owner.
> Can you make jsvc create them as the -user ?

I'm no C expert but my reading of
https://github.com/apache/commons-daemon/blob/master/src/native/unix/native/jsvc-unix.c#L1039
is no.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Question around catalina.policy change back with 9.0.33, etc.

[Mark Thomas]

On 26/06/2020 00:15, jonmcalexan...@wellsfargo.com.INVALID wrote:
> I have a developer that is asking WHY the following policies were set to read 
> only. The Change Log doesn't illuminate why.
> 
> // The cookie code needs these.
> permission java.util.PropertyPermission
>  "org.apache.catalina.STRICT_SERVLET_COMPLIANCE", "read";
> permission java.util.PropertyPermission
>  "org.apache.tomcat.util.http.ServerCookie.STRICT_NAMING", "read";
> permission java.util.PropertyPermission
>  "org.apache.tomcat.util.http.ServerCookie.FWD_SLASH_IS_SEPARATOR", 
> "read";
> 
> Any information I can share with her?

Those permissions were removed, not set to read only, for 9.0.9 onwards.

It was the result of a refactoring:
https://github.com/apache/tomcat/commit/6ceb931e1aac0355e0980d09814559f24406a14a

I made the change but it was 2 years ago. I don't recall the motivation
off-hand. /me heads off to look at the archives...

..and that is why we have the archives.

It appears to stem from this issue:
https://bz.apache.org/bugzilla/show_bug.cgi?id=43925

The fix for that issue led to this:
https://tomcat.markmail.org/thread/mab6jbyb57phslwk

Rather than add a permission, the code was refactored so the additional
permission (and some of the existing permissions) were no longer required.

It isn't documented, but I strongly suspect that got me looking at other
permissions which led to the refactoring that allowed the removal of the
cookie permissions.

The general principle behind all of this being the fewer explicit
permissions you need to give to applications the better.

HTH,

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Connection Closure due to Fatal Stream with HTTP2

[Mark Thomas]

Hi,

Thanks for the additional information. The GC roots were particularly
informative.

Those RequestInfo objects are associated with HTTP/1.1 requests, not
HTTP/2 requests.

Some further questions to try and track down what is going on:

- Are you using h2c or h2 in your test?

- Do you see the same issue if you switch the the NIO connector? Note
  performance differences between NIO and NIO2 are very small.

- How long does a single request take to process?

Thanks,

Mark

On 26/06/2020 09:24, Chirag Dewan wrote:
> Thanks Mark.
> 
> *What is the typical response size for one of these requests? *
> It' basically a dummy response JSON of ~300bytes. I expect 2300bytes of
> response in my production use case, but the purpose here was to isolate as
> many things as possible. Hence a dummy response.
> 
> *How long does a typical test take to process? *
> I see Tomcat's memory reaching 28G in about 2hours at 19K TPS. The number
> of streams on my client was 500 - which means about 40 connections per
> second.
> 
> * What are the GC roots for those RequestInfo objects?*
> https://ibb.co/fMRmCXZ
> 
> I hope I was able to answer everything as expected. Thanks.
> 
> On Thu, Jun 25, 2020 at 8:30 PM Mark Thomas  wrote:
> 
>> Thanks.
>>
>> I've looked at the code and I have tried various tests but I am unable
>> to re-create a memory leak.
>>
>> The code used to (before I made a few changes this afternoon) retain a
>> lot more memory per Stream and it is possible that what you are seeing
>> is a system that doesn't have enough memory to achieve steady state.
>>
>> If you are able to build the latest 9.0.x and test that, that could be
>> helpful. Alternatively, I could provide a test build for you to
>> experiment with.
>>
>> Some additional questions that might aid understanding:
>>
>> - What is the typical response size for one of these requests?
>> - How long does a typical test take to process?
>> - What are the GC roots for those RequestInfo objects?
>>
>> Thanks again,
>>
>> Mark
>>
>>
>>
>>
>> On 25/06/2020 15:10, Chirag Dewan wrote:
>>> Hi Mark,
>>>
>>> Its the default APR connector with 150 Threads.
>>>
>>> Chirag
>>>
>>> On Thu, 25 Jun, 2020, 7:30 pm Mark Thomas,  wrote:
>>>
 On 25/06/2020 11:00, Chirag Dewan wrote:
> Thanks for the quick check Mark.
>
> These are the images I tried referring to:
>
> https://ibb.co/LzKtRgh
>
> https://ibb.co/2s7hqRL
>
> https://ibb.co/KmKj590
>
>
> The last one is the MAT screenshot showing many RequestInfo objects.

 Thanks. That certainly looks like a memory leak. I'll take a closer
 look. Out of interest, how many threads is the Connector configured to
>> use?

 Mark


>
>
> Thanks,
>
> Chirag
>
> On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas  wrote:
>
>> On 24/06/2020 12:17, Mark Thomas wrote:
>>> On 22/06/2020 11:06, Chirag Dewan wrote:
 Hi,

 Update: We found that Tomcat goes OOM when a client closes and opens
 new
 connections every second. In the memory dump, we see a lot of
 RequestInfo objects that are causing the memory spike.

 After a while, Tomcat goes OOM and start rejecting request(I get a
 request timed out on my client). This seems like a bug to me.

 For better understanding, let me explain my use case again:

 I have a jetty client that sends HTTP2 requests to Tomcat. My
 requirement is to close a connection after a configurable(say 5000)
 number of requests/streams and open a new connection that continues
>> to
 send requests. I close a connection by sending a GoAway frame.

 When I execute this use case under load, I see that after ~2hours my
 requests fail and I get a series of errors like request
 timeouts(5seconds), invalid window update frame, and connection
>> close
 exception on my client.
 On further debugging, I found that it's a Tomcat memory problem and
>> it
 goes OOM after sometime under heavy load with multiple connections
 being
 re-established by the clients.

 image.png

 image.png

 Is this a known issue? Or a known behavior with Tomcat?
>>>
>>> Embedded images get dropped by the list software. Post those images
>>> somewhere we can see them.
>>>
 Please let me know if you any experience with such a situation.
>> Thanks
 in advance.
>>>
>>> Nothing comes to mind.
>>>
>>> I'll try some simple tests with HTTP/2.
>>
>> I don't see a memory leak (the memory is reclaimed eventually) but I
>> do
>> see possibilities to release memory associated with request processing
>> sooner.
>>
>> Right now you need to allocate more memory to the Java process to
>> enable
>> Tomcat to handle the HTTP/2 load it is presented with.

Re: Connection Closure due to Fatal Stream with HTTP2

[Chirag Dewan]

Thanks Mark.

*What is the typical response size for one of these requests? *
It' basically a dummy response JSON of ~300bytes. I expect 2300bytes of
response in my production use case, but the purpose here was to isolate as
many things as possible. Hence a dummy response.

*How long does a typical test take to process? *
I see Tomcat's memory reaching 28G in about 2hours at 19K TPS. The number
of streams on my client was 500 - which means about 40 connections per
second.

* What are the GC roots for those RequestInfo objects?*
https://ibb.co/fMRmCXZ

I hope I was able to answer everything as expected. Thanks.

On Thu, Jun 25, 2020 at 8:30 PM Mark Thomas  wrote:

> Thanks.
>
> I've looked at the code and I have tried various tests but I am unable
> to re-create a memory leak.
>
> The code used to (before I made a few changes this afternoon) retain a
> lot more memory per Stream and it is possible that what you are seeing
> is a system that doesn't have enough memory to achieve steady state.
>
> If you are able to build the latest 9.0.x and test that, that could be
> helpful. Alternatively, I could provide a test build for you to
> experiment with.
>
> Some additional questions that might aid understanding:
>
> - What is the typical response size for one of these requests?
> - How long does a typical test take to process?
> - What are the GC roots for those RequestInfo objects?
>
> Thanks again,
>
> Mark
>
>
>
>
> On 25/06/2020 15:10, Chirag Dewan wrote:
> > Hi Mark,
> >
> > Its the default APR connector with 150 Threads.
> >
> > Chirag
> >
> > On Thu, 25 Jun, 2020, 7:30 pm Mark Thomas,  wrote:
> >
> >> On 25/06/2020 11:00, Chirag Dewan wrote:
> >>> Thanks for the quick check Mark.
> >>>
> >>> These are the images I tried referring to:
> >>>
> >>> https://ibb.co/LzKtRgh
> >>>
> >>> https://ibb.co/2s7hqRL
> >>>
> >>> https://ibb.co/KmKj590
> >>>
> >>>
> >>> The last one is the MAT screenshot showing many RequestInfo objects.
> >>
> >> Thanks. That certainly looks like a memory leak. I'll take a closer
> >> look. Out of interest, how many threads is the Connector configured to
> use?
> >>
> >> Mark
> >>
> >>
> >>>
> >>>
> >>> Thanks,
> >>>
> >>> Chirag
> >>>
> >>> On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas  wrote:
> >>>
>  On 24/06/2020 12:17, Mark Thomas wrote:
> > On 22/06/2020 11:06, Chirag Dewan wrote:
> >> Hi,
> >>
> >> Update: We found that Tomcat goes OOM when a client closes and opens
> >> new
> >> connections every second. In the memory dump, we see a lot of
> >> RequestInfo objects that are causing the memory spike.
> >>
> >> After a while, Tomcat goes OOM and start rejecting request(I get a
> >> request timed out on my client). This seems like a bug to me.
> >>
> >> For better understanding, let me explain my use case again:
> >>
> >> I have a jetty client that sends HTTP2 requests to Tomcat. My
> >> requirement is to close a connection after a configurable(say 5000)
> >> number of requests/streams and open a new connection that continues
> to
> >> send requests. I close a connection by sending a GoAway frame.
> >>
> >> When I execute this use case under load, I see that after ~2hours my
> >> requests fail and I get a series of errors like request
> >> timeouts(5seconds), invalid window update frame, and connection
> close
> >> exception on my client.
> >> On further debugging, I found that it's a Tomcat memory problem and
> it
> >> goes OOM after sometime under heavy load with multiple connections
> >> being
> >> re-established by the clients.
> >>
> >> image.png
> >>
> >> image.png
> >>
> >> Is this a known issue? Or a known behavior with Tomcat?
> >
> > Embedded images get dropped by the list software. Post those images
> > somewhere we can see them.
> >
> >> Please let me know if you any experience with such a situation.
> Thanks
> >> in advance.
> >
> > Nothing comes to mind.
> >
> > I'll try some simple tests with HTTP/2.
> 
>  I don't see a memory leak (the memory is reclaimed eventually) but I
> do
>  see possibilities to release memory associated with request processing
>  sooner.
> 
>  Right now you need to allocate more memory to the Java process to
> enable
>  Tomcat to handle the HTTP/2 load it is presented with.
> 
>  It looks like a reasonable chunk of memory is released when the
>  Connection closes that could be released earlier when the associated
>  Stream closes. I'll take a look at what can be done in that area. In
> the
>  meantime, reducing the number of Streams you allow on a Connection
>  before it is closed should reduce overall memory usage.
> 
>  Mark
> 
>  -
>  To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>  For additional commands, e-mail: 

jsvc - non root - log as root

[Jürgen Weber]

Hi,

when you run tomcat with jsvc and have jsvc drop privileges to a
different user, stdout and stderr log files are still created with
root as owner.
Can you make jsvc create them as the -user ?

weberjn@beo:~/apache-tomcat-9.0.36/logs$ ll
total 20
-rw--- 1 weberjn weberjn 4630 Jun 26 08:28 catalina.2020-06-26.log
-rw--- 1 rootroot4630 Jun 26 08:28 catalina.err
-rw--- 1 rootroot  28 Jun 26 08:28 catalina.out
-rw--- 1 weberjn weberjn0 Jun 26 08:28 host-manager.2020-06-26.log
-rw--- 1 weberjn weberjn0 Jun 26 08:28 localhost.2020-06-26.log
-rw--- 1 weberjn weberjn0 Jun 26 08:28
localhost_access_log.2020-06-26.txt
-rw--- 1 weberjn weberjn0 Jun 26 08:28 manager.2020-06-26.log

jsvc \
-classpath 
$CATALINA_HOME/bin/bootstrap.jar:$CATALINA_HOME/bin/tomcat-juli.jar
\
-outfile $CATALINA_BASE/logs/catalina.out \
-errfile $CATALINA_BASE/logs/catalina.err \
-java-home /usr/lib/jvm/java-11-openjdk-amd64 \
-user weberjn \
-Dcatalina.home=$CATALINA_HOME \
-Dcatalina.base=$CATALINA_BASE \
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager \
-Djava.util.logging.config.file=$CATALINA_BASE/conf/logging.properties \
org.apache.catalina.startup.Bootstrap

jsvc (Apache Commons Daemon) 1.2.3-dev

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org