Re: Tomcat SecurityListener

[Mark Eggers]

Shawn,

On 10/12/2020 12:59 PM, Beard, Shawn wrote:

Tomcat 9.0.31.0 loads a org.apache.catalina.security.SecurityListener by 
default in the catalina.sh file.

This SecurityListener also sets the UMASK of files to 0027. This has the effect 
of any file tomcat creates or the app running in tomcat creates with 
permissions or -rw-r-

This is causing a problem for us as it prevents certain people from being able 
to read log files or read any file the application might create. Putting these 
users in the group of the user that tomcat runs as is not an option.

I’ve tried changing the catalina.sh to set the UMASK to something like 0022 but 
that prevents tomcat from starting with an error that it has to me at least as 
restrictive as 0027.

I’ve also tried setting the UMASK to 0022 in the setenv.sh with same results.

I’m hesitant to comment out the loading of the security listener in catalina.sh 
as I don’t want to disable anything else important that it may be doing from a 
security standpoint.

Does anyone have any ideas as to a workaround?
 ​

Shawn   Beard‑ Sr. Systems Engineer

Middleware Engineering

[cid:image624238.png@1BC27BA2.B6427C15]
3840 109th Street   ,   Urbandale   ,   IA  50322

Phone: +1-515-564-2528
Email:  sbe...@wrberkley.com

Website: https://berkleytechnologyservices.com/




[cid:image040736.jpg@BA9411B9.333ADE5A]

Technology Leadership Unleashing Business Potential







CONFIDENTIALITY NOTICE: This e-mail and the transmitted documents contain 
private, privileged and confidential information belonging to the sender. The 
information therein is solely for the use of the addressee. If your receipt of 
this transmission has occurred as the result of an error, please immediately 
notify us so we can arrange for the return of the documents. In such 
circumstances, you are advised that you may not disclose, copy, distribute or 
take any other action in reliance on the information transmitted.



I don't know what your security or audit requirements are. These are 
some options off the top of my head.


1. Service account for the user that runs Tomcat
You don't run Tomcat as root, correct?

You could then have a list of authorized sudoers, use two factor 
authentication (maybe for both the users and the service account), and 
audit both the service account and the sudoers accounts.


Prevent the service account from being accessed directly.

2. Remote logging
This would take care of needing to access log files on the server, but 
it would not allow anyone to audit application-created files.


Speaking of application-created files, I hope that these are not 
user-provided files that are then directly accessible. Without careful 
auditing, that can lead to some pretty serious security breaches.


. . . just my two cents.
/mde/


OpenPGP_0x41466EC60D793C2D.asc
Description: application/pgp-keys


OpenPGP_signature
Description: OpenPGP digital signature

Tomcat SecurityListener

[Beard, Shawn]

Tomcat 9.0.31.0 loads a org.apache.catalina.security.SecurityListener by 
default in the catalina.sh file.

This SecurityListener also sets the UMASK of files to 0027. This has the effect 
of any file tomcat creates or the app running in tomcat creates with 
permissions or -rw-r-

This is causing a problem for us as it prevents certain people from being able 
to read log files or read any file the application might create. Putting these 
users in the group of the user that tomcat runs as is not an option.

I’ve tried changing the catalina.sh to set the UMASK to something like 0022 but 
that prevents tomcat from starting with an error that it has to me at least as 
restrictive as 0027.

I’ve also tried setting the UMASK to 0022 in the setenv.sh with same results.

I’m hesitant to comment out the loading of the security listener in catalina.sh 
as I don’t want to disable anything else important that it may be doing from a 
security standpoint.

Does anyone have any ideas as to a workaround?
​

Shawn   Beard‑ Sr. Systems Engineer

Middleware Engineering

[cid:image624238.png@1BC27BA2.B6427C15]
3840 109th Street   ,   Urbandale   ,   IA  50322

Phone: +1-515-564-2528
Email:  sbe...@wrberkley.com

Website: https://berkleytechnologyservices.com/




[cid:image040736.jpg@BA9411B9.333ADE5A]

Technology Leadership Unleashing Business Potential







CONFIDENTIALITY NOTICE: This e-mail and the transmitted documents contain 
private, privileged and confidential information belonging to the sender. The 
information therein is solely for the use of the addressee. If your receipt of 
this transmission has occurred as the result of an error, please immediately 
notify us so we can arrange for the return of the documents. In such 
circumstances, you are advised that you may not disclose, copy, distribute or 
take any other action in reliance on the information transmitted.

Re: Deploying war, Negative Date exception

[Mark Thomas]

On 12/10/2020 18:48, Christopher Schultz wrote:



 > There is already a check for -1 for the last-modified time for the file
> in the ZIP archive. Also for 0 for some reason (sorry, Brian Kernighan,
> you can't store your first file in a ZIP file!).

I've tracked that change back to this:
https://bz.apache.org/bugzilla/show_bug.cgi?id=33636

I don't see why "0" is not allowed (but neither do I see not allowing it
being a significant problem).

> I think that any negative value should be ignored, so the expanded-file
> on the disk should get "now" as its last-updated time. So just change
> the comparison to:
> 
>   if(lastModified > 0) {
> expandedFile.setLastModified(lastModified);
>   }

Hmm. I do wonder about silently swallowing what is almost certainly some
some of build system error (and breaking caching as per BZ 33636) when
lastModified is negative.

Ignoring files that return -1 (not specified) seems reasonable.

I think an exception is better than silently swallowing here.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Deploying war, Negative Date exception

[Christopher Schultz]

Mark,

On 10/12/20 09:50, Mark Thomas wrote:
> On 12/10/2020 13:53, Mark Thomas wrote:
>> On 12/10/2020 12:49, Mark Thomas wrote:
>>> On 12/10/2020 12:19, Peter Henderson wrote:
 Hello fellow tomcat users.

 My environment.
 Tomcat: 9.0.39
 Java: openjdk 11.0.8 2020-07-14
 OS: Ubuntu 18.04.5 LTS

 Source code [0]

 When deploying this war [1], by copying it into the webapps directory,
 I get this exception. [2]
 java.lang.IllegalArgumentException: Negative time


 I only started seeing this exception when I upgraded my projects build tool
 version
 from
 sbt.version=1.3.10
 to
 sbt.version=1.4.0


 Is this a tomcat bug, a build tool bug or most likely something I'm doing
 wrong?
>>>
>>> Looks like an issue with the dates of files in the WAR.
>>>
>>> If you look at the dates of the files in the WAR nearly all of them are
>>> in the future (07 Feb 2106, 06:28).
>>>
>>> It looks like something is overflowing but a Java long shouldn't do that.
>>>
>>> Need to dig into exactly what is going on as this looks like it should
>>> work - even if the WAR contains files created almost a century in the
>>> future.
>>
>> Hmm. I see the 2106 date on the file system and with Java 8 but with
>> Java 11 I see 1969-dec-31 23:00 which gives -360 which triggers the
>> exception.
>>
>> The root cause appears to be in the JRE at this point. Whether it is in
>> the JRE used to create the WAR or the JRE reading the WAR is TBD.
>>
>> I think I am going to have to look at the raw bytes and the zip file
>> spec to figure out where the root cause is.
> 
> That was fun.
> 
> Per the zip spec, the last modified time on those files is:
> 
> 1969-Dec-31 23:00:00 UTC
> 
> i.e. 1 hour before the Epoch at 1970-Jan-01 00:00:00 UTC
> 
> It is stored as a signed 32-bit int (F1F0) which is -3600 (zip
> timestamps are in seconds since the Epoch).
> 
> Java 8 reads this incorrectly as an unsigned int (4294963696) which,
> when taken as seconds since Epoch, gives 2016-Feb-07 05:28:16 UTC.

Underflow! :(

> (Incidently the archiver that ships with Ubuntu appears to make the same
> error)
> 
> Java 11 reads this correctly but Java does not let you set times before
> the Epoch so the exception is triggered.
> 
> The short version is that the modification times of the files in the WAR
> are being set to "1969-Dec-31 23:00:00 UTC" which Java doesn't like.
> 
> Tomcat could handle this more gracefully but the end result will be the
> same - the WAR isn't going to deploy. I'm not convinced it is worth
> doing anything here.
> 
> It looks like the fix will be somewhere in the build system used to
> create the WAR.

There is already a check for -1 for the last-modified time for the file
in the ZIP archive. Also for 0 for some reason (sorry, Brian Kernighan,
you can't store your first file in a ZIP file!).

I think that any negative value should be ignored, so the expanded-file
on the disk should get "now" as its last-updated time. So just change
the comparison to:

  if(lastModified > 0) {
expandedFile.setLastModified(lastModified);
  }

-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Deploying war, Negative Date exception

[Peter Henderson]

On Mon, 12 Oct 2020 at 14:50, Mark Thomas  wrote:

> On 12/10/2020 13:53, Mark Thomas wrote:
> > On 12/10/2020 12:49, Mark Thomas wrote:
> >> On 12/10/2020 12:19, Peter Henderson wrote:
> >>> Hello fellow tomcat users.
> >>>
> >>> My environment.
> >>> Tomcat: 9.0.39
> >>> Java: openjdk 11.0.8 2020-07-14
> >>> OS: Ubuntu 18.04.5 LTS
> >>>
> >>> Source code [0]
> >>>
> >>> When deploying this war [1], by copying it into the webapps directory,
> >>> I get this exception. [2]
> >>> java.lang.IllegalArgumentException: Negative time
> >>>
> >>>
> >>> I only started seeing this exception when I upgraded my projects build
> tool
> >>> version
> >>> from
> >>> sbt.version=1.3.10
> >>> to
> >>> sbt.version=1.4.0
> >>>
> >>>
> >>> Is this a tomcat bug, a build tool bug or most likely something I'm
> doing
> >>> wrong?
> >>
> >> Looks like an issue with the dates of files in the WAR.
> >>
> >> If you look at the dates of the files in the WAR nearly all of them are
> >> in the future (07 Feb 2106, 06:28).
> >>
> >> It looks like something is overflowing but a Java long shouldn't do
> that.
> >>
> >> Need to dig into exactly what is going on as this looks like it should
> >> work - even if the WAR contains files created almost a century in the
> >> future.
> >
> > Hmm. I see the 2106 date on the file system and with Java 8 but with
> > Java 11 I see 1969-dec-31 23:00 which gives -360 which triggers the
> > exception.
> >
> > The root cause appears to be in the JRE at this point. Whether it is in
> > the JRE used to create the WAR or the JRE reading the WAR is TBD.
> >
> > I think I am going to have to look at the raw bytes and the zip file
> > spec to figure out where the root cause is.
>
> That was fun.
>
> Per the zip spec, the last modified time on those files is:
>
> 1969-Dec-31 23:00:00 UTC
>
> i.e. 1 hour before the Epoch at 1970-Jan-01 00:00:00 UTC
>
> It is stored as a signed 32-bit int (F1F0) which is -3600 (zip
> timestamps are in seconds since the Epoch).
>
> Java 8 reads this incorrectly as an unsigned int (4294963696) which,
> when taken as seconds since Epoch, gives 2016-Feb-07 05:28:16 UTC.
>
> (Incidently the archiver that ships with Ubuntu appears to make the same
> error)
>
> Java 11 reads this correctly but Java does not let you set times before
> the Epoch so the exception is triggered.
>
> The short version is that the modification times of the files in the WAR
> are being set to "1969-Dec-31 23:00:00 UTC" which Java doesn't like.
>
> Tomcat could handle this more gracefully but the end result will be the
> same - the WAR isn't going to deploy. I'm not convinced it is worth
> doing anything here.
>
> It looks like the fix will be somewhere in the build system used to
> create the WAR.
>

Thanks for digging into this.

For anyone else who runs into this.

When upgrading to sbt >= 1.4.0
An environment variable needs to be set
SOURCE_DATE_EPOCH
[0]

I suspect the bug is in [1] with the orElse(0L) start of epoch looks
familiar.


[0] https://reproducible-builds.org/docs/source-date-epoch/
[1]
https://github.com/sbt/sbt/pull/5344/commits/1d0a41520071c2fcf694d6b68e4b5e7721f7c321

Peter.






>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

-- 
Peter Henderson

Director
Starjar Ltd.
www.starjar.com
0330 088 1662

Re: java.lang.OutOfMemoryError: PermGen space when we redeploy same application multiple times

[Prabhu Gurunathan]

Thanks Manuel Dominguez

On Mon, Oct 12, 2020 at 5:51 PM Manuel Dominguez Sarmiento
 wrote:
>
> The articles in this page will be helpful:
> https://java.jiderhamn.se/category/classloader-leaks/
>
> On 12/10/2020 04:19, Mark Thomas wrote:
> > On 11/10/2020 02:39, Prabhu Gurunathan wrote:
> >> Hi All,
> >>
> >> We have an setup where we are using OpenJDK 1.7 and Tomcat 7.0.100 ,
> >> in CentOs 7 Env . and we have many application deployed in
> >> Tomcat/webapps and the common lib's needed for those apps are kept in
> >> Tomcat/lib directory like log4j , commons-fileupload  ,xerces , Xalan
> >> .. etc
> >>
> >> The problem here is When we try to undeploy and deploy same
> >> applications multiple time we are  getting into
> >> java.lang.OutOfMemoryError: PermGen space very quickly . Want to know
> >> is it very generic problem on this deployment model or is this can be
> >> fixed anyway ?
> > The memory leak could be in any of:
> > - the application code
> > - a library the application depends on
> > - the JVM
> > - Tomcat
> >
> > In all cases, it should be possible to fix it. There might even be a
> > short-term workaround available.
> >
> > First of all, make sure that the JreMemoryLeakPreventionListener is
> > enabled in your configuration.
> >
> > Secondly, make sure that you don't get any warnings about possible
> > memory leaks in the logs when you reload an application. If you do, fix
> > the leaks identified.
> >
> > If you still see issues, the short version is:
> > - user a profiler
> > - reload each app in turn until you see more strong references to
> >org.apache.catalina.loader.[Parallel]WebappClassLoader instances in
> >memory than you have web applications
> > - find the one where started = false
> > - trace its GC roots
> > - that will tell you where the memory leak is
> > - how it was created might be harder to track down
> >
> > The long version is in a presentation linked from the Tomcat web site:
> > https://home.apache.org/~markt/presentations/2010-08-05-Memory-Leaks-JavaOne-60mins.pdf
> >
> > If you have any questions, you can ask here.
> >
> > Mark
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
> >
>

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)

[Mark Thomas]

On 12/10/2020 08:02, Arshiya Shariff wrote:
> Hi Mark , 
> 
> The issue is reproduced with version 9.0.39 as well. Max threads in Tomcat is 
> 200.
> 
> Please find the case:
> Client:JMeter 5.2.1 (With http2 plugin)
> TPS: around 20
> No of users from JMeter : 700
> Message payload size: 6 KB to 34 KB
> Loop: Infinite 
> We let the loop run infinitely and see the java.lang.StackOverflowError trace 
> printed multiple times in the log within few minutes of starting the test.

POSTing to what URL?

What does that URL do with the POSTed content? Ignore it? Read it from
an InputStream? Read it via getParameter()?

Is JMeter run on the same machine as Tomcat?

Do you use the JMeter GUI or the command line?

What are the specs of the server(s) being used?

You need to provide the exact steps to recreate this issue on a clean
install of Tomcat 9.0.39 as provided by the ASF.

Mark


> Please help us with this . What is the impact of StackOverflowError ?
> 
> Thanks and Regards
> Arshiya Shariff
> 
> -Original Message-
> From: Mark Thomas  
> Sent: Friday, October 9, 2020 5:31 PM
> To: users@tomcat.apache.org
> Subject: Re: HTTP2: memory filled up fast on increasing the connections to 
> 1000/2000 (Embedded tomcat 9.0.38)
> 
> On 09/10/2020 12:32, Arshiya Shariff wrote:
>> Hi,
>>
>> Mark , with the test runs that I performed over clean 9.0.x branch I was not 
>> able to reproduce this.
> 
> Good. But I'd really like to understand why...
> 
>> But with 9.0.38 and the jars built from 9.0.x with hash: 
>> c8ec2d4cde3a31b0e9df9a30e7915d77ba725545  , with 700 or 1000 users 
>> (connections) and on sending 1000 Requests per second (or even lesser) , 
>> payload of 16K  from JMeter I can see that this Exception occurs within few 
>> minutes of starting the test . The maxThreads configured in tomcat is 200 .
>>
>> How often do you see these errors in your test run?
>> Randomly, at times 2 or 3 such traces.
> 
> OK. Definitely a timing issue then.
> 
>> Do you have the other end of that stack trace?
>> It is only the two lines that is recursively printed till the end about  
>> ~500 times in one trace  :
>> at 
>> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
>> at 
>> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandl
>> er.completed(SocketWrapperBase.java:1100)
> 
> Doesn't tell me much unfortunately.
> 
>> I see the trace starting with :
>> Exception in thread "http-nio-x.y.z-1090-exec-107" 
>> java.lang.StackOverflowError 
>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:446)
>> at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174)
>> at 
>> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468)
>> at 
>> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandl
>> er.completed(SocketWrapperBase.java:1100)
>>
>>  (OR)
>>
>> Exception in thread "http-nio-x.y.z-1090-exec-87" 
>> java.lang.StackOverflowError
>> at sun.nio.ch.IOVecWrapper.get(IOVecWrapper.java:96)
>> at sun.nio.ch.IOUtil.read(IOUtil.java:240)
>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:440)
>> at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174)
>> at 
>> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468)
>> at 
>> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
>> at 
>> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
>> at 
>> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
>> .
>> .
>> .
>> .
>> at 
>> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
>> at 
>> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
>> at 
>> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
>> at 
>> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandl
>> er.completed(SocketWrapperBase.java:1100)
>>
>> Is there anything that was fixed around this in latest 9.0.x branch ?
> 
> Not obviously. I've reviewed every commit since c8ec2d4c. There is nothing 
> that directly works with the I/O. There is 1e97ab2 which fixes a relatively 
> recent regression in the HTTP/2 code. I guess it is possible (but it seems a 
> bit of a stretch) that that bug is triggering an issue in JMeter which in 
> turn is sending invalid HTTP/2 packets.
> 
> I think at this point, given the relatively small number of commits between 
> c8ec2d4c and HEAD, the most useful thing you could do is run a 

Re: Deploying war, Negative Date exception

[Mark Thomas]

On 12/10/2020 13:53, Mark Thomas wrote:
> On 12/10/2020 12:49, Mark Thomas wrote:
>> On 12/10/2020 12:19, Peter Henderson wrote:
>>> Hello fellow tomcat users.
>>>
>>> My environment.
>>> Tomcat: 9.0.39
>>> Java: openjdk 11.0.8 2020-07-14
>>> OS: Ubuntu 18.04.5 LTS
>>>
>>> Source code [0]
>>>
>>> When deploying this war [1], by copying it into the webapps directory,
>>> I get this exception. [2]
>>> java.lang.IllegalArgumentException: Negative time
>>>
>>>
>>> I only started seeing this exception when I upgraded my projects build tool
>>> version
>>> from
>>> sbt.version=1.3.10
>>> to
>>> sbt.version=1.4.0
>>>
>>>
>>> Is this a tomcat bug, a build tool bug or most likely something I'm doing
>>> wrong?
>>
>> Looks like an issue with the dates of files in the WAR.
>>
>> If you look at the dates of the files in the WAR nearly all of them are
>> in the future (07 Feb 2106, 06:28).
>>
>> It looks like something is overflowing but a Java long shouldn't do that.
>>
>> Need to dig into exactly what is going on as this looks like it should
>> work - even if the WAR contains files created almost a century in the
>> future.
> 
> Hmm. I see the 2106 date on the file system and with Java 8 but with
> Java 11 I see 1969-dec-31 23:00 which gives -360 which triggers the
> exception.
> 
> The root cause appears to be in the JRE at this point. Whether it is in
> the JRE used to create the WAR or the JRE reading the WAR is TBD.
> 
> I think I am going to have to look at the raw bytes and the zip file
> spec to figure out where the root cause is.

That was fun.

Per the zip spec, the last modified time on those files is:

1969-Dec-31 23:00:00 UTC

i.e. 1 hour before the Epoch at 1970-Jan-01 00:00:00 UTC

It is stored as a signed 32-bit int (F1F0) which is -3600 (zip
timestamps are in seconds since the Epoch).

Java 8 reads this incorrectly as an unsigned int (4294963696) which,
when taken as seconds since Epoch, gives 2016-Feb-07 05:28:16 UTC.

(Incidently the archiver that ships with Ubuntu appears to make the same
error)

Java 11 reads this correctly but Java does not let you set times before
the Epoch so the exception is triggered.

The short version is that the modification times of the files in the WAR
are being set to "1969-Dec-31 23:00:00 UTC" which Java doesn't like.

Tomcat could handle this more gracefully but the end result will be the
same - the WAR isn't going to deploy. I'm not convinced it is worth
doing anything here.

It looks like the fix will be somewhere in the build system used to
create the WAR.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Deploying war, Negative Date exception

[Mark Thomas]

On 12/10/2020 12:49, Mark Thomas wrote:
> On 12/10/2020 12:19, Peter Henderson wrote:
>> Hello fellow tomcat users.
>>
>> My environment.
>> Tomcat: 9.0.39
>> Java: openjdk 11.0.8 2020-07-14
>> OS: Ubuntu 18.04.5 LTS
>>
>> Source code [0]
>>
>> When deploying this war [1], by copying it into the webapps directory,
>> I get this exception. [2]
>> java.lang.IllegalArgumentException: Negative time
>>
>>
>> I only started seeing this exception when I upgraded my projects build tool
>> version
>> from
>> sbt.version=1.3.10
>> to
>> sbt.version=1.4.0
>>
>>
>> Is this a tomcat bug, a build tool bug or most likely something I'm doing
>> wrong?
> 
> Looks like an issue with the dates of files in the WAR.
> 
> If you look at the dates of the files in the WAR nearly all of them are
> in the future (07 Feb 2106, 06:28).
> 
> It looks like something is overflowing but a Java long shouldn't do that.
> 
> Need to dig into exactly what is going on as this looks like it should
> work - even if the WAR contains files created almost a century in the
> future.

Hmm. I see the 2106 date on the file system and with Java 8 but with
Java 11 I see 1969-dec-31 23:00 which gives -360 which triggers the
exception.

The root cause appears to be in the JRE at this point. Whether it is in
the JRE used to create the WAR or the JRE reading the WAR is TBD.

I think I am going to have to look at the raw bytes and the zip file
spec to figure out where the root cause is.

Mark
=

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Deploying war, Negative Date exception

[Martin Grigorov]

Hi Peter,

On Mon, Oct 12, 2020 at 2:20 PM Peter Henderson 
wrote:

> Hello fellow tomcat users.
>
> My environment.
> Tomcat: 9.0.39
> Java: openjdk 11.0.8 2020-07-14
> OS: Ubuntu 18.04.5 LTS
>
> Source code [0]
>
> When deploying this war [1], by copying it into the webapps directory,
> I get this exception. [2]
> java.lang.IllegalArgumentException: Negative time
>
>
> I only started seeing this exception when I upgraded my projects build tool
> version
> from
> sbt.version=1.3.10
> to
> sbt.version=1.4.0
>
>
> Is this a tomcat bug, a build tool bug or most likely something I'm doing
> wrong?
>
> Thanks
> Peter.
>
>
> [0]
> https://github.com/bollinger/NegativeDate
>
> [1]
> https://github.com/bollinger/NegativeDate/blob/master/Negative.war
>
> [2]
> 2020-10-12 11:41:35.932 SEVERE oacs.HostConfig Error deploying web
> application archive [/home/peter/apache-tomcat-9.0.39/webapps/Negative.war]
> java.lang.IllegalStateException: Error starting child
> at
>
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:720)
> at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:690)
> at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:706)
> at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:978)
> at
> org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1848)
> at
>
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
>
> org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
> at
>
> java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
> at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:773)
> at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:427)
> at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1620)
> at
> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:305)
> at
>
> org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
> at
>
> org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1151)
> at
>
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1353)
> at
>
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1357)
> at
>
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1335)
> at
>
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> at
>
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at
>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at
>
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: org.apache.catalina.LifecycleException: Failed to start
> component
>
> [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/Negative]]
> at
>
> org.apache.catalina.util.LifecycleBase.handleSubClassException(LifecycleBase.java:440)
> at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:198)
> at
>
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:717)
> ... 24 more
> Caused by: java.lang.IllegalArgumentException: Negative time
> at java.base/java.io.File.setLastModified(File.java:1441)
>

The negative time comes from jarEntry.getTime() -
https://github.com/apache/tomcat/blob/d6e5d838a9565c0c4bfeca4396e6266d21a80288/java/org/apache/catalina/startup/ExpandWar.java#L167
I guess SBT creates the JAR/WAR so most probably SBT 1.4.0 is to blame here.

Martin


> at org.apache.catalina.startup.ExpandWar.expand(ExpandWar.java:169)
> at
>
> org.apache.catalina.startup.ContextConfig.fixDocBase(ContextConfig.java:820)
> at
>
> org.apache.catalina.startup.ContextConfig.beforeStart(ContextConfig.java:958)
> at
>
> org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:305)
> at
>
> org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
> at
>
> org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423)
> at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:182)
> ... 25 more
>
> --
> Peter Henderson
>

Re: Deploying war, Negative Date exception

[Mark Thomas]

On 12/10/2020 12:19, Peter Henderson wrote:
> Hello fellow tomcat users.
> 
> My environment.
> Tomcat: 9.0.39
> Java: openjdk 11.0.8 2020-07-14
> OS: Ubuntu 18.04.5 LTS
> 
> Source code [0]
> 
> When deploying this war [1], by copying it into the webapps directory,
> I get this exception. [2]
> java.lang.IllegalArgumentException: Negative time
> 
> 
> I only started seeing this exception when I upgraded my projects build tool
> version
> from
> sbt.version=1.3.10
> to
> sbt.version=1.4.0
> 
> 
> Is this a tomcat bug, a build tool bug or most likely something I'm doing
> wrong?

Looks like an issue with the dates of files in the WAR.

If you look at the dates of the files in the WAR nearly all of them are
in the future (07 Feb 2106, 06:28).

It looks like something is overflowing but a Java long shouldn't do that.

Need to dig into exactly what is going on as this looks like it should
work - even if the WAR contains files created almost a century in the
future.

Mark




> 
> Thanks
> Peter.
> 
> 
> [0]
> https://github.com/bollinger/NegativeDate
> 
> [1]
> https://github.com/bollinger/NegativeDate/blob/master/Negative.war
> 
> [2]
> 2020-10-12 11:41:35.932 SEVERE oacs.HostConfig Error deploying web
> application archive [/home/peter/apache-tomcat-9.0.39/webapps/Negative.war]
> java.lang.IllegalStateException: Error starting child
> at
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:720)
> at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:690)
> at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:706)
> at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:978)
> at
> org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1848)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
> at
> java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
> at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:773)
> at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:427)
> at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1620)
> at
> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:305)
> at
> org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
> at
> org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1151)
> at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1353)
> at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1357)
> at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1335)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: org.apache.catalina.LifecycleException: Failed to start
> component
> [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/Negative]]
> at
> org.apache.catalina.util.LifecycleBase.handleSubClassException(LifecycleBase.java:440)
> at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:198)
> at
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:717)
> ... 24 more
> Caused by: java.lang.IllegalArgumentException: Negative time
> at java.base/java.io.File.setLastModified(File.java:1441)
> at org.apache.catalina.startup.ExpandWar.expand(ExpandWar.java:169)
> at
> org.apache.catalina.startup.ContextConfig.fixDocBase(ContextConfig.java:820)
> at
> org.apache.catalina.startup.ContextConfig.beforeStart(ContextConfig.java:958)
> at
> org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:305)
> at
> org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
> at
> org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423)
> at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:182)
> ... 25 more
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: java.lang.OutOfMemoryError: PermGen space when we redeploy same application multiple times

[Manuel Dominguez Sarmiento]

The articles in this page will be helpful:
https://java.jiderhamn.se/category/classloader-leaks/

On 12/10/2020 04:19, Mark Thomas wrote:

On 11/10/2020 02:39, Prabhu Gurunathan wrote:

Hi All,

We have an setup where we are using OpenJDK 1.7 and Tomcat 7.0.100 ,
in CentOs 7 Env . and we have many application deployed in
Tomcat/webapps and the common lib's needed for those apps are kept in
Tomcat/lib directory like log4j , commons-fileupload  ,xerces , Xalan
.. etc

The problem here is When we try to undeploy and deploy same
applications multiple time we are  getting into
java.lang.OutOfMemoryError: PermGen space very quickly . Want to know
is it very generic problem on this deployment model or is this can be
fixed anyway ?

The memory leak could be in any of:
- the application code
- a library the application depends on
- the JVM
- Tomcat

In all cases, it should be possible to fix it. There might even be a
short-term workaround available.

First of all, make sure that the JreMemoryLeakPreventionListener is
enabled in your configuration.

Secondly, make sure that you don't get any warnings about possible
memory leaks in the logs when you reload an application. If you do, fix
the leaks identified.

If you still see issues, the short version is:
- user a profiler
- reload each app in turn until you see more strong references to
   org.apache.catalina.loader.[Parallel]WebappClassLoader instances in
   memory than you have web applications
- find the one where started = false
- trace its GC roots
- that will tell you where the memory leak is
- how it was created might be harder to track down

The long version is in a presentation linked from the Tomcat web site:
https://home.apache.org/~markt/presentations/2010-08-05-Memory-Leaks-JavaOne-60mins.pdf

If you have any questions, you can ask here.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Deploying war, Negative Date exception

[Peter Henderson]

Hello fellow tomcat users.

My environment.
Tomcat: 9.0.39
Java: openjdk 11.0.8 2020-07-14
OS: Ubuntu 18.04.5 LTS

Source code [0]

When deploying this war [1], by copying it into the webapps directory,
I get this exception. [2]
java.lang.IllegalArgumentException: Negative time


I only started seeing this exception when I upgraded my projects build tool
version
from
sbt.version=1.3.10
to
sbt.version=1.4.0


Is this a tomcat bug, a build tool bug or most likely something I'm doing
wrong?

Thanks
Peter.


[0]
https://github.com/bollinger/NegativeDate

[1]
https://github.com/bollinger/NegativeDate/blob/master/Negative.war

[2]
2020-10-12 11:41:35.932 SEVERE oacs.HostConfig Error deploying web
application archive [/home/peter/apache-tomcat-9.0.39/webapps/Negative.war]
java.lang.IllegalStateException: Error starting child
at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:720)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:690)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:706)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:978)
at
org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1848)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at
java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:773)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:427)
at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1620)
at
org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:305)
at
org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1151)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1353)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1357)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1335)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at
java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.catalina.LifecycleException: Failed to start
component
[StandardEngine[Catalina].StandardHost[localhost].StandardContext[/Negative]]
at
org.apache.catalina.util.LifecycleBase.handleSubClassException(LifecycleBase.java:440)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:198)
at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:717)
... 24 more
Caused by: java.lang.IllegalArgumentException: Negative time
at java.base/java.io.File.setLastModified(File.java:1441)
at org.apache.catalina.startup.ExpandWar.expand(ExpandWar.java:169)
at
org.apache.catalina.startup.ContextConfig.fixDocBase(ContextConfig.java:820)
at
org.apache.catalina.startup.ContextConfig.beforeStart(ContextConfig.java:958)
at
org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:305)
at
org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
at
org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:182)
... 25 more

-- 
Peter Henderson

Re: java.lang.OutOfMemoryError: PermGen space when we redeploy same application multiple times

[Prabhu Gurunathan]

Thanks Mark , for your thoughts , will explore more and get back .

On Mon, Oct 12, 2020 at 12:49 PM Mark Thomas  wrote:
>
> On 11/10/2020 02:39, Prabhu Gurunathan wrote:
> > Hi All,
> >
> > We have an setup where we are using OpenJDK 1.7 and Tomcat 7.0.100 ,
> > in CentOs 7 Env . and we have many application deployed in
> > Tomcat/webapps and the common lib's needed for those apps are kept in
> > Tomcat/lib directory like log4j , commons-fileupload  ,xerces , Xalan
> > .. etc
> >
> > The problem here is When we try to undeploy and deploy same
> > applications multiple time we are  getting into
> > java.lang.OutOfMemoryError: PermGen space very quickly . Want to know
> > is it very generic problem on this deployment model or is this can be
> > fixed anyway ?
>
> The memory leak could be in any of:
> - the application code
> - a library the application depends on
> - the JVM
> - Tomcat
>
> In all cases, it should be possible to fix it. There might even be a
> short-term workaround available.
>
> First of all, make sure that the JreMemoryLeakPreventionListener is
> enabled in your configuration.
>
> Secondly, make sure that you don't get any warnings about possible
> memory leaks in the logs when you reload an application. If you do, fix
> the leaks identified.
>
> If you still see issues, the short version is:
> - user a profiler
> - reload each app in turn until you see more strong references to
>   org.apache.catalina.loader.[Parallel]WebappClassLoader instances in
>   memory than you have web applications
> - find the one where started = false
> - trace its GC roots
> - that will tell you where the memory leak is
> - how it was created might be harder to track down
>
> The long version is in a presentation linked from the Tomcat web site:
> https://home.apache.org/~markt/presentations/2010-08-05-Memory-Leaks-JavaOne-60mins.pdf
>
> If you have any questions, you can ask here.
>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

[SECURITY] CVE-2020-13943 Apache Tomcat HTTP/2 Request mix-up

[Mark Thomas]

CVE-2020-13943 Apache Tomcat HTTP/2 Request mix-up

Severity: Moderate

Vendor: The Apache Software Foundation

Versions Affected:
Apache Tomcat 10.0.0-M1 to 10.0.0-M7
Apache Tomcat 9.0.0.M5 to 9.0.37
Apache Tomcat 8.5.1 to 8.5.57

Description:
If an HTTP/2 client exceeded the agreed maximum number of concurrent
streams for a connection (in violation of the HTTP/2 protocol), it was
possible that a subsequent request made on that connection could contain
HTTP headers - including HTTP/2 pseudo headers - from a previous request
rather than the intended headers. This could lead to users seeing
responses for unexpected resources.

Mitigation:
- Upgrade to Apache Tomcat 10.0.0-M8 or later
- Upgrade to Apache Tomcat 9.0.38 or later
- Upgrade to Apache Tomcat 8.5.58 or later

Credit:
This issue was identified by the Apache Tomcat Security Team.

References:
[1] http://tomcat.apache.org/security-10.html
[2] http://tomcat.apache.org/security-9.html
[3] http://tomcat.apache.org/security-8.html

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

[ANN] Apache Tomcat 8.5.59 available

[Mark Thomas]

The Apache Tomcat team announces the immediate availability of Apache
Tomcat 8.5.59.

Apache Tomcat 8 is an open source software implementation of the Java
Servlet, JavaServer Pages, Java Unified Expression Language, Java
WebSocket and Java Authentication Service Provider Interface for
Containers technologies.

Apache Tomcat 8.5.x replaces 8.0.x and includes new features pulled
forward from the 9.0.x branch. The notable changes since 8.5.58 include:

- Refactor the handling of closed HTTP/2 streams to reduce the heap
  usage associated with used streams and to retain information for more
  streams in the priority tree.

- Deprecate the JDBCRealm.

- Ensure that none of the methods on a ServletContext instance always
  fail when running under a SecurityManager. Pull request provided by
  Kyle Stiemann.

Please refer to the change log for the complete list of changes:
http://tomcat.apache.org/tomcat-8.5-doc/changelog.html


Downloads:
http://tomcat.apache.org/download-80.cgi

Migration guides from Apache Tomcat 7.x and 8.0.x:
http://tomcat.apache.org/migration.html

Enjoy!

- The Apache Tomcat team


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

[ANN] Apache Tomcat 9.0.39 available

[Mark Thomas]

The Apache Tomcat team announces the immediate availability of Apache
Tomcat 9.0.39.

Apache Tomcat 9 is an open source software implementation of the Java
Servlet, JavaServer Pages, Java Unified Expression Language, Java
WebSocket and JASPIC technologies.

Apache Tomcat 9.0.39 is a bugfix and feature release. The notable
changes compared to 9.0.38 include:

- Refactor the handling of closed HTTP/2 streams to reduce the heap
  usage associated with used streams and to retain information for more
  streams in the priority tree.

- Allow using the utility executor for annotation scanning. Patch
  provided by Jatin Kamnani.

- Add a bloom filter to speed up archive lookup and improve deployment
  speed of applications with a large number of JARs. Patch provided by
  Jatin Kamnani.

Please refer to the change log for the complete list of changes:
http://tomcat.apache.org/tomcat-9.0-doc/changelog.html


Downloads:
http://tomcat.apache.org/download-90.cgi

Migration guides from Apache Tomcat 7.x and 8.x:
http://tomcat.apache.org/migration.html

Enjoy!

- The Apache Tomcat team


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

[ANN] Apache Tomcat 10.0.0-M9 available

[Mark Thomas]

The Apache Tomcat team announces the immediate availability of Apache
Tomcat 10.0.0-M9.

Apache Tomcat 10 is an open source software implementation of the
Jakarta Servlet, Jakarta Server Pages, Jakarta Expression Language,
Jakarta WebSocket, Jakarta Authentication and Jakarta Annotations
specifications.

Users of Tomcat 10 onwards should be aware that, as a result of the move
from Java EE to Jakarta EE as part of the transfer of Java EE to the
Eclipse Foundation, the primary package for all implemented APIs has
changed from javax.* to jakarta.*. This will almost certainly require
code changes to enable applications to migrate from Tomcat 9 and earlier
to Tomcat 10 and later. A migration tool is under development to aid
this process.

Apache Tomcat 10.0.0-M9 is a milestone release of the 10.0.x
branch and has been made to provide users with early access to the new
features in Apache Tomcat 10.0.x so that they may provide feedback. The
notable changes compared to 10.0.0-M8 include:

- Refactor the handling of closed HTTP/2 streams to reduce the heap
  usage associated with used streams and to retain information for more
  streams in the priority tree.

- Allow using the utility executor for annotation scanning. Patch
  provided by Jatin Kamnani.

- Add a bloom filter to speed up archive lookup and improve deployment
  speed of applications with a large number of JARs. Patch provided by
  Jatin Kamnani.

Please refer to the change log for the complete list of changes:
http://tomcat.apache.org/tomcat-10.0-doc/changelog.html

Downloads:
http://tomcat.apache.org/download-10.cgi

Migration guides from Apache Tomcat 7.0.x, 8.5.x and 9.0.x:
http://tomcat.apache.org/migration.html

Enjoy!

- The Apache Tomcat team

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: java.lang.OutOfMemoryError: PermGen space when we redeploy same application multiple times

[Mark Thomas]

On 11/10/2020 02:39, Prabhu Gurunathan wrote:
> Hi All,
> 
> We have an setup where we are using OpenJDK 1.7 and Tomcat 7.0.100 ,
> in CentOs 7 Env . and we have many application deployed in
> Tomcat/webapps and the common lib's needed for those apps are kept in
> Tomcat/lib directory like log4j , commons-fileupload  ,xerces , Xalan
> .. etc
> 
> The problem here is When we try to undeploy and deploy same
> applications multiple time we are  getting into
> java.lang.OutOfMemoryError: PermGen space very quickly . Want to know
> is it very generic problem on this deployment model or is this can be
> fixed anyway ?

The memory leak could be in any of:
- the application code
- a library the application depends on
- the JVM
- Tomcat

In all cases, it should be possible to fix it. There might even be a
short-term workaround available.

First of all, make sure that the JreMemoryLeakPreventionListener is
enabled in your configuration.

Secondly, make sure that you don't get any warnings about possible
memory leaks in the logs when you reload an application. If you do, fix
the leaks identified.

If you still see issues, the short version is:
- user a profiler
- reload each app in turn until you see more strong references to
  org.apache.catalina.loader.[Parallel]WebappClassLoader instances in
  memory than you have web applications
- find the one where started = false
- trace its GC roots
- that will tell you where the memory leak is
- how it was created might be harder to track down

The long version is in a presentation linked from the Tomcat web site:
https://home.apache.org/~markt/presentations/2010-08-05-Memory-Leaks-JavaOne-60mins.pdf

If you have any questions, you can ask here.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)

[Arshiya Shariff]

Hi Mark , 

The issue is reproduced with version 9.0.39 as well. Max threads in Tomcat is 
200.

Please find the case:
Client:JMeter 5.2.1 (With http2 plugin)
TPS: around 20
No of users from JMeter : 700
Message payload size: 6 KB to 34 KB
Loop: Infinite 
We let the loop run infinitely and see the java.lang.StackOverflowError trace 
printed multiple times in the log within few minutes of starting the test.

Please help us with this . What is the impact of StackOverflowError ?

Thanks and Regards
Arshiya Shariff

-Original Message-
From: Mark Thomas  
Sent: Friday, October 9, 2020 5:31 PM
To: users@tomcat.apache.org
Subject: Re: HTTP2: memory filled up fast on increasing the connections to 
1000/2000 (Embedded tomcat 9.0.38)

On 09/10/2020 12:32, Arshiya Shariff wrote:
> Hi,
> 
> Mark , with the test runs that I performed over clean 9.0.x branch I was not 
> able to reproduce this.

Good. But I'd really like to understand why...

> But with 9.0.38 and the jars built from 9.0.x with hash: 
> c8ec2d4cde3a31b0e9df9a30e7915d77ba725545  , with 700 or 1000 users 
> (connections) and on sending 1000 Requests per second (or even lesser) , 
> payload of 16K  from JMeter I can see that this Exception occurs within few 
> minutes of starting the test . The maxThreads configured in tomcat is 200 .
> 
> How often do you see these errors in your test run?
> Randomly, at times 2 or 3 such traces.

OK. Definitely a timing issue then.

> Do you have the other end of that stack trace?
> It is only the two lines that is recursively printed till the end about  ~500 
> times in one trace  :
> at 
> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
> at 
> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandl
> er.completed(SocketWrapperBase.java:1100)

Doesn't tell me much unfortunately.

> I see the trace starting with :
> Exception in thread "http-nio-x.y.z-1090-exec-107" 
> java.lang.StackOverflowError 
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:446)
> at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174)
> at 
> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468)
> at 
> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandl
> er.completed(SocketWrapperBase.java:1100)
> 
>   (OR)
> 
> Exception in thread "http-nio-x.y.z-1090-exec-87" java.lang.StackOverflowError
> at sun.nio.ch.IOVecWrapper.get(IOVecWrapper.java:96)
> at sun.nio.ch.IOUtil.read(IOUtil.java:240)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:440)
> at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174)
> at 
> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468)
> at 
> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
> at 
> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
> at 
> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
> .
> .
> .
> .
> at 
> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
> at 
> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
> at 
> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
> at 
> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandl
> er.completed(SocketWrapperBase.java:1100)
> 
> Is there anything that was fixed around this in latest 9.0.x branch ?

Not obviously. I've reviewed every commit since c8ec2d4c. There is nothing that 
directly works with the I/O. There is 1e97ab2 which fixes a relatively recent 
regression in the HTTP/2 code. I guess it is possible (but it seems a bit of a 
stretch) that that bug is triggering an issue in JMeter which in turn is 
sending invalid HTTP/2 packets.

I think at this point, given the relatively small number of commits between 
c8ec2d4c and HEAD, the most useful thing you could do is run a binary search to 
find out at which commit the issue is fixed. If we know which commit to look at 
that should help track down the root cause.

Mark

> 
> Thanks and Regards
> Arshiya Shariff
> 
> -Original Message-
> From: Mark Thomas 
> Sent: Monday, October 5, 2020 9:52 PM
> To: users@tomcat.apache.org
> Subject: Re: HTTP2: memory filled up fast on increasing the 
> connections to 1000/2000 (Embedded tomcat 9.0.38)
> 
> On 05/10/2020 10:56, Arshiya Shariff wrote:
>> Hi All,
>>
>> Thank you so much Mark . 
>> We tested the jars built from latest 9.0.x  with 2000 / 5000 users
>>