Re: Cannot upgrade past Tomcat 6 due to massive memory increase

[Mark Thomas]

On 18/02/2016 22:57, Ty wrote:
> I've re-run my tests using 8.0.x trunk (8.0.33-dev) using the same
> procedure from the first message of this thread, and summarized the results
> below.  Heap usage is vastly improved both with and without JAR scanning.
> Heap size and startup time are both reasonable and it's now feasible to
> re-visit our upgrade plan, so we will be putting 8.0.33 into our next cycle
> (once it's officially released).
> 
> Thanks again for looking into this, and wow, addressing it so quickly.

Great. That is good news.

I'm not finished with this. There were a number of things in the heap
dump that looked to be worthy of further consideration. I hope to get to
them in time for 8.0.33 but regardless of whether I do, looking at this
some more is on my TODO list.

Mark


> 
> Cheers,
> Ty
> 
> 
> Case 1 (baseline):
> 
> +-+--+---+
> | version | startup time | heap usage after major GC |
> +-+--+---+
> | tomcat6 | 36,711ms | 21,163,288|
> | tomcat7 | 104,517ms| 489,992,264   |
> | tomcat8 | 156,094ms| 1,010,512,568 |
> | tomcat8-dev | 139,858ms| 218,976,528   |
> +-+--+---+
> 
> 
> Case 2 (Tomcat 7 and 8 with “jarsToSkip=*”)
> 
> +-+--+---+
> | version | startup time | heap usage after major GC |
> +-+--+---+
> | tomcat6 | 36,711ms | 21,163,288|
> | tomcat7 | 38,979ms | 72,359,840|
> | tomcat8 | 52,040ms | 633,682,336   |
> | tomcat8-dev | 66,662ms | 97,365,048|
> +-+--+---+
> 
> 
> 
> 
> On Thu, Feb 18, 2016 at 10:12 AM Ty  wrote:
> 
>>> Are you able to build Tomcat 8.0.x from source to test locally or would
>>> you like me to provide you with a test build?
>>
>> Looks like building is pretty straightforward.  I'll dive in and post back
>> with my testing results.
>>
>>
>> On Thu, Feb 18, 2016 at 9:11 AM Mark Thomas  wrote:
>>
>>> Another update.
>>>
>>> Before any fixes, my tests showed a heap of ~1GB with ~30% used by
>>> WebappClassLoader and 50% by JarResourceSet
>>>
>>> The WebappClassLoader issue was triggered by JAR scanning. Disabling /
>>> limiting JAR scanning will reduce the impact of this issue.
>>> The issue has been fixed in 9.0.x and 8.0.x and will be included in
>>> 9.0.0.M4 and 8.0.33 onwards.
>>>
>>> The JarResourceSet issue was hard coded. There is no way to work around
>>> it. This has also been fixed in 9.0.x and 8.0.x and will be included in
>>> 9.0.0.M4 and 8.0.33 onwards.
>>>
>>> Regressions are possible but unlikely. All the unit tests pass and I
>>> have been using Atlassian Jira as a test app to check for changes in
>>> start time and as a smoke test. There is a small increase in start time
>>> with fix for the JarResourceSet issue (~5%) but the reduction in memory
>>> usage more than makes up for it.
>>>
>>> Are you able to build Tomcat 8.0.x from source to test locally or would
>>> you like me to provide you with a test build?
>>>
>>> Looking at the heap dump, there are still a few areas where it may be
>>> possible to reclaim more memory. This is on my TODO list and I hope to
>>> get to it before the next releases.
>>>
>>> Mark
>>>
>>> -
>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>>
>>>
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Cannot upgrade past Tomcat 6 due to massive memory increase

[Ty]

I've re-run my tests using 8.0.x trunk (8.0.33-dev) using the same
procedure from the first message of this thread, and summarized the results
below.  Heap usage is vastly improved both with and without JAR scanning.
Heap size and startup time are both reasonable and it's now feasible to
re-visit our upgrade plan, so we will be putting 8.0.33 into our next cycle
(once it's officially released).

Thanks again for looking into this, and wow, addressing it so quickly.

Cheers,
Ty


Case 1 (baseline):

+-+--+---+
| version | startup time | heap usage after major GC |
+-+--+---+
| tomcat6 | 36,711ms | 21,163,288|
| tomcat7 | 104,517ms| 489,992,264   |
| tomcat8 | 156,094ms| 1,010,512,568 |
| tomcat8-dev | 139,858ms| 218,976,528   |
+-+--+---+


Case 2 (Tomcat 7 and 8 with “jarsToSkip=*”)

+-+--+---+
| version | startup time | heap usage after major GC |
+-+--+---+
| tomcat6 | 36,711ms | 21,163,288|
| tomcat7 | 38,979ms | 72,359,840|
| tomcat8 | 52,040ms | 633,682,336   |
| tomcat8-dev | 66,662ms | 97,365,048|
+-+--+---+




On Thu, Feb 18, 2016 at 10:12 AM Ty  wrote:

> > Are you able to build Tomcat 8.0.x from source to test locally or would
> > you like me to provide you with a test build?
>
> Looks like building is pretty straightforward.  I'll dive in and post back
> with my testing results.
>
>
> On Thu, Feb 18, 2016 at 9:11 AM Mark Thomas  wrote:
>
>> Another update.
>>
>> Before any fixes, my tests showed a heap of ~1GB with ~30% used by
>> WebappClassLoader and 50% by JarResourceSet
>>
>> The WebappClassLoader issue was triggered by JAR scanning. Disabling /
>> limiting JAR scanning will reduce the impact of this issue.
>> The issue has been fixed in 9.0.x and 8.0.x and will be included in
>> 9.0.0.M4 and 8.0.33 onwards.
>>
>> The JarResourceSet issue was hard coded. There is no way to work around
>> it. This has also been fixed in 9.0.x and 8.0.x and will be included in
>> 9.0.0.M4 and 8.0.33 onwards.
>>
>> Regressions are possible but unlikely. All the unit tests pass and I
>> have been using Atlassian Jira as a test app to check for changes in
>> start time and as a smoke test. There is a small increase in start time
>> with fix for the JarResourceSet issue (~5%) but the reduction in memory
>> usage more than makes up for it.
>>
>> Are you able to build Tomcat 8.0.x from source to test locally or would
>> you like me to provide you with a test build?
>>
>> Looking at the heap dump, there are still a few areas where it may be
>> possible to reclaim more memory. This is on my TODO list and I hope to
>> get to it before the next releases.
>>
>> Mark
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>>

Re: Cannot upgrade past Tomcat 6 due to massive memory increase

[Ty]

> Are you able to build Tomcat 8.0.x from source to test locally or would
> you like me to provide you with a test build?

Looks like building is pretty straightforward.  I'll dive in and post back
with my testing results.

On Thu, Feb 18, 2016 at 9:11 AM Mark Thomas  wrote:

> Another update.
>
> Before any fixes, my tests showed a heap of ~1GB with ~30% used by
> WebappClassLoader and 50% by JarResourceSet
>
> The WebappClassLoader issue was triggered by JAR scanning. Disabling /
> limiting JAR scanning will reduce the impact of this issue.
> The issue has been fixed in 9.0.x and 8.0.x and will be included in
> 9.0.0.M4 and 8.0.33 onwards.
>
> The JarResourceSet issue was hard coded. There is no way to work around
> it. This has also been fixed in 9.0.x and 8.0.x and will be included in
> 9.0.0.M4 and 8.0.33 onwards.
>
> Regressions are possible but unlikely. All the unit tests pass and I
> have been using Atlassian Jira as a test app to check for changes in
> start time and as a smoke test. There is a small increase in start time
> with fix for the JarResourceSet issue (~5%) but the reduction in memory
> usage more than makes up for it.
>
> Are you able to build Tomcat 8.0.x from source to test locally or would
> you like me to provide you with a test build?
>
> Looking at the heap dump, there are still a few areas where it may be
> possible to reclaim more memory. This is on my TODO list and I hope to
> get to it before the next releases.
>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Cannot upgrade past Tomcat 6 due to massive memory increase

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Mark,

On 2/18/16 10:11 AM, Mark Thomas wrote:
> Another update.
> 
> Before any fixes, my tests showed a heap of ~1GB with ~30% used by 
> WebappClassLoader and 50% by JarResourceSet
> 
> The WebappClassLoader issue was triggered by JAR scanning.
> Disabling / limiting JAR scanning will reduce the impact of this
> issue. The issue has been fixed in 9.0.x and 8.0.x and will be
> included in 9.0.0.M4 and 8.0.33 onwards.
> 
> The JarResourceSet issue was hard coded. There is no way to work
> around it. This has also been fixed in 9.0.x and 8.0.x and will be
> included in 9.0.0.M4 and 8.0.33 onwards.
> 
> Regressions are possible but unlikely. All the unit tests pass and
> I have been using Atlassian Jira as a test app to check for changes
> in start time and as a smoke test. There is a small increase in
> start time with fix for the JarResourceSet issue (~5%) but the
> reduction in memory usage more than makes up for it.

Nice. Thanks for diving into this one.

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlbF4OIACgkQ9CaO5/Lv0PBjTACcCcDphK4esKtoC92i9p5iRu+R
fqMAn0GmGulhAwirSyDRcqIK6lFMO5Pf
=BKY6
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Cannot upgrade past Tomcat 6 due to massive memory increase

[Mark Thomas]

Another update.

Before any fixes, my tests showed a heap of ~1GB with ~30% used by
WebappClassLoader and 50% by JarResourceSet

The WebappClassLoader issue was triggered by JAR scanning. Disabling /
limiting JAR scanning will reduce the impact of this issue.
The issue has been fixed in 9.0.x and 8.0.x and will be included in
9.0.0.M4 and 8.0.33 onwards.

The JarResourceSet issue was hard coded. There is no way to work around
it. This has also been fixed in 9.0.x and 8.0.x and will be included in
9.0.0.M4 and 8.0.33 onwards.

Regressions are possible but unlikely. All the unit tests pass and I
have been using Atlassian Jira as a test app to check for changes in
start time and as a smoke test. There is a small increase in start time
with fix for the JarResourceSet issue (~5%) but the reduction in memory
usage more than makes up for it.

Are you able to build Tomcat 8.0.x from source to test locally or would
you like me to provide you with a test build?

Looking at the heap dump, there are still a few areas where it may be
possible to reclaim more memory. This is on my TODO list and I hope to
get to it before the next releases.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Cannot upgrade past Tomcat 6 due to massive memory increase

[Mark Thomas]

On 18/02/2016 00:26, Ty wrote:
>>> For Ty's purposes, simply avoid JAR scanning entirely should eliminate
>>> the heap issues, right? Moving from Tomcat 6 means that JAR scanning
>>> is not required for correct operation of their application.
> 
>> Yes. http://wiki.apache.org/tomcat/HowTo/FasterStartUp has the details
>> of the configuration required.
> 
> I might be missing something, if so please correct me, but it seems that
> there's a huge difference with Tomcat 8, even when JAR scanning is taken
> out of the picture (jarsToSkip=*).
> 
> For my test case, Tomcat 8 has a heap size of ~1GB with JAR scanning
> enabled, and ~600MB with jarsToSkip=*.  That's a 40% improvement by
> disabling JAR scanning, sure, but for comparison, same test case, Tomcat
> 6's heap is ~20MB while Tomcat 7's (with jarsToSkip=*) is ~70MB.  We're
> still back to a 30x increase in heap utilization from Tomcat 6 to Tomcat 8,
> and a 10x increase from Tomcat 7 to Tomcat 8, without having JAR scanning
> in the mix.

Sorry. Not enough sleep when I wrote that. The WebappClassLoader issue
is addressed by disabling JAR scanning but the JarResourceSet issue is not.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Cannot upgrade past Tomcat 6 due to massive memory increase

[Ty]

>> For Ty's purposes, simply avoid JAR scanning entirely should eliminate
>> the heap issues, right? Moving from Tomcat 6 means that JAR scanning
>> is not required for correct operation of their application.

> Yes. http://wiki.apache.org/tomcat/HowTo/FasterStartUp has the details
> of the configuration required.

I might be missing something, if so please correct me, but it seems that
there's a huge difference with Tomcat 8, even when JAR scanning is taken
out of the picture (jarsToSkip=*).

For my test case, Tomcat 8 has a heap size of ~1GB with JAR scanning
enabled, and ~600MB with jarsToSkip=*.  That's a 40% improvement by
disabling JAR scanning, sure, but for comparison, same test case, Tomcat
6's heap is ~20MB while Tomcat 7's (with jarsToSkip=*) is ~70MB.  We're
still back to a 30x increase in heap utilization from Tomcat 6 to Tomcat 8,
and a 10x increase from Tomcat 7 to Tomcat 8, without having JAR scanning
in the mix.


On Wed, Feb 17, 2016 at 6:14 PM Mark Thomas  wrote:

> On 18/02/2016 00:05, Christopher Schultz wrote:
> > Mark,
> >
> > On 2/17/16 5:04 PM, Mark Thomas wrote:
> >> On 17/02/2016 11:11, Mark Thomas wrote:
> >>> On 16/02/2016 22:07, Ty wrote:
> > Exactly which versions were you using when you ran your
> > tests?
> 
>  According to my notes:  7.0.59, which should've been the most
>  recent version available at the time (about this time last
>  year).
> 
>  To clarify:  that was the version we did a heap dump analysis
>  of, and noted the memory increase related to Manifest objects.
>  The version used for the test procedure in this thread's
>  original message -- last performed last week -- was 7.0.67 and
>  we did not do a heap analysis of it (only noting that at a
>  glance it resulted in approximately the same amount of heap
>  increase correlating with JAR scanning as we saw with last
>  year's version).
> >>>
> >>> Thanks for the version info.
> >>>
> >>> I've started looking at this. The bulk of the memory is used by
> >>> JarResourceSet. It is caching a JarFileEntry object for each
> >>> class in the JAR.
> >>>
> >>> I wrote this code but I don't recall why those entries were
> >>> retained. I need to look back through svn to figure out what
> >>> problem it was solving and then see if I can find a better
> >>> solution.
> >
> >> Quick update.
> >
> >> There are two separate issues, both triggered by JAR scanning.
> >
> >> The JarResourceSet issue above accounts for ~50% of a 1GB heap.
> >> There is also a WebappClassLoader issue that accounts for ~30% of
> >> the same heap.
> >
> >> The WebappClassLoader issue is the easier to fix. I have fixed this
> >> in trunk and am just waiting for the tests to confirm all is well.
> >> I plan to back-port this to 8.0.x for the next release.
> >
> >> The JarResourceSet is going to be trickier. The code is the way it
> >> is partly for performance, partly to avoid file locking on Windows.
> >> It is going to take a little longer to put together a fix for this
> >> one.
> >
> > For Ty's purposes, simply avoid JAR scanning entirely should eliminate
> > the heap issues, right? Moving from Tomcat 6 means that JAR scanning
> > is not required for correct operation of their application.
>
> Yes. http://wiki.apache.org/tomcat/HowTo/FasterStartUp has the details
> of the configuration required.
>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Cannot upgrade past Tomcat 6 due to massive memory increase

[Mark Thomas]

On 18/02/2016 00:05, Christopher Schultz wrote:
> Mark,
> 
> On 2/17/16 5:04 PM, Mark Thomas wrote:
>> On 17/02/2016 11:11, Mark Thomas wrote:
>>> On 16/02/2016 22:07, Ty wrote:
> Exactly which versions were you using when you ran your
> tests?

 According to my notes:  7.0.59, which should've been the most
 recent version available at the time (about this time last
 year).

 To clarify:  that was the version we did a heap dump analysis
 of, and noted the memory increase related to Manifest objects.
 The version used for the test procedure in this thread's
 original message -- last performed last week -- was 7.0.67 and
 we did not do a heap analysis of it (only noting that at a
 glance it resulted in approximately the same amount of heap 
 increase correlating with JAR scanning as we saw with last
 year's version).
>>>
>>> Thanks for the version info.
>>>
>>> I've started looking at this. The bulk of the memory is used by 
>>> JarResourceSet. It is caching a JarFileEntry object for each
>>> class in the JAR.
>>>
>>> I wrote this code but I don't recall why those entries were
>>> retained. I need to look back through svn to figure out what
>>> problem it was solving and then see if I can find a better
>>> solution.
> 
>> Quick update.
> 
>> There are two separate issues, both triggered by JAR scanning.
> 
>> The JarResourceSet issue above accounts for ~50% of a 1GB heap. 
>> There is also a WebappClassLoader issue that accounts for ~30% of
>> the same heap.
> 
>> The WebappClassLoader issue is the easier to fix. I have fixed this
>> in trunk and am just waiting for the tests to confirm all is well.
>> I plan to back-port this to 8.0.x for the next release.
> 
>> The JarResourceSet is going to be trickier. The code is the way it
>> is partly for performance, partly to avoid file locking on Windows.
>> It is going to take a little longer to put together a fix for this
>> one.
> 
> For Ty's purposes, simply avoid JAR scanning entirely should eliminate
> the heap issues, right? Moving from Tomcat 6 means that JAR scanning
> is not required for correct operation of their application.

Yes. http://wiki.apache.org/tomcat/HowTo/FasterStartUp has the details
of the configuration required.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Cannot upgrade past Tomcat 6 due to massive memory increase

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Ty,

On 2/16/16 4:51 PM, Ty wrote:
>> if Tomcat had the option to ignore META-INF/resources in JAR
>> files
> 
> Perhaps interesting:  of the 27 JAR files in my dummy application,
> not a single one has a META-INF/resources.

Obviously, markt is following the real story, here. I was just
speculating based upon your comments and Mark's (previously untested)
thoughts on the matter.

It still seems that getting to work on disabling JAR scanning is going
to help you immensely.

- -chris

> On Tue, Feb 16, 2016 at 3:40 PM Christopher Schultz < 
> ch...@christopherschultz.net> wrote:
> 
> Ty,
> 
> On 2/16/16 3:11 PM, Ty wrote:
 George, thanks for sharing your experience, good to know I'm
 not the first down this path.  I'd be interested to know if
 you've tested at all with Tomcat 8 and what your findings
 were, if so. Based on my experience I'd expect you to be in a
 similar boat to us.
 
 Mark, thanks for the reply.  I've uploaded my "dummy" webapp
 here:
 
 http://filebin.ca/2XCPi8oAdy84/test.war
 
 As for the JAR scanning / startup times, with Tomcat 8's
 jarsToScan setting I'm confident we'd be able to manage that,
 and am happy to leapfrog over Tomcat 7. It's the memory
 utilization in Tomcat 8 specifically, independent of JAR
 scanning, that's the show-stopper for us.  Let me know if I
 can provide any other info, and thanks again.
> 
> I'm not sure if others will agree, but if Tomcat had the option to 
> ignore META-INF/resources in JAR files, you could probably save
> some memory. Since you are currently using Tomcat 6, this setting
> wouldn't affect you at all, since Tomcat 6 didn't expose those
> files in the way Mark describes. On the other hand, since it's a
> spec-requirement, there may be some pushback amongst the team here
> as to whether or not we should implement such a workaround.
> 
> We have lots of other settings that can technically be used to
> violate spec-defined behavior, so it's not like there isn't
> precedent for such things.
> 
> -chris
>> 
>> -
>>
>> 
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>> 
>> 
> 
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlbFCyEACgkQ9CaO5/Lv0PC4BQCfa6BD9xNHdnb9SOpajipk++wq
0AYAoIu1/kKrK9/ol/YnJxGKEfMcqsgO
=dtK0
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Cannot upgrade past Tomcat 6 due to massive memory increase

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Mark,

On 2/17/16 5:04 PM, Mark Thomas wrote:
> On 17/02/2016 11:11, Mark Thomas wrote:
>> On 16/02/2016 22:07, Ty wrote:
 Exactly which versions were you using when you ran your
 tests?
>>> 
>>> According to my notes:  7.0.59, which should've been the most
>>> recent version available at the time (about this time last
>>> year).
>>> 
>>> To clarify:  that was the version we did a heap dump analysis
>>> of, and noted the memory increase related to Manifest objects.
>>> The version used for the test procedure in this thread's
>>> original message -- last performed last week -- was 7.0.67 and
>>> we did not do a heap analysis of it (only noting that at a
>>> glance it resulted in approximately the same amount of heap 
>>> increase correlating with JAR scanning as we saw with last
>>> year's version).
>> 
>> Thanks for the version info.
>> 
>> I've started looking at this. The bulk of the memory is used by 
>> JarResourceSet. It is caching a JarFileEntry object for each
>> class in the JAR.
>> 
>> I wrote this code but I don't recall why those entries were
>> retained. I need to look back through svn to figure out what
>> problem it was solving and then see if I can find a better
>> solution.
> 
> Quick update.
> 
> There are two separate issues, both triggered by JAR scanning.
> 
> The JarResourceSet issue above accounts for ~50% of a 1GB heap. 
> There is also a WebappClassLoader issue that accounts for ~30% of
> the same heap.
> 
> The WebappClassLoader issue is the easier to fix. I have fixed this
> in trunk and am just waiting for the tests to confirm all is well.
> I plan to back-port this to 8.0.x for the next release.
> 
> The JarResourceSet is going to be trickier. The code is the way it
> is partly for performance, partly to avoid file locking on Windows.
> It is going to take a little longer to put together a fix for this
> one.

For Ty's purposes, simply avoid JAR scanning entirely should eliminate
the heap issues, right? Moving from Tomcat 6 means that JAR scanning
is not required for correct operation of their application.

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlbFCtMACgkQ9CaO5/Lv0PAZ6QCdHDVAEamLknKC6//lBxhfY7cX
/qQAn0FqyDoE+92/ivKZbxgX9PtEjzeD
=9sLw
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Cannot upgrade past Tomcat 6 due to massive memory increase

[Mark Thomas]

On 17/02/2016 11:11, Mark Thomas wrote:
> On 16/02/2016 22:07, Ty wrote:
>>> Exactly which versions were you using when you ran your tests?
>>
>> According to my notes:  7.0.59, which should've been the most recent
>> version available at the time (about this time last year).
>>
>> To clarify:  that was the version we did a heap dump analysis of, and noted
>> the memory increase related to Manifest objects.  The version used for the
>> test procedure in this thread's original message -- last performed last
>> week -- was 7.0.67 and we did not do a heap analysis of it (only noting
>> that at a glance it resulted in approximately the same amount of heap
>> increase correlating with JAR scanning as we saw with last year's version).
> 
> Thanks for the version info.
> 
> I've started looking at this. The bulk of the memory is used by
> JarResourceSet. It is caching a JarFileEntry object for each class in
> the JAR.
> 
> I wrote this code but I don't recall why those entries were retained. I
> need to look back through svn to figure out what problem it was solving
> and then see if I can find a better solution.

Quick update.

There are two separate issues, both triggered by JAR scanning.

The JarResourceSet issue above accounts for ~50% of a 1GB heap.
There is also a WebappClassLoader issue that accounts for ~30% of the
same heap.

The WebappClassLoader issue is the easier to fix. I have fixed this in
trunk and am just waiting for the tests to confirm all is well. I plan
to back-port this to 8.0.x for the next release.

The JarResourceSet is going to be trickier. The code is the way it is
partly for performance, partly to avoid file locking on Windows. It is
going to take a little longer to put together a fix for this one.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Cannot upgrade past Tomcat 6 due to massive memory increase

[Mark Thomas]

On 16/02/2016 22:07, Ty wrote:
>> Exactly which versions were you using when you ran your tests?
> 
> According to my notes:  7.0.59, which should've been the most recent
> version available at the time (about this time last year).
> 
> To clarify:  that was the version we did a heap dump analysis of, and noted
> the memory increase related to Manifest objects.  The version used for the
> test procedure in this thread's original message -- last performed last
> week -- was 7.0.67 and we did not do a heap analysis of it (only noting
> that at a glance it resulted in approximately the same amount of heap
> increase correlating with JAR scanning as we saw with last year's version).

Thanks for the version info.

I've started looking at this. The bulk of the memory is used by
JarResourceSet. It is caching a JarFileEntry object for each class in
the JAR.

I wrote this code but I don't recall why those entries were retained. I
need to look back through svn to figure out what problem it was solving
and then see if I can find a better solution.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Cannot upgrade past Tomcat 6 due to massive memory increase

[Ty]

> Exactly which versions were you using when you ran your tests?

According to my notes:  7.0.59, which should've been the most recent
version available at the time (about this time last year).

To clarify:  that was the version we did a heap dump analysis of, and noted
the memory increase related to Manifest objects.  The version used for the
test procedure in this thread's original message -- last performed last
week -- was 7.0.67 and we did not do a heap analysis of it (only noting
that at a glance it resulted in approximately the same amount of heap
increase correlating with JAR scanning as we saw with last year's version).



On Tue, Feb 16, 2016 at 3:55 PM Mark Thomas  wrote:

> On 16/02/2016 20:14, Ty wrote:
> >> JAR scanning should be transient. What was it that was causing the 10-20
> >> fold increase?
> >
> > From memory (this was a while ago), it was related to
> > java.util.jar.Manifest objects, or what they contained/referenced.  I'd
> be
> > glad to do another heap analysis and summarize it if it would help, but
> > even more glad to just skip Tomcat 7 altogether.
>
> There was an issue related to caching of Manifest instances but that
> should have been fixed. Exactly which versions were you using when you
> ran your tests?
>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Cannot upgrade past Tomcat 6 due to massive memory increase

[Mark Thomas]

On 16/02/2016 20:14, Ty wrote:
>> JAR scanning should be transient. What was it that was causing the 10-20
>> fold increase?
> 
> From memory (this was a while ago), it was related to
> java.util.jar.Manifest objects, or what they contained/referenced.  I'd be
> glad to do another heap analysis and summarize it if it would help, but
> even more glad to just skip Tomcat 7 altogether.

There was an issue related to caching of Manifest instances but that
should have been fixed. Exactly which versions were you using when you
ran your tests?

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Cannot upgrade past Tomcat 6 due to massive memory increase

[Ty]

>  if Tomcat had the option to ignore META-INF/resources in JAR files

Perhaps interesting:  of the 27 JAR files in my dummy application, not a
single one has a META-INF/resources.

On Tue, Feb 16, 2016 at 3:40 PM Christopher Schultz <
ch...@christopherschultz.net> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Ty,
>
> On 2/16/16 3:11 PM, Ty wrote:
> > George, thanks for sharing your experience, good to know I'm not
> > the first down this path.  I'd be interested to know if you've
> > tested at all with Tomcat 8 and what your findings were, if so.
> > Based on my experience I'd expect you to be in a similar boat to
> > us.
> >
> > Mark, thanks for the reply.  I've uploaded my "dummy" webapp here:
> >
> > http://filebin.ca/2XCPi8oAdy84/test.war
> >
> > As for the JAR scanning / startup times, with Tomcat 8's jarsToScan
> > setting I'm confident we'd be able to manage that, and am happy to
> > leapfrog over Tomcat 7. It's the memory utilization in Tomcat 8
> > specifically, independent of JAR scanning, that's the show-stopper
> > for us.  Let me know if I can provide any other info, and thanks
> > again.
>
> I'm not sure if others will agree, but if Tomcat had the option to
> ignore META-INF/resources in JAR files, you could probably save some
> memory. Since you are currently using Tomcat 6, this setting wouldn't
> affect you at all, since Tomcat 6 didn't expose those files in the way
> Mark describes. On the other hand, since it's a spec-requirement,
> there may be some pushback amongst the team here as to whether or not
> we should implement such a workaround.
>
> We have lots of other settings that can technically be used to violate
> spec-defined behavior, so it's not like there isn't precedent for such
> things.
>
> - -chris
> -BEGIN PGP SIGNATURE-
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEARECAAYFAlbDlzAACgkQ9CaO5/Lv0PBWDgCfV3efH9stmGMVPFtch9unNeJl
> Dq4AoJwlx4NbRa9NfHzu10XHLV86FBGf
> =hafd
> -END PGP SIGNATURE-
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Cannot upgrade past Tomcat 6 due to massive memory increase

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Ty,

On 2/16/16 3:11 PM, Ty wrote:
> George, thanks for sharing your experience, good to know I'm not
> the first down this path.  I'd be interested to know if you've
> tested at all with Tomcat 8 and what your findings were, if so.
> Based on my experience I'd expect you to be in a similar boat to
> us.
> 
> Mark, thanks for the reply.  I've uploaded my "dummy" webapp here:
> 
> http://filebin.ca/2XCPi8oAdy84/test.war
> 
> As for the JAR scanning / startup times, with Tomcat 8's jarsToScan
> setting I'm confident we'd be able to manage that, and am happy to
> leapfrog over Tomcat 7. It's the memory utilization in Tomcat 8
> specifically, independent of JAR scanning, that's the show-stopper
> for us.  Let me know if I can provide any other info, and thanks
> again.

I'm not sure if others will agree, but if Tomcat had the option to
ignore META-INF/resources in JAR files, you could probably save some
memory. Since you are currently using Tomcat 6, this setting wouldn't
affect you at all, since Tomcat 6 didn't expose those files in the way
Mark describes. On the other hand, since it's a spec-requirement,
there may be some pushback amongst the team here as to whether or not
we should implement such a workaround.

We have lots of other settings that can technically be used to violate
spec-defined behavior, so it's not like there isn't precedent for such
things.

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlbDlzAACgkQ9CaO5/Lv0PBWDgCfV3efH9stmGMVPFtch9unNeJl
Dq4AoJwlx4NbRa9NfHzu10XHLV86FBGf
=hafd
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Cannot upgrade past Tomcat 6 due to massive memory increase

[George Sexton]

On 2/16/2016 1:11 PM, Ty wrote:

George, thanks for sharing your experience, good to know I'm not the first
down this path.  I'd be interested to know if you've tested at all with
Tomcat 8 and what your findings were, if so.  Based on my experience I'd
expect you to be in a similar boat to us.


I was doing some work for someone, and I thought about upgrading to 
Tomcat 8, but it doesn't offer anything I need. I'm just going to wait 
for Tomcat 9, which will have something compelling for me (SNI support).





Mark, thanks for the reply.  I've uploaded my "dummy" webapp here:

http://filebin.ca/2XCPi8oAdy84/test.war

As for the JAR scanning / startup times, with Tomcat 8's jarsToScan setting
I'm confident we'd be able to manage that, and am happy to leapfrog over
Tomcat 7. It's the memory utilization in Tomcat 8 specifically, independent
of JAR scanning, that's the show-stopper for us.  Let me know if I can
provide any other info, and thanks again.



On Tue, Feb 16, 2016 at 12:58 PM Mark Thomas  wrote:


On 16/02/2016 16:13, Ty wrote:

Summary:

Because of our use case (hundreds of webapps per instance), we cannot
feasibly upgrade our Tomcat 6 containers to Tomcat 7 or Tomcat 8, due to
the massive increase in memory the upgrade causes.

Background:

We’ve been running Tomcat 6 for several years, on perhaps an unusual

scale:

several dozen Tomcat instances, each running up to a *few hundred

webapps*

in a single container.  (Picture a cloud service provider with several
products and hundreds of customers, with one dedicated webapp
per-customer-per-product, resulting in many thousands of web
applications.  It’s
not ideal but it’s how the software was designed).  Typical max heap

sizes

per container are 1GB to 4GB depending on utilization, and max container
startup times are in the neighborhood of 5 to 10 minutes.

· When Tomcat 7 was released, our testing showed a 10-to-20-fold
increase in memory consumption and a 3-to-5-fold increase in startup time
for our test case (100 deployed webapps).  After some digging we

determined

that these increases were related to JAR scanning
(tomcat.util.scan.DefaultJarScanner).

JAR scanning should be transient. What was it that was causing the 10-20
fold increase?


· When Tomcat 8 was released, we noted that it included a
“whitelist” version of the jarsToSkip Property, jarsToScan.  Our hopes

were

high that we could skip from Tomcat 6 to Tomcat 8, but our test case

(again

100 deployed webapps) quickly deflated those hopes:  similar startup

times

but at least a *30-fold increase in memory consumption*, regardless of

how

we configure jarsToScan or jarsToSkip.

· A heap dump analysis showed that the Tomcat 8 memory increase

was

largely due to the size/count of
org.apache.catalina.webresources.JarResourceSet objects.

Those are only going to be created when a JAR is found that contains
META-INF/resources. The spec requires Tomcat to expose those as static
resources and that requires plumbing. Note Tomcat 6 will just ignore
these because the spec is to old for this feature.


We attempted to
reduce this by setting the Context/Resources@cachingAllowed atttribute

to

“false” and also tried to tune the “cacheMaxSize” and

“cacheObjectMaxSize”

attributes.

Those attributes should have no impact on the number or size of the
JarResourceSet instances.


The only effect that came from these changes was a change in
the object that took up all the space:  instead of
org.apache.catalina.webresources.JarResourceSet objects, they are
org.apache.catalina.webresources.StandardResourceSet objects, taking up
roughly the same amount of space.

Strange. Very strange.


Question:

Is there anything else we can adjust to make Tomcat 8’s memory

consumption

closer to that of Tomcat 6’s, for our use case?

It is hard to see exactly what is going on here. The symptoms you
describe and the resulting changes with configuration are not consistent
with how I would expect Tomcat to behave nor can I see how it might
behave in the way described.


If not, we are faced with
either running Tomcat 6 past its EOL, or possibly maintaining a very

large

jarsToSkip blacklist in Tomcat 7 (until its EOL).  It is not economically
feasible for us to increase the physical memory of our servers by 30x so

we

can run Tomcat 8.

Test procedure:


- Create a "dummy" webapp (using default Maven archetype) named

test.war

- Add several popular Maven dependencies such that the total size of

the

JARs in WEB-INF/lib is about 30MB

Can you provide the sample project? It makes sense to ensure we are
working from the same baseline. I wonder if the libraries themselves are
a factor here.


- Download the latest versions of Tomcat 6, 7, and 8.
- Set Java environment variables for a JMX listener and a 3GB max heap
- Make 100 copies of test.war (test1.war, test2.war, …, test100.war)

and

drop them in /webapps.
- Start the container, 

Re: Cannot upgrade past Tomcat 6 due to massive memory increase

[Ty]

George, thanks for sharing your experience, good to know I'm not the first
down this path.  I'd be interested to know if you've tested at all with
Tomcat 8 and what your findings were, if so.  Based on my experience I'd
expect you to be in a similar boat to us.

Mark, thanks for the reply.  I've uploaded my "dummy" webapp here:

http://filebin.ca/2XCPi8oAdy84/test.war

As for the JAR scanning / startup times, with Tomcat 8's jarsToScan setting
I'm confident we'd be able to manage that, and am happy to leapfrog over
Tomcat 7. It's the memory utilization in Tomcat 8 specifically, independent
of JAR scanning, that's the show-stopper for us.  Let me know if I can
provide any other info, and thanks again.



On Tue, Feb 16, 2016 at 12:58 PM Mark Thomas  wrote:

> On 16/02/2016 16:13, Ty wrote:
> > Summary:
> >
> > Because of our use case (hundreds of webapps per instance), we cannot
> > feasibly upgrade our Tomcat 6 containers to Tomcat 7 or Tomcat 8, due to
> > the massive increase in memory the upgrade causes.
> >
> > Background:
> >
> > We’ve been running Tomcat 6 for several years, on perhaps an unusual
> scale:
> > several dozen Tomcat instances, each running up to a *few hundred
> webapps*
> > in a single container.  (Picture a cloud service provider with several
> > products and hundreds of customers, with one dedicated webapp
> > per-customer-per-product, resulting in many thousands of web
> > applications.  It’s
> > not ideal but it’s how the software was designed).  Typical max heap
> sizes
> > per container are 1GB to 4GB depending on utilization, and max container
> > startup times are in the neighborhood of 5 to 10 minutes.
> >
> > · When Tomcat 7 was released, our testing showed a 10-to-20-fold
> > increase in memory consumption and a 3-to-5-fold increase in startup time
> > for our test case (100 deployed webapps).  After some digging we
> determined
> > that these increases were related to JAR scanning
> > (tomcat.util.scan.DefaultJarScanner).
>
> JAR scanning should be transient. What was it that was causing the 10-20
> fold increase?
>
> > · When Tomcat 8 was released, we noted that it included a
> > “whitelist” version of the jarsToSkip Property, jarsToScan.  Our hopes
> were
> > high that we could skip from Tomcat 6 to Tomcat 8, but our test case
> (again
> > 100 deployed webapps) quickly deflated those hopes:  similar startup
> times
> > but at least a *30-fold increase in memory consumption*, regardless of
> how
> > we configure jarsToScan or jarsToSkip.
> >
> > · A heap dump analysis showed that the Tomcat 8 memory increase
> was
> > largely due to the size/count of
> > org.apache.catalina.webresources.JarResourceSet objects.
>
> Those are only going to be created when a JAR is found that contains
> META-INF/resources. The spec requires Tomcat to expose those as static
> resources and that requires plumbing. Note Tomcat 6 will just ignore
> these because the spec is to old for this feature.
>
> > We attempted to
> > reduce this by setting the Context/Resources@cachingAllowed atttribute
> to
> > “false” and also tried to tune the “cacheMaxSize” and
> “cacheObjectMaxSize”
> > attributes.
>
> Those attributes should have no impact on the number or size of the
> JarResourceSet instances.
>
> > The only effect that came from these changes was a change in
> > the object that took up all the space:  instead of
> > org.apache.catalina.webresources.JarResourceSet objects, they are
> > org.apache.catalina.webresources.StandardResourceSet objects, taking up
> > roughly the same amount of space.
>
> Strange. Very strange.
>
> > Question:
> >
> > Is there anything else we can adjust to make Tomcat 8’s memory
> consumption
> > closer to that of Tomcat 6’s, for our use case?
>
> It is hard to see exactly what is going on here. The symptoms you
> describe and the resulting changes with configuration are not consistent
> with how I would expect Tomcat to behave nor can I see how it might
> behave in the way described.
>
> > If not, we are faced with
> > either running Tomcat 6 past its EOL, or possibly maintaining a very
> large
> > jarsToSkip blacklist in Tomcat 7 (until its EOL).  It is not economically
> > feasible for us to increase the physical memory of our servers by 30x so
> we
> > can run Tomcat 8.
> >
> > Test procedure:
> >
> >
> >- Create a "dummy" webapp (using default Maven archetype) named
> test.war
> >- Add several popular Maven dependencies such that the total size of
> the
> >JARs in WEB-INF/lib is about 30MB
>
> Can you provide the sample project? It makes sense to ensure we are
> working from the same baseline. I wonder if the libraries themselves are
> a factor here.
>
> >- Download the latest versions of Tomcat 6, 7, and 8.
> >- Set Java environment variables for a JMX listener and a 3GB max heap
> >- Make 100 copies of test.war (test1.war, test2.war, …, test100.war)
> and
> >drop them in /webapps.
> >- Start 

Re: Cannot upgrade past Tomcat 6 due to massive memory increase

[Ty]

> JAR scanning should be transient. What was it that was causing the 10-20
> fold increase?

>From memory (this was a while ago), it was related to
java.util.jar.Manifest objects, or what they contained/referenced.  I'd be
glad to do another heap analysis and summarize it if it would help, but
even more glad to just skip Tomcat 7 altogether.

On Tue, Feb 16, 2016 at 2:11 PM Ty  wrote:

> George, thanks for sharing your experience, good to know I'm not the first
> down this path.  I'd be interested to know if you've tested at all with
> Tomcat 8 and what your findings were, if so.  Based on my experience I'd
> expect you to be in a similar boat to us.
>
> Mark, thanks for the reply.  I've uploaded my "dummy" webapp here:
>
> http://filebin.ca/2XCPi8oAdy84/test.war
>
> As for the JAR scanning / startup times, with Tomcat 8's jarsToScan
> setting I'm confident we'd be able to manage that, and am happy to leapfrog
> over Tomcat 7. It's the memory utilization in Tomcat 8 specifically,
> independent of JAR scanning, that's the show-stopper for us.  Let me know
> if I can provide any other info, and thanks again.
>
>
>
> On Tue, Feb 16, 2016 at 12:58 PM Mark Thomas  wrote:
>
>> On 16/02/2016 16:13, Ty wrote:
>> > Summary:
>> >
>> > Because of our use case (hundreds of webapps per instance), we cannot
>> > feasibly upgrade our Tomcat 6 containers to Tomcat 7 or Tomcat 8, due to
>> > the massive increase in memory the upgrade causes.
>> >
>> > Background:
>> >
>> > We’ve been running Tomcat 6 for several years, on perhaps an unusual
>> scale:
>> > several dozen Tomcat instances, each running up to a *few hundred
>> webapps*
>> > in a single container.  (Picture a cloud service provider with several
>> > products and hundreds of customers, with one dedicated webapp
>> > per-customer-per-product, resulting in many thousands of web
>> > applications.  It’s
>> > not ideal but it’s how the software was designed).  Typical max heap
>> sizes
>> > per container are 1GB to 4GB depending on utilization, and max container
>> > startup times are in the neighborhood of 5 to 10 minutes.
>> >
>> > · When Tomcat 7 was released, our testing showed a 10-to-20-fold
>> > increase in memory consumption and a 3-to-5-fold increase in startup
>> time
>> > for our test case (100 deployed webapps).  After some digging we
>> determined
>> > that these increases were related to JAR scanning
>> > (tomcat.util.scan.DefaultJarScanner).
>>
>> JAR scanning should be transient. What was it that was causing the 10-20
>> fold increase?
>>
>> > · When Tomcat 8 was released, we noted that it included a
>> > “whitelist” version of the jarsToSkip Property, jarsToScan.  Our hopes
>> were
>> > high that we could skip from Tomcat 6 to Tomcat 8, but our test case
>> (again
>> > 100 deployed webapps) quickly deflated those hopes:  similar startup
>> times
>> > but at least a *30-fold increase in memory consumption*, regardless of
>> how
>> > we configure jarsToScan or jarsToSkip.
>> >
>> > · A heap dump analysis showed that the Tomcat 8 memory increase
>> was
>> > largely due to the size/count of
>> > org.apache.catalina.webresources.JarResourceSet objects.
>>
>> Those are only going to be created when a JAR is found that contains
>> META-INF/resources. The spec requires Tomcat to expose those as static
>> resources and that requires plumbing. Note Tomcat 6 will just ignore
>> these because the spec is to old for this feature.
>>
>> > We attempted to
>> > reduce this by setting the Context/Resources@cachingAllowed atttribute
>> to
>> > “false” and also tried to tune the “cacheMaxSize” and
>> “cacheObjectMaxSize”
>> > attributes.
>>
>> Those attributes should have no impact on the number or size of the
>> JarResourceSet instances.
>>
>> > The only effect that came from these changes was a change in
>> > the object that took up all the space:  instead of
>> > org.apache.catalina.webresources.JarResourceSet objects, they are
>> > org.apache.catalina.webresources.StandardResourceSet objects, taking up
>> > roughly the same amount of space.
>>
>> Strange. Very strange.
>>
>> > Question:
>> >
>> > Is there anything else we can adjust to make Tomcat 8’s memory
>> consumption
>> > closer to that of Tomcat 6’s, for our use case?
>>
>> It is hard to see exactly what is going on here. The symptoms you
>> describe and the resulting changes with configuration are not consistent
>> with how I would expect Tomcat to behave nor can I see how it might
>> behave in the way described.
>>
>> > If not, we are faced with
>> > either running Tomcat 6 past its EOL, or possibly maintaining a very
>> large
>> > jarsToSkip blacklist in Tomcat 7 (until its EOL).  It is not
>> economically
>> > feasible for us to increase the physical memory of our servers by 30x
>> so we
>> > can run Tomcat 8.
>> >
>> > Test procedure:
>> >
>> >
>> >- Create a "dummy" webapp (using default Maven archetype) named
>> test.war
>> >- Add 

Re: Cannot upgrade past Tomcat 6 due to massive memory increase

[Mark Thomas]

On 16/02/2016 16:13, Ty wrote:
> Summary:
> 
> Because of our use case (hundreds of webapps per instance), we cannot
> feasibly upgrade our Tomcat 6 containers to Tomcat 7 or Tomcat 8, due to
> the massive increase in memory the upgrade causes.
> 
> Background:
> 
> We’ve been running Tomcat 6 for several years, on perhaps an unusual scale:
> several dozen Tomcat instances, each running up to a *few hundred webapps*
> in a single container.  (Picture a cloud service provider with several
> products and hundreds of customers, with one dedicated webapp
> per-customer-per-product, resulting in many thousands of web
> applications.  It’s
> not ideal but it’s how the software was designed).  Typical max heap sizes
> per container are 1GB to 4GB depending on utilization, and max container
> startup times are in the neighborhood of 5 to 10 minutes.
> 
> · When Tomcat 7 was released, our testing showed a 10-to-20-fold
> increase in memory consumption and a 3-to-5-fold increase in startup time
> for our test case (100 deployed webapps).  After some digging we determined
> that these increases were related to JAR scanning
> (tomcat.util.scan.DefaultJarScanner).

JAR scanning should be transient. What was it that was causing the 10-20
fold increase?

> · When Tomcat 8 was released, we noted that it included a
> “whitelist” version of the jarsToSkip Property, jarsToScan.  Our hopes were
> high that we could skip from Tomcat 6 to Tomcat 8, but our test case (again
> 100 deployed webapps) quickly deflated those hopes:  similar startup times
> but at least a *30-fold increase in memory consumption*, regardless of how
> we configure jarsToScan or jarsToSkip.
> 
> · A heap dump analysis showed that the Tomcat 8 memory increase was
> largely due to the size/count of
> org.apache.catalina.webresources.JarResourceSet objects.

Those are only going to be created when a JAR is found that contains
META-INF/resources. The spec requires Tomcat to expose those as static
resources and that requires plumbing. Note Tomcat 6 will just ignore
these because the spec is to old for this feature.

> We attempted to
> reduce this by setting the Context/Resources@cachingAllowed atttribute to
> “false” and also tried to tune the “cacheMaxSize” and “cacheObjectMaxSize”
> attributes.

Those attributes should have no impact on the number or size of the
JarResourceSet instances.

> The only effect that came from these changes was a change in
> the object that took up all the space:  instead of
> org.apache.catalina.webresources.JarResourceSet objects, they are
> org.apache.catalina.webresources.StandardResourceSet objects, taking up
> roughly the same amount of space.

Strange. Very strange.

> Question:
> 
> Is there anything else we can adjust to make Tomcat 8’s memory consumption
> closer to that of Tomcat 6’s, for our use case?

It is hard to see exactly what is going on here. The symptoms you
describe and the resulting changes with configuration are not consistent
with how I would expect Tomcat to behave nor can I see how it might
behave in the way described.

> If not, we are faced with
> either running Tomcat 6 past its EOL, or possibly maintaining a very large
> jarsToSkip blacklist in Tomcat 7 (until its EOL).  It is not economically
> feasible for us to increase the physical memory of our servers by 30x so we
> can run Tomcat 8.
> 
> Test procedure:
> 
> 
>- Create a "dummy" webapp (using default Maven archetype) named test.war
>- Add several popular Maven dependencies such that the total size of the
>JARs in WEB-INF/lib is about 30MB

Can you provide the sample project? It makes sense to ensure we are
working from the same baseline. I wonder if the libraries themselves are
a factor here.

>- Download the latest versions of Tomcat 6, 7, and 8.
>- Set Java environment variables for a JMX listener and a 3GB max heap
>- Make 100 copies of test.war (test1.war, test2.war, …, test100.war) and
>drop them in /webapps.
>- Start the container, note the reported startup time, perform an
>explicit major GC, and note the heap utilization.
>- Run each test several times and average the results

That is a nice simple test we should be able to repeat.

> Test results:
> 
> Case 1 (baseline):
> 
> 
> 
> +-+--+---+
> | version | startup time | heap usage after major GC |
> +-+--+---+
> | tomcat6 | 36,711ms | 21,163,288|
> | tomcat7 | 104,517ms| 489,992,264   |
> | tomcat8 | 156,094ms| 1,010,512,568 |
> +-+--+---+
> 
> 
> Case 2 (Tomcat 7 and 8 with “jarsToSkip=*”)
> 
> +-+--+---+
> | version | startup time | heap usage after major GC |
> +-+--+---+
> | tomcat6 | 36,711ms | 21,163,288|
> | tomcat7 | 

Re: Cannot upgrade past Tomcat 6 due to massive memory increase

[George Sexton]

On 2/16/2016 9:13 AM, Ty wrote:

· When Tomcat 7 was released, our testing showed a 10-to-20-fold
increase in memory consumption and a 3-to-5-fold increase in startup time
for our test case (100 deployed webapps).  After some digging we determined
that these increases were related to JAR scanning
(tomcat.util.scan.DefaultJarScanner).  When setting jarsToSkip=*, the
memory consumption was decreased to something somewhat more reasonable
(2-3x increase over Tomcat 6) and the startup time went back to what we
were used to with Tomcat 6.  However at that time we opted to stay on
Tomcat 6, because our apps were not functional with jarsToSkip=*, and we
deemed it impractical to manage a very large “blacklist” in the jarsToSkip
Property.




I'll just throw in my experiences. I'm running around 800+ copies of a 
web app on tomcat 7.


First, to get Tomcat 7 to start in under a century, I had to do 
jarsToSkip. It hasn't been a big problem to maintain since. Fortunately 
for me as well, my app doesn't use annotations in any significant way so 
I'm pretty lucky there.


Second, I moved all of the generic jars into ${catalina.base}/lib. My 
web app's jar uses statics, so each webapp has perhaps a single 1.5MB 
jar in the context WEB-INF/lib. This is bad because now I pretty much 
have to upgrade all instances in lock-step otherwise I run into version 
dependencies with ${catalina.base}/lib files. Still, it's an acceptable 
tradeoff. If this were possible for you, I'd definitely give this a go.


My startup times are around 30 seconds. A huge part of my remaining 
startup times are caused by loading each instance's unique database. 
When I put the databases on SSD, things were lots better :)


Right now, I have 48GB allocated to the JVM. Really, it's using 16GB of 
eden space, and it's got around 32GB dedicated to Old Gen, but it's 
really only using 5GB of space in Old Gen. Survivor space is under 
256MB. I suppose I could do some tuning with various parameters to force 
more allocation to eden space, and less to old gen, but performance is 
really good, and I don't have any motivation to do it.



--
George Sexton
*MH Software, Inc.*
Voice: 303 438 9585
http://www.mhsoftware.com

Cannot upgrade past Tomcat 6 due to massive memory increase

[Ty]

Summary:

Because of our use case (hundreds of webapps per instance), we cannot
feasibly upgrade our Tomcat 6 containers to Tomcat 7 or Tomcat 8, due to
the massive increase in memory the upgrade causes.

Background:

We’ve been running Tomcat 6 for several years, on perhaps an unusual scale:
several dozen Tomcat instances, each running up to a *few hundred webapps*
in a single container.  (Picture a cloud service provider with several
products and hundreds of customers, with one dedicated webapp
per-customer-per-product, resulting in many thousands of web
applications.  It’s
not ideal but it’s how the software was designed).  Typical max heap sizes
per container are 1GB to 4GB depending on utilization, and max container
startup times are in the neighborhood of 5 to 10 minutes.

· When Tomcat 7 was released, our testing showed a 10-to-20-fold
increase in memory consumption and a 3-to-5-fold increase in startup time
for our test case (100 deployed webapps).  After some digging we determined
that these increases were related to JAR scanning
(tomcat.util.scan.DefaultJarScanner).  When setting jarsToSkip=*, the
memory consumption was decreased to something somewhat more reasonable
(2-3x increase over Tomcat 6) and the startup time went back to what we
were used to with Tomcat 6.  However at that time we opted to stay on
Tomcat 6, because our apps were not functional with jarsToSkip=*, and we
deemed it impractical to manage a very large “blacklist” in the jarsToSkip
Property.

· When Tomcat 8 was released, we noted that it included a
“whitelist” version of the jarsToSkip Property, jarsToScan.  Our hopes were
high that we could skip from Tomcat 6 to Tomcat 8, but our test case (again
100 deployed webapps) quickly deflated those hopes:  similar startup times
but at least a *30-fold increase in memory consumption*, regardless of how
we configure jarsToScan or jarsToSkip.

· A heap dump analysis showed that the Tomcat 8 memory increase was
largely due to the size/count of
org.apache.catalina.webresources.JarResourceSet objects.  We attempted to
reduce this by setting the Context/Resources@cachingAllowed atttribute to
“false” and also tried to tune the “cacheMaxSize” and “cacheObjectMaxSize”
attributes.  The only effect that came from these changes was a change in
the object that took up all the space:  instead of
org.apache.catalina.webresources.JarResourceSet objects, they are
org.apache.catalina.webresources.StandardResourceSet objects, taking up
roughly the same amount of space.

Question:

Is there anything else we can adjust to make Tomcat 8’s memory consumption
closer to that of Tomcat 6’s, for our use case?  If not, we are faced with
either running Tomcat 6 past its EOL, or possibly maintaining a very large
jarsToSkip blacklist in Tomcat 7 (until its EOL).  It is not economically
feasible for us to increase the physical memory of our servers by 30x so we
can run Tomcat 8.

Test procedure:


   - Create a "dummy" webapp (using default Maven archetype) named test.war
   - Add several popular Maven dependencies such that the total size of the
   JARs in WEB-INF/lib is about 30MB
   - Download the latest versions of Tomcat 6, 7, and 8.
   - Set Java environment variables for a JMX listener and a 3GB max heap
   - Make 100 copies of test.war (test1.war, test2.war, …, test100.war) and
   drop them in /webapps.
   - Start the container, note the reported startup time, perform an
   explicit major GC, and note the heap utilization.
   - Run each test several times and average the results

Test results:

Case 1 (baseline):



+-+--+---+
| version | startup time | heap usage after major GC |
+-+--+---+
| tomcat6 | 36,711ms | 21,163,288|
| tomcat7 | 104,517ms| 489,992,264   |
| tomcat8 | 156,094ms| 1,010,512,568 |
+-+--+---+


Case 2 (Tomcat 7 and 8 with “jarsToSkip=*”)

+-+--+---+
| version | startup time | heap usage after major GC |
+-+--+---+
| tomcat6 | 36,711ms | 21,163,288|
| tomcat7 | 38,979ms | 72,359,840|
| tomcat8 | 52,040ms | 633,682,336   |
+-+--+---+