Re: Cannot upgrade past Tomcat 6 due to massive memory increase
On 18/02/2016 22:57, Ty wrote: > I've re-run my tests using 8.0.x trunk (8.0.33-dev) using the same > procedure from the first message of this thread, and summarized the results > below. Heap usage is vastly improved both with and without JAR scanning. > Heap size and startup time are both reasonable and it's now feasible to > re-visit our upgrade plan, so we will be putting 8.0.33 into our next cycle > (once it's officially released). > > Thanks again for looking into this, and wow, addressing it so quickly. Great. That is good news. I'm not finished with this. There were a number of things in the heap dump that looked to be worthy of further consideration. I hope to get to them in time for 8.0.33 but regardless of whether I do, looking at this some more is on my TODO list. Mark > > Cheers, > Ty > > > Case 1 (baseline): > > +-+--+---+ > | version | startup time | heap usage after major GC | > +-+--+---+ > | tomcat6 | 36,711ms | 21,163,288| > | tomcat7 | 104,517ms| 489,992,264 | > | tomcat8 | 156,094ms| 1,010,512,568 | > | tomcat8-dev | 139,858ms| 218,976,528 | > +-+--+---+ > > > Case 2 (Tomcat 7 and 8 with “jarsToSkip=*”) > > +-+--+---+ > | version | startup time | heap usage after major GC | > +-+--+---+ > | tomcat6 | 36,711ms | 21,163,288| > | tomcat7 | 38,979ms | 72,359,840| > | tomcat8 | 52,040ms | 633,682,336 | > | tomcat8-dev | 66,662ms | 97,365,048| > +-+--+---+ > > > > > On Thu, Feb 18, 2016 at 10:12 AM Ty wrote: > >>> Are you able to build Tomcat 8.0.x from source to test locally or would >>> you like me to provide you with a test build? >> >> Looks like building is pretty straightforward. I'll dive in and post back >> with my testing results. >> >> >> On Thu, Feb 18, 2016 at 9:11 AM Mark Thomas wrote: >> >>> Another update. >>> >>> Before any fixes, my tests showed a heap of ~1GB with ~30% used by >>> WebappClassLoader and 50% by JarResourceSet >>> >>> The WebappClassLoader issue was triggered by JAR scanning. Disabling / >>> limiting JAR scanning will reduce the impact of this issue. >>> The issue has been fixed in 9.0.x and 8.0.x and will be included in >>> 9.0.0.M4 and 8.0.33 onwards. >>> >>> The JarResourceSet issue was hard coded. There is no way to work around >>> it. This has also been fixed in 9.0.x and 8.0.x and will be included in >>> 9.0.0.M4 and 8.0.33 onwards. >>> >>> Regressions are possible but unlikely. All the unit tests pass and I >>> have been using Atlassian Jira as a test app to check for changes in >>> start time and as a smoke test. There is a small increase in start time >>> with fix for the JarResourceSet issue (~5%) but the reduction in memory >>> usage more than makes up for it. >>> >>> Are you able to build Tomcat 8.0.x from source to test locally or would >>> you like me to provide you with a test build? >>> >>> Looking at the heap dump, there are still a few areas where it may be >>> possible to reclaim more memory. This is on my TODO list and I hope to >>> get to it before the next releases. >>> >>> Mark >>> >>> - >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> >>> > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Cannot upgrade past Tomcat 6 due to massive memory increase
I've re-run my tests using 8.0.x trunk (8.0.33-dev) using the same procedure from the first message of this thread, and summarized the results below. Heap usage is vastly improved both with and without JAR scanning. Heap size and startup time are both reasonable and it's now feasible to re-visit our upgrade plan, so we will be putting 8.0.33 into our next cycle (once it's officially released). Thanks again for looking into this, and wow, addressing it so quickly. Cheers, Ty Case 1 (baseline): +-+--+---+ | version | startup time | heap usage after major GC | +-+--+---+ | tomcat6 | 36,711ms | 21,163,288| | tomcat7 | 104,517ms| 489,992,264 | | tomcat8 | 156,094ms| 1,010,512,568 | | tomcat8-dev | 139,858ms| 218,976,528 | +-+--+---+ Case 2 (Tomcat 7 and 8 with “jarsToSkip=*”) +-+--+---+ | version | startup time | heap usage after major GC | +-+--+---+ | tomcat6 | 36,711ms | 21,163,288| | tomcat7 | 38,979ms | 72,359,840| | tomcat8 | 52,040ms | 633,682,336 | | tomcat8-dev | 66,662ms | 97,365,048| +-+--+---+ On Thu, Feb 18, 2016 at 10:12 AM Ty wrote: > > Are you able to build Tomcat 8.0.x from source to test locally or would > > you like me to provide you with a test build? > > Looks like building is pretty straightforward. I'll dive in and post back > with my testing results. > > > On Thu, Feb 18, 2016 at 9:11 AM Mark Thomas wrote: > >> Another update. >> >> Before any fixes, my tests showed a heap of ~1GB with ~30% used by >> WebappClassLoader and 50% by JarResourceSet >> >> The WebappClassLoader issue was triggered by JAR scanning. Disabling / >> limiting JAR scanning will reduce the impact of this issue. >> The issue has been fixed in 9.0.x and 8.0.x and will be included in >> 9.0.0.M4 and 8.0.33 onwards. >> >> The JarResourceSet issue was hard coded. There is no way to work around >> it. This has also been fixed in 9.0.x and 8.0.x and will be included in >> 9.0.0.M4 and 8.0.33 onwards. >> >> Regressions are possible but unlikely. All the unit tests pass and I >> have been using Atlassian Jira as a test app to check for changes in >> start time and as a smoke test. There is a small increase in start time >> with fix for the JarResourceSet issue (~5%) but the reduction in memory >> usage more than makes up for it. >> >> Are you able to build Tomcat 8.0.x from source to test locally or would >> you like me to provide you with a test build? >> >> Looking at the heap dump, there are still a few areas where it may be >> possible to reclaim more memory. This is on my TODO list and I hope to >> get to it before the next releases. >> >> Mark >> >> - >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >>
Re: WebEx meeting invitation: Apache Tomcat: Connector Selection
The recording is now on YouTube: https://www.youtube.com/watch?v=LBSWixIwMmU Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Updating SSL certificates
On 18/02/2016 22:03, James H. H. Lampert wrote: > Out of morbid curiosity, is there a way to make a certificate update > take effect without restarting Tomcat? Sort of. Set bindOnInit on the connector to false. Modify the config via JMX. Then you should be able to use JMX to call stop() followed by start() on the TLS connector which should re-initialise the TLS settings from the in-memory config. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Updating SSL certificates
Out of morbid curiosity, is there a way to make a certificate update take effect without restarting Tomcat? -- JHHL - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Session on Tomcat 7 manager screem
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Qadeer,
On 2/18/16 12:03 PM, Qadeer Khan wrote:
> Ok I got a little more information the user
Get this, too: What exact version of Tomcat. 7.0.what?
> On clicking on the session number '0', he is getting the following
> error. Any idea?
>
> FAIL - Encountered exception org.apache.jasper.JasperException:
> Unable to compile class for JSP:
>
> An error occurred at line: [18] in the generated java file:
> [/opt/jboss/jws-3.0/tomcat7/work/Catalina/localhost/manager/org/apache
/jsp/WEB_002dINF/jsp/sessionsList_jsp.java]
Okay,
>
so this is JBoss. Has Tomcat itself been modified at all? Or is
it a (mostly) stock Tomcat?
> Only a type can be imported.
> org.apache.catalina.ha.session.DeltaSession resolves to a package
>
> An error occurred at line: 109 in the jsp file:
> /WEB-INF/jsp/sessionsList.jsp DeltaSession cannot be resolved to a
> type 106:Session currentSession = (Session) iter.next();
> 107:String currentSessionId =
> JspHelper.escapeXml(currentSession.getId()); 108:String
> type; 109:if (currentSession instanceof DeltaSession) {
> 110:if (((DeltaSession)
> currentSession).isPrimarySession()) { 111:type =
> "Primary"; 112:} else {
>
>
> An error occurred at line: 110 in the jsp file:
> /WEB-INF/jsp/sessionsList.jsp DeltaSession cannot be resolved to a
> type 107:String currentSessionId =
> JspHelper.escapeXml(currentSession.getId()); 108:String
> type; 109:if (currentSession instanceof DeltaSession) {
> 110:if (((DeltaSession)
> currentSession).isPrimarySession()) { 111:type =
> "Primary"; 112:} else { 113:type =
> "Backup";
This suggests to me that something is broken in the Tomcat
installation itself (meaning that the customer - or you - have broken
something). Can you confirm that the manager web application matches
the Tomcat version you are running? If you copied the manager from
another Tomcat version, it might not work very well -- or at all.
Back to your original question:
> So there are sessions running by users but no increment shown
> under 'session'
>
> Please throw some light as why it is not increment and what can be
> done to fix it?
Why are you so sure that there are user sessions? Is some other
component telling you there *are* sessions? If so, which component is
telling you that there are sessions in memory?
- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlbGGvUACgkQ9CaO5/Lv0PCpFQCfeqjXU92mDmfpuCXM3Rn7sDFR
0WIAn0wk2sFTWQU5j2VGWPJ1pvVlOAf6
=cWwW
-END PGP SIGNATURE-
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Session on Tomcat 7 manager screem
On 2/18/2016 11:55 AM, Qadeer Khan wrote: I am not sure why so aggravated you get... I am not claiming any bugs or anything and I am no expert in Tomcat 7 application server. I am at client site and they are simply asking me a question as I explained before. They have java applications deployed and creating more than one sessions of application and as I said earlier session is not incrementing And Mark is telling you that most likely, they are not actually creating sessions. "Session" has a specific definition in the http world, and not every user, application or call will create a session. It is as simple as that!! - Original Message - From: "Mark Thomas" To: "Tomcat Users List" Sent: Thursday, February 18, 2016 11:50:24 AM Subject: Re: Session on Tomcat 7 manager screem On 18/02/2016 16:45, Qadeer Khan wrote: Can you tell what a session definition is? You are claiming there is a bug with session handling yet you don't know what a session is? Oh dear. Time for you to spend some time reading the Servlet specification. Mark On 18/02/2016 16:28, Qadeer Khan wrote: Hey Guys, Someone please throw some light as I am being asked about it several time now Then maybe try reading the responses you received to your previous post and answering the questions you were asked. There is are several running java applications deployed and running on tomcat server 7 but the 'session' field on the Manager screen always show a '0". OK. So there aren't any sessions. So there are sessions running by users but no increment shown under 'session' You need to provide some evidence to back up that statement. You have been asked several times for this and still have not provided any. What have you done to confirm that sessions are created? Please throw some light as why it is not increment and what can be done to fix it? You have yet to demonstrate that there is anything to fix. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Session on Tomcat 7 manager screem
On 18/02/2016 16:55, Qadeer Khan wrote: > I am not sure why so aggravated you get... I am not claiming any bugs or > anything Then you need to modify your language rather than suggesting that there is something wrong in Tomcat that needs fixing. > and I am no expert in Tomcat 7 application server. That much is clear. > I am at client site and they are simply asking me a question as I explained > before. So you are being paid to fix a problem in a system you know next to nothing about and you are expecting us to take the time to provide you with the answer for free. And you wonder why people get aggravated when you can't even be bothered to answer the one question you have been asked over and over again. > They have java applications deployed and creating more than one sessions of > application and as I said earlier session is not incrementing > > It is as simple as that!! I am reminded of the late, great Terry Pratchett and his views on the use of multiple exclamation marks. [1] [1] http://wiki.lspace.org/mediawiki/Multiple_exclamation_marks > > > > > - Original Message - > From: "Mark Thomas" > To: "Tomcat Users List" > Sent: Thursday, February 18, 2016 11:50:24 AM > Subject: Re: Session on Tomcat 7 manager screem > > On 18/02/2016 16:45, Qadeer Khan wrote: >> Can you tell what a session definition is? > > You are claiming there is a bug with session handling yet you don't know > what a session is? Oh dear. > > Time for you to spend some time reading the Servlet specification. > > Mark > > >> >> >> On 18/02/2016 16:28, Qadeer Khan wrote: >>> Hey Guys, >>> >>> Someone please throw some light as I am being asked about it several time >>> now >> >> Then maybe try reading the responses you received to your previous post >> and answering the questions you were asked. >> >>> There is are several running java applications deployed and running on >>> tomcat server 7 but the 'session' field on the Manager screen always show >>> a '0". >> >> OK. So there aren't any sessions. >> >>> So there are sessions running by users but no increment shown under >>> 'session' >> >> You need to provide some evidence to back up that statement. You have >> been asked several times for this and still have not provided any. What >> have you done to confirm that sessions are created? >> >>> Please throw some light as why it is not increment and what can be done to >>> fix it? >> >> You have yet to demonstrate that there is anything to fix. >> >> Mark >> >> >> - >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> >> - >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Session on Tomcat 7 manager screem
Ok I got a little more information the user
On cllicking on the session number '0', he is getting the following error. Any
idea?
FAIL - Encountered exception org.apache.jasper.JasperException: Unable to
compile class for JSP:
An error occurred at line: [18] in the generated java file:
[/opt/jboss/jws-3.0/tomcat7/work/Catalina/localhost/manager/org/apache/jsp/WEB_002dINF/jsp/sessionsList_jsp.java]
Only a type can be imported. org.apache.catalina.ha.session.DeltaSession
resolves to a package
An error occurred at line: 109 in the jsp file: /WEB-INF/jsp/sessionsList.jsp
DeltaSession cannot be resolved to a type
106:Session currentSession = (Session) iter.next();
107:String currentSessionId =
JspHelper.escapeXml(currentSession.getId());
108:String type;
109:if (currentSession instanceof DeltaSession) {
110:if (((DeltaSession) currentSession).isPrimarySession()) {
111:type = "Primary";
112:} else {
An error occurred at line: 110 in the jsp file: /WEB-INF/jsp/sessionsList.jsp
DeltaSession cannot be resolved to a type
107:String currentSessionId =
JspHelper.escapeXml(currentSession.getId());
108:String type;
109:if (currentSession instanceof DeltaSession) {
110:if (((DeltaSession) currentSession).isPrimarySession()) {
111:type = "Primary";
112:} else {
113:type = "Backup";
Stacktrace:
Thanks
Original Message -
From: "Mark Thomas"
To: "Tomcat Users List"
Sent: Thursday, February 18, 2016 11:50:24 AM
Subject: Re: Session on Tomcat 7 manager screem
On 18/02/2016 16:45, Qadeer Khan wrote:
> Can you tell what a session definition is?
You are claiming there is a bug with session handling yet you don't know
what a session is? Oh dear.
Time for you to spend some time reading the Servlet specification.
Mark
>
>
> On 18/02/2016 16:28, Qadeer Khan wrote:
>> Hey Guys,
>>
>> Someone please throw some light as I am being asked about it several time now
>
> Then maybe try reading the responses you received to your previous post
> and answering the questions you were asked.
>
>> There is are several running java applications deployed and running on
>> tomcat server 7 but the 'session' field on the Manager screen always show a
>> '0".
>
> OK. So there aren't any sessions.
>
>> So there are sessions running by users but no increment shown under 'session'
>
> You need to provide some evidence to back up that statement. You have
> been asked several times for this and still have not provided any. What
> have you done to confirm that sessions are created?
>
>> Please throw some light as why it is not increment and what can be done to
>> fix it?
>
> You have yet to demonstrate that there is anything to fix.
>
> Mark
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Session on Tomcat 7 manager screem
I am not sure why so aggravated you get... I am not claiming any bugs or anything and I am no expert in Tomcat 7 application server. I am at client site and they are simply asking me a question as I explained before. They have java applications deployed and creating more than one sessions of application and as I said earlier session is not incrementing It is as simple as that!! - Original Message - From: "Mark Thomas" To: "Tomcat Users List" Sent: Thursday, February 18, 2016 11:50:24 AM Subject: Re: Session on Tomcat 7 manager screem On 18/02/2016 16:45, Qadeer Khan wrote: > Can you tell what a session definition is? You are claiming there is a bug with session handling yet you don't know what a session is? Oh dear. Time for you to spend some time reading the Servlet specification. Mark > > > On 18/02/2016 16:28, Qadeer Khan wrote: >> Hey Guys, >> >> Someone please throw some light as I am being asked about it several time now > > Then maybe try reading the responses you received to your previous post > and answering the questions you were asked. > >> There is are several running java applications deployed and running on >> tomcat server 7 but the 'session' field on the Manager screen always show a >> '0". > > OK. So there aren't any sessions. > >> So there are sessions running by users but no increment shown under 'session' > > You need to provide some evidence to back up that statement. You have > been asked several times for this and still have not provided any. What > have you done to confirm that sessions are created? > >> Please throw some light as why it is not increment and what can be done to >> fix it? > > You have yet to demonstrate that there is anything to fix. > > Mark > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Session on Tomcat 7 manager screem
On 18/02/2016 16:45, Qadeer Khan wrote: > Can you tell what a session definition is? You are claiming there is a bug with session handling yet you don't know what a session is? Oh dear. Time for you to spend some time reading the Servlet specification. Mark > > > On 18/02/2016 16:28, Qadeer Khan wrote: >> Hey Guys, >> >> Someone please throw some light as I am being asked about it several time now > > Then maybe try reading the responses you received to your previous post > and answering the questions you were asked. > >> There is are several running java applications deployed and running on >> tomcat server 7 but the 'session' field on the Manager screen always show a >> '0". > > OK. So there aren't any sessions. > >> So there are sessions running by users but no increment shown under 'session' > > You need to provide some evidence to back up that statement. You have > been asked several times for this and still have not provided any. What > have you done to confirm that sessions are created? > >> Please throw some light as why it is not increment and what can be done to >> fix it? > > You have yet to demonstrate that there is anything to fix. > > Mark > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Session on Tomcat 7 manager screem
Can you tell what a session definition is? On 18/02/2016 16:28, Qadeer Khan wrote: > Hey Guys, > > Someone please throw some light as I am being asked about it several time now Then maybe try reading the responses you received to your previous post and answering the questions you were asked. > There is are several running java applications deployed and running on tomcat > server 7 but the 'session' field on the Manager screen always show a '0". OK. So there aren't any sessions. > So there are sessions running by users but no increment shown under 'session' You need to provide some evidence to back up that statement. You have been asked several times for this and still have not provided any. What have you done to confirm that sessions are created? > Please throw some light as why it is not increment and what can be done to > fix it? You have yet to demonstrate that there is anything to fix. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Session on Tomcat 7 manager screem
On 18/02/2016 16:28, Qadeer Khan wrote: > Hey Guys, > > Someone please throw some light as I am being asked about it several time now Then maybe try reading the responses you received to your previous post and answering the questions you were asked. > There is are several running java applications deployed and running on tomcat > server 7 but the 'session' field on the Manager screen always show a '0". OK. So there aren't any sessions. > So there are sessions running by users but no increment shown under 'session' You need to provide some evidence to back up that statement. You have been asked several times for this and still have not provided any. What have you done to confirm that sessions are created? > Please throw some light as why it is not increment and what can be done to > fix it? You have yet to demonstrate that there is anything to fix. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Session on Tomcat 7 manager screem
Hey Guys, Someone please throw some light as I am being asked about it several time now There is are several running java applications deployed and running on tomcat server 7 but the 'session' field on the Manager screen always show a '0". So there are sessions running by users but no increment shown under 'session' Please throw some light as why it is not increment and what can be done to fix it? Help. Please! Thanks - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Close_wait state
On 18.02.2016 16:50, Elias, Michael wrote: Hi - We are running tomcat version 7.0.50. Starting 2 days ago are application stopped responding to requests. Our investigation showed us that we are not closing connections. We see after 300 tcp sessions, for the tomcat PID, in CLOSE_WAIT state out app stops responding. Restarting the app clears the state. We took tcpdumps between our web layer and our tomcat layer. What we see in a successful connection is, after the response, tomcat sends a FIN, web ACK's, then web sends a FIN and Tomcat ACK's.. connection closes In a bad connection, tomcat does not send its FIN after the response, after 3 minutes, the WEB sends a FIN and tomcat ACK's. the connection goes into CLOSE_WAIT and stays in that state until restart of tomcat. Any help would be greatly appreciated. I have a question, and a story to this : Question : what happens to your connections in CLOSE_WAIT, if you force Tomcat (or rather, its JVM) to do a GC (garbage collection) ? (There are probably different ways to do that, but I know only one and it is lengthy to set up. Maybe someone has a quick suggestion ?) Story : One case in the past in which I had a similar issue, was with a webapp which : - created an object which itself created a TCP connection to some external process - used that object (its methods) to access that connection - and when the time came to close this connection, it just "forgot" the object, and left it to the JVM to cleanup when it destroyed the object And the JVM ended up with hundreds of connections in the CLOSE_WAIT state, up to a point (under Linux) where the entire TCP stack became unresponsive. My interpretation of what happened then is : Because in Java the garbage collection is asynchronous with the rest and only happens when needed, this unreferenced object could stay on the heap for quite a while. (As a matter of fact, the more comfortable the heap, the longer it stays). And because the JVM, to create a socket, uses ultimately some native code and some underlying native socket structure, this underlying OS-level socket remained also, in its CLOSE_WAIT state, for a long time after the original java object and wrapped connection had long ceased to be used by the webapp. A GC cleared that, because it finally eliminates and destroys unreferenced objects, and their linked native structures at the same time, which has the effect of finally closing the connection properly. So a GC magically deleted these hundreds of CLOSE_WAIT connections. Maybe your case is similar ? The proper solution of course is to make sure that the webapp properly closes the underlying connection before it drops the object that encapsulates it. An unproper and temporary (but in the meantime working) solution for me - because we had no access to the bad code - was to write a script which ran regularly, and forced the Tomcat JVM to do a GC. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Close_wait state
Hi - We are running tomcat version 7.0.50. Starting 2 days ago are application stopped responding to requests. Our investigation showed us that we are not closing connections. We see after 300 tcp sessions, for the tomcat PID, in CLOSE_WAIT state out app stops responding. Restarting the app clears the state. We took tcpdumps between our web layer and our tomcat layer. What we see in a successful connection is, after the response, tomcat sends a FIN, web ACK's, then web sends a FIN and Tomcat ACK's.. connection closes In a bad connection, tomcat does not send its FIN after the response, after 3 minutes, the WEB sends a FIN and tomcat ACK's. the connection goes into CLOSE_WAIT and stays in that state until restart of tomcat. Any help would be greatly appreciated. Thanks Michael
Re: Cannot upgrade past Tomcat 6 due to massive memory increase
> Are you able to build Tomcat 8.0.x from source to test locally or would > you like me to provide you with a test build? Looks like building is pretty straightforward. I'll dive in and post back with my testing results. On Thu, Feb 18, 2016 at 9:11 AM Mark Thomas wrote: > Another update. > > Before any fixes, my tests showed a heap of ~1GB with ~30% used by > WebappClassLoader and 50% by JarResourceSet > > The WebappClassLoader issue was triggered by JAR scanning. Disabling / > limiting JAR scanning will reduce the impact of this issue. > The issue has been fixed in 9.0.x and 8.0.x and will be included in > 9.0.0.M4 and 8.0.33 onwards. > > The JarResourceSet issue was hard coded. There is no way to work around > it. This has also been fixed in 9.0.x and 8.0.x and will be included in > 9.0.0.M4 and 8.0.33 onwards. > > Regressions are possible but unlikely. All the unit tests pass and I > have been using Atlassian Jira as a test app to check for changes in > start time and as a smoke test. There is a small increase in start time > with fix for the JarResourceSet issue (~5%) but the reduction in memory > usage more than makes up for it. > > Are you able to build Tomcat 8.0.x from source to test locally or would > you like me to provide you with a test build? > > Looking at the heap dump, there are still a few areas where it may be > possible to reclaim more memory. This is on my TODO list and I hope to > get to it before the next releases. > > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Close_wait state
Hi - We are running tomcat version 7.0.50. Starting 2 days ago are application stopped responding to requests. Our investigation showed us that we are not closing connections. We see after 300 tcp sessions, for the tomcat PID, in CLOSE_WAIT state out app stops responding. Restarting the app clears the state. We took tcpdumps between our web layer and our tomcat layer. What we see in a successful connection is, after the response, tomcat sends a FIN, web ACK's, then web sends a FIN and Tomcat ACK's.. connection closes In a bad connection, tomcat does not send its FIN after the response, after 3 minutes, the WEB sends a FIN and tomcat ACK's. the connection goes into CLOSE_WAIT and stays in that state until restart of tomcat. Any help would be greatly appreciated. Thanks Michael
Re: Cannot upgrade past Tomcat 6 due to massive memory increase
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Mark, On 2/18/16 10:11 AM, Mark Thomas wrote: > Another update. > > Before any fixes, my tests showed a heap of ~1GB with ~30% used by > WebappClassLoader and 50% by JarResourceSet > > The WebappClassLoader issue was triggered by JAR scanning. > Disabling / limiting JAR scanning will reduce the impact of this > issue. The issue has been fixed in 9.0.x and 8.0.x and will be > included in 9.0.0.M4 and 8.0.33 onwards. > > The JarResourceSet issue was hard coded. There is no way to work > around it. This has also been fixed in 9.0.x and 8.0.x and will be > included in 9.0.0.M4 and 8.0.33 onwards. > > Regressions are possible but unlikely. All the unit tests pass and > I have been using Atlassian Jira as a test app to check for changes > in start time and as a smoke test. There is a small increase in > start time with fix for the JarResourceSet issue (~5%) but the > reduction in memory usage more than makes up for it. Nice. Thanks for diving into this one. - -chris -BEGIN PGP SIGNATURE- Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlbF4OIACgkQ9CaO5/Lv0PBjTACcCcDphK4esKtoC92i9p5iRu+R fqMAn0GmGulhAwirSyDRcqIK6lFMO5Pf =BKY6 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Cannot upgrade past Tomcat 6 due to massive memory increase
Another update. Before any fixes, my tests showed a heap of ~1GB with ~30% used by WebappClassLoader and 50% by JarResourceSet The WebappClassLoader issue was triggered by JAR scanning. Disabling / limiting JAR scanning will reduce the impact of this issue. The issue has been fixed in 9.0.x and 8.0.x and will be included in 9.0.0.M4 and 8.0.33 onwards. The JarResourceSet issue was hard coded. There is no way to work around it. This has also been fixed in 9.0.x and 8.0.x and will be included in 9.0.0.M4 and 8.0.33 onwards. Regressions are possible but unlikely. All the unit tests pass and I have been using Atlassian Jira as a test app to check for changes in start time and as a smoke test. There is a small increase in start time with fix for the JarResourceSet issue (~5%) but the reduction in memory usage more than makes up for it. Are you able to build Tomcat 8.0.x from source to test locally or would you like me to provide you with a test build? Looking at the heap dump, there are still a few areas where it may be possible to reclaim more memory. This is on my TODO list and I hope to get to it before the next releases. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Cannot upgrade past Tomcat 6 due to massive memory increase
On 18/02/2016 00:26, Ty wrote: >>> For Ty's purposes, simply avoid JAR scanning entirely should eliminate >>> the heap issues, right? Moving from Tomcat 6 means that JAR scanning >>> is not required for correct operation of their application. > >> Yes. http://wiki.apache.org/tomcat/HowTo/FasterStartUp has the details >> of the configuration required. > > I might be missing something, if so please correct me, but it seems that > there's a huge difference with Tomcat 8, even when JAR scanning is taken > out of the picture (jarsToSkip=*). > > For my test case, Tomcat 8 has a heap size of ~1GB with JAR scanning > enabled, and ~600MB with jarsToSkip=*. That's a 40% improvement by > disabling JAR scanning, sure, but for comparison, same test case, Tomcat > 6's heap is ~20MB while Tomcat 7's (with jarsToSkip=*) is ~70MB. We're > still back to a 30x increase in heap utilization from Tomcat 6 to Tomcat 8, > and a 10x increase from Tomcat 7 to Tomcat 8, without having JAR scanning > in the mix. Sorry. Not enough sleep when I wrote that. The WebappClassLoader issue is addressed by disabling JAR scanning but the JarResourceSet issue is not. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
