Re: Tomcat 5.5.25 | Memory leak in Web Application
Hello all, Can one of you please help answer the last set of queries I have on this please? Thanks Anurag On Mon, Oct 18, 2010 at 3:02 PM, Anurag Kapur anuragka...@gmail.com wrote: Thanks Chris/Mark/Charles/Pid for your help with this. The issue has been fixed after using the JVM argument :- -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true I get a healthy heap utilization graph on the same web application under similar load test conditions as indicated in the graph in the link below http://3.bp.blogspot.com/_Oq9OlP-8ap0/TLxLM-dsf_I/CPk/SolVOyYpAuI/s1600/heap_utilization_post_fix.jpg The application sustains the load over a 2 hour period without showing any signs of degradation as it did earlier. Before closing this chapter I wanted better clarity on a couple of questions:- Also, yes, ours being a content management application, we have large body tags. We use several tag libraries that come with the CMS implementation to fetch content from the CMS. Usually CMS applications /manage/ content, rather than hard-coding it. What kind of huge body tags do you have? 1. I am not quite sure if I have the correct answer to your question but I think the most probable reason is that we use tags provided by the CMS to cache the HTML response (JSP caches). The body of these tags can hold large chunks of HTML response. Can this be a suitable explanation to your question? 2. The large character arrays I saw in the heap dumps had a lot of empty elements. For example, the first 500 or so elements in the array were empty and then the HTML response was seen in some elements again followed by empty elements and then actual HTML content again. This pattern was spread across the entire character array. Can these correspond to white spaces? Can this be because I do not have *trimSpaces *option enabled as mentioned here:- http://tomcat.apache.org/tomcat-5.5-doc/jasper-howto.html#Production Configurationhttp://tomcat.apache.org/tomcat-5.5-doc/jasper-howto.html#Production+Configuration ? 3. What does the JVM argument actually do? -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true I understand it does not turn off tag pooling and instead limits the size of the buffer. Can you please elaborate what this means? What happens when a body tag which exceeds a certain threshold of size? Regards Anurag On Wed, Oct 13, 2010 at 7:16 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Anurag, On 10/12/2010 5:47 PM, Anurag Kapur wrote: I have probably attached an incomplete snapshot of the memory utilization graph. I'm only looking at what is on your blog. That graph looks good. Perhaps you could update your blog with a graph that illustrates your conclusion. I have done a few tests with the JVM argument suggested in the bug report. The initial tests look good. Good. Also, yes, ours being a content management application, we have large body tags. We use several tag libraries that come with the CMS implementation to fetch content from the CMS. Usually CMS applications /manage/ content, rather than hard-coding it. What kind of huge body tags do you have? Good luck, - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAky193MACgkQ9CaO5/Lv0PC8ZwCeOCRg4pGeK7QjTVkeV7tNJUcD GBgAnRO63f4ed/ZRewJcgLC7FwJwLg20 =ZC7S -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 5.5.25 | Memory leak in Web Application
Anurag, Anurag Kapur wrote: ... 3. What does the JVM argument actually do? -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true I understand it does not turn off tag pooling and instead limits the size of the buffer. Can you please elaborate what this means? What happens when a body tag which exceeds a certain threshold of size? I don't understand the matter at hand, but when I search Google for org.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER I am getting a page which seems to describe this. Maybe it is the same for your other questions ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 5.5.25 | Memory leak in Web Application
Thanks Chris/Mark/Charles/Pid for your help with this. The issue has been fixed after using the JVM argument :- -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true I get a healthy heap utilization graph on the same web application under similar load test conditions as indicated in the graph in the link below http://3.bp.blogspot.com/_Oq9OlP-8ap0/TLxLM-dsf_I/CPk/SolVOyYpAuI/s1600/heap_utilization_post_fix.jpg The application sustains the load over a 2 hour period without showing any signs of degradation as it did earlier. Before closing this chapter I wanted better clarity on a couple of questions:- Also, yes, ours being a content management application, we have large body tags. We use several tag libraries that come with the CMS implementation to fetch content from the CMS. Usually CMS applications /manage/ content, rather than hard-coding it. What kind of huge body tags do you have? 1. I am not quite sure if I have the correct answer to your question but I think the most probable reason is that we use tags provided by the CMS to cache the HTML response (JSP caches). The body of these tags can hold large chunks of HTML response. Can this be a suitable explanation to your question? 2. The large character arrays I saw in the heap dumps had a lot of empty elements. For example, the first 500 or so elements in the array were empty and then the HTML response was seen in some elements again followed by empty elements and then actual HTML content again. This pattern was spread across the entire character array. Can these correspond to white spaces? Can this be because I do not have *trimSpaces *option enabled as mentioned here:- http://tomcat.apache.org/tomcat-5.5-doc/jasper-howto.html#Production Configuration? 3. What does the JVM argument actually do? -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true I understand it does not turn off tag pooling and instead limits the size of the buffer. Can you please elaborate what this means? What happens when a body tag which exceeds a certain threshold of size? Regards Anurag On Wed, Oct 13, 2010 at 7:16 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Anurag, On 10/12/2010 5:47 PM, Anurag Kapur wrote: I have probably attached an incomplete snapshot of the memory utilization graph. I'm only looking at what is on your blog. That graph looks good. Perhaps you could update your blog with a graph that illustrates your conclusion. I have done a few tests with the JVM argument suggested in the bug report. The initial tests look good. Good. Also, yes, ours being a content management application, we have large body tags. We use several tag libraries that come with the CMS implementation to fetch content from the CMS. Usually CMS applications /manage/ content, rather than hard-coding it. What kind of huge body tags do you have? Good luck, - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAky193MACgkQ9CaO5/Lv0PC8ZwCeOCRg4pGeK7QjTVkeV7tNJUcD GBgAnRO63f4ed/ZRewJcgLC7FwJwLg20 =ZC7S -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 5.5.25 | Memory leak in Web Application
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Mark, On 10/12/2010 5:28 PM, Mark Thomas wrote: On 12/10/2010 19:45, Christopher Schultz wrote: markt marked this bug as FIXED, but I see no indication of a resolution. Perhaps that was meant to be WONTFIX? Nope. I meant FIXED. As in There is now an option you can use to disable this behaviour if you don't like it. Gotcha. It wasn't clear from the comment that the fix was somewhat of a workaround. Any reason those buffers never release their memory? Was the complexity of the proposed fix deemed too high for the rare cases where this problem occurs (that is, shoddy JSPs)? Finally, is this same technique used in Tomcat 6.x and is the same workaround available? Thanks, - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAky19ugACgkQ9CaO5/Lv0PBRFACgistXA3G55lUJ9hE4Yp43Op/9 CJAAn03E1e/3lxhu6sz3d+/Cj1H+hiP0 =3ZzW -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 5.5.25 | Memory leak in Web Application
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Anurag, On 10/12/2010 5:47 PM, Anurag Kapur wrote: I have probably attached an incomplete snapshot of the memory utilization graph. I'm only looking at what is on your blog. That graph looks good. Perhaps you could update your blog with a graph that illustrates your conclusion. I have done a few tests with the JVM argument suggested in the bug report. The initial tests look good. Good. Also, yes, ours being a content management application, we have large body tags. We use several tag libraries that come with the CMS implementation to fetch content from the CMS. Usually CMS applications /manage/ content, rather than hard-coding it. What kind of huge body tags do you have? Good luck, - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAky193MACgkQ9CaO5/Lv0PC8ZwCeOCRg4pGeK7QjTVkeV7tNJUcD GBgAnRO63f4ed/ZRewJcgLC7FwJwLg20 =ZC7S -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 5.5.25 | Memory leak in Web Application
On 13/10/2010 19:14, Christopher Schultz wrote: Mark, On 10/12/2010 5:28 PM, Mark Thomas wrote: On 12/10/2010 19:45, Christopher Schultz wrote: markt marked this bug as FIXED, but I see no indication of a resolution. Perhaps that was meant to be WONTFIX? Nope. I meant FIXED. As in There is now an option you can use to disable this behaviour if you don't like it. Gotcha. It wasn't clear from the comment that the fix was somewhat of a workaround. Any reason those buffers never release their memory? I believe performance. Was the complexity of the proposed fix deemed too high for the rare cases where this problem occurs (that is, shoddy JSPs)? I believe so but to be honest I haven't looked at the proposed fix for quite some time. Finally, is this same technique used in Tomcat 6.x and is the same workaround available? 6.0.x and 7.0.x. Same workaround. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 5.5.25 | Memory leak in Web Application
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Anurag, On 10/11/2010 12:30 PM, Anurag Kapur wrote: I have added my problem statement with Images to by blog here: http://anuragkapur-techbytes.blogspot.com/2010/10/tomcat-5527-memory-leak-in-escenic-cms.html The memory profile you show there appears to have a good GC curve: tenured generation stabilizes over time, and eden space sees regular activity that is eventually collected. The process repeats itself as GCs take place. I see no problem, here, other than your OOME. I disagree with your analysis that this heap utilization graph suggests a memory leak type pattern. The objects holding references to the character arrays that ultimately consume all the memory are of type org.apache.jasper.runtime.BodyContentImpl as indicated in the object reference tree below: There was a bug reported in Tomcat 5.5.9 which says The problem is that this huge array never gets reset due to the object pooling implementation in Jasper (JspFactoryImpl maintains a pool of PageContextImpl objects. Each PageContextImpl object maintains an array of BodyContentImpl objects), so the memory it consumed is never returned to the heap. https://issues.apache.org/bugzilla/show_bug.cgi?id=37793 We do not use the suggest JVM argument in our Tomcat 5.5.27 JVM -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true Could adding this argument to the Tomcat JVM resolve the problem? Certainly you could try it. It appears from this bug report that this should only be a problem with JSPs where there are large body tags. Are you using such body tags? If so, then this may help. Otherwise, it probably will not help. markt marked this bug as FIXED, but I see no indication of a resolution. Perhaps that was meant to be WONTFIX? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAky0rLIACgkQ9CaO5/Lv0PAGfwCeONuTmfs0mew6DLeYCgod5MKu JcAAn14OemGYLUtcpXWucCszxqVuQfvC =2G2E -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 5.5.25 | Memory leak in Web Application
On 12/10/2010 19:45, Christopher Schultz wrote: markt marked this bug as FIXED, but I see no indication of a resolution. Perhaps that was meant to be WONTFIX? Nope. I meant FIXED. As in There is now an option you can use to disable this behaviour if you don't like it. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 5.5.25 | Memory leak in Web Application
Thanks for your inputs. I have probably attached an incomplete snapshot of the memory utilization graph. What happens as more time progresses is that the utilization keeps increasing and the amount of heap that gets collected keeps decreasing. After some time the system starts doing Full GCs every second and the application stops responding. The graph I had previously attached was from an instance where I simulated the behaviour in our test env but did not wait for the system to fail over. I stopped the test since I was seeing the same pattern which ultimately falls over. I have done a few tests with the JVM argument suggested in the bug report. The initial tests look good. The memory utilization seems healthy. The heap, unlike the snapshot I sent previously, seems to get collected by pretty much the same amount after each Full GC cycle. So there is no upward trend in the utilization as seen in the previous snapshots. Also, yes, ours being a content management application, we have large body tags. We use several tag libraries that come with the CMS implementation to fetch content from the CMS. I will be doing some more thorough tests tomorrow before reaching a conclusion. Thank you for all your inputs so far. I will share my final findings in-case they can be useful to someone in the future with a similar issue. On Tue, Oct 12, 2010 at 10:28 PM, Mark Thomas ma...@apache.org wrote: On 12/10/2010 19:45, Christopher Schultz wrote: markt marked this bug as FIXED, but I see no indication of a resolution. Perhaps that was meant to be WONTFIX? Nope. I meant FIXED. As in There is now an option you can use to disable this behaviour if you don't like it. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 5.5.25 | Memory leak in Web Application
Thanks Chris and Chuck for your help so far. I have added my problem statement with Images to by blog here: http://anuragkapur-techbytes.blogspot.com/2010/10/tomcat-5527-memory-leak-in-escenic-cms.html http://anuragkapur-techbytes.blogspot.com/2010/10/tomcat-5527-memory-leak-in-escenic-cms.html On Sat, Oct 9, 2010 at 12:13 AM, Caldarale, Charles R chuck.caldar...@unisys.com wrote Attachments are stripped: there is no image to view, here. I have attached the heap usage graph as a file this time (heap_usage.jpg) Please read Chris' statement again: attachments are stripped. If you want us to look at the pictures, you'll have to post them in some public location on the web. Heap utilization is like this: http://3.bp.blogspot.com/_Oq9OlP-8ap0/TLMAno5Z2lI/CPA/pUr70osY3HU/s1600/heap_usage.jpg An object call reference tree snapshot that I wanted to show you is here: http://2.bp.blogspot.com/_Oq9OlP-8ap0/TLMBRLiUzTI/CPE/ogO2HI2uYig/s1600/object_reference_tree.jpg Or can the address of the object change in the lifetime of the tomcat jvm? An object can move on each GC. So how should I determine if an object (HashMap entry or character array) that I saw in the first heap dump exists in the next hump dump I obtained after the 4th Full GC? Struggling to determine the root cause. Any further pointers would be much appreciated. Thanks Anurag
Re: Tomcat 5.5.25 | Memory leak in Web Application
The objects holding references to the character arrays that ultimately consume all the memory are of type org.apache.jasper.runtime.BodyContentImpl as indicated in the object reference tree below: There was a bug reported in Tomcat 5.5.9 which says The problem is that this huge array never gets reset due to the object pooling implementation in Jasper (JspFactoryImpl maintains a pool of PageContextImpl objects. Each PageContextImpl object maintains an array of BodyContentImpl objects), so the memory it consumed is never returned to the heap. https://issues.apache.org/bugzilla/show_bug.cgi?id=37793 We do not use the suggest JVM argument in our Tomcat 5.5.27 JVM -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true Could adding this argument to the Tomcat JVM resolve the problem? -Anurag On Mon, Oct 11, 2010 at 1:55 PM, Anurag Kapur anuragka...@gmail.com wrote: Thanks Chris and Chuck for your help so far. I have added my problem statement with Images to by blog here: http://anuragkapur-techbytes.blogspot.com/2010/10/tomcat-5527-memory-leak-in-escenic-cms.html On Sat, Oct 9, 2010 at 12:13 AM, Caldarale, Charles R chuck.caldar...@unisys.com wrote Attachments are stripped: there is no image to view, here. I have attached the heap usage graph as a file this time (heap_usage.jpg) Please read Chris' statement again: attachments are stripped. If you want us to look at the pictures, you'll have to post them in some public location on the web. Heap utilization is like this: http://3.bp.blogspot.com/_Oq9OlP-8ap0/TLMAno5Z2lI/CPA/pUr70osY3HU/s1600/heap_usage.jpg An object call reference tree snapshot that I wanted to show you is here: http://2.bp.blogspot.com/_Oq9OlP-8ap0/TLMBRLiUzTI/CPE/ogO2HI2uYig/s1600/object_reference_tree.jpg Or can the address of the object change in the lifetime of the tomcat jvm? An object can move on each GC. So how should I determine if an object (HashMap entry or character array) that I saw in the first heap dump exists in the next hump dump I obtained after the 4th Full GC? Struggling to determine the root cause. Any further pointers would be much appreciated. Thanks Anurag - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 5.5.25 | Memory leak in Web Application
On 11/10/2010 17:30, Anurag Kapur wrote: The objects holding references to the character arrays that ultimately consume all the memory are of type org.apache.jasper.runtime.BodyContentImpl as indicated in the object reference tree below: There was a bug reported in Tomcat 5.5.9 which says The problem is that this huge array never gets reset due to the object pooling implementation in Jasper (JspFactoryImpl maintains a pool of PageContextImpl objects. Each PageContextImpl object maintains an array of BodyContentImpl objects), so the memory it consumed is never returned to the heap. https://issues.apache.org/bugzilla/show_bug.cgi?id=37793 We do not use the suggest JVM argument in our Tomcat 5.5.27 JVM -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true Could adding this argument to the Tomcat JVM resolve the problem? I suspect it would only work if the app has JSPs which were pre-compiled against 5.5.9, rather than a newer version. p 0x62590808.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature
Re: Tomcat 5.5.25 | Memory leak in Web Application
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Anurag, On 10/8/2010 3:15 PM, Anurag Kapur wrote: _Problem Statement_ Memory leak in web application running on Tomcat _Solution Statement_ Fix your memory leak _System Information_ Tomcat 5.5.27 This statement plus the subject line are confusing me. _Observations and Investigation Done_ Under load, the web application slowly runs out of heap space. The heap utilization graph suggests a memory leak type pattern. Attachments are stripped: there is no image to view, here. Heap dumps were gathered to determine the root cause and observations are as below from the dumps: 1. HashMap entries from the below object reference tree seem to consume over 80% of the used tenured generation space. org/apache/jasper/compiler/JspRuntimeContext Java/util/Collections$SynchronizedMap Java/util/HashMap Java/util/HashMap$Entry If this doesn't grow, then it doesn't sound like a memory leak. 2. Heap dumps gathered during different times and after a Full GC always indicate 100 entries This indicates zero growth, so I would guess no memory leak. 3. The number of objects referenced by each HashMap entry vary between 2-3 1. These are either a String and JspServletWrapper object or 2. A String, JspServletWrapper and _another HashMap Entry_. This call reference tree of HashMap entries referenced by another HashMap entry can repeat to a depth of approximately 8-10 nodes That sounds weird. The above is indicated in an object reference tree obtained after analyzing the Heap dump as shown below 4. The maximum percentage of the memory occupied by the HashMap entry object is by a character array that seems like the JSP servlet response. HTML response (tags with content) can be seen in the character array Does this response text stay in memory indefinitely? When you say it's the response, is it the actual response to a particular client, or is it just static text from the JSP source? If the latter, this is completely expected: the JSP compiles-in all static text. _Questions_ 1. I am stuck at this point and have run out of ideas on how to get to the root cause of this issue. Do you have any ideas/suggestions to help identify the root cause? Finding the roots to those references should help. For instance, what page is holding on to all those entries? It's possible that the JSPRuntimeContext requires these data even after the compiler has compiled the code. 2. I google and found the following interesting links 1. http://www.mail-archive.com/d...@tomcat.apache.org/msg03395.html Is there any know issues, configuration that causes a memory leak by caching of servlet responses in the container and not flushing the cached objects? The above seems to be a complaint about how JSPs work: they are translated into .java files, then compiled and loaded. These loaded class files contain lots of static strings. There's nothing to be done about this. 2. http://mail-archives.apache.org/mod_mbox/tomcat-users/200303.mbox/%3c000c01c2ee2c$41c079a0$03b69...@garethdqw0t9if%3e We use a lot of JSTL in our web application. Are there any known issues around memory issues as indicated on the link? This one is seriously old: 2003 was a long time ago (even older than your version of Tomcat). I don't think that post is relevant anymore. You should consider upgrading. Check the changelog (http://tomcat.apache.org/tomcat-5.5-doc/changelog.html) for changes to Jasper between your version (5.5.25 or 5.5.27) and the latest (5.5.30). You also might consider upgrading to Tomcat 6.0. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkyviC8ACgkQ9CaO5/Lv0PCHQQCeKH/X4Ee4nN1Z3x19wCNkWW3Q pf0An2e+6trJtZezARIhx3GXifLUVpWc =Id6K -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 5.5.25 | Memory leak in Web Application
Thanks for your response Chris. On Fri, Oct 8, 2010 at 10:07 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Anurag, On 10/8/2010 3:15 PM, Anurag Kapur wrote: _Problem Statement_ Memory leak in web application running on Tomcat _Solution Statement_ Fix your memory leak That's the plan :) _System Information_ Tomcat 5.5.27 This statement plus the subject line are confusing me. Apologies. I have made a typo in the subject line. I am using Tomcat 5.5.27 _Observations and Investigation Done_ Under load, the web application slowly runs out of heap space. The heap utilization graph suggests a memory leak type pattern. Attachments are stripped: there is no image to view, here. I have attached the heap usage graph as a file this time (heap_usage.jpg) Heap dumps were gathered to determine the root cause and observations are as below from the dumps: 1. HashMap entries from the below object reference tree seem to consume over 80% of the used tenured generation space. org/apache/jasper/compiler/JspRuntimeContext Java/util/Collections$SynchronizedMap Java/util/HashMap Java/util/HashMap$Entry If this doesn't grow, then it doesn't sound like a memory leak. The number of entries under the HashMap referenced by JspRuntimeContent - Collections$SynchronizedMap does not increase. However the size of these 100 entries increases with time. In my last test, after the first full GC the size of this HashMap was around 350MB. With time and under load, this increased. Next heap dump gathered after the 4th Full GC showed that the size of this hash map had doubled to around 730MB. The number of entries immediately under this HashMap however remained constant. 2. Heap dumps gathered during different times and after a Full GC always indicate 100 entries This indicates zero growth, so I would guess no memory leak. Yes, the number of immediate entries in the Hashmap remained constant however the size these occupy in the heap increased. 3. The number of objects referenced by each HashMap entry vary between 2-3 1. These are either a String and JspServletWrapper object or 2. A String, JspServletWrapper and _another HashMap Entry_. This call reference tree of HashMap entries referenced by another HashMap entry can repeat to a depth of approximately 8-10 nodes That sounds weird. To make this a little clear, I have attached a screen shot of the object reference tree from IBM Heap analyzer I used on the heap dumps obtained. (object_reference_tree.jpg) The above is indicated in an object reference tree obtained after analyzing the Heap dump as shown below 4. The maximum percentage of the memory occupied by the HashMap entry object is by a character array that seems like the JSP servlet response. HTML response (tags with content) can be seen in the character array Does this response text stay in memory indefinitely? When you say it's the response, is it the actual response to a particular client, or is it just static text from the JSP source? If the latter, this is completely expected: the JSP compiles-in all static text I am having difficulties at this stage determining this with 100% confidence. I am mostly seeing static HTML tags in the character array that would be coming from the static JSP source. However I have also seen some references to dynamic content that is sent back to the client in these charcter arrays. There is atleast 1 such character array in each of these 100 has map entries. Sometime more in cases where there are more hash map entries within an entry. I have not seen the size of all these charcter arrays but a few are as big as 7MB. . _Questions_ 1. I am stuck at this point and have run out of ideas on how to get to the root cause of this issue. Do you have any ideas/suggestions to help identify the root cause? Finding the roots to those references should help. For instance, what page is holding on to all those entries? It's possible that the JSPRuntimeContext requires these data even after the compiler has compiled the code. I can also see the name of the custom JSP files in the object call reference trees under most of these 100 hash map entries. However some of them are relatively simple and I am unable to think of a reason why such a JSP would hold reference in the heap causing the leak. As you suggested, I am now trying to search for objects I see in the heap dump obtained after the first Full GC in the dump obtained after the 4th Full GC. Thanks for your inputs so far. Any other inputs while I do the investigation highlighted above? 2. I google and found the following interesting links 1. http://www.mail-archive.com/d...@tomcat.apache.org/msg03395.html Is there any
Re: Tomcat 5.5.25 | Memory leak in Web Application
On Fri, Oct 8, 2010 at 11:04 PM, Anurag Kapur anuragka...@gmail.com wrote: Thanks for your response Chris. On Fri, Oct 8, 2010 at 10:07 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Anurag, On 10/8/2010 3:15 PM, Anurag Kapur wrote: _Problem Statement_ Memory leak in web application running on Tomcat _Solution Statement_ Fix your memory leak That's the plan :) _System Information_ Tomcat 5.5.27 This statement plus the subject line are confusing me. Apologies. I have made a typo in the subject line. I am using Tomcat 5.5.27 _Observations and Investigation Done_ Under load, the web application slowly runs out of heap space. The heap utilization graph suggests a memory leak type pattern. Attachments are stripped: there is no image to view, here. I have attached the heap usage graph as a file this time (heap_usage.jpg) Heap dumps were gathered to determine the root cause and observations are as below from the dumps: 1. HashMap entries from the below object reference tree seem to consume over 80% of the used tenured generation space. org/apache/jasper/compiler/JspRuntimeContext Java/util/Collections$SynchronizedMap Java/util/HashMap Java/util/HashMap$Entry If this doesn't grow, then it doesn't sound like a memory leak. The number of entries under the HashMap referenced by JspRuntimeContent - Collections$SynchronizedMap does not increase. However the size of these 100 entries increases with time. In my last test, after the first full GC the size of this HashMap was around 350MB. With time and under load, this increased. Next heap dump gathered after the 4th Full GC showed that the size of this hash map had doubled to around 730MB. The number of entries immediately under this HashMap however remained constant. 2. Heap dumps gathered during different times and after a Full GC always indicate 100 entries This indicates zero growth, so I would guess no memory leak. Yes, the number of immediate entries in the Hashmap remained constant however the size these occupy in the heap increased. 3. The number of objects referenced by each HashMap entry vary between 2-3 1. These are either a String and JspServletWrapper object or 2. A String, JspServletWrapper and _another HashMap Entry_. This call reference tree of HashMap entries referenced by another HashMap entry can repeat to a depth of approximately 8-10 nodes That sounds weird. To make this a little clear, I have attached a screen shot of the object reference tree from IBM Heap analyzer I used on the heap dumps obtained. (object_reference_tree.jpg) The above is indicated in an object reference tree obtained after analyzing the Heap dump as shown below 4. The maximum percentage of the memory occupied by the HashMap entry object is by a character array that seems like the JSP servlet response. HTML response (tags with content) can be seen in the character array Does this response text stay in memory indefinitely? When you say it's the response, is it the actual response to a particular client, or is it just static text from the JSP source? If the latter, this is completely expected: the JSP compiles-in all static text I am having difficulties at this stage determining this with 100% confidence. I am mostly seeing static HTML tags in the character array that would be coming from the static JSP source. However I have also seen some references to dynamic content that is sent back to the client in these charcter arrays. There is atleast 1 such character array in each of these 100 has map entries. Sometime more in cases where there are more hash map entries within an entry. I have not seen the size of all these charcter arrays but a few are as big as 7MB. . _Questions_ 1. I am stuck at this point and have run out of ideas on how to get to the root cause of this issue. Do you have any ideas/suggestions to help identify the root cause? Finding the roots to those references should help. For instance, what page is holding on to all those entries? It's possible that the JSPRuntimeContext requires these data even after the compiler has compiled the code. I can also see the name of the custom JSP files in the object call reference trees under most of these 100 hash map entries. However some of them are relatively simple and I am unable to think of a reason why such a JSP would hold reference in the heap causing the leak. As you suggested, I am now trying to search for objects I see in the heap dump obtained after the first Full GC in the dump obtained after the 4th Full GC. I did a quick check on the heap dumps. I noted the address of the character array referenced by the largest entry in the hash map from the first dump. I
RE: Tomcat 5.5.25 | Memory leak in Web Application
From: Anurag Kapur [mailto:anuragka...@gmail.com] Subject: Re: Tomcat 5.5.25 | Memory leak in Web Application Attachments are stripped: there is no image to view, here. I have attached the heap usage graph as a file this time (heap_usage.jpg) Please read Chris' statement again: attachments are stripped. If you want us to look at the pictures, you'll have to post them in some public location on the web. Or can the address of the object change in the lifetime of the tomcat jvm? An object can move on each GC. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
