Re: Memory Leak(?) causing tomcat to store 57610801 tomcat objects in ONE request
I'd like to thank everyone involved, as many of you suspected it turned out to be an infinite loop in a component of our webapp, i've tracked the issue down to the places where it is most possible and gave it over to the owner of the component. Thank all of you for great help finding it ! Leon On Thu, Jan 15, 2009 at 4:40 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Leon, Leon Rosenberg wrote: The last thing I can imagine is that the request somehow gets redirected or forwarded infinitely during processing, but is not leaving tomcat entirely and just reenters the processing, keeping the outer request object, where the mimeheader fields are stored. Is something like this possible? Unlikely. Here's why: 1. A redirect would end the request, freeing all these objects. You'd probably also notice that many requests were being made. Also, most browsers give up after a certain number of redirects in a row. 2. Internal forwards (achieved by calling RequestDispatcher.forward) result in a deeper call stack. Thus, infinite forwards would likely result in a StackOverflowError. My guess is that you have an infinite loop that might not look like a loop (maybe the loop body is several screens long or whatever). What do thread dumps look like for this thread? Presumably, this request has a thread stuck to it which is actually executing code. Chuck's suggestion to modify MimeHeaders.createHeader is a good one, though hacking and recompiling Tomcat isn't my idea of a fun time. Instead, I'd write a filter that wraps the response with a HttpServletResponseWrapper whose setHeader/addHeader methods have been overridden to spit out a timestamp and maybe even a stack trace. I think you'll be able to find out some good information from that. Good luck, - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAklvVLcACgkQ9CaO5/Lv0PBgyQCgvBSezsoZB64/StEtqeRxX+SI GloAmQFZai+UtfLqTX5mqEZDYJWmqZAk =ojk9 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Memory Leak(?) causing tomcat to store 57610801 tomcat objects in ONE request
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Leon, Leon Rosenberg wrote: The last thing I can imagine is that the request somehow gets redirected or forwarded infinitely during processing, but is not leaving tomcat entirely and just reenters the processing, keeping the outer request object, where the mimeheader fields are stored. Is something like this possible? Unlikely. Here's why: 1. A redirect would end the request, freeing all these objects. You'd probably also notice that many requests were being made. Also, most browsers give up after a certain number of redirects in a row. 2. Internal forwards (achieved by calling RequestDispatcher.forward) result in a deeper call stack. Thus, infinite forwards would likely result in a StackOverflowError. My guess is that you have an infinite loop that might not look like a loop (maybe the loop body is several screens long or whatever). What do thread dumps look like for this thread? Presumably, this request has a thread stuck to it which is actually executing code. Chuck's suggestion to modify MimeHeaders.createHeader is a good one, though hacking and recompiling Tomcat isn't my idea of a fun time. Instead, I'd write a filter that wraps the response with a HttpServletResponseWrapper whose setHeader/addHeader methods have been overridden to spit out a timestamp and maybe even a stack trace. I think you'll be able to find out some good information from that. Good luck, - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAklvVLcACgkQ9CaO5/Lv0PBgyQCgvBSezsoZB64/StEtqeRxX+SI GloAmQFZai+UtfLqTX5mqEZDYJWmqZAk =ojk9 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Memory Leak(?) causing tomcat to store 57610801 tomcat objects in ONE request
A small update on the matter: I've managed to track it down, but the details are rather application specific, so I don't want to bore you, but the scenario looks as follows: - something posts a rather normal looking request to an url that is yet supported but obsolete (should be actually served by another pool of tomcats). - this request probably comes without cookies - the webapp tries to set a vi - visitor id - cookie till infinity, the requests lasts and lasts and lasts (longest detected was 25h). - at one point the amount of cookies in the response is larger than the free memory, so old gen is running full, tomcat freezes, asta la vista... Now, the problem is, that there is no evidence in the thread dumps of infinite loops or anything. Maybe because most thread dumps are taken at the moment when the tomcat is frozen and aren't reliable anymore. However it it were recursion, it would be aborted faster (stackoverflow) or visible in the thread dump, but it's not. The other explanation would be a direct infinite loop in the application, but that should also be visible in the thread dump and its not. The last thing I can imagine is that the request somehow gets redirected or forwarded infinitely during processing, but is not leaving tomcat entirely and just reenters the processing, keeping the outer request object, where the mimeheader fields are stored. Is something like this possible? The (probably most outer request) in manager which was found in all cases is always shown as S(ervice), 0Kb size, and being processed very very long time... Any ideas, how an infinite redirection/forward loop can be created without leaving tomcat? regards Leon - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Memory Leak(?) causing tomcat to store 57610801 tomcat objects in ONE request
From: Leon Rosenberg [mailto:rosenberg.l...@googlemail.com] Subject: Re: Memory Leak(?) causing tomcat to store 57610801 tomcat objects in ONE request The other explanation would be a direct infinite loop in the application, but that should also be visible in the thread dump and its not. Not necessarily; there might be a retry loop or something else more subtle inside the webapp. Since you know where the headers get remembered: org.apache.tomcat.util.http.MimeHeaders#createHeader you could put some debug code in there to display the current call stack. Don't think you'll be able to get to the Request object from there, but at least you'll know who's making the calls and you can add more debug code there. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Memory Leak(?) causing tomcat to store 57610801 tomcat objects in ONE request
2009/1/10 Caldarale, Charles R chuck.caldar...@unisys.com: From: Leon Rosenberg [mailto:rosenberg.l...@googlemail.com] Subject: Memory Leak(?) causing tomcat to store 57610801 tomcat objects in ONE request HOW can it actually happen that a response object contains 8.000.000 mime headers? Must be a very productive servlet... Sounds like you've got a rather ugly (and probably infinite) loop in your webapp. Might want to put a trap in org.apache.tomcat.util.http.MimeHeaders to catch generation of an excessive number of header fields. I agree with Charles. But, to note: the count of MimeHeaderField instances mentioned in the message is of the same range as the count of Cookie instances. Thus I think that all those header fields are derivatives of the enormous count of cookies that your application created. I remembered about the following incident from two months ago, when programming error caused a session to be reestablished many times in a row: http://www.nabble.com/multiple-Set-Cookie-headers-in-initial-http-response-td20437761.html (multiple Set-Cookie headers in initial http response). Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Memory Leak(?) causing tomcat to store 57610801 tomcat objects in ONE request
From: Leon Rosenberg [mailto:rosenberg.l...@googlemail.com] I was recently hunting what I thought to be a memory leak in our application. What happens is that the Old Gen Space is running full at once and then tomcat freezes because java is busy with Full GC all the time. I've managed to create a memory dump shortly before the crash and the top5 looked like that: 14481362 instances of class org.apache.tomcat.util.buf.ByteChunk 14471293 instances of class org.apache.tomcat.util.buf.CharChunk 14464658 instances of class org.apache.tomcat.util.buf.MessageBytes 7113296 instances of class org.apache.tomcat.util.http.MimeHeaderField 7080192 instances of class javax.servlet.http.Cookie Other classes looked as normal (I was making jmap histograms all the time and comparing). Now, the funny thing is that all that data was bound to ONE request. I checked the manager status page previously and it showed only one active request, but lasting more than 5 hours: S18757746 ms0 KB0 KBXXX.XXX.XXX.XXX www.site.com POST /searchboxright.html?linkId=XXX|SearchBoxRight|NA| HTTP/1.1 By analyzing the dump I found that all MimeHeaderFields were in one large array belonging to one response object, belonging to the request, with remote ip equal to the XXX.XXX.XXX.XXX ip of the request in the manager. I understand that sicne the request lasted 5 hours all the data went into old space filling it out and causing the problem, but HOW can it actually happen that a response object contains 8.000.000 mime headers? App/Tomcat stuck in a loop adding cookie headers? Do you have any thread dumps? Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Memory Leak(?) causing tomcat to store 57610801 tomcat objects in ONE request
Hi, I was recently hunting what I thought to be a memory leak in our application. What happens is that the Old Gen Space is running full at once and then tomcat freezes because java is busy with Full GC all the time. I've managed to create a memory dump shortly before the crash and the top5 looked like that: 14481362 instances of class org.apache.tomcat.util.buf.ByteChunk 14471293 instances of class org.apache.tomcat.util.buf.CharChunk 14464658 instances of class org.apache.tomcat.util.buf.MessageBytes 7113296 instances of class org.apache.tomcat.util.http.MimeHeaderField 7080192 instances of class javax.servlet.http.Cookie Other classes looked as normal (I was making jmap histograms all the time and comparing). Now, the funny thing is that all that data was bound to ONE request. I checked the manager status page previously and it showed only one active request, but lasting more than 5 hours: S18757746 ms0 KB0 KBXXX.XXX.XXX.XXX www.site.com POST /searchboxright.html?linkId=XXX|SearchBoxRight|NA| HTTP/1.1 By analyzing the dump I found that all MimeHeaderFields were in one large array belonging to one response object, belonging to the request, with remote ip equal to the XXX.XXX.XXX.XXX ip of the request in the manager. I understand that sicne the request lasted 5 hours all the data went into old space filling it out and causing the problem, but HOW can it actually happen that a response object contains 8.000.000 mime headers? regards Leon - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Memory Leak(?) causing tomcat to store 57610801 tomcat objects in ONE request
From: Leon Rosenberg [mailto:rosenberg.l...@googlemail.com] Subject: Memory Leak(?) causing tomcat to store 57610801 tomcat objects in ONE request HOW can it actually happen that a response object contains 8.000.000 mime headers? Must be a very productive servlet... Sounds like you've got a rather ugly (and probably infinite) loop in your webapp. Might want to put a trap in org.apache.tomcat.util.http.MimeHeaders to catch generation of an excessive number of header fields. There's an interesting comment in there: /* XXX XXX XXX Need a major rewrite */ - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
