We've seen errors where individual queries are improperly bounded or unbounded allowing requests to attempt to retrieve unreasonable amounts of data. A single request like this can take an otherwise stable system and throw it into all sorts of bad behaviors and error conditions.
To assist with this debugging you may try to run your jvm with gc logging turned on, and log the http requests coming in. If you then have an OOM crash you can correlate the spike in memory allocation (on a timeline) with certain requests that may have caused it. you can use tools like gcviewer to post-process your gc log files to see what was going on with your memory over time. -John On 2/7/07, Sharon French <[EMAIL PROTECTED]> wrote:
> Uhhh - no. The JVM will not return an OOME until the GC > is completely finished and the request still cannot be > satisfied. Even with concurrent GC, failing allocation > requests are suspended until a full GC has completed. > The exception to the above is if a request is made that > could never be satisfied; that will be given an OOME > immediately. This is very helpful to know. So if after receiving such OOMEs, I run jmap -heap and there is plenty of memory, I'm assuming that whomever was sucking up the memory has either given it back or died off - right? (The dying off would line up with fact that the number of processors tomcat says he's started is less than the number of threads I currently see running.) Now it's down to how do I catch the culprit? Since I don't see steady memory growth, I believe that there must be some error condition that is handled improperly that throws us into some sort of memory cosumption craziness. Since by the time the symptom is present, there is plenty of memory available, I'm having trouble catching the case. Any suggestions? >> After an OOM is thrown, the JVM is basically completely hosed. > Definitely not true. The JVM is still functional, although the app > (Tomcat, in this case) may not be able to proceed. This also lines up with my observations. The first several OOMEs don't seem to trip us up, however at some point we hit a threshold that does hose us. Given that after such an error occurs, we have plently of memory, and plenty of threads sitting in await(), any ideas on why the connections to apache end up in SYN_RECV? Am I right in thinking that this indicates that the tomcat listener thread has accepted the connection but then fails to pass it off to a processor thread to handle? Or does the pass off happen but for some reason the socket is hosed? Any thoughts here? > > If so, how do we combat this behavior? > > Fix resource leaks, allocate an appropriate amount of memory, > and limit resource use to avoid OOMs in the first place. > However, the 64m given to the PermGen is rather small for an app server. We don't seem to ever get to a usage of > 20M for PermGen, and the OutOfMemoryErrors always indicate "java heap space". Is this error message reliable? Can I count on the fact that it's really heap space I'm out of (I've seen reports of this indicating other problems, but all seem to be pre-1.5 jvm). Thanks for the help thus far! --------------------------------------------------------------------- To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
-- John Hayward
