On 25.09.2009 09:12, Darren Kukulka wrote: > There are no OOME messages in the logs...the application still runs, > albeit very slowly...GC is not kicking in any more frequently...I was > under the impression that GC doesn't touch OldGen though...?
It does. So it's strange you know your memory is exhausted and that's the reason for the slowness, although if it were exhausted, you would either get an OutOfMemoryError or GC has to kick in. The picture is not complete here. Regards, Rainer > -----Original Message----- > From: Rainer Jung [mailto:rainer.j...@kippdata.de] > Sent: 25 September 2009 08:09 > To: Tomcat Users List > Subject: Re: Clustering Question... > > On 25.09.2009 09:01, Darren Kukulka wrote: >> Thanks for the suggestions Chris. >> >> Unfortunately, the memory that was exhausted was the OldGen heap area, > not PermGen, which doesn't show up in the Catalina log. >> >> The heap allocation is quite hefty as this is a 64-bit > environment...we need to get our developers to look into the application > behaviour, but in the meantime I was looking for a way of dealing with > the problem. > > So did you get an OutOfMemoryError, or was there only a shortage of > memory resulting in more or less continuous garbage collection runs? > > Regards, > > Rainer > >> -----Original Message----- >> From: Christopher Schultz [mailto:ch...@christopherschultz.net] >> Sent: 24 September 2009 19:59 >> To: Tomcat Users List >> Subject: Re: Clustering Question... >> >> Darren, >> >> On 9/24/2009 10:21 AM, Darren Kukulka wrote: >>> In a 2-node scenario, where both nodes are configured identically and >>> load balanced via Apache based on availability, how can we configure > the >>> cluster to deal with situations where one node has exhausted its Old > Gen >>> heap allocation. >> >> Hmm... this is a situations that is difficult to detect using remote >> code (like mod_jk). >> >>> In such situations we've observed that the application being served > by >>> the cluster slow down considerably. I can understand why this would > be >>> the case for sessions on the degraded node, but why would sessions on >>> the good node suffer? >> >> Are you using session replication? If so, the "good" Tomcat may be >> slowing down attempting to replicate session changes to the "damaged" >> Tomcat that is either not responding, or responding slowly, or >> responding in confusing ways. >> >>> How can we modify our configuration to deal with such occurrences > more >>> effectively? >> >> After we had some trouble with OOMEs in production (legit ones, >> actually: we just needed more heap), I implemented a quick-and-dirty >> OOME checker. All it does is "grep OutOfMemoryError catalina.out" and, >> if found, sends an email to someone. >> >> Instead of emailing, you could have your OOME checker actually shut > down >> (or forceably terminate) the damaged Tomcat, and then the cluster > should >> stabilize. With only two nodes, this might be a problem, as the good >> Tomcat will take over and might, under the new load of 100% of your >> traffic, experience its own pergmen exhaustion and also be shut down. >> >> You should consider adjusting your pergmen allocation (duh!) as well > as >> perhaps your heap allocation as well. >> >> -chris > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > Connaught plc is a FTSE 250 company. We are the UK's leading provider of > integrated services operating in the compliance, social housing and > public sector markets. > > Please visit our website to see a full list of Connaught's Registered > Companies www.connaught.plc.uk/group/aboutconnaught/registeredcompanies > > Disclaimer: > > The information transmitted is intended only for the person or entity to > which it is addressed and may contain confidential and/or privileged > material. Any review, retransmission, dissemination or other use of, or > taking of any action in reliance upon, this information by persons or > entities other than the intended recipient is prohibited. If you > received this in error, please contact the sender and delete this > message. > > Connaught plc, Head Office 01392 444546 --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
