Re: Registry problems followed by OutOfMemory errors

[Asim Alp]

Thanks for the quick reply!

How can I turn off JMX request registration?  I tried to find it in the
Tomcat documentation, but all I could find was the MBean Descriptor How
To which wasn't very helpful.  I doubt that we need it.

I will look into the resource problem.  I doubt that we're running out
of threads, because 1000 of them should be more than enough.  Another
resource problem could be the number of SQL connections, but on our SQL
Side (MySQL), max_connections is set to 5000 which is about 7 times
more than we need.  Number of file descriptors might be a problem.  For
each web application, we have to load properties files and these files
are automatically checked once every minute for possible updates.  How
can I configure the number of file descriptors?

Thank you very much!

Asim

On Dec 10, 2004, at 11:02 AM, Shapira, Yoav wrote:

Hi,
Do you need JMX request registration? If not, turn it off by setting it
to false in the configuration file. That'll save you some memory by
itself.

Your OOME is the root cause, so if you fix that the hundreds of others
errors will likely go away.  The OOME, in turn, is probably caused not
because you're out of heap memory, but because you're out of another
resource, such as file descriptors or threads.

Yoav Shapira http://www.yoavshapira.com

-----Original Message-----
From: Asim Alp [mailto:[EMAIL PROTECTED]
Sent: Friday, December 10, 2004 10:59 AM
To: Tomcat Users List
Subject: Registry problems followed by OutOfMemory errors

Here is our configuration:

Windows Server 2003
Apache 2.0.49 (Win32) mod_ssl/2.0.51 OpenSSL/0.9.7d DAV/2 mod_jk/1.2.6
Tomcat 5.5.4
sun jdk 1.5.0-b64

We have a 2 node tomcat cluster each running with the -Xms128m
-Xmx1024m options.  We have the following thread options set in the
AJP/1.3 connector:
maxThreads="1000" minSpareThreads="50" maxSpareThreads="300"

And the following corresponding values in our workers.properties (same
for each tomcat)
worker.tomcat1.type=ajp13
worker.tomcat1.cachesize=300
worker.tomcat1.cache_timeout=60
worker.worker1.local_worker=1
worker.tomcat1.lbfactor=1
worker.tomcat1.connect_timeout = 1000
worker.tomcat1.prepost_timeout = 1000
worker.tomcat1.reply_timeout = 7000
...
worker.loadbalancer.local_worker_only=0

We have a heavily loaded database application running.  Same
application runs on about 180 different virtual hosts on each Tomcat.

We did profiling with JProfile and couldn't find any memory leaks in
our application.  Each Tomcat works perfect for about 8 to 10 hours,
then all of a sudden, they start hanging (not necessarily at the same
time).  We monitor our heap memory very closely and we usually have
enough FREE memory (more than 25%) when the following errors occur:

First, we get a couple of SEVERE registering errors:

<record>
 <date>2004-12-09T18:19:51</date>
 <millis>1102634391333</millis>
 <sequence>270</sequence>
 <logger>org.apache.commons.modeler.Registry</logger>
 <level>SEVERE</level>
 <class>org.apache.commons.modeler.Registry</class>
 <method>registerComponent</method>
 <thread>44</thread>
 <message>Error registering
Catalina:type=RequestProcessor,worker=jk-8009,name=JkRequest2291</
message>
 <exception>
   <message>javax.management.InstanceAlreadyExistsException:
Catalina:type=RequestProcessor,worker=jk-8009,name=JkRequest2291</
message>
   <frame>
     <class>com.sun.jmx.mbeanserver.RepositorySupport</class>
     <method>addMBean</method>
     <line>452</line>
   </frame>
   <frame>

<class>com.sun.jmx.interceptor.DefaultMBeanServerInterceptor</class>
     <method>internal_addObject</method>
     <line>1410</line>
   </frame>
   ....
 </exception>
</record>
<record>
 <date>2004-12-09T18:19:51</date>
 <millis>1102634391333</millis>
 <sequence>271</sequence>
 <logger>org.apache.jk.common.ChannelSocket</logger>
 <level>WARNING</level>
 <class>org.apache.jk.common.ChannelSocket</class>
 <method>registerRequest</method>
 <thread>44</thread>
 <message>Error registering request</message>
</record>

Followed by a couple of java.lang.OutOfMemoryError: PermGen space
messages
<record>
 <date>2004-12-09T21:48:25</date>
 <millis>1102646905849</millis>
 <sequence>294</sequence>
 <logger>StandardWrapper[/apps:jsp]</logger>
 <level>SEVERE</level>
 <class>org.apache.catalina.core.StandardWrapperValve</class>
 <method>invoke</method>
 <thread>47</thread>
 <message>Servlet.service() for servlet jsp threw exception</message>
 <exception>
   <message>java.lang.OutOfMemoryError: PermGen space</message>
 </exception>
</record>
...

Followed by HUNDREDS (basically for each thread) of Error
unregistering mbean messages
<record>
 <date>2004-12-09T21:53:29</date>
 <millis>1102647209630</millis>
 <sequence>302</sequence>
 <logger>org.apache.commons.modeler.Registry</logger>
 <level>SEVERE</level>
 <class>org.apache.commons.modeler.Registry</class>
 <method>unregisterComponent</method>
 <thread>20</thread>
 <message>Error unregistering mbean </message>
 <exception>
   <message>javax.management.RuntimeOperationsException: Object name
cannot be null</message>
   <frame>

<class>com.sun.jmx.interceptor.DefaultMBeanServerInterceptor</class>
     <method>isRegistered</method>
     <line>545</line>
   </frame>
   <frame>
     <class>com.sun.jmx.mbeanserver.JmxMBeanServer</class>
     <method>isRegistered</method>
     <line>619</line>
   </frame>
   <frame>
     <class>org.apache.commons.modeler.Registry</class>
     <method>unregisterComponent</method>
     <line>642</line>
   </frame>
   ...
 </exception>
</record>

Any ideas?  Is this something to do with our thread counts
(maxThreads="1000" minSpareThreads="50" maxSpareThreads="300")?  If
so, how can we determine these numbers for fastest performance.  We
get about 5 hits every second and we want our Tomcats to serve static
files as well (such as image files), so we want to make sure that we
have enough threads.  It's very important for our pages to load fast
on the client side.

Thank you very much!  Any help would be greatly appreciated!

Asim

---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you.

---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]