Registry problems followed by OutOfMemory errors
Here is our configuration: Windows Server 2003 Apache 2.0.49 (Win32) mod_ssl/2.0.51 OpenSSL/0.9.7d DAV/2 mod_jk/1.2.6 Tomcat 5.5.4 sun jdk 1.5.0-b64 We have a 2 node tomcat cluster each running with the -Xms128m -Xmx1024m options. We have the following thread options set in the AJP/1.3 connector: maxThreads=1000 minSpareThreads=50 maxSpareThreads=300 And the following corresponding values in our workers.properties (same for each tomcat) worker.tomcat1.type=ajp13 worker.tomcat1.cachesize=300 worker.tomcat1.cache_timeout=60 worker.worker1.local_worker=1 worker.tomcat1.lbfactor=1 worker.tomcat1.connect_timeout = 1000 worker.tomcat1.prepost_timeout = 1000 worker.tomcat1.reply_timeout = 7000 ... worker.loadbalancer.local_worker_only=0 We have a heavily loaded database application running. Same application runs on about 180 different virtual hosts on each Tomcat. We did profiling with JProfile and couldn't find any memory leaks in our application. Each Tomcat works perfect for about 8 to 10 hours, then all of a sudden, they start hanging (not necessarily at the same time). We monitor our heap memory very closely and we usually have enough FREE memory (more than 25%) when the following errors occur: First, we get a couple of SEVERE registering errors: record date2004-12-09T18:19:51/date millis1102634391333/millis sequence270/sequence loggerorg.apache.commons.modeler.Registry/logger levelSEVERE/level classorg.apache.commons.modeler.Registry/class methodregisterComponent/method thread44/thread messageError registering Catalina:type=RequestProcessor,worker=jk-8009,name=JkRequest2291/ message exception messagejavax.management.InstanceAlreadyExistsException: Catalina:type=RequestProcessor,worker=jk-8009,name=JkRequest2291/ message frame classcom.sun.jmx.mbeanserver.RepositorySupport/class methodaddMBean/method line452/line /frame frame classcom.sun.jmx.interceptor.DefaultMBeanServerInterceptor/class methodinternal_addObject/method line1410/line /frame /exception /record record date2004-12-09T18:19:51/date millis1102634391333/millis sequence271/sequence loggerorg.apache.jk.common.ChannelSocket/logger levelWARNING/level classorg.apache.jk.common.ChannelSocket/class methodregisterRequest/method thread44/thread messageError registering request/message /record Followed by a couple of java.lang.OutOfMemoryError: PermGen space messages record date2004-12-09T21:48:25/date millis1102646905849/millis sequence294/sequence loggerStandardWrapper[/apps:jsp]/logger levelSEVERE/level classorg.apache.catalina.core.StandardWrapperValve/class methodinvoke/method thread47/thread messageServlet.service() for servlet jsp threw exception/message exception messagejava.lang.OutOfMemoryError: PermGen space/message /exception /record ... Followed by HUNDREDS (basically for each thread) of Error unregistering mbean messages record date2004-12-09T21:53:29/date millis1102647209630/millis sequence302/sequence loggerorg.apache.commons.modeler.Registry/logger levelSEVERE/level classorg.apache.commons.modeler.Registry/class methodunregisterComponent/method thread20/thread messageError unregistering mbean /message exception messagejavax.management.RuntimeOperationsException: Object name cannot be null/message frame classcom.sun.jmx.interceptor.DefaultMBeanServerInterceptor/class methodisRegistered/method line545/line /frame frame classcom.sun.jmx.mbeanserver.JmxMBeanServer/class methodisRegistered/method line619/line /frame frame classorg.apache.commons.modeler.Registry/class methodunregisterComponent/method line642/line /frame ... /exception /record Any ideas? Is this something to do with our thread counts (maxThreads=1000 minSpareThreads=50 maxSpareThreads=300)? If so, how can we determine these numbers for fastest performance. We get about 5 hits every second and we want our Tomcats to serve static files as well (such as image files), so we want to make sure that we have enough threads. It's very important for our pages to load fast on the client side. Thank you very much! Any help would be greatly appreciated! Asim - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Registry problems followed by OutOfMemory errors
Hi, Do you need JMX request registration? If not, turn it off by setting it to false in the configuration file. That'll save you some memory by itself. Your OOME is the root cause, so if you fix that the hundreds of others errors will likely go away. The OOME, in turn, is probably caused not because you're out of heap memory, but because you're out of another resource, such as file descriptors or threads. Yoav Shapira http://www.yoavshapira.com -Original Message- From: Asim Alp [mailto:[EMAIL PROTECTED] Sent: Friday, December 10, 2004 10:59 AM To: Tomcat Users List Subject: Registry problems followed by OutOfMemory errors Here is our configuration: Windows Server 2003 Apache 2.0.49 (Win32) mod_ssl/2.0.51 OpenSSL/0.9.7d DAV/2 mod_jk/1.2.6 Tomcat 5.5.4 sun jdk 1.5.0-b64 We have a 2 node tomcat cluster each running with the -Xms128m -Xmx1024m options. We have the following thread options set in the AJP/1.3 connector: maxThreads=1000 minSpareThreads=50 maxSpareThreads=300 And the following corresponding values in our workers.properties (same for each tomcat) worker.tomcat1.type=ajp13 worker.tomcat1.cachesize=300 worker.tomcat1.cache_timeout=60 worker.worker1.local_worker=1 worker.tomcat1.lbfactor=1 worker.tomcat1.connect_timeout = 1000 worker.tomcat1.prepost_timeout = 1000 worker.tomcat1.reply_timeout = 7000 ... worker.loadbalancer.local_worker_only=0 We have a heavily loaded database application running. Same application runs on about 180 different virtual hosts on each Tomcat. We did profiling with JProfile and couldn't find any memory leaks in our application. Each Tomcat works perfect for about 8 to 10 hours, then all of a sudden, they start hanging (not necessarily at the same time). We monitor our heap memory very closely and we usually have enough FREE memory (more than 25%) when the following errors occur: First, we get a couple of SEVERE registering errors: record date2004-12-09T18:19:51/date millis1102634391333/millis sequence270/sequence loggerorg.apache.commons.modeler.Registry/logger levelSEVERE/level classorg.apache.commons.modeler.Registry/class methodregisterComponent/method thread44/thread messageError registering Catalina:type=RequestProcessor,worker=jk-8009,name=JkRequest2291/ message exception messagejavax.management.InstanceAlreadyExistsException: Catalina:type=RequestProcessor,worker=jk-8009,name=JkRequest2291/ message frame classcom.sun.jmx.mbeanserver.RepositorySupport/class methodaddMBean/method line452/line /frame frame classcom.sun.jmx.interceptor.DefaultMBeanServerInterceptor/class methodinternal_addObject/method line1410/line /frame /exception /record record date2004-12-09T18:19:51/date millis1102634391333/millis sequence271/sequence loggerorg.apache.jk.common.ChannelSocket/logger levelWARNING/level classorg.apache.jk.common.ChannelSocket/class methodregisterRequest/method thread44/thread messageError registering request/message /record Followed by a couple of java.lang.OutOfMemoryError: PermGen space messages record date2004-12-09T21:48:25/date millis1102646905849/millis sequence294/sequence loggerStandardWrapper[/apps:jsp]/logger levelSEVERE/level classorg.apache.catalina.core.StandardWrapperValve/class methodinvoke/method thread47/thread messageServlet.service() for servlet jsp threw exception/message exception messagejava.lang.OutOfMemoryError: PermGen space/message /exception /record ... Followed by HUNDREDS (basically for each thread) of Error unregistering mbean messages record date2004-12-09T21:53:29/date millis1102647209630/millis sequence302/sequence loggerorg.apache.commons.modeler.Registry/logger levelSEVERE/level classorg.apache.commons.modeler.Registry/class methodunregisterComponent/method thread20/thread messageError unregistering mbean /message exception messagejavax.management.RuntimeOperationsException: Object name cannot be null/message frame classcom.sun.jmx.interceptor.DefaultMBeanServerInterceptor/class methodisRegistered/method line545/line /frame frame classcom.sun.jmx.mbeanserver.JmxMBeanServer/class methodisRegistered/method line619/line /frame frame classorg.apache.commons.modeler.Registry/class methodunregisterComponent/method line642/line /frame ... /exception /record Any ideas? Is this something to do with our thread counts (maxThreads=1000 minSpareThreads=50 maxSpareThreads=300)? If so, how can we determine these numbers for fastest performance. We get about 5 hits every second and we want our Tomcats to serve static files as well (such as image files), so we want to make sure that we have enough threads. It's very important for our pages to load fast on the client side. Thank you very much! Any help would be greatly appreciated! Asim
Re: Registry problems followed by OutOfMemory errors
Thanks for the quick reply! How can I turn off JMX request registration? I tried to find it in the Tomcat documentation, but all I could find was the MBean Descriptor How To which wasn't very helpful. I doubt that we need it. I will look into the resource problem. I doubt that we're running out of threads, because 1000 of them should be more than enough. Another resource problem could be the number of SQL connections, but on our SQL Side (MySQL), max_connections is set to 5000 which is about 7 times more than we need. Number of file descriptors might be a problem. For each web application, we have to load properties files and these files are automatically checked once every minute for possible updates. How can I configure the number of file descriptors? Thank you very much! Asim On Dec 10, 2004, at 11:02 AM, Shapira, Yoav wrote: Hi, Do you need JMX request registration? If not, turn it off by setting it to false in the configuration file. That'll save you some memory by itself. Your OOME is the root cause, so if you fix that the hundreds of others errors will likely go away. The OOME, in turn, is probably caused not because you're out of heap memory, but because you're out of another resource, such as file descriptors or threads. Yoav Shapira http://www.yoavshapira.com -Original Message- From: Asim Alp [mailto:[EMAIL PROTECTED] Sent: Friday, December 10, 2004 10:59 AM To: Tomcat Users List Subject: Registry problems followed by OutOfMemory errors Here is our configuration: Windows Server 2003 Apache 2.0.49 (Win32) mod_ssl/2.0.51 OpenSSL/0.9.7d DAV/2 mod_jk/1.2.6 Tomcat 5.5.4 sun jdk 1.5.0-b64 We have a 2 node tomcat cluster each running with the -Xms128m -Xmx1024m options. We have the following thread options set in the AJP/1.3 connector: maxThreads=1000 minSpareThreads=50 maxSpareThreads=300 And the following corresponding values in our workers.properties (same for each tomcat) worker.tomcat1.type=ajp13 worker.tomcat1.cachesize=300 worker.tomcat1.cache_timeout=60 worker.worker1.local_worker=1 worker.tomcat1.lbfactor=1 worker.tomcat1.connect_timeout = 1000 worker.tomcat1.prepost_timeout = 1000 worker.tomcat1.reply_timeout = 7000 ... worker.loadbalancer.local_worker_only=0 We have a heavily loaded database application running. Same application runs on about 180 different virtual hosts on each Tomcat. We did profiling with JProfile and couldn't find any memory leaks in our application. Each Tomcat works perfect for about 8 to 10 hours, then all of a sudden, they start hanging (not necessarily at the same time). We monitor our heap memory very closely and we usually have enough FREE memory (more than 25%) when the following errors occur: First, we get a couple of SEVERE registering errors: record date2004-12-09T18:19:51/date millis1102634391333/millis sequence270/sequence loggerorg.apache.commons.modeler.Registry/logger levelSEVERE/level classorg.apache.commons.modeler.Registry/class methodregisterComponent/method thread44/thread messageError registering Catalina:type=RequestProcessor,worker=jk-8009,name=JkRequest2291/ message exception messagejavax.management.InstanceAlreadyExistsException: Catalina:type=RequestProcessor,worker=jk-8009,name=JkRequest2291/ message frame classcom.sun.jmx.mbeanserver.RepositorySupport/class methodaddMBean/method line452/line /frame frame classcom.sun.jmx.interceptor.DefaultMBeanServerInterceptor/class methodinternal_addObject/method line1410/line /frame /exception /record record date2004-12-09T18:19:51/date millis1102634391333/millis sequence271/sequence loggerorg.apache.jk.common.ChannelSocket/logger levelWARNING/level classorg.apache.jk.common.ChannelSocket/class methodregisterRequest/method thread44/thread messageError registering request/message /record Followed by a couple of java.lang.OutOfMemoryError: PermGen space messages record date2004-12-09T21:48:25/date millis1102646905849/millis sequence294/sequence loggerStandardWrapper[/apps:jsp]/logger levelSEVERE/level classorg.apache.catalina.core.StandardWrapperValve/class methodinvoke/method thread47/thread messageServlet.service() for servlet jsp threw exception/message exception messagejava.lang.OutOfMemoryError: PermGen space/message /exception /record ... Followed by HUNDREDS (basically for each thread) of Error unregistering mbean messages record date2004-12-09T21:53:29/date millis1102647209630/millis sequence302/sequence loggerorg.apache.commons.modeler.Registry/logger levelSEVERE/level classorg.apache.commons.modeler.Registry/class methodunregisterComponent/method thread20/thread messageError unregistering mbean /message exception messagejavax.management.RuntimeOperationsException: Object name cannot be null/message frame classcom.sun.jmx.interceptor.DefaultMBeanServerInterceptor/class methodisRegistered/method line545/line /frame frame
Re: Registry problems followed by OutOfMemory errors
Asim Alp [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED] Thanks for the quick reply! How can I turn off JMX request registration? I tried to find it in the Tomcat documentation, but all I could find was the MBean Descriptor How To which wasn't very helpful. I doubt that we need it. Set request.registerRequests=false on the Connector. I will look into the resource problem. I doubt that we're running out of threads, because 1000 of them should be more than enough. Another resource problem could be the number of SQL connections, but on our SQL Side (MySQL), max_connections is set to 5000 which is about 7 times more than we need. Number of file descriptors might be a problem. For each web application, we have to load properties files and these files are automatically checked once every minute for possible updates. How can I configure the number of file descriptors? Thank you very much! Asim On Dec 10, 2004, at 11:02 AM, Shapira, Yoav wrote: Hi, Do you need JMX request registration? If not, turn it off by setting it to false in the configuration file. That'll save you some memory by itself. Your OOME is the root cause, so if you fix that the hundreds of others errors will likely go away. The OOME, in turn, is probably caused not because you're out of heap memory, but because you're out of another resource, such as file descriptors or threads. Yoav Shapira http://www.yoavshapira.com -Original Message- From: Asim Alp [mailto:[EMAIL PROTECTED] Sent: Friday, December 10, 2004 10:59 AM To: Tomcat Users List Subject: Registry problems followed by OutOfMemory errors Here is our configuration: Windows Server 2003 Apache 2.0.49 (Win32) mod_ssl/2.0.51 OpenSSL/0.9.7d DAV/2 mod_jk/1.2.6 Tomcat 5.5.4 sun jdk 1.5.0-b64 We have a 2 node tomcat cluster each running with the -Xms128m -Xmx1024m options. We have the following thread options set in the AJP/1.3 connector: maxThreads=1000 minSpareThreads=50 maxSpareThreads=300 And the following corresponding values in our workers.properties (same for each tomcat) worker.tomcat1.type=ajp13 worker.tomcat1.cachesize=300 worker.tomcat1.cache_timeout=60 worker.worker1.local_worker=1 worker.tomcat1.lbfactor=1 worker.tomcat1.connect_timeout = 1000 worker.tomcat1.prepost_timeout = 1000 worker.tomcat1.reply_timeout = 7000 ... worker.loadbalancer.local_worker_only=0 We have a heavily loaded database application running. Same application runs on about 180 different virtual hosts on each Tomcat. We did profiling with JProfile and couldn't find any memory leaks in our application. Each Tomcat works perfect for about 8 to 10 hours, then all of a sudden, they start hanging (not necessarily at the same time). We monitor our heap memory very closely and we usually have enough FREE memory (more than 25%) when the following errors occur: First, we get a couple of SEVERE registering errors: record date2004-12-09T18:19:51/date millis1102634391333/millis sequence270/sequence loggerorg.apache.commons.modeler.Registry/logger levelSEVERE/level classorg.apache.commons.modeler.Registry/class methodregisterComponent/method thread44/thread messageError registering Catalina:type=RequestProcessor,worker=jk-8009,name=JkRequest2291/ message exception messagejavax.management.InstanceAlreadyExistsException: Catalina:type=RequestProcessor,worker=jk-8009,name=JkRequest2291/ message frame classcom.sun.jmx.mbeanserver.RepositorySupport/class methodaddMBean/method line452/line /frame frame classcom.sun.jmx.interceptor.DefaultMBeanServerInterceptor/class methodinternal_addObject/method line1410/line /frame /exception /record record date2004-12-09T18:19:51/date millis1102634391333/millis sequence271/sequence loggerorg.apache.jk.common.ChannelSocket/logger levelWARNING/level classorg.apache.jk.common.ChannelSocket/class methodregisterRequest/method thread44/thread messageError registering request/message /record Followed by a couple of java.lang.OutOfMemoryError: PermGen space messages record date2004-12-09T21:48:25/date millis1102646905849/millis sequence294/sequence loggerStandardWrapper[/apps:jsp]/logger levelSEVERE/level classorg.apache.catalina.core.StandardWrapperValve/class methodinvoke/method thread47/thread messageServlet.service() for servlet jsp threw exception/message exception messagejava.lang.OutOfMemoryError: PermGen space/message /exception /record ... Followed by HUNDREDS (basically for each thread) of Error unregistering mbean messages record date2004-12-09T21:53:29/date millis1102647209630/millis sequence302/sequence loggerorg.apache.commons.modeler.Registry/logger levelSEVERE/level classorg.apache.commons.modeler.Registry/class methodunregisterComponent/method thread20/thread messageError unregistering mbean /message exception
