RE: OutOfMemory / JMeter / Profiler questions
Thanks Harry for your input. But we just can't do any upgrades to the production environment at this point :( Hopefully soon. Guillaume -Original Message- From: Harry Mantheakis [mailto:[EMAIL PROTECTED] Sent: 03 March 2005 10:45 To: Tomcat Users List Subject: Re: OutOfMemory / JMeter / Profiler questions We've started performance testing one of our REMOTE web apps using JMeter. We're gathering benchmark data before doing further fine tuning. Details: Win2K only have ssh + cygwin access to this remote server JDK 1.4.1_03 Tomcat 4.1.26, running as a service: a.. Use security manager 1 b.. Security policy file D:\Tomcat4\conf\catalina.policy c.. Initial heap 256 d.. Max heap 512 e.. Stack size 256 f.. JVM server I cannot help you with respect to your OOM problems, but I would recommend that you consider upgrading to JVM 1.4.2. You should also be able to migrate your app to TC 5.0.28 without too much grief. You should probably avoid TC 5.5.x for the meantime. JVM 1.4.2 and TC 5.0.28 play well together. Good luck in case! Harry Mantheakis London, UK - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemory / JMeter / Profiler questions
Alan, Thanks for your feedback. You got me curious here: Why does/would Tomcat reload sessions after startup? Aren't the sessions destroyed upon Tomcat shutdown? Also, I could only find a $TOMCAT_HOME/work/Standalone/localhost/prti-rpt-engine.2.4.2.b17/SESSIONS.se r, which is NOT the context I have been load testing. Sessions should be serialized per context, shouldn't they? We do not store anything in the session (our test just calls a servlet which generate a PDF report) so I can't see why the sessions would be using so much memory. But I'll definitely keep your idea in mind for other tests. I should hopefully be able to run the same tests on our staging platform and see if we get similar behaviors. Thanks for your inputs. Please keep them coming! Cheers, Guillaume -Original Message- From: Flisch, Alan [mailto:[EMAIL PROTECTED] Sent: 03 March 2005 12:17 To: Tomcat Users List Subject: RE: OutOfMemory / JMeter / Profiler questions You probably have either very large or very many sessions which Tomcat is attempting to reload on startup. Tomcat serialises sessions into files called SESSIONS.ser (in the application directories under the work dir) and then when it is restarted it attempts to reload them all. I presume this behaviour can be turned off. In terms of testing your app, you want to figure out whether you have a memory leak issue with your app, or simply that your max heap size is set too low for the load you are running. To check for memory leaks, you could run jmeter reasonably (although not too hard) excercising as much of your app as you can, repetitively and for an extended period. If it eventually keels over then you may need to investigate memory leaks with a profiler. Another possibility is that you may not be invalidating sessions and they are just being left to expire naturally. This can use up a lot of memory if you aren't careful and I supect is a quite likely source of your problems. Cheers! -Original Message- From: Guillaume Lahitette [mailto:[EMAIL PROTECTED] Sent: 03 March 2005 09:51 To: tomcat-user@jakarta.apache.org Subject: OutOfMemory / JMeter / Profiler questions Hello Tomcat'oids! We've started performance testing one of our REMOTE web apps using JMeter. We're gathering benchmark data before doing further fine tuning. Details: Win2K only have ssh + cygwin access to this remote server JDK 1.4.1_03 Tomcat 4.1.26, running as a service: a.. Use security manager 1 b.. Security policy file D:\Tomcat4\conf\catalina.policy c.. Initial heap 256 d.. Max heap 512 e.. Stack size 256 f.. JVM server Issue: We are getting OutOfMemory errors with very few threads simulated (as low as 5). More problematic, we've seen the OOM just after a Tomcat service restart! From the stack trace below, you can see we get the OOM before any of our code is executed :( Questions: a.. Anyone has seen this behavior upon Tomcat start up? b.. Anything particular to watch for in our JMeter test plan? c.. Would a profiler help? Could it profile a remote Tomcat installation? Any +/- feedback on Eclipse Profiler plug in (http://eclipsecolorer.sourceforge.net/index_profiler.html)? We'll work on gathering more data (e.g. periodic free / allocated memory dumps). Untill then, thank you for sharing your experiences, suggestions, code,...! Cheers, Guillaume javax.servlet.ServletException: Servlet execution threw an exception at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(A pplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.access$000(Applica tionFilterChain.java:98) at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationF ilterChain.java:176) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ApplicationFilterChain.doFilter(Applicati onFilterChain.java:172) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapp erValve.java:256) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveCon text.invokeNext(StandardPipeline.java:643) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline. java:480) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) at org.apache.catalina.core.StandardContextValve.invoke(StandardConte xtValve.java:191) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveCon text.invokeNext(StandardPipeline.java:643) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline. java:480) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) at org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2416) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValv e.java:180) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveCon text.invokeNext(StandardPipeline.java:643
RE: OutOfMemory / JMeter / Profiler questions
Hi, Tomcat always serializes sessions on shutdown and reloads them on startup. This is the default behaviour but can be changed. You are right in thinking that sessions are serialized per context though. Are you using the standard manager or the persistent manager as they are stored differently with the persistent manager. Although you don't store anything in a session, do you perhaps instanciate it? The number of sessions you currently have can be viewed from the manager application. Ta Matt -Original Message- From: Guillaume Lahitette [mailto:[EMAIL PROTECTED] Sent: 04 March 2005 15:15 To: Tomcat Users List Subject: RE: OutOfMemory / JMeter / Profiler questions Alan, Thanks for your feedback. You got me curious here: Why does/would Tomcat reload sessions after startup? Aren't the sessions destroyed upon Tomcat shutdown? Also, I could only find a $TOMCAT_HOME/work/Standalone/localhost/prti-rpt-engine.2.4.2.b17/SESSIONS.se r, which is NOT the context I have been load testing. Sessions should be serialized per context, shouldn't they? We do not store anything in the session (our test just calls a servlet which generate a PDF report) so I can't see why the sessions would be using so much memory. But I'll definitely keep your idea in mind for other tests. I should hopefully be able to run the same tests on our staging platform and see if we get similar behaviors. Thanks for your inputs. Please keep them coming! Cheers, Guillaume -Original Message- From: Flisch, Alan [mailto:[EMAIL PROTECTED] Sent: 03 March 2005 12:17 To: Tomcat Users List Subject: RE: OutOfMemory / JMeter / Profiler questions You probably have either very large or very many sessions which Tomcat is attempting to reload on startup. Tomcat serialises sessions into files called SESSIONS.ser (in the application directories under the work dir) and then when it is restarted it attempts to reload them all. I presume this behaviour can be turned off. In terms of testing your app, you want to figure out whether you have a memory leak issue with your app, or simply that your max heap size is set too low for the load you are running. To check for memory leaks, you could run jmeter reasonably (although not too hard) excercising as much of your app as you can, repetitively and for an extended period. If it eventually keels over then you may need to investigate memory leaks with a profiler. Another possibility is that you may not be invalidating sessions and they are just being left to expire naturally. This can use up a lot of memory if you aren't careful and I supect is a quite likely source of your problems. Cheers! -Original Message- From: Guillaume Lahitette [mailto:[EMAIL PROTECTED] Sent: 03 March 2005 09:51 To: tomcat-user@jakarta.apache.org Subject: OutOfMemory / JMeter / Profiler questions Hello Tomcat'oids! We've started performance testing one of our REMOTE web apps using JMeter. We're gathering benchmark data before doing further fine tuning. Details: Win2K only have ssh + cygwin access to this remote server JDK 1.4.1_03 Tomcat 4.1.26, running as a service: a.. Use security manager 1 b.. Security policy file D:\Tomcat4\conf\catalina.policy c.. Initial heap 256 d.. Max heap 512 e.. Stack size 256 f.. JVM server Issue: We are getting OutOfMemory errors with very few threads simulated (as low as 5). More problematic, we've seen the OOM just after a Tomcat service restart! From the stack trace below, you can see we get the OOM before any of our code is executed :( Questions: a.. Anyone has seen this behavior upon Tomcat start up? b.. Anything particular to watch for in our JMeter test plan? c.. Would a profiler help? Could it profile a remote Tomcat installation? Any +/- feedback on Eclipse Profiler plug in (http://eclipsecolorer.sourceforge.net/index_profiler.html)? We'll work on gathering more data (e.g. periodic free / allocated memory dumps). Untill then, thank you for sharing your experiences, suggestions, code,...! Cheers, Guillaume javax.servlet.ServletException: Servlet execution threw an exception at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(A pplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.access$000(Applica tionFilterChain.java:98) at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationF ilterChain.java:176) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ApplicationFilterChain.doFilter(Applicati onFilterChain.java:172) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapp erValve.java:256) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveCon text.invokeNext(StandardPipeline.java:643) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline. java:480) at org.apache.catalina.core.ContainerBase.invoke
RE: OutOfMemory / JMeter / Profiler questions
Matt, Thanks for your feedbackwhich triggered more questions below! -Original Message- From: Dale, Matt [mailto:[EMAIL PROTECTED] Sent: 04 March 2005 15:26 To: Tomcat Users List Subject: RE: OutOfMemory / JMeter / Profiler questions Hi, Tomcat always serializes sessions on shutdown and reloads them on startup. Can anyone explain what is the usage for this? You can not re-use a session after a restart, can you??? I would think new session ID is created. This is the default behaviour but can be changed. Does anyone know how? If I can't see any usage for this, I'm very keen on disabling it. You are right in thinking that sessions are serialized per context though. Are you using the standard manager or the persistent manager as they are stored differently with the persistent manager. First time I hear about these! I could see a commented PersistentManager in server.xml so I guess I am using the standard manager. Can you tell me more about these? Why/When would I use the PersistentManager? Although you don't store anything in a session, do you perhaps instanciate it? The number of sessions you currently have can be viewed from the manager application. Yes, I've been using it a lot for context stop / start / reload and monitoring the sessions... Cheers, G Ta Matt -Original Message- From: Guillaume Lahitette [mailto:[EMAIL PROTECTED] Sent: 04 March 2005 15:15 To: Tomcat Users List Subject: RE: OutOfMemory / JMeter / Profiler questions Alan, Thanks for your feedback. You got me curious here: Why does/would Tomcat reload sessions after startup? Aren't the sessions destroyed upon Tomcat shutdown? Also, I could only find a $TOMCAT_HOME/work/Standalone/localhost/prti-rpt-engine.2.4.2.b17/S ESSIONS.se r, which is NOT the context I have been load testing. Sessions should be serialized per context, shouldn't they? We do not store anything in the session (our test just calls a servlet which generate a PDF report) so I can't see why the sessions would be using so much memory. But I'll definitely keep your idea in mind for other tests. I should hopefully be able to run the same tests on our staging platform and see if we get similar behaviors. Thanks for your inputs. Please keep them coming! Cheers, Guillaume -Original Message- From: Flisch, Alan [mailto:[EMAIL PROTECTED] Sent: 03 March 2005 12:17 To: Tomcat Users List Subject: RE: OutOfMemory / JMeter / Profiler questions You probably have either very large or very many sessions which Tomcat is attempting to reload on startup. Tomcat serialises sessions into files called SESSIONS.ser (in the application directories under the work dir) and then when it is restarted it attempts to reload them all. I presume this behaviour can be turned off. In terms of testing your app, you want to figure out whether you have a memory leak issue with your app, or simply that your max heap size is set too low for the load you are running. To check for memory leaks, you could run jmeter reasonably (although not too hard) excercising as much of your app as you can, repetitively and for an extended period. If it eventually keels over then you may need to investigate memory leaks with a profiler. Another possibility is that you may not be invalidating sessions and they are just being left to expire naturally. This can use up a lot of memory if you aren't careful and I supect is a quite likely source of your problems. Cheers! -Original Message- From: Guillaume Lahitette [mailto:[EMAIL PROTECTED] Sent: 03 March 2005 09:51 To: tomcat-user@jakarta.apache.org Subject: OutOfMemory / JMeter / Profiler questions Hello Tomcat'oids! We've started performance testing one of our REMOTE web apps using JMeter. We're gathering benchmark data before doing further fine tuning. Details: Win2K only have ssh + cygwin access to this remote server JDK 1.4.1_03 Tomcat 4.1.26, running as a service: a.. Use security manager 1 b.. Security policy file D:\Tomcat4\conf\catalina.policy c.. Initial heap 256 d.. Max heap 512 e.. Stack size 256 f.. JVM server Issue: We are getting OutOfMemory errors with very few threads simulated (as low as 5). More problematic, we've seen the OOM just after a Tomcat service restart! From the stack trace below, you can see we get the OOM before any of our code is executed :( Questions: a.. Anyone has seen this behavior upon Tomcat start up? b.. Anything particular to watch for in our JMeter test plan? c.. Would a profiler help? Could it profile a remote Tomcat installation? Any +/- feedback on Eclipse Profiler plug in (http://eclipsecolorer.sourceforge.net/index_profiler.html)? We'll work on gathering more data (e.g. periodic free / allocated memory dumps). Untill then, thank you for sharing your experiences, suggestions, code
RE: OutOfMemory / JMeter / Profiler questions
Hi, This can be very useful behaviour. It means a tomcat instance can be bounced with only a short loss of service to the users. The sessions CAN be reused after a restart. Information on the session managers and how to disable persistence can be found in the tomcat documentation so it's a case of RTFM. http://jakarta.apache.org/tomcat/tomcat-5.0-doc/config/manager.html should get you started. It sounds like it is not a case of session persistence or sessions not being invalidated that is causing your problem however. I recommend, as always, that you get jvmstat from sun and monitor your tomcat instance with it. It will allow you to observe the jvm and the various parts of the heap and diagnose where the problem is. Ta Matt -Original Message- From: Guillaume Lahitette [mailto:[EMAIL PROTECTED] Sent: 04 March 2005 17:58 To: Tomcat Users List Subject: RE: OutOfMemory / JMeter / Profiler questions Matt, Thanks for your feedbackwhich triggered more questions below! -Original Message- From: Dale, Matt [mailto:[EMAIL PROTECTED] Sent: 04 March 2005 15:26 To: Tomcat Users List Subject: RE: OutOfMemory / JMeter / Profiler questions Hi, Tomcat always serializes sessions on shutdown and reloads them on startup. Can anyone explain what is the usage for this? You can not re-use a session after a restart, can you??? I would think new session ID is created. This is the default behaviour but can be changed. Does anyone know how? If I can't see any usage for this, I'm very keen on disabling it. You are right in thinking that sessions are serialized per context though. Are you using the standard manager or the persistent manager as they are stored differently with the persistent manager. First time I hear about these! I could see a commented PersistentManager in server.xml so I guess I am using the standard manager. Can you tell me more about these? Why/When would I use the PersistentManager? Although you don't store anything in a session, do you perhaps instanciate it? The number of sessions you currently have can be viewed from the manager application. Yes, I've been using it a lot for context stop / start / reload and monitoring the sessions... Cheers, G Ta Matt -Original Message- From: Guillaume Lahitette [mailto:[EMAIL PROTECTED] Sent: 04 March 2005 15:15 To: Tomcat Users List Subject: RE: OutOfMemory / JMeter / Profiler questions Alan, Thanks for your feedback. You got me curious here: Why does/would Tomcat reload sessions after startup? Aren't the sessions destroyed upon Tomcat shutdown? Also, I could only find a $TOMCAT_HOME/work/Standalone/localhost/prti-rpt-engine.2.4.2.b17/S ESSIONS.se r, which is NOT the context I have been load testing. Sessions should be serialized per context, shouldn't they? We do not store anything in the session (our test just calls a servlet which generate a PDF report) so I can't see why the sessions would be using so much memory. But I'll definitely keep your idea in mind for other tests. I should hopefully be able to run the same tests on our staging platform and see if we get similar behaviors. Thanks for your inputs. Please keep them coming! Cheers, Guillaume -Original Message- From: Flisch, Alan [mailto:[EMAIL PROTECTED] Sent: 03 March 2005 12:17 To: Tomcat Users List Subject: RE: OutOfMemory / JMeter / Profiler questions You probably have either very large or very many sessions which Tomcat is attempting to reload on startup. Tomcat serialises sessions into files called SESSIONS.ser (in the application directories under the work dir) and then when it is restarted it attempts to reload them all. I presume this behaviour can be turned off. In terms of testing your app, you want to figure out whether you have a memory leak issue with your app, or simply that your max heap size is set too low for the load you are running. To check for memory leaks, you could run jmeter reasonably (although not too hard) excercising as much of your app as you can, repetitively and for an extended period. If it eventually keels over then you may need to investigate memory leaks with a profiler. Another possibility is that you may not be invalidating sessions and they are just being left to expire naturally. This can use up a lot of memory if you aren't careful and I supect is a quite likely source of your problems. Cheers! -Original Message- From: Guillaume Lahitette [mailto:[EMAIL PROTECTED] Sent: 03 March 2005 09:51 To: tomcat-user@jakarta.apache.org Subject: OutOfMemory / JMeter / Profiler questions Hello Tomcat'oids! We've started performance testing one of our REMOTE web apps using JMeter. We're gathering benchmark data before doing further fine tuning. Details: Win2K only have ssh + cygwin access to this remote server JDK 1.4.1_03 Tomcat 4.1.26, running as a service
Re: OutOfMemory / JMeter / Profiler questions
We've started performance testing one of our REMOTE web apps using JMeter. We're gathering benchmark data before doing further fine tuning. Details: Win2K only have ssh + cygwin access to this remote server JDK 1.4.1_03 Tomcat 4.1.26, running as a service: a.. Use security manager 1 b.. Security policy file D:\Tomcat4\conf\catalina.policy c.. Initial heap 256 d.. Max heap 512 e.. Stack size 256 f.. JVM server I cannot help you with respect to your OOM problems, but I would recommend that you consider upgrading to JVM 1.4.2. You should also be able to migrate your app to TC 5.0.28 without too much grief. You should probably avoid TC 5.5.x for the meantime. JVM 1.4.2 and TC 5.0.28 play well together. Good luck in case! Harry Mantheakis London, UK - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemory / JMeter / Profiler questions
You probably have either very large or very many sessions which Tomcat is attempting to reload on startup. Tomcat serialises sessions into files called SESSIONS.ser (in the application directories under the work dir) and then when it is restarted it attempts to reload them all. I presume this behaviour can be turned off. In terms of testing your app, you want to figure out whether you have a memory leak issue with your app, or simply that your max heap size is set too low for the load you are running. To check for memory leaks, you could run jmeter reasonably (although not too hard) excercising as much of your app as you can, repetitively and for an extended period. If it eventually keels over then you may need to investigate memory leaks with a profiler. Another possibility is that you may not be invalidating sessions and they are just being left to expire naturally. This can use up a lot of memory if you aren't careful and I supect is a quite likely source of your problems. Cheers! -Original Message- From: Guillaume Lahitette [mailto:[EMAIL PROTECTED] Sent: 03 March 2005 09:51 To: tomcat-user@jakarta.apache.org Subject: OutOfMemory / JMeter / Profiler questions Hello Tomcat'oids! We've started performance testing one of our REMOTE web apps using JMeter. We're gathering benchmark data before doing further fine tuning. Details: Win2K only have ssh + cygwin access to this remote server JDK 1.4.1_03 Tomcat 4.1.26, running as a service: a.. Use security manager 1 b.. Security policy file D:\Tomcat4\conf\catalina.policy c.. Initial heap 256 d.. Max heap 512 e.. Stack size 256 f.. JVM server Issue: We are getting OutOfMemory errors with very few threads simulated (as low as 5). More problematic, we've seen the OOM just after a Tomcat service restart! From the stack trace below, you can see we get the OOM before any of our code is executed :( Questions: a.. Anyone has seen this behavior upon Tomcat start up? b.. Anything particular to watch for in our JMeter test plan? c.. Would a profiler help? Could it profile a remote Tomcat installation? Any +/- feedback on Eclipse Profiler plug in (http://eclipsecolorer.sourceforge.net/index_profiler.html)? We'll work on gathering more data (e.g. periodic free / allocated memory dumps). Untill then, thank you for sharing your experiences, suggestions, code,...! Cheers, Guillaume javax.servlet.ServletException: Servlet execution threw an exception at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:98) at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:176) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:172) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) at org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2416) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643) at org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:171) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:577) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480) at
