RE: Off Topic: Tomcat monitoring tools
Thanks to all for your help: I now need to evaluate them. Peter, I'll see you on the jmeter list for a question! Cheers, Guillaume -Original Message- From: Peter Lin [mailto:[EMAIL PROTECTED] Sent: 10 May 2005 14:22 To: Tomcat Users List Subject: Re: Off Topic: Tomcat monitoring tools there's also this, which I haven't tried. http://mc4j.sourceforge.net/ peter On 5/10/05, Didier McGillis [EMAIL PROTECTED] wrote: Maybe something like this will work. http://www.hyperic.net Its not just for Tomcat. From: Guillaume Lahitette [EMAIL PROTECTED] Reply-To: Tomcat Users List tomcat-user@jakarta.apache.org To: tomcat-user@jakarta.apache.org Subject: RE: Off Topic: Tomcat monitoring tools Date: Tue, 10 May 2005 00:18:25 +0100 Thank you very much Peter. Yes indeed, we've used JMeter to do our testing but I wasn't aware of this monitoring capability. I'm trying to install Tomcat 5 to give it a try since JMeter can't play with our Tomcat 4.1.26 for monitoring purposes. Any other candidates out there? TIA. Guillaume -Original Message- From: Peter Lin [mailto:[EMAIL PROTECTED] Sent: 09 May 2005 18:47 To: Tomcat Users List Subject: Re: Off Topic: Tomcat monitoring tools there's a tool call jmeter from a group call Jakarta, you might have heard of them. Jmeter has the ability to monitor multiple tomcat's and show the performance in a graph. http://jakarta.apache.org/jmeter/ http://jakarta.apache.org/jmeter/usermanual/build-monitor-test-plan.html have fun peter lin On 5/9/05, Guillaume Lahitette [EMAIL PROTECTED] wrote: Hi, I am looking for feedback on (preferably free) tools to monitor the performance of Tomcat during stress testing. We're running Tomcat 4.1.26 in production on Linux and Windows (as a service). Ideally, the tool would support connecting to multiple, remote servers. TIA. Guillaume - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Off Topic: Tomcat monitoring tools
Hi, I am looking for feedback on (preferably free) tools to monitor the performance of Tomcat during stress testing. We're running Tomcat 4.1.26 in production on Linux and Windows (as a service). Ideally, the tool would support connecting to multiple, remote servers. TIA. Guillaume - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Off Topic: Tomcat monitoring tools
Thank you very much Peter. Yes indeed, we've used JMeter to do our testing but I wasn't aware of this monitoring capability. I'm trying to install Tomcat 5 to give it a try since JMeter can't play with our Tomcat 4.1.26 for monitoring purposes. Any other candidates out there? TIA. Guillaume -Original Message- From: Peter Lin [mailto:[EMAIL PROTECTED] Sent: 09 May 2005 18:47 To: Tomcat Users List Subject: Re: Off Topic: Tomcat monitoring tools there's a tool call jmeter from a group call Jakarta, you might have heard of them. Jmeter has the ability to monitor multiple tomcat's and show the performance in a graph. http://jakarta.apache.org/jmeter/ http://jakarta.apache.org/jmeter/usermanual/build-monitor-test-plan.html have fun peter lin On 5/9/05, Guillaume Lahitette [EMAIL PROTECTED] wrote: Hi, I am looking for feedback on (preferably free) tools to monitor the performance of Tomcat during stress testing. We're running Tomcat 4.1.26 in production on Linux and Windows (as a service). Ideally, the tool would support connecting to multiple, remote servers. TIA. Guillaume - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Need your help to troubleshoot a Tomcat crash
Hello gurus, * SETUP This is Tomcat 4.1.26 on Win2K running with Tomcat Service Manager from http://web.bvu.edu/staff/david/tcservcfg/ JDK 1.4.1_03 with options -Xms512 -Xmx1024 -Xss2048 -showversion -verbose:gc -Xloggc:c:\winnt\system32\PRTI-Report.log Connector config: Connector className=org.apache.coyote.tomcat4.CoyoteConnector port=18580 minProcessors=5 maxProcessors=150 enableLookups=true redirectPort=8443 acceptCount=100 debug=0 connectionTimeout=2 useURIValidationHack=false disableUploadTimeout=true / * PROBLEM We're facing a problem whereby when we stress test our web application, the Tomcat service brutally crashes. Our problem is that it has been impossible to identify what made Tomcat (or the JVM?) crash: nothing shows in the logs, no JVM crash report in $TOMCAT_HOME/bin nor C:/WINNT/system32. We don't know why but removing the JVM option -server has enabled the performance test to complete successfully. * QUESTIONS * can I force Tomcat to give details about why when it crashes? Ideally, we'd need a stack trace... * is there a negative impact of running Windows as a service? * is anyone facing issues with JDK 1.4.1_03 running with the -server switch? * any recommendations for improving our setup? Thanks in advance, Guillaume - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: How to enable GC with Tomcat Service Manager
Folks, The closer we could get so far is to define the following JVM Option to the Tomcat Manager Service: -Xloggc:path_to_log_file Which does not explain me why it does not pick up the -verbose:gc . Maybe I need to wait for another life! Guillaume - Original Message - From: Guillaume Lahitette [EMAIL PROTECTED] To: Tomcat Users List tomcat-user@jakarta.apache.org Subject: How to enable GC with Tomcat Service Manager Date: Wed, 20 Apr 2005 22:37:02 +0100 Hi, I'm running Tomcat as a Windows service using Tomcat Service Manager 0.2.8 (http://web.bvu.edu/staff/david/index.jsp?section=softwaresubsection=tcservcfgpage=overview). I'm now trying to enable the garbage collector output but am simply not getting any GC info in /cygdrive/D/jakarta-tomcat-4.1.24/logs/Tomcat4124_out.log. I've pasted the config below. Has anyone got any success with it? Guillaume. - Configuration Service name Tomcat4124 Catalina home D:\jakarta-tomcat-4.1.24 Catalina base D:\jakarta-tomcat-4.1.24 Tomcat config file D:\jakarta-tomcat-4.1.24\conf\server.xml Use security manager 0 Security policy file D:\jakarta-tomcat-4.1.24\conf\catalina.policy Java home c:\j2sdk1.4.1 JVM Path c:\j2sdk1.4.1\jre\bin\server\jvm.dll Classpath D:\jakarta-tomcat-4.1.24\bin\bootstrap.jar;D:\jakarta-tomcat-4.1.24\common\lib;c:\j2sdk1.4.1\lib\tools.jar Temp directory D:\jakarta-tomcat-4.1.24\temp System.out file D:\jakarta-tomcat-4.1.24\logs System.err file D:\jakarta-tomcat-4.1.24\logs Initial heap Max heap Stack size JVM server Additional JVM option -verbose:gc - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Heap not expanding beyond -Xms512m
Hello Tomcat-oids, I'd really appreciate any help with this...Thanks in advance. We've been running performance tests against Tomcat 4.1.26 on Linux and Windows. We've tuned heap sizes to -Xms512m -Xmx1024m. Monitoring the JVM memory usage with a daemon thread calling Runtime.getRuntime().freeMemory() / totalMemory() / maxMemory(), we've noticed memory was never expanded beyond 512MB. Instead, when usage was getting close to 512MB, we could see the garbage collector kick in. We never got OOM errors. Can anyone share experiences about this? Do I have to use -Xms1024m -Xmx1024m (which seems to defeat the purpose of having ms and mx...)? Guillaume - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Heap not expanding beyond -Xms512m
Quinten, Matt, Thank you for your feedback and the JVM reference. Guillaume - Original Message - From: Quinten Verheyen [EMAIL PROTECTED] To: Tomcat Users List tomcat-user@jakarta.apache.org Subject: RE: Heap not expanding beyond -Xms512m Date: Wed, 20 Apr 2005 15:32:12 +0200 I agree, it's always a good idea to let the JVM do what it is supposed to do instead of creating a growing overhead. The only exception to that rule is if you think you'll reach 1024 MB, but as you say yourself, 512 seems to do fine as -Xms. btw, here is an article that goes into detail (ratio's etc.) using jvm 1.4.2 http://java.sun.com/docs/hotspot/gc1.4.2/index.html Q -Original Message- From: Dale, Matt [mailto:[EMAIL PROTECTED] Sent: 20 April 2005 15:24 To: Tomcat Users List Subject: RE: Heap not expanding beyond -Xms512m This is expected behaviour. The heap will not grow if the memory required can be gained through a garbage collection. The heap will only grow if after a garbage collection it still cannot allocate the required memory or is over a certain percentage (I forget what the default is). I would suggest that you don't in fact need 512MB if your app runs fine with no OOM in 512MB. If you really want to use the full 1GB the you'll have to use -Xms as you suggested. Ta Matt -Original Message- From: Guillaume Lahitette [mailto:[EMAIL PROTECTED] Sent: 20 April 2005 13:54 To: tomcat-user@jakarta.apache.org Subject: Heap not expanding beyond -Xms512m Hello Tomcat-oids, I'd really appreciate any help with this...Thanks in advance. We've been running performance tests against Tomcat 4.1.26 on Linux and Windows. We've tuned heap sizes to -Xms512m -Xmx1024m. Monitoring the JVM memory usage with a daemon thread calling Runtime.getRuntime().freeMemory() / totalMemory() / maxMemory(), we've noticed memory was never expanded beyond 512MB. Instead, when usage was getting close to 512MB, we could see the garbage collector kick in. We never got OOM errors. Can anyone share experiences about this? Do I have to use -Xms1024m -Xmx1024m (which seems to defeat the purpose of having ms and mx...)? Guillaume - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
How to enable GC with Tomcat Service Manager
Hi, I'm running Tomcat as a Windows service using Tomcat Service Manager 0.2.8 (http://web.bvu.edu/staff/david/index.jsp?section=softwaresubsection=tcservcfgpage=overview). I'm now trying to enable the garbage collector output but am simply not getting any GC info in /cygdrive/D/jakarta-tomcat-4.1.24/logs/Tomcat4124_out.log. I've pasted the config below. Has anyone got any success with it? Guillaume. - Configuration Service name Tomcat4124 Catalina home D:\jakarta-tomcat-4.1.24 Catalina base D:\jakarta-tomcat-4.1.24 Tomcat config file D:\jakarta-tomcat-4.1.24\conf\server.xml Use security manager 0 Security policy file D:\jakarta-tomcat-4.1.24\conf\catalina.policy Java home c:\j2sdk1.4.1 JVM Path c:\j2sdk1.4.1\jre\bin\server\jvm.dll Classpath D:\jakarta-tomcat-4.1.24\bin\bootstrap.jar;D:\jakarta-tomcat-4.1.24\common\lib;c:\j2sdk1.4.1\lib\tools.jar Temp directory D:\jakarta-tomcat-4.1.24\temp System.out file D:\jakarta-tomcat-4.1.24\logs System.err file D:\jakarta-tomcat-4.1.24\logs Initial heap Max heap Stack size JVM server Additional JVM option -verbose:gc - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Over 1000 threads running in production
Chuck, Many thanks for your feeback. From: Guillaume Lahitette [mailto:[EMAIL PROTECTED] Subject: Over 1000 threads running in production Our operations team discovered over 1000 java threads running on each production server. It appears that about 20 threads are added to this count every hour. Since no one else seems to be reporting this kind of problem, odds are it's a bug in one or more of the applications you're running under Tomcat. This is what I am suspecting too. It's a large application with third party modules (open source, outsourced,...). I am not too sure on how to best track it down. Pointers welcome! There's an off chance it could be a problem in the JRE, since you're using a level that's been out of date and unsupported for quite some time. We'll try to run some tests with the latest 1.4.2 JDK...If performance increases, it will be an easy sell (although we'll want plenty of regression testing). If not, operation teams will argue about the upgrade cost, benefits The number of threads Tomcat uses to service requests is controlled by the maxProcessors parameter of the Connector element you're using; for 4.1, the default value is 20, and it would be very, very unusual to ever need to set it to much over 100. Since you don't seem to be aware of this configuration parameter, you're probably using the default. We're using the default values for TC4.1.26: minProcessors=5 maxProcessors=75. Other than a few created for internal purposes by Tomcat and the JVM itself, additional threads must come from the applications. How can I tell? Can I monitor who/when these are created? I would be highly suspicious of some piece of code kicking off a thread and then losing track of it and the thread not having any logic to terminate itself when it's no longer needed. Slightly off topic: a third party module uses RMI: would RMI kick off a new thread for each new RMI connection? We're suspecting this module does NOT release RMI connections correctly and would explain these accumulation of threads. I'm trying to get this confirmed... e.. Any known issues with our setup? Can't tell unless you pass on the .xml files. The server.xml is per default, except for port numbers. Any other relevant XML files? Tons of thanks for your feedback Chuck! Guillaume - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Over 1000 threads running in production
Tim, Thank you. This is very helpful. I've tried kill -3 pid (following a google hit: http://www.findarticles.com/p/articles/mi_m0MLW/is_3_3/ai_114495281/print) but can't see the thread dump in catalina.out nor localhost_logtxt files. What did I miss (this is Linux)? If you could share your perl / grep / wc scripts, that would surely jumpstart me and give me some ideas! Thanks in advance. Cheers, Guillaume -Original Message- From: Tim Funk [mailto:[EMAIL PROTECTED] Sent: 15 March 2005 11:03 To: Tomcat Users List Subject: Re: Over 1000 threads running in production Take periodic thread dumps and read them. perl, grep, and wc are your friends to parse the data without going insane. The thread dumps should give you a general idea of who created them. -Tim Guillaume Lahitette wrote: Hello Tomcat'oids, This is Tomcat 4.1.26 on Linux with JDK 1.4.1_03, load balanced accross 2 servers by a BigIP device. User base is around 50. Our operations team discovered over 1000 java threads running on each production server. It appears that about 20 threads are added to this count every hour. Can someone clarify: a.. how are threads open? Is it 1 thread per HTTP request? Can I monitor this / gather more details? b.. how and when are threads closed? garbage collected? c.. are threads re-used / pooled? d.. how to tune the number of threads Tomcat uses appropriately? e.. Any known issues with our setup? Any other recommendations, best practices and war stories are greatly appreciated! Many thanks in advance. Guillaume - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Over 1000 threads running in production
Found the solution: to know which PID to kill -3 for, use ps -ef --forest to detect the parent thread which has the *most* child threads. Then the thread dump shows in catalina.out. This is where I found the answer: http://groups-beta.google.com/group/weblogic.developer.interest.performance/ browse_thread/thread/336d16199150720f/40be68d56007a1ea?q=periodic+%22thread+ dumps%22+linux+%22kill+-3%22#40be68d56007a1ea Guillaume -Original Message- From: Guillaume Lahitette [mailto:[EMAIL PROTECTED] Sent: 15 March 2005 11:34 To: Tomcat Users List Subject: RE: Over 1000 threads running in production Tim, Thank you. This is very helpful. I've tried kill -3 pid (following a google hit: http://www.findarticles.com/p/articles/mi_m0MLW/is_3_3/ai_11449528 1/print) but can't see the thread dump in catalina.out nor localhost_logtxt files. What did I miss (this is Linux)? If you could share your perl / grep / wc scripts, that would surely jumpstart me and give me some ideas! Thanks in advance. Cheers, Guillaume -Original Message- From: Tim Funk [mailto:[EMAIL PROTECTED] Sent: 15 March 2005 11:03 To: Tomcat Users List Subject: Re: Over 1000 threads running in production Take periodic thread dumps and read them. perl, grep, and wc are your friends to parse the data without going insane. The thread dumps should give you a general idea of who created them. -Tim Guillaume Lahitette wrote: Hello Tomcat'oids, This is Tomcat 4.1.26 on Linux with JDK 1.4.1_03, load balanced accross 2 servers by a BigIP device. User base is around 50. Our operations team discovered over 1000 java threads running on each production server. It appears that about 20 threads are added to this count every hour. Can someone clarify: a.. how are threads open? Is it 1 thread per HTTP request? Can I monitor this / gather more details? b.. how and when are threads closed? garbage collected? c.. are threads re-used / pooled? d.. how to tune the number of threads Tomcat uses appropriately? e.. Any known issues with our setup? Any other recommendations, best practices and war stories are greatly appreciated! Many thanks in advance. Guillaume - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Over 1000 threads running in production
Hello Tomcat'oids, This is Tomcat 4.1.26 on Linux with JDK 1.4.1_03, load balanced accross 2 servers by a BigIP device. User base is around 50. Our operations team discovered over 1000 java threads running on each production server. It appears that about 20 threads are added to this count every hour. Can someone clarify: a.. how are threads open? Is it 1 thread per HTTP request? Can I monitor this / gather more details? b.. how and when are threads closed? garbage collected? c.. are threads re-used / pooled? d.. how to tune the number of threads Tomcat uses appropriately? e.. Any known issues with our setup? Any other recommendations, best practices and war stories are greatly appreciated! Many thanks in advance. Guillaume
RE: OutOfMemory / JMeter / Profiler questions
Thanks Harry for your input. But we just can't do any upgrades to the production environment at this point :( Hopefully soon. Guillaume -Original Message- From: Harry Mantheakis [mailto:[EMAIL PROTECTED] Sent: 03 March 2005 10:45 To: Tomcat Users List Subject: Re: OutOfMemory / JMeter / Profiler questions We've started performance testing one of our REMOTE web apps using JMeter. We're gathering benchmark data before doing further fine tuning. Details: Win2K only have ssh + cygwin access to this remote server JDK 1.4.1_03 Tomcat 4.1.26, running as a service: a.. Use security manager 1 b.. Security policy file D:\Tomcat4\conf\catalina.policy c.. Initial heap 256 d.. Max heap 512 e.. Stack size 256 f.. JVM server I cannot help you with respect to your OOM problems, but I would recommend that you consider upgrading to JVM 1.4.2. You should also be able to migrate your app to TC 5.0.28 without too much grief. You should probably avoid TC 5.5.x for the meantime. JVM 1.4.2 and TC 5.0.28 play well together. Good luck in case! Harry Mantheakis London, UK - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemory / JMeter / Profiler questions
Alan, Thanks for your feedback. You got me curious here: Why does/would Tomcat reload sessions after startup? Aren't the sessions destroyed upon Tomcat shutdown? Also, I could only find a $TOMCAT_HOME/work/Standalone/localhost/prti-rpt-engine.2.4.2.b17/SESSIONS.se r, which is NOT the context I have been load testing. Sessions should be serialized per context, shouldn't they? We do not store anything in the session (our test just calls a servlet which generate a PDF report) so I can't see why the sessions would be using so much memory. But I'll definitely keep your idea in mind for other tests. I should hopefully be able to run the same tests on our staging platform and see if we get similar behaviors. Thanks for your inputs. Please keep them coming! Cheers, Guillaume -Original Message- From: Flisch, Alan [mailto:[EMAIL PROTECTED] Sent: 03 March 2005 12:17 To: Tomcat Users List Subject: RE: OutOfMemory / JMeter / Profiler questions You probably have either very large or very many sessions which Tomcat is attempting to reload on startup. Tomcat serialises sessions into files called SESSIONS.ser (in the application directories under the work dir) and then when it is restarted it attempts to reload them all. I presume this behaviour can be turned off. In terms of testing your app, you want to figure out whether you have a memory leak issue with your app, or simply that your max heap size is set too low for the load you are running. To check for memory leaks, you could run jmeter reasonably (although not too hard) excercising as much of your app as you can, repetitively and for an extended period. If it eventually keels over then you may need to investigate memory leaks with a profiler. Another possibility is that you may not be invalidating sessions and they are just being left to expire naturally. This can use up a lot of memory if you aren't careful and I supect is a quite likely source of your problems. Cheers! -Original Message- From: Guillaume Lahitette [mailto:[EMAIL PROTECTED] Sent: 03 March 2005 09:51 To: tomcat-user@jakarta.apache.org Subject: OutOfMemory / JMeter / Profiler questions Hello Tomcat'oids! We've started performance testing one of our REMOTE web apps using JMeter. We're gathering benchmark data before doing further fine tuning. Details: Win2K only have ssh + cygwin access to this remote server JDK 1.4.1_03 Tomcat 4.1.26, running as a service: a.. Use security manager 1 b.. Security policy file D:\Tomcat4\conf\catalina.policy c.. Initial heap 256 d.. Max heap 512 e.. Stack size 256 f.. JVM server Issue: We are getting OutOfMemory errors with very few threads simulated (as low as 5). More problematic, we've seen the OOM just after a Tomcat service restart! From the stack trace below, you can see we get the OOM before any of our code is executed :( Questions: a.. Anyone has seen this behavior upon Tomcat start up? b.. Anything particular to watch for in our JMeter test plan? c.. Would a profiler help? Could it profile a remote Tomcat installation? Any +/- feedback on Eclipse Profiler plug in (http://eclipsecolorer.sourceforge.net/index_profiler.html)? We'll work on gathering more data (e.g. periodic free / allocated memory dumps). Untill then, thank you for sharing your experiences, suggestions, code,...! Cheers, Guillaume javax.servlet.ServletException: Servlet execution threw an exception at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(A pplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.access$000(Applica tionFilterChain.java:98) at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationF ilterChain.java:176) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ApplicationFilterChain.doFilter(Applicati onFilterChain.java:172) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapp erValve.java:256) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveCon text.invokeNext(StandardPipeline.java:643) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline. java:480) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) at org.apache.catalina.core.StandardContextValve.invoke(StandardConte xtValve.java:191) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveCon text.invokeNext(StandardPipeline.java:643) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline. java:480) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) at org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2416) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValv e.java:180) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveCon text.invokeNext(StandardPipeline.java:643
RE: OutOfMemory / JMeter / Profiler questions
Matt, Thanks for your feedbackwhich triggered more questions below! -Original Message- From: Dale, Matt [mailto:[EMAIL PROTECTED] Sent: 04 March 2005 15:26 To: Tomcat Users List Subject: RE: OutOfMemory / JMeter / Profiler questions Hi, Tomcat always serializes sessions on shutdown and reloads them on startup. Can anyone explain what is the usage for this? You can not re-use a session after a restart, can you??? I would think new session ID is created. This is the default behaviour but can be changed. Does anyone know how? If I can't see any usage for this, I'm very keen on disabling it. You are right in thinking that sessions are serialized per context though. Are you using the standard manager or the persistent manager as they are stored differently with the persistent manager. First time I hear about these! I could see a commented PersistentManager in server.xml so I guess I am using the standard manager. Can you tell me more about these? Why/When would I use the PersistentManager? Although you don't store anything in a session, do you perhaps instanciate it? The number of sessions you currently have can be viewed from the manager application. Yes, I've been using it a lot for context stop / start / reload and monitoring the sessions... Cheers, G Ta Matt -Original Message- From: Guillaume Lahitette [mailto:[EMAIL PROTECTED] Sent: 04 March 2005 15:15 To: Tomcat Users List Subject: RE: OutOfMemory / JMeter / Profiler questions Alan, Thanks for your feedback. You got me curious here: Why does/would Tomcat reload sessions after startup? Aren't the sessions destroyed upon Tomcat shutdown? Also, I could only find a $TOMCAT_HOME/work/Standalone/localhost/prti-rpt-engine.2.4.2.b17/S ESSIONS.se r, which is NOT the context I have been load testing. Sessions should be serialized per context, shouldn't they? We do not store anything in the session (our test just calls a servlet which generate a PDF report) so I can't see why the sessions would be using so much memory. But I'll definitely keep your idea in mind for other tests. I should hopefully be able to run the same tests on our staging platform and see if we get similar behaviors. Thanks for your inputs. Please keep them coming! Cheers, Guillaume -Original Message- From: Flisch, Alan [mailto:[EMAIL PROTECTED] Sent: 03 March 2005 12:17 To: Tomcat Users List Subject: RE: OutOfMemory / JMeter / Profiler questions You probably have either very large or very many sessions which Tomcat is attempting to reload on startup. Tomcat serialises sessions into files called SESSIONS.ser (in the application directories under the work dir) and then when it is restarted it attempts to reload them all. I presume this behaviour can be turned off. In terms of testing your app, you want to figure out whether you have a memory leak issue with your app, or simply that your max heap size is set too low for the load you are running. To check for memory leaks, you could run jmeter reasonably (although not too hard) excercising as much of your app as you can, repetitively and for an extended period. If it eventually keels over then you may need to investigate memory leaks with a profiler. Another possibility is that you may not be invalidating sessions and they are just being left to expire naturally. This can use up a lot of memory if you aren't careful and I supect is a quite likely source of your problems. Cheers! -Original Message- From: Guillaume Lahitette [mailto:[EMAIL PROTECTED] Sent: 03 March 2005 09:51 To: tomcat-user@jakarta.apache.org Subject: OutOfMemory / JMeter / Profiler questions Hello Tomcat'oids! We've started performance testing one of our REMOTE web apps using JMeter. We're gathering benchmark data before doing further fine tuning. Details: Win2K only have ssh + cygwin access to this remote server JDK 1.4.1_03 Tomcat 4.1.26, running as a service: a.. Use security manager 1 b.. Security policy file D:\Tomcat4\conf\catalina.policy c.. Initial heap 256 d.. Max heap 512 e.. Stack size 256 f.. JVM server Issue: We are getting OutOfMemory errors with very few threads simulated (as low as 5). More problematic, we've seen the OOM just after a Tomcat service restart! From the stack trace below, you can see we get the OOM before any of our code is executed :( Questions: a.. Anyone has seen this behavior upon Tomcat start up? b.. Anything particular to watch for in our JMeter test plan? c.. Would a profiler help? Could it profile a remote Tomcat installation? Any +/- feedback on Eclipse Profiler plug in (http://eclipsecolorer.sourceforge.net/index_profiler.html)? We'll work on gathering more data (e.g. periodic free / allocated memory dumps). Untill then, thank you for sharing your experiences, suggestions, code
OutOfMemory / JMeter / Profiler questions
Hello Tomcat'oids! We've started performance testing one of our REMOTE web apps using JMeter. We're gathering benchmark data before doing further fine tuning. Details: Win2K only have ssh + cygwin access to this remote server JDK 1.4.1_03 Tomcat 4.1.26, running as a service: a.. Use security manager 1 b.. Security policy file D:\Tomcat4\conf\catalina.policy c.. Initial heap 256 d.. Max heap 512 e.. Stack size 256 f.. JVM server Issue: We are getting OutOfMemory errors with very few threads simulated (as low as 5). More problematic, we've seen the OOM just after a Tomcat service restart! From the stack trace below, you can see we get the OOM before any of our code is executed :( Questions: a.. Anyone has seen this behavior upon Tomcat start up? b.. Anything particular to watch for in our JMeter test plan? c.. Would a profiler help? Could it profile a remote Tomcat installation? Any +/- feedback on Eclipse Profiler plug in (http://eclipsecolorer.sourceforge.net/index_profiler.html)? We'll work on gathering more data (e.g. periodic free / allocated memory dumps). Untill then, thank you for sharing your experiences, suggestions, code,...! Cheers, Guillaume javax.servlet.ServletException: Servlet execution threw an exception at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:98) at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:176) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:172) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) at org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2416) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643) at org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:171) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:577) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174) at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:223) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:601) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:392) at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:565) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:619) at java.lang.Thread.run(Thread.java:536) root cause java.lang.OutOfMemoryError
