Need help with performance issue - Tomcat 4.1.X
With Tomcat 4.1.x We've recently run into an issue where some of our pages either a) take a really long time to come up (20+ seconds) or b) come up, but never really finish loading (the status bar in IE shows the the response hasn't finished). These are very simple pages (for example, a login page) and our server load is minimal (maybe 10 concurrent users). Then, mysteriously, the problem will go away by itself. It will also return. Given the information above, I'm not expecting any solutions, but I could use some help steering me in the right direction with things to check. We're hooking up monitoring on the systems (Linux) to examine memory and CPU utilization during the slowdowns. Any things in particular we should be examining to try to find the problem? Any idea why the pages would never fully return from the server? TIA -- Dave - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Need help with performance issue - Tomcat 4.1.X
Yes, these occur internally and externally. There does not appear to be any network traffic that is causing this (according to the network people...). We're also seeing this on two different environments on two distinct networks. After some more digging, there is just NOTHING happening on the machine. Dual CPU, with both cpu's sitting at about 99% idle. 1 GB of RAM, and it's not even being all used. Tomcat is using about 327MB of RAM with maybe 40(?) threads running. When I submit my request to get the login page (not to process the login, just to serve the login page), I couple of the java threads will spin up and take maybe 3% of the cpu, but it still takes over 30 seconds to get the page. Still stuggling for ideas. -- Dave Are these results occurring when tested on a local area network as well? I have had an issue like that, but it turned out to be a networking issue..not tomcat. --- [EMAIL PROTECTED] wrote: With Tomcat 4.1.x We've recently run into an issue where some of our pages either a) take a really long time to come up (20+ seconds) or b) come up, but never really finish loading (the status bar in IE shows the the response hasn't finished). These are very simple pages (for example, a login page) and our server load is minimal (maybe 10 concurrent users). Then, mysteriously, the problem will go away by itself. It will also return. Given the information above, I'm not expecting any solutions, but I could use some help steering me in the right direction with things to check. We're hooking up monitoring on the systems (Linux) to examine memory and CPU utilization during the slowdowns. Any things in particular we should be examining to try to find the problem? Any idea why the pages would never fully return from the server? TIA -- Dave - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] __ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Need help with performance issue - Tomcat 4.1.X
I don't believe that the host lookup or db connection is the issue. The page we are serving up is the login page. I will create a test hello world jsp to see if the problem still occurs with that page. -- Dave In a situation like that, the two things I check first are: 1) DNS lookup failing, especially a reverse lookup..do you have Tomcat trying to look up a hostname for access logging, perhaps? 2) database connection timing out Also, I would look at garbage collection, but I doubt that's it. John On Fri, 11 Jul 2003 19:22:41 +, [EMAIL PROTECTED] wrote: Yes, these occur internally and externally. There does not appear to be any network traffic that is causing this (according to the network people...). We're also seeing this on two different environments on two distinct networks. After some more digging, there is just NOTHING happening on the machine. Dual CPU, with both cpu's sitting at about 99% idle. 1 GB of RAM, and it's not even being all used. Tomcat is using about 327MB of RAM with maybe 40(?) threads running. When I submit my request to get the login page (not to process the login, just to serve the login page), I couple of the java threads will spin up and take maybe 3% of the cpu, but it still takes over 30 seconds to get the page. Still stuggling for ideas. -- Dave Are these results occurring when tested on a local area network as well? I have had an issue like that, but it turned out to be a networking issue..not tomcat. --- [EMAIL PROTECTED] wrote: With Tomcat 4.1.x We've recently run into an issue where some of our pages either a) take a really long time to come up (20+ seconds) or b) come up, but never really finish loading (the status bar in IE shows the the response hasn't finished). These are very simple pages (for example, a login page) and our server load is minimal (maybe 10 concurrent users). Then, mysteriously, the problem will go away by itself. It will also return. Given the information above, I'm not expecting any solutions, but I could use some help steering me in the right direction with things to check. We're hooking up monitoring on the systems (Linux) to examine memory and CPU utilization during the slowdowns. Any things in particular we should be examining to try to find the problem? Any idea why the pages would never fully return from the server? TIA -- Dave - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] __ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/ - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Need help with performance issue - Tomcat 4.1.X
We are currently starting up with -Xmx256M. Java is currently using about 327MB and has been that way for hours. I haven't really seen any fluxuations at all, which leans me away from the garbage collection issue. I created a hello.jsp, which is completely separate from our application. Same results. It takes about 30 seconds to return. In the hello.jsp, the results of the page return instantly to the browser, but then it's as if tomcat just won't close the connection to the browser. The page is there, fully rendered, but just sits there waiting for the server to tell it it is done. Is there some way that the tomcat connections could be failing to terminate properly? Our current maxProcessors = 75 acceptCount = 10 (which is probably low but right now we have only a couple users on the system). We're not using 1.4 so those options are out. -- Dave Could be a Memory Leak/Garbage Collection issue (we had a similar problem with a memory intensive app), maybe your heap is too small and java is running many Full GC's. Start java with -verbose:gc and look in tomcat/logs/catalina.out for Garbage Collections. (set Environment Variables JAVA_OPTS or CATALINA_OPTS in bin/catalina.sh to do this in Tomcat) Try using a bigger Heapsize (though if you've got a Memory Leak that will only delay your problem) or set initial Heapsize to same as maximum, for example 128MB: -Xmx128m -Xms128m Modify the Connector you are using to access Tomcat (in tomcat/conf/server.xml) and try using more Tomcat Processors (maxProcessors=XX) or a bigger accept queue length (acceptCount=XX) (test value for acceptCount: at least concurrent users x 3) Try using another method of garbage collection, if you're using JDK 1.4.1 i'd try either ConcurrentGC with ParNewGC (ParNewGC on Multi-CPU machines): -XX:+UseConcMarkSweepGC -XX:+UseParNewGC or ParallelGC with AdaptiveSizePolicy (saves you the work of Java Heap usage analyzing :-) : -XX:UseParallelGC -XX:+UseAdaptiveSizePolicy A good article on GC can be found here: http://wireless.java.sun.com/midp/articles/garbagecollection2/ At 18:04 11.07.2003 +, you wrote: With Tomcat 4.1.x We've recently run into an issue where some of our pages either a) take a really long time to come up (20+ seconds) or b) come up, but never really finish loading (the status bar in IE shows the the response hasn't finished). These are very simple pages (for example, a login page) and our server load is minimal (maybe 10 concurrent users). Then, mysteriously, the problem will go away by itself. It will also return. Given the information above, I'm not expecting any solutions, but I could use some help steering me in the right direction with things to check. We're hooking up monitoring on the systems (Linux) to examine memory and CPU utilization during the slowdowns. Any things in particular we should be examining to try to find the problem? Any idea why the pages would never fully return from the server? TIA -- Dave - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Need help with performance issue - Tomcat 4.1.X
Another followup. If we try to access the root context of the server, which is not available, the same thing happens. I get back a 500 error immediately, but then the page doesn't finish loading (i.e. get all the response back from the server) for another 20 seconds or so. This seems like a connection problem rather than a memory problem. Thoughts? -- Dave We are currently starting up with -Xmx256M. Java is currently using about 327MB and has been that way for hours. I haven't really seen any fluxuations at all, which leans me away from the garbage collection issue. I created a hello.jsp, which is completely separate from our application. Same results. It takes about 30 seconds to return. In the hello.jsp, the results of the page return instantly to the browser, but then it's as if tomcat just won't close the connection to the browser. The page is there, fully rendered, but just sits there waiting for the server to tell it it is done. Is there some way that the tomcat connections could be failing to terminate properly? Our current maxProcessors = 75 acceptCount = 10 (which is probably low but right now we have only a couple users on the system). We're not using 1.4 so those options are out. -- Dave Could be a Memory Leak/Garbage Collection issue (we had a similar problem with a memory intensive app), maybe your heap is too small and java is running many Full GC's. Start java with -verbose:gc and look in tomcat/logs/catalina.out for Garbage Collections. (set Environment Variables JAVA_OPTS or CATALINA_OPTS in bin/catalina.sh to do this in Tomcat) Try using a bigger Heapsize (though if you've got a Memory Leak that will only delay your problem) or set initial Heapsize to same as maximum, for example 128MB: -Xmx128m -Xms128m Modify the Connector you are using to access Tomcat (in tomcat/conf/server.xml) and try using more Tomcat Processors (maxProcessors=XX) or a bigger accept queue length (acceptCount=XX) (test value for acceptCount: at least concurrent users x 3) Try using another method of garbage collection, if you're using JDK 1.4.1 i'd try either ConcurrentGC with ParNewGC (ParNewGC on Multi-CPU machines): -XX:+UseConcMarkSweepGC -XX:+UseParNewGC or ParallelGC with AdaptiveSizePolicy (saves you the work of Java Heap usage analyzing :-) : -XX:UseParallelGC -XX:+UseAdaptiveSizePolicy A good article on GC can be found here: http://wireless.java.sun.com/midp/articles/garbagecollection2/ At 18:04 11.07.2003 +, you wrote: With Tomcat 4.1.x We've recently run into an issue where some of our pages either a) take a really long time to come up (20+ seconds) or b) come up, but never really finish loading (the status bar in IE shows the the response hasn't finished). These are very simple pages (for example, a login page) and our server load is minimal (maybe 10 concurrent users). Then, mysteriously, the problem will go away by itself. It will also return. Given the information above, I'm not expecting any solutions, but I could use some help steering me in the right direction with things to check. We're hooking up monitoring on the systems (Linux) to examine memory and CPU utilization during the slowdowns. Any things in particular we should be examining to try to find the problem? Any idea why the pages would never fully return from the server? TIA -- Dave - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Need help with performance issue - Tomcat 4.1.X
I definitely don't know enough about how the memory settings in java to be of much use on this part. I do not know if we've tried 512. We probably should. As for the 327, is that actually unreasonable? The mx setting specfies the maximum heap size. Can the thread stack (specified using -Xss, which we're not using) be taking up the remaining 75MB? We have about 40 threads started right now. Like I said, I don't know enough about java's memory allocation, but I didn't think that the -Xmx set the maximum allowable total memory for the JVM. Am I mistaken? Even if memory were a problem, which it may be, would that account for a 20 second delay between serving a page and closing the connection? Remember that we currently have no load and only a couple users on the system. -- Dave I'm certainly no guru, but if you are setting a max of 256 and then your app is soaking up 327 with no load whatsoever, I'd say you have a problem. Have you tried a max of 512? John On Fri, 11 Jul 2003 20:08:10 +, [EMAIL PROTECTED] wrote: We are currently starting up with -Xmx256M. Java is currently using about 327MB and has been that way for hours. I haven't really seen any fluxuations at all, which leans me away from the garbage collection issue. I created a hello.jsp, which is completely separate from our application. Same results. It takes about 30 seconds to return. In the hello.jsp, the results of the page return instantly to the browser, but then it's as if tomcat just won't close the connection to the browser. The page is there, fully rendered, but just sits there waiting for the server to tell it it is done. Is there some way that the tomcat connections could be failing to terminate properly? Our current maxProcessors = 75 acceptCount = 10 (which is probably low but right now we have only a couple users on the system). We're not using 1.4 so those options are out. -- Dave Could be a Memory Leak/Garbage Collection issue (we had a similar problem with a memory intensive app), maybe your heap is too small and java is running many Full GC's. Start java with -verbose:gc and look in tomcat/logs/catalina.out for Garbage Collections. (set Environment Variables JAVA_OPTS or CATALINA_OPTS in bin/catalina.sh to do this in Tomcat) Try using a bigger Heapsize (though if you've got a Memory Leak that will only delay your problem) or set initial Heapsize to same as maximum, for example 128MB: -Xmx128m -Xms128m Modify the Connector you are using to access Tomcat (in tomcat/conf/server.xml) and try using more Tomcat Processors (maxProcessors=XX) or a bigger accept queue length (acceptCount=XX) (test value for acceptCount: at least concurrent users x 3) Try using another method of garbage collection, if you're using JDK 1.4.1 i'd try either ConcurrentGC with ParNewGC (ParNewGC on Multi-CPU machines): -XX:+UseConcMarkSweepGC -XX:+UseParNewGC or ParallelGC with AdaptiveSizePolicy (saves you the work of Java Heap usage analyzing :-) : -XX:UseParallelGC -XX:+UseAdaptiveSizePolicy A good article on GC can be found here: http://wireless.java.sun.com/midp/articles/garbagecollection2/ At 18:04 11.07.2003 +, you wrote: With Tomcat 4.1.x We've recently run into an issue where some of our pages either a) take a really long time to come up (20+ seconds) or b) come up, but never really finish loading (the status bar in IE shows the the response hasn't finished). These are very simple pages (for example, a login page) and our server load is minimal (maybe 10 concurrent users). Then, mysteriously, the problem will go away by itself. It will also return. Given the information above, I'm not expecting any solutions, but I could use some help steering me in the right direction with things to check. We're hooking up monitoring on the systems (Linux) to examine memory and CPU utilization during the slowdowns. Any things in particular we should be examining to try to find the problem? Any idea why the pages would never fully return from the server? TIA -- Dave - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Re: Need help with performance issue - Tomcat 4.1.X
But again, the mx is only setting the heap size, not the whole JVM. The thread stack is controlled with an entirely different parameter for example. I found this with a quick search: The -Xmx setting controls the maximum size of the Java heap. However, the Java heap represents only part of the memory taken up by a running Java application. There is space required for code, data space used by the JIT, plus any memory malloced by native code drivers associated with the application running in the JVM. So the overall memory picture for Java applications turns out to be more complex than just the -Xmx value. -- Dave I have no idea, but based on past experience, if you are using more RAM than you've allocated, then you have a swap situation. If the max setting wasn't an actual max, and you could blow past it whenever you needed it, why on earth would you need it in the first place? John On Fri, 11 Jul 2003 20:24:48 +, [EMAIL PROTECTED] wrote: I definitely don't know enough about how the memory settings in java to be of much use on this part. I do not know if we've tried 512. We probably should. As for the 327, is that actually unreasonable? The mx setting specfies the maximum heap size. Can the thread stack (specified using -Xss, which we're not using) be taking up the remaining 75MB? We have about 40 threads started right now. Like I said, I don't know enough about java's memory allocation, but I didn't think that the -Xmx set the maximum allowable total memory for the JVM. Am I mistaken? Even if memory were a problem, which it may be, would that account for a 20 second delay between serving a page and closing the connection? Remember that we currently have no load and only a couple users on the system. -- Dave I'm certainly no guru, but if you are setting a max of 256 and then your app is soaking up 327 with no load whatsoever, I'd say you have a problem. Have you tried a max of 512? John On Fri, 11 Jul 2003 20:08:10 +, [EMAIL PROTECTED] wrote: We are currently starting up with -Xmx256M. Java is currently using about 327MB and has been that way for hours. I haven't really seen any fluxuations at all, which leans me away from the garbage collection issue. I created a hello.jsp, which is completely separate from our application. Same results. It takes about 30 seconds to return. In the hello.jsp, the results of the page return instantly to the browser, but then it's as if tomcat just won't close the connection to the browser. The page is there, fully rendered, but just sits there waiting for the server to tell it it is done. Is there some way that the tomcat connections could be failing to terminate properly? Our current maxProcessors = 75 acceptCount = 10 (which is probably low but right now we have only a couple users on the system). We're not using 1.4 so those options are out. -- Dave Could be a Memory Leak/Garbage Collection issue (we had a similar problem with a memory intensive app), maybe your heap is too small and java is running many Full GC's. Start java with -verbose:gc and look in tomcat/logs/catalina.out for Garbage Collections. (set Environment Variables JAVA_OPTS or CATALINA_OPTS in bin/catalina.sh to do this in Tomcat) Try using a bigger Heapsize (though if you've got a Memory Leak that will only delay your problem) or set initial Heapsize to same as maximum, for example 128MB: -Xmx128m -Xms128m Modify the Connector you are using to access Tomcat (in tomcat/conf/server.xml) and try using more Tomcat Processors (maxProcessors=XX) or a bigger accept queue length (acceptCount=XX) (test value for acceptCount: at least concurrent users x 3) Try using another method of garbage collection, if you're using JDK 1.4.1 i'd try either ConcurrentGC with ParNewGC (ParNewGC on Multi-CPU machines): -XX:+UseConcMarkSweepGC -XX:+UseParNewGC or ParallelGC with AdaptiveSizePolicy (saves you the work of Java Heap usage analyzing :-) : -XX:UseParallelGC -XX:+UseAdaptiveSizePolicy A good article on GC can be found here: http://wireless.java.sun.com/midp/articles/garbagecollection2/ At 18:04 11.07.2003 +, you wrote: With Tomcat 4.1.x We've recently run into an issue where some of our pages either a) take a really long time to come up (20+ seconds) or b) come up, but never really finish loading (the status bar in IE shows the the response hasn't finished). These are very simple pages (for example, a login page) and our server load is minimal (maybe 10 concurrent users). Then, mysteriously, the problem will go away by
Re: Need help with performance issue - Tomcat 4.1.X
Yes, still running on the same tomcat, which is standalone BTW. The problem occurs with both IE and mozilla, but a little differently. Mozilla tends to render the results immediately, then sit there for 20 seconds. IE will wait a bit, render the results, then sit there. I'm assuming the difference is simply an artifact of when the browser chooses to render its cache on an open connection. Neither browser is closing the connection to fast... Opposite problem :-) And no, I haven't tried any of your other recommended steps yet. We're trying to do as much as we can without bringing down the system, since it's a production system (and we have yet to reproduce on a test system). -- Dave If your application is running on same Tomcat/JVM then blocking GC calls could still lead to that effect for hello.jsp too. Are you using Apache/mod_jk/Tomcat or Tomcat standalone? HTTPS Connections? Does the Problem occur only with IE or with other Browsers (Mozilla) too? I read once something about a problem of IE closing http connections too fast when it thinks the page transfer is finished. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
