Re: [Resin-interest] Perf Issues
It's also worth mentioning that for serious application tuning, there is no substitute for a good profiling tool. In the past I have had great results with JProfiler: http://www.ej-technologies.com/products/jprofiler/overview.html I've even gone so far as to take one node in a production cluster, give it a reduced load profile at the balancer (because profiling slows things down considerably), and run the profiler against it live. You'll be amazed what you discover. Jeff On Sat, Mar 21, 2009 at 7:39 AM, Adam Allgaier allgai...@yahoo.com wrote: I would plug jconsole into your resin instance and watch what's happening to the JVM memory. Could be loose open threads (and large thread size) that grows over time and eats up free memory. Restarting would kill all the threads and free the memory. http://java.sun.com/j2se/1.5.0/docs/guide/management/jconsole.html Adam - Original Message From: Stargazer starga...@blueyonder.co.uk To: General Discussion for the Resin application server resin-interest@caucho.com Sent: Saturday, March 21, 2009 4:39:56 AM Subject: Re: [Resin-interest] Perf Issues Ronan Lucio wrote: Hi, We have had a perf issue. Our servers have 4Gb RAM. It has just Resin and Apache installed. The problem is, when I start Resin, the whole server uses about 2.5 Gb RAM After that memory usage keep growing til it reachs the 4Gb RAM, use swap and so on. After few hours the application start getting slow. Analyzing the server sources, it's using so low CPU, load about 1... I see none overload evidence, except for RAM memory. So I just restart Resin and/or Apache and application gets fast again, but few ours later it will raise the same issue. I have been suffering from exactly those symptoms for years. Do your httpd processes consume all the swap, with top showing some at 450Mb? (default httpd.conf values) I.e does restarting apache alone, and not resin, cause the swap to drop back down to normal until the next slow growth starts it all again? That is my pattern. So when I profile resin theres no unusual growth even through the whole server is effectiviley dead until restart. Given that, whos to blame? Could mod_caucho somehow be at fault even though resin itself is ok? I have another identical server running a couple of PHP CMS apps, no resin or java in sight - because of this problem actually. The plan is to move everything over when stable but these have become too critical to play with. Their typical httpd swap use is 25Mb, and its the default httpd.conf. I would dearly love to know what the httpd on the failing server thinks it needs to hold onto 450Mb for, without tweaking there could be 20 of these. There are other non-quercus PHP apps running on that server btw. My solution is to kill child httpd processes at a far quicker rate than you'd normally want, and it works of sorts: IfModule prefork.c StartServers 8 MinSpareServers 5 MaxSpareServers 20 ServerLimit 256 MaxClients 256 MaxRequestsPerChild 96 /IfModule IfModule worker.c StartServers 2 MaxClients 150 MinSpareThreads 25 MaxSpareThreads 75 ThreadsPerChild 25 MaxRequestsPerChild 96 /IfModule This problem usually happens on peaks hours. So we upgraded RAM memory to 8Gb with a PAE kernel. Although it doesn't reach the 8G RAM, slow moments gots for frequent. It seems to work worse that way (8Gb + PAE kernel). The question is: Is there everyone having the same issue with Resin (3.1.6)? My doubt if such problem resides either on Resin or on my application. All versions prior to 3.1.6, and currently with 3.2.1 Pro Thanks, Ronan ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.0.238 / Virus Database: 270.11.21/2014 - Release Date: 03/20/09 06:59:00 ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Perf Issues
Adam Allgaier wrote: I would plug jconsole into your resin instance and watch what's happening to the JVM memory. Could be loose open threads (and large thread size) that grows over time and eats up free memory. Restarting would kill all the threads and free the memory. http://java.sun.com/j2se/1.5.0/docs/guide/management/jconsole.html Adam Thanks, but as I'd hoped I'd made clear Resin itself shows flat memory use, whether using jconsole, jprofiler, /resin-admin or any of the other jvmti tools I've used to try to fix this. The memory consumption occurs outside java. - Original Message From: Stargazer starga...@blueyonder.co.uk To: General Discussion for the Resin application server resin-interest@caucho.com Sent: Saturday, March 21, 2009 4:39:56 AM Subject: Re: [Resin-interest] Perf Issues Ronan Lucio wrote: Hi, We have had a perf issue. Our servers have 4Gb RAM. It has just Resin and Apache installed. The problem is, when I start Resin, the whole server uses about 2.5 Gb RAM After that memory usage keep growing til it reachs the 4Gb RAM, use swap and so on. After few hours the application start getting slow. Analyzing the server sources, it's using so low CPU, load about 1... I see none overload evidence, except for RAM memory. So I just restart Resin and/or Apache and application gets fast again, but few ours later it will raise the same issue. I have been suffering from exactly those symptoms for years. Do your httpd processes consume all the swap, with top showing some at 450Mb? (default httpd.conf values) I.e does restarting apache alone, and not resin, cause the swap to drop back down to normal until the next slow growth starts it all again? That is my pattern. So when I profile resin theres no unusual growth even through the whole server is effectiviley dead until restart. Given that, whos to blame? Could mod_caucho somehow be at fault even though resin itself is ok? I have another identical server running a couple of PHP CMS apps, no resin or java in sight - because of this problem actually. The plan is to move everything over when stable but these have become too critical to play with. Their typical httpd swap use is 25Mb, and its the default httpd.conf. I would dearly love to know what the httpd on the failing server thinks it needs to hold onto 450Mb for, without tweaking there could be 20 of these. There are other non-quercus PHP apps running on that server btw. My solution is to kill child httpd processes at a far quicker rate than you'd normally want, and it works of sorts: IfModule prefork.c StartServers 8 MinSpareServers5 MaxSpareServers 20 ServerLimit 256 MaxClients 256 MaxRequestsPerChild 96 /IfModule IfModule worker.c StartServers 2 MaxClients 150 MinSpareThreads 25 MaxSpareThreads 75 ThreadsPerChild 25 MaxRequestsPerChild 96 /IfModule This problem usually happens on peaks hours. So we upgraded RAM memory to 8Gb with a PAE kernel. Although it doesn't reach the 8G RAM, slow moments gots for frequent. It seems to work worse that way (8Gb + PAE kernel). The question is: Is there everyone having the same issue with Resin (3.1.6)? My doubt if such problem resides either on Resin or on my application. All versions prior to 3.1.6, and currently with 3.2.1 Pro Thanks, Ronan ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.0.238 / Virus Database: 270.11.21/2014 - Release Date: 03/20/09 06:59:00 ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.0.238 / Virus Database: 270.11.22/2015 - Release Date: 03/20/09 19:01:00 ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Perf Issues
From our years using Resin and other containers, even though basically Resin, we have found basically 2 things that containers do not like: .- Swap memory. I know it's stating the obvious, but when your app is running out of memory and the GC starts working like frenzy, the worst it can happen is that the memory it is working with is on disk. That's where the CPU usage starts going up the roof and there's no way to stop the JVM but with a hard kill. .- Restarting the context. Yeah, contexts should not be restarted in production, but sometimes it happens. No matter how good a container is or how careful you are with your application, there are many third-party libraries that were not really well thought-out for a multi-classloader environment and they leave dangling references, which in many cases means the old classloader is not properly cleaned and retains, sort of, a copy of all your WEB-INF/lib/.jar's in memory. Unless you do all on your own, we consider this one a lost battle, because there are so many popular libraries that cause such problems... We don't use mod_caucho, so we don't usually have problems with Apache, so in our case we simply make sure to stay in the safe RAM memory zone and restart the servers once a day just to stay on the safe side as well. There are some nodes we don't restart and can stay up for months, but our environment is many small applications and they don't need to be 24/7, so that works better for us. From your description it looks more probable to be a problem of your application, but even if it is not, as the other people recommended, a memory profiler should do wonders to find out what's going on. Take a memory snapshot when the application is in regular usage and then another when the memory starts growing too much and compare them. I've used YourKit Java Profiler in those situation with the compare-snapshots option and worked quite well. Many other profilers also have that capability. S! D. S'està citant Jeff Schnitzer j...@infohazard.org: It's also worth mentioning that for serious application tuning, there is no substitute for a good profiling tool. In the past I have had great results with JProfiler: http://www.ej-technologies.com/products/jprofiler/overview.html I've even gone so far as to take one node in a production cluster, give it a reduced load profile at the balancer (because profiling slows things down considerably), and run the profiler against it live. You'll be amazed what you discover. Jeff On Sat, Mar 21, 2009 at 7:39 AM, Adam Allgaier allgai...@yahoo.com wrote: I would plug jconsole into your resin instance and watch what's happening to the JVM memory. Could be loose open threads (and large thread size) that grows over time and eats up free memory. Restarting would kill all the threads and free the memory. http://java.sun.com/j2se/1.5.0/docs/guide/management/jconsole.html Adam - Original Message From: Stargazer starga...@blueyonder.co.uk To: General Discussion for the Resin application server resin-interest@caucho.com Sent: Saturday, March 21, 2009 4:39:56 AM Subject: Re: [Resin-interest] Perf Issues Ronan Lucio wrote: Hi, We have had a perf issue. Our servers have 4Gb RAM. It has just Resin and Apache installed. The problem is, when I start Resin, the whole server uses about 2.5 Gb RAM After that memory usage keep growing til it reachs the 4Gb RAM, use swap and so on. After few hours the application start getting slow. Analyzing the server sources, it's using so low CPU, load about 1... I see none overload evidence, except for RAM memory. So I just restart Resin and/or Apache and application gets fast again, but few ours later it will raise the same issue. I have been suffering from exactly those symptoms for years. Do your httpd processes consume all the swap, with top showing some at 450Mb? (default httpd.conf values) I.e does restarting apache alone, and not resin, cause the swap to drop back down to normal until the next slow growth starts it all again? That is my pattern. So when I profile resin theres no unusual growth even through the whole server is effectiviley dead until restart. Given that, whos to blame? Could mod_caucho somehow be at fault even though resin itself is ok? I have another identical server running a couple of PHP CMS apps, no resin or java in sight - because of this problem actually. The plan is to move everything over when stable but these have become too critical to play with. Their typical httpd swap use is 25Mb, and its the default httpd.conf. I would dearly love to know what the httpd on the failing server thinks it needs to hold onto 450Mb for, without tweaking there could be 20 of these. There are other non-quercus PHP apps running on that server btw. My solution is to kill child httpd processes at a far quicker rate than you'd normally want, and it works of
Re: [Resin-interest] Perf Issues
On Mar 21, 2009, at 14:59, Stargazer wrote: Adam Allgaier wrote: I would plug jconsole into your resin instance and watch what's happening to the JVM memory. Could be loose open threads (and large thread size) that grows over time and eats up free memory. Restarting would kill all the threads and free the memory. http://java.sun.com/j2se/1.5.0/docs/guide/management/jconsole.html Adam Thanks, but as I'd hoped I'd made clear Resin itself shows flat memory use, whether using jconsole, jprofiler, /resin-admin or any of the other jvmti tools I've used to try to fix this. The memory consumption occurs outside java. I don't know about Ronan's problem, but yours sounds like non-java- heap memory or native memory is being exhausted. In my experience, the biggest consumers of that are threads and graphic operations. Depending on the system you're running on, your default stack size could be quite large. Have you tried setting your stack size to a much smaller value via -Xss? Unless you're doing a lot of heavy recursion, the chance that you'll need a large stack is fairly remote. On our (very busy) Windows boxes, we use a stack size of 128K (-Xss128K). Keep in mind that on a 32-bit system (not sure what you're running on), you have only 2G of addressable memory per process. So if you define a larg(ish) java heap, you have only 2G - YourHeapSize to use for allocating memory for threads and other non-java operations, such as some graphics operations. Rob - Original Message From: Stargazer starga...@blueyonder.co.uk To: General Discussion for the Resin application server resin-interest@caucho.com Sent: Saturday, March 21, 2009 4:39:56 AM Subject: Re: [Resin-interest] Perf Issues Ronan Lucio wrote: Hi, We have had a perf issue. Our servers have 4Gb RAM. It has just Resin and Apache installed. The problem is, when I start Resin, the whole server uses about 2.5 Gb RAM After that memory usage keep growing til it reachs the 4Gb RAM, use swap and so on. After few hours the application start getting slow. Analyzing the server sources, it's using so low CPU, load about 1... I see none overload evidence, except for RAM memory. So I just restart Resin and/or Apache and application gets fast again, but few ours later it will raise the same issue. I have been suffering from exactly those symptoms for years. Do your httpd processes consume all the swap, with top showing some at 450Mb? (default httpd.conf values) I.e does restarting apache alone, and not resin, cause the swap to drop back down to normal until the next slow growth starts it all again? That is my pattern. So when I profile resin theres no unusual growth even through the whole server is effectiviley dead until restart. Given that, whos to blame? Could mod_caucho somehow be at fault even though resin itself is ok? I have another identical server running a couple of PHP CMS apps, no resin or java in sight - because of this problem actually. The plan is to move everything over when stable but these have become too critical to play with. Their typical httpd swap use is 25Mb, and its the default httpd.conf. I would dearly love to know what the httpd on the failing server thinks it needs to hold onto 450Mb for, without tweaking there could be 20 of these. There are other non-quercus PHP apps running on that server btw. My solution is to kill child httpd processes at a far quicker rate than you'd normally want, and it works of sorts: IfModule prefork.c StartServers 8 MinSpareServers5 MaxSpareServers 20 ServerLimit 256 MaxClients 256 MaxRequestsPerChild 96 /IfModule IfModule worker.c StartServers 2 MaxClients 150 MinSpareThreads 25 MaxSpareThreads 75 ThreadsPerChild 25 MaxRequestsPerChild 96 /IfModule This problem usually happens on peaks hours. So we upgraded RAM memory to 8Gb with a PAE kernel. Although it doesn't reach the 8G RAM, slow moments gots for frequent. It seems to work worse that way (8Gb + PAE kernel). The question is: Is there everyone having the same issue with Resin (3.1.6)? My doubt if such problem resides either on Resin or on my application. All versions prior to 3.1.6, and currently with 3.2.1 Pro Thanks, Ronan ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.0.238 / Virus Database: 270.11.21/2014 - Release Date: 03/20/09 06:59:00 ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
