From our years using Resin and other containers, even though
basically Resin, we have found basically 2 things that containers do
.- Swap memory. I know it's stating the obvious, but when your app is
running out of memory and the GC starts working like frenzy, the worst
it can happen is that the memory it is working with is on disk. That's
where the CPU usage starts going up the roof and there's no way to
stop the JVM but with a hard kill.
.- Restarting the context. Yeah, contexts should not be restarted in
production, but sometimes it happens. No matter how good a container
is or how careful you are with your application, there are many
third-party libraries that were not really well thought-out for a
multi-classloader environment and they leave dangling references,
which in many cases means the old classloader is not properly cleaned
and retains, sort of, a copy of all your WEB-INF/lib/.jar's in memory.
Unless you do all on your own, we consider this one a lost battle,
because there are so many popular libraries that cause such problems...
We don't use mod_caucho, so we don't usually have problems with
Apache, so in our case we simply make sure to stay in the "safe" RAM
memory zone and restart the servers once a day just to stay on the
safe side as well. There are some nodes we don't restart and can stay
up for months, but our environment is many small applications and they
don't need to be 24/7, so that works better for us.
From your description it looks more probable to be a problem of your
application, but even if it is not, as the other people recommended, a
memory profiler should do wonders to find out what's going on. Take a
memory snapshot when the application is in regular usage and then
another when the memory starts growing too much and compare them. I've
used YourKit Java Profiler in those situation with the
compare-snapshots option and worked quite well. Many other profilers
also have that capability.
S'està citant Jeff Schnitzer <j...@infohazard.org>:
> It's also worth mentioning that for serious application tuning, there
> is no substitute for a good profiling tool. In the past I have had
> great results with JProfiler:
> I've even gone so far as to take one node in a production cluster,
> give it a reduced load profile at the balancer (because profiling
> slows things down considerably), and run the profiler against it live.
> You'll be amazed what you discover.
> On Sat, Mar 21, 2009 at 7:39 AM, Adam Allgaier <allgai...@yahoo.com> wrote:
>> I would plug jconsole into your resin instance and watch what's
>> happening to the JVM memory. Could be loose open threads (and
>> large thread size) that grows over time and eats up free memory.
>> Restarting would kill all the threads and free the memory.
>> ----- Original Message ----
>> From: Stargazer <starga...@blueyonder.co.uk>
>> To: General Discussion for the Resin application server
>> Sent: Saturday, March 21, 2009 4:39:56 AM
>> Subject: Re: [Resin-interest] Perf Issues
>> Ronan Lucio wrote:
>>> We have had a perf issue.
>>> Our servers have 4Gb RAM. It has just Resin and Apache installed.
>>> The problem is, when I start Resin, the whole server uses about 2.5 Gb RAM
>>> After that memory usage keep growing til it reachs the 4Gb RAM, use swap
>>> and so on.
>>> After few hours the application start getting slow.
>>> Analyzing the server sources, it's using so low CPU, load about 1... I
>>> see none overload evidence, except for RAM memory.
>>> So I just restart Resin and/or Apache and application gets fast again,
>>> but few ours later it will raise the same issue.
>> I have been suffering from exactly those symptoms for years.
>> Do your httpd processes consume all the swap, with top showing some at
>> 450Mb? (default httpd.conf values) I.e does restarting apache alone,
>> and not resin, cause the swap to drop back down to normal until the next
>> slow growth starts it all again?
>> That is my pattern. So when I profile resin theres no unusual growth
>> even through the whole server is effectiviley dead until restart. Given
>> that, whos to blame? Could mod_caucho somehow be at fault even though
>> resin itself is ok?
>> I have another identical server running a couple of PHP CMS apps, no
>> resin or java in sight - because of this problem actually. The plan is
>> to move everything over when stable but these have become too critical
>> to play with. Their typical httpd swap use is 25Mb, and its the default
>> httpd.conf. I would dearly love to know what the httpd on the failing
>> server thinks it needs to hold onto 450Mb for, without tweaking there
>> could be 20 of these. There are other non-quercus PHP apps running on
>> that server btw.
>> My "solution" is to kill child httpd processes at a far quicker rate
>> than you'd normally want, and it works of sorts:
>> <IfModule prefork.c>
>> StartServers 8
>> MinSpareServers 5
>> MaxSpareServers 20
>> ServerLimit 256
>> MaxClients 256
>> MaxRequestsPerChild 96
>> <IfModule worker.c>
>> StartServers 2
>> MaxClients 150
>> MinSpareThreads 25
>> MaxSpareThreads 75
>> ThreadsPerChild 25
>> MaxRequestsPerChild 96
>>> This problem usually happens on peaks hours.
>>> So we upgraded RAM memory to 8Gb with a PAE kernel.
>>> Although it doesn't reach the 8G RAM, slow moments gots for frequent. It
>>> seems to work worse that way (8Gb + PAE kernel).
>>> The question is:
>>> Is there everyone having the same issue with Resin (3.1.6)?
>>> My doubt if such problem resides either on Resin or on my application.
>> All versions prior to 3.1.6, and currently with 3.2.1 Pro
resin-interest mailing list