RE: Reactor Performance was RE: [Reactor For CF] Been out of touch, but trying to get back

Fri, 26 May 2006 13:07:42 -0700

Dan there are basically two flavors of Garbage Collection (GC) in the JVM.  The ongoing GC's (not Full) are going on all the time and are typically miliseconds in duration.  For instance on Doug's server there were 2,569 GC's in a 12.8 hour period, 56 of which were Full GC's.
 
Full GC's in this case were what the JVM decided was necessary.  When I first looked and prior to adding the disable explicit GC's there were 786 Full GC's in just over 8 hours.  Because they were being called explicitly probably, by RMI.
 
Full GC's are at best a necessary evil because literally everything stops so they need to be minimzed to run when needed and should always reclaim some memory, if they do not they should not be running.  And yes your assumption is correct in a Full GC all objects are inspaected, as Sean rightly says in his last excellent post, this is a very dense subject.
 
One last thing I will emphasize here for now.  In all CFMX installations the JVM is as close to a beating heart as CF gets (this is true of all J2EE - JavaEE applications and servers).
 
Kind Regards - Mike Brunt
CTO and Lead Architect - Webapper Services LLC
Tel 562.243.6255
Web Site: http://www.webapper.com
Blog: http://www.webapper.net
Tools: http://www.seefusion.com
 

From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Dan Wilson
Sent: Friday, May 26, 2006 12:02 PM
To: [email protected]
Subject: Re: Reactor Performance was RE: [Reactor For CF] Been out of touch, but trying to get back

So to the layperson, Explicit Garbage collection would infer a complete review of all objects held in memory and a complete review of all currently held references. ( sounds like VACUUM FULL in PostgreSQL, Expensive and Important. )

Thinking out loud, I remember an object caching mechanism in CF. If the frequency of GCs before the change was sub-optimally high, and disabling explicit Garbage collection at the JVM layer improved the memory management situation, was there an impact to any of the default, standard, behaviour of ColdFusion server?

When I re-read the original summation, it sounded like disabling Explicit Garbage collection was something you recommended to all your clients.  Does this pertain to just Reactor users, or any CFserver user?

If this is not the right time, place, or questions for this list, please let me know.

I remain highly interested in this issue and would like to say thanks for the transparent manner in which this is being handled.

Dan Wilson





On 5/26/06, Mike Brunt - Webapper Services, LLC <[EMAIL PROTECTED]> wrote:
As a quick explanation about garbage collection, objects are inspected to see if there are any references to them if there are not they are garbage collected and the memory is freed up.  So if the references to any objects are not being released effectively in an application they will hang around.  That is a vey basic explanation of why objects are not collected and memory is not freed up. 
 
I think there is another thought point here, Jeff Lester mentioned he does NOT have similar issues (correct me if I am wrong Jeff) does Jeff also use the Blog Application?
 
Kind Regards - Mike Brunt
CTO and Lead Architect - Webapper Services LLC
Tel 562.243.6255
 

From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Doug Hughes
Sent: Friday, May 26, 2006 8:32 AM

To: [email protected]
Subject: RE: Reactor Performance was RE: [Reactor For CF] Been out of touch, but trying to get back

Mike – As always:  Awesome.  I've been using a tool called Jprobe to watch what variables CF creates when using my blog.  I can essentially run one use case and see what's around when I'm done.  Based on this, I'm beginning to suspect that this is in the OO queries…. I think that some of the various objects (join, field, order, etc) used by the queries are not released when the query is added back into the resource pool.  I'm not sure, however, how this could create such a leak.  OO queries are where I'll be focusing a lot of my time this weekend with reactor dev.  Maybe I'll track it down?  We'll see.

 

Doug

 

From: [EMAIL PROTECTED] [mailto: [EMAIL PROTECTED]] On Behalf Of Mike Brunt - Webapper Services, LLC
Sent: Friday, May 26, 2006 11:05 AM
To: [email protected]
Subject: RE: Reactor Performance was RE: [Reactor For CF] Been out of touch, but trying to get back

 

Just a quick update on what I found this morning.  Doug last restarted his CFMX instance here 05/25 21:47:47, the first Full GC ocurred 5.4 hours later (which is early morning?) and as with the previous behavior did reclaim a good amount of memory (around 220MB).  However, in the total time the system has been up there have been 56 Full GC's and as before the most recent ones reclaim virtually no memory.

 

As of 10 minutes ago there was 220MB of available memory which is reasonable.  I will be sending Doug some other suggestions as tweaks to the JVM arguments in a little while to see if we can control those Full GC's more effectively.  I will also have some observations about the New and Old memory segments.

 

Kind Regards - Mike Brunt

CTO and Lead Architect - Webapper Services LLC

Tel 562.243.6255

Web Site: http://www.webapper.com

Blog: http://www.webapper.net

Tools: http://www.seefusion.com

 

 

From: [EMAIL PROTECTED] [mailto: [EMAIL PROTECTED]] On Behalf Of Brian Kotek
Sent: Thursday, May 25, 2006 2:44 PM
To: [email protected]
Subject: Re: Reactor Performance was RE: [Reactor For CF] Been out of touch, but trying to get back

Mike I'm afraid I can't add to the technical discussion becuase the innards of the JVM are way over my head. But I am learning a lot from hearing this back and forth and I wanted to thank you for stepping up and helping Doug work through this.

Thanks,

Brian

On 5/25/06, Mike Brunt - Webapper Services, LLC < [EMAIL PROTECTED]> wrote:

I thought I would add a bit to the subject line so that we know this is
about the performance review we are carrying out for Reactor at Doug's
place.  If you recall yesterday we observed the following as far as Full
Garbage Collections (Full GC's) on Doug's instance running Reactor:

"In 8.9 hours of uptime there were 786 Full GC's, that is one every 41
seconds the latest ones are taking 3.8 seconds"

Doug applied a change to the jvm.config file to disable Explicit Full GC's.
Explicit Full GC's are Full GC's called either by an application, a command
or a process such as RMI (Remote Method Invocation). After the change
applied by Doug there is a very noticeable change.

In the 4.08 hours since Doug started the instance with Explicit Full GC's
disabled there have been 19 Full GC's.  We can be fairly certain that these
Full GC's are being activated by the JVM itself.  19 is much better however
there is still evidence of them not being very effective, eventually and we
will watch that.  Here are some details:

The first Full GC occurred after 3.02 hours of uptime and was efficient (as
a comparison before these changes the first Full GC occurred at 0.012
seconds after restart and continued unabated).  This ([Full GC
515767K->256620K) tells us that before the Full GC 516MB of Ram was being
used by the JVM and after that reduced to 257MB.  At 3.6 hours after the
restart the Full GC's although still running are not reclaiming much if any
RAM.  So we have made some progress by disabling Explicit Full GC's and we
recommend that our clients do that unless there is a compelling not to do so
that they are aware of.

Doug, do you have any idea what your traffic level is like on this instance?

Kind Regards - Mike Brunt
CTO and Lead Architect - Webapper Services LLC
Tel 562.243.6255
Web Site: http://www.webapper.com
Blog: http://www.webapper.net
Tools: http://www.seefusion.com


-- Reactor for ColdFusion Mailing List -- [email protected] -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/

-- Reactor for ColdFusion Mailing List -- [email protected] -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/ -- Reactor for ColdFusion Mailing List -- [email protected] -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/
-- Reactor for ColdFusion Mailing List -- [email protected] -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/



--
"Go for it, its later than you think" -- Reactor for ColdFusion Mailing List -- [email protected] -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/ -- Reactor for ColdFusion Mailing List -- [email protected] -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/

Reply via email to