One of our CF servers keeps needing to have the CF service rebooted on it
in order to work and continue serving our sites 1-2x / week. Fortunately
we do have a web service on one of the other box that monitors this machine
to let us know when it starts bugging out again.
The error I am getting on the server has to do with an out of memory perm
gen space issue, something that there is tons of stuff online about but
nothing that I've been able to succinctly tell is a good idea to look
toward resolving the issue. Here are the errors we get before the service
fails.
javax.servlet.ServletException: ROOT CAUSE: java.lang.OutOfMemoryError:
PermGen space
12/06 16:24:46 Error [jrpp-3954] - PermGen space The specific sequence of
files included or processed is:
F:\...................\case-studies\ENDO-enduring-hot-seat-discussion\index.cfm''
12/06 16:24:46 error ROOT CAUSE: java.lang.OutOfMemoryError: PermGen space
javax.servlet.ServletException: ROOT CAUSE: java.lang.OutOfMemoryError:
PermGen space
Server: Windows Server 2008 R2 64bit
ColdFusion 9,0,1,274733 Standard
Java Version: 1.6.0_24
RAM 8GB JVM Arguments
-server -Dsun.io.useCanonCaches=false -XX:MaxPermSize=192m
-XX:+UseParallelGC -Xbatch -Dcoldfusion.rootDir={application.home}/../
-Dcoldfusion.libPath={application.home}/../lib
Despite my attendance of multiple Charlie lectures and/or because of
such attendance I certainly do not consider myself an expert in the memory
underpinnings of CF. What I do believe is happening through some research
and late night testing is that essentially the server stores information
(in think in cfclasses) that is essentially compiled java code. It does
this to speed the delivery of data generated by CF in a caching like
mechanism.
There is a checkmark in the settings that says "Save Class Files" which I
suspect would solve the problem if indeed this area is getting overflowed
and not properly Garbage Collected (if GC even runs in PermGen). However,
unchecking that is not recommended for a production machine, and it would
obviously slow down all of the sites. Similarly I've seen posts that say
to just increase the MaxPermSize, but I find many posts after those saying
that is not solve and just delaying the problem. I'd rather solve the
problem.
Additionally, when observing the websites when it is acting up we get a lot
of totally blank pages along with 200 OK http response codes. I have also
gotten partial pages at times, but that all leads me to some sort of
partial template loading until the permgen runs out of memory.
Finally, I don't know that it is a 100% deal, but we have launched a new
application on this server that makes use of the new ORM Hibernate
functionality. This happened right around the same time we started
observing these issues. This was my first shot at ORM within CF and I know
I need to go back and re-look at how I have the lazy-loading set up. I
have a feeling that it can definitely be tuned better and that it's
possible this type of code may make heavy use of the permgen space with
dynamically loading data in the way that it works. This heavy banging in
the permgen along with an untuned JVM may be the combo that is causing our
problems, or maybe I'm way off. Figured I'd throw it out there to group to
see what thoughts and solutions might be out there beyond my teams
continued googling of words like "permgen", "coldfusion", "hibernate",
"orm" and "arehart" :)
- Cheyenne Throckmorton
P.S. Looking forward to seeing folks at Whirlyball tomorrow night.
--
Cheyenne Throckmorton - Atlanta, GA
Blog : www.CheyenneJack.com
Twitter : @cheyennejack
Founder : www.AtlantaUserGroups.com
www.TheTallStreetJournal.com
www.MohawksRock.com
