I've got a problem on a production server that has been running since April.
The problem has been ongoing for the last 6 days or so and everything I've
tried doesn't seem to do anything to alleviate the problem.
The server is a Win2003R2 server running a fully patched CF8.01 install. Until
4 days ago, the server was running under JVM 1.6.0_10. Having read through the
change logs for the more recent JVM's there are several memory leaks that have
been fixed in the JVM itself so we upgraded it to JVM 1.6.0_16-b01.
The error I'm seeing in the hotspot JVM error log is:
# A fatal error has been detected by the Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 5242880 bytes for GrET in
C:\BUILD_AREA\jdk6_16\hotspot\src\share\vm\utilities\growableArray.cpp. Out of
swap space?
I hooked up JConsole to the production server and watch the system through
several crashes using the following info
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9500
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
The only thing that climbs is memory usage.
Memory Pool "CMS Old Gen" just keep on climbing until about 1.2Gb is used, the
server then runs at this for an hour or two (sometimes less) before restarting
the jrun service.
Average Par Eden memory usage is @ 35 Mb
Average Par Survivor memory usage is @ 6Mb
Average Code Cache memory usage is @ 10Mb
Average CMS Perm Gen is @ 47Mb
Active threads sits at around 90 - 110
Loaded classes sits at around 9 to 10,000
Nothing spikes, there are no hints as to what is happening other that the
aggressive memory leak causing the Out of Memory errors.
I've also tinkered with the JVM arguments and this is what they're set to at
the moment.
-server -Dsun.io.useCanonCaches=false
-XX:MaxPermSize=384m
-Dcoldfusion.rootDir={application.home}/../
-Dcoldfusion.libPath={application.home}/../lib
-XX:+UseParNewGC
-XX:+CMSParallelRemarkEnabled
-XX:+UseConcMarkSweepGC
-XX:+HeapDumpOnOutOfMemoryError
Unfortunately the HeapDumpOnOutOfMemoryError directive seems not to work :-( as
I wanted to examine the dump with JHat but there are no dump files on the
server and we've had 6 JVM crashes today.
The server is hosting 3 sites all of which are running the same code, two are
very light traffic, the third is a medium heavy traffic site with around 20,000
visitors a day averaging out at around 700 sessions in existence at any given
time.
I've been looking through the server logs, the SVN logs etc to see if anything
significant has changed over the last week and as far as I can tell, nothing
has.
The leak certainly appears to be request driven as at low traffic times between
1am and 6am, the site stay up just fine.
The architecture of the platform running on the server is a predominantly
singleton based object hierarchy using a custom object factory to handle the
generation of objects in the system most of the objects provide functions to
interface with the DB and aren't creating lots of objects themselves.
I've also looked at SeeFusion and FusionReactor trials to see if they can help
me but neither of them seem to be able to tell me how much memory each request
is using etc, they both seem to be too general for my needs.
If anyone has any idea as to what I can do to chase this down I'd very much
appreciate it.
TIA
Paul
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Want to reach the ColdFusion community with something they want? Let them know
on the House of Fusion mailing lists
Archive:
http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:326272
Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
