Well in the end it was a rookie mistake... DOH!
One of the most heavily used queries in the system was set up to be cached
for 10 minutes at a time. The only problem was that one of the parameters I
was passing into the query was a date/time of Now().
Changing the parameter being passed in from Now() to
CreateDate(Year(Now()),Month(Now()),Day(Now())) has solved the memory leak
completely.
Phew!
Paul
> -----Original Message-----
> From: Paul Vernon
> Sent: Monday, September 14, 2009 6:55 PM
> To: cf-talk
> Subject: Need help pinpointing an aggressive memory leak
>
>
> I've got a problem on a production server that has been running since
> April. The problem has been ongoing for the last 6 days or so and
> everything I've tried doesn't seem to do anything to alleviate the
> problem.
>
> The server is a Win2003R2 server running a fully patched CF8.01
> install. Until 4 days ago, the server was running under JVM 1.6.0_10.
> Having read through the change logs for the more recent JVM's there are
> several memory leaks that have been fixed in the JVM itself so we
> upgraded it to JVM 1.6.0_16-b01.
>
> The error I'm seeing in the hotspot JVM error log is:
>
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # java.lang.OutOfMemoryError: requested 5242880 bytes for GrET in
> C:\BUILD_AREA\jdk6_16\hotspot\src\share\vm\utilities\growableArray.cpp.
> Out of swap space?
>
> I hooked up JConsole to the production server and watch the system
> through several crashes using the following info
>
> -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.port=9500
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false
>
> The only thing that climbs is memory usage.
>
> Memory Pool "CMS Old Gen" just keep on climbing until about 1.2Gb is
> used, the server then runs at this for an hour or two (sometimes less)
> before restarting the jrun service.
>
> Average Par Eden memory usage is @ 35 Mb
> Average Par Survivor memory usage is @ 6Mb
> Average Code Cache memory usage is @ 10Mb
> Average CMS Perm Gen is @ 47Mb
>
> Active threads sits at around 90 - 110
> Loaded classes sits at around 9 to 10,000
>
> Nothing spikes, there are no hints as to what is happening other that
> the aggressive memory leak causing the Out of Memory errors.
>
> I've also tinkered with the JVM arguments and this is what they're set
> to at the moment.
>
> -server -Dsun.io.useCanonCaches=false
> -XX:MaxPermSize=384m
> -Dcoldfusion.rootDir={application.home}/../
> -Dcoldfusion.libPath={application.home}/../lib
> -XX:+UseParNewGC
> -XX:+CMSParallelRemarkEnabled
> -XX:+UseConcMarkSweepGC
> -XX:+HeapDumpOnOutOfMemoryError
>
> Unfortunately the HeapDumpOnOutOfMemoryError directive seems not to
> work :-( as I wanted to examine the dump with JHat but there are no
> dump files on the server and we've had 6 JVM crashes today.
>
> The server is hosting 3 sites all of which are running the same code,
> two are very light traffic, the third is a medium heavy traffic site
> with around 20,000 visitors a day averaging out at around 700 sessions
> in existence at any given time.
>
> I've been looking through the server logs, the SVN logs etc to see if
> anything significant has changed over the last week and as far as I can
> tell, nothing has.
>
> The leak certainly appears to be request driven as at low traffic times
> between 1am and 6am, the site stay up just fine.
>
> The architecture of the platform running on the server is a
> predominantly singleton based object hierarchy using a custom object
> factory to handle the generation of objects in the system most of the
> objects provide functions to interface with the DB and aren't creating
> lots of objects themselves.
>
> I've also looked at SeeFusion and FusionReactor trials to see if they
> can help me but neither of them seem to be able to tell me how much
> memory each request is using etc, they both seem to be too general for
> my needs.
>
> If anyone has any idea as to what I can do to chase this down I'd very
> much appreciate it.
>
> TIA
>
> Paul
>
>
>
>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Want to reach the ColdFusion community with something they want? Let them know
on the House of Fusion mailing lists
Archive:
http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:326285
Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4
