Oh, I'm not blaming the JVM. I'm sure it is working as designed. But, it sounds like you are sure that GC is running. I'll look into the hibernate session and Quercus Env objects and see what the heap dump shows relative to them.

Andrew

Scott Ferguson wrote:

On Apr 8, 2008, at 7:55 AM, Andrew Fritz wrote:

Well, based on previous advice we are inspection both the profile and heap dump. Both point to a problem with one of our business objects (referenced previous: ShallowSongBO) so I implemented a simple instance counter on that object.

There is a static variable. It is incremented in the constructor and decremented by finalize. Both are synchronized on the class "ShallowSongBO" so the counter should be thread safe. I simply print the count each time an instance is created of finalized.

On my box and on production to start with the instance count is well behaved. It goes up for a while and then at some point it will fall off. On production, eventually it begins to ONLY increase and never go down. I suspect that other objects have the same problem, it is just that we load many many songs compared to other classes. In fact the heap dump shows that the object we load the second most is also hanging around.

Good.  That's the first step in tracking down the problem.

It looks to me like garbage collection just stops at some point.

Probably not.  At this point in debugging, it's better to assume the JVM is running perfectly, no matter how temping it might be to blame it. :)

Most likely, there's some reference somewhere to your ShallowSongBO that shouldn't exist.  This is where a more sophisticated profiler would help tremendously, although it's still possible to track down.

Some things to check: 
  1) is the number of Hibernate sessions expected, or is it larger than normal?  If you see 1000s of sessions, then something is holding the sessions beyond the end of the request, which are holding your object.
  2) is the number of Quercus Env objects expected?  Similarly, if something's holding the Env beyond the request, that might be either holding your object directly, or holding the Hibernate sessions (if this is happening, it might be a Quercus bug.)
  3) etc, look for other containers, or things that should only have 1 object per request.  If the number of those objects is significantly bigger than the number of requests, that's the place to start.

You might want to use <jvm-arg>-Xrunhprof:heap=sites</jvm-arg>.  It'll give you similar information as Resin's heap dump, but might be more useful (although it's a bit more complicated to debug.)

-- Scott



As an experiment I've implemented a url  that allows me to execute System.gc(); This is probably a placebo and should be needed but I'll know today or tonight if requesting garbage collection has any effect. I have to wait for the problem to manifest itself again some time later today or tonight to know if this will have any effect.

Andrew

Bill Au wrote:
I would take some threads dump during heavy load to see what is going on.
Jconsole with the JTop plug-in can show you the threads that are using the
most CPU.

Bill

On Tue, Apr 8, 2008 at 12:06 AM, Knut Forkalsrud <[EMAIL PROTECTED]> wrote:

On Apr 7, 2008, at 9:18 AM, Sandeep Ghael wrote:
            <jvm-arg>-Xdebug</jvm-arg>

In my experience the debug switch sometimes causes the JVM to behave erratically under heavy load.  I would get rid of it and try again.

-Knut



_______________________________________________
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest



_______________________________________________ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
_______________________________________________
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest


_______________________________________________ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest




_______________________________________________
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest

Reply via email to