our TC-based webapplication performs well but the java-processes concerned are showing increasing memory usage over time. For tracing we already stripped the app down to the very basic to get a clue. Wasn't successful enough.
Have you looked at the memory over a long time, including several large GCs? I've had to do this in the past, and it's no fun without a profiler.
No, it isn't indeed. I've been checking LINUX' 'top' over a period of time. A mem-hole is apparent. Unfortunately our app makes use of a (self-developed) JNI-Module (C) which makes it even more difficult because the C-code also runs in the Java-context.
Ooh, JNI is always a wild-card. I wrote some JNI code that we were using within a servlet container as well. I tested the hell out of it, especially with threaded use (that's very important). I could only get it to fail when it had trouble writing to files at very high thread rates (I made each thread write to files on a floppy disk, which would run out of space and cause an I/O error).
I'd highly recommend testing your native library separately from the use of JNI and the servlet. Try setting up a test harness for the native code to see if there are easier memory leaks to find. Adding JNI and the VM on top of all that it tough.
Some profiles claim to allow you to watch native code as well as Java code (which sounds really cool!), but again they are expensive. You can use gdb and clever use of tests that are likely to break your code without getting Java involved.
Unfortunately, profiling tools can get expensive. Anyone know of any decent OSS ones out there?
I don't. Anyone have any ideas? Some intern at my old company wrote one from scratch, apparently, so I'm guessing they aren't TOO hard. He used the JVMDP or whatever, which is a debugging protocol that you use to attach to a running VM process.
'JProbe' and 'OptimizeIt!' are really no bargains !
Well, that depends on how much money you are going to spend to find the problem otherwise. How much do you (and your team) cost per hour to your employer? If it's going to take you a week (40 hours * x team members), then how does that compare to the cost of an OptimizeIt license...?
The problem is that the [thread] dump written only contains the hex-adresses of the modules being involved. No possibilty to trace those adressesdown to the functions or even modules involved.
Yeah, that's a problem. You'd have to do a lot of investigation to figure out what's actually being executed.
Even more difficult: the root cause given by this dump is not the function(s)/method(s) causing the memory problems.
Well, once you get an OutOfMemoryException, you pretty much can't trust anything the VM says after that. What you're seeing is just a guide; they might be total lies. :)
There is a tool provided by SUN itself called 'HAT' (HeapAnalyzingTool) http://java.sun.com/people/billf/heap/ but this seems to be only applicable to java programs, not servlets. With Tomcat there were errors. May be in different cases this would help.
Well, Tomcat is a Java program, so you should be able to use it, right? It's just a matter of figuring out how to hook it into your startup process. I hacked through the OptimizeIt instructions and created an ant target to start my Tomcat instance running inside of OptimizeIt. Very sexy.
In our case it was helpful to operate java with the -Xincgc option that causes the GC to collect smaller amounts of mem; but does this more often. It gives a more accurate impression in the current memory status.
What we did is the following: - separate the JNI-module and write a little main()-frame around it to run it alone. - compile this with the 'dmalloc'-library to identify mem not being free'd there. - running the Servlet in one scenario (without JNI) and examine the few methods being touched. - rewrite java-code being in question. - test it on mem-behavior. - iterate with the next scenario etc.
Could be more satisfying, this kind of work, but it should be straight-forward.
Well, that's the approach I'd take without a profiler.
Good luck. Let us know what you find.
-chris
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
