Richard (and Ben),
Appears like a lot of good work has gone into trying to resolve / fix
this....
Thanks for the info. I hope my comments in-line can assist....
--Nikolaos
Richard Hauswald wrote:
I spent some time investigating the problem - results:
Can you provide a little more information on the test system... OS and
version, JDK version, web container and version,...
Can you provide the complete list of JVM options you use? I'm curious
to see your heap settings, gc algorithm and options, gc ratios, thread
stack size, etc... If you are using tomcat then simply have it echo the
command it uses to launch the JVM....
Also have you tried enabling gc logging so that you can see when the
minor and major gc's occur i.e. logging how the GC behaves as you run
your test is typically enlightening, i.e. is it new generation or old
generation objects that are growing, etc...
-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
Also have you tried running with jconsole (its an Agent that gets added
a JVM param on web container startup) to better analyze what objects are
being retained and by what?
- The problem is not related to thread locals.
OK
- The memory costs using layout tag with ~2MB sized pages is
significant. Without layout tag the test (20 threads parallel same
page) took around 35MB, with layout tag around 190MB
That is significant. It would be interesting to repeat the test with
small pages e.g. 50K or whatever to see if the issue still appears in
the same proportion.
ASIDE:: JSPs use by default an 8K buffer IIRC to hold the page contents
before returning it.... although upping this buffer dramatically is not
recommended, as your pages are huge using a 32K or 64K buffer on a page
will mean less flushes so it might improve your 2MB page regardless of
this issue... of course it increases the delay in getting the data to
the web browser... but it might be worth trying...
- After this test has passed the memory usage did not drop below 100MB
using stripes layout, without layout tag it did not drop under 10MB
This would appear to rule out the container / buffers not releasing
memory... i.e. all things equal and merely interpreting your results
only... this would point to something in the Stripes layout tag is not
releasing memory...
(after requesting gc multiple times)
I assume this is by doing a "System.gc()"? Keep in mind that this can
be totally ignored by the GC i.e. that the call is a "hint" or "can you
please collect the garbage when you get a chance" upon which the GC can
totally ignore the request, act on it later, etc... i.e. don't rely or
expect that this will make any difference....
- The app crashed with heap space exceptions when running with 100MB
max heap and stripes layout tag. It did not without stripes layout
tag.
Well that is good in a way that you can get it to fall over. There are
several things here:
1) Out of memory comes in various flavours (i.e. it could be related to
java heap, perm gen which for most jvm's is OUTSIDE the java heap,
etc...) can you provide a copy of the out of memory stack trace?
2) If the jvm crashed with out of memory exception then there should be
a hs_err_pid<pid>.log or similar file in your working directory. This
file will among other things will show what the state of the GC was at
before running out of memory. Can you please provide this file from
your crashed run.
3) I think you mentioned that you had a heap dump if so I assume you
used: -XX:-HeapDumpOnOutOfMemoryError If so and you have a heap dump
what are the top 50 or so objects by total size, total count, etc...?
It is important to state that the app crashed after 40 requests.
So the memory got filled up with each request.
If that truly was true then the long running test would consume tons of
memory and eventually crash too... but it didn't which doesn't mean
memory isn't increasing with each request but instead it means to me
that memory increases to a certain threshold... and the 40 requests just
got it to that point...
- I ran a test using stripes layout tag with 2 threads running request
after request for over 25 hours and the app is not leaking. The memory
usage did not drop below 100MB (after requesting gc multiple times).
Conclusion
There is no memory leak is stripes layout tag (currently ;-P )
I guess that depends on how you define a memory leak A memory leak does
not have to leak like an oil well IMO. There is a difference of 90MB of
data that is being retained based on your results and by what appears to
be the Stripes layout tag... .
I wish Java came up with a different term and didn't borrow memory leak
from say C/C++... because in Java a memory leak in my opinion is simply
retaining references to objects that are no longer being used... just
because this doesn't grow in size doesn't mean it isn't a memory leak...
it could merely mean that the components that are holding the data
beyond its useful life are simply swapping in new equally sized chunks
of data that come in on the next request or whatever....
So to say that the Stripes framework layout tag is not leaking memory is
not something I would agree with... if it doesn't release the increased
memory it uses and is doing nothing with what it is retaining then it is
a memory leak AFAIK.
There are still open questions:
- Why is the memory not released by gc after a request is finished
Remember GC and request processing are unrelated. When an object has
zero references it is a candidate for GC. Just because the request has
completed does not mean the GC will kick in and clean it up. GC depends
highly on the algorithm used and other inner workings that unless your a
VM engineer we need not worry or concern ourselves with i.e. IMO best to
consider GC a black box that will do its thing when it wants to based on
how you told it you wanted it to do it but with no guarantees on when /
what gets done.
The only thing we should concern ourselves is with is WHAT is holding on
to that data that is not getting GC'd (and what that data is).
- Why is the memory not released by gc after a request is finished is
increasing until reaching some magic random numbers around 100MB
Well it appears that... that Stripes layout tag is retaining something
like data related to the last request OR X requests or whatever....
What happens if you run this ONE test while having gc logging enabled...
1) Do your 2MB Stripes layout requests steady for say 1 hour....
2) Then without restart do a 1MB Stripes layout requests steady for say
1 hour....
What is your memory at the end of 1) and at the end of 2)? Is it still
100MB or is it 50MB or something else?
- Can this behavior be reproduced in other containers eg. Jetty?
If you provide the 2 test cases I can try it on glassfish v3 final...
I think it is related to the tomcat buffer, the limit buffer = true
setting and that stripes layout tag is somehow referencing/using this
buffer.
If the Stripes layout tag is somehow still referencing or using the
tomcat buffer then it is a Stripes memory leak. No?
Some other things to help with debugging this:
- http://happygiraffe.net/blog/2009/04/02/heap-dump-analysis/
- http://java.sun.com/j2se/1.5/pdf/jdk50_ts_guide.pdf
--> Great read but also try something called: jconsole and jmap
-histo
-
http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp#DebuggingOptions
I would personally like to see this issue solved and will help as much
as I can.
Thanks for the work you have done to date!
--Nikolaos
Many thanks to Ben,
Richard
On Wed, May 12, 2010 at 10:27 AM, M.C.S. <m...@syn-online.de> wrote:
Hi Ben,
could you please check in the patch into the 1.5.x and, if needed, the
1.4.x branches? Currently we are not using the version on trunk, but
also would like to benefit from the fix asap.
Marcus
Am 11.05.2010 17:10, schrieb Ben Gunter:
I did a little poking around and found that sometime in 2008,
LayoutComponentTag was changed such that it does not null its
reference to its BodyContent. Apparently it had to do with a bug in
Resin, but I suspect it might have led to a memory leak. The JSP spec
is clear that setBodyContent is to be called after each call to
doStartTag() that returns EVAL_BODY_BUFFERED so there's no reason to
retain a reference to the BodyContent. I've reverted that change in
the trunk. Can you check out and build the trunk and see if that helps
the memory issues?
-Ben
------------------------------------------------------------------------------
_______________________________________________
Stripes-users mailing list
Stripes-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/stripes-users
------------------------------------------------------------------------------
_______________________________________________
Stripes-users mailing list
Stripes-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/stripes-users
