Thanks a lot for all your information. my comments in line
At 07:38 PM 3/21/2003 +0100, Santiago Gala wrote:
Shan Gopalakrishnan wrote:We observed few more things. I'm posting this to the best interest to mature the framework further than what it is currently today.
Definitely this is not criticism.
Feel free to criticize. It is the only way we have to improve Jetspeed.
--------->>Thanks and i will exchange whatever we learn part of our analysis and testing
We are seeing the Garbage collector (Major GC) getting triggered frequently as I think huge set of objects are getting created.
Some profiling could be handy. AFAIK:
--------->> We will be doing this with OptimizeIt. will share that with you.
Wrappers for security are not recycled (they *must* be inmutable)
PortletSets are created new for every request, due to threading issues.
--> Both kinds of objects should be claimed from "young" space, unless we are missing some "nullification" in permanent objects.
--------->>As we see the minor GC is not clearing up much objects, we still think there is some missing nullification in the permanent objects
as we also notice the memory keeps growing.
Every 30m you will have tomcat sessions expiring. If your testing does not keeps the sessions, you will be filling a lot of memory with fake sessions which will never be reused. I imagine you are using a testing framework that keeps sessions for a number of hits per "user".
--------->>we ensured this . the session id is passed through a cookie and this is reused between requests from the same user
A lot of other objects could come from either Velocity ASTs, Torque peers or other places. Even from your portlets ;-)--------->> This really makes us think twice whether we can use it for production. I would appreciate if profiling/tuning can be treated much frequently to develop a healthy framework. Hopefully some of our results /analysis will help the development team.
About Spring 2001 some profiling and tuning was done, by the IBM team, David, Raphael and myself. I don't know of other such effort since then. This led to rundata and some other objects pooling in turbine, and some changes of String concatenation to StringBuffer.append(). We also discovered a lot of duplicated initializations and similar stuff. It is always funny to see how big bugs can hide in code.
But the behaviour you are seeing looks more like the bug I pointed below or a bug in the hotspot compiler. Could you switch temporarily to jdk 1.3.1_07 or 1.4.0_XX and see if the same happens? (According to reports in cocoon-dev, 1.4.0 does not have the StringBuffer bug)
--------->> I tried changing this. unfortunately changing it to 1.4.0.xx resulted in a slower performance which is substantial and noticeable. i couldn't believe. looked it and tried more than once to ensure it happens. and yes it happens.
----->>>Good point. I used to wonder how Jetspeed (team) will handle for the inherent problems of Turbine ??This could be inherited from Turbine implementation. We are now playing with the various options in the Garbage Collector
part of the JDK 1.4.1, which is running the GC parallel, setting high heap size, giving some % for young generation Vs old generation etc.
Most likely to my knowledge the minor GC looks only the younger generation area as we didn't see any benefits in increase the
% of younger generation. Apparently the spike still remains and gives poor response time just that the symptom happens way early
or later during the tests depending on the heap size and GC configuration.
If the problem is that some "bunches" of objects get into "old" space because they are pointed by persistent objects, then a juditious ammount of "= null" on recycling can help a lot. I've just checked DefaultJetspeedRunData and it is disposed properly (unless something is broken down in Turbine and it is not really disposed).
Thanks a lot for all your information.
- Shan
