Hi David, Memory dump was insightful, thanks for sharing. Seems that Jetty is keeping sessions open for some reason; I see almost two gigs of org.mortbay.jetty.servlety.HashSessionManager$Session classes open, with no link back to our code. A quick question on #jetty revealed this bug, which looks like the one I'm seeing in this dump file:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=354297 In short, jetty defaults to session lengths of -1, which is never. Resolution is to set session lengths to something like 30 mins. (note: I don't really know how we're using jetty at all, it's a black box to me, so this next bit is just from browsing the code to get a better grip) I think we use pax web to realise the http service, which uses jetty. I think this means that you can configure jetty by changing the last line of /felix/conf/bin/start_matterhorn.sh to read: java $DEBUG_OPTS $GRAPHICS_OPTS $MAVEN_ARG $JAVA_OPTS -Dorg.ops4j.pax.web.config.file=/opt/matterhorn/felix/conf/felix.xml "$FELIX_FILEINSTALL_OPTS" "$PAX_CONFMAN_OPTS" $PAX_LOGGING_OPTS $ECLIPSELINK_LOGGING_OPTS "$MATTERH ORN_LOGGING_OPTS" "$UTIL_LOGGING_OPTS" -jar "$FELIX_HOME/bin/felix.jar" "$FELIX_CACHE" Then, create a file in /opt/matterhorn/felix/conf/felix.xml with some magic configuration lines. The magic configuration lines is where I'm not quite at I'm afraid. Anyone want to share if a) this is the best way to configure our web server? b) does anyone know the magic juice to set the felix to time out sessions? Regards, Chris On Mon, 03 Oct 2011 16:37:50 +0200 David Horwitz <[email protected]> wrote: > Hi All, > > Dropped the file here: > http://source.cet.uct.ac.za/opencastproject/memDump/HeapDump.hprof > > Regards > > D > > On 10/03/2011 03:55 PM, Micah Sutton wrote: > > I would also like to see the memory dump! > > > > Micah > > > > On Oct 3, 2011, at 5:07 AM, [email protected] wrote: > > > >> Hi david, > >> > >> I'll bite on this if you can make the dump available... > >> > >> chris > >> > >> > >> Quoting David Horwitz<[email protected]>: > >> > >>> Hi All > >>> > >>> As mentioned in a thread last week I took a heap dump from our > >>> production Matterhorn instance that had become unresponsive (100% > >>> CPU but low load) and tried to run it through the eclipse memory > >>> stand alone analyser. On my 32 bit desktop machine I can't > >>> allocate enough heap to the analyser to be able to parse the > >>> file. As this tool could successfully parse dumps from Sakai > >>> instances with a very bad memory leak I would consider this > >>> ominous. > >>> > >>> I would gladly share the dump (3.5Gb) with others > >>> > >>> Regards > >>> > >>> David > >>> _______________________________________________ > >>> Matterhorn mailing list > >>> [email protected] > >>> http://lists.opencastproject.org/mailman/listinfo/matterhorn > >>> > >>> > >>> To unsubscribe please email > >>> [email protected] > >>> _______________________________________________ > >>> > >> > >> > >> _______________________________________________ > >> Matterhorn mailing list > >> [email protected] > >> http://lists.opencastproject.org/mailman/listinfo/matterhorn > >> > >> > >> To unsubscribe please email > >> [email protected] > >> _______________________________________________ > > _______________________________________________ > > Matterhorn mailing list > > [email protected] > > http://lists.opencastproject.org/mailman/listinfo/matterhorn > > > > > > To unsubscribe please email > > [email protected] > > _______________________________________________ > _______________________________________________ > Matterhorn mailing list > [email protected] > http://lists.opencastproject.org/mailman/listinfo/matterhorn > > > To unsubscribe please email > [email protected] > _______________________________________________ -- Christopher Brooks, BSc, MSc ARIES Laboratory, University of Saskatchewan Web: http://www.cs.usask.ca/~cab938 Phone: 1.306.966.1442 Mail: Advanced Research in Intelligent Educational Systems Laboratory Department of Computer Science University of Saskatchewan 176 Thorvaldson Building 110 Science Place Saskatoon, SK S7N 5C9 _______________________________________________ Matterhorn mailing list [email protected] http://lists.opencastproject.org/mailman/listinfo/matterhorn To unsubscribe please email [email protected] _______________________________________________
