Hello Jan, I really appreciate your prompt reply. let me answer some of the questions below. - Deadlock: I was thinking about this (race condition?) as well. Not sure how exactly I would verify if that theory is correct Stress test was performed on a public instance against STK project pages in a loop: /magnoliaPublic/demo-project.html /magnoliaPublic/demo-project/about.html /magnoliaPublic/demo-project/news-and-events.html /magnoliaPublic/demo-project/multimedia.html /magnoliaPublic/demo-project/service.html /magnoliaPublic/demo-project/service/search-result.html?queryStr=preceded All of them have Gaussian Timeout set to 1000ms offset and 1500ms deviation.
90% responses are in 954ms, that's why I chose 800 threads to handle requests. - Was content cached? I assume yes, because SQL Admin showed no activity during the test - all MySQL connections were idle for a while at the moment. - I had another application deployed and its not responding during that "stall" period. But if I deploy that application to tomcat separately (or I can use tomcat's JSP samples) the Tomcat instance will handle up to 1000 threads with 80% of CPU load. - I have other applications deployed on the same box - they won't be accessible as well - not just magnolia. - I tried with regular GC and other versions (_13, 32-bit) of JDK with no success. Thank you. Regards, Denis On Mon, Oct 5, 2009 at 9:37 AM, Jan Haderka <[email protected]>wrote: > > Interesting tests. Would like to get more details (like detailed timings > for those concurrent users or at least summary stats) once you are done > with it if you don't mind sharing. > > Now to explain your results: > - the DB connections are normal. JR maintains one connection per > workspace no matter what the load, so the amount of connections will not > change. What might change is how busy those connections are (i.e. how > many requests they are executing against the DB) > - the connections being idle and CPU usage being 0% suggests waiting for > some timeout ... it might be a deadlock, but if that is somewhere in > Magnolia itself I would expect it to be rather busy re-trying to either > save or read the content rather then simply sleeping. I might be able to > tell more if I knew the details of the tests - was it on public or on > the author box? Was it just reading the content? Was the content cached > by Magnolia? Were you hitting just single piece of content or > distributing the requests all over the instance (i.e. randomly > retrieving/editing different pieces of content). > - Just because you have 800 threads, doesn't mean you are able to serve > 800 concurrent users. If the processing of the request takes more time > then the average time between requests ... Might be that you simply > filled the queue to the limit and Tomcat stopped queuing further > requests until the queue was emptied. If you have Tomcat management > console installed it would help to see the info from there as well. Also > you can try to have some minimal webapp deployed next to Magnolia and > see if that one will be still responding even when Magnolia is not ... > that should rule out the tomcat involvement. > > JR uses internal in-memory cache for the items, but it is somewhat > limited to about 30MB of content. It would help to know what kind of > content were you retrieving in your test - binary or text? > > Another thing related to threads ... for some ops Magnolia spawn > additional threads and so does the JackRabbit. Not sure if those counts > into the 800 threads limit or not. If so and if you were updating the > content as part of your test, it might be that JR has simply stopped at > some point as it was not able to create indexer threads for re-indexing > the updated content. > > It would also help to see what happened just after the silence period > was over ... what was written in FS, what was the first few DB requests > sent ... > > Also during the test, do you see memory consumption growing or do you > see response times getting longer before the silence? > Have you (or can you) try with different GC collectors (e.g. the default > one)? > > Cheers & good luck with more testing. > Jan > > > > On Mon, 2009-10-05 at 09:15 -0400, Denis Demichev wrote: > > Hello All, > > > > I've been playing with Magnolia for a while trying to asses its > > stability. Eventually I put it under the stress on previous week - 800 > > concurrent users on a single box. > > > > Server: Windows Server 2007(?) SP1 64 bit, Intel Xeon 2 Cores, 4Gb > > RAM > > JDK 1.6.0_16 64-bit version > > Magnolia 4.1 Community Edition (STK Installed, Debug log is disabled) > > + MySQL 5.1.39 > > Tomcat: maxThreads="800" minSpareThreads="25" maxSpareThreads="75" > > Tomcat startup params: > > JAVA_OPTS=%JAVA_OPTS% -Xms2G -Xmx2G -XX:MaxPermSize=1024M -XX: > > +UseConcMarkSweepGC -XX:+CMSIncrementalMode > > -Djava.net.preferIPv4Stack=true -XX:+DoEscapeAnalysis > > Client - J-Meter, 800 Concurrent threads, 5 static links + 1 search > > link, Gaussian timer set for 1000ms+-1500ms > > Response time 954ms, Errors 17% > > Avg. CPU utilization ~85%, RAM Utilization ~2Gb > > > > Now the strange behavior I noticed is following: Magnolia is > > responding to incoming requests normally for about 2-5 minutes (more > > or less) and then just freezes. CPU usage = 0%, RAM is the same (GC is > > obviously not running). No responses at all. > > After removing the stress from Magnolia it starts responding in 30-40 > > seconds. > > 500 Concurrent users will produce the same effect with the only > > exception it might take 50-100 minutes (maybe more) to reproduce. > > Tried to look at MySQL connection stack using remote admin - 8 > > connections all of them are idle. Seems like all the content was > > cached by JCR previously. > > > > Question: What exactly might cause this strange freezing? It looks > > like tomcat has enough maximum threads, connection pool to MySQL is > > not that busy... Could it be some internal cache of jackrabbit? Not > > sure but it looks like JVM is not the rootcause in this case. > > > > Would appreciate your thoughts on this point. > > Thank you in advance! > > > > Regards, > > Denis > > > ---------------------------------------------------------------- > For list details see > http://www.magnolia-cms.com/home/community/mailing-lists.html > To unsubscribe, E-mail to: <[email protected]> > ---------------------------------------------------------------- > > ---------------------------------------------------------------- For list details see http://www.magnolia-cms.com/home/community/mailing-lists.html To unsubscribe, E-mail to: <[email protected]> ----------------------------------------------------------------
