I guess an immediate answer would be to create a new velocity engine per request but macros.vm would then be parsed per request which would be a hit to performance. It would be nice to see macros (and velocity code) parsed and cached with the document which I think would improve performance over the current setup but the velocity engine code is resistant to extension so the job means lots or rewriting/copy-paste.
I can take a look at this after I have something to show for invitation, right now I have to put my full attention there so it progresses at a medium speed crawl ;) Caleb Ludovic Dubost wrote: > > Indeed very interesting. Here are the results for myxwiki.org > > 2 engines (default and one for Velocity engine: skins/albatross/macros.vm) > 703 and 5 namespaces in each engine > 3477 macros total (I added a total counter) > > I found by looking at it a bad use case. On my blog (which runs on > myxwiki.org) I have a panel which is in syntax 1.0 calling in > includeInContext which has a macro definition which it uses iteself. > It seems this generates one namespace per page using the panel. > > This accounts for 1200 of the macros in the 3477 just for my blog > (ludovic.org) which has a custom panel. > > But this case is also present on other blogs running on myxwiki.org > (vincent's for instance) which use XWiki's Blog. > > Here is an output: > ludovic-3:myxwikiorg ludovic$ grep "massol:" velo.txt > Namespace: massol:Blog.MovingBlog (42 entries) > Namespace: massol:Blog.SVNSearch (42 entries) > Namespace: massol:Blog.BlogScripts (1 entries) > Namespace: massol:Blog.Migration (42 entries) > Namespace: massol:Blog.WikiVsCmsUsi2009 (42 entries) > Namespace: massol:Blog.Open Source (42 entries) > Namespace: massol:Blog.XWikiStats200901 (42 entries) > Namespace: massol:XWiki.eirbjo (42 entries) > Namespace: massol:Main.WebHome (42 entries) > Namespace: massol:Blog.BlogTemplate (42 entries) > Namespace: massol:Blog.ParisJUGSecondAnniversary (42 entries) > Namespace: massol:XWiki.DefaultSkin (42 entries) > Namespace: massol:Blog.OSSGTPSoireeOpenSource (42 entries) > Namespace: massol:Blog.WebHome (42 entries) > Namespace: massol:Blog.XWikiSASAndOpenSource (42 entries) > Namespace: massol:Blog.Personal Events (42 entries) > Namespace: massol:Blog.Archive (45 entries) > Namespace: massol:Blog.BlogRss (42 entries) > Namespace: massol:Blog.XWikiAtJazoon2009 (42 entries) > > 42 blog macros are loaded in their own namespace for each page in the > blog that is called. > > I believe we have something big here. The namespace system generates a > cache for macros which is not under control. While it might be normal > that there is a different namespace for each page holding macros, there > need to keep this under control. > This namespace should have a limited livespan which should be about the > same as the life of the page in the XWiki Cache. If we could find a way > to remove this namespace from the velocity engine when the document gets > out of the cache this would keep velocity's memory usage under control > > Right now it looks like the more pages, the more the velocity engine > macro namespace system will grow indefinitively. > > Ludovic > > Le 15/05/10 19:53, Caleb James DeLisle a écrit : >> 170Mb of macros sounds like a leak. >> >> I spent some time trying to figure out how to write the invitation >> app more defensively and started looking for the cause of >> http://jira.xwiki.org/jira/browse/XWIKI-4934 >> I wrote up a test which dumps all of the names of the working >> velocity engines and the namespaces in each engine, the macros in >> each namespace and the memory location of the name of the macro >> (using identityHashCode which gives the location in most java >> implementations). The macro names appear to be interned so counting >> references to the memory location in the heap analysis tool should >> show any duplicate macro leaks. >> >> {{html clean=false}}<style>p {font-family:monospace;}</style>{{/html}} >> >> {{groovy}} >> vf = >> com.xpn.xwiki.web.Utils.getComponent(org.xwiki.velocity.VelocityFactory.class); >> println(vf.velocityEngines.size()); >> for (String engineName : vf.velocityEngines.keySet()) { >> println("Velocity engine: " + engineName); >> vmm = >> vf.velocityEngines.get(engineName).engine.ri.vmFactory.vmManager; >> println(" Velocity Macro Namespaces:") >> for (String x : vmm.namespaceHash.keySet()) { >> if (vmm.globalNamespace.equals(vmm.namespaceHash.get(x))) { >> println(" Global Macro Namespace: " + x + " (" + >> vmm.namespaceHash.get(x).size() + " entries)"); >> } else { >> println(" Namespace: " + x + " (" + >> vmm.namespaceHash.get(x).size() + " entries)"); >> } >> for (String y : vmm.namespaceHash.get(x).keySet()) { >> print(" #" + y); >> for (int i = y.length(); i < 30; i++) { >> print(" "); >> } >> println("java.lang.String@" + >> Integer.toString(System.identityHashCode(y), 16)); >> } >> println("\n\n"); >> } >> } >> {{/groovy}} >> >> >> Not much to see on my local installation except that all syntax 2.0 >> velocity macros are in the same namespace. Might be more interesting >> running on a large system. >> >> >> Caleb >> >> >> Ludovic Dubost wrote: >> >>> Hi developers, >>> >>> It would be great to see some developers be interested in this thread. >>> We need to better understand memory usage by XWiki in order to achieve >>> higher throughput with controled memory usage. >>> >>> I've found an additional interesting tools to use, the Eclipse Memory >>> Analyzer which works with a dump retrived using the command "jmap >>> -heap:format=b <processid>" >>> (This is practical because we can get such a dump on any running VM, and >>> we can even configure the VM to give such a dump when hitting OutOfMemory) >>> >>> It gives some interesting results. I retrieved a dump from myxwiki.org >>> and analyzed it a bit >>> >>> http://www.zapnews.tv/xwiki/bin/download/Admin/MemoryUsage/myxwikiorgmem.png >>> >>> >>> As the following image shows it, we have a significant amount of memory >>> in the velocity package in a structure meant to store all velocity macros. >>> It's 170Mb which represents 37% of the heap and which is more than the >>> document size. >>> >>> I suspect that if we are able to achieve this amount we can achieve more >>> and reach OutOfMemory with only this module. >>> There is a chance that it is linked to MultiWiki usage where macros are >>> kept in a different context for each wiki, but it could be also >>> something growing regularly every time a macro is found in a page. >>> Even if it is growing by number of wikis, it is still potentially a >>> scalability issue. I already analyzed memory a long time ago and did not >>> see Velocity as storing a lot of information. This could be linked to >>> the new implementation in component mode. >>> >>> Velocity+JBoss cache seem to hold at least 70% of the consumed heap. >>> This is clearly the area to focus on and verify that we can keep it in >>> control. >>> >>> Ludovic >>> >>> Le 07/05/10 16:50, Ludovic Dubost a écrit : >>> >>>> Hi developers, >>>> >>>> A while ago I was looking for some ways to track how much memory is >>>> used by our internal cache and was not able to find anything. >>>> I've tried it again and this time I found the following code: >>>> >>>> http://www.javamex.com/classmexer/ >>>> >>>> This requires a simple instrumentation to work, but I was able to get >>>> some results out of it to measure the size of our documents in cache. >>>> >>>> You can see the result on a personal server: >>>> >>>> Measuring one page: >>>> http://www.zapnews.tv/xwiki/bin/view/Admin/MemoryUsage >>>> >>>> Measuring all pages in cache: >>>> http://www.zapnews.tv/xwiki/bin/view/Admin/MemoryUsage?page=all >>>> >>>> The first results I can see, is that with no surprise the items taking >>>> most memory are: >>>> >>>> - attachment content >>>> - attachment archive >>>> - archive >>>> >>>> What I was able to see is that as expected these fields won't consume >>>> memory until we are asking for the data. >>>> And after a while, the memory is indeed discarded for these fields, so >>>> the usage of SoftReferences for them seem to work. >>>> >>>> Now what I can see is that the attachment archive can be very very >>>> very costly in memory. >>>> Also it does not seem clear how the memory from these fields is >>>> garbage collected (a GC did not recover it). >>>> >>>> With some experience of massive loading of attachments that lead to >>>> OutOfMemory errors in the server, I do suspect that the SoftReferences >>>> are not necessarly discarded fast enough to avoid the OutOfMemory. I >>>> also believe that a search engine that is walking all our pages >>>> including our archive pages can genearate important memory usage that >>>> could lead to problems. But this is only an intuition that needs to be >>>> proved. >>>> >>>> I believe we need to run some testing under stress to see if the cache >>>> and memory usage do behave properly and if the cache(s) are never able >>>> to go over the memory usage. >>>> >>>> We also should try the classmexer on servers that are heavily used an >>>> be able to look at the memory usage and see if we are "controling" it. >>>> I'm not 100% sure how intrusive the "instrumentation" module is but I >>>> believe it's quite light. >>>> >>>> We could try it on xwiki.org or on myxwiki.org. >>>> >>>> WDYT ? >>>> >>>> Ludovic >>>> >>>> >>>> _______________________________________________ >>>> devs mailing list >>>> [email protected] >>>> http://lists.xwiki.org/mailman/listinfo/devs >>>> >>>> >>> >>> ------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> devs mailing list >>> [email protected] >>> http://lists.xwiki.org/mailman/listinfo/devs >>> >> _______________________________________________ >> devs mailing list >> [email protected] >> http://lists.xwiki.org/mailman/listinfo/devs >> > > > -- > Ludovic Dubost > Blog: http://blog.ludovic.org/ > XWiki: http://www.xwiki.com > Skype: ldubost GTalk: ldubost > > > -- > Ludovic Dubost > Blog: http://blog.ludovic.org/ > XWiki: http://www.xwiki.com > Skype: ldubost GTalk: ldubost > _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
