On Mon, Nov 30, 2009 at 2:51 PM, Markus Kohler <[email protected]>wrote:
> Hi all, > Here come some details about my memory allocation analysis done after > Vassil > fixed the most important issue. > > -we are at 4,93Mbyte temporary allocated for one message send > - 2,7Mbyte alone are allocated for char[]. There's seems to be some easy > optimization possible, because 1,5Mbyte is allocated in > StringBuilder.expandCapacity. How many of these allocations escape the eden pool? Have you run any of this on a 1.6_16 JVM (the one with escape analysis)? If so, does it make any difference? Also, doing a length on a List is O(n), so I'm not sure the value of getting the length as part of a pre-allocation strategy. Is the mkString stuff happening as part of the parse combinator? If so, we can change up some strategies pretty easily. > This is caused by scala not preallocating the > StringBuilder with the correct size in List.mkString. The reason for this > behaviour is that mkString is not implemented in List but in the more > abstract Iterator. Iterator doesn't have a length (for a good reason) and > therefore mkString has no idea how large the StringBuilder has to be at the > end. > I believe (without looking at the Scala source code so far) that mkString > should be reimplemented in List. We can probably go around this issue > somehow, until some does implement a better List.mkString in scala (have to > check the scala sources, whether that's possible at all). > I think as soon as we fixed this, we should maybe stop with the memory > allocation optimizations, because we are already very good, and we should > do > more tests before we put more effort into this topic. > > The rest of the allocations are pretty minor. It's interesting the > ListBuffer is used not sure whether that is the best choice here. I will > provide an profiler report, for those how want to look into this. > > I will send the corresponding profiling reports to Dick. > > - I haven't done a heap dump yet. I want to do 2 of them to compute the > amount of memory for one user. > > Regards, > Markus > > "The best way to predict the future is to invent it" -- Alan Kay > > > On Fri, Nov 27, 2009 at 5:28 PM, Markus Kohler <[email protected] > >wrote: > > > Hi all, > > In short, We are down from 90Mbyte to *19Mbyte* for sending one message > > and I believe it should be possible to get under 10Mbyte without too > much > > effort. With 10 Mbyte we would comply to the performance standard of > some > > major ERP vendor, but honestly I believe we should be able to do even > better > > ;-) > > > > Out of the 19Mbyte(maybe even only 11, it's bit hard to get reproducible > > results) 9,9 are consumed by scala.xml.XML$.loadString(java.lang.String) > > mainly because every time the routine is called, a new Parser is > > instantiated. In addition > javax.xml.parsers.SAXParserFactory.newInstance() > > is called, which hits (unless that has changed in recent JDK's) the file > > system to find out which XML parser is configured. *I think ideally this > > should be fixed in the Scala sources. * > > > > The issue with the message formatted 300 times when all users are logged > on > > still seems to be there, at least the GC log suggests that, but I guess > > David will attack that soon. I will send around the profiling reports > later > > on. > > > > Regards, > > Markus > > > > > > "The best way to predict the future is to invent it" -- Alan Kay > > > -- Lift, the simply functional web framework http://liftweb.net Beginning Scala http://www.apress.com/book/view/1430219890 Follow me: http://twitter.com/dpp Surf the harmonics
