Hi Folks, Its great to see such enthusiasm - thanks for listening. Just got a coupe of comments and a simple idea that might help towards the solution - if you scroll down.
Gerhard Froehlich wrote: > Hi, > > >From: Stefano Mazzocchi [mailto:[EMAIL PROTECTED]] > > > >Gerhard Froehlich wrote: > >> > >> Hi, > >> > >> >From: Peter Hargreaves [mailto:[EMAIL PROTECTED]] > >> > > >> >Hi Gerhard and Team, > >> > > >> >I'm posting in this group now I think I found a design problem with > >> >store-janitor. > >> > >> Slowly but constant this Store issues are a PITA for me :-/. I don't > know > >> why this is in Java that difficult to control a little bit memory. > >> That's the price for automatic Memory Management! > >> (Nothing against you Peter!) > > > >Eh, if you look at the VM problems in Linux 2.4 you'll see that this > >realm is a complex one itself and now just because it's java (sure, java > >doesn't help you when it hides everything from you, like in this case). > > Complex is the point. > > >> >I think store-janitor works fine when it can free sufficient memory by > >> >calling the finalize and garbage collection. But if it needs to > free the > >> >store it locks up! > >> > > >> >The reason it appears to lock up (I think), is because it calls the > >> >garbage collector too often - i.e. every time it frees an item from a > >> >store. Once called the store-janitor will free an item and call the > >> >garbage collector, repeating both until it has freed sufficient > memory. > >> >If one item frees 4.44k and the garbage collector takes 2844ms then > the > >> >time taken to free 20m will be 3.5 hours - during which time the > system > >> >appears to be locked! > > > >Ouch! that hurts! > > Yep, I still recover from that ;). > > >> >The following is a brief summary of my analysis: > >> > > >> >With the following settings: > >> >Xms=0m > >> >Xmx=140m > >> >heapsize =100000000 > >> >freememory=20000000 > >> > > >> >Using Optimizeit I watched my java heap 'total' and 'in-use' growing. > >> >When total heapsize gets to 100m the store-janitor kicks in and > frees a > >> >chunk of memory. This repeats every ten seconds but each time it frees > >> >less memory until it is no longer able to free memory and locks up. > When > >> >locked up the garbage collector is running about 99% of the time. > > > >That's *very* bad! > > Indeed! > > >> >Looking at the store-janitor code and my debug log I can see exactly > >> >what happens when it locks. > >> > > >> >I notice, examining Store-janitorImpl.run() that, if the garbage > >> >collector alone manages to clear sufficient memory, no store items are > >> >cleared (as expected). This is possibly all that is happening when > >> >store-janitor appears to work! However if that is not sufficient the > >> >following loop is started. > >> > > >> > synchronized (this) { > >> > while (this.memoryLow() && > >> >this.getStoreList().size() > 0) { > >> > this.freeMemory(); > >> > } > >> > this.resetIndex(); > >> > } > >> > > >> >this.memoryLow() tests to see if sufficient is free and prints > Debug info. > >> >this.freeMemory() removes an item, runs garbage collection, and prints > >> >Debug info. > >> > > >> > From my log files I can see this loop being executed. Each time round > >> >the loop I can see how much memory is freed and how long it takes. > >> >Typically it can take seconds to free one item which can be only a few > >> >kb. So, in my case with 'freememory' set to 20m, it is likely to > take a > >> >few hours to free sufficient memory!! > >> > > >> >I've noticed when running Optimizeit that although garbage collection > >> >can be quick, it sometimes takes a few seconds. If it has just been > run, > >> >and you force it to run again, then it seems to take longer. If it > can't > >> >find much to freeup, it seems to do a more time consuming search for > >> >garbage. > >> > >> This all makes sense too me. > >> Then we need a limit how often the GC can be called. Maybe 100 loops > >> should be enough. > > > >I don't get it: why more than one loop? I mean, if you call the GC, it > >should free all the memory that it can... doing any more loops at that > >time, will not change anything. > > Maybe I did something wrong. But the idea was that: > > while(MemoryIsLow) { > free objects out of the store //physicall > run gc > } > > I run the GC after each "free objects" to clean away > does not needed objects! > > The problem is now, that when he doesn't has anything > to free from the registered stores, he runs the GC > all the time. And then we have this endless loop! > > Therefore I should check if the Stores are empty. If they > are empty and there is nothing to free anymore we should > leave that loop. Is this where you mean: while (this.memoryLow() && this.getStoreList().size() > 0) { this.freeMemory(); } I assumed that "this.getStoreList().size()" returned 0 when all the stores were empty. If this is not the case (as you suggest) then it looks like a permanent loop. I have seen it come out of this loop however. Perhaps another thread freed up memory letting the gc become successful at breaking out of the loop through "this.memoryLow()". I saw it break out of the loop when I had the jvm -Xnoclassgc option set. With this option classes that haven't been used for a while get dropped from memory don't they? So, if stuck in a loop, it won't be long before there are a lot of classes that haven't been used for a while! When I saw it come out, it was spectacular - a sudden big drop in memory in use! > > >> After that it waits as configured (threadinterval) > >> before the next execution. > >> > >> Maybe there is a more elegent solution. Hava you considered a sort of binary approach to emptying stores? while (MemoryIsLow) { free half of the items in the next store run gc } > > > >Well, this is a very difficult realm, as I said, but sure, we can start > >a redesign phase on this. > > We should! > > >> I actually develop in jakarta-commons a store (again) with Weak > References. > >> This is a much better solution. No Threads, no GC calls. When it is > ready > >> to introduce I'll explain you! > > > >Be careful when going down this road: the JLS doesn't say anything about > >'how' the weak references should be cleaned. And Java 1.3 implements it > >in the most memory efficient possible way: if the GC gets a weak > >reference, all of them are removed. > > > >Now, imagine the case: you have spent a couple of hours building up and > >tuning your cache using highly complex statistical analysis... but you > >saved them into a store that is implemented using weak references. > > > >You hit the heap ceiling and the GC kicks in. > > > >If finds one of those weak references, but it doesn't have the ability > >to ask you (well, the store) about 'which is weaker', so, wooosh, > >everything is gone. > > > >Sure, you probably collect 40Mb or heap, but then your perceived > >performance drops and your cache system works in a 'saw-like' figure > > > > > >cache size > > ^ > > | /| /| > > | / | / | > > | / | / | > > | / |/ |/ > > +-----------------> time > > ^ > > GC kicks in > > > >and since the cache size is linearly proportional with the perceived > >efficiency of the cache, Cocoon will appear to be 'fast at times' (to be > >kind) or 'awefully bumping' (to be more reasonable). > > > >The solution I thought for this problem is simple but I don't know if it > >works (never tested it real life): > > > > 1) partition the memory store into 'clusters' which are connected using > >hard references. > > That is the idea of jakarta-commons simplestore. To build up a store > chain! > > > 2) these 'clusters' are the real containers of the stored entities and > >such entities are weak-referenced from the 'clusters' > > > >The idea is to have something like this: > > > > store +==> cluster +---> entity > > | +---> entity > > | +---> entity > > | > > +==> cluster +---> entity > > +---> entity > > +---> entity > > > >where: > > > > ===> hard reference > > ---> soft reference > > > >and the key concept is that, whenever the GC needs to free some memory > >that is occupied by weak references, the avalanche cleanup will be > >'stopped' at a cluster level, saving the other clusters. > > > >Now, I'm not sure this works, but if it does, we have a reasonable > >solution to the problem. > > We had the same idea and we are working on that. > ==> jakarta-commons-sandbox/simplestore > "simple" to integrate "complex" to code ;-). > > >> Sorry for all that crap with Store and everything. > > > >Don't worry, it takes time to do it right but I'm sure we'll get there. > > > >Probably we need to discuss its design *before* implementing it :) > > Indeed ;-). > > Gerhard > > "The most important question when any new > computer architecture is introduced is 'So what?'" > Peter. --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, email: [EMAIL PROTECTED]
