Hi,
>From: Stefano Mazzocchi [mailto:[EMAIL PROTECTED]]
>
>Gerhard Froehlich wrote:
>>
>> Hi,
>>
>> >From: Peter Hargreaves [mailto:[EMAIL PROTECTED]]
>> >
>> >Hi Gerhard and Team,
>> >
>> >I'm posting in this group now I think I found a design problem with
>> >store-janitor.
>>
>> Slowly but constant this Store issues are a PITA for me :-/. I don't know
>> why this is in Java that difficult to control a little bit memory.
>> That's the price for automatic Memory Management!
>> (Nothing against you Peter!)
>
>Eh, if you look at the VM problems in Linux 2.4 you'll see that this
>realm is a complex one itself and now just because it's java (sure, java
>doesn't help you when it hides everything from you, like in this case).
Complex is the point.
>> >I think store-janitor works fine when it can free sufficient memory by
>> >calling the finalize and garbage collection. But if it needs to free the
>> >store it locks up!
>> >
>> >The reason it appears to lock up (I think), is because it calls the
>> >garbage collector too often - i.e. every time it frees an item from a
>> >store. Once called the store-janitor will free an item and call the
>> >garbage collector, repeating both until it has freed sufficient memory.
>> >If one item frees 4.44k and the garbage collector takes 2844ms then the
>> >time taken to free 20m will be 3.5 hours - during which time the system
>> >appears to be locked!
>
>Ouch! that hurts!
Yep, I still recover from that ;).
>> >The following is a brief summary of my analysis:
>> >
>> >With the following settings:
>> >Xms=0m
>> >Xmx=140m
>> >heapsize =100000000
>> >freememory=20000000
>> >
>> >Using Optimizeit I watched my java heap 'total' and 'in-use' growing.
>> >When total heapsize gets to 100m the store-janitor kicks in and frees a
>> >chunk of memory. This repeats every ten seconds but each time it frees
>> >less memory until it is no longer able to free memory and locks up. When
>> >locked up the garbage collector is running about 99% of the time.
>
>That's *very* bad!
Indeed!
>> >Looking at the store-janitor code and my debug log I can see exactly
>> >what happens when it locks.
>> >
>> >I notice, examining Store-janitorImpl.run() that, if the garbage
>> >collector alone manages to clear sufficient memory, no store items are
>> >cleared (as expected). This is possibly all that is happening when
>> >store-janitor appears to work! However if that is not sufficient the
>> >following loop is started.
>> >
>> > synchronized (this) {
>> > while (this.memoryLow() &&
>> >this.getStoreList().size() > 0) {
>> > this.freeMemory();
>> > }
>> > this.resetIndex();
>> > }
>> >
>> >this.memoryLow() tests to see if sufficient is free and prints Debug info.
>> >this.freeMemory() removes an item, runs garbage collection, and prints
>> >Debug info.
>> >
>> > From my log files I can see this loop being executed. Each time round
>> >the loop I can see how much memory is freed and how long it takes.
>> >Typically it can take seconds to free one item which can be only a few
>> >kb. So, in my case with 'freememory' set to 20m, it is likely to take a
>> >few hours to free sufficient memory!!
>> >
>> >I've noticed when running Optimizeit that although garbage collection
>> >can be quick, it sometimes takes a few seconds. If it has just been run,
>> >and you force it to run again, then it seems to take longer. If it can't
>> >find much to freeup, it seems to do a more time consuming search for
>> >garbage.
>>
>> This all makes sense too me.
>> Then we need a limit how often the GC can be called. Maybe 100 loops
>> should be enough.
>
>I don't get it: why more than one loop? I mean, if you call the GC, it
>should free all the memory that it can... doing any more loops at that
>time, will not change anything.
Maybe I did something wrong. But the idea was that:
while(MemoryIsLow) {
free objects out of the store //physicall
run gc
}
I run the GC after each "free objects" to clean away
does not needed objects!
The problem is now, that when he doesn't has anything
to free from the registered stores, he runs the GC
all the time. And then we have this endless loop!
Therefore I should check if the Stores are empty. If they
are empty and there is nothing to free anymore we should
leave that loop.
>> After that it waits as configured (threadinterval)
>> before the next execution.
>>
>> Maybe there is a more elegent solution.
>
>Well, this is a very difficult realm, as I said, but sure, we can start
>a redesign phase on this.
We should!
>> I actually develop in jakarta-commons a store (again) with Weak References.
>> This is a much better solution. No Threads, no GC calls. When it is ready
>> to introduce I'll explain you!
>
>Be careful when going down this road: the JLS doesn't say anything about
>'how' the weak references should be cleaned. And Java 1.3 implements it
>in the most memory efficient possible way: if the GC gets a weak
>reference, all of them are removed.
>
>Now, imagine the case: you have spent a couple of hours building up and
>tuning your cache using highly complex statistical analysis... but you
>saved them into a store that is implemented using weak references.
>
>You hit the heap ceiling and the GC kicks in.
>
>If finds one of those weak references, but it doesn't have the ability
>to ask you (well, the store) about 'which is weaker', so, wooosh,
>everything is gone.
>
>Sure, you probably collect 40Mb or heap, but then your perceived
>performance drops and your cache system works in a 'saw-like' figure
>
>
>cache size
> ^
> | /| /|
> | / | / |
> | / | / |
> | / |/ |/
> +-----------------> time
> ^
> GC kicks in
>
>and since the cache size is linearly proportional with the perceived
>efficiency of the cache, Cocoon will appear to be 'fast at times' (to be
>kind) or 'awefully bumping' (to be more reasonable).
>
>The solution I thought for this problem is simple but I don't know if it
>works (never tested it real life):
>
> 1) partition the memory store into 'clusters' which are connected using
>hard references.
That is the idea of jakarta-commons simplestore. To build up a store
chain!
> 2) these 'clusters' are the real containers of the stored entities and
>such entities are weak-referenced from the 'clusters'
>
>The idea is to have something like this:
>
> store +==> cluster +---> entity
> | +---> entity
> | +---> entity
> |
> +==> cluster +---> entity
> +---> entity
> +---> entity
>
>where:
>
> ===> hard reference
> ---> soft reference
>
>and the key concept is that, whenever the GC needs to free some memory
>that is occupied by weak references, the avalanche cleanup will be
>'stopped' at a cluster level, saving the other clusters.
>
>Now, I'm not sure this works, but if it does, we have a reasonable
>solution to the problem.
We had the same idea and we are working on that.
==> jakarta-commons-sandbox/simplestore
"simple" to integrate "complex" to code ;-).
>> Sorry for all that crap with Store and everything.
>
>Don't worry, it takes time to do it right but I'm sure we'll get there.
>
>Probably we need to discuss its design *before* implementing it :)
Indeed ;-).
Gerhard
"The most important question when any new
computer architecture is introduced is 'So what?'"
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, email: [EMAIL PROTECTED]
