Re: Memory Leak in PropertyCache: need ideas

[Andreas Delmelle]

On Aug 21, 2008, at 18:47, Jeremias Maerki wrote:

I've tested only on Windows with Sun JDK 6.0_03 (dual core,
non-multithreaded in MemoryEater). The reporter will have had a
different setup but I don't know what it is without asking back. I
believe it's JVM-independent. cleanSegment() does get called but I
suspect it doesn't get called regularly for all segments to have the
wanted effect of a clean-up.

That's a possibility. I somehow wondered about this: in the long run, especially in multi-session environments, get() will virtually always return a value and put() may not be called frequently enough anymore... As long as there is no put(), the cleanup will not be triggered as often. For EnumProperty, the issue most likely does not appear since the enums are all created when initializing FOPropertyMapping. So no new instances are added during normal processing, and the cache only serves unsynchronized get-requests.

I'm attaching my changes I've done while debugging. It might help you if

you want to write out those statistics I've shown. Please note that I
made CacheEntry static which makes the instance smaller.

Thanks for this! I was unaware of that particular effect.

Good luck and thanks for helping!

The bugger is already showing in the very first few cleanups:

(filtered to show only the relevant buckets)

bucket #3
before: 35 entries, 32 stale
after: 19 entries, 16 stale

bucket #2
before: 43 entries, 41 stale
after: 27 entries, 25 stale

bucket #7
before: 49 entries, 47 stale
after: 33 entries, 31 stale

bucket #4
before: 43 entries, 42 stale
after: 27 entries, 26 stale

bucket #2
before: 39 entries, 37 stale
after: 24 entries, 22 stale

bucket #6
before: 48 entries, 46 stale
after: 32 entries, 30 stale

It really seems like I made a serious error in there... :-/










Hope this helps some.

In the meantime, I'm observing the effect here too. Only following the StringProperty cache, where the issue is most manifest. So, I was wondering about the role of the reference queue. Calling poll() returns the first reference /if/ one is available, so it seems likely that cleanSegment() does get called frequently enough, but there's always the possibility that it returns without a reference being available, yet. Or, as it happens, after throwing away only roughly half of the references. Maybe it's only those that are also effectively enqueued at the time the cleanup is triggered (didn't try adding this to the stats yet). In the meantime, I've made some changes, bypassing the ReferenceQueue, and simply cleaning all buckets that map to the segment in question, and now the behavior already seems more like what I had in mind. The first runs, each segment corresponds to one bucket, and the output now shows the number dropping to 0% for the related bucket. See the attached patch. Still needs some cleanup, and haven't done any long-running tests, nor checked the cache for other types. I copied some, but not all of the added debug-code. There are still a few things that bug me about StringProperty's cache, since for example, it is used in a lot of cases where not the property but its String value is attached to the FOs (reducing the benefits of caching somewhat, and probably explaining the large number of stale entries here...)

propcache_leakfix.diff
Description: Binary data

Cheers

Andreas