On Wed, Apr 25, 2012 at 4:45 PM, Emmanuel Lécharny <[email protected]> wrote: > Le 4/5/12 1:09 AM, Emmanuel Lécharny a écrit : >> >> Le 4/5/12 12:43 AM, Selcuk AYA a écrit : >>> >>> On Wed, Apr 4, 2012 at 3:22 PM, Emmanuel Lécharny<[email protected]> >>> wrote: >>>> >>>> It's systematic, and I guess that the fact we now pond the RdnIndex >>>> table >>>> way more often than before (just because we don't call anymore the >>>> OneLevelIndex) cause the cache to get filled and not released fast >>>> enough. >>> >>> do we hold a cursor open while this code gets stuck? I would think we >>> hold a cursor open and moduify quite a bit of jdbm btree pages for >>> this kind of behavior to happen. >> >> >> I'll check that. >>>> >>>> As we don't set any size for the cache, its default size is 1024. For >>>> some >>>> of the tests, this mightnot be enough, as we load a lot of entries >>>> (typically the schema elements) plus many others that get added and >>>> removed >>>> while running tests in revert mode. >>>> >>>> If I increase the default size to 65536, the tests are passing. >>>> >>>> Ok, now, I have to admit I haven't - yet - looked at the LRUCache code, >>>> and >>>> my analysis is just based on what I saw by quickly looking at the code, >>>> the >>>> stack traces I have added and some few blind guesses. >>>> However, I think we have a serious issue here. As far as I can tel, the >>>> code >>>> itself is probably not responsible for this behaviour, but the way we >>>> use it >>>> is. >>>> >>>> Did I missed something ? Is there anything we can do - except increase >>>> the >>>> cache size - to get the tests passing fine ? >>>> >>>> I'm more concern about what could occur in real life, when some users >>>> will >>>> load the server up to a point it just stop responding... >>> >>> to aovid this issue, we can let the writers allocate more cache >>> pages(rather than keeping the cache size fixed) so that they do not >>> loop waiting for a replaceable cache. However, I would again suggest >>> making sure we do not forget the cursor open. If we forget a cursor >>> open and keep allocating new cache pages for writes, we will have >>> other problems. >> >> Yeah, I can see how it may affect the tests. I'll definitively investigate >> this first, before going any further in another direction. >> >> ATM, I'm using a not committed version of JDBM were the default cache size >> has been changed. >> >> Thanks a lot Selcuk ! > > > So I still have the LRUCache size issue, after having removed the SubLevel > index. Once I increased the size to 1 << 16, tests are passing. > > The failing tests are the SearchAuthorizationIT class' tests. > > What happens is that when I add an entry, I update many elements in the > RdnIndex, as I have to modify the nbDescendant in all its parents. As those > tests are injecting a lot of entries, so they do a lot of modifications in > the RdnIndex. > > I checked that all the cursors are correctly closed. > > Any clue ?
Can you provide the following details: -on which jdbm table are you having the problem(rdn index, main table?)? - approximately how many modifications are you doing on this table while you are holding a cursor open( even if the cursor is held open legally). Knowing this number would help a lot. - is the same problem you had before or did closing the cursors in the previous case solve your problem? this kind of problem can currently occur if a thread is holding a cursor on one table(not necessarily illegally) and the "same" thread is modifying the "same" table with many add/delete/update operations. I am wondering whether we have a use case like this now. If we have, then I can change the code to account for it. > > -- > Regards, > Cordialement, > Emmanuel Lécharny > www.iktek.com > thanks Selcuk
