Howard, Now it fits under my expectation. Under the initial load without the fix I had slapd not respecting any boundary and then all DN information being cached.
In this way the first query would take longer since the information wasn't yet cached. At that time I had, without cache, a search for around 1million entrances in around 7 minutes and after cached this time reduce to around half. So now with the new load I was expecting something a little longer as the initial not cached since now slapd will not be able to cache all information and so the cache will be often overwritten. Now in my tests I obtained : First time: real 6m50.017s user 0m17.346s sys 0m9.159s Second time: real 6m47.790s user 0m17.032s sys 0m9.137s Third time: real 7m17.459s user 0m17.412s sys 0m9.320s This is more than perfect. Amazing how you guys could solve this problem so quickly. I saw sometimes the DN cache allocating a little more memory since it sometimes pass a little the boundary, like seen below : olmBDBEntryCache: 1000 olmBDBDNCache: 1000247 olmBDBIDLCache: 1 The maximum I saw was 1000268 but nothing considerable. In this way now slapd is keeping memory allocation stable as it is supposed to behave. The only comment is if dn cache is passed this memory allocated appears to never be released. The maximum size slapd obtained was : PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 10195 ldap 18 0 385m 266m 68m S 0 2.2 20:54.16 slapd This is more than perfect ! Thanks for all your quick solution of this problem. I believe many users of openLDAP will see their system more stable with this fix. Just some last questions. Is there any expectation for a new release, for example 2.4.15, with this ITS included? Just to use a formal release. By the tests I was expecting the cachefree as the cache size bringing some improvements, at least in a full query, since system would "cycle" pointer memory and overwritten all cache at once. What is the purpose of cachefree? Just to understand better and then tune later with some better idea about cache purposes. Best Regards, Rodrigo. --- On Mon, 1/26/09, Howard Chu <[email protected]> wrote: > From: Howard Chu <[email protected]> > Subject: Re: (ITS#5860) slapd memeory leak under openldap 2.4 > To: [email protected] > Cc: [email protected] > Date: Monday, January 26, 2009, 6:00 PM > Rodrigo Costa wrote: > > Howard, > > > > I do not think the ITS is completely solved. Let me > try to explain better. > > > > I have the system respecting now the DN cachesize > boundary. At the same time I wasn't sure about the > cachefree idea and now I changed to 1, or equal the default > value. > > > > In any case what happens is : > > > > 1) I have the system doing sequential search in a > reasonable speed until the dncachesize is reached; > > 2) After dncchesize is reached the sequential search > hangs, or the output from the search get stuck for a long > time(I'm forwarding to a file so I do not have screen > actualization delays); > > Ex: > > [r...@brtldp11 backup]# date; cat temp.ldif |grep -e > '^pnnum*' |wc -l > > Mon Jan 26 17:22:21 BRST 2009 > > 250016 > > [r...@brtldp11 backup]# date; cat temp.ldif |grep -e > '^pnnum*' |wc -l > > Mon Jan 26 17:23:00 BRST 2009 > > 250016 > > See above that even almost after 1 minute passed not > new LDIF entrance was included. > > 3) Then query get stuck and looks like deterministic > it time by time only dumps 16 entrances and get stuck. This > behavior repeats and with these stuck that sometime gets > minutes the query never ends. > > > > olmBDBEntryCache: 884 > > olmBDBDNCache: 1000261 > > olmBDBIDLCache: 1 > > > > olmBDBEntryCache: 611 > > olmBDBDNCache: 1000261 > > olmBDBIDLCache: 1 > > > >Even with cachesize as 1000 and cachefree as 1, the > olmBDBEntryCache continues > to decrease, just slow now. > > > > I was expecting that a cachefree as 1000 would purge > all entrances and then > > cache again all 1000 new with in sequence always > answering the search. So > > the search would never hangs like it is happening now. > > I see. OK, this is now fixed in HEAD. > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP > http://www.openldap.org/project/
