[EMAIL PROTECTED] wrote:
Update of /repo/OpenLDAP/pkg/ldap/servers/slapd
Modified Files:
attr.c 1.113 -> 1.114
entry.c 1.149 -> 1.150
Log Message:
Entry/Attribute struct caching, to minimize malloc fragmentation
Note: this breaks LDAP_COMP_MATCH and SLAP_ZONE_MALLOC. But they
were probably broken already anyway.
So the most important part here is to always attr_alloc/attr_free and
entry_alloc/entry_free rather than calling a generic malloc/free for
these items now. There's some cruft in here that might be better suited
to a general purpose object manager, but it's working well for now.
Heap profiling with valgrind/massif reconfirmed what we already knew
when originally writing back-bdb: the actual data is only 50-60% of the
volume, 40-50% is consumed by Attribute and berval structures to point
to the data. Allocating them all one at a time imposes a significant
overhead. Originally entry_encode/decode for back-bdb tried to shove
everything into the DB, so that we could just allocate one chunk of
memory from the DB and carve it up into the appropriate structures. But
this basically made the DB twice as large, with the majority of space
filled with zeroes. This approach was abandoned before OpenLDAP 2.1 was
released.
Next the chunk was split into two, with the raw data going into the DB,
plus a few counters/lengths so we could let the DB alloc one chunk, and
entry_decode would alloc another single chunk for the Entry and its
Attribute and berval structures. That's been the approach from 2.1 to
2.3, but it turns out to be nasty in terms of heap fragmentation when
you have a lot of searches operating over a DB much larger than the
entry cache.
The new code here preallocates Entry and Attribute structures in chunks
of 1000 structs per malloc and keeps them on their own private free
lists. In addition, the back-bdb db_open function preallocates as many
Entries as its entry cache is set to, and 20x that many Attributes. So
in general, this part should have been removed from the fragmentation
equation. The only thing that remains is to allocate the vector of
struct bervals for pointing to the actual data. Again, that's done as a
single malloc per entry, and a new Entry field e->e_abv is used to
remember this pointer so it can be freed by entry_free.
This change has been sufficient to keep fragmentation away in a variety
of tests, though it's still a problem when the entry size varies greatly
in a DB. Really fixing that requires a smarter malloc.
--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc
OpenLDAP Core Team http://www.openldap.org/project/
