Quanah Gibson-Mount wrote:
--On Tuesday, August 28, 2007 2:44 PM -0400 matthew sporleder
<[EMAIL PROTECTED]> wrote:
That's an unfortunate development from oracle considering the
presentation that was released showing bdb4.6 to be very fast.
Yup. I believe Howard is working with Sleepycat to find out what was done
to 4.6 to make it suddenly be 3-4 times slower instead of significantly
faster as it was in the early releases.
Yes, it's been a long conversation so far. While their new memory manager
(inspired by Jong-Hyuk Choi's research on BerkeleyDB performance in OpenLDAP)
is a great improvement, their new lock manager is not.
This may not be an issue on all platforms. On x86 though, they have a hybrid
mutex mechanism which is enabled by default. It uses both assembly language
spinlocks and pthread mutexes, first spinning in the assembly language lock
some number of times before falling back onto the mutex. They claimed that this
improved performance in their tests, because pthread mutexes can be very
expensive on some platforms. In my tests on x86_64 Linux however, it just
forced CPU usage to 100% (200% actually, dualcore) and slowed down overall
throughput.
You can give an explicit "--with-mutex=POSIX/pthread" argument when configuring
BerkeleyDB to avoid the hybrid mutex scheme, in which case performance of BDB
4.6.19 seems to match what we obtained with BDB 4.6.3.
If you were using BerkeleyDB 4.5.20 before, the early BDB 4.6 releases will
work well for you. The only difference between 4.5.20 and 4.6.1 was in the
memory manager. 4.6.2 just tweaked some portability issues to support BREW
(Qualcomm's cellphone programming environment). 4.6.3 added the ability to
specify different cache priorities per database operation. (I didn't track what
changed between 4.6.3 and 4.6.18.)
And for anyone curious - you can read ITS#3851
http://www.openldap.org/its/index.cgi/Development?id=3851 for the background on
the problems in BerkeleyDB's memory allocator. While there was a fair amount of
debate as to whether Jong's proposed solution was of any benefit, it was pretty
clear that the existing code in BerkeleyDB could be improved. But the changes
in the memory allocator may not visibly affect you if you're running a
small-to-medium sized database. It's only apparent when the total volume of
data is much larger than the BerkeleyDB cache, because that's the condition
that leads to memory fragmentation, and it's the fragmentation that causes the
allocator's performance to degrade.
The same kind of problem affects the slapd entry cache, when the number of
active entries is much larger than the entry cache, and you're using the glibc
malloc. (At least, for glibc 2.3 and older. I haven't retested with glibc 2.5 yet.)
To sum up - if you're already using BDB 4.5.20 with OpenLDAP 2.3 and the
performance is acceptable, I wouldn't be in any hurry to upgrade to BDB 4.6.
Releases 4.6.18 and 4.6.19 are incompatible with OpenLDAP 2.3. BDB 4.6.1 is a
drop-in replacement for 4.5.20 though.
--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
