So I upgraded one of my test systems to 2.5, from 2.4. Doing a quick
basic functionality check after the upgrade, I noticed that the
performance on the upgraded system was significantly slower.
When the system was running 2.4, searching for my entry after a cold
start took a little more than a second:
# time ldapsearch -x -b dc=cpp,dc=edu -H ldap://localhost/ uid=henson
[...]
dn: uid=henson,ou=user,dc=cpp,dc=edu
[...]
real 0m1.274s
and then subsequent searches generally less than 1/10th of a second:
real 0m0.010s
real 0m0.009s
real 0m0.007s
real 0m0.006s
real 0m0.010s
real 0m0.008s
After upgrading to 2.5, searches, both cold start and subsequent, all
took between 1.5-2 seconds:
real 0m1.513s
real 0m1.523s
real 0m1.308s
real 0m1.874s
real 0m2.150s
real 0m1.406s
These are test systems, so really have no load. I noticed on the
upgraded box that the CPU kept spiking to 100% every few seconds, and
tracked it down to the load balancer health checks, which basically run:
# time /opt/symas/bin/ldapsearch -x -b dc=cpp,dc=edu
'(objectClass=dcObject)'
On the upgraded system, this again takes about 1.5 seconds on every search:
real 0m1.483s
whereas on the 2.4 system, less than 1/10th of a second:
real 0m0.006s
It feels like something weird is going on with indexing. I turned on
debugging, and this is what the index activity looked like on the 2.4
system:
Oct 21 15:27:06 ldap-dev-02 slapd[73033]: <= mdb_index_read: failed (-30798)
Oct 21 15:27:06 ldap-dev-02 slapd[73033]: <= mdb_index_read: failed (-30798)
Oct 21 15:27:12 ldap-dev-02 slapd[73033]: <= mdb_index_read: failed (-30798)
Oct 21 15:27:12 ldap-dev-02 slapd[73033]: <= mdb_index_read 1 candidates
Which makes sense, there's only one of me :). On the other hand, on the
upgraded system:
Oct 21 15:25:55 ldap-dev-03 slapd[42259]: <= mdb_index_read: failed (-30798)
Oct 21 15:25:55 ldap-dev-03 slapd[42259]: <= mdb_index_read: failed (-30798)
Oct 21 15:26:19 ldap-dev-03 slapd[42259]: <= mdb_index_read 132354
candidates
It looks like it's doing a full directory search for my entry? My
configuration on the two systems is exactly the same, other than the
changed paths for the 2.5 directory layout, removing the password policy
schema include, and removing a pres index on automountInformation for
the upgrade advice.
I have tested this both with using the existing binary mdb database
files from the 2.4 system as well as dumping them out via slapcat,
deleting the old ones, and creating new ones with the 2.5 version of
slapadd, with the exact same behavior
Any thoughts on what might be going on or how I can debug it to track
down exactly what is causing it? There was obviously a lot more debug
info in the logs that I didn't include, but none of it jumped out to me
as "smoking gun".
Thanks much…
