I found the problem. Stupid problem, this took us several days! Since it has been so many days, I quote the full original question for your reference, see the bottom for the last jigsaw of the puzzle
Zhang Weiwu wrote: > We have a development server 'emerson' with roughly one third of the > performance of our productional server 'bossdog', it runs ldap query 10 > times faster than the productional server. I checked everything I can to > find out the difference, it seems both are equally configured. > > Test method: > > emerson:~# time for (( i=0; i<100; i++)); do ldapsearch -xD > cn=manager,dc=eoa,dc=cn -w masked -b ou=contacts,ou=realss.com,dc=eoa,dc=cn > '(uidNumber=7)' dn; done; > [snip] > # zhangweiwu, contacts, realss.com, eoa.cn > dn: uid=zhangweiwu,ou=contacts,ou=realss.com,dc=eoa,dc=cn > > # search result > search: 2 > result: 0 Success > > # numResponses: 2 > # numEntries: 1 > > real 0m1.047s > user 0m0.320s > sys 0m0.352s > > bossdog:~# time for (( i=0; i<100; i++)); do ldapsearch -xD > cn=manager,dc=eoa,dc=cn -w masked -b ou=contacts,ou=realss.com,dc=eoa,dc=cn > '(uidNumber=7)' dn; done; > [snip] > > real 0m41.790s > user 0m0.232s > sys 0m0.464s > > > What I have compared: > > 1. Both server uses Debian Lenney > 2. Both server have same DB_CONFIG in /var/lib/ldap > 3. Both server have same ACL settings. In fact we tried to reduce the > ACL setting on the slow preforming productional server without > improvement result. > 4. Both server uses bdb > 5. Compare syslog at log level 256 both server produce same log messages > 6. The productional server shows harddisk is twice as fast as > development server (hdparm), free memory more than the development > server, and its dual-core xeon cpu should outperform the VIA CPU > used in the development server, yet slower. The productional > server average load is 0.5 which is not high for dual-CPU (with > hyperthreading display as 4 CPUs to Linux). > 7. Productional server DB is then re-built (by rm -rf & slapadd) > without noticeable change in performance. > > This performance difference resulted the productional server user > interface much slower than user's patience. > > What would you check further in this case? Thanks in advance The problem is we have been looking into openldap and system resource/kernel so much that we forgot other software directly related to performance. There had been something wrong made syslog low performance, change loglevel to none instantly solves the problem.
