Hi folks,
While trying to profile ApacheDS I was playing with tunable cache
parameters. Only recently, we made the entry cache and index cache
sizes tunable so I was curious to see how these effected performance.
For performance tests I started working with 10,000 entries and tried
various configurations of the server's memory along with these cache
sizes. The key was to determine how fast I can get the server to add,
delete and return all these entries.
I found that the existing defaults in RC3 were way less than optimal.
Namely we had far too much cache allocated to indices which were robbing
the server of memory causing it to have major GCs way to often. The
balance between the cache sizes and max heap size is critical. It makes
a big difference to performance.
After several configurations I arrived at one which seemed to lead to
the best performance when using one single threaded client. The entry
cache size was set to 10K and the index cache sizes were set to 10 with
a heap size of 96MB. Increasing the heap above this value had no
benefit with this simplified testing scenario. Of course as we increase
the number of threads and clients this may change.
Here's the machine's processor:
[EMAIL PROTECTED]:~$ cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 6
model : 10
model name : AMD Athlon(TM) XP
stepping : 0
cpu MHz : 2500.561
cache size : 512 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 mmx fxsr sse syscall mmxext 3dnowext 3dnow
bogomips : 5005.28
Here's are the performance stats I got on that machine:
[NOTE: EPS means entries per second]
Add 10K entries => 1m19.774s (125 EPS)
Search 10K entries (1st try) => 0m12.208s (819 EPS)
Search 10K entries (2nd try) => 0m4.716s (2120 EPS)
Search 10K entries (3rd try) => 0m4.861s (2057 EPS)
Del 10K entries => 0m41.049s (244 EPS)
As you can see the throughput experiment for a single client for these
simple operations is pretty damn good. Btw I got better performance
when I piped the client's output to /dev/null from the ldapXXXX clients:
I had quoted higher values to Emmanuel over IM.
At ApacheCon EU Emmanuel did a presentation with a different setup.
First of all the server could not be tuned properly so it had issues
with memory management and performance would be effected. Secondly the
nature of the experiments were different. The common case where a bind,
simple one entry lookup, followed by an unbind was done. The number of
times a server can do this was measured per second. ApacheDS did not
perform that well here and some profiling indicates this may be due to
an overhead in the optimizer. This optimizer does some calculations
before beginning a search to increase performance however it has a cost.
For simple lookups returning a single entry this may not be optimal.
I think adding an option to disable the optimizer is a good idea.
Just for a point of reference I decided to try these experiments with
OpenLDAP. Ok some word of caution before I reveal these stats. I
freaked when I first did a run with OpenLDAP because the performance had
to be way too low. So then I started looking into seriously tuning the
server with the bdb backend. I wanted to make sure it had enough cache
to keep all 10K entries in memory as did ApacheDS. I adjusted these
values and only got a 30% improvement. In conclusion I feel that I must
be doing something wrong.
*Quanah* this is where we may need your expertise! I'll publish these
sorry values for now but I'd like to make sure people understand I'm not
as experienced with tuning OpenLDAP (although I've done it a few times
in the past). So they may not be figures from the most optimial
configuration:
Add 10K entries => 6m13.013s (27 EPS)
Search 10K entries (2nd try) => 19m14.748s (9 EPS)
Search 10K entries (2nd try) => 13m57.378s (12 EPS)
Search 10K entries (3rd try) => 13m41.407s (12 EPS)
Del 10K entries => 4m46.699s (35 EPS)
*Yes these are all in minutes!*
Again to be fair I am piping stdout output to /dev/null. The following
commands were used for these respective operations:
[For OpenLDAP running on port 389]
----------------------------------
time ldapadd -h localhost -p 389 -D 'cn=admin,dc=example,dc=com' -x -w
secret -f add.10k.ldif >> /dev/null
time ldapsearch -h localhost -p 389 -D 'cn=admin,dc=example,dc=com' -x
-w secret -b 'dc=example,dc=com' 1.1 > /dev/null
*[0]*
time ldapdelete -h localhost -p 389 -D 'cn=admin,dc=example,dc=com' -x
-w secret -f del.10k.ldif
Check out the following wiki page for more information about these
tests: *[1]*. Here you can find the ApacheDS installer which I used for
these tests: *[2]*.
Alex
-----
*[0]* - We are not even returning whole entries! '1.1' for the attribute
list only returns the DN of the entry. I made sure we have presence and
eq matching indices especially for objectClass since this search
defaults to an (objectClass=*) filter. This was done before doing
anything with the server: namely creating the new bdb database files and
adding the entries.
*[1]* - http://monretia.notlong.com/
*[2]* - http://dulcaned.notlong.com/
