RE: pls help: performance tuning
Hi all, Thanks for your help. Regarding the "too many executing" source, it is an anti-spam/anti-virus gateway blackbox that performs recipient checking using ldap query. I'm afraid it's difficult to change her behaviour to split out hundreds of operations over multiple connections. I'm check the product manual of this blackbox, but probably won't be able to change it. Besides, there are some other deferring operations logged from the same source: 11:May 10 03:09:38 fiesta slapd[31600]: connection_input: conn=633362 deferring operation: pending operations 11:May 10 03:04:56 fiesta slapd[31600]: connection_input: conn=712193 deferring operation: binding The slapd doesn't respond or respond slowly when the problem occurs. The servers each has 4 GB RAM. Here comes some setting in slapd.conf and DB_CONFIG: slapd.conf: sizeLimit 50 timeLimit 600 cacheSize 100 checkPoint 1024 10 IDLcacheSize3000 threads 32 sockbuf_max_incoming4194303 DB_CONFIG: set_cachesize 2 0 1 set_lg_bsize 20971520 set_lg_max 83886080 set_lg_regionmax 2621440 set_lk_max_locks 1 set_lk_max_objects 5000 Would like to know if there is any problem with the setting. Thanks again. /ST Wong -Original Message- From: [email protected] [mailto:[email protected]] On Behalf Of Howard Chu Sent: Wednesday, May 12, 2010 2:25 AM To: Aaron Richton Cc: ST Wong (ITSC); [email protected] Subject: Re: pls help: performance tuning Aaron Richton wrote: > On Tue, 11 May 2010, ST Wong (ITSC) wrote: > >> I currently set thread in slapd.conf to 32 (default x 2). While I >> think shouldn't modify SLAP_MAX_WORKER_THREADS, I tried to increase >> no. of threads, at the cost of read performance. Besides, after >> increasing thread to over 32, say 100, I got warning of "threads >> larger than twice the default". >> >> Would like to know the recommended way to tune the system in our case. >> Would anyone pls help? > > "Recommended way to tune the system" is quite the slippery slope, but > here goes nothing... Note that the "too many executing" message refers to the number of operations executing on a single connection. The first thing to do is find out why this single connection is so busy. Well behaved clients would not be spitting out hundreds of operations over a single connection all at once. > It sounds like you're focusing primarily on parallelism, which > certainly has its merits. But I might sooner look at latency per > operation. For example, with loglevel stats, you should have visiblity > of client operations that are taking too long (i.e. delay until RESULT lines). > Perhaps an application started searching a field that isn't properly > indexed, for example. > > Also, you don't mention results from looking for obvious bottlenecks > (spindles, bus, cpu, etc.). For example, if you're waiting on hard > drives, adding more threads is unlikely to help and in many cases > would hurt. The likely solution there, short of a hardware upgrade, is > adding more replicas (which also has benefits for redundancy/HA > concerns, in addition to alleviating hardware limitations). > > Finally, you mention that you're using BerkeleyDB. You should post the > size of your working set and your DB_CONFIG parameters. A 4-core > system sounds fairly modern, it generally shouldn't be going to > disk...if DB_CONFIG is not tuned accordingly, that would drastically > affect performance. You can get some ideas for DB_CONFIG tuning in the > FAQ-O-Matic and list archives. > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Re: pls help: performance tuning
Aaron Richton wrote: On Tue, 11 May 2010, ST Wong (ITSC) wrote: I currently set thread in slapd.conf to 32 (default x 2). While I think shouldn't modify SLAP_MAX_WORKER_THREADS, I tried to increase no. of threads, at the cost of read performance. Besides, after increasing thread to over 32, say 100, I got warning of "threads larger than twice the default". Would like to know the recommended way to tune the system in our case. Would anyone pls help? "Recommended way to tune the system" is quite the slippery slope, but here goes nothing... Note that the "too many executing" message refers to the number of operations executing on a single connection. The first thing to do is find out why this single connection is so busy. Well behaved clients would not be spitting out hundreds of operations over a single connection all at once. It sounds like you're focusing primarily on parallelism, which certainly has its merits. But I might sooner look at latency per operation. For example, with loglevel stats, you should have visiblity of client operations that are taking too long (i.e. delay until RESULT lines). Perhaps an application started searching a field that isn't properly indexed, for example. Also, you don't mention results from looking for obvious bottlenecks (spindles, bus, cpu, etc.). For example, if you're waiting on hard drives, adding more threads is unlikely to help and in many cases would hurt. The likely solution there, short of a hardware upgrade, is adding more replicas (which also has benefits for redundancy/HA concerns, in addition to alleviating hardware limitations). Finally, you mention that you're using BerkeleyDB. You should post the size of your working set and your DB_CONFIG parameters. A 4-core system sounds fairly modern, it generally shouldn't be going to disk...if DB_CONFIG is not tuned accordingly, that would drastically affect performance. You can get some ideas for DB_CONFIG tuning in the FAQ-O-Matic and list archives. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Re: pls help: performance tuning
On Tue, 11 May 2010, ST Wong (ITSC) wrote: I currently set thread in slapd.conf to 32 (default x 2). While I think shouldn't modify SLAP_MAX_WORKER_THREADS, I tried to increase no. of threads, at the cost of read performance. Besides, after increasing thread to over 32, say 100, I got warning of "threads larger than twice the default". Would like to know the recommended way to tune the system in our case. Would anyone pls help? "Recommended way to tune the system" is quite the slippery slope, but here goes nothing... It sounds like you're focusing primarily on parallelism, which certainly has its merits. But I might sooner look at latency per operation. For example, with loglevel stats, you should have visiblity of client operations that are taking too long (i.e. delay until RESULT lines). Perhaps an application started searching a field that isn't properly indexed, for example. Also, you don't mention results from looking for obvious bottlenecks (spindles, bus, cpu, etc.). For example, if you're waiting on hard drives, adding more threads is unlikely to help and in many cases would hurt. The likely solution there, short of a hardware upgrade, is adding more replicas (which also has benefits for redundancy/HA concerns, in addition to alleviating hardware limitations). Finally, you mention that you're using BerkeleyDB. You should post the size of your working set and your DB_CONFIG parameters. A 4-core system sounds fairly modern, it generally shouldn't be going to disk...if DB_CONFIG is not tuned accordingly, that would drastically affect performance. You can get some ideas for DB_CONFIG tuning in the FAQ-O-Matic and list archives.
