Hmm... Thanks again for the data. I will see what happens with a modification of those attributes.
However, I tried running the modifications with and without the index (I deleted it entirely). When it was present, it matched the default index which only uses the eq index type. I think it went slightly faster without the index but the difference was negligible compared to the overall time. Regards, Russ. On Apr 19, 2012, at 11:47 AM, Andrey Ivanov wrote: > I've forgotten Linked Attributes plugin, you could also disable it. > > Don't you have some exotic type of index activated for uniqueMember (like > substring)? The default value is only the equality index in dse.ldif > > dn: cn=uniquemember,cn=default indexes,cn=config,cn=ldbm database,cn=plugins,c > n=config > objectClass: top > objectClass: nsIndex > cn: uniquemember > nsSystemIndex: false > nsIndexType: eq > > > In any case, batch write loads are quite particular. You could try to play > with nsslapd-db-checkpoint-interval and nsslapd-db-durable-transaction config > attributes while you run your batch uniqueMember modifications. You could > also try disabling completely or limiting logging intensity > (http://directory.fedoraproject.org/wiki/Named_Pipe_Log_Script) . > > In my VM tests (RHEL5, 1 vCPU Xeon E5640 @ 2.67GHz, 1Gb mem, 389DS > v1.2.10.6) with our production environment (~20k users, groups of ~6000 > members created as in your case with perl scripts using ldapmodify running on > the same VM) a group of 6000 uniqueMembers is created in 3 minutes 10 sec > (190s) from scratch. Using "dstat" i see that the main problem is disk writes > (transaction logs of db4): > ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system-- > usr sys idl wai hiq siq| read writ| recv send| in out | int csw > 91 0 0 9 0 0| 0 40M| 120B 978B| 0 0 | 316 325 > 90 3 0 7 0 0| 0 40M| 60B 310B| 0 0 | 184 294 > 92 2 0 5 0 1| 0 40M| 60B 310B| 0 0 | 160 297 > 93 1 0 5 0 1| 0 40M| 60B 310B| 0 0 | 159 312 > 93 3 1 1 0 2| 0 24M| 60B 310B| 0 0 | 206 372 > 76 4 2 18 0 0| 0 40M| 60B 310B| 0 0 | 165 265 > 94 0 0 6 0 0| 0 40M| 60B 310B| 0 0 | 221 275 > 90 1 0 7 1 1| 0 41M| 60B 310B| 0 0 | 479 313 > 86 2 0 11 1 0| 0 40M| 60B 310B| 0 0 | 403 306 > 93 0 0 6 0 1| 0 20M| 120B 364B| 0 0 | 489 298 > 90 1 0 9 0 0| 0 40M| 60B 310B| 0 0 | 389 296 > 88 1 0 11 0 0| 0 40M| 60B 310B| 0 0 | 358 319 > 76 0 0 23 1 0| 0 41M| 60B 310B| 0 0 | 403 303 > > @+ > > > > Le 19 avril 2012 18:50, Russell Beall <[email protected]> a écrit : > Thanks for the tips. I scanned the dse.ldif for those plugins and I found > definitions for them all, but they all have nsslapd-pluginEnabled: off. > > There is something special about the uniquemember attribute that requires > additional processing different from other attributes... Ldapmodify of other > attributes runs pretty quick. > > Regards, > Russ. > > On Apr 19, 2012, at 2:20 AM, Andrey Ivanov wrote: > >> Hi Russel, >> >> >> Le 18 avril 2012 23:06, Russell Beall <[email protected]> a écrit : >> On Apr 18, 2012, at 11:15 AM, Rich Megginson wrote: >>> Yeah, this particular operation has not been optimized. I believe SunDS >>> added explicit optimizations for this particular case. >> >> >> It is becoming painfully apparent as I write more detailed tests. 389 takes >> time to add or delete uniquemember values proportionate to the number of >> values being operated on and is using about twice as much time to delete as >> it does to add. Sun DS appears to have perhaps an almost O(1) algorithm in >> play on both adding and deleting values. >> >> Is there perhaps some kind of referential integrity setting that is being >> used and forcing some kind of lookup of each value, one that we could >> perhaps turn off? We wouldn't need such a check because our metadirectory >> process handles the integrity/consistency checking already. >> >> There is memberOf plugin that maintains the memberOf attribute for groups. I >> don't know whether it is activated by default or not. You could try to >> disable it. There is also referential integrity plugin, attribute uniqueness >> plugin, maybe USN plugin or custom indexes that could consume a lot of CPU. >> Make sure you've disabled them if you don't need them. >> >> @+ >> -- >> 389 users mailing list >> [email protected] >> https://admin.fedoraproject.org/mailman/listinfo/389-users > > > -- > 389 users mailing list > [email protected] > https://admin.fedoraproject.org/mailman/listinfo/389-users > > -- > 389 users mailing list > [email protected] > https://admin.fedoraproject.org/mailman/listinfo/389-users
-- 389 users mailing list [email protected] https://admin.fedoraproject.org/mailman/listinfo/389-users
