Soisik Froger wrote: > On 01/09/2021 17:17, Quanah Gibson-Mount wrote: >> >> >> --On Wednesday, September 1, 2021 2:07 PM +0100 Mark Cairney >> <[email protected]> wrote: >> >>> Hi, >>> >>> I've been out the LDAP loop for a bit but the recent discussion of the >>> memberof overlay on 2.5 piqued my curiosity. Having upgraded a Dev box, >>> removed the memberof elements from the database and replaced the >>> memberof overlay with dynlist the queries appear to work as expected but >>> are both a) slow and b) heavily CPU-intensive on the LDAP server. >> >> >> As an aside, I would note that you appear to be indexing "pres" >> unnecessarily. Please read >> <https://www.openldap.org/doc/admin25/tuning.html#Presence%20indexing> >> >> If the group object is large you may be having slow searches due to indices >> being collapsed to a range. You would need to run the search with trace >> logging to determine if that's the case as was recently discussed on the >> list. >> >> Regards, >> Quanah >> > > Hi, > > We are also observing very long query time involving memberOf when using > dynlist migrating a 2.4 OpenLDAP docker install to 2.5.11. We have followed > community > recommendation and replaced the deprecated memberOf overlay with dynlist to > calculate memberOf attributes, and removed the memberof elements from the data > before importing it. > > Using dynlist, we have noticed an important drop of performance in queries > involving the memberOf attributes on (relatively) large database (about 36000 > users > and 9000 groups dispatched in 150 branches) compared to 2.4: Queries on > memberOf attributes that used to be processed in 0.200 seconds now takes > about 6-7 > seconds. In the example below, the search base is a branch with a relatively > small number of users (about 350) and the targeted group contains 260 users: > > time ldapsearch -x -h 172.17.0.2 -D "cn=admin,dc=domain,dc=com" -w "secret" > -b "ou=people,dc=branch,dc=domain,dc=com" > "(memberof=cn=dev,ou=groups,dc=branch,dc=domain,dc=com)" uid > > real 0m6,358s > user 0m0,024s > sys 0m0,020s > > If the group used in memberOf has 1200 members, the query takes over 30 > seconds. These results are obtained using an OpenLDAP running in a simple > docker > container on a host with 8CPU/16GB RAM. > > When using the memberOf overlay instead of dynlist to calculate memberOf, the > performance are back to what is used to be in 2.4 (< 0.200 second for the > above > ldapsearch). > > We believe our config to be very standard: member attribute is indexed and > dynlist is loaded and configured as below > > dn: olcOverlay={3}dynlist,olcDatabase={1}mdb,cn=config > objectClass: olcOverlayConfig > objectClass: olcDynListConfig > olcOverlay: {3}dynlist > olcDynListAttrSet: {0}groupOfURLs memberURL member+memberOf@groupOfNames > > Are this performance issues an expected side-effect of switching to dynlist - > as the memberOf attributes are now dynamically calculated while the memberOf > overlay used to writes these attributes - or something is more likely wrong > with our install or could be tuned to improve these performances?
That's just the cost of dynamically calculating these attributes, yes. It's possible we can make some other optimizations to the overlay to reduce the amount of work in calculating memberOf if you're only using it in a filter and not returning all of its values. Looks like it could be a fairly significant refactoring to achieve that though. > > Some organizations have applications that relies on memberOf queries to be > fast, and expects to keep these performances when upgrading to 2.5. From what > we > observed, we can only advise them to stick with the legacy memberOf overlay > if they need these queries to stay fast, or rethink their approach to avoid > querying > memberOf on large databases. Makes sense. > > Regards, > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
