Re: dynlist vs memberof performance issues

[Paul B. Henson]

On Tue, Jan 10, 2023 at 01:41:58PM +0100, Ondřej Kuzník wrote:

> The latest manpage update should make it clearer how dynamic *lists*
> differ from dynamic *groups*. And yes, no need to change config for
> groups.

Cool, thank you both for the clarification.

> If you can give the current 2.5/2.6 branch a go, or if you decide to
> wait until we get a call for testing out, feedback is always welcome.

Will do, thanks again...

Re: dynlist vs memberof performance issues

[Ondřej Kuzník]

On Mon, Jan 09, 2023 at 11:39:47AM -0800, Paul B. Henson wrote:
> We're currently using dynlist to add the memberOf attribute to users.
> One thing that was nice about the new dynlist implementation in 2.5 was
> that it allows searching on the dynamic memberOf attribute which we
> couldn't do in 2.4. Looking at the commit diff, there's one part that
> says:
> 
> "To enable filter evaluation on the dynamic list, the configuration must
> be changed to explicitly map the dynamic attributes to be filtered"
> 
> This just applies to groups created dynamically, right, not static
> objects that get dynamic attributes added? Our current config is:

The latest manpage update should make it clearer how dynamic *lists*
differ from dynamic *groups*. And yes, no need to change config for
groups.

If you can give the current 2.5/2.6 branch a go, or if you decide to
wait until we get a call for testing out, feedback is always welcome.

Regards,

-- 
Ondřej Kuzník
Senior Software Engineer
Symas Corporation   http://www.symas.com
Packaged, certified, and supported LDAP solutions powered by OpenLDAP

Re: dynlist vs memberof performance issues

[Shawn McKinney]

> On Jan 9, 2023, at 1:39 PM, Paul B. Henson  wrote:
> 
> On Mon, Jan 09, 2023 at 09:26:44AM -0600, Shawn McKinney wrote:
> 
>> Yes (both)
> 
> Sweet :). We've had some performance issues with the new dynlist
> implementation since we upgraded to 2.5, I look forward to trying this
> out.
> 
> We're currently using dynlist to add the memberOf attribute to users.
> One thing that was nice about the new dynlist implementation in 2.5 was
> that it allows searching on the dynamic memberOf attribute which we
> couldn't do in 2.4. Looking at the commit diff, there's one part that
> says:
> 
> "To enable filter evaluation on the dynamic list, the configuration must
> be changed to explicitly map the dynamic attributes to be filtered"
> 
> This just applies to groups created dynamically, right, not static
> objects that get dynamic attributes added? Our current config is:
> 
>   dynlist-attrset groupOfURLs memberURL member+memberOf@groupOfNames
> 
> It doesn't require any changes to keep working with a searchable
> memberOf attribute?
> 

Correct, no changes to the config of dynlist.

> Thanks much...

Re: dynlist vs memberof performance issues

[Paul B. Henson]

On Mon, Jan 09, 2023 at 09:26:44AM -0600, Shawn McKinney wrote:

> Yes (both)

Sweet :). We've had some performance issues with the new dynlist
implementation since we upgraded to 2.5, I look forward to trying this
out.

We're currently using dynlist to add the memberOf attribute to users.
One thing that was nice about the new dynlist implementation in 2.5 was
that it allows searching on the dynamic memberOf attribute which we
couldn't do in 2.4. Looking at the commit diff, there's one part that
says:

"To enable filter evaluation on the dynamic list, the configuration must
be changed to explicitly map the dynamic attributes to be filtered"

This just applies to groups created dynamically, right, not static
objects that get dynamic attributes added? Our current config is:

dynlist-attrset groupOfURLs memberURL member+memberOf@groupOfNames

It doesn't require any changes to keep working with a searchable
memberOf attribute?

Thanks much...

Re: dynlist vs memberof performance issues

[Shawn McKinney]

> On Jan 7, 2023, at 8:44 PM, Paul B. Henson  wrote:
> 
> On Tue, Jan 03, 2023 at 11:44:30AM -0600, Shawn McKinney wrote:
> 
>> Some work on the dynlist code of late to boost performance.
>> 
>> So, it might be worthwhile to give it a try. MR576[1] in the next release.
> 
> Is this going to hit the 2.5 LTS train or just 2.6?


Yes (both)

Re: dynlist vs memberof performance issues

[Paul B. Henson]

On Tue, Jan 03, 2023 at 11:44:30AM -0600, Shawn McKinney wrote:

> Some work on the dynlist code of late to boost performance.
> 
> So, it might be worthwhile to give it a try. MR576[1] in the next release.

Is this going to hit the 2.5 LTS train or just 2.6?

Thanks...

Re: dynlist vs memberof performance issues

[Shawn McKinney]

> On Jul 25, 2022, at 2:51 PM, David Coutadeur  
> wrote:
> 
> I have worked with Soisik on this particular topic.
> As you mentionned, in many cases we can just query directly the group.
> In our particular scenario, the software only allows one request for getting 
> the mails of the users present in the dynamic group.
> 
> Thus I think it would be great to have a server solution (if possible in 
> 2.6), especially if dynlist is officially the memberof replacement.

Hello,

Some work on the dynlist code of late to boost performance.

So, it might be worthwhile to give it a try. MR576[1] in the next release.

—
Shawn

[1] https://git.openldap.org/openldap/openldap/-/merge_requests/576

Re: dynlist vs memberof performance issues

[Paul B. Henson]

On 7/26/2022 1:56 AM, Wilkinson, Hugo (IT Dept) wrote:
If you have the ability to do so and your kernel is v5+, out-of-hours 
experiment with disabling system swap entirely (vm.swapiness=0 and 
'swapoff -a' )  and then simulate a run of requests you'd expect to 
encounter the page errors with and see if it stops happening.


We use to run our LDAP servers under Gentoo with a 5.4 kernel at the 
time, but currently are using Rocky Linux 8 which comes with a modified 
4.18 kernel.


I did actually run a production system with no swap for a while, it 
still occasionally had slow responses for queries requesting the 
memberOf attribute.


Part of the problem is I don't expect to ever see these slow response 
times 8-/. As of yet I have not figured out what is going on when they 
are slow, I definitely don't have a reproducible query test case to make 
it happen .


We were running 2.4 under rocky for a while before upgrading to 2.5 and 
never saw this problem, so I'm pretty sure it's the result of some 
change in the dynlist implementation.

Re: dynlist vs memberof performance issues

[Wilkinson, Hugo (IT Dept)]

If you have the ability to do so and your kernel is v5+, out-of-hours 
experiment with disabling system swap entirely (vm.swapiness=0 and 'swapoff -a' 
)  and then simulate a run of requests you'd expect to encounter the page 
errors with and see if it stops happening.

Depending on what else is going on this might not be a production-compatible 
strategy, but it will either rule that out as a potential flaw/bug in 2.5 
and/or give you a workaround until it's fixed.



From: Paul B. Henson 
Sent: 26 July 2022 00:16
To: openldap-technical@openldap.org 
Subject: Re: dynlist vs memberof performance issues

On 7/25/2022 10:38 AM, Shawn McKinney wrote:

> As you (and others) have pointed out, there's a significant
> performance penalty for searching attributes generated by dylist.

I'm still seeing performance issues with queries that simply return
memberOf, with no reference to it in the actual search filter.

For example, this query which searches on the static uid attribute and
returns memberOf:

time ldapsearch -H ldapi:/// uid=henson memberOf

Most of the time it completes in fractions of a second:

real0m0.187s
user0m0.005s
sys 0m0.003s

But sometimes it takes 5 seconds, 10 seconds, or even more. These
extremely slow response times coordinate with a high read I/O percentage
on the server and the high number of page faults on the slapd process.

When I first deployed 2.5, sometimes the server would get into a state
where every query that requested memberOf would take in excess of 30
seconds to return until the server was restarted. I cranked up the
memory on the servers and at this point I have had no more reoccurrences
of that behavior, but I am still regularly seeing occasional slow
performance on the queries and high read I/O percentages.

The servers have way more memory now than they should need to fully
cache the entire database:

# du -sh /var/symas/openldap-data-cpp
2.6G/var/symas/openldap-data-cpp

# free -m
   totalusedfree  shared  buff/cache
available
Mem:   4818 703 124   03991
3831
Swap:  2047 1361911

I haven't been able to correlate the slow response times with any other
external criteria such as updates or query load. Sometimes it's just
slow 8-/. We never saw this problem under 2.4 which used the previous
implementation of dynlist to generate memberOf. I definitely appreciate
the ability to query on dynamic memberOf that was added in 2.5, but it
would be nice to sort out this performance issue.




** Internet Email Confidentiality Footer **

Privileged/Confidential information may be contained in this message.
If you have received this message in error you should, without taking any 
copies, immediately delete it from your system and kindly notify the sender by 
reply email. Opinions, conclusions and other information in this message that 
do not relate to the official business of this company shall be understood as 
neither given nor endorsed by it. 
All contracts for goods and services are subject to the terms and conditions of 
our relevant company, a copy of which is available on request or by visiting 
the company’s website.

Amari Metals Ltd. Registered in England and Wales number 2023155.
Registered Office: Parkway House, Unit 6, Parkway Industrial Estate, Pacific 
Avenue, Wednesbury, WS10 7WP.

Re: dynlist vs memberof performance issues

[Mark Cairney]

Hi both,

Just to let you know that with my dataset I see a consistently slow
(10-20 secs) lookup of a user's memberOf group membership using dynamic
dynlist overlays on 2.5 and 2.6 compared with the old "memberof" overlay.

My dynlist config is as below:

dn: olcOverlay={2}dynlist,olcDatabase={1}mdb,cn=config
objectClass: olcOverlayConfig
objectClass: olcDynListConfig
olcOverlay: {2}dynlist
olcDynListAttrSet: {0}groupOfURLs memberURL member+memberOf@groupOfNames
structuralObjectClass: olcDynListConfig

It's fairly common to use the memberOf attribute to look up the groups a
particular user is a member of e.g. for access control purposes. We use
it routinely for this purpose on our Shibboleth IdPs which use OpenLDAP
as a datasource.

In terms of data size/complexity we've currently got around 500K users
(each with their own unique group in a separate OU) with the bulk of our
group membership managed by Grouper. We've got about 100K groups in
there(!) ranging in size from a single "placeholder" member up to about
200K members for our applicant/alumni groups.

The total DIT size (dumped via slapcat) is just under 2GB and our
(freshly slapcat'ed and slapadd'ed mdb database size is currently
sitting at 5.1GB). We're running on Centos 7 boxes with 12vCPUs and 64GB
RAM each which I think should be enough CPU/memory resource for our usage?

On a related note for a production environment should we be targeting
the 2.5 or 2.6 branch? I'd like to upgrade our production servers off
2.4 before the start of next term.


Kind regards,

Mark


On 26/07/2022 00:16, Paul B. Henson wrote:

This email was sent to you by someone outside the University.
You should only click on links or attachments if you are certain that
the email is genuine and the content is safe.

On 7/25/2022 10:38 AM, Shawn McKinney wrote:


As you (and others) have pointed out, there's a significant
performance penalty for searching attributes generated by dylist.


I'm still seeing performance issues with queries that simply return
memberOf, with no reference to it in the actual search filter.

For example, this query which searches on the static uid attribute and
returns memberOf:

time ldapsearch -H ldapi:/// uid=henson memberOf

Most of the time it completes in fractions of a second:

real0m0.187s
user0m0.005s
sys 0m0.003s

But sometimes it takes 5 seconds, 10 seconds, or even more. These
extremely slow response times coordinate with a high read I/O percentage
on the server and the high number of page faults on the slapd process.

When I first deployed 2.5, sometimes the server would get into a state
where every query that requested memberOf would take in excess of 30
seconds to return until the server was restarted. I cranked up the
memory on the servers and at this point I have had no more reoccurrences
of that behavior, but I am still regularly seeing occasional slow
performance on the queries and high read I/O percentages.

The servers have way more memory now than they should need to fully
cache the entire database:

# du -sh /var/symas/openldap-data-cpp
2.6G/var/symas/openldap-data-cpp

# free -m
  totalusedfree  shared buff/cache
available
Mem:   4818 703 124   0 3991
   3831
Swap:  2047 1361911

I haven't been able to correlate the slow response times with any other
external criteria such as updates or query load. Sometimes it's just
slow 8-/. We never saw this problem under 2.4 which used the previous
implementation of dynlist to generate memberOf. I definitely appreciate
the ability to query on dynamic memberOf that was added in 2.5, but it
would be nice to sort out this performance issue.


--
/

Mark Cairney
ITI Enterprise Services
Information Services
University of Edinburgh

Tel: 0131 650 6565
Email: mark.cair...@ed.ac.uk

***/

The University of Edinburgh is a charitable body, registered in Scotland, with 
registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh 
Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.

Re: dynlist vs memberof performance issues

[David Coutadeur]

Hello,

Le 26/07/2022 à 00:04, Howard Chu a écrit :

David Coutadeur wrote:

Hello,

I have worked with Soisik on this particular topic.
As you mentionned, in many cases we can just query directly the group.
In our particular scenario, the software only allows one request for getting 
the mails of the users present in the dynamic group.

Have you already tried using the deref control to return a mail attribute 
instead?


Thanks for the idea.

I suppose it could work indeed. I have not tested it yet, but it could 
be interresting to have the performance associated to such request.


However, in my specific use case, it is not possible to configure a LDAP 
control in the software.



Thus I think it would be great to have a server solution (if possible in 2.6), 
especially if dynlist is officially the memberof replacement.

Regards,

David



Le 25 juillet 2022 19:38:52 GMT+02:00, Shawn McKinney  a 
écrit :


 On May 23, 2022, at 3:10 AM, Soisik Froger  
wrote:

 I've just sent you sample slapd.conf and a data ldif that illustrate 
the long query time on large database. Thank you very much to take a look at 
this !


 Hi Soisik,

 I've completed a review/test of dynlist, following your excellent example 
as sent off list.

 Here are my findings…

 As a baseline, here’s an example of a 'normal' search across a well 
populated user tree:

 # time ldapsearch -x -LLL -H ldap://m01:389 -D "dc=example,dc=com" -W -s sub -b 
'ou=people,dc=example,dc=com' "(cn=load01-TEST1-*)" | grep dn: | wc -l
 17470

 real0m0.191s
 user0m0.128s
 sys 0m0.132s

 Compared with performing a search of the memberof attribute, generated by 
dynast. This pulls back the members of a large group (about 25K members):
 # time ldapsearch -x -LLL -H ldap://m01:389 -D "dc=example,dc=com" -W -s sub -b 
'ou=people,dc=example,dc=com' "(memberof=cn=load01-test1-2,ou=groups,dc=example,dc=com)" 
| grep dn: | wc -l
 23869
 real4m6.167s
 user0m2.289s
 sys 0m2.037s

 That's 4 minutes to search across 100K users. The other search took < 200 
ms.

 As you (and others) have pointed out, there's a significant performance 
penalty for searching attributes generated by dylist.

 It's possible that we can get modest improvements within a 2.7 timeframe. 
Unfortunately, we don't anticipate being able to get it back into range of a 
'normal' search.

 So, that gets us into looking at mitigation.  One approach, focus on what 
the client does. For example, instead of searching across the user tree for 
membership to a particular group, it's far more efficient to just pull back 
that list from the group entry itself.

 This is a technique that of course we’re already familiar with. I bring it 
up here for the sake of argument.

 The question: can we sidestep the memberOf performance problems by 
focusing on the the client?

 Or, is there a usecase that I've missed for which there's no remedy?

 Thanks

 —
 Shawn




--
David Coutadeur | IAM integrator

david.coutad...@worteks.com
+33 7 88 46 85 34
137 boulevard de Magenta, Paris 75010

Worteks | https://www.worteks.com

Re: dynlist vs memberof performance issues

[Paul B. Henson]

On 7/25/2022 10:38 AM, Shawn McKinney wrote:


As you (and others) have pointed out, there's a significant
performance penalty for searching attributes generated by dylist.


I'm still seeing performance issues with queries that simply return 
memberOf, with no reference to it in the actual search filter.


For example, this query which searches on the static uid attribute and 
returns memberOf:


time ldapsearch -H ldapi:/// uid=henson memberOf

Most of the time it completes in fractions of a second:

real0m0.187s
user0m0.005s
sys 0m0.003s

But sometimes it takes 5 seconds, 10 seconds, or even more. These 
extremely slow response times coordinate with a high read I/O percentage 
on the server and the high number of page faults on the slapd process.


When I first deployed 2.5, sometimes the server would get into a state 
where every query that requested memberOf would take in excess of 30 
seconds to return until the server was restarted. I cranked up the 
memory on the servers and at this point I have had no more reoccurrences 
of that behavior, but I am still regularly seeing occasional slow 
performance on the queries and high read I/O percentages.


The servers have way more memory now than they should need to fully 
cache the entire database:


# du -sh /var/symas/openldap-data-cpp
2.6G/var/symas/openldap-data-cpp

# free -m
  totalusedfree  shared  buff/cache 
available
Mem:   4818 703 124   03991 
   3831

Swap:  2047 1361911

I haven't been able to correlate the slow response times with any other 
external criteria such as updates or query load. Sometimes it's just 
slow 8-/. We never saw this problem under 2.4 which used the previous 
implementation of dynlist to generate memberOf. I definitely appreciate 
the ability to query on dynamic memberOf that was added in 2.5, but it 
would be nice to sort out this performance issue.

Re: dynlist vs memberof performance issues

[Howard Chu]

David Coutadeur wrote:
> Hello,
> 
> I have worked with Soisik on this particular topic.
> As you mentionned, in many cases we can just query directly the group.
> In our particular scenario, the software only allows one request for getting 
> the mails of the users present in the dynamic group.

Have you already tried using the deref control to return a mail attribute 
instead?
> 
> Thus I think it would be great to have a server solution (if possible in 
> 2.6), especially if dynlist is officially the memberof replacement.
> 
> Regards,
> 
> David
> 
> 
> 
> Le 25 juillet 2022 19:38:52 GMT+02:00, Shawn McKinney  a 
> écrit :
> 
> 
> On May 23, 2022, at 3:10 AM, Soisik Froger 
>  wrote:
> 
> I've just sent you sample slapd.conf and a data ldif that illustrate 
> the long query time on large database. Thank you very much to take a look at 
> this !
> 
> 
> Hi Soisik,
> 
> I've completed a review/test of dynlist, following your excellent example 
> as sent off list.
> 
> Here are my findings… 
> 
> As a baseline, here’s an example of a 'normal' search across a well 
> populated user tree:
> 
> # time ldapsearch -x -LLL -H ldap://m01:389 -D "dc=example,dc=com" -W -s 
> sub -b 'ou=people,dc=example,dc=com' "(cn=load01-TEST1-*)" | grep dn: | wc -l
> 17470
> 
> real0m0.191s
> user0m0.128s
> sys 0m0.132s
> 
> Compared with performing a search of the memberof attribute, generated by 
> dynast. This pulls back the members of a large group (about 25K members):
> # time ldapsearch -x -LLL -H ldap://m01:389 -D "dc=example,dc=com" -W -s 
> sub -b 'ou=people,dc=example,dc=com' 
> "(memberof=cn=load01-test1-2,ou=groups,dc=example,dc=com)" | grep dn: | wc -l
> 23869
> real4m6.167s
> user0m2.289s
> sys 0m2.037s
> 
> That's 4 minutes to search across 100K users. The other search took < 200 
> ms.
> 
> As you (and others) have pointed out, there's a significant performance 
> penalty for searching attributes generated by dylist.
> 
> It's possible that we can get modest improvements within a 2.7 timeframe. 
> Unfortunately, we don't anticipate being able to get it back into range of a 
> 'normal' search.
> 
> So, that gets us into looking at mitigation.  One approach, focus on what 
> the client does. For example, instead of searching across the user tree for 
> membership to a particular group, it's far more efficient to just pull back 
> that list from the group entry itself.
> 
> This is a technique that of course we’re already familiar with. I bring 
> it up here for the sake of argument. 
> 
> The question: can we sidestep the memberOf performance problems by 
> focusing on the the client?
> 
> Or, is there a usecase that I've missed for which there's no remedy?
> 
> Thanks
> 
> —
> Shawn
> 


-- 
  -- Howard Chu
  CTO, Symas Corp.   http://www.symas.com
  Director, Highland Sun http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

Re: dynlist vs memberof performance issues

[David Coutadeur]

Hello,

I have worked with Soisik on this particular topic.
As you mentionned, in many cases we can just query directly the group.
In our particular scenario, the software only allows one request for getting 
the mails of the users present in the dynamic group.

Thus I think it would be great to have a server solution (if possible in 2.6), 
especially if dynlist is officially the memberof replacement.

Regards,

David



Le 25 juillet 2022 19:38:52 GMT+02:00, Shawn McKinney  a 
écrit :
>
>> On May 23, 2022, at 3:10 AM, Soisik Froger  wrote:
>> 
>> I've just sent you sample slapd.conf and a data ldif that illustrate the 
>> long query time on large database. Thank you very much to take a look at 
>> this !
>
>Hi Soisik,
>
>I've completed a review/test of dynlist, following your excellent example as 
>sent off list.
>
>Here are my findings… 
>
>As a baseline, here’s an example of a 'normal' search across a well populated 
>user tree:
>
># time ldapsearch -x -LLL -H ldap://m01:389 -D "dc=example,dc=com" -W -s sub 
>-b 'ou=people,dc=example,dc=com' "(cn=load01-TEST1-*)" | grep dn: | wc -l
>17470
>
>real0m0.191s
>user0m0.128s
>sys 0m0.132s
>
>Compared with performing a search of the memberof attribute, generated by 
>dynast. This pulls back the members of a large group (about 25K members):
># time ldapsearch -x -LLL -H ldap://m01:389 -D "dc=example,dc=com" -W -s sub 
>-b 'ou=people,dc=example,dc=com' 
>"(memberof=cn=load01-test1-2,ou=groups,dc=example,dc=com)" | grep dn: | wc -l
>23869
>real4m6.167s
>user0m2.289s
>sys 0m2.037s
>
>That's 4 minutes to search across 100K users. The other search took < 200 ms.
>
>As you (and others) have pointed out, there's a significant performance 
>penalty for searching attributes generated by dylist.
>
>It's possible that we can get modest improvements within a 2.7 timeframe. 
>Unfortunately, we don't anticipate being able to get it back into range of a 
>'normal' search.
>
>So, that gets us into looking at mitigation.  One approach, focus on what the 
>client does. For example, instead of searching across the user tree for 
>membership to a particular group, it's far more efficient to just pull back 
>that list from the group entry itself.
>
>This is a technique that of course we’re already familiar with. I bring it up 
>here for the sake of argument. 
>
>The question: can we sidestep the memberOf performance problems by focusing on 
>the the client?
>
>Or, is there a usecase that I've missed for which there's no remedy?
>
>Thanks
>
>—
>Shawn

Re: dynlist vs memberof performance issues

[Shawn McKinney]

> On May 23, 2022, at 3:10 AM, Soisik Froger  wrote:
> 
> I've just sent you sample slapd.conf and a data ldif that illustrate the long 
> query time on large database. Thank you very much to take a look at this !

Hi Soisik,

I've completed a review/test of dynlist, following your excellent example as 
sent off list.

Here are my findings… 

As a baseline, here’s an example of a 'normal' search across a well populated 
user tree:

# time ldapsearch -x -LLL -H ldap://m01:389 -D "dc=example,dc=com" -W -s sub -b 
'ou=people,dc=example,dc=com' "(cn=load01-TEST1-*)" | grep dn: | wc -l
17470

real0m0.191s
user0m0.128s
sys 0m0.132s

Compared with performing a search of the memberof attribute, generated by 
dynast. This pulls back the members of a large group (about 25K members):
# time ldapsearch -x -LLL -H ldap://m01:389 -D "dc=example,dc=com" -W -s sub -b 
'ou=people,dc=example,dc=com' 
"(memberof=cn=load01-test1-2,ou=groups,dc=example,dc=com)" | grep dn: | wc -l
23869
real4m6.167s
user0m2.289s
sys 0m2.037s

That's 4 minutes to search across 100K users. The other search took < 200 ms.

As you (and others) have pointed out, there's a significant performance penalty 
for searching attributes generated by dylist.

It's possible that we can get modest improvements within a 2.7 timeframe. 
Unfortunately, we don't anticipate being able to get it back into range of a 
'normal' search.

So, that gets us into looking at mitigation.  One approach, focus on what the 
client does. For example, instead of searching across the user tree for 
membership to a particular group, it's far more efficient to just pull back 
that list from the group entry itself.

This is a technique that of course we’re already familiar with. I bring it up 
here for the sake of argument. 

The question: can we sidestep the memberOf performance problems by focusing on 
the the client?

Or, is there a usecase that I've missed for which there's no remedy?

Thanks

—
Shawn

Re: Antw: [EXT] Re: dynlist vs memberof performance issues

[Ulrich Windl]

>>> Howard Chu  schrieb am 23.05.2022 um 18:04 in Nachricht
:
> Ulrich Windl wrote:
> "Paul B. Henson"  schrieb am 22.05.2022 um 04:51 in 
> Nachricht
>> <5d343067-aef3-b499-63e3-996f3d680...@acm.org>:
>>> On 5/11/2022 3:48 AM, Soisik Froger wrote:
>>>
 Are this performance issues an expected side-effect of switching to 
 dynlist - as the memberOf attributes are now dynamically calculated 
 while the memberOf overlay used to writes these attributes - or 
>>>
>>> I am also having ongoing sporadic issues with memberOf performance using 
>>> the new dynlist overlay. Initially, randomly a server would get into a 
>>> state where any query requesting the memberOf attribute would take in 
>>> excess of 30 seconds, whereas normally it would only take a fraction of 
>>> a second. The symptoms were the same, free memory, no swapping, but 
>>> insanely high read IO load.
>> 
>> I'm wondering: If you'd make a core dump when the issue happens, how big 
> would such a core dump be with MDB?
>> I'm afraid it would be insanely large, containing the whole database. Am I 
> wrong?
> 
> Yes, you are wrong.
> 
> mmap'd memory isn't included in core dumps by default.

OK, but that makes it hard to debug any MDB issues from the core dump unless 
significant parts are copied in to private memory (which I think is not the 
idea behind MDB).

> 
> -- 
>   -- Howard Chu
>   CTO, Symas Corp.   http://www.symas.com 
>   Director, Highland Sun http://highlandsun.com/hyc/ 
>   Chief Architect, OpenLDAP  http://www.openldap.org/project/ 

Re: Antw: [EXT] Re: dynlist vs memberof performance issues

[Howard Chu]

Ulrich Windl wrote:
 Howard Chu  schrieb am 23.05.2022 um 18:04 in Nachricht
> :
>> Ulrich Windl wrote:
>> "Paul B. Henson"  schrieb am 22.05.2022 um 04:51 in 
>> Nachricht
>>> <5d343067-aef3-b499-63e3-996f3d680...@acm.org>:
 On 5/11/2022 3:48 AM, Soisik Froger wrote:

> Are this performance issues an expected side-effect of switching to 
> dynlist - as the memberOf attributes are now dynamically calculated 
> while the memberOf overlay used to writes these attributes - or 

 I am also having ongoing sporadic issues with memberOf performance using 
 the new dynlist overlay. Initially, randomly a server would get into a 
 state where any query requesting the memberOf attribute would take in 
 excess of 30 seconds, whereas normally it would only take a fraction of 
 a second. The symptoms were the same, free memory, no swapping, but 
 insanely high read IO load.
>>>
>>> I'm wondering: If you'd make a core dump when the issue happens, how big 
>> would such a core dump be with MDB?
>>> I'm afraid it would be insanely large, containing the whole database. Am I 
>> wrong?
>>
>> Yes, you are wrong.
>>
>> mmap'd memory isn't included in core dumps by default.
> 
> OK, but that makes it hard to debug any MDB issues from the core dump unless 
> significant parts are copied in to private memory (which I think is not the 
> idea behind MDB).
> 
Thanks for your concern.


-- 
  -- Howard Chu
  CTO, Symas Corp.   http://www.symas.com
  Director, Highland Sun http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

Re: Antw: [EXT] Re: dynlist vs memberof performance issues

[Howard Chu]

Ulrich Windl wrote:
 "Paul B. Henson"  schrieb am 22.05.2022 um 04:51 in 
 Nachricht
> <5d343067-aef3-b499-63e3-996f3d680...@acm.org>:
>> On 5/11/2022 3:48 AM, Soisik Froger wrote:
>>
>>> Are this performance issues an expected side-effect of switching to 
>>> dynlist - as the memberOf attributes are now dynamically calculated 
>>> while the memberOf overlay used to writes these attributes - or 
>>
>> I am also having ongoing sporadic issues with memberOf performance using 
>> the new dynlist overlay. Initially, randomly a server would get into a 
>> state where any query requesting the memberOf attribute would take in 
>> excess of 30 seconds, whereas normally it would only take a fraction of 
>> a second. The symptoms were the same, free memory, no swapping, but 
>> insanely high read IO load.
> 
> I'm wondering: If you'd make a core dump when the issue happens, how big 
> would such a core dump be with MDB?
> I'm afraid it would be insanely large, containing the whole database. Am I 
> wrong?

Yes, you are wrong.

mmap'd memory isn't included in core dumps by default.

-- 
  -- Howard Chu
  CTO, Symas Corp.   http://www.symas.com
  Director, Highland Sun http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

Antw: [EXT] Re: dynlist vs memberof performance issues

[Ulrich Windl]

>>> "Paul B. Henson"  schrieb am 22.05.2022 um 04:51 in 
>>> Nachricht
<5d343067-aef3-b499-63e3-996f3d680...@acm.org>:
> On 5/11/2022 3:48 AM, Soisik Froger wrote:
> 
>> Are this performance issues an expected side-effect of switching to 
>> dynlist - as the memberOf attributes are now dynamically calculated 
>> while the memberOf overlay used to writes these attributes - or 
> 
> I am also having ongoing sporadic issues with memberOf performance using 
> the new dynlist overlay. Initially, randomly a server would get into a 
> state where any query requesting the memberOf attribute would take in 
> excess of 30 seconds, whereas normally it would only take a fraction of 
> a second. The symptoms were the same, free memory, no swapping, but 
> insanely high read IO load.

I'm wondering: If you'd make a core dump when the issue happens, how big would 
such a core dump be with MDB?
I'm afraid it would be insanely large, containing the whole database. Am I 
wrong?

> 
> I cranked up the memory, which not did resolve the issue, but did help, 
> it doesn't happen nearly as often. But still, every now and again, a 
> server demonstrates a high read IO rate and severely degraded memberOf 
> query performance. At this point, I just have a monitoring check that 
> alerts on slow query performance and high read I/O and go restart them 
> when it happens, as the additional memory made the issue go from a 
> couple of times a week to every month or three.
> 
> I did notice now that when the issue occurs the box with the slow 
> queries does have less memory available then when it is working 
> normally, but still a good gigabyte of free memory not being used.
> 
> Even when the systems don't completely blow up, there are occasional 
> slower than normal queries. Typically the test query I am doing 
> literally takes fractions of a second:
> 
> May 21 19:47:22 ldap-01 slapd[1223]: conn=849157 op=1 SEARCH RESULT 
> tag=101 err=0 qtime=0.15 etime=0.198042 nentries=1 text=
> 
> Every now and again for no discernible reason it might take 5 to 10 seconds.

Re: dynlist vs memberof performance issues

[Paul B. Henson]

On 5/11/2022 3:48 AM, Soisik Froger wrote:

Are this performance issues an expected side-effect of switching to 
dynlist - as the memberOf attributes are now dynamically calculated 
while the memberOf overlay used to writes these attributes - or 


I am also having ongoing sporadic issues with memberOf performance using 
the new dynlist overlay. Initially, randomly a server would get into a 
state where any query requesting the memberOf attribute would take in 
excess of 30 seconds, whereas normally it would only take a fraction of 
a second. The symptoms were the same, free memory, no swapping, but 
insanely high read IO load.


I cranked up the memory, which not did resolve the issue, but did help, 
it doesn't happen nearly as often. But still, every now and again, a 
server demonstrates a high read IO rate and severely degraded memberOf 
query performance. At this point, I just have a monitoring check that 
alerts on slow query performance and high read I/O and go restart them 
when it happens, as the additional memory made the issue go from a 
couple of times a week to every month or three.


I did notice now that when the issue occurs the box with the slow 
queries does have less memory available then when it is working 
normally, but still a good gigabyte of free memory not being used.


Even when the systems don't completely blow up, there are occasional 
slower than normal queries. Typically the test query I am doing 
literally takes fractions of a second:


May 21 19:47:22 ldap-01 slapd[1223]: conn=849157 op=1 SEARCH RESULT 
tag=101 err=0 qtime=0.15 etime=0.198042 nentries=1 text=


Every now and again for no discernible reason it might take 5 to 10 seconds.

Re: dynlist vs memberof performance issues

[Shawn McKinney]

> On May 11, 2022, at 5:48 AM, Soisik Froger  wrote:
> 
> If the group used in memberOf has 1200 members, the query takes over 30 
> seconds. These results are obtained using an OpenLDAP running in a simple 
> docker container on a host with 8CPU/16GB RAM.
> 
> When using the memberOf overlay instead of dynlist to calculate memberOf, the 
> performance are back to what is used to be in 2.4 (< 0.200 second for the 
> above ldapsearch).
> 
> We believe our config to be very standard: member attribute is indexed and 
> dynlist is loaded and configured as below
> 
> dn: olcOverlay={3}dynlist,olcDatabase={1}mdb,cn=config
> objectClass: olcOverlayConfig
> objectClass: olcDynListConfig
> olcOverlay: {3}dynlist
> olcDynListAttrSet: {0}groupOfURLs memberURL member+memberOf@groupOfNames
> 


Hi Soisik,

I’d like to recreate this in a test env in order to analyze and look for 
solutions.  Can you send me your (complete) test server config and specifics on 
the tests, like ldif, sample data and operations?

Thanks

—
Shawn

Re: dynlist vs memberof performance issues

[Howard Chu]

Soisik Froger wrote:
> On 01/09/2021 17:17, Quanah Gibson-Mount wrote:
>>
>>
>> --On Wednesday, September 1, 2021 2:07 PM +0100 Mark Cairney
>>  wrote:
>>
>>> Hi,
>>>
>>> I've been out the LDAP loop for a bit but the recent discussion of the
>>> memberof overlay on 2.5 piqued my curiosity. Having upgraded a Dev box,
>>> removed the memberof elements from the database and replaced the
>>> memberof overlay with dynlist the queries appear to work as expected but
>>> are both a) slow and b) heavily CPU-intensive on the LDAP server.
>>
>>
>> As an aside, I would note that you appear to be indexing "pres"
>> unnecessarily.  Please read
>> 
>>
>> If the group object is large you may be having slow searches due to indices
>> being collapsed to a range.  You would need to run the search with trace
>> logging to determine if that's the case as was recently discussed on the
>> list.
>>
>> Regards,
>> Quanah
>>
> 
> Hi,
> 
> We are also observing very long query time involving memberOf when using 
> dynlist migrating a 2.4 OpenLDAP docker install to 2.5.11. We have followed 
> community
> recommendation and replaced the deprecated memberOf overlay with dynlist to 
> calculate memberOf attributes, and removed the memberof elements from the data
> before importing it.
> 
> Using dynlist, we have noticed an important drop of performance in queries 
> involving the memberOf attributes on (relatively) large database (about 36000 
> users
> and 9000 groups dispatched in 150 branches) compared to 2.4: Queries on 
> memberOf attributes that used to be processed in 0.200 seconds now takes 
> about 6-7
> seconds. In the example below, the search base is a branch with a relatively 
> small number of users (about 350) and the targeted group contains 260 users:
> 
> time ldapsearch -x -h 172.17.0.2 -D "cn=admin,dc=domain,dc=com" -w "secret" 
> -b "ou=people,dc=branch,dc=domain,dc=com"
> "(memberof=cn=dev,ou=groups,dc=branch,dc=domain,dc=com)" uid
> 
> real    0m6,358s
> user    0m0,024s
> sys    0m0,020s
> 
> If the group used in memberOf has 1200 members, the query takes over 30 
> seconds. These results are obtained using an OpenLDAP running in a simple 
> docker
> container on a host with 8CPU/16GB RAM.
> 
> When using the memberOf overlay instead of dynlist to calculate memberOf, the 
> performance are back to what is used to be in 2.4 (< 0.200 second for the 
> above
> ldapsearch).
> 
> We believe our config to be very standard: member attribute is indexed and 
> dynlist is loaded and configured as below
> 
> dn: olcOverlay={3}dynlist,olcDatabase={1}mdb,cn=config
> objectClass: olcOverlayConfig
> objectClass: olcDynListConfig
> olcOverlay: {3}dynlist
> olcDynListAttrSet: {0}groupOfURLs memberURL member+memberOf@groupOfNames
> 
> Are this performance issues an expected side-effect of switching to dynlist - 
> as the memberOf attributes are now dynamically calculated while the memberOf
> overlay used to writes these attributes - or something is more likely wrong 
> with our install or could be tuned to improve these performances?

That's just the cost of dynamically calculating these attributes, yes.

It's possible we can make some other optimizations to the overlay to reduce the 
amount of work in calculating memberOf if you're only
using it in a filter and not returning all of its values. Looks like it could 
be a fairly significant refactoring to achieve that though.
> 
> Some organizations have applications that relies on memberOf queries to be 
> fast, and expects to keep these performances when upgrading to 2.5. From what 
> we
> observed, we can only advise them to stick with the legacy memberOf overlay 
> if they need these queries to stay fast, or rethink their approach to avoid 
> querying
> memberOf on large databases.

Makes sense.
> 
> Regards,
> 


-- 
  -- Howard Chu
  CTO, Symas Corp.   http://www.symas.com
  Director, Highland Sun http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

Re: dynlist vs memberof performance issues

[Soisik Froger]

On 01/09/2021 17:17, Quanah Gibson-Mount wrote:



--On Wednesday, September 1, 2021 2:07 PM +0100 Mark Cairney
 wrote:


Hi,

I've been out the LDAP loop for a bit but the recent discussion of the
memberof overlay on 2.5 piqued my curiosity. Having upgraded a Dev box,
removed the memberof elements from the database and replaced the
memberof overlay with dynlist the queries appear to work as expected but
are both a) slow and b) heavily CPU-intensive on the LDAP server.



As an aside, I would note that you appear to be indexing "pres"
unnecessarily.  Please read


If the group object is large you may be having slow searches due to indices
being collapsed to a range.  You would need to run the search with trace
logging to determine if that's the case as was recently discussed on the
list.

Regards,
Quanah



Hi,

We are also observing very long query time involving memberOf when using 
dynlist migrating a 2.4 OpenLDAP docker install to 2.5.11. We have followed 
community recommendation and replaced the deprecated memberOf overlay with 
dynlist to calculate memberOf attributes, and removed the memberof elements 
from the data before importing it.

Using dynlist, we have noticed an important drop of performance in queries 
involving the memberOf attributes on (relatively) large database (about 36000 
users and 9000 groups dispatched in 150 branches) compared to 2.4: Queries on 
memberOf attributes that used to be processed in 0.200 seconds now takes about 
6-7 seconds. In the example below, the search base is a branch with a 
relatively small number of users (about 350) and the targeted group contains 
260 users:

time ldapsearch -x -h 172.17.0.2 -D "cn=admin,dc=domain,dc=com" -w "secret" -b 
"ou=people,dc=branch,dc=domain,dc=com" "(memberof=cn=dev,ou=groups,dc=branch,dc=domain,dc=com)" uid

real0m6,358s
user0m0,024s
sys 0m0,020s

If the group used in memberOf has 1200 members, the query takes over 30 
seconds. These results are obtained using an OpenLDAP running in a simple 
docker container on a host with 8CPU/16GB RAM.

When using the memberOf overlay instead of dynlist to calculate memberOf, the 
performance are back to what is used to be in 2.4 (< 0.200 second for the above 
ldapsearch).

We believe our config to be very standard: member attribute is indexed and 
dynlist is loaded and configured as below

dn: olcOverlay={3}dynlist,olcDatabase={1}mdb,cn=config
objectClass: olcOverlayConfig
objectClass: olcDynListConfig
olcOverlay: {3}dynlist
olcDynListAttrSet: {0}groupOfURLs memberURL member+memberOf@groupOfNames

Are this performance issues an expected side-effect of switching to dynlist - 
as the memberOf attributes are now dynamically calculated while the memberOf 
overlay used to writes these attributes - or something is more likely wrong 
with our install or could be tuned to improve these performances?

Some organizations have applications that relies on memberOf queries to be 
fast, and expects to keep these performances when upgrading to 2.5. From what 
we observed, we can only advise them to stick with the legacy memberOf overlay 
if they need these queries to stay fast, or rethink their approach to avoid 
querying memberOf on large databases.

Regards,

--
Soisik Froger

Worteks | https://www.worteks.com

Re: dynlist vs memberof performance issues

[Quanah Gibson-Mount]

--On Wednesday, September 1, 2021 2:07 PM +0100 Mark Cairney 
 wrote:



Hi,

I've been out the LDAP loop for a bit but the recent discussion of the
memberof overlay on 2.5 piqued my curiosity. Having upgraded a Dev box,
removed the memberof elements from the database and replaced the
memberof overlay with dynlist the queries appear to work as expected but
are both a) slow and b) heavily CPU-intensive on the LDAP server.



As an aside, I would note that you appear to be indexing "pres" 
unnecessarily.  Please read 



If the group object is large you may be having slow searches due to indices 
being collapsed to a range.  You would need to run the search with trace 
logging to determine if that's the case as was recently discussed on the 
list.


Regards,
Quanah

--

Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:

dynlist vs memberof performance issues

[Mark Cairney]

Hi,

I've been out the LDAP loop for a bit but the recent discussion of the
memberof overlay on 2.5 piqued my curiosity. Having upgraded a Dev box,
removed the memberof elements from the database and replaced the
memberof overlay with dynlist the queries appear to work as expected but
are both a) slow and b) heavily CPU-intensive on the LDAP server.

2021-09-01T12:47:17.603513+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 fd=12 ACCEPT from IP=192.168.152.33:58738
(IP=129.215.17.9:636)
2021-09-01T12:47:17.687488+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 fd=12 TLS established tls_ssf=256 ssf=256
tls_proto=TLSv1.3 tls_cipher=TLS_AES_256_GCM_SHA384
2021-09-01T12:47:17.688032+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=0 SRCH base="" scope=0 deref=0
filter="(objectClass=*)"
2021-09-01T12:47:17.688470+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=0 SRCH attr=supportedSASLMechanisms
2021-09-01T12:47:17.688878+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=0 SEARCH RESULT tag=101 err=0 qtime=0.14
etime=0.000214 nentries=1 text=
2021-09-01T12:47:17.811279+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=1 BIND dn="" method=163
2021-09-01T12:47:17.819249+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=1 RESULT tag=97 err=14 qtime=0.30
etime=0.009084 text=SASL(0): successful result:
2021-09-01T12:47:17.908889+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=2 BIND dn="" method=163
2021-09-01T12:47:17.909836+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=2 RESULT tag=97 err=14 qtime=0.31
etime=0.000181 text=SASL(0): successful result:
2021-09-01T12:47:17.938839+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=3 BIND dn="" method=163
2021-09-01T12:47:17.939621+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=3 BIND authcid="mcair...@ease.ed.ac.uk"
authzid="mcair...@ease.ed.ac.uk"
2021-09-01T12:47:17.940213+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=3 BIND
dn="uid=mcairney,ou=people,ou=central,dc=authorise-dev,dc=ed,dc=ac,dc=uk"
mech=GSSAPI bind_ssf=256 ssf=256
2021-09-01T12:47:17.940616+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=3 RESULT tag=97 err=0 qtime=0.24
etime=0.000409 text=
2021-09-01T12:47:18.227342+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=4 SRCH
base="dc=authorise-dev,dc=ed,dc=ac,dc=uk" scope=2 deref=0
filter="(uid=mcairney)"
2021-09-01T12:47:18.227703+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=4 SRCH attr=* +
2021-09-01T12:47:31.392255+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=5 UNBIND
2021-09-01T12:47:31.460705+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=4 SEARCH RESULT tag=101 err=0 qtime=0.31
etime=13.233679 nentries=1 text=
2021-09-01T12:47:31.461098+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 fd=12 closed

I'm guessing that as the values are computed that this will be heavier
on the CPU but it seems a bit excessive? Has anyone else noticed any
similar performance issues?

This is a relatively low-specced DEV server (2 vCPUs, 4GB RAM) so I
guess this could be a factor but there's no io waiting on the server and
no swapping?

The database is on a par in size with our Production service ( about
400K user objects with 1 group object per user and then about 80K actual
groups of users)

The config for the primary DB (ACLs and rootPW redacted) is:

dn: olcDatabase={1}mdb,cn=config
objectClass: olcDatabaseConfig
objectClass: olcMdbConfig
olcDatabase: {1}mdb
olcDbDirectory: /opt/openldap/var/openldap-data/authorise
olcSuffix: dc=authorise-dev,dc=ed,dc=ac,dc=uk

olcAddContentAcl: FALSE
olcLastMod: TRUE
olcMaxDerefDepth: 2
olcReadOnly: FALSE
olcSecurity: ssf=1
olcSecurity: update_ssf=112
olcSecurity: simple_bind=64
olcSizeLimit: unlimited
olcSyncUseSubentry: FALSE
olcTimeLimit: unlimited
olcMonitoring: TRUE
olcDbEnvFlags: writemap
olcDbEnvFlags: nometasync
olcDbNoSync: FALSE
olcDbIndex: objectClass eq
olcDbIndex: entryUUID eq
olcDbIndex: entryCSN eq
olcDbIndex: cn pres,eq,sub
olcDbIndex: uid pres,eq,sub
olcDbIndex: uidNumber pres,eq
olcDbIndex: gidNumber pres,eq
olcDbIndex: eduniType eq
olcDbIndex: gecos pres,eq,sub
olcDbIndex: eduniCategory eq
olcDbIndex: mail pres,eq,sub
olcDbIndex: eduniSchoolCode eq
olcDbIndex: eduniIDStatus eq
olcDbIndex: eduniCollegeCode eq
olcDbIndex: eduniOrgCode eq
olcDbIndex: memberOf pres,eq
olcDbIndex: eduniLibraryBarcode pres,eq
olcDbIndex: eduniOrganisation pres,eq,sub
olcDbIndex: eduniServiceCode pres,eq
olcDbIndex: krbName pres,eq
olcDbIndex: eduPersonAffiliation pres,eq
olcDbIndex: eduPersonEntitlement pres,eq
olcDbIndex: sn pres,eq,sub
olcDbIndex: eduniIdmsId pres,eq
olcDbIndex: member pres,eq
olcDbIndex: memberUid pres,eq
olcDbIndex: eduniRefNo pres,eq
olcDbIndex: eduniTitle pres,eq
olcDbIndex: title