[389-users] Re: Experiences with Large Groups (>100k Members)

[Ludwig Krispenz]

On 04/26/2018 08:06 PM, Fong, Trevor wrote:


Hi Everyone,

I was wondering what experiences people have had with large groups (> 
100k members) in 389 DS?


Particularly interested in loading, managing and syncing them.

WRT syncing – how do people efficiently sync large groups?  Most sync 
utilities sync at the attribute level; if the changed attribute (eg 
member) is multivalued, it just replaces all values.  That’s OK if 
there’s only a few values, but is not efficient when there are a large 
number of them.  A more efficient way would be to diff the 2 
attributes and add/delete the differences; does anyone know of any 
sync tools that do something like this?


389-ds does syncing/replicaten by replictaing the operation, so if you 
add one member only this modify/add operation will be replicated and 
applied. So the efficiency depends on your client application, if it 
always does a replace the the full group will be replicated, if it does 
add/del of single members the replication footprint is very small


Background:

I have a few particularly large groups of > 500k members that are 
currently handled in a DBMS, but want to migrate them to LDAP instead.


When I try to load them via ldapmodify, doing an add:member per member 
was going to take more than 24 hrs at rate of processing at the time 
of abort.


Trying to add all members instead, with a single add:member and 
listing all members after that instruction, eventually ended with an 
Operations Error.  Turning on housekeeping error level showed it was 
getting “Lock table is out of available lock entries” error – I’m in 
the process of retrying with adjusted nsslapd-db-locks in 
cn=config,cn=ldbm database,cn=plugins,cn=config.


Thanks,

Trev

_

*Trevor Fong*

/Senior Programmer Analyst/

Information Technology | Engage. Envision. Enable.

*The University of British Columbia*

trevor.f...@ubc.ca  | 1-604-827-5247 
 | it.ubc.ca 




___
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org


--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

___
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org

[389-users] Re: Experiences with Large Groups (>100k Members)

[Michael Ströder]

Michael Ströder wrote:
> Fong, Trevor wrote:
>> Just wondering what others have encountered with large groups and
>> syncing between LDAP <--> RDBMS / LDAP <--> LDAP.
> 
> Simply write a custom script in your favourite scripting language.
> It's not that hard.
> 
> If you're familiar with Python:
> 
> https://python-ldap.org/en/latest/reference/ldap-modlist.html#ldap.modlist.modifyModlist

Ah, sorry. The above function is not the smart variant doing
delete-by-value and add-by-value.

But it's really not that hard to achieve that.

Ciao, Michael.
___
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org

[389-users] Re: Experiences with Large Groups (>100k Members)

[Michael Ströder]

Fong, Trevor wrote:
> Just wondering what others have encountered with large groups and
> syncing between LDAP <--> RDBMS / LDAP <--> LDAP.

Simply write a custom script in your favourite scripting language.
It's not that hard.

If you're familiar with Python:

https://python-ldap.org/en/latest/reference/ldap-modlist.html#ldap.modlist.modifyModlist

Ciao, Michael.
___
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org

[389-users] Re: Experiences with Large Groups (>100k Members)

[Fong, Trevor]

Thanks very much for your reply Trevor.
Just to expand a bit on my bit about sync’ing – We’ve been running on 389 DS 
for about 5 years now, and it has been solid for the most part.
Our LDAP cluster (multi-master replicated providers, hubs, and consumers) is 
sync’d to from our RDBMS based identity service via LSC (lsc-project.org).
LSC is normally trouble free, but it doesn’t do group replication very well due 
to it treating multi-value attributes in a monolithic manner (so we don’t do 
group sync yet).

Just wondering what others have encountered with large groups and syncing 
between LDAP <--> RDBMS / LDAP <--> LDAP.

Thanks,
Trev

From: "Wendt, Trevor" <trevor.we...@blackhillscorp.com>
Reply-To: "389-users@lists.fedoraproject.org" 
<389-users@lists.fedoraproject.org>
Date: Thursday, April 26, 2018 at 1:35 PM
To: "389-users@lists.fedoraproject.org" <389-users@lists.fedoraproject.org>
Subject: [389-users] Re: Experiences with Large Groups (>100k Members)

Trev, Was going to suggest splitting and batching your imports, good call.
We have had over 650k and not had any issues with 389ds. Started back on 
fedora-ds and migrated along with changes. Replication (master/master) is 
solid, keeps up with day to day changes fine, very stable. Overall no issues 
for 10+ years using it with steady growth. We are moving away to another 
solution but by not because of 389ds.  Good luck. -Trevor


From: Fong, Trevor [mailto:trevor.f...@ubc.ca]
Sent: Thursday, April 26, 2018 2:08 PM
To: General discussion list for the 389 Directory server project. 
<389-users@lists.fedoraproject.org>
Subject: [389-users] Re: Experiences with Large Groups (>100k Members)

Just an update:
I was successful in loading 532k members into a group, in ~45 mins, via 
ldapmodify by segmenting the ldif into 5 separate add:member sections, of ~100k 
each.  I also set nsslapd-db-locks in cn=config,cn=ldbm 
database,cn=plugins,cn=config to 40 – not sure which made any difference.

Still interested in other people’s experience with large groups.

From: "Fong, Trevor" <trevor.f...@ubc.ca>
Reply-To: "389-users@lists.fedoraproject.org" 
<389-users@lists.fedoraproject.org>
Date: Thursday, April 26, 2018 at 11:06 AM
To: "389-users@lists.fedoraproject.org" <389-users@lists.fedoraproject.org>
Subject: [389-users] Experiences with Large Groups (>100k Members)

Hi Everyone,

I was wondering what experiences people have had with large groups (> 100k 
members) in 389 DS?
Particularly interested in loading, managing and syncing them.
WRT syncing – how do people efficiently sync large groups?  Most sync utilities 
sync at the attribute level; if the changed attribute (eg member) is 
multivalued, it just replaces all values.  That’s OK if there’s only a few 
values, but is not efficient when there are a large number of them.  A more 
efficient way would be to diff the 2 attributes and add/delete the differences; 
does anyone know of any sync tools that do something like this?

Background:
I have a few particularly large groups of > 500k members that are currently 
handled in a DBMS, but want to migrate them to LDAP instead.
When I try to load them via ldapmodify, doing an add:member per member was 
going to take more than 24 hrs at rate of processing at the time of abort.
Trying to add all members instead, with a single add:member and listing all 
members after that instruction, eventually ended with an Operations Error.  
Turning on housekeeping error level showed it was getting “Lock table is out of 
available lock entries” error – I’m in the process of retrying with adjusted 
nsslapd-db-locks in cn=config,cn=ldbm database,cn=plugins,cn=config.

Thanks,
Trev


_
Trevor Fong
Senior Programmer Analyst
Information Technology | Engage. Envision. Enable.
The University of British Columbia
trevor.f...@ubc.ca<mailto:trevor.f...@ubc.ca> | 
1-604-827-5247 | it.ubc.ca<http://it.ubc.ca>




This electronic message transmission contains information from Black Hills 
Corporation, its affiliate or subsidiary, which may be confidential or 
privileged. The information is intended to be for the use of the individual or 
entity named above. If you are not the intended recipient, be aware the 
disclosure, copying, distribution or use of the contents of this information is 
prohibited. If you received this electronic transmission in error, please reply 
to sender immediately; then delete this message without copying it or further 
reading.

___
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org

[389-users] Re: Experiences with Large Groups (>100k Members)

[Wendt, Trevor]

Trev, Was going to suggest splitting and batching your imports, good call.
We have had over 650k and not had any issues with 389ds. Started back on 
fedora-ds and migrated along with changes. Replication (master/master) is 
solid, keeps up with day to day changes fine, very stable. Overall no issues 
for 10+ years using it with steady growth. We are moving away to another 
solution but by not because of 389ds.  Good luck. -Trevor


From: Fong, Trevor [mailto:trevor.f...@ubc.ca]
Sent: Thursday, April 26, 2018 2:08 PM
To: General discussion list for the 389 Directory server project. 
<389-users@lists.fedoraproject.org>
Subject: [389-users] Re: Experiences with Large Groups (>100k Members)

Just an update:
I was successful in loading 532k members into a group, in ~45 mins, via 
ldapmodify by segmenting the ldif into 5 separate add:member sections, of ~100k 
each.  I also set nsslapd-db-locks in cn=config,cn=ldbm 
database,cn=plugins,cn=config to 40 – not sure which made any difference.

Still interested in other people’s experience with large groups.

From: "Fong, Trevor" <trevor.f...@ubc.ca>
Reply-To: "389-users@lists.fedoraproject.org" 
<389-users@lists.fedoraproject.org>
Date: Thursday, April 26, 2018 at 11:06 AM
To: "389-users@lists.fedoraproject.org" <389-users@lists.fedoraproject.org>
Subject: [389-users] Experiences with Large Groups (>100k Members)

Hi Everyone,

I was wondering what experiences people have had with large groups (> 100k 
members) in 389 DS?
Particularly interested in loading, managing and syncing them.
WRT syncing – how do people efficiently sync large groups?  Most sync utilities 
sync at the attribute level; if the changed attribute (eg member) is 
multivalued, it just replaces all values.  That’s OK if there’s only a few 
values, but is not efficient when there are a large number of them.  A more 
efficient way would be to diff the 2 attributes and add/delete the differences; 
does anyone know of any sync tools that do something like this?

Background:
I have a few particularly large groups of > 500k members that are currently 
handled in a DBMS, but want to migrate them to LDAP instead.
When I try to load them via ldapmodify, doing an add:member per member was 
going to take more than 24 hrs at rate of processing at the time of abort.
Trying to add all members instead, with a single add:member and listing all 
members after that instruction, eventually ended with an Operations Error.  
Turning on housekeeping error level showed it was getting “Lock table is out of 
available lock entries” error – I’m in the process of retrying with adjusted 
nsslapd-db-locks in cn=config,cn=ldbm database,cn=plugins,cn=config.

Thanks,
Trev


_
Trevor Fong
Senior Programmer Analyst
Information Technology | Engage. Envision. Enable.
The University of British Columbia
trevor.f...@ubc.ca<mailto:trevor.f...@ubc.ca> | 
1-604-827-5247 | it.ubc.ca<http://it.ubc.ca>




This electronic message transmission contains information from Black Hills 
Corporation, its affiliate or subsidiary, which may be confidential or 
privileged. The information is intended to be for the use of the individual or 
entity named above. If you are not the intended recipient, be aware the 
disclosure, copying, distribution or use of the contents of this information is 
prohibited. If you received this electronic transmission in error, please reply 
to sender immediately; then delete this message without copying it or further 
reading.
___
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org

[389-users] Re: Experiences with Large Groups (>100k Members)

[Fong, Trevor]

Just an update:
I was successful in loading 532k members into a group, in ~45 mins, via 
ldapmodify by segmenting the ldif into 5 separate add:member sections, of ~100k 
each.  I also set nsslapd-db-locks in cn=config,cn=ldbm 
database,cn=plugins,cn=config to 40 – not sure which made any difference.

Still interested in other people’s experience with large groups.

From: "Fong, Trevor" 
Reply-To: "389-users@lists.fedoraproject.org" 
<389-users@lists.fedoraproject.org>
Date: Thursday, April 26, 2018 at 11:06 AM
To: "389-users@lists.fedoraproject.org" <389-users@lists.fedoraproject.org>
Subject: [389-users] Experiences with Large Groups (>100k Members)

Hi Everyone,

I was wondering what experiences people have had with large groups (> 100k 
members) in 389 DS?
Particularly interested in loading, managing and syncing them.
WRT syncing – how do people efficiently sync large groups?  Most sync utilities 
sync at the attribute level; if the changed attribute (eg member) is 
multivalued, it just replaces all values.  That’s OK if there’s only a few 
values, but is not efficient when there are a large number of them.  A more 
efficient way would be to diff the 2 attributes and add/delete the differences; 
does anyone know of any sync tools that do something like this?

Background:
I have a few particularly large groups of > 500k members that are currently 
handled in a DBMS, but want to migrate them to LDAP instead.
When I try to load them via ldapmodify, doing an add:member per member was 
going to take more than 24 hrs at rate of processing at the time of abort.
Trying to add all members instead, with a single add:member and listing all 
members after that instruction, eventually ended with an Operations Error.  
Turning on housekeeping error level showed it was getting “Lock table is out of 
available lock entries” error – I’m in the process of retrying with adjusted 
nsslapd-db-locks in cn=config,cn=ldbm database,cn=plugins,cn=config.

Thanks,
Trev


_
Trevor Fong
Senior Programmer Analyst
Information Technology | Engage. Envision. Enable.
The University of British Columbia
trevor.f...@ubc.ca | 
1-604-827-5247 | it.ubc.ca

___
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org