Searches with dereferncing causing high CPU load.

[Mark Cairney]

Hi,

We're having severe performance issues for any query with alias
dereferencing set to "always".

Any query with this causes the CPU to spin up to 100% and if we have a
number of these concurrently the machine will become unresponsive.

We're using OpenLDAP 2.4.42 with the old hdb backend.

We do have a large number of aliases (~63,000). Could this be the cause?

Our olcMaxDerefDepth is currently set to "1"

-- 
/

Mark Cairney
ITI UNIX Section
Information Services
University of Edinburgh

Tel: 0131 650 6565
Email: mark.cair...@ed.ac.uk
PGP: 0x435A9621

***/

The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.



signature.asc
Description: OpenPGP digital signature

Re: OpenLDAP installation. Am I missing something?

[David Gabriel]

Hi Sherman,

I am also beginner ...
I advise you to use this tutorial

.

Regards,

2015-11-13 15:29 GMT+01:00 Sherman Lilly :

> I may have this totally wrong but why is there no installation
> documentation that tells somebody how to setup OpenLDAP the right way.
> After installing OpenLDAP you have no slapd.conf file so that direction is
> not happening. If you modify any file in the slapd.d directory, startup
> will complain about bad checksum. Yes I know you can regenerate the
> checksum and fix that but why? I can't find any where that tells you how to
> modify the base dn, rootdn, and root password without editing the files in
> the slapd.d manually. Am I missing something? I have check Google, Youtube,
> and other places and they all say manually edit files in slapd.d. That
> can't be the right way if openldap server is complaining about doing it.
>
> Sherman Lilly
>
>

Integrate Openldap and Windows Active Directory Server

[Kaushal Shriyan]

Hi,

Is there a way to integrate Openldap ldap server with Windows Server Active
Directory wherein AD will act as Authentication and Openldap will be setup
for Authorization?

Any help will be highly appreciable.

Regards,

Kaushal

Re: LMDB file size, again

[Christian Sell]

well, beyond the requirement about copyability of the file, there is also the
requirement that the app coexist well with other processes that are using the
disk and not grab all disk space while running. And then, there is the
requirement that the database should still be able to grow up to the limit of
the available resources if need be.

This is a desktop app, and the database can be looked upon as a "project file".
If user copies it in a inconsistent state, it's their problem.

I have decided to implement a database growth algorithm. There is an initial map
size and a threshhold. Whenever a write transaction is started, I check that
enough space is available and enlarge the map if needed. The app may also also
request more space for a transaction if known beforehand. If the transaction
hits the MAP_FULL anyway, bad luck.

regards + thanks,
Chris

> Klaus Malorny  hat am 16. November 2015 um 11:04
> geschrieben:
>
>
> On 11.11.2015 23:51, Christian Sell wrote:
> >> You're being stupid.
> >
> > let's say I am stubborn. As I mentioned, I have the requirement that the
> > user be
> > able to copy the database file away on their USB stick (or send it via
> > email,
> > for that matter). With SQLite, that is no problem. But with LMDB under
> > Windows
> > the user is looking at a huge file, which will be unacceptable in all
> > likelihood. I also must allow the user to copy using file system tools, and
> > not
> > some custom utility.
> >
> > So I take it that LMDB is simply not designed or suitable for such a
> > scenario.
> >
>
> Maybe you should consider to write your own exporter/importer. It shouldn't be
> too hard to walk through all key/value pairs and write out the data (prepended
> by their respective lengths), plus some metadata at the beginning to recognize
> the format on import and to know the number of records (if there is no special
> end-of-file marker).
>
> Just my two cents.
>
> Klaus
>

Re: Searches with dereferncing causing high CPU load.

[Andrew Findlay]

On Mon, Nov 16, 2015 at 03:13:11PM +, Mark Cairney wrote:

> We're having severe performance issues for any query with alias
> dereferencing set to "always".
> 
> Any query with this causes the CPU to spin up to 100% and if we have a
> number of these concurrently the machine will become unresponsive.

I hit something similar a while ago using mdb:

http://www.openldap.org/its/index.cgi/Software%20Bugs?id=8146

> We're using OpenLDAP 2.4.42 with the old hdb backend.
> 
> We do have a large number of aliases (~63,000). Could this be the cause?

It would be worth checking that you have indexed the objectclass attribute.

I prefer to avoid aliases...

Andrew
-- 
---
| From Andrew Findlay, Skills 1st Ltd |
| Consultant in large-scale systems, networks, and directory services |
| http://www.skills-1st.co.uk/+44 1628 782565 |
---

Re: Searches with dereferncing causing high CPU load.

[Mark Cairney]

Hi Andrew,

Thanks for getting back. I saw your report for mdb actually. I can
confirm that I've got "olcDBIndex objectlass eq" set on my servers.

Everyone keeps telling me that about aliases but unfortunately we've got
a group of users who require them to act in lieu of groups to support
their application i.e. they have OUs filled with aliases back to user
accounts in the main user OU.

We've started deleting old/hanging OUs and it's made a small improvement
but it's still taking 20-30s per query rather than returning almost
instantly like it was before.



On 16/11/15 18:10, Andrew Findlay wrote:
> On Mon, Nov 16, 2015 at 03:13:11PM +, Mark Cairney wrote:
> 
>> We're having severe performance issues for any query with alias
>> dereferencing set to "always".
>>
>> Any query with this causes the CPU to spin up to 100% and if we have a
>> number of these concurrently the machine will become unresponsive.
> 
> I hit something similar a while ago using mdb:
> 
> http://www.openldap.org/its/index.cgi/Software%20Bugs?id=8146
> 
>> We're using OpenLDAP 2.4.42 with the old hdb backend.
>>
>> We do have a large number of aliases (~63,000). Could this be the cause?
> 
> It would be worth checking that you have indexed the objectclass attribute.
> 
> I prefer to avoid aliases...
> 
> Andrew
> 

-- 
/

Mark Cairney
ITI UNIX Section
Information Services
University of Edinburgh

Tel: 0131 650 6565
Email: mark.cair...@ed.ac.uk
PGP: 0x435A9621

***/

The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.



signature.asc
Description: OpenPGP digital signature

RE: OpenLDAP installation. Am I missing something?

[Quanah Gibson-Mount]

--On Monday, November 16, 2015 4:32 PM + Sherman Lilly 
 wrote:




Lets try this out. Ubuntu apparently has gotten it together and fixed the
issue with dpkg-reconfigure. Red Hat has not. And wouldn't you guess I am
using Red Hat. I looked at the Red Hat documentation and it is exactly
what I am fed up with. Convert slapd.conf is the wrong way.


Don't use RedHat's builds, they have numerous issues.  You should start 
with the LTB project builds, or if you need support, contact Symas and use 
their builds.


--Quanah


--

Quanah Gibson-Mount
Platform Architect
Zimbra, Inc.

Zimbra ::  the leader in open source messaging and collaboration

RE: OpenLDAP installation. Am I missing something?

[Sherman Lilly]

Lets try this out. Ubuntu apparently has gotten it together and fixed the issue 
with dpkg-reconfigure. Red Hat has not. And wouldn't you guess I am using Red 
Hat. I looked at the Red Hat documentation and it is exactly what I am fed up 
with. Convert slapd.conf is the wrong way.

I have started slapd and tried running ldapmodify with the LDIF below and I get 
an error.

dn: olcDatabase={1}hdb,cn=config
changetype: modify
replace: olcSuffix
olcSuffix: dc=lillyrnd,dc=org

Error

[root@here ~]# ldapmodify -Y EXTERNAL -H ldapi:/// -f setup.ldif
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
modifying entry "olcDatabase={1}hdb,cn=config"
ldap_modify: No such object (32)
matched DN: cn=config

Sherman Lilly


From: David Gabriel [davidgab...@gmail.com]
Sent: Monday, November 16, 2015 3:35 AM
To: Sherman Lilly
Cc: OpenLDAP, Technical ‎[openldap-technical@openldap.org]‎
Subject: Re: OpenLDAP installation. Am I missing something?

Hi Sherman,

I am also beginner ...
I advise you to use this 
tutorial.

Regards,

2015-11-13 15:29 GMT+01:00 Sherman Lilly 
>:
I may have this totally wrong but why is there no installation documentation 
that tells somebody how to setup OpenLDAP the right way. After installing 
OpenLDAP you have no slapd.conf file so that direction is not happening. If you 
modify any file in the slapd.d directory, startup will complain about bad 
checksum. Yes I know you can regenerate the checksum and fix that but why? I 
can't find any where that tells you how to modify the base dn, rootdn, and root 
password without editing the files in the slapd.d manually. Am I missing 
something? I have check Google, Youtube, and other places and they all say 
manually edit files in slapd.d. That can't be the right way if openldap server 
is complaining about doing it.

Sherman Lilly

Re: RE24 testing call (2.4.43)

[Marco Schirrmeister]

> On 11 Nov 2015, at 17:28, Quanah Gibson-Mount  wrote:
> 
> If you know how to build OpenLDAP manually, and would like to participate in 
> testing the next set of code for the 2.4.43 release, please do so.
> 

Build and tests are successful on CentOS 6/7 64bit.
The RPMs are also installed on our test environment (C6).


Marco



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: LMDB file size, again

[Klaus Malorny]

On 11.11.2015 23:51, Christian Sell wrote:

You're being stupid.


let's say I am stubborn. As I mentioned, I have the requirement that the user be
able to copy the database file away on their USB stick (or send it via email,
for that matter). With SQLite, that is no problem. But with LMDB under Windows
the user is looking at a huge file, which will be unacceptable in all
likelihood. I also must allow the user to copy using file system tools, and not
some custom utility.

So I take it that LMDB is simply not designed or suitable for such a scenario.



Maybe you should consider to write your own exporter/importer. It shouldn't be 
too hard to walk through all key/value pairs and write out the data (prepended 
by their respective lengths), plus some metadata at the beginning to recognize 
the format on import and to know the number of records (if there is no special 
end-of-file marker).


Just my two cents.

Klaus

Re: LMDB file size, again

[Klaus Malorny]

On 16.11.2015 11:04, Klaus Malorny wrote:

On 11.11.2015 23:51, Christian Sell wrote:

You're being stupid.


let's say I am stubborn. As I mentioned, I have the requirement that the user be
able to copy the database file away on their USB stick (or send it via email,
for that matter). With SQLite, that is no problem. But with LMDB under Windows
the user is looking at a huge file, which will be unacceptable in all
likelihood. I also must allow the user to copy using file system tools, and not
some custom utility.

So I take it that LMDB is simply not designed or suitable for such a scenario.



Maybe you should consider to write your own exporter/importer. It shouldn't be
too hard to walk through all key/value pairs and write out the data (prepended
by their respective lengths), plus some metadata at the beginning to recognize
the format on import and to know the number of records (if there is no special
end-of-file marker).

Just my two cents.

Klaus


Hmm, I missed the requirement "I also must allow the user to copy using file 
system tools, and not some custom utility." However, I think this is not a good 
idea for any database since there is no guarantee that you get a valid state of 
the database without proper synchronization with the running process(es). Even 
if you are using the database with an application that does not run permanently 
(like a GUI app), I don't know whether users are smart enough to stop the 
application before performing the copy.


Regards,

Klaus