Re: Very slow ldapserach
On 9 April 2015 at 17:57, Quanah Gibson-Mount qua...@zimbra.com wrote: --On Thursday, April 09, 2015 10:21 AM +0200 Saša-Stjepan Bakša ssba...@gmail.com wrote: And search for all still crashes openldap server ldapsearch -h 10.14.252.103 -p 389 -D cn=admin,dc=spr -w siemens -s sub -a always -b dc=SPR objectClass=* I must wait for 2.4.41 release then. In the mean time I will use LTB project build with HDB to see can I have that version in working state for my colleagues test suite. Hi Sasa, Thanks for checking. Looks like there is some more work to do in this area before 2.4.41 releases. I wil be asking you to do further testing in the near future. If it were possible for you to provide me an example DB and your slapd configuration that replicates this behavior, it would help with a resolution. --Quanah Hi Quanah, I will be more than happy to provide help with testing. I am at home now but tomorrow morning (8:00 CET) I will provide as much data as I can to you. As some of data can be of sensitive nature, may be I will be obliged to send it directly to you. Is that Ok with you? Best regards, Sasa
Re: Very slow ldapserach
On 9 April 2015 at 17:57, Quanah Gibson-Mount qua...@zimbra.com wrote: --On Thursday, April 09, 2015 10:21 AM +0200 Saša-Stjepan Bakša ssba...@gmail.com wrote: And search for all still crashes openldap server I must wait for 2.4.41 release then. In the mean time I will use LTB project build with HDB to see can I have that version in working state for my colleagues test suite. Hi Sasa, Thanks for checking. Looks like there is some more work to do in this area before 2.4.41 releases. I wil be asking you to do further testing in the near future. If it were possible for you to provide me an example DB and your slapd configuration that replicates this behavior, it would help with a resolution. --Quanah And again I did stupid thing to send withoth checking recipient. I hate this Gmail web interface. Sorry.
Re: Very slow ldapserach
On 9 April 2015 at 02:37, Quanah Gibson-Mount qua...@zimbra.com wrote:
--On Wednesday, April 08, 2015 8:51 PM +0200 Saša-Stjepan Bakša
ssba...@gmail.com wrote:
I am sorry for this mistake with answering to you and not to the group.
It was unintentional mistake.
Ok, I will check ITS#7657.
Please try current RE24 code. It is believed the fixes for ITS#8011 which
are scheduled to be part of the 2.4.41 relase should resolve the issue.
Thanks,
Quanah
At 8:00 local time:
git checkout OPENLDAP_REL_ENG_2_4
git pull
Build + test
root@test:~# date; ldapsearch -h 10.14.252.103 -p 389 -D cn=admin,dc=spr -w
siemens -s sub -a always -b msisdn=38521209,dc=MSISDN,dc=SPR
objectClass=*; date
*Thu Apr 9 08:22:46 CEST 2015*
# extended LDIF
#
# LDAPv3
# base msisdn=38521209,dc=MSISDN,dc=SPR with scope subtree
# filter: objectClass=*
# requesting: ALL
#
- cut--
# search result
search: 2
result: 0 Success
# numResponses: 9
# numEntries: 8
*Thu Apr 9 08:23:38 CEST 2015*
Nothing changed yet - 52 seconds
And search for all still crashes openldap server
ldapsearch -h 10.14.252.103 -p 389 -D cn=admin,dc=spr -w siemens -s sub -a
always -b dc=SPR objectClass=*
552641e7 dnPrettyNormal: dc=SPR
552641e7 dnPrettyNormal: dc=SPR, dc=spr
ber_scanf fmt (m) ber:
ber_scanf fmt ({M}}) ber:
552641e7 = mdb_search
552641e7 mdb_dn2entry(dc=spr)
552641e7 = mdb_dn2id(dc=spr)
552641e7 = mdb_dn2id: got id=0x1
552641e7 = mdb_entry_decode:
552641e7 = mdb_entry_decode
552641e7 search_candidates: base=dc=spr (0x0001) scope=2
552641e7 = mdb_equality_candidates (objectClass)
552641e7 = key_read
552641e7 = mdb_index_read 1000 candidates
552641e7 = mdb_equality_candidates: id=-1, first=16, last=1015
Segmentation fault
I must wait for 2.4.41 release then. In the mean time I will use LTB
project build with HDB to see can I have that version in working state for
my colleagues test suite.
Br
Sasa
Re: Very slow ldapserach
--On Thursday, April 09, 2015 10:21 AM +0200 Saša-Stjepan Bakša
ssba...@gmail.com wrote:
And search for all still crashes openldap server
ldapsearch -h 10.14.252.103 -p 389 -D cn=admin,dc=spr -w siemens -s sub
-a always -b dc=SPR objectClass=*
552641e7 dnPrettyNormal: dc=SPR
552641e7 dnPrettyNormal: dc=SPR, dc=spr
ber_scanf fmt (m) ber:
ber_scanf fmt ({M}}) ber:
552641e7 = mdb_search
552641e7 mdb_dn2entry(dc=spr)
552641e7 = mdb_dn2id(dc=spr)
552641e7 = mdb_dn2id: got id=0x1
552641e7 = mdb_entry_decode:
552641e7 = mdb_entry_decode
552641e7 search_candidates: base=dc=spr (0x0001) scope=2
552641e7 = mdb_equality_candidates (objectClass)
552641e7 = key_read
552641e7 = mdb_index_read 1000 candidates
552641e7 = mdb_equality_candidates: id=-1, first=16, last=1015
Segmentation fault
I must wait for 2.4.41 release then. In the mean time I will use LTB
project build with HDB to see can I have that version in working state
for my colleagues test suite.
Hi Sasa,
Thanks for checking. Looks like there is some more work to do in this area
before 2.4.41 releases. I wil be asking you to do further testing in the
near future. If it were possible for you to provide me an example DB and
your slapd configuration that replicates this behavior, it would help with
a resolution.
--Quanah
--
Quanah Gibson-Mount
Platform Architect
Zimbra, Inc.
Zimbra :: the leader in open source messaging and collaboration
Re: Very slow ldapserach
On 8 April 2015 at 18:05, Quanah Gibson-Mount qua...@zimbra.com wrote: --On Wednesday, April 08, 2015 11:58 AM +0200 Saša-Stjepan Bakša ssba...@gmail.com wrote: I have tested search with your suggestion. No crash - slapd crashes only when deref is on or in best case it is slow (10 to 20 seconds for answer). As I have said, when database is small (relative size) it is quick as we expected. So, which alternatives do I have? Back to hdb? Uh, I don't like it to much. Is this long standing issue a big problem to solve? I am not developer so maybe this was a rude question (sorry!). Please keep replies on the list if you want help. My guess is this is ITS#7657. --Quanah I am sorry for this mistake with answering to you and not to the group. It was unintentional mistake. Ok, I will check ITS#7657. Sasa
Re: Very slow ldapserach
--On Wednesday, April 08, 2015 8:51 PM +0200 Saša-Stjepan Bakša ssba...@gmail.com wrote: I am sorry for this mistake with answering to you and not to the group. It was unintentional mistake. Ok, I will check ITS#7657. Please try current RE24 code. It is believed the fixes for ITS#8011 which are scheduled to be part of the 2.4.41 relase should resolve the issue. Thanks, Quanah -- Quanah Gibson-Mount Platform Architect Zimbra, Inc. Zimbra :: the leader in open source messaging and collaboration
Re: Very slow ldapserach
Hi,
It was long weekend with Easter holiday so...
To answer Mattes, and Ulrich first, I will do as suggested by you.
But something new first. If I have tried to add less data to database and
then I have conducted the search. Till some point it works and then fails.
It looks like when database is filed with more than some amount of data it
stops to send data back.
To be sure that my build is not the culprit I have installed in parallel on
similar server Ubuntu 14.04 with their slapd package (I know, it is not
good but just for this test...) and this happened when I have tried this:
ldapsearch -h 10.14.252.104 -p 389 -D cn=admin,dc=spr -w password -s sub -a
always -b dc=SPR objectClass=*
My build on centos 6.6 chrashed with this message:
5523ac18 slap_listener(ldap:///)
5523ac18 connection_get(11): got connid=1001
5523ac18 connection_read(11): checking for input on id=1001
ber_get_next
ber_get_next: tag 0x30 len 34 contents:
5523ac18 op tag 0x60, time 1428401176
ber_get_next
5523ac18 conn=1001 op=0 do_bind
ber_scanf fmt ({imt) ber:
ber_scanf fmt (m}) ber:
5523ac18 dnPrettyNormal: cn=admin,dc=spr
5523ac18 dnPrettyNormal: cn=admin,dc=spr, cn=admin,dc=spr
5523ac18 do_bind: version=3 dn=cn=admin,dc=spr method=128
5523ac18 do_bind: v3 bind: cn=admin,dc=spr to cn=admin,dc=spr
5523ac18 send_ldap_result: conn=1001 op=0 p=3
5523ac18 send_ldap_response: msgid=1 tag=97 err=0
ber_flush2: 14 bytes to sd 11
5523ac18 connection_get(11): got connid=1001
5523ac18 connection_read(11): checking for input on id=1001
ber_get_next
ber_get_next: tag 0x30 len 43 contents:
5523ac18 op tag 0x63, time 1428401176
ber_get_next
5523ac18 conn=1001 op=1 do_search
ber_scanf fmt ({mb) ber:
5523ac18 dnPrettyNormal: dc=SPR
5523ac18 dnPrettyNormal: dc=SPR, dc=spr
ber_scanf fmt (m) ber:
ber_scanf fmt ({M}}) ber:
5523ac18 = mdb_search
5523ac18 mdb_dn2entry(dc=spr)
5523ac18 = mdb_dn2id(dc=spr)
5523ac18 = mdb_dn2id: got id=0x1
5523ac18 = mdb_entry_decode:
5523ac18 = mdb_entry_decode
5523ac18 search_candidates: base=dc=spr (0x0001) scope=2
5523ac18 = mdb_equality_candidates (objectClass)
5523ac18 = key_read
5523ac18 = mdb_index_read 1000 candidates
5523ac18 = mdb_equality_candidates: id=-1, first=16, last=1015
Segmentation fault
On Ubuntu 14.04 passed through but no result:
Apr 7 12:51:36 spr2 slapd[8432]: conn=1001 op=1 SRCH base=dc=SPR scope=2
deref=3 filter=(objectClass=*)
Apr 7 12:51:36 spr2 slapd[8432]: = mdb_search
Apr 7 12:51:36 spr2 slapd[8432]: mdb_dn2entry(dc=spr)
Apr 7 12:51:36 spr2 slapd[8432]: = mdb_dn2id(dc=spr)
Apr 7 12:51:36 spr2 slapd[8432]: = mdb_dn2id: got id=0x1
Apr 7 12:51:36 spr2 slapd[8432]: = mdb_entry_decode:
Apr 7 12:51:36 spr2 slapd[8432]: = mdb_entry_decode
Apr 7 12:51:36 spr2 slapd[8432]: = access_allowed: search access to
dc=SPR entry requested
Apr 7 12:51:36 spr2 slapd[8432]: = root access granted
Apr 7 12:51:36 spr2 slapd[8432]: = access_allowed: search access granted
by manage(=mwrscxd)
Apr 7 12:51:36 spr2 slapd[8432]: search_candidates: base=dc=spr
(0x0001) scope=2
Apr 7 12:51:36 spr2 slapd[8432]: = mdb_filter_candidates
Apr 7 12:51:36 spr2 slapd[8432]: #011EQUALITY
Apr 7 12:51:36 spr2 slapd[8432]: = mdb_equality_candidates (objectClass)
Apr 7 12:51:36 spr2 slapd[8432]: = key_read
Apr 7 12:51:36 spr2 slapd[8432]: mdb_idl_fetch_key: [01872a84]
Apr 7 12:51:36 spr2 slapd[8432]: = mdb_index_read 294110 candidates
Apr 7 12:51:36 spr2 slapd[8432]: = mdb_equality_candidates: id=-1,
first=16, last=294125
Apr 7 12:51:36 spr2 slapd[8432]: = mdb_filter_candidates: id=-1 first=16
last=294125
Apr 7 12:51:36 spr2 slapd[8432]: mdb_search_candidates: failed (rc=-30798)
Apr 7 12:51:36 spr2 slapd[8432]: mdb_search: no candidates
Apr 7 12:51:36 spr2 slapd[8432]: send_ldap_result: conn=1001 op=1 p=3
Apr 7 12:51:36 spr2 slapd[8432]: send_ldap_result: err=0 matched= text=
Apr 7 12:51:36 spr2 slapd[8432]: send_ldap_response: msgid=2 tag=101 err=0
Apr 7 12:51:36 spr2 slapd[8432]: conn=1001 op=1 SEARCH RESULT tag=101
err=0 nentries=0 text=
Apr 7 12:51:36 spr2 slapd[8432]: daemon: activity on 1 descriptor
Apr 7 12:51:36 spr2 slapd[8432]: daemon: activity on:
Apr 7 12:51:36 spr2 slapd[8432]: 13r
For test I have cut data from 2 mil lines to 30 lines and database is
now just 841 M in size (was 2,1 G)
root@spr2:/var/log# du -hs /var/lib/ldap/*
841M/var/lib/ldap/data.mdb
4.0K/var/lib/ldap/lock.mdb
And data is accessible:
cut-
# search result
search: 2
result: 0 Success
# numResponses: 22076
# numEntries: 22075
It looks like this is happening again:
http://www.openldap.org/lists/openldap-technical/201304/msg00198.html
Br
Sasa
Re: Very slow ldapserach
On 7 April 2015 at 11:36, Saša-Stjepan Bakša ssba...@gmail.com wrote: Hi, It was long weekend with Easter holiday so... To answer Mattes, and Ulrich first, I will do as suggested by you. But something new first. If I have tried to add less data to database and then I have conducted the search. Till some point it works and then fails. It looks like when database is filed with more than some amount of data it stops to send data back. To be sure that my build is not the culprit I have installed in parallel on similar server Ubuntu 14.04 with their slapd package (I know, it is not good but just for this test...) and this happened when I have tried this: ldapsearch -h 10.14.252.104 -p 389 -D cn=admin,dc=spr -w password -s sub -a always -b dc=SPR objectClass=* My build on centos 6.6 chrashed with this message: cut For test I have cut data from 2 mil lines to 30 lines and database is now just 841 M in size (was 2,1 G) root@spr2:/var/log# du -hs /var/lib/ldap/* 841M/var/lib/ldap/data.mdb 4.0K/var/lib/ldap/lock.mdb And data is accessible: cut- # search result search: 2 result: 0 Success # numResponses: 22076 # numEntries: 22075 It looks like this is happening again: http://www.openldap.org/lists/openldap-technical/201304/msg00198.html Just to add, after deleting big database from centos build openldap server, ldapsearch for all objects works as expected - no crashes. cut # search result search: 2 result: 0 Success # numResponses: 22076 # numEntries: 22075 Br Sasa
Re: Very slow ldapserach
--On Tuesday, April 07, 2015 12:36 PM +0200 Saša-Stjepan Bakša ssba...@gmail.com wrote: Hi, It was long weekend with Easter holiday so... To answer Mattes, and Ulrich first, I will do as suggested by you. But something new first. If I have tried to add less data to database and then I have conducted the search. Till some point it works and then fails. It looks like when database is filed with more than some amount of data it stops to send data back. To be sure that my build is not the culprit I have installed in parallel on similar server Ubuntu 14.04 with their slapd package (I know, it is not good but just for this test...) and this happened when I have tried this: ldapsearch -h 10.14.252.104 -p 389 -D cn=admin,dc=spr -w password -s sub -a always -b dc=SPR objectClass=* Sounds like there's something wrong with your openldap build or server. Zimbra has clients with DBs of various sizes and entry counts, often in the multi-GBs, and we're not seeing any such issues. --Quanah -- Quanah Gibson-Mount Platform Architect Zimbra, Inc. Zimbra :: the leader in open source messaging and collaboration
Re: Very slow ldapserach
--On Tuesday, April 07, 2015 1:10 PM -0700 Quanah Gibson-Mount qua...@zimbra.com wrote: ldapsearch -h 10.14.252.104 -p 389 -D cn=admin,dc=spr -w password -s sub -a always -b dc=SPR objectClass=* Sounds like there's something wrong with your openldap build or server. Zimbra has clients with DBs of various sizes and entry counts, often in the multi-GBs, and we're not seeing any such issues. Actually, you are doing alias deref. I believe there's a known issue with back-mdb and alias deref. Do you see the same behavior if you select -a never instead? --Quanah -- Quanah Gibson-Mount Platform Architect Zimbra, Inc. Zimbra :: the leader in open source messaging and collaboration
Re: Very slow ldapserach
Am Donnerstag, 02. April 2015 13:40 CEST, Saša-Stjepan Bakša ssba...@gmail.com schrieb: On 31 March 2015 at 23:25, Geoff Swan gsw...@bigpond.net.au wrote: Does the server have access to a nameserver? I've seen dns timeouts cause this kind of thing. Yes it has. Also, I have created distinct entry's in hosts file to make name resolving quicker and to make sure that name resolving is not the main culprit for this slow down. When I start slapd in debug mode, normally all answers are so quick written on the screen that I can't read without stopping slapd. With our setup, I can read line by line. One shure performance killer I encountered was activating gccess logging. If this isn0t the case you might need to set up some profiling. Since you are running under Linux perf might be a good tool. It might be a good idea to recompile your binary with -fno-omit-frame-pointer to get meaningful/correct call graph reports. You then can do: perf record -g -p pid-of-ldapd and then later on perf report --stdio perf report --stdio --sort=dso -g none perf report --stdio -g none perf report --stdio -g (have a look at https://perf.wiki.kernel.org/index.php/Tutorial) HTH Ralf Mattes
Re: Very slow ldapserach
--On Tuesday, March 31, 2015 4:09 PM +0200 Saša-Stjepan Bakša ssba...@gmail.com wrote: Hi, Year ago we have tested openldap with back_mdb and it was fantastic. Search worked as a charm. Database was filled with 20 mil. users and serach returned some 20 k results per sec (my colegue did the test). Now we need that setup for some tests and we encountered very slow response - 1 search for user data with some aliased data need 8 to 20 seconds to be retrieved. ldapsearch -h 10.14.252.104 -p 389 -D cn=admin,dc=spr -w test -s sub -a always -b num=1234563123,dc=num,dc=SPR ObjectClass=* num=1234563123,dc=num,dc=SPR is alias to uid aliasedObjectName: uid=1234563123,ds=USERS,o=STANDARD,dc=spr Have you tried using writemap for the lmdb db? --Quanah -- Quanah Gibson-Mount Platform Architect Zimbra, Inc. Zimbra :: the leader in open source messaging and collaboration
Re: Very slow ldapserach
Does the server have access to a nameserver?
I've seen dns timeouts cause this kind of thing.
On 1/04/2015 12:09 AM, Saša-Stjepan Bakša wrote:
Hi,
Year ago we have tested openldap with back_mdb and it was fantastic.
Search worked as a charm. Database was filled with 20 mil. users and
serach returned some 20 k results per sec (my colegue did the test).
Now we need that setup for some tests and we encountered very slow
response - 1 search for user data with some aliased data need 8 to 20
seconds to be retrieved.
ldapsearch -h 10.14.252.104 -p 389 -D cn=admin,dc=spr -w test -s sub
-a always -b num=1234563123,dc=num,dc=SPR ObjectClass=*
num=1234563123,dc=num,dc=SPR is alias to uid
aliasedObjectName: uid=1234563123,ds=USERS,o=STANDARD,dc=spr
We build our openldap from git source. We have tried new as older
versions as well and no change is seen.
Hardware: SuperMicro, 2xQuad core, 32 GB RAM, RAID 10 storage.
HP blade 2xQuad core, 64 GB RAM sorage 2 disks in mirror.
Results are the same and not depending on hardware.
Openldap ver:
root@centdevel openldap# git log
commit 68d9aa207f51b4d1ef29bb9876e7da8c7eaf0eee
Author: Quanah Gibson-Mount qua...@openldap.org
mailto:qua...@openldap.org
Date: Tue Apr 8 21:16:52 2014 -0500
ITS#7430, ITS#6359
OS is Centos 6.4 (also tryed on Centos 6.6)numx
mdb config part is:
[root@spr2 cn=config]# cat olcDatabase\=\{1\}mdb.ldif
# AUTO-GENERATED FILE - DO NOT EDIT!! Use ldapmodify.
# CRC32 2c245069
dn: olcDatabase={1}mdb
objectClass: olcDatabaseConfig
objectClass: olcMdbConfig
olcDatabase: {1}mdb
olcDbDirectory: /opt/openldap/var/openldap-data
olcSuffix: dc=spr
olcAccess: {0}to attrs=userPassword,shadowLastChange by self write by
anonymou
s auth by dn=cn=admin,dc=spr write by * none
olcAccess: {1}to attrs=shadowLastChange by self write by * read
olcAccess: {2}to dn.base= by * read
olcAccess: {3}to * by self write by dn=cn=admin,dc=spr write by * read
olcLastMod: TRUE
olcRootDN: cn=admin,dc=spr
olcRootPW:: xyzdgsdsadeew
olcDbCheckpoint: 4096 10
olcDbNoSync: TRUE
olcDbIndex: objectClass eq
olcDbIndex: uid eq
olcDbIndex: num eq
olcDbIndex: numx eq
olcDbIndex: Username eq
olcDbIndex: entryCSN eq
olcDbIndex: entryUUID eq
olcDbIndex: contextCSN eq
olcDbMaxSize: 16106127360
structuralObjectClass: olcMdbConfig
entryUUID: 21ac150c-6b30-1034-9009-81396a683c5e
creatorsName: cn=admin,cn=config
createTimestamp: 20150330135513Z
entryCSN: 20150330135513.544218Z#00#000#00
modifiersName: cn=admin,cn=config
modifyTimestamp: 20150330135513Z
MDB database stat:
[root@spr2 openldap]# /opt/openldap/sbin/mdb_stat
/opt/openldap/var/openldap-data/ -e -rr -a
Environment Info
Map address: (nil)
Map size: 16106127360
Page size: 4096
Max pages: 3932160
Number of pages used: 1523336
Last transaction ID: 16058165
Max readers: 126
Number of readers used: 0
Reader Table Status
(no active readers)
0 stale readers cleared.
(no active readers)
Status of Main DB
Tree depth: 1
Branch pages: 0
Leaf pages: 1
Overflow pages: 0
Entries: 11
Status of ad2i
Tree depth: 1
Branch pages: 0
Leaf pages: 1
Overflow pages: 0
Entries: 38
Status of contextCSN
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of dn2i
Tree depth: 4
Branch pages: 2937
Leaf pages: 38
Overflow pages: 0
Entries: 1669
Status of entryCSN
Tree depth: 3
Branch pages: 3
Leaf pages: 307
Overflow pages: 0
Entries: 834
Status of entryUUID
Tree depth: 3
Branch pages: 259
Leaf pages: 62932
Overflow pages: 0
Entries: 834
Status of id2e
Tree depth: 4
Branch pages: 4446
Leaf pages: 105
Overflow pages: 0
Entries: 834
Status of numx
Tree depth: 3
Branch pages: 128
Leaf pages: 22295
Overflow pages: 0
Entries: 204
Status of num
Tree depth: 3
Branch pages: 129
Leaf pages: 22325
Overflow pages: 0
Entries: 204
Status of objectClass
Tree depth: 1
Branch pages: 0
Leaf pages: 1
Overflow pages: 0
Entries: 29
Status of Username
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of uid
Tree depth: 3
Branch pages: 34
Leaf pages: 7883
Overflow pages: 0
Entries: 104
Build config:
make clean
./configure --enable-hdb=no \
--enable-bdb=no \
--enable-monitor=yes \
--prefix=/opt/openldap \
--enable-local=yes \
--enable-accesslog=yes \
--enable-syncprov=yes \
--enable-debug=yes
make depend
make #STRIP=''
rm -r /opt/openldap/etc/openldap/schema
make install #STRIP=''
removing debug has no efect
Do you have any hint for us?
Br
Sasa
