Error apperas when number of users is higher than 21800 (total users 21900) and difference between successful query and bad one is:
Good 517a35fd => access_allowed: search access to "uid=1000,ds=USERS,o=STANDARD,dc=SPR" "entry" requested 517a35fd <= root access granted 517a35fd => access_allowed: search access granted by manage(=mwrscxd) 517a35fd search_candidates: base="uid=1000,ds=users,o=standard,dc=spr" (0x00002afa) scope=2 517a35fd => mdb_filter_candidates 517a35fd EQUALITY 517a35fd => mdb_equality_candidates (objectClass) 517a35fd => key_read 517a35fd mdb_idl_fetch_key: [01872a84] 517a35fd <= mdb_index_read 30000 candidates 517a35fd <= mdb_equality_candidates: id=30000, first=21, last=110012 517a35fd <= mdb_filter_candidates: id=30000 first=21 last=110012 517a35fd => mdb_filter_candidates 517a35fd PRESENT 517a35fd => mdb_presence_candidates (objectClass) 517a35fd <= mdb_filter_candidates: id=-1 first=1 last=-1 517a35fd mdb_search_candidates: id=-1 first=1 last=-1 517a35fd => test_filter 517a35fd PRESENT 517a35fd => access_allowed: search access to "uid=1000,ds=USERS,o=STANDARD,dc=SPR" "objectClass" requested 517a35fd <= root access granted 517a35fd => access_allowed: search access granted by manage(=mwrscxd) 517a35fd <= test_filter 6 517a35fd => send_search_entry: conn 1000 dn="uid=1000,ds=USERS,o=STANDARD,dc=SPR" Bad: 517a4879 => access_allowed: search access to "uid=1000,ds=USERS,o=STANDARD,dc=SPR" "entry" requested 517a4879 <= root access granted 517a4879 => access_allowed: search access granted by manage(=mwrscxd) 517a4879 search_candidates: base="uid=1000,ds=users,o=standard,dc=spr" (0x00002afa) scope=2 517a4879 => mdb_filter_candidates 517a4879 EQUALITY 517a4879 => mdb_equality_candidates (objectClass) 517a4879 => key_read 517a4879 mdb_idl_fetch_key: [01872a84] 517a4879 <= mdb_index_read 240892 candidates 517a4879 <= mdb_equality_candidates: id=-1, first=21, last=240912 517a4879 <= mdb_filter_candidates: id=-1 first=21 last=240912 517a4879 mdb_search_candidates: failed (rc=-30798) 517a4879 mdb_search: no candidates 517a4879 send_ldap_result: conn=1021 op=1 p=3 517a4879 send_ldap_result: err=0 matched="" text="" 517a4879 send_ldap_response: msgid=2 tag=101 err=0 On Fri, Apr 26, 2013 at 8:51 AM, Saša-Stjepan Bakša <[email protected]>wrote: > If I may add some of my observations, > > Problem (for me) is starting to show when I have more than 20000 users > inserted in LMDB database. > > I am testing with latest (almost) OpenLDAP compiled from source (git pull > 24.04.2013). > > Test was done by adding 10k users first then adding 5k at a time. Between > 20k and 25k ldapsearch started to return only this: > > root@test7kde:~# ldapsearch -x -h 172.17.103.200 -D cn=admin,dc=spr -s > sub -a always -w secret -b pfUsername=user20000,dc=USERNAME,dc=spr > # extended LDIF > # > # LDAPv3 > # base <pfUsername=user20000,dc=USERNAME,dc=spr> with scope subtree > # filter: (objectclass=*) > # requesting: ALL > # > > # search result > search: 2 > result: 0 Success > > # numResponses: 1 > > I have attached ldapsearch results for reference. > > You can see data structure from ldapsearch. > > LMDB is configured like this: > > dn: olcDatabase={1}mdb,cn=config > objectClass: olcDatabaseConfig > objectClass: olcMdbConfig > olcDatabase: {1}mdb > olcDbDirectory: /opt/openldap/var/openldap-data > olcSuffix: dc=spr > olcAccess: {0}to attrs=userPassword,shadowLastChange by self write by > anonymou > s auth by dn="cn=admin,dc=spr" write by * none > olcAccess: {1}to attrs=shadowLastChange by self write by * read > olcAccess: {2}to dn.base="" by * read > olcAccess: {3}to * by self write by dn="cn=admin,dc=spr" write by * read > olcLastMod: TRUE > olcRootDN: cn=admin,dc=spr > olcRootPW:: e1NTSEF9eDRVRTBiV2Z4YnFSNnZDVDdKRWEwSWRhWFRhMDN2M3I= > olcDbCheckpoint: 4096 10 > olcDbNoSync: TRUE > olcDbMaxSize: 38654705664 > structuralObjectClass: olcMdbConfig > entryUUID: 804a8ede-2cc3-1032-9b7a-c7b2eb845e03 > creatorsName: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth > createTimestamp: 20130329135129Z > olcDbIndex: objectClass eq > olcDbIndex: uid eq > olcDbIndex: MSISDN eq > olcDbIndex: IMSI eq > olcDbIndex: pfUsername eq > olcDbIndex: entryCSN eq > olcDbIndex: entryUUID eq > olcDbIndex: contextCSN eq > olcMirrorMode: TRUE > olcSyncrepl: {0}rid=003 provider=ldap://spr1.lab.os > binddn="cn=admin,dc=spr" b > indmethod=simple credentials=siemens searchbase="dc=spr" type=refreshOnly > int > erval=00:00:00:10 retry="5 5 300 5" timeout=1 > olcSyncrepl: {1}rid=004 provider=ldap://spr2.lab.os > binddn="cn=admin,dc=spr" b > indmethod=simple credentials=siemens searchbase="dc=spr" type=refreshOnly > int > erval=00:00:00:10 retry="5 5 300 5" timeout=1 > entryCSN: 20130329141637.442296Z#000000#002#000000 > modifiersName: cn=admin,cn=config > modifyTimestamp: 20130329141637Z > > > If you need more data I wiil be more than happy to provide. > > > On Wed, Apr 24, 2013 at 11:06 AM, Howard Chu <[email protected]> wrote: > >> [email protected] wrote: >> >>> Hi Michael, >>> >>> NSS results must not be dependent on the backend database a directory >>> service uses. >>> >>> I activated connection logging and here's the proof that NSS is not the >>> culprit. >>> >>> Searches initiated by NSS are identical and exactly this behavior can >>> also be seen when using ldapsearch from command line with parameters from >>> the log: >>> >>> # running 'getent passwd' with hdb backend: >>> Apr 24 09:53:54 openldap-dev slapd[19240]: conn=1000 op=0 BIND >>> dn="cn=itsAgent,ou=**customerAgent,dc=scom" method=128 >>> Apr 24 09:53:54 openldap-dev slapd[19240]: conn=1000 op=0 BIND >>> dn="cn=itsAgent,ou=**customerAgent,dc=scom" mech=SIMPLE ssf=0 >>> Apr 24 09:53:54 openldap-dev slapd[19240]: conn=1000 op=0 RESULT tag=97 >>> err=0 text= >>> Apr 24 09:53:54 openldap-dev slapd[19240]: conn=1000 op=1 SRCH >>> base="ou=account,dc=its,dc=**scom" scope=1 deref=3 filter="(objectClass= >>> **posixAccount)" >>> Apr 24 09:53:54 openldap-dev slapd[19240]: conn=1000 op=1 SRCH attr=uid >>> userPassword uidNumber gidNumber cn homeDirectory x-LinuxLoginShell gecos >>> description objectClass >>> Apr 24 09:53:54 openldap-dev slapd[19240]: conn=1000 op=1 SEARCH RESULT >>> tag=101 err=0 nentries=656 text= >>> Apr 24 09:53:54 openldap-dev slapd[19240]: conn=1000 fd=13 closed >>> (connection lost) >>> >>> # running 'getent passwd' with mdb backend: >>> Apr 24 10:00:17 openldap-dev slapd[19300]: conn=1002 op=0 BIND >>> dn="cn=itsAgent,ou=**customerAgent,dc=scom" method=128 >>> Apr 24 10:00:17 openldap-dev slapd[19300]: conn=1002 op=0 BIND >>> dn="cn=itsAgent,ou=**customerAgent,dc=scom" mech=SIMPLE ssf=0 >>> Apr 24 10:00:17 openldap-dev slapd[19300]: conn=1002 op=0 RESULT tag=97 >>> err=0 text= >>> Apr 24 10:00:17 openldap-dev slapd[19300]: conn=1002 op=1 SRCH >>> base="ou=account,dc=its,dc=**scom" scope=1 deref=3 filter="(objectClass= >>> **posixAccount)" >>> Apr 24 10:00:17 openldap-dev slapd[19300]: conn=1002 op=1 SRCH attr=uid >>> userPassword uidNumber gidNumber cn homeDirectory x-LinuxLoginShell gecos >>> description objectClass >>> Apr 24 10:00:17 openldap-dev slapd[19300]: conn=1002 op=1 SEARCH RESULT >>> tag=101 err=0 text= >>> Apr 24 10:00:17 openldap-dev slapd[19300]: conn=1002 fd=13 closed >>> (connection lost) >>> >>> I suspect that aliases are not handled the same way in hdb and mdb as I >>> am using aliases here and deref=3 in both searches, example: >>> >>> dn: uid=joe,ou=Account,dc=its,dc=**scom >>> objectClass: alias >>> objectClass: extensibleObject >>> uid: joe >>> aliasedObjectName: uid=joe,ou=Person,dc=its,dc=**scom >>> structuralObjectClass: alias >>> >>> When using hdb, the alias is dereferenced correctly (nentries=656), when >>> using mdb it seems not to be dereferenced at all (nentries=0). >>> >>> Maybe there's a parameter around for mdb which I couldn't find in the >>> docs >>> >> to prevent this, but if not I consider this as a bug. >> >> There is no parameter. Seems like you've found a bug, please submit the >> info to the ITS. >> >> >>> Regards >>> >>> Juergen >>> >>> [email protected] wrote: >>> >>>> It would certainly help if you could examine the issue with pure LDAP >>>> search >>>> operations preferably with OpenLDAP's ldapsearch command-line tool. >>>> >>>> When looking at NSS results too many things can go wrong with other >>>> components' configuration. >>>> >>>> Ciao, Michael. >>>> >>>> [email protected] wrote: >>>> >>>>> Hi, >>>>> >>>>> I have running OpenDLAP 2.4.35 on Gentoo Linux and wanted to make >>>>> some tests with mdb. >>>>> >>>>> Slapd was running fine with hdb, no problems so far. >>>>> Then I exported contents via slapcat and switched config to mdb. >>>>> When slapd started using mdb no users from directory were shown by >>>>> 'getent passwd': >>>>> >>>>> ### hdb part #### >>>>> # using hdb parameters >>>>> database hdb >>>>> dirtyread >>>>> cachesize 150000 >>>>> cachefree 100 >>>>> idlcachesize 450000 >>>>> dncachesize 100000 >>>>> >>>>> # slapadd from backup and run slapd with hdb backend >>>>> /etc/init.d/unscd stop >>>>> /etc/init.d/slapd stop >>>>> rm /var/lib/openldap-data/* >>>>> rm -rf /etc/openldap/slapd.d/* >>>>> cp -p /etc/openldap/DB_CONFIG /var/lib/openldap-data/ >>>>> cp -p /etc/openldap/slapd.conf.hdb /etc/openldap/slapd.conf >>>>> su ldap -c '/usr/sbin/slapadd -f /etc/openldap/slapd.conf -l >>>>> odsldap-dev.ldif' >>>>> /etc/init.d/slapd start >>>>> /etc/init.d/unscd start >>>>> slapcat -f /etc/openldap/slapd.conf -b dc=scom | md5sum >>>>> # 73850f9a3f7ff9d3d1ddb7663cd046**a6 - >>>>> >>>>> getent passwd >>>>> # all users shown, everything ok >>>>> >>>>> ### mdb part #### >>>>> # using mdb paramters >>>>> database mdb >>>>> dbnosync >>>>> maxsize 2094967296 >>>>> searchstack 64 >>>>> >>>>> # slapadd from backup and run slapd with mdb backend >>>>> /etc/init.d/unscd stop >>>>> /etc/init.d/slapd stop >>>>> rm /var/lib/openldap-data/* >>>>> rm -rf /etc/openldap/slapd.d/* >>>>> cp -p /etc/openldap/slapd.conf.mdb /etc/openldap/slapd.conf >>>>> su ldap -c '/usr/sbin/slapadd -f /etc/openldap/slapd.conf -l >>>>> odsldap-dev.ldif' >>>>> /etc/init.d/slapd start >>>>> /etc/init.d/unscd start >>>>> slapcat -f /etc/openldap/slapd.conf -b dc=scom | md5sum >>>>> # 73850f9a3f7ff9d3d1ddb7663cd046**a6 - >>>>> >>>> <> >>> >>>> getent passwd >>>>> # no users from ldap shown >>>>> >>>>> Am I missing something when setting up and using mdb? >>>>> Both backends have exactly the same content, and so the results for >>>>> searches should also be identical. >>>>> >>>>> Regards >>>>> >>>>> Jürgen Sprenger >>>>> >>>> >>> >>> >>> >> >> -- >> -- Howard Chu >> CTO, Symas Corp. http://www.symas.com >> Director, Highland Sun http://highlandsun.com/hyc/ >> Chief Architect, OpenLDAP >> http://www.openldap.org/**project/<http://www.openldap.org/project/> >> >> >
