Re: (ITS#9051) slapo-accesslog fails to log compare, search

2019-07-08 Thread quanah 
--On Monday, July 08, 2019 11:33 PM + qua...@openldap.org wrote:

Note: compare operations are confirmed also to not log.

:~# ldapcompare -Y EXTERNAL -H ldapi:/// cn=admin,dc=rb,dc=symas,dc=net 
cn:admin
TRUE

ldapsearch -Y EXTERNAL -H ldapi:/// -b cn=accesslog -LLL
dn: cn=accesslog
objectClass: auditContainer
cn: accesslog

dn: reqStart=2019070817.00Z,cn=accesslog
objectClass: auditBind
reqStart: 2019070817.00Z
reqEnd: 2019070817.01Z
reqType: bind
reqSession: 1014
reqAuthzID:
reqDN: cn=admin,dc=rb,dc=symas,dc=net
reqResult: 0
reqVersion: 3
reqMethod: SIMPLE

dn: reqStart=20190709001033.00Z,cn=accesslog
objectClass: auditBind
reqStart: 20190709001033.00Z
reqEnd: 20190709001033.01Z
reqType: bind
reqSession: 1019
reqAuthzID:
reqDN: cn=admin,dc=rb,dc=symas,dc=net
reqResult: 0
reqVersion: 3
reqMethod: SIMPLE


--Quanah



--

Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:

(ITS#9051) slapo-accesslog fails to log compare, search

2019-07-08 Thread quanah 
Full_Name: Quanah Gibson-Mount
Version: 2.4.47
OS: N/A
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (47.208.128.44)


In testing out various logging scenarios with the accesslog overlay, it has been
found that it fails to log certain operations in the underlying accesslog DB. 
Specifically, it fails to log "reads" or "compare" + "search".  It does however
log "writes" or "bind".

Example configuration:

dn: olcDatabase={2}mdb,cn=config
objectClass: olcMdbConfig
objectClass: olcDatabaseConfig
olcDatabase: {2}mdb
olcDbDirectory: /var/lib/ldap/accesslog
olcAddContentAcl: FALSE
olcDbIndex: default eq
olcDbIndex: objectClass
olcDbIndex: entryUUID
olcDbIndex: entryCSN
olcDbIndex: reqStart
olcDbIndex: reqEnd
olcDbIndex: reqResult
olcDbIndex: reqDN
olcDbMaxReaders: 0
olcDbMaxSize: 512
olcDbMode: 0600
olcDbNoSync: FALSE
olcDbRtxnSize: 1
olcDbSearchStack: 16
olcLastMod: TRUE
olcMaxDerefDepth: 15
olcMonitoring: TRUE
olcReadOnly: FALSE
olcRootDN: cn=config
olcSuffix: cn=accesslog
olcSyncUseSubentry: FALSE
olcAccess: {0}to * by dn.exact=gidNumber=0+uidNumber=0,cn=peercred,cn=extern
 al,cn=auth manage by * break


dn: olcOverlay={0}accesslog,olcDatabase={1}mdb,cn=config
objectClass: olcAccessLogConfig
objectClass: olcOverlayConfig
olcAccessLogDB: cn=accesslog
olcOverlay: {0}accesslog
olcAccessLogOps: reads
olcAccessLogOps: writes
olcAccessLogPurge: 1+00:00 1+00:00
olcAccessLogSuccess: TRUE


Performing a search in this scenario results in nothing logged.

Changing it to "reads" only (no writes), nothing logged
Changing it to "bind, search, compare", the bind is logged, but nothing else:

root@anvil3:~/accesslog-testing# ldapsearch -Y EXTERNAL -H ldapi:/// -b
cn=accesslog -LLL -Q

dn: cn=accesslog
objectClass: auditContainer
cn: accesslog

dn: reqStart=2019070817.00Z,cn=accesslog
objectClass: auditBind
reqStart: 2019070817.00Z
reqEnd: 2019070817.01Z
reqType: bind
reqSession: 1014
reqAuthzID:
reqDN: cn=admin,dc=rb,dc=symas,dc=net
reqResult: 0
reqVersion: 3
reqMethod: SIMPLE


Search is clearly logged at STATS level logging:

Jul  8 15:22:17 anvil3 slapd[12993]: conn=1014 fd=12 ACCEPT from IP=[::1]:51644
(IP=[::]:389)
Jul  8 15:22:17 anvil3 slapd[12993]: conn=1014 op=0 BIND
dn="cn=admin,dc=rb,dc=symas,dc=net" method=128
Jul  8 15:22:17 anvil3 slapd[12993]: conn=1014 op=0 BIND
dn="cn=admin,dc=rb,dc=symas,dc=net" mech=SIMPLE ssf=0
Jul  8 15:22:17 anvil3 slapd[12993]: conn=1014 op=0 RESULT tag=97 err=0 text=
Jul  8 15:22:17 anvil3 slapd[12993]: conn=1014 op=1 SRCH
base="dc=rb,dc=symas,dc=net" scope=2 deref=0 filter="(objectClass=*)"
Jul  8 15:22:17 anvil3 slapd[12993]: conn=1014 op=1 SEARCH RESULT tag=101 err=0
nentries=2 text=
Jul  8 15:22:17 anvil3 slapd[12993]: conn=1014 op=2 UNBIND
Jul  8 15:22:17 anvil3 slapd[12993]: conn=1014 fd=12 closed

Re: (ITS#8875) [Patch] Performance problems in back-mdb with large DITs and many aliases

2019-07-08 Thread henrik . bohnenkamp 
On Thu, Jun 27, 2019 at 10:34:19PM -0700, Quanah Gibson-Mount wrote:
> --On Sunday, April 07, 2019 10:15 AM + henrik.bohnenk...@ionos.com
> wrote:
> 
> > On Thu, Apr 04, 2019 at 05:32:16PM +0100, Howard Chu wrote:
> 
> Had a customer who was hitting this issue try out these patches -- It
> greatly decreases the search time (from unknown/infinite to 1 minute).
> Unfortunately slapd then segv's.  Working on getting a test database to
> reproduce the issue with for a good backtrace.

oh, that's too bad. I'd be happy to participate in the bug hunt. May I
ask: which version did your customer use? I have modified the patch to
work with the recent changes on master, but I did not dig in very deep
to understand these changes. So I must admit that these modifications
to the patch are completely untested (except the tests in the tests
subdir).

Things might work better with the 2.4.46/47 patch, if that is an
option for your customer.

Wrt getting a test-tree, maybe the python script I have in the github
repo for the patch can provide
one. 
(https://github.com/hbo/openldap-mdb-deref-problem/blob/master/scripts/write_tree.py)

I'm of course very interested to keep this patch alive, so if you have
details on the segv, please let me know.

Henrik


> 
> --Quanah
> 
> 
> --
> 
> Quanah Gibson-Mount
> Product Architect
> Symas Corporation
> Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
> 

-- 
Dr. rer. nat. Henrik Bohnenkamp

Senior Sysadmin
IT Operations Monitoring & Infrastructure

1&1 IONOS SE | Brauerstraße 48 | 76135 Karlsruhe | Germany
Phone: +49 721 91374 - 4159
E-mail: henrik.bohnenk...@ionos.com | Web: www.ionos.de

Hauptsitz Montabaur, Amtsgericht Montabaur, HRB 24498

Vorstand: Dr. Christian Böing, Hüseyin Dogan, Hans-Henning Kettler, Matthias 
Steinberg, Achim Weiß
Aufsichtsratsvorsitzender: Markus Kadelke

Member of United Internet

Diese E-Mail kann vertrauliche und/oder gesetzlich geschützte Informationen 
enthalten. Wenn Sie nicht der bestimmungsgemäße Adressat sind oder diese 
E-Mail irrtümlich erhalten haben, unterrichten Sie bitte den Absender und 
vernichten Sie diese E-Mail. Anderen als dem bestimmungsgemäßen Adressaten 
ist untersagt, diese E-Mail zu speichern, weiterzuleiten oder ihren Inhalt auf 
welche Weise auch immer zu verwenden.

This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient of this e-mail, you are hereby notified that saving, 
distribution or use of the content of this e-mail in any way is prohibited. If 
you have received this e-mail in error, please notify the sender and delete the 
e-mail.

ITS#9047 published: SECURITY: Coverity report on LMDB

2019-07-08 Thread openldap-its 
Private ITS#9047 is now publicly readable

URL: http://www.OpenLDAP.org/its/?findid=9047

Re: (ITS#8977) Make IDL sizes configurable in back-mdb

2019-07-08 Thread quanah 
--On Thursday, June 27, 2019 8:56 PM + qua...@symas.com wrote:

> --On Thursday, June 27, 2019 8:35 PM + h...@symas.com wrote:
>
>> No, because order is irrelevant for these.
>
> Cool, thanks!  I'll continue on with deeper testing then. :)

Given the current implementation of OpenLDAP, this feature is impossible to 
use w/o recompiling OpenLDAP when a change to the IDL size is made.  This 
is because LDAP_PVT_THREAD_STACK_SIZE must be adjusted as well and that 
requires a recompile.

The default size for LDAP_PVT_THREAD_STACK_SIZE is:

( 1 * 1024 * 1024 * sizeof(void *) )

which works for an IDL size of 16 (2^16) which is 65536.

If you change the IDL size, say to 22, then the new IDL size is: 4,194,304. 
We then use this difference to find the offset we need to adjust 
LDAP_PVT_THREAD_STACK_SIZE by:

4194304/65536 = 64

So it needs to be 64 time larger:

( 64 * 1024 * 1024 * sizeof(void *) )


Generally, this feature is simply unusuable (currently) as a tunable given 
the requirement for recompiling OpenLDAP to use it.

--QUanah


--

Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP: