On 06/18/2012 01:55 PM, [email protected] wrote: > Jan Synacek wrote: >> On 06/15/2012 03:16 PM, [email protected] wrote: >>> [email protected] wrote: >>>> Full_Name: Jan Synacek >>>> Version: git (c73ec15) >>>> OS: linux-fedora17 >>>> URL: >>>> http://jsynacek.fedorapeople.org/openldap/leak/openldap-mmr-leak.tar.gz >>>> Submission from: (NULL) (209.132.186.34) >>>> >>>> >>>> I'm using a 2-node mmr setup on my local machine - configuration files and >>>> 'uploader' scripts are provided in the archive. >>>> >>>> 1) I have the two nodes running. >>>> 2) Execute run.sh (only a wrapper for ldapusradm.sh) and start monitoring >>>> slapd's memory usage. >>>> 3) After some time (at about 2k users on my system), slapd consumes a large >>>> amount of memory which is still growing >>>> >>>> Note that not using ldapmodify to add members to 'cn=users,dc=yes,dc=my', >>>> but >>>> using it e.g. for modifying each user's email, does NOT result in any >>>> memory >>>> leakage. >>>> >>>> I have also created a massif output using valgrind's massif tool: >>>> http://jsynacek.fedorapeople.org/openldap/leak/massif.out.17906 >>>> >>>> I found a very similar bug (#7292), but I'm not sure if it's related. >>> >>> Running RE24 with valgrind --leak-check=full I see no leak when running your >>> test. That should be the same as git c73ec15. No idea what leak you're >>> seeing. >>> >> >> The memory consumption still grows and slapd eventually gets killed by >> oom-killer. > > Perhaps you're seeing fragmentation in the glibc malloc library. Please test > RE24, and try with some other malloc (e.g. tcmalloc). I've run your entire > test with both valgrind and with tcmalloc's heap checker and neither one > shows > any leak. >
I've done more tests and have some additional observations to add: * tcmalloc doesn't make any difference * if syncing finishes, memory usage drops considerably * putting sleep between ldapadd and ldapmodify in the script seems to stop the memory consumption * also, generating one big ldif, that is used modify all the users in groupOfNames just once (calling ldapmodify only once that is, not calling it once for every user), seems to help as well; the ldif is formatted like so: dn: cn=users,dc=yes,dc=my changetype:modify add: member member: [email protected],dc=yes,dc=my - add: member member: [email protected],dc=yes,dc=my - ... I'm also including my configure setup: export CFLAGS="$(rpm --eval %{optflags}) -O2 -fPIC -D_REENTRANT -DHAVE_TLS -DHAVE_MOZNSS" export LIBS="-lpthread -lssl3 -lsmime3 -lnss3 -lnssutil3 -lplds4 -lplc4 -lnspr4" CFLAGS="${CFLAGS:--g -pipe -Wall }"; export CFLAGS; CXXFLAGS="${CXXFLAGS:--g -pipe -Wall }"; export CXXFLAGS; FFLAGS="${FFLAGS:--g -pipe -Wall }"; export FFLAGS; ./configure \ --with-threads=posix --disable-static --with-tls=no \ --with-pic --with-cyrus-sasl \ --enable-bdb \ --enable-overlays --enable-slapd --enable-spasswd --enable-modules --enable-slapi \ --enable-accesslog \ --enable-wrappers -- Jan Synacek Software Engineer, BaseOS team Brno, Red Hat
