Well, To resume my tests:
provider and consumer in 2.4.24 -> same behaviour Get rid of session log -> same behaviour If I removed one of the providers of my consumer, everything went fine. Even, if I stop the consumer during mass deletes, it sync very fast as soon as it starts again That's related to several providers for one consumer and may be something else I don't see. The second provider (first rid in my conf) has less that 15 entries and no updates at all. I changed the order of the providers in my slapd.conf. There is no difference. Dom Mar 24 11:36:32 ldaprelay slapd[5444]: slapd starting Mar 24 11:36:32 ldaprelay slapd[5444]: do_syncrep2: rid=020 LDAP_RES_INTERMEDIATE - SYNC_ID_SET Mar 24 11:36:32 ldaprelay slapd[5444]: do_syncrep2: rid=020 LDAP_RES_INTERMEDIATE - SYNC_ID_SET Mar 24 11:36:32 ldaprelay slapd[5444]: do_syncrep2: rid=040 LDAP_RES_INTERMEDIATE - REFRESH_DELETE Mar 24 11:36:32 ldaprelay slapd[5444]: do_syncrep2: rid=020 LDAP_RES_INTERMEDIATE - SYNC_ID_SET Mar 24 11:36:33 ldaprelay slapd[5444]: last message repeated 200 times Mar 24 11:36:33 ldaprelay slapd[5444]: do_syncrep2: rid=020 LDAP_RES_INTERMEDIATE - REFRESH_PRESENT Mar 24 11:36:33 ldaprelay slapd[5444]: do_syncrep2: rid=020 cookie=rid=020,sid=020,csn=20110324103606.185647Z#000000#020#000000 Mar 24 11:36:33 ldaprelay slapd[5444]: slap_queue_csn: queing 0x227dfe0 20110324103606.185647Z#000000#020#000000 Mar 24 11:36:33 ldaprelay slapd[5444]: slap_graduate_commit_csn: removing 0x227e0a0 20110324103606.185647Z#000000#020#000000 # consumer # big provider and mass deletes serverid 0x20 ldap://anutest.univ.fr/ syncrepl rid=020 provider=ldap://anutest.univ.fr/ type=refreshAndPersist searchbase="dc=univ,dc=fr" retry="60 10 300 +" scope=sub filter="(objectClass=*)" attrs="*,+" schemachecking=off bindmethod=simple # small provider serverid 0x40 ldap://ldapmaitre.univ-a.fr/ syncrepl rid=040 provider=ldap://ldapmaitre.univ-a.fr/ type=refreshAndPersist searchbase="dc=univ-a,dc=fr" retry="60 10 300 +" scope=sub filter="(objectClass=*)" attrs="*,+" schemachecking=off bindmethod=simple context on the servers # univ.fr dn: dc=univ,dc=fr contextCSN: 20110324103606.185647Z#000000#020#000000 # univ-a.fr dn: dc=univ-a,dc=fr contextCSN: 20110323101636.221613Z#000000#040#000000 dn: dc=fr contextCSN: 20110324103606.185647Z#000000#020#000000 contextCSN: 20110323101636.221613Z#000000#040#000000 2011/3/24 LALOT Dominique <[email protected]> > Here is my provider, and we use syncprov-sessionlog > > database bdb > suffix "dc=univ-yy,dc=fr" > directory /var/ldap/base/ > sizelimit -1 > cachesize 50000 > dbconfig set_cachesize 0 140000000 1 > dbconfig set_flags DB_LOG_AUTOREMOVE > dbconfig set_tas_spins 1 > dbconfig set_lg_regionmax 262144 > idlcachesize 150000 > checkpoint 1024 5 > # flush tous les 1Mo ou 5 minutes > #dbnosync > syncprov-checkpoint 100 10 > syncprov-sessionlog 100 > index objectClass,entryCSN,entryUUID eq > > ... > > Should we use a session log? I'll try without. That's tricky to understand > each parameter. > Would be great to have something anaylizing more than syntax, relations > between parameters and database cache optimization. > > > Dom > > > > 2011/3/23 Howard Chu <[email protected]> > >> LALOT Dominique wrote: >> >>> Howard, >>> >>> I was obliged to remove slapd package on consumer. Then compile in 2.4.24 >>> and >>> restart. doing the same tests, there was nothing diffferent. provider is >>> still >>> 2.4.23 >>> My test: delete 30000 entries, stop consumer when deleting; start again >>> consumer when it's finished. Consumer is then out of sync. >>> >> >> You haven't posted your provider config. It appears you're using >> syncprov-sessionlog. Obviously both ends of this system are vital to solving >> the puzzle. >> >> Mar 23 17:27:50 ldaprelay slapd[4358]: slapd stopped. >>> Mar 23 17:34:45 ldaprelay slapd[4577]: @(#) $OpenLDAP: slapd 2.4.24 (Mar >>> 23 >>> 2011 16:52:04) >>> $#012#[email protected]: >>> /usr/local/src/openldap-2.4.24/servers/slapd >>> Mar 23 17:34:45 ldaprelay slapd[4579]: slapd starting >>> Mar 23 17:34:45 ldaprelay slapd[4579]: do_syncrep2: rid=040 >>> LDAP_RES_INTERMEDIATE - REFRESH_DELETE >>> Mar 23 17:34:45 ldaprelay slapd[4579]: do_syncrep2: rid=020 >>> LDAP_RES_INTERMEDIATE - SYNC_ID_SET >>> Mar 23 17:34:48 ldaprelay slapd[4579]: last message repeated 175 times >>> Mar 23 17:34:48 ldaprelay slapd[4579]: do_syncrep2: rid=020 >>> LDAP_RES_INTERMEDIATE - REFRESH_PRESENT >>> Mar 23 17:34:48 ldaprelay slapd[4579]: do_syncrep2: rid=020 >>> cookie=rid=020,sid=020,csn=20110323163416.105518Z#000000#020#000000 >>> Mar 23 17:34:48 ldaprelay slapd[4579]: slap_queue_csn: queing 0x1a35560 >>> 20110323163416.105518Z#000000#020#000000 >>> Mar 23 17:34:48 ldaprelay slapd[4579]: slap_graduate_commit_csn: removing >>> 0x1a35620 20110323163416.105518Z#000000#020#000000 >>> >>> I changed the provider to 2.4.24 that makes deletes. Hopefully this one >>> was >>> built on tar.gz >>> That makes no difference, after restarting the consumer, it does not >>> delete >>> extra entries >>> >>> Dom >>> >>> 2011/3/23 LALOT Dominique <[email protected] <mailto: >>> [email protected]>> >>> >>> >>> Hi Howard, >>> >>> We were told to migrate to 2.4.23 sometimes ago, and we did some work >>> to >>> update our production servers. Can I try 2.4.24 only on the consumer >>> side? >>> It would be a pain to migrate all servers to 2.4.24 without package. >>> >>> is this related to the last fixes? >>> >>> Fixed slapd syncrepl reuse of presence list (ITS#6707) >>> Fixed slapd syncrepl uninitialized return code (ITS#6719) >>> Fixed slapd syncrepl variable initialization (ITS#6739) >>> >>> >>> Fixed slapd syncrepl refresh to use complete cookie (ITS#6807) >>> >>> Thanks >>> >>> Dom >>> >>> >>> 2011/3/23 Howard Chu <[email protected] <mailto:[email protected]>> >>> >>> >>> LALOT Dominique wrote: >>> >>> Hello, >>> >>> I am testing the replication feature in a multimaster >>> environment >>> replicating >>> into a single database. As stated before, I added serverid to >>> my >>> providers. I >>> just have two providers for test purpose. >>> I tested mass updates on a provider, stopped my replica during >>> updates, then >>> start again and it's OK, it updates the entries >>> If I do the same for mass deletes. I deleted 40000 entries >>> while >>> stopping the >>> consumer. My consumer is still with 30000 undeleted entries. I >>> left the >>> consumer for hours, restarting it twice. >>> It seems there is no regular compare between consumer or >>> provider >>> in such >>> situation. I'll simplify to test in a single provider setup, >>> to >>> see if it works. >>> >>> All servers are 2.4.23 >>> >>> >>> Please try your test with 2.4.24 instead. >>> >>> >> -- >> -- Howard Chu >> CTO, Symas Corp. http://www.symas.com >> Director, Highland Sun http://highlandsun.com/hyc/ >> Chief Architect, OpenLDAP http://www.openldap.org/project/ >> > > > > -- > Dominique LALOT > Ingénieur Systèmes et Réseaux > http://annuaire.univmed.fr/showuser.php?uid=lalot > -- Dominique LALOT Ingénieur Systèmes et Réseaux http://annuaire.univmed.fr/showuser.php?uid=lalot
