[Freeipa-users] replicating cn=accounts, dc=ipa, dc=example, dc=com tree to a read-only instance of 389ds on our mailserver
Hi guys, our current setup consists of 3 replicated free-ipa servers in a master-master configuration. What we are currently trying to do, is to add a standalone 389-ds on our mailserver which should only readonly-replicate cn=accounts,dc=ipa,dc=example,dc=com to enable our mailserver to have a local ldap cache (for alias/mailbox mapping in postfix/dovecot) and to be able to add a local ldap-addressbook to our mailserver without the need to have it on our ipa-servers. Our environment is: 3 free-ipa servers (centos7, 389-ds-base.x86_64 1.3.5.10-20.el7_3) 1 Mailserver (debian stretch, 389-ds 1.3.5.15-2) What we did do: Basically following this guide: https://access.redhat.com/documentation/en-us/red_hat_directory_server/10/html/administration_guide/Managing_Replication-Configuring-Replication-cmd#Configuring-Replication-Suppliers-cmd on consumer (our mailserver): ...first we created the missing root (cn=accounts,dc=ipa,dc=example,dc=com) by hand # readonly replication manager dn: cn=readonly replication manager,cn=config objectClass: inetorgperson objectClass: person objectClass: top cn: readonly replication manager sn: RORM userPassword: NotTheRealPassword passwordExpirationTime: 20380119031407Z nsIdleTimeout: 0 Replication Entry: # no dc=ipa in the dn! dn: cn=replica,cn=dc\=example\,dc\=com,cn=mapping tree,cn=config changetype: add objectclass: top objectclass: nsds5replica objectclass: extensibleObject cn: replica nsds5replicaid: 65535 nsds5replicaroot: cn=accounts,dc=ipa,dc=example,dc=com nsds5replicatype: 2 nsds5ReplicaPurgeDelay: 604800 nsds5ReplicaBindDN: cn=replication manager,cn=config nsds5flags: 1 # on supplier (one of our IPA-servers) # on our IPA-servers, dc=ipa is included dn: cn=accountsToMail,cn=replica,cn=dc\=ipa\,dc\=example\,dc\=com,cn=mapping tree,cn=config objectclass: top objectclass: nsds5ReplicationAgreement cn: accounts2hermes nsds5replicahost: mail.example.com nsds5replicaport: 389 nsds5ReplicaBindDN: cn=readonly replication manager,cn=config nsds5replicabindmethod: SIMPLE nsds5replicaroot: cn=accounts,dc=ipa,dc=example,dc=com description: replicate cn=accounts from ipa to hermes nsds5replicatedattributelist: (objectclass=*) $ EXCLUDE authorityRevocationList accountUnlockTime memberof nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE accountUnlockTime nsds5replicacredentials: notTheRealButSameAsAbove nsds5ReplicaIgnoreMissingChange: once nsds5BeginReplicaRefresh: start After some log-entries regarding the schema versions, we stopped the consumer and copied the schema from the supplier to the consumer by hand... This fixed most of the noise in the log, but we are still getting the following error: [18/May/2017:10:23:41.311816674 +0200] NSMMReplicationPlugin - agmt="cn=accountsToMail" (mail:389): The remote replica has a different database generation ID tha n the local database. You may have to reinitialize the remote replica, or the local replica. Of course, we tried to re-initialize the remote-replica by, dn: cn=accountsToMail,cn=replica,cn=dc\=ipa\,dc\=example\,dc\=com,cn=mapping tree,cn=config changetype: modify replace: nsds5BeginReplicaRefresh nsds5BeginReplicaRefresh: start What are we missing? Best regards, Bernhard -- Bernhard Kneip Systemadministration E-Mail: bernhard.kn...@isa.de.com Tel: +49(0)3677/46929-144 Internet: www.isa.de.com ISA Institut für Serviceautomation GmbH & Co. KG Ziolkowskistraße 8, 98693 Ilmenau Amtsgericht Jena, HRA 301735 persönlich haftende Gesellschafterin: ISA GmbH Amtsgericht Jena, HRB 306708 Geschäftsführer: Dr.-Ing. Walther Spies, Dipl.-Ing. (FH) Peter Mayer Member of SIELAFF GROUP -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] Replica cannot be reinitialized after upgrade
On 05/18/2017 10:13 PM, Goran Marik wrote: Thanks Ludwig for the suggestion and thanks to Maciej for the confirmation from his end. This issue is happening for us for several weeks, so I don’t think this is a transient problem. What is the best way to sanitize the logs without removing useful info before sending them your way? Will the files mentioned on "https://www.freeipa.org/page/Files_to_be_attached_to_bug_report -> Directory server failed" be sufficient? yes, but we need soem additional info on the replication config and state, you could add /etc/dirsrv/slapd-*/dse.ldif and the result of these query ldapsearch -o ldif-wrap=no -D "cn=directory manager" ... -b "cn=config" "objectclass=nsds5replica" \* nsds50ruv But looking again at the csn reorted missing it is from June, 2016. So I wonder if this is for an stale/removed replica and cleaning the ruvs would help I’ve also run the ipa_consistency_check script, and the output shows that something is indeed wrong with the sync: “”” FreeIPA servers:inf01inf01inf02inf02STATE = Active Users15 15 15 15 OK Stage Users 0000OK Preserved Users 3333OK User Groups 9999OK Hosts 45 45 45 46 FAIL Host Groups 7777OK HBAC Rules 6666OK SUDO Rules 7777OK DNS Zones 33 33 33 33 OK LDAP Conflicts NO NO NO NO OK Ghost Replicas 2222FAIL Anonymous BIND YES YES YES YES OK Replication Status inf01.prod 0inf01.dev 0inf01.dev 0inf01.dev 0 inf02.dev 0inf02.dev 0inf01.prod 0inf01.prod 0 inf02.prod 0inf02.prod 0inf02.prod 0inf02.dev 0 = “”” Thanks, Goran On May 15, 2017, at 6:35 AM, Ludwig Krispenz wrote: The messages you see could be transient messages, and if replication is working than this seems to be the case. If not we would need more data to investigate: deployment info, relicaIDs of all servers, ruvs, logs,. Here is some background info: there are some scenarios where a csn could not be found in the changelog, eg if updates were aplied on the supplier during a total init, they could be part of the data and database ruv, but not in the changelog of the initialized replica. ds did try to use an alternative csn in cases where it could not be found, but this had the risk of missing updates, so we decided to change it and make this misssing csn a non fatal error, backoff and retry, if another supplier would have updated the replica in between, the starting csn could have changed and be found. so if the reported missing csns change and replication continues everything is ok, although I think the messages should stop at some point. There is a configuration parameter for a replciation agreement to trigger the previous behaviour of picking an alternative csn: nsds5ReplicaIgnoreMissingChange with potential values "once", "always". where "once" just tries to kickstart replication by using another csn and "always" changes the default behaviour On 05/11/2017 06:53 PM, Goran Marik wrote: Hi, After an upgrade to Centos 7.3.1611 with “yum update", we started seeing the following messages in the logs: “”” May 9 21:58:28 inf01 ns-slapd[4323]: [09/May/2017:21:58:28.519724479 +] NSMMReplicationPlugin - changelog program - agmt="cn=cloneAgreement1-inf02.dev.ecobee.com-pki-tomcat" (inf02:389): CSN 576b34e8000a050f not found, we aren't as up to date, or we purged May 9 21:58:28 inf01 ns-slapd[4323]: [09/May/2017:21:58:28.550459233 +] NSMMReplicationPlugin - agmt="cn=cloneAgreement1-inf02.dev.ecobee.com-pki-tomcat" (inf02:389): Data required to update replica has been purged from the changelog. The replica must be reinitialized. May 9 21:58:32 inf01 ns-slapd[4323]: [09/May/2017:21:58:32.588245476 +] agmt="cn=cloneAgreement1-inf02.dev.ecobee.com-pki-tomcat" (inf02:389) - Can't locate CSN 576b34e8000a050f in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized. May 9 21:58:32 inf01 ns-slapd[4323]: [09/May/2017:21:58:32.611400689 +] NSMMReplicationPlugin - changelog program - agmt="cn=cloneAgreement1-inf02.dev.ecobee.com-pki-tomcat" (inf02:389): CSN 576b34e8000a050f not found, we aren't as up to date, or we purged May 9 21:58:32 inf01 ns-slapd[4323]: [09/May/2017:21:58:32.642226385 +] NSMMReplicationPlugin - agmt="cn=cloneAgreement1-inf02.dev.ecobee.com-pki-tomcat" (inf02:389): Data required to update replica has been purged from the chang
[Freeipa-users] replicating cn=accounts, dc=ipa, dc=example, dc=com tree to a read-only instance of 389ds on our mailserver
Hi guys, our current setup consists of 3 replicated free-ipa servers in a master-master configuration. What we are currently trying to do, is to add a standalone 389-ds on our mailserver which should only readonly-replicate cn=accounts,dc=ipa,dc=example,dc=com to enable our mailserver to have a local ldap cache (for alias/mailbox mapping in postfix/dovecot) and to be able to add a local ldap-addressbook to our mailserver without the need to have it on our ipa-servers. Our environment is: 3 free-ipa servers (centos7, 389-ds-base.x86_64 1.3.5.10-20.el7_3) 1 Mailserver (debian stretch, 389-ds 1.3.5.15-2) What we did do: Basically following this guide: https://access.redhat.com/documentation/en-us/red_hat_directory_server/10/html/administration_guide/Managing_Replication-Configuring-Replication-cmd#Configuring-Replication-Suppliers-cmd on consumer (our mailserver): ...first we created the missing root (cn=accounts,dc=ipa,dc=example,dc=com) by hand # readonly replication manager dn: cn=readonly replication manager,cn=config objectClass: inetorgperson objectClass: person objectClass: top cn: readonly replication manager sn: RORM userPassword: NotTheRealPassword passwordExpirationTime: 20380119031407Z nsIdleTimeout: 0 Replication Entry: # no dc=ipa in the dn! dn: cn=replica,cn=dc\=example\,dc\=com,cn=mapping tree,cn=config changetype: add objectclass: top objectclass: nsds5replica objectclass: extensibleObject cn: replica nsds5replicaid: 65535 nsds5replicaroot: cn=accounts,dc=ipa,dc=example,dc=com nsds5replicatype: 2 nsds5ReplicaPurgeDelay: 604800 nsds5ReplicaBindDN: cn=replication manager,cn=config nsds5flags: 1 # on supplier (one of our IPA-servers) # on our IPA-servers, dc=ipa is included dn: cn=accountsToMail,cn=replica,cn=dc\=ipa\,dc\=example\,dc\=com,cn=mapping tree,cn=config objectclass: top objectclass: nsds5ReplicationAgreement cn: accounts2hermes nsds5replicahost: mail.example.com nsds5replicaport: 389 nsds5ReplicaBindDN: cn=readonly replication manager,cn=config nsds5replicabindmethod: SIMPLE nsds5replicaroot: cn=accounts,dc=ipa,dc=example,dc=com description: replicate cn=accounts from ipa to hermes nsds5replicatedattributelist: (objectclass=*) $ EXCLUDE authorityRevocationList accountUnlockTime memberof nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE accountUnlockTime nsds5replicacredentials: notTheRealButSameAsAbove nsds5ReplicaIgnoreMissingChange: once nsds5BeginReplicaRefresh: start After some log-entries regarding the schema versions, we stopped the consumer and copied the schema from the supplier to the consumer by hand... This fixed most of the noise in the log, but we are still getting the following error: [18/May/2017:10:23:41.311816674 +0200] NSMMReplicationPlugin - agmt="cn=accountsToMail" (mail:389): The remote replica has a different database generation ID tha n the local database. You may have to reinitialize the remote replica, or the local replica. Of course, we tried to re-initialize the remote-replica by, dn: cn=accountsToMail,cn=replica,cn=dc\=ipa\,dc\=example\,dc\=com,cn=mapping tree,cn=config changetype: modify replace: nsds5BeginReplicaRefresh nsds5BeginReplicaRefresh: start What are we missing? Best regards, Bernhard -- Bernhard Kneip Systemadministration E-Mail: bernhard.kn...@isa.de.com Tel: +49(0)3677/46929-144 Internet: www.isa.de.com ISA Institut für Serviceautomation GmbH & Co. KG Ziolkowskistraße 8, 98693 Ilmenau Amtsgericht Jena, HRA 301735 persönlich haftende Gesellschafterin: ISA GmbH Amtsgericht Jena, HRB 306708 Geschäftsführer: Dr.-Ing. Walther Spies, Dipl.-Ing. (FH) Peter Mayer Member of SIELAFF GROUP -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] Cant locate CSN after yum update
Dear Ludwig, Thank you for the explanations. Now I understand. Strangely then, the problem csn was on the replica that we had to reinitialize. How could such a csn disappear? Thanks again for the help. Much appreciated. Sent from my iPhone > On 19 May 2017, at 08:47, Ludwig Krispenz wrote: > > >> On 05/18/2017 05:35 PM, Christophe TREFOIS wrote: >> Dear Ludwig, >> >> Thanks for your help in IRC to guide me in running the right commands. >> >> Here is the output, toto1 and toto2 are CA master, and toto3 and toto4 are >> non CA master. The problematic replica was toto3, and after re-init, we >> haven’t seen any errors in the log anymore. >> >> https://paste.fedoraproject.org/paste/j8c30CZPyh8rPymjbKSvZF5M1UNdIGYhyRLivL9gydE= >> >> I also ran ipa-replica-manage on all replicas to all replicas, so total of >> 16 command, and found all of them reported “incremental update succeeded”. >> >> As discussed, I’m not sure what I’m looking at with the RUV stuff above, and >> any explanation for a newcomer to ldap / ds / freeipa would be greatly >> appreciated. > ok, here is a quick explanation of the csn/ruv stuff. > > each change applied on a server gets a CSN (change sequence number), it > basically consists of a timestamp and an identifier of the replica where it > was originally applied, so in 59095fe1000b0012 there is a time stamp: > 59095fe1 and a replicaid: 0012 == 18, the rest of the csn isused to serialize > csns within the one second resolution of a timestamp. > a change is applied to the main database and written to the changelog, with > the csn as key. > > now each replica keeps track of the latest csn it has seen for each > replicaID, so you get a vector of max csns, this is called RUV (replica > update vector). > In a replication session, the supplier compares its own ruv with the ruv of > the consumer and so decides if it has changes which the consumer has not yet > seen. > based on the consumer ruv it determines the start csn to send updates. > > >> >> Thanks a lot for your help! >> >> Kind regards, >> Christophe aka Trefex >> >>> On 18 May 2017, at 17:04, Christophe TREFOIS >>> wrote: >>> >>> Hi Ludwig, >>> >>> Since we were scared, we did a full re-init of that specific replica from >>> the CA master, and it looks like the issue is not appearing anymore. >>> >>> Is this sufficient, or should we still investigate ? >>> >>> Thanks for your help! >>> Christophe >>> -- >>> >>> Dr Christophe Trefois, Dipl.-Ing. >>> Technical Specialist / Post-Doc >>> >>> UNIVERSITÉ DU LUXEMBOURG >>> >>> LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE >>> Campus Belval | House of Biomedicine >>> 6, avenue du Swing >>> L-4367 Belvaux >>> T: +352 46 66 44 6124 >>> F: +352 46 66 44 6949 >>> http://www.uni.lu/lcsb >>> >>> >>> >>> >>> >>> This message is confidential and may contain privileged information. >>> It is intended for the named recipient only. >>> If you receive it in error please notify me and permanently delete the >>> original message and any copies. >>> >>> >>> On 18 May 2017, at 16:11, Ludwig Krispenz wrote: hi, there was a change that in the case of a missing csn ds would not silently use a "close" one and continue, but log an error, backoff and retry - after updates on other masters the staring csn coudl change and replication continue. Now, in your case the csn reported missing: 59095fe1000b0012 has a time stamp from May,3rd, so it could very well be correct that this csn is no longer found in the changelog. To continue analysis, could you provide the replicaids of all your current replicas, and which is the replicaid of the sever logging the change and the ruvs of the replicas from all servers. ldapsearch -D "cn=directory manager" -b cn=config "objectclass=nsds5replica" nsds50ruv Regards, Ludwig > On 05/18/2017 03:09 PM, Christophe TREFOIS wrote: > Hi all, > > Did a yum update on one of my replicas, non CA master, and upgrade was > successful (ipupgrade.log) said so. > > > Hwoever, now every few seconds I get the following message. > https://paste.fedoraproject.org/paste/wS4x9KvD3EB0gv2HAsj6X15M1UNdIGYhyRLivL9gydE= > > Does anybody know how to proceed and if this is important? > ipa-replica-manage says, backing off, retrying later, so not sure if > replication happens successfully or not and what to do ?? > > Setup: CentOS 7.3 all up-to-date, 2 CA master, 2 non CA master in diamond > replication. > > Remaining replicas were upgraded today as well, and don’t seem to > complain. Only 1 of them complains. > > 389-ds-base-libs-1.3.5.10-20.el7_3.x86_64 > 389-ds-base-1.3.5.10-20.el7_3.x86_64 > > > [root@lums3 ~]# rpm -qa | grep ipa > libipa_hbac-1.14.0-43.el7_3.14.x86_64 > p