> From: Rob Crittenden 
> Martin Babinsky wrote:
> > On 12/01/2016 01:28 PM, Neal Harrington | i-Neda Ltd wrote:
> >> Hi IPA Gurus,
> >>
> >>
> >> I had a 3 site multi master IPA replication setup (1 office and 2
> >> datacentres) with 2 IPA servers at each site. Each server was
> >> replicating successfully to 3 other servers (the other local site
> >> server and one server at each of the two remote sites). Everything is
> >> running on the default packages from CentOS 7.2 and each server is a
> >> full replica (ipa-replica-install
> >> /var/lib/ipa/replica-info-id-myserver.fqdn.com.gpg  --setup-ca
> >> --setup-dns --mkhomedir --forwarder 8.8.8.8)
> >>
> >>
> >> Everything was ticking over nicely until we had notice that the
> >> office site was moving on short notice.
> >>
> >>
> >> I successfully created IPA servers at the new site, setup replication
> >> again between the new office and the two datacentres that were to
> >> remain online, tested and everything worked as expected -
> >> unfortunately in the rush I did not have time to properly retire the
> >> IPA servers in the old office.
> >>
> >>
> >> The problem this has caused is that I only ever created users in one
> >> of the IPA servers in the original office - so only those servers
> >> have a DNA range and I am now unable to create new users on the active
> servers.
> >> The original office servers are still in the IPA replication and
> >> powered on but offline so potential split brain?
> >>
> >>
> >> I now have two things I would like to know before proceeding:
> >>
> >>   * Is the best fix here to force remove the original IPA servers and
> >>     manually add a new dna range significantly different from the
> >>     original to avoid overlaps?
> >>   * Is there anything else I should check? I can't see any issues
> >>     however did not notice the DNA range until I tried to create a user.
> >>
> >> Any pointers greatly appreciated.
> >>
> >>
> >> Thanks,
> >>
> >> Neal.
> >>
> >>
> >>
> >>
> >>
> >>
> >
> > Hi Neal,
> >
> > If you already disconnected/decomissioned the old masters then I thnk
> > the best you can do is option a, i.e. re-set DNA ranges on replicas to
> > new values while avioding overlap with old ranges.
> >
> > We have an upstream document[1] describing the procedure. Hope it
> helps.
> >
> > Also make sure that you migrated CA renewal and CRL master
> > responsibilities to the new replicas, otherwise you may get problems
> > with expiring certificates which are really hard to solve. See the
> > following guide for details. [2]
> >
> > [1] http://www.freeipa.org/page/V3/Recover_DNA_Ranges
> > [2]
> >
> http://www.freeipa.org/page/Howto/Promote_CA_to_Renewal_and_CRL_
> Master
> >
> 
> You may want to look at this too, http://blog-rcritten.rhcloud.com/?p=50
> 
> rob

Hi Rob & Martin,

Thanks for the pointers, I am now able to create new users on different servers 
- however everything to do with replication seems to be failing.

I have changed my replication from a mesh to a long chain and run 
"ipa-replica-manage -v re-initialize --from <server>" and the same for 
ipa-csreplica-manage along the chain which succeeds (and any passwords/user 
creation etc I have done at the start of the chain is pulled through) however 
replication fails immediately after. I was hoping that re-initializing the 
chain like this would flush out any "bad" entries - probably wishful thinking.

"Ipa-replica-manage -v list" only shows servers in the chain. 
"Ipa-replica-manage list-ruv" did show the two original servers which I lost 
connection to and I removed those which successfully removed them from all 
servers so that part of replication seems to be working. When I do an LDAP 
search I still see those old masters though (and also see one previously 
retired server with two different ID's - blue-auth01). Will I need to manually 
delete these? (example search and output below)

Apart from manually deleting the dead servers from LDAP, what else should I do 
to get replication working again? I'm watching for the CentOS 7.3 release to be 
able to upgrade to IPA 4.3 as I've seen a few posts about the better handling 
of replication etc in that version. In the meantime the errors log (copy below) 
indicates I need to re-initialize which I've done several times without any 
improvement.

Thanks in advance,
Neal.

[root@office-auth04 ~]# ldapsearch -h $(hostname -f)  -D "cn=directory manager" 
-W  -b "o=ipaca" 
"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" 
nscpentrywsi
Enter LDAP Password: 
# extended LDIF
#
# LDAPv3
# base <o=ipaca> with scope subtree
# filter: 
(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))
# requesting: nscpentrywsi 
#

# replica, o\3Dipaca, mapping tree, config
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nscpentrywsi: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nscpentrywsi: cn: replica
nscpentrywsi: createTimestamp: 20161122150144Z
nscpentrywsi: creatorsName: cn=directory manager
nscpentrywsi: modifiersName: cn=Multimaster Replication 
Plugin,cn=plugins,cn=config
nscpentrywsi: modifyTimestamp: 20161204152409Z
nscpentrywsi: nsDS5Flags: 1
nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager 
cloneAgreement1-office-auth04.int.i-neda.com-pki-tomcat,ou=csusers,cn=config
nscpentrywsi: nsDS5ReplicaBindDN: cn=replication manager,cn=config
nscpentrywsi: nsDS5ReplicaId: 1495
nscpentrywsi: nsDS5ReplicaName: 725aa31e-b0c411e6-b5d989ac-8f24d4e5
nscpentrywsi: nsDS5ReplicaRoot: o=ipaca
nscpentrywsi: nsDS5ReplicaType: 3
nscpentrywsi: nsState:: 1wUAAAAAAAAXNURYAAAAAAAAAAAAAAAAUCAAAAAAAAACAAAAAAAAAA==
nscpentrywsi: objectClass: top
nscpentrywsi: objectClass: nsDS5Replica
nscpentrywsi: objectClass: extensibleobject
nscpentrywsi: numSubordinates: 1
nscpentrywsi: nsds50ruv: {replicageneration} 575ee7d2000000600000               
                                                                                
                                            [0/694]
nscpentrywsi: nsds50ruv: {replica 1495 ldap://office-auth04.int.i-neda.com:389} 
58347e61000005d70000 58445505000005d70000
nscpentrywsi: nsds50ruv: {replica 91 ldap://power-auth01.int.i-neda.com:389} 
577549ad0000005b0000 583457150008005b0000
nscpentrywsi: nsds50ruv: {replica 86 ldap://power-auth02.int.i-neda.com:389} 
57754d7b000000560000 583469ad000000560000
nscpentrywsi: nsds50ruv: {replica 1395 ldap://blue-auth04.int.i-neda.com:389} 
583469b5000005730000 5841c354000305730000
nscpentrywsi: nsds50ruv: {replica 96 ldap://office-auth01.int.i-neda.com:389} 
575ee96e000000600000 58349d69000400600000
nscpentrywsi: nsds50ruv: {replica 97 ldap://office-auth02.int.i-neda.com:389} 
575ee993000000610000 5820979e000700610000
nscpentrywsi: nsds50ruv: {replica 1095 ldap://blue-auth02.int.i-neda.com:389} 
5783b6fb000004470000 582de13b000d04470000
nscpentrywsi: nsds50ruv: {replica 81 ldap://blue-auth01.int.i-neda.com:389} 
5783b6fc000800510000 5783b719000800510000
nscpentrywsi: nsds50ruv: {replica 1195 ldap://blue-auth01.int.i-neda.com:389} 
5784d185000004ab0000 5784d1af005604ab0000
nscpentrywsi: nsds50ruv: {replica 76 ldap://blue-auth03.int.i-neda.com:389} 
5819fea00000004c0000 58306e950000004c0000
nscpentrywsi: nsds50ruv: {replica 1295 ldap://office-auth03.int.i-neda.com:389} 
582f1d660009050f0000 582f1d7c000a050f0000
nscpentrywsi: nsruvReplicaLastModified: {replica 1495 
ldap://office-auth04.int.i-neda.com:389} 00000000
nscpentrywsi: nsruvReplicaLastModified: {replica 91 
ldap://power-auth01.int.i-neda.com:389} 00000000
nscpentrywsi: nsruvReplicaLastModified: {replica 86 
ldap://power-auth02.int.i-neda.com:389} 00000000
nscpentrywsi: nsruvReplicaLastModified: {replica 1395 
ldap://blue-auth04.int.i-neda.com:389} 00000000
nscpentrywsi: nsruvReplicaLastModified: {replica 96 
ldap://office-auth01.int.i-neda.com:389} 00000000
nscpentrywsi: nsruvReplicaLastModified: {replica 97 
ldap://office-auth02.int.i-neda.com:389} 00000000
nscpentrywsi: nsruvReplicaLastModified: {replica 1095 
ldap://blue-auth02.int.i-neda.com:389} 00000000
nscpentrywsi: nsruvReplicaLastModified: {replica 81 
ldap://blue-auth01.int.i-neda.com:389} 00000000
nscpentrywsi: nsruvReplicaLastModified: {replica 1195 
ldap://blue-auth01.int.i-neda.com:389} 00000000
nscpentrywsi: nsruvReplicaLastModified: {replica 76 
ldap://blue-auth03.int.i-neda.com:389} 00000000
nscpentrywsi: nsruvReplicaLastModified: {replica 1295 
ldap://office-auth03.int.i-neda.com:389} 00000000
nscpentrywsi: nsds5ReplicaChangeCount: 2
nscpentrywsi: nsds5replicareapactive: 0

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1


Extract from /var/log/dirsrv/REALM/errors from power-auth01 which is one down 
from the "start" of the chain (office-auth04):
[07/Dec/2016:12:07:16 +0000] NSMMReplicationPlugin - 
multimaster_be_state_change: replica dc=int,dc=i-neda,dc=com is going offline; 
disabling replication
[07/Dec/2016:12:07:17 +0000] - WARNING: Import is running with 
nsslapd-db-private-import-mem on; No other process is allowed to access the 
database
[07/Dec/2016:12:07:21 +0000] - import userRoot: Workers finished; cleaning up...
[07/Dec/2016:12:07:21 +0000] - import userRoot: Workers cleaned up.
[07/Dec/2016:12:07:21 +0000] - import userRoot: Indexing complete.  
Post-processing...
[07/Dec/2016:12:07:21 +0000] - import userRoot: Generating numsubordinates 
(this may take several minutes to complete)...
[07/Dec/2016:12:07:21 +0000] - import userRoot: Generating numSubordinates 
complete.
[07/Dec/2016:12:07:21 +0000] - import userRoot: Gathering ancestorid non-leaf 
IDs...
[07/Dec/2016:12:07:21 +0000] - import userRoot: Finished gathering ancestorid 
non-leaf IDs.
[07/Dec/2016:12:07:21 +0000] - import userRoot: Creating ancestorid index (new 
idl)...
[07/Dec/2016:12:07:21 +0000] - import userRoot: Created ancestorid index (new 
idl).
[07/Dec/2016:12:07:21 +0000] - import userRoot: Flushing caches...
[07/Dec/2016:12:07:21 +0000] - import userRoot: Closing files...
[07/Dec/2016:12:07:22 +0000] - import userRoot: Import complete.  Processed 777 
entries in 5 seconds. (155.40 entries/sec)
[07/Dec/2016:12:07:22 +0000] NSMMReplicationPlugin - 
multimaster_be_state_change: replica dc=int,dc=i-neda,dc=com is coming online; 
enabling replication
[07/Dec/2016:12:07:22 +0000] NSMMReplicationPlugin - replica_reload_ruv: 
Warning: new data for replica dc=int,dc=i-neda,dc=com does not match the data 
in the changelog.
 Recreating the changelog file. This could affect replication with replica's  
consumers in which case the consumers should be reinitialized.
[07/Dec/2016:12:07:22 +0000] - Skipping CoS Definition cn=Password 
Policy,cn=accounts,dc=int,dc=i-neda,dc=com--no CoS Templates found, which 
should be added before the CoS Definition.
[07/Dec/2016:12:07:22 +0000] NSACLPlugin - The ACL target 
cn=groups,cn=compat,dc=int,dc=i-neda,dc=com does not exist
[07/Dec/2016:12:07:22 +0000] NSACLPlugin - The ACL target 
cn=computers,cn=compat,dc=int,dc=i-neda,dc=com does not exist
[07/Dec/2016:12:07:22 +0000] NSACLPlugin - The ACL target 
cn=ng,cn=compat,dc=int,dc=i-neda,dc=com does not exist
[07/Dec/2016:12:07:22 +0000] NSACLPlugin - The ACL target 
ou=sudoers,dc=int,dc=i-neda,dc=com does not exist
[07/Dec/2016:12:07:22 +0000] NSACLPlugin - The ACL target 
cn=users,cn=compat,dc=int,dc=i-neda,dc=com does not exist
[07/Dec/2016:12:07:22 +0000] NSACLPlugin - The ACL target 
cn=vaults,cn=kra,dc=int,dc=i-neda,dc=com does not exist
[07/Dec/2016:12:07:22 +0000] NSACLPlugin - The ACL target 
cn=vaults,cn=kra,dc=int,dc=i-neda,dc=com does not exist
[07/Dec/2016:12:07:22 +0000] NSACLPlugin - The ACL target 
cn=vaults,cn=kra,dc=int,dc=i-neda,dc=com does not exist
[07/Dec/2016:12:07:22 +0000] NSACLPlugin - The ACL target 
cn=vaults,cn=kra,dc=int,dc=i-neda,dc=com does not exist
[07/Dec/2016:12:07:22 +0000] NSACLPlugin - The ACL target 
cn=vaults,cn=kra,dc=int,dc=i-neda,dc=com does not exist
[07/Dec/2016:12:07:22 +0000] NSACLPlugin - The ACL target 
cn=vaults,cn=kra,dc=int,dc=i-neda,dc=com does not exist
[07/Dec/2016:12:07:22 +0000] NSACLPlugin - The ACL target 
cn=vaults,cn=kra,dc=int,dc=i-neda,dc=com does not exist
[07/Dec/2016:12:07:22 +0000] NSACLPlugin - The ACL target 
cn=vaults,cn=kra,dc=int,dc=i-neda,dc=com does not exist
[07/Dec/2016:12:07:22 +0000] NSACLPlugin - The ACL target 
cn=vaults,cn=kra,dc=int,dc=i-neda,dc=com does not exist
[07/Dec/2016:12:07:22 +0000] NSACLPlugin - The ACL target 
cn=vaults,cn=kra,dc=int,dc=i-neda,dc=com does not exist
[07/Dec/2016:12:07:22 +0000] NSACLPlugin - The ACL target 
cn=vaults,cn=kra,dc=int,dc=i-neda,dc=com does not exist
[07/Dec/2016:12:07:22 +0000] NSACLPlugin - The ACL target 
cn=ad,cn=etc,dc=int,dc=i-neda,dc=com does not exist
[07/Dec/2016:12:07:22 +0000] NSACLPlugin - The ACL target cn=casigningcert 
cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=int,dc=i-neda,dc=com does not exist
[07/Dec/2016:12:07:22 +0000] NSACLPlugin - The ACL target cn=casigningcert 
cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=int,dc=i-neda,dc=com does not exist
[07/Dec/2016:12:07:22 +0000] agmt="cn=meTopower-auth02.int.i-neda.com" 
(power-auth02:389) - Can't locate CSN 5847f3fd0000000d0000 in the changelog (DB 
rc=-30988). If replication stops, the consumer may need to be reinitialized.
[07/Dec/2016:12:07:22 +0000] NSMMReplicationPlugin - changelog program - 
agmt="cn=meTopower-auth02.int.i-neda.com" (power-auth02:389): CSN 
5847f3fd0000000d0000 not found, we aren't as up to date, or we purged
[07/Dec/2016:12:07:22 +0000] NSMMReplicationPlugin - 
agmt="cn=meTopower-auth02.int.i-neda.com" (power-auth02:389): Data required to 
update replica has been purged. The replica must be reinitialized.
[07/Dec/2016:12:07:22 +0000] NSMMReplicationPlugin - 
agmt="cn=meTopower-auth02.int.i-neda.com" (power-auth02:389): Incremental 
update failed and requires administrator action
[07/Dec/2016:12:07:22 +0000] agmt="cn=meTooffice-auth04.int.i-neda.com" 
(office-auth04:389) - Can't locate CSN 584412d7000200050000 in the changelog 
(DB rc=-30988). If replication stops, the consumer may need to be reinitialized.
[07/Dec/2016:12:07:22 +0000] NSMMReplicationPlugin - changelog program - 
agmt="cn=meTooffice-auth04.int.i-neda.com" (office-auth04:389): CSN 
584412d7000200050000 not found, we aren't as up to date, or we purged
[07/Dec/2016:12:07:22 +0000] NSMMReplicationPlugin - 
agmt="cn=meTooffice-auth04.int.i-neda.com" (office-auth04:389): Data required 
to update replica has been purged. The replica must be reinitialized.
[07/Dec/2016:12:07:22 +0000] NSMMReplicationPlugin - 
agmt="cn=meTooffice-auth04.int.i-neda.com" (office-auth04:389): Incremental 
update failed and requires administrator action
[07/Dec/2016:12:07:30 +0000] attrlist_replace - attr_replace (nsslapd-referral, 
ldap://office-auth04.int.i-neda.com:389/o%3Dipaca) failed.
[07/Dec/2016:12:07:30 +0000] attrlist_replace - attr_replace (nsslapd-referral, 
ldap://office-auth04.int.i-neda.com:389/o%3Dipaca) failed.
[07/Dec/2016:12:07:30 +0000] attrlist_replace - attr_replace (nsslapd-referral, 
ldap://office-auth04.int.i-neda.com:389/o%3Dipaca) failed.
[07/Dec/2016:12:07:30 +0000] NSMMReplicationPlugin - 
multimaster_be_state_change: replica o=ipaca is going offline; disabling 
replication
[07/Dec/2016:12:07:31 +0000] - Skipping CoS Definition cn=Password 
Policy,cn=accounts,dc=int,dc=i-neda,dc=com--no CoS Templates found, which 
should be added before the CoS Definition.
[07/Dec/2016:12:07:31 +0000] - WARNING: Import is running with 
nsslapd-db-private-import-mem on; No other process is allowed to access the 
database
[07/Dec/2016:12:07:36 +0000] attrlist_replace - attr_replace (nsslapd-referral, 
ldap://office-auth04.int.i-neda.com:389/o%3Dipaca) failed.
[07/Dec/2016:12:07:36 +0000] attrlist_replace - attr_replace (nsslapd-referral, 
ldap://office-auth04.int.i-neda.com:389/o%3Dipaca) failed.
[07/Dec/2016:12:07:36 +0000] attrlist_replace - attr_replace (nsslapd-referral, 
ldap://office-auth04.int.i-neda.com:389/o%3Dipaca) failed.
[07/Dec/2016:12:07:36 +0000] - import ipaca: Workers finished; cleaning up...
[07/Dec/2016:12:07:36 +0000] - import ipaca: Workers cleaned up.
[07/Dec/2016:12:07:36 +0000] - import ipaca: Indexing complete.  
Post-processing...
[07/Dec/2016:12:07:36 +0000] - import ipaca: Generating numsubordinates (this 
may take several minutes to complete)...
[07/Dec/2016:12:07:36 +0000] - import ipaca: Generating numSubordinates 
complete.
[07/Dec/2016:12:07:36 +0000] - import ipaca: Gathering ancestorid non-leaf 
IDs...
[07/Dec/2016:12:07:36 +0000] - import ipaca: Finished gathering ancestorid 
non-leaf IDs.
[07/Dec/2016:12:07:36 +0000] - import ipaca: Creating ancestorid index (new 
idl)...
[07/Dec/2016:12:07:36 +0000] - import ipaca: Created ancestorid index (new idl).
[07/Dec/2016:12:07:36 +0000] - import ipaca: Flushing caches...
[07/Dec/2016:12:07:36 +0000] - import ipaca: Closing files...
[07/Dec/2016:12:07:36 +0000] - import ipaca: Import complete.  Processed 375 
entries in 6 seconds. (62.50 entries/sec)
[07/Dec/2016:12:07:36 +0000] NSMMReplicationPlugin - 
multimaster_be_state_change: replica o=ipaca is coming online; enabling 
replication
[07/Dec/2016:12:07:36 +0000] NSMMReplicationPlugin - replica_reload_ruv: 
Warning: new data for replica o=ipaca does not match the data in the changelog.
 Recreating the changelog file. This could affect replication with replica's  
consumers in which case the consumers should be reinitialized.
[07/Dec/2016:12:07:36 +0000] - Skipping CoS Definition cn=Password 
Policy,cn=accounts,dc=int,dc=i-neda,dc=com--no CoS Templates found, which 
should be added before the CoS Definition.

Thanks in advance for any hints,
Neal.

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Reply via email to