Re: [Freeipa-users] Loss of initial master in multi master setup
> From: Rob Crittenden > Martin Babinsky wrote: > > On 12/01/2016 01:28 PM, Neal Harrington | i-Neda Ltd wrote: > >> Hi IPA Gurus, > >> > >> > >> I had a 3 site multi master IPA replication setup (1 office and 2 > >> datacentres) with 2 IPA servers at each site. Each server was > >> replicating successfully to 3 other servers (the other local site > >> server and one server at each of the two remote sites). Everything is > >> running on the default packages from CentOS 7.2 and each server is a > >> full replica (ipa-replica-install > >> /var/lib/ipa/replica-info-id-myserver.fqdn.com.gpg --setup-ca > >> --setup-dns --mkhomedir --forwarder 8.8.8.8) > >> > >> > >> Everything was ticking over nicely until we had notice that the > >> office site was moving on short notice. > >> > >> > >> I successfully created IPA servers at the new site, setup replication > >> again between the new office and the two datacentres that were to > >> remain online, tested and everything worked as expected - > >> unfortunately in the rush I did not have time to properly retire the > >> IPA servers in the old office. > >> > >> > >> The problem this has caused is that I only ever created users in one > >> of the IPA servers in the original office - so only those servers > >> have a DNA range and I am now unable to create new users on the active > servers. > >> The original office servers are still in the IPA replication and > >> powered on but offline so potential split brain? > >> > >> > >> I now have two things I would like to know before proceeding: > >> > >> * Is the best fix here to force remove the original IPA servers and > >> manually add a new dna range significantly different from the > >> original to avoid overlaps? > >> * Is there anything else I should check? I can't see any issues > >> however did not notice the DNA range until I tried to create a user. > >> > >> Any pointers greatly appreciated. > >> > >> > >> Thanks, > >> > >> Neal. > >> > >> > >> > >> > >> > >> > > > > Hi Neal, > > > > If you already disconnected/decomissioned the old masters then I thnk > > the best you can do is option a, i.e. re-set DNA ranges on replicas to > > new values while avioding overlap with old ranges. > > > > We have an upstream document[1] describing the procedure. Hope it > helps. > > > > Also make sure that you migrated CA renewal and CRL master > > responsibilities to the new replicas, otherwise you may get problems > > with expiring certificates which are really hard to solve. See the > > following guide for details. [2] > > > > [1] http://www.freeipa.org/page/V3/Recover_DNA_Ranges > > [2] > > > http://www.freeipa.org/page/Howto/Promote_CA_to_Renewal_and_CRL_ > Master > > > > You may want to look at this too, http://blog-rcritten.rhcloud.com/?p=50 > > rob Hi Rob & Martin, Thanks for the pointers, I am now able to create new users on different servers - however everything to do with replication seems to be failing. I have changed my replication from a mesh to a long chain and run "ipa-replica-manage -v re-initialize --from " and the same for ipa-csreplica-manage along the chain which succeeds (and any passwords/user creation etc I have done at the start of the chain is pulled through) however replication fails immediately after. I was hoping that re-initializing the chain like this would flush out any "bad" entries - probably wishful thinking. "Ipa-replica-manage -v list" only shows servers in the chain. "Ipa-replica-manage list-ruv" did show the two original servers which I lost connection to and I removed those which successfully removed them from all servers so that part of replication seems to be working. When I do an LDAP search I still see those old masters though (and also see one previously retired server with two different ID's - blue-auth01). Will I need to manually delete these? (example search and output below) Apart from manually deleting the dead servers from LDAP, what else should I do to get replication working again? I'm watching for the CentOS 7.3 release to be able to upgrade to IPA 4.3 as I've seen a few posts about the better handling of replication etc in that version. In the meantime the errors log (copy below) indicates I need to re-initialize which I've done several times without any improvement. Thanks in advance, Neal. [root@office-auth04 ~]# ldapsearch -h $(hostname -f) -D "cn=directory manager" -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=---))" nscpentrywsi Enter LDAP Password: # extended LDIF # # LDAPv3 # base
Re: [Freeipa-users] Loss of initial master in multi master setup
> > Hi IPA Gurus, > > > > > > I had a 3 site multi master IPA replication setup (1 office and 2 > > datacentres) with 2 IPA servers at each site. Each server was > > replicating successfully to 3 other servers (the other local site > > server and one server at each of the two remote sites). Everything is > > running on the default packages from CentOS 7.2 and each server is a > > full replica (ipa-replica-install > > /var/lib/ipa/replica-info-id-myserver.fqdn.com.gpg --setup-ca > > --setup-dns --mkhomedir --forwarder 8.8.8.8) > > > > > > Everything was ticking over nicely until we had notice that the office > > site was moving on short notice. > > > > > > I successfully created IPA servers at the new site, setup replication > > again between the new office and the two datacentres that were to > > remain online, tested and everything worked as expected - > > unfortunately in the rush I did not have time to properly retire the > > IPA servers in the old office. > > > > > > The problem this has caused is that I only ever created users in one > > of the IPA servers in the original office - so only those servers have > > a DNA range and I am now unable to create new users on the active > servers. > > The original office servers are still in the IPA replication and > > powered on but offline so potential split brain? > > > > > > I now have two things I would like to know before proceeding: > > > > * Is the best fix here to force remove the original IPA servers and > > manually add a new dna range significantly different from the > > original to avoid overlaps? > > * Is there anything else I should check? I can't see any issues > > however did not notice the DNA range until I tried to create a user. > > > > Any pointers greatly appreciated. > > > > > > Thanks, > > > > Neal. > > Hi Neal, > > If you already disconnected/decomissioned the old masters then I thnk the > best you can do is option a, i.e. re-set DNA ranges on replicas to new values > while avioding overlap with old ranges. > > We have an upstream document[1] describing the procedure. Hope it helps. > > Also make sure that you migrated CA renewal and CRL master responsibilities > to the new replicas, otherwise you may get problems with expiring > certificates which are really hard to solve. See the following guide for > details. > [2] > > [1] http://www.freeipa.org/page/V3/Recover_DNA_Ranges > [2] > http://www.freeipa.org/page/Howto/Promote_CA_to_Renewal_and_CRL_ > Master > > -- > Martin^3 Babinsky Hi Martin & Rob, Thank you very much for the pointers. I have added a new range to a IPA server I used the top half of the previous range, I only had 30 ish ID's used so far) # ipa-replica-manage dnarange-set office03.fqdn.com 31030-31039 and this has allowed me to add a user on that server. However when I try to add a user on a different server it still fails with "allocation of new value for range". I was expecting this to request a new range and halve the currently assigned range. Robs link included this command: # ldapsearch -x -D 'cn=Directory Manager' -W -b cn=posix-ids,cn=dna,cn=ipa,cn=etc,dc=int,dc=i-neda,dc=com ...Which seems to list all of the other servers, including office03.fqdn.com which it shows as having 9 dnaRemainingValues (all the rest have 0) so the server that cannot add users can see office03 has 9 unused. However of more immediate concern now I can create user accounts is the CA replication which I seem to have completely messed up. Most CA replication went back to the (now offline) office and even what I have does not seem to work as expected. Eg on Office03: # ldapsearch -H ldap://$HOSTNAME -D 'cn=Directory Manager' -W -b 'cn=masters,cn=ipa,cn=etc,dc=example,dc=com' '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn search result search: 2 result: 32 No such object Following the instructions to set the master seems to work at first (no errors) but the ldap search for renewal master still returns "result: 32 No Such Object" # ipa-csreplica-manage set-renewal-master ipa: WARNING: session memcached servers not running Directory Manager password: office03.fqdn.com is now the renewal master re running the set-renwal-master command reports that this server is already the renewal master. I think I need to reinitialize the CA replication and connect everything up in a redundant loop as I have with the main replication - however the LDAP query not returning the replication master does not seem right. I have not added any IPA servers since these network changes happened a week ago, is it reasonably safe to assume no certificates will have been created so all servers are effectively in sync? Your help with this is greatly appreciated. On the plus side the systems we use this for are all dev, not live, so it is a good learning experience for me if nothing else! Best Regards, Neal. -- Manage your subscription for the Freeipa-users mailing list:
Re: [Freeipa-users] Loss of initial master in multi master setup
Martin Babinsky wrote: > On 12/01/2016 01:28 PM, Neal Harrington | i-Neda Ltd wrote: >> Hi IPA Gurus, >> >> >> I had a 3 site multi master IPA replication setup (1 office and 2 >> datacentres) with 2 IPA servers at each site. Each server was >> replicating successfully to 3 other servers (the other local site server >> and one server at each of the two remote sites). Everything is running >> on the default packages from CentOS 7.2 and each server is a full >> replica (ipa-replica-install >> /var/lib/ipa/replica-info-id-myserver.fqdn.com.gpg --setup-ca >> --setup-dns --mkhomedir --forwarder 8.8.8.8) >> >> >> Everything was ticking over nicely until we had notice that the >> office site was moving on short notice. >> >> >> I successfully created IPA servers at the new site, setup replication >> again between the new office and the two datacentres that were to remain >> online, tested and everything worked as expected - unfortunately in the >> rush I did not have time to properly retire the IPA servers in the old >> office. >> >> >> The problem this has caused is that I only ever created users in one of >> the IPA servers in the original office - so only those servers have a >> DNA range and I am now unable to create new users on the active servers. >> The original office servers are still in the IPA replication and powered >> on but offline so potential split brain? >> >> >> I now have two things I would like to know before proceeding: >> >> * Is the best fix here to force remove the original IPA servers and >> manually add a new dna range significantly different from the >> original to avoid overlaps? >> * Is there anything else I should check? I can't see any issues >> however did not notice the DNA range until I tried to create a user. >> >> Any pointers greatly appreciated. >> >> >> Thanks, >> >> Neal. >> >> >> >> >> >> > > Hi Neal, > > If you already disconnected/decomissioned the old masters then I thnk > the best you can do is option a, i.e. re-set DNA ranges on replicas to > new values while avioding overlap with old ranges. > > We have an upstream document[1] describing the procedure. Hope it helps. > > Also make sure that you migrated CA renewal and CRL master > responsibilities to the new replicas, otherwise you may get problems > with expiring certificates which are really hard to solve. See the > following guide for details. [2] > > [1] http://www.freeipa.org/page/V3/Recover_DNA_Ranges > [2] http://www.freeipa.org/page/Howto/Promote_CA_to_Renewal_and_CRL_Master > You may want to look at this too, http://blog-rcritten.rhcloud.com/?p=50 rob -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] Loss of initial master in multi master setup
On 12/01/2016 01:28 PM, Neal Harrington | i-Neda Ltd wrote: Hi IPA Gurus, I had a 3 site multi master IPA replication setup (1 office and 2 datacentres) with 2 IPA servers at each site. Each server was replicating successfully to 3 other servers (the other local site server and one server at each of the two remote sites). Everything is running on the default packages from CentOS 7.2 and each server is a full replica (ipa-replica-install /var/lib/ipa/replica-info-id-myserver.fqdn.com.gpg --setup-ca --setup-dns --mkhomedir --forwarder 8.8.8.8) Everything was ticking over nicely until we had notice that the office site was moving on short notice. I successfully created IPA servers at the new site, setup replication again between the new office and the two datacentres that were to remain online, tested and everything worked as expected - unfortunately in the rush I did not have time to properly retire the IPA servers in the old office. The problem this has caused is that I only ever created users in one of the IPA servers in the original office - so only those servers have a DNA range and I am now unable to create new users on the active servers. The original office servers are still in the IPA replication and powered on but offline so potential split brain? I now have two things I would like to know before proceeding: * Is the best fix here to force remove the original IPA servers and manually add a new dna range significantly different from the original to avoid overlaps? * Is there anything else I should check? I can't see any issues however did not notice the DNA range until I tried to create a user. Any pointers greatly appreciated. Thanks, Neal. Hi Neal, If you already disconnected/decomissioned the old masters then I thnk the best you can do is option a, i.e. re-set DNA ranges on replicas to new values while avioding overlap with old ranges. We have an upstream document[1] describing the procedure. Hope it helps. Also make sure that you migrated CA renewal and CRL master responsibilities to the new replicas, otherwise you may get problems with expiring certificates which are really hard to solve. See the following guide for details. [2] [1] http://www.freeipa.org/page/V3/Recover_DNA_Ranges [2] http://www.freeipa.org/page/Howto/Promote_CA_to_Renewal_and_CRL_Master -- Martin^3 Babinsky -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
[Freeipa-users] Loss of initial master in multi master setup
Hi IPA Gurus, I had a 3 site multi master IPA replication setup (1 office and 2 datacentres) with 2 IPA servers at each site. Each server was replicating successfully to 3 other servers (the other local site server and one server at each of the two remote sites). Everything is running on the default packages from CentOS 7.2 and each server is a full replica (ipa-replica-install /var/lib/ipa/replica-info-id-myserver.fqdn.com.gpg --setup-ca --setup-dns --mkhomedir --forwarder 8.8.8.8) Everything was ticking over nicely until we had notice that the office site was moving on short notice. I successfully created IPA servers at the new site, setup replication again between the new office and the two datacentres that were to remain online, tested and everything worked as expected - unfortunately in the rush I did not have time to properly retire the IPA servers in the old office. The problem this has caused is that I only ever created users in one of the IPA servers in the original office - so only those servers have a DNA range and I am now unable to create new users on the active servers. The original office servers are still in the IPA replication and powered on but offline so potential split brain? I now have two things I would like to know before proceeding: * Is the best fix here to force remove the original IPA servers and manually add a new dna range significantly different from the original to avoid overlaps? * Is there anything else I should check? I can't see any issues however did not notice the DNA range until I tried to create a user. Any pointers greatly appreciated. Thanks, Neal. -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
