[Freeipa-users] What to do next???: IPA replica host entry is removed on web UI by mistake
Hi all, I accidentally removed one of my IPA replica host on IPA web UI by mistake, on the host list I planed to remove ipaclient02.example.com, but accidentally the mouse moved to ipareplica02.example.com and the latter got removed without a prompt. I realized the mistake and tried to recover from this disaster but it was already too late, the change propagated to all the replicas and the poor ipareplica02 now stops functioning. [root@ipareplica02 slapd-EXAMPLE-COM]# ipa service-find ipa: ERROR: cannot connect to u'https://ipareplica02.qe9.jigsaw.com/ipa/xml': Internal Server Error [root@ipareplica02 slapd-EXAMPLE-COM]# ipa user-find ipa: ERROR: cannot connect to u'https://ipareplica02.qe9.jigsaw.com/ipa/xml': Internal Server Error [root@ipareplica02 slapd-EXAMPLE-COM]# ipa host-find ipa: ERROR: cannot connect to u'https://ipareplica02.qe9.jigsaw.com/ipa/xml': Internal Server Error [root@ipareplica02 slapd-EXAMPLE-COM]# On the IPA master, It was found that ipareplica02 didn't show up in 'host-find' list or 'service-find' list. Though it still showed in the master list reported by 'ipa-replica-manage' and 'ipa-csreplica-manage', the real command 'ipa-replica-manage list ipareplica02' fails with LDAP could't reach error. What should I do now? Is there are any other ways to recover besides uninstall and reinstall of IPA replica ipareplica02? BTW, it will be more than appreciated if the web UI could pop up a warning prompt when removing host/services entries associated with IPA masters and IPA replicas. Thanks. --David From: Rich Megginson rmegg...@redhat.com To: Ben Ho ben1...@hotmail.com Cc: freeipa-users@redhat.com Sent: Tuesday, May 15, 2012 5:33 PM Subject: Re: [Freeipa-users] Help with ipa-replica-manage On 05/15/2012 02:49 PM, Ben Ho wrote: This is the information I retrieved about my server. ipa-server-selinux-2.1.3-9.el6.x86_64 ipa-client-2.1.3-9.el6.x86_64 ipa-server-2.1.3-9.el6.x86_64 CentOS release 6.2 389-ds-base-1.2.9.14-1.el6_2.2.x86_64 Thanks again. Is replication otherwise working? -Ben Date: Tue, 15 May 2012 13:15:46 -0600 From: rmegg...@redhat.com To: ben1...@hotmail.com CC: freeipa-users@redhat.com Subject: Re: [Freeipa-users] Help with ipa-replica-manage On 05/15/2012 01:00 PM, Ben Ho wrote: Hello, I am pretty new to IPA. Right now I have three servers that are running IPA. I am trying to replicate one server to two other servers. I use this command: ipa-replica-manage re-initialize --from example2.edu On the first server I need to replicate, it works fine. However, on the second server I get this message in my log files. The errors get printed out once every 1 to 5 minutes. [15/May/2012:14:22:43 -0400] NSMMReplicationPlugin - agmt=cn=meToexample1.edu (example1:389): Schema replication update failed: Type or value exists [15/May/2012:14:22:43 -0400] NSMMReplicationPlugin - agmt=cn=meToexample1.edu (example1:389): Warning: unable to replicate schema: rc=1 [15/May/2012:14:22:47 -0400] NSMMReplicationPlugin - agmt=cn=meToexample2.edu (example2:389): Schema replication update failed: Type or value exists [15/May/2012:14:22:47 -0400] NSMMReplicationPlugin - agmt=cn=meToexample2.edu (example2:389): Warning: unable to replicate schema: rc=1 Again, I am pretty new to this, so any help or tips would be appreciated. What platform and what version of 389-ds-base and ipa-server for all of your servers? Thanks! -Ben ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] What to do next???: IPA replica host entry is removed on web UI by mistake
On May 16, 2012, at 12:23 PM, David Copperfield wrote: Hi all, I accidentally removed one of my IPA replica host on IPA web UI by mistake, on the host list I planed to remove ipaclient02.example.com, but accidentally the mouse moved to ipareplica02.example.com and the latter got removed without a prompt. I realized the mistake and tried to recover from this disaster but it was already too late, the change propagated to all the replicas and the poor ipareplica02 now stops functioning. [root@ipareplica02 slapd-EXAMPLE-COM]# ipa service-find ipa: ERROR: cannot connect to u'https://ipareplica02.qe9.jigsaw.com/ipa/xml': Internal Server Error [root@ipareplica02 slapd-EXAMPLE-COM]# ipa user-find ipa: ERROR: cannot connect to u'https://ipareplica02.qe9.jigsaw.com/ipa/xml': Internal Server Error [root@ipareplica02 slapd-EXAMPLE-COM]# ipa host-find ipa: ERROR: cannot connect to u'https://ipareplica02.qe9.jigsaw.com/ipa/xml': Internal Server Error [root@ipareplica02 slapd-EXAMPLE-COM]# On the IPA master, It was found that ipareplica02 didn't show up in 'host-find' list or 'service-find' list. Though it still showed in the master list reported by 'ipa-replica-manage' and 'ipa-csreplica-manage', the real command 'ipa-replica-manage list ipareplica02' fails with LDAP could't reach error. What should I do now? Is there are any other ways to recover besides uninstall and reinstall of IPA replica ipareplica02? BTW, it will be more than appreciated if the web UI could pop up a warning prompt when removing host/services entries associated with IPA masters and IPA replicas. Been there... Done that... The bug is fixed in 2.2... It will prompt and prevent you from deleting a replica host if there is an agreement. To clean up... 0. On the master replica: ipa-replica-manage del ipareplica02.example.com --force -This will delete the replica agreement for the host. 1. $ ldapsearch -xLLL -D cn=directory manager -W -b dc=example,dc=com \ '((nsuniqueid=---)(objectclass=nstombstone))' Look for your your nsds50ruv that matches your ghost replica. 2. Create an ldif following the directions here: http://directory.fedoraproject.org/wiki/Howto:CLEANRUV Something like: $ cat cleanup.ldif dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config changetype: modify replace: nsds5task nsds5task: CLEANRUV## - ## == The ReplicaID number for the ghost replica. 3. Run on all of the remaining replicas: ldapmodify -x -D cn=directory manager -W -f fixed.ldif - This removes the ghost entry. 4. on the broken replica: ipa-server-install --uninstall 5. Follow the normal directions for 'installing a replica' - on master: ipa-replica-prepare ipareplica02.example.com - scp /path/to/ipareplica02.example.com.gpg ipareplica02.example.com: ipareplica02.example.com.gpg - on replica: ipa-replica-install ipareplica02.example.com --whatever_options_you_used_previously 6. Check to make sure the server was built correctly and command work as expected: kinit admin, ipa user-find, ipa host-find, id admin, etc etc 7. Sigh and drink coffee Thanks. --David From: Rich Megginson rmegg...@redhat.com To: Ben Ho ben1...@hotmail.com Cc: freeipa-users@redhat.com Sent: Tuesday, May 15, 2012 5:33 PM Subject: Re: [Freeipa-users] Help with ipa-replica-manage On 05/15/2012 02:49 PM, Ben Ho wrote: This is the information I retrieved about my server. ipa-server-selinux-2.1.3-9.el6.x86_64 ipa-client-2.1.3-9.el6.x86_64 ipa-server-2.1.3-9.el6.x86_64 CentOS release 6.2 389-ds-base-1.2.9.14-1.el6_2.2.x86_64 Thanks again. Is replication otherwise working? -Ben Date: Tue, 15 May 2012 13:15:46 -0600 From: rmegg...@redhat.com To: ben1...@hotmail.com CC: freeipa-users@redhat.com Subject: Re: [Freeipa-users] Help with ipa-replica-manage On 05/15/2012 01:00 PM, Ben Ho wrote: Hello, I am pretty new to IPA. Right now I have three servers that are running IPA. I am trying to replicate one server to two other servers. I use this command: ipa-replica-manage re-initialize --from example2.edu On the first server I need to replicate, it works fine. However, on the second server I get this message in my log files. The errors get printed out once every 1 to 5 minutes. [15/May/2012:14:22:43 -0400] NSMMReplicationPlugin - agmt=cn=meToexample1.edu (example1:389): Schema replication update failed: Type or value exists [15/May/2012:14:22:43 -0400] NSMMReplicationPlugin - agmt=cn=meToexample1.edu (example1:389): Warning: unable to replicate schema: rc=1 [15/May/2012:14:22:47 -0400] NSMMReplicationPlugin - agmt=cn=meToexample2.edu (example2:389): Schema replication update failed: Type or value exists [15/May/2012:14:22:47 -0400] NSMMReplicationPlugin - agmt=cn=meToexample2.edu (example2:389): Warning: unable to replicate schema: rc=1 Again, I am pretty new to this, so any help or tips would be
Re: [Freeipa-users] What to do next???: IPA replica host entry is removed on web UI by mistake
David Copperfield wrote: Hi all, I accidentally removed one of my IPA replica host on IPA web UI by mistake, on the host list I planed to remove ipaclient02.example.com, but accidentally the mouse moved to ipareplica02.example.com and the latter got removed without a prompt. I realized the mistake and tried to recover from this disaster but it was already too late, the change propagated to all the replicas and the poor ipareplica02 now stops functioning. [root@ipareplica02 slapd-EXAMPLE-COM]# ipa service-find ipa: ERROR: cannot connect to u'https://ipareplica02.qe9.jigsaw.com/ipa/xml': Internal Server Error [root@ipareplica02 slapd-EXAMPLE-COM]# ipa user-find ipa: ERROR: cannot connect to u'https://ipareplica02.qe9.jigsaw.com/ipa/xml': Internal Server Error [root@ipareplica02 slapd-EXAMPLE-COM]# ipa host-find ipa: ERROR: cannot connect to u'https://ipareplica02.qe9.jigsaw.com/ipa/xml': Internal Server Error [root@ipareplica02 slapd-EXAMPLE-COM]# On the IPA master, It was found that ipareplica02 didn't show up in 'host-find' list or 'service-find' list. Though it still showed in the master list reported by 'ipa-replica-manage' and 'ipa-csreplica-manage', the real command 'ipa-replica-manage list ipareplica02' fails with LDAP could't reach error. What should I do now? Is there are any other ways to recover besides uninstall and reinstall of IPA replica ipareplica02? BTW, it will be more than appreciated if the web UI could pop up a warning prompt when removing host/services entries associated with IPA masters and IPA replicas. Thanks. --David On a working master try re-creating the host and re-adding the services. You'll probably want to use the fqdn in places of ipareplica02 here. The case of the services is important. I'm assuming this master is not running dogtag or DNS. # ipa host-add ipareplica02 # ipa service-add ldap/ipareplica02 # ipa service-add HTTP/ipareplica02 # mkdir /tmp/ipareplica02 # ipa-getkeytab -s master -k /tmp/ipareplica02/ds.keytab -p ldap/ipareplica02 # ipa-getkeytab -s master -k /tmp/ipareplica02/ipa.keytab -p HTTP/ipareplica02 Copy these files to ipareplica02. ds.keytab goes in /etc/dirsrv/ ipa.keytab goes in /etc/httpd/conf/ I'd run restorecon on both. Perms should be 0600 dirsrv:dirsrv on ds.keytab 0600 root:root on ipa.keytab # ipactl restart You'll need to restart the dirsrv service (or ipactl restart) on all your other masters to pick up the new ldap service principal. In theory you should have a working system again. The only downside is the certs being used aren't reflected in your service entries any more. I don't believe this will affect automated renewal so if you don't care about that you're done. If you are using dogtag as your CA your SSL certs have been revoked though. To fix this we can try to get certmonger to refresh them. # ipa-getcert list find the ID for the /etc/dirsrv/slapd-YOURINSTANCE cert # ipa-getcert resubmit -i ID Run ipa-getcert list again to see the status. It should be MONITORING and the expires date should have changed. Assuming that worked do the same for the Apache cert (in /etc/httpd/alias). Restart dirsrv and httpd services or ipactl restart. We block deleting master hosts and services in FreeIPA 2.2. rob ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] What to do next???: IPA replica host entry is removed on web UI by mistake
Hi JR, Thanks a lot! It works perfectly. The only extra thing probably goes with 2.1.3 only: I need to find and clear ghost RUV records for CA database, and remove it from master and all other live replicas as well. BTW, on 2.2.0 the two database backends still are separate, or merged into one? Thanks. --David From: JR Aquino jr.aqu...@citrix.com To: David Copperfield cao2...@yahoo.com Cc: FreeIPAUsers freeipa-users@redhat.com Sent: Wednesday, May 16, 2012 12:57 PM Subject: Re: [Freeipa-users] What to do next???: IPA replica host entry is removed on web UI by mistake On May 16, 2012, at 12:23 PM, David Copperfield wrote: Hi all, I accidentally removed one of my IPA replica host on IPA web UI by mistake, on the host list I planed to remove ipaclient02.example.com, but accidentally the mouse moved to ipareplica02.example.com and the latter got removed without a prompt. I realized the mistake and tried to recover from this disaster but it was already too late, the change propagated to all the replicas and the poor ipareplica02 now stops functioning. [root@ipareplica02 slapd-EXAMPLE-COM]# ipa service-find ipa: ERROR: cannot connect to u'https://ipareplica02.qe9.jigsaw.com/ipa/xml': Internal Server Error [root@ipareplica02 slapd-EXAMPLE-COM]# ipa user-find ipa: ERROR: cannot connect to u'https://ipareplica02.qe9.jigsaw.com/ipa/xml': Internal Server Error [root@ipareplica02 slapd-EXAMPLE-COM]# ipa host-find ipa: ERROR: cannot connect to u'https://ipareplica02.qe9.jigsaw.com/ipa/xml': Internal Server Error [root@ipareplica02 slapd-EXAMPLE-COM]# On the IPA master, It was found that ipareplica02 didn't show up in 'host-find' list or 'service-find' list. Though it still showed in the master list reported by 'ipa-replica-manage' and 'ipa-csreplica-manage', the real command 'ipa-replica-manage list ipareplica02' fails with LDAP could't reach error. What should I do now? Is there are any other ways to recover besides uninstall and reinstall of IPA replica ipareplica02? BTW, it will be more than appreciated if the web UI could pop up a warning prompt when removing host/services entries associated with IPA masters and IPA replicas. Been there... Done that... The bug is fixed in 2.2... It will prompt and prevent you from deleting a replica host if there is an agreement. To clean up... 0. On the master replica: ipa-replica-manage del ipareplica02.example.com --force -This will delete the replica agreement for the host. 1. $ ldapsearch -xLLL -D cn=directory manager -W -b dc=example,dc=com \ '((nsuniqueid=---)(objectclass=nstombstone))' Look for your your nsds50ruv that matches your ghost replica. 2. Create an ldif following the directions here: http://directory.fedoraproject.org/wiki/Howto:CLEANRUV Something like: $ cat cleanup.ldif dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config changetype: modify replace: nsds5task nsds5task: CLEANRUV## - ## == The ReplicaID number for the ghost replica. 3. Run on all of the remaining replicas: ldapmodify -x -D cn=directory manager -W -f fixed.ldif - This removes the ghost entry. 4. on the broken replica: ipa-server-install --uninstall 5. Follow the normal directions for 'installing a replica' - on master: ipa-replica-prepare ipareplica02.example.com - scp /path/to/ipareplica02.example.com.gpg ipareplica02.example.com: ipareplica02.example.com.gpg - on replica: ipa-replica-install ipareplica02.example.com --whatever_options_you_used_previously 6. Check to make sure the server was built correctly and command work as expected: kinit admin, ipa user-find, ipa host-find, id admin, etc etc 7. Sigh and drink coffee Thanks. --David From: Rich Megginson rmegg...@redhat.com To: Ben Ho ben1...@hotmail.com Cc: freeipa-users@redhat.com Sent: Tuesday, May 15, 2012 5:33 PM Subject: Re: [Freeipa-users] Help with ipa-replica-manage On 05/15/2012 02:49 PM, Ben Ho wrote: This is the information I retrieved about my server. ipa-server-selinux-2.1.3-9.el6.x86_64 ipa-client-2.1.3-9.el6.x86_64 ipa-server-2.1.3-9.el6.x86_64 CentOS release 6.2 389-ds-base-1.2.9.14-1.el6_2.2.x86_64 Thanks again. Is replication otherwise working? -Ben Date: Tue, 15 May 2012 13:15:46 -0600 From: rmegg...@redhat.com To: ben1...@hotmail.com CC: freeipa-users@redhat.com Subject: Re: [Freeipa-users] Help with ipa-replica-manage On 05/15/2012 01:00 PM, Ben Ho wrote: Hello, I am pretty new to IPA. Right now I have three servers that are running IPA. I am trying to replicate one server to two other servers. I use this command: ipa-replica-manage re-initialize --from example2.edu On the first server I need to replicate, it works fine. However, on the second server I get this message in my log files. The errors get printed out once every 1 to 5 minutes. [15/May/2012:14:22:43 -0400]
