Rob and Rich, >> ipa-replica-manage del should have cleaned things up. You can clear out >> old RUVs with ipa-replica-manage too via list-ruv and clean-ruv. You use >> list-ruv to get the id# to clean and clean-ruv to do the actual cleaning. > > I remember having previously tried this task, but it had failed on > older RUV's which were not even active (the KDC was under some strain > so ipa queries were timing out). However, I ran it again and have > been able to delete all but 1 (it's still running) RUV referencing the > previous replica. > > I'll report back once the tasks finishes or fails.
The last RUV is "stuck" on another replica. It fails with the following error: [23/Oct/2014:08:55:09 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Initiating CleanAllRUV Task... [23/Oct/2014:08:55:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Retrieving maxcsn... [23/Oct/2014:08:55:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Found maxcsn (5447f861000000180000) [23/Oct/2014:08:55:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Cleaning rid (24)... [23/Oct/2014:08:55:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Waiting to process all the updates from the deleted replica... [23/Oct/2014:08:55:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Waiting for all the replicas to be online... [23/Oct/2014:08:55:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Waiting for all the replicas to receive all the deleted replica updates... [23/Oct/2014:08:55:11 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Replica maxcsn (5447f56b000200180000) is not caught up with deleted replica's maxcsn(5447f861000000180000) [23/Oct/2014:08:55:11 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Replica not caught up (agmt="cn=meToiparepbackup.our.personal.domain" (iparepbackup:389)) [23/Oct/2014:08:55:11 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Not all replicas caught up, retrying in 10 seconds [23/Oct/2014:08:55:23 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Replica maxcsn (5447f56b000200180000) is not caught up with deleted replica's maxcsn(5447f861000000180000) [23/Oct/2014:08:55:23 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Replica not caught up (agmt="cn=meToiparepbackup.our.personal.domain" (iparepbackup:389)) [23/Oct/2014:08:55:23 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Not all replicas caught up, retrying in 20 seconds I then abort the task since the retrying went up to 14400 seconds. Would this be a simple re-initialization from the master on the host "iparepbackup"? Thanks, John DeSantis 2014-10-22 16:03 GMT-04:00 John Desantis <desan...@mail.usf.edu>: > Rob and Rich, > >> ipa-replica-manage del should have cleaned things up. You can clear out >> old RUVs with ipa-replica-manage too via list-ruv and clean-ruv. You use >> list-ruv to get the id# to clean and clean-ruv to do the actual cleaning. > > I remember having previously tried this task, but it had failed on > older RUV's which were not even active (the KDC was under some strain > so ipa queries were timing out). However, I ran it again and have > been able to delete all but 1 (it's still running) RUV referencing the > previous replica. > > I'll report back once the tasks finishes or fails. > > Thanks, > John DeSantis > > > 2014-10-22 15:49 GMT-04:00 Rob Crittenden <rcrit...@redhat.com>: >> Rich Megginson wrote: >>> On 10/22/2014 12:55 PM, John Desantis wrote: >>>> Richard, >>>> >>>>> You should remove the unused ruv elements. I'm not sure why they >>>>> were not >>>>> cleaned. You may have to use cleanallruv manually. >>>>> https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/9.0/html/Administration_Guide/Managing_Replication-Solving_Common_Replication_Conflicts.html#cleanruv >>>>> >>>>> >>>>> note - use the cleanallruv procedure, not cleanruv. >>>> I'll try that, thanks for the guidance. >>>> >>>>> What is the real problem you have? Did replication stop working? Are >>>>> you >>>>> getting error messages? >>>> I cannot get the host to be a replica. Each time I run >>>> `ipa-replica-install >>>> replica-info-host-in-question.our.personal.domain.gpg' it fails. I >>>> had assumed it was due to the fact that the host was already a >>>> replica, but had to be taken offline due to a hard disk failing. The >>>> machine was re-provisioned after the new hard drive was installed. >>> >>> Ok. I don't know if we have a documented procedure for that case. I >>> assumed that if you first ran ipa-replica-manage del, then >>> ipa-replica-prepare, then ipa-replica-install, that would take care of >>> that. >> >> ipa-replica-manage del should have cleaned things up. You can clear out >> old RUVs with ipa-replica-manage too via list-ruv and clean-ruv. You use >> list-ruv to get the id# to clean and clean-ruv to do the actual cleaning. >> >>>> >>>> When I enabled extra debugging during the installation process, the >>>> initial error was that the dirsrv instance couldn't be started. I >>>> checked into this and found that there were missing files in >>>> /etc/dirsrv/slapd-BLAH directory. I was then able to start dirsrv >>>> after copying some schema files from another replica. The install did >>>> move forward but then failed with Apache and its IPA configuration. >>>> >>>> I performed several uninstalls and re-installs, and at one point I got >>>> error code 3 from ipa-replica-install, which is why I was thinking >>>> that the old RUV's and tombstones were to blame. >>> >>> It could be. I'm really not sure what the problem is at this point. >> >> I think we'd need to see ipareplica-install.log to know for sure. It >> could be the sort of thing where it fails early but doesn't kill the >> install, so the last error is a red herring. >> >> rob >> >>> >>>> >>>> Thanks, >>>> John DeSantis >>>> >>>> >>>> 2014-10-22 12:51 GMT-04:00 Rich Megginson <rmegg...@redhat.com>: >>>>> On 10/22/2014 10:31 AM, John Desantis wrote: >>>>>> Richard, >>>>>> >>>>>> You helped me before in #freeipa, so I appreciate the assistance again. >>>>>> >>>>>>> What version of 389 are you using? >>>>>>> rpm -q 389-ds-base >>>>>> 389-ds-base-1.2.11.15-34.el6_5 >>>>>> >>>>>> Thanks, >>>>>> John DeSantis >>>>>> >>>>>> 2014-10-22 12:09 GMT-04:00 Rich Megginson <rmegg...@redhat.com>: >>>>>>> On 10/22/2014 09:42 AM, John Desantis wrote: >>>>>>>> Hello all, >>>>>>>> >>>>>>>> First and foremost, a big "thank you!" to the FreeIPA developers >>>>>>>> for a >>>>>>>> great product! >>>>>>>> >>>>>>>> Now, to the point! >>>>>>>> >>>>>>>> We're trying to re-provision a previous replica using the standard >>>>>>>> documentation via the Red Hat site: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Identity_Management_Guide/Setting_up_IPA_Replicas.html >>>>>>>> >>>>>>>> >>>>>>>> However, we're running into errors during the import process. The >>>>>>>> errors are varied and fail at random steps; there was an issue with >>>>>>>> NTP or HTTP or LDAP, etc. This did not happen when we promoted a >>>>>>>> separate node to become a replica. >>>>>>>> >>>>>>>> We had previously removed the replica via `ipa-replica-manage del` >>>>>>>> and >>>>>>>> ensured that no trace of it being a replica existed: removed DNS >>>>>>>> records and verified that the host enrollment was not present. I did >>>>>>>> not use the "--force" and "--cleanup" options. >>>>>>> >>>>>>> What version of 389 are you using? >>>>>>> rpm -q 389-ds-base >>>>> >>>>> You should remove the unused ruv elements. I'm not sure why they >>>>> were not >>>>> cleaned. You may have to use cleanallruv manually. >>>>> https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/9.0/html/Administration_Guide/Managing_Replication-Solving_Common_Replication_Conflicts.html#cleanruv >>>>> >>>>> >>>>> note - use the cleanallruv procedure, not cleanruv. >>>>> >>>>>>>> When I check RUV's against the host in question, there are >>>>>>>> several. I >>>>>>>> also queried the tombstones against the host and found two entries >>>>>>>> which have valid hex time stamps; coincidentally, out of the 9 >>>>>>>> tombstone entries, 2 have "nsds50ruv" time stamps. I'll paste >>>>>>>> sanitized output below: >>>>>>>> >>>>>>>> # ldapsaerch -x -W -LLL -D "cn=directory manager" -b >>>>>>>> "dc=our,dc=personal,dc=domain" '(objectclass=nsTombstone)' >>>>>>>> Enter LDAP Password: >>>>>>>> dn: >>>>>>>> >>>>>>>> nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,dc=our,dc=personal,dc=domain >>>>>>>> >>>>>>>> objectClass: top >>>>>>>> objectClass: nsTombstone >>>>>>>> objectClass: extensibleobject >>>>>>>> nsds50ruv: {replicageneration} 50ef13ae000000040000 >>>>>>>> nsds50ruv: {replica 4 ldap://master.our.personal.domain:389} >>>>>>>> 5164d147000000040000 5447bda 8000100040000 >>>>>>>> nsds50ruv: {replica 22 ldap://separatenode.our.personal.domain:389} >>>>>>>> 54107f9f000000160000 54436b 25000000160000 >>>>>>>> nsds50ruv: {replica 21 ldap://iparepbackup.our.personal.domain:389} >>>>>>>> 51b734de000000150000 51b7 34ef000200150000 >>>>>>>> nsds50ruv: {replica 19 >>>>>>>> ldap://host-in-question.our.personal.domain:389} 510d56c9000100130000 >>>>>>>> 510d82 be000200130000 >>>>>>>> nsds50ruv: {replica 18 >>>>>>>> ldap://host-in-question.our.personal.domain:389} >>>>>>>> nsds50ruv: {replica 17 >>>>>>>> ldap://host-in-question.our.personal.domain:389} >>>>>>>> nsds50ruv: {replica 16 >>>>>>>> ldap://host-in-question.our.personal.domain:389} >>>>>>>> nsds50ruv: {replica 15 >>>>>>>> ldap://host-in-question.our.personal.domain:389} >>>>>>>> nsds50ruv: {replica 14 >>>>>>>> ldap://host-in-question.our.personal.domain:389} >>>>>>>> nsds50ruv: {replica 13 >>>>>>>> ldap://host-in-question.our.personal.domain:389} >>>>>>>> nsds50ruv: {replica 12 >>>>>>>> ldap://host-in-question.our.personal.domain:389} >>>>>>>> nsds50ruv: {replica 23 >>>>>>>> ldap://host-in-question.our.personal.domain:389} 54187702000200170000 >>>>>>>> 541878 9a000000170000 >>>>>>>> dc: our >>>>>>>> nsruvReplicaLastModified: {replica 4 >>>>>>>> ldap://master.our.personal.domain:389} 5447bce8 >>>>>>>> nsruvReplicaLastModified: {replica 22 >>>>>>>> ldap://separatenode.our.personal.domain:389} 54436a5e >>>>>>>> nsruvReplicaLastModified: {replica 21 >>>>>>>> ldap://iparepbackup.our.personal.domain:389} 00000000 >>>>>>>> nsruvReplicaLastModified: {replica 19 >>>>>>>> ldap://host-in-question.our.personal.domain:389} 00000000 >>>>>>>> nsruvReplicaLastModified: {replica 18 >>>>>>>> ldap://host-in-question.our.personal.domain:389} 00000000 >>>>>>>> nsruvReplicaLastModified: {replica 17 >>>>>>>> ldap://host-in-question.our.personal.domain:389} 00000000 >>>>>>>> nsruvReplicaLastModified: {replica 16 >>>>>>>> ldap://host-in-question.our.personal.domain:389} 00000000 >>>>>>>> nsruvReplicaLastModified: {replica 15 >>>>>>>> ldap://host-in-question.our.personal.domain:389} 00000000 >>>>>>>> nsruvReplicaLastModified: {replica 14 >>>>>>>> ldap://host-in-question.our.personal.domain:389} 00000000 >>>>>>>> nsruvReplicaLastModified: {replica 13 >>>>>>>> ldap://host-in-question.our.personal.domain:389} 00000000 >>>>>>>> nsruvReplicaLastModified: {replica 12 >>>>>>>> ldap://host-in-question.our.personal.domain:389} 00000000 >>>>>>>> nsruvReplicaLastModified: {replica 23 >>>>>>>> ldap://host-in-question.our.personal.domain:389} 00000000 >>>>>>>> >>>>>>>> dn: >>>>>>>> >>>>>>>> nsuniqueid=c08a2803-5b5a11e2-a527ce8b-8fa47d35,cn=host-in-question.our.personal.domain,cn=maste >>>>>>>> >>>>>>>> rs,cn=ipa,cn=etc,dc=our,dc=personal,dc=domain >>>>>>>> objectClass: top >>>>>>>> objectClass: nsContainer >>>>>>>> objectClass: nsTombstone >>>>>>>> cn: host-in-question.our.personal.domain >>>>>>>> nsParentUniqueId: e6fa9418-5b5711e2-a1a9825b-daf5b5b0 >>>>>>>> >>>>>>>> dn: >>>>>>>> >>>>>>>> nsuniqueid=664c4383-6d6311e2-8db6e946-de27dd8d,cn=host-in-question.our.personal.domain,cn=maste >>>>>>>> >>>>>>>> rs,cn=ipa,cn=etc,dc=our,dc=personal,dc=domain >>>>>>>> objectClass: top >>>>>>>> objectClass: nsContainer >>>>>>>> objectClass: nsTombstone >>>>>>>> cn: host-in-question.our.personal.domain >>>>>>>> nsParentUniqueId: e6fa9418-5b5711e2-a1a9825b-daf5b5b0 >>>>>>>> >>>>>>>> As you can see, the "host-in-question" has many RUV's and of which >>>>>>>> two >>>>>>>> appear to be "active" and two entries which I believe (pardon my >>>>>>>> ignorance) possibly correlate with the "active" entries of the >>>>>>>> "host-in-question". >>>>>>>> >>>>>>>> Do these two tombstone entries need to be deleted with ldapdelete >>>>>>>> before we can re-provision "host-in-question" and add it back as a >>>>>>>> replica? >>>>> >>>>> No, you cannot delete tombstones manually. They will be cleaned up >>>>> at some >>>>> point by the dirsrv tombstone reap thread, and they should not be >>>>> interfering with anything. >>>>> >>>>> What is the real problem you have? Did replication stop working? Are >>>>> you >>>>> getting error messages? >>>>> >>>>>>>> Thank you, >>>>>>>> John DeSantis >>>>>>>> >>>>>>> -- >>>>>>> Manage your subscription for the Freeipa-users mailing list: >>>>>>> https://www.redhat.com/mailman/listinfo/freeipa-users >>>>>>> Go To http://freeipa.org for more info on the project >>>>> >>>>> -- >>>>> Manage your subscription for the Freeipa-users mailing list: >>>>> https://www.redhat.com/mailman/listinfo/freeipa-users >>>>> Go To http://freeipa.org for more info on the project >>> >> -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go To http://freeipa.org for more info on the project