I think I've finally started to make some progress on this. I did a lot of googling and found some stuff to run manually in 389 ds through ldapmodify commands to clean RUVs. During this process the server crashed and when it came back online, suddenly all my ghost RUVs were visible through ipa-replica-manage list-ruv. It was really strange, I had like 5 of them from winsync agreements that kept failing and needing re-initialization, and another 5 from my earlier re-installations of the 2 other domain controllers.
I ran some more ruv cleanup commands through ldap and they all appear to be gone. I'm not sure how the crash suddenly made them visible though or why they had to be cleaned through ldapmodify directly and ipa-replica-manage could neither see nor clean them. Console logs below in case anyone can shed some light on it. I've re-installed the replicas again, and I'm hoping it doesn't crash in 12 hours like last time ... --- console output --- [root@dc1-ipa-dev-van slapd-mydomain-NET]# ipa-replica-manage del dc2-ipa-dev-nvan.mydomain.net --force --cleanup Connection to 'dc2-ipa-dev-nvan.mydomain.net' failed: Insufficient access: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context Invalid credentials Forcing removal of dc2-ipa-dev-nvan.mydomain.net Skipping calculation to determine if one or more masters would be orphaned. Deleting replication agreements between dc2-ipa-dev-nvan.mydomain.net and dc1-ipa-dev-van.mydomain.net, dc1-ipa-dev-nvan.mydomain.net Failed to get list of agreements from 'dc2-ipa-dev-nvan.mydomain.net': Insufficient access: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context Invalid credentials Forcing removal on 'dc1-ipa-dev-van.mydomain.net' Any DNA range on 'dc2-ipa-dev-nvan.mydomain.net' will be lost Deleted replication agreement from 'dc1-ipa-dev-van.mydomain.net' to 'dc2-ipa-dev-nvan.mydomain.net' Failed to determine agreement type for 'dc2-ipa-dev-nvan.mydomain.net': Insufficient access: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context Invalid credentials There were issues removing a connection for dc2-ipa-dev-nvan.mydomain.net from dc1-ipa-dev-nvan.mydomain.net: local variable 'type1' referenced before assignment Background task created to clean replication data. This may take a while. This may be safely interrupted with Ctrl+C [root@dc1-ipa-dev-van slapd-mydomain-NET]# [root@dc2-ipa-dev-nvan slapd-mydomain-NET]# ipa-server-install --uninstall This is a NON REVERSIBLE operation and will delete all data and configuration! Are you sure you want to continue with the uninstall procedure? [no]: yes Replication agreements with the following IPA masters found: dc1-ipa-dev-van .mydomain.net. Removing any replication agreements before uninstalling the server is strongly recommended. You can remove replication agreements by running the following command on any other IPA master: $ ipa-replica-manage del dc2-ipa-dev-nvan.mydomain.net Are you sure you want to continue with the uninstall procedure? [no]: yes Shutting down all IPA services Removing IPA client configuration Unconfiguring ntpd Configuring certmonger to stop tracking system certificates for KRA Configuring certmonger to stop tracking system certificates for CA Unconfiguring CA Unconfiguring named Unconfiguring ipa-dnskeysyncd Unconfiguring web server Unconfiguring krb5kdc Unconfiguring kadmin Unconfiguring directory server ipa : ERROR Instance removal failed. ipa : ERROR Failed to remove DS instance. You may need to remove instance data manually Unconfiguring ipa_memcached Unconfiguring ipa-otpd [root@dc2-ipa-dev-nvan slapd-mydomain-NET]# [root@dc1-ipa-dev-van slapd-mydomain-NET]# ipa-csreplica-manage del dc2-ipa-dev-nvan.mydomain.net --force -v Directory Manager password: Unable to connect to replica dc2-ipa-dev-nvan.mydomain.net, forcing removal Failed to get data from 'dc2-ipa-dev-nvan.mydomain.net': cannot connect to 'ldap://dc2-ipa-dev-nvan.mydomain.net:389': Forcing removal on 'dc1-ipa-dev-van.mydomain.net' There were issues removing a connection: 'NoneType' object has no attribute 'port' [root@dc1-ipa-dev-van slapd-mydomain-NET]# ldapsearch -D "cn=directory manager" -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" nscpentrywsi Enter LDAP Password: # extended LDIF # # LDAPv3 # base <o=ipaca> with scope subtree # filter: (&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff)) # requesting: nscpentrywsi # # replica, o\3Dipaca, mapping tree, config dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nscpentrywsi: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nscpentrywsi: objectClass: top nscpentrywsi: objectClass: nsDS5Replica nscpentrywsi: objectClass: extensibleobject nscpentrywsi: nsDS5ReplicaRoot: o=ipaca nscpentrywsi: nsDS5ReplicaType: 3 nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc1-ipa-dev-nvan.mydomain.net-pki-tomcat,ou=csusers,cn=config nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc2-ipa-dev-nvan.mydomain.net-pki-tomcat,ou=csusers,cn=config nscpentrywsi: cn: replica nscpentrywsi: nsDS5ReplicaId: 96 nscpentrywsi: nsDS5Flags: 1 nscpentrywsi: creatorsName: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: modifiersName: cn=Multimaster Replication Plugin,cn=plugins,cn=c onfig nscpentrywsi: createTimestamp: 20160114034427Z nscpentrywsi: modifyTimestamp: 20160115034515Z nscpentrywsi: nsState:: YAAAAAAAAAA3a5hWAAAAAAAAAAAAAAAAAgAAAAAAAAAAAAAAAAAAAA == nscpentrywsi: nsDS5ReplicaName: 0c97968e-ba7111e5-b1f1cd78-f19552bb nscpentrywsi: numSubordinates: 1 nscpentrywsi: nsds50ruv: {replicageneration} 5697199b000000600000 nscpentrywsi: nsds50ruv: {replica 96 ldap://dc1-ipa-dev-van.mydomain.net:389} 569719a0000000600000 56986b35000000600000 nscpentrywsi: nsds50ruv: {replica 76 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 56976b310000004c0000 56976b5c0002004c0000 nscpentrywsi: nsds50ruv: {replica 81 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 5697661a000000510000 56986b55000000510000 nscpentrywsi: nsds50ruv: {replica 86 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 569761d2000000560000 5697620b000500560000 nscpentrywsi: nsds50ruv: {replica 91 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 569738560000005b0000 569738790004005b0000 nscpentrywsi: nsds50ruv: {replica 97 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 569719a4000000610000 569719e6001100610000 nscpentrywsi: nsds5agmtmaxcsn: o=ipaca;masterAgreement1-dc1-ipa-dev-nvan.mydomain.net-pki-tomcat;dc1-ipa-dev-nvan.mydomain.net;389;81;56986b3 5000000600000 nscpentrywsi: nsruvReplicaLastModified: {replica 96 ldap://dc1-ipa-dev-van.mydomain.net:389} 56986b33 nscpentrywsi: nsruvReplicaLastModified: {replica 76 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 56976b68 nscpentrywsi: nsruvReplicaLastModified: {replica 81 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 56986b54 nscpentrywsi: nsruvReplicaLastModified: {replica 86 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 56976208 nscpentrywsi: nsruvReplicaLastModified: {replica 91 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 56973881 nscpentrywsi: nsruvReplicaLastModified: {replica 97 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 00000000 nscpentrywsi: nsds5ReplicaChangeCount: 1464 nscpentrywsi: nsds5replicareapactive: 0 # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 [root@dc1-ipa-dev-van slapd-mydomain-NET]# ipa-replica-manage list-ruv dc2-ipa-dev-nvan.mydomain.net:389: 10 dc1-ipa-dev-van.mydomain.net:389: 4 dc1-ipa-dev-nvan.mydomain.net:389: 9 [root@dc1-ipa-dev-van slapd-mydomain-NET]# ipa-replica-manage clean-ruv 10 Clean the Replication Update Vector for dc2-ipa-dev-nvan.mydomain.net:389 Cleaning the wrong replica ID will cause that server to no longer replicate so it may miss updates while the process is running. It would need to be re-initialized to maintain consistency. Be very careful. Continue to clean? [no]: yes Background task created to clean replication data. This may take a while. This may be safely interrupted with Ctrl+C Cleanup task created [root@dc1-ipa-dev-van slapd-mydomain-NET]# ipa-replica-manage list-ruv dc1-ipa-dev-van.mydomain.net:389: 4 dc1-ipa-dev-nvan.mydomain.net:389: 9 [root@dc1-ipa-dev-van slapd-mydomain-NET]# [root@dc1-ipa-dev-van slapd-mydomain-NET]# ldapsearch -D "cn=directory manager" -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" nscpentrywsi Enter LDAP Password: # extended LDIF # # LDAPv3 # base <o=ipaca> with scope subtree # filter: (&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff)) # requesting: nscpentrywsi # # replica, o\3Dipaca, mapping tree, config dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nscpentrywsi: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nscpentrywsi: objectClass: top nscpentrywsi: objectClass: nsDS5Replica nscpentrywsi: objectClass: extensibleobject nscpentrywsi: nsDS5ReplicaRoot: o=ipaca nscpentrywsi: nsDS5ReplicaType: 3 nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc1-ipa-dev-nvan.mydomain.net-pki-tomcat,ou=csusers,cn=config nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc2-ipa-dev-nvan.mydomain.net-pki-tomcat,ou=csusers,cn=config nscpentrywsi: cn: replica nscpentrywsi: nsDS5ReplicaId: 96 nscpentrywsi: nsDS5Flags: 1 nscpentrywsi: creatorsName: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: modifiersName: cn=Multimaster Replication Plugin,cn=plugins,cn=c onfig nscpentrywsi: createTimestamp: 20160114034427Z nscpentrywsi: modifyTimestamp: 20160115034515Z nscpentrywsi: nsState:: YAAAAAAAAAA3a5hWAAAAAAAAAAAAAAAAAgAAAAAAAAAAAAAAAAAAAA == nscpentrywsi: nsDS5ReplicaName: 0c97968e-ba7111e5-b1f1cd78-f19552bb nscpentrywsi: numSubordinates: 1 nscpentrywsi: nsds50ruv: {replicageneration} 5697199b000000600000 nscpentrywsi: nsds50ruv: {replica 96 ldap://dc1-ipa-dev-van.mydomain.net:389} 569719a0000000600000 56986b35000000600000 nscpentrywsi: nsds50ruv: {replica 76 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 56976b310000004c0000 56976b5c0002004c0000 nscpentrywsi: nsds50ruv: {replica 81 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 5697661a000000510000 56986b55000000510000 nscpentrywsi: nsds50ruv: {replica 86 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 569761d2000000560000 5697620b000500560000 nscpentrywsi: nsds50ruv: {replica 91 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 569738560000005b0000 569738790004005b0000 nscpentrywsi: nsds50ruv: {replica 97 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 569719a4000000610000 569719e6001100610000 nscpentrywsi: nsds5agmtmaxcsn: o=ipaca;masterAgreement1-dc1-ipa-dev-nvan.mydomain.net-pki-tomcat;dc1-ipa-dev-nvan.mydomain.net;389;81;56986b3 5000000600000 nscpentrywsi: nsruvReplicaLastModified: {replica 96 ldap://dc1-ipa-dev-van.mydomain.net:389} 56986b33 nscpentrywsi: nsruvReplicaLastModified: {replica 76 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 56976b68 nscpentrywsi: nsruvReplicaLastModified: {replica 81 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 56986b54 nscpentrywsi: nsruvReplicaLastModified: {replica 86 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 56976208 nscpentrywsi: nsruvReplicaLastModified: {replica 91 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 56973881 nscpentrywsi: nsruvReplicaLastModified: {replica 97 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 00000000 nscpentrywsi: nsds5ReplicaChangeCount: 1464 nscpentrywsi: nsds5replicareapactive: 0 # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 [root@dc1-ipa-dev-van slapd-mydomain-NET]# [root@dc1-ipa-dev-van slapd-mydomain-NET]# ipa-replica-manage del dc1-ipa-dev-nvan.mydomain.net --force --cleanup Connection to 'dc1-ipa-dev-nvan.mydomain.net' failed: Insufficient access: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context Invalid credentials Forcing removal of dc1-ipa-dev-nvan.mydomain.net Skipping calculation to determine if one or more masters would be orphaned. Deleting replication agreements between dc1-ipa-dev-nvan.mydomain.net and dc1-ipa-dev-van.mydomain.net Failed to get list of agreements from 'dc1-ipa-dev-nvan.mydomain.net': Insufficient access: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context Invalid credentials Forcing removal on 'dc1-ipa-dev-van.mydomain.net' Any DNA range on 'dc1-ipa-dev-nvan.mydomain.net' will be lost Deleted replication agreement from 'dc1-ipa-dev-van.mydomain.net' to 'dc1-ipa-dev-nvan.mydomain.net' Background task created to clean replication data. This may take a while. This may be safely interrupted with Ctrl+C Failed to cleanup dc1-ipa-dev-nvan.mydomain.net entries: Operations error: You may need to manually remove them from the tree [root@dc1-ipa-dev-van slapd-mydomain-NET]# [root@dc1-ipa-dev-van slapd-mydomain-NET]# ipa-csreplica-manage del dc1-ipa-dev-nvan.mydomain.net --force Directory Manager password: Unable to connect to replica dc1-ipa-dev-nvan.mydomain.net, forcing removal Failed to get data from 'dc1-ipa-dev-nvan.mydomain.net': cannot connect to 'ldap://dc1-ipa-dev-nvan.mydomain.net:389': Forcing removal on 'dc1-ipa-dev-van.mydomain.net' There were issues removing a connection: 'NoneType' object has no attribute 'port' [root@dc1-ipa-dev-nvan slapd-mydomain-NET]# ipa-server-install --uninstall This is a NON REVERSIBLE operation and will delete all data and configuration! Are you sure you want to continue with the uninstall procedure? [no]: yes Replication agreements with the following IPA masters found: dc1-ipa-dev-van .mydomain.net. Removing any replication agreements before uninstalling the server is strongly recommended. You can remove replication agreements by running the following command on any other IPA master: $ ipa-replica-manage del dc1-ipa-dev-nvan.mydomain.net Are you sure you want to continue with the uninstall procedure? [no]: yes Shutting down all IPA services Removing IPA client configuration Unconfiguring ntpd Configuring certmonger to stop tracking system certificates for KRA Configuring certmonger to stop tracking system certificates for CA Unconfiguring CA Unconfiguring named Unconfiguring ipa-dnskeysyncd Unconfiguring web server ipa : ERROR Command ''/bin/systemctl' 'restart' 'httpd.service'' returned non-zero exit status 1 [root@dc1-ipa-dev-nvan slapd-mydomain-NET]# ipa-server-install --uninstall This is a NON REVERSIBLE operation and will delete all data and configuration! Are you sure you want to continue with the uninstall procedure? [no]: yes WARNING: Failed to connect to Directory Server to find information about replication agreements. Uninstallation will continue despite the possible existing replication agreements. Shutting down all IPA services Removing IPA client configuration Configuring certmonger to stop tracking system certificates for KRA Configuring certmonger to stop tracking system certificates for CA Unconfiguring krb5kdc Unconfiguring kadmin Unconfiguring directory server ipa : ERROR Instance removal failed. ipa : ERROR Failed to remove DS instance. You may need to remove instance data manually Unconfiguring ipa_memcached Unconfiguring ipa-otpd [root@dc1-ipa-dev-nvan slapd-mydomain-NET]# ipa-server-install --uninstall This is a NON REVERSIBLE operation and will delete all data and configuration! Are you sure you want to continue with the uninstall procedure? [no]: yes WARNING: Failed to connect to Directory Server to find information about replication agreements. Uninstallation will continue despite the possible existing replication agreements. Shutting down all IPA services Removing IPA client configuration Configuring certmonger to stop tracking system certificates for KRA Configuring certmonger to stop tracking system certificates for CA [root@dc1-ipa-dev-nvan slapd-mydomain-NET]# [root@dc2-ipa-dev-nvan slapd-mydomain-NET]# ipa-server-install --uninstall This is a NON REVERSIBLE operation and will delete all data and configuration! Are you sure you want to continue with the uninstall procedure? [no]: yes Replication agreements with the following IPA masters found: dc1-ipa-dev-van .mydomain.net. Removing any replication agreements before uninstalling the server is strongly recommended. You can remove replication agreements by running the following command on any other IPA master: $ ipa-replica-manage del dc2-ipa-dev-nvan.mydomain.net Are you sure you want to continue with the uninstall procedure? [no]: yes Shutting down all IPA services Removing IPA client configuration Unconfiguring ntpd Configuring certmonger to stop tracking system certificates for KRA Configuring certmonger to stop tracking system certificates for CA Unconfiguring CA Unconfiguring named Unconfiguring ipa-dnskeysyncd Unconfiguring web server Unconfiguring krb5kdc Unconfiguring kadmin Unconfiguring directory server ipa : ERROR Instance removal failed. ipa : ERROR Failed to remove DS instance. You may need to remove instance data manually Unconfiguring ipa_memcached Unconfiguring ipa-otpd [root@dc2-ipa-dev-nvan slapd-mydomain-NET]# ipa-server-install --uninstall This is a NON REVERSIBLE operation and will delete all data and configuration! Are you sure you want to continue with the uninstall procedure? [no]: yes WARNING: Failed to connect to Directory Server to find information about replication agreements. Uninstallation will continue despite the possible existing replication agreements. Shutting down all IPA services Removing IPA client configuration Configuring certmonger to stop tracking system certificates for KRA Configuring certmonger to stop tracking system certificates for CA [root@dc2-ipa-dev-nvan slapd-mydomain-NET]# [root@dc1-ipa-dev-van slapd-mydomain-NET]# ipa-replica-manage clean-all-ruv Usage: ipa-replica-manage [options] ipa-replica-manage: error: must provide a command [clean-ruv | dnarange-set | list-ruv | dnarange-show | connect | force-sync | list-clean-ruv | disconnect | list | dnanextrange-set | dnanextrange-show | del | re-initialize | abort-clean-ruv] [root@dc1-ipa-dev-van slapd-mydomain-NET]# ipa-replica-manage clean-ruv 9 Clean the Replication Update Vector for dc1-ipa-dev-nvan.mydomain.net:389 Cleaning the wrong replica ID will cause that server to no longer replicate so it may miss updates while the process is running. It would need to be re-initialized to maintain consistency. Be very careful. Continue to clean? [no]: yes Background task created to clean replication data. This may take a while. This may be safely interrupted with Ctrl+C Cleanup task created [root@dc1-ipa-dev-van slapd-mydomain-NET]# ipa-replica-manage list-ruv unexpected error: Insufficient access: SASL(-14): authorization failure: Invalid credentials [root@dc1-ipa-dev-van slapd-mydomain-NET]# kdestroy [root@dc1-ipa-dev-van slapd-mydomain-NET]# kinit nathan.peters Password for nathan.pet...@mydomain.net: [root@dc1-ipa-dev-van slapd-mydomain-NET]# ipa-replica-manage list-ruv dc1-ipa-dev-van.mydomain.net:389: 4 [root@dc1-ipa-dev-van slapd-mydomain-NET]# [root@dc1-ipa-dev-van slapd-mydomain-NET]# ldapsearch -D "cn=directory manager" -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" nscpentrywsi Enter LDAP Password: # extended LDIF # # LDAPv3 # base <o=ipaca> with scope subtree # filter: (&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff)) # requesting: nscpentrywsi # # replica, o\3Dipaca, mapping tree, config dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nscpentrywsi: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nscpentrywsi: objectClass: top nscpentrywsi: objectClass: nsDS5Replica nscpentrywsi: objectClass: extensibleobject nscpentrywsi: nsDS5ReplicaRoot: o=ipaca nscpentrywsi: nsDS5ReplicaType: 3 nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc1-ipa-dev-nvan.mydomain.net-pki-tomcat,ou=csusers,cn=config nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc2-ipa-dev-nvan.mydomain.net-pki-tomcat,ou=csusers,cn=config nscpentrywsi: cn: replica nscpentrywsi: nsDS5ReplicaId: 96 nscpentrywsi: nsDS5Flags: 1 nscpentrywsi: creatorsName: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: modifiersName: cn=Multimaster Replication Plugin,cn=plugins,cn=c onfig nscpentrywsi: createTimestamp: 20160114034427Z nscpentrywsi: modifyTimestamp: 20160115040015Z nscpentrywsi: nsState:: YAAAAAAAAAC3bphWAAAAAAAAAAAAAAAAAgAAAAAAAAACAAAAAAAAAA == nscpentrywsi: nsDS5ReplicaName: 0c97968e-ba7111e5-b1f1cd78-f19552bb nscpentrywsi: nsds50ruv: {replicageneration} 5697199b000000600000 nscpentrywsi: nsds50ruv: {replica 96 ldap://dc1-ipa-dev-van.mydomain.net:389} 569719a0000000600000 56986eb9000000600000 nscpentrywsi: nsds50ruv: {replica 76 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 56976b310000004c0000 56976b5c0002004c0000 nscpentrywsi: nsds50ruv: {replica 81 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 5697661a000000510000 56986b55000000510000 nscpentrywsi: nsds50ruv: {replica 86 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 569761d2000000560000 5697620b000500560000 nscpentrywsi: nsds50ruv: {replica 91 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 569738560000005b0000 569738790004005b0000 nscpentrywsi: nsds50ruv: {replica 97 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 569719a4000000610000 569719e6001100610000 nscpentrywsi: nsruvReplicaLastModified: {replica 96 ldap://dc1-ipa-dev-van.mydomain.net:389} 56986eb7 nscpentrywsi: nsruvReplicaLastModified: {replica 76 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 56976b68 nscpentrywsi: nsruvReplicaLastModified: {replica 81 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 56986b54 nscpentrywsi: nsruvReplicaLastModified: {replica 86 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 56976208 nscpentrywsi: nsruvReplicaLastModified: {replica 91 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 56973881 nscpentrywsi: nsruvReplicaLastModified: {replica 97 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 00000000 nscpentrywsi: nsds5ReplicaChangeCount: 1465 nscpentrywsi: nsds5replicareapactive: 0 # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 [root@dc1-ipa-dev-van slapd-mydomain-NET]# dn: cn=clean 76, cn=cleanallruv, cn=tasks, cn=config objectclass: extensibleObject replica-base-dn: dc=dev-mydomain,dc=net replica-id: 76 replica-force-cleaning: yes cn: clean 76 ldapmodify -x -D "cn=directory manager" -W <<EOF dn: cn=replica,cn=dc\3Ddev-mydomain\2Cdc\3Dnet,cn=mapping tree,cn=config changetype: modify replace: nsds5task nsds5task: CLEANRUV76 EOF ldapmodify -x -D "cn=directory manager" -W <<EOF dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config changetype: modify replace: nsds5task nsds5task: CLEANRUV76 EOF ldapmodify -x -D "cn=directory manager" -W <<EOF dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config changetype: modify replace: nsds5task nsds5task: CLEANRUV81 EOF ldapmodify -x -D "cn=directory manager" -W <<EOF dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config changetype: modify replace: nsds5task nsds5task: CLEANRUV91 EOF ==== SERVER CRASHED HERE ==== [15/Jan/2016:05:21:46 +0000] - acquire_replica, supplier RUV is newer [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - windows sync - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): Cancelling linger on the connection [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - windows sync - windows_acquire_replica returned success (101) [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - windows sync - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): State: ready_to_acquire_replica -> sending_updates [15/Jan/2016:05:21:46 +0000] - csngen_adjust_time: gen state before 569882e20004:1452835306:0:248 [15/Jan/2016:05:21:46 +0000] - csngen_adjust_time: gen state after 569882e20004:1452835306:0:248 [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - changelog program - _cl5GetDBFile: found DB object 7ffa5b17b8f0 for database /var/lib/dirsrv/slapd-DEV-mydomain-NET/cldb/e054c085-ede211e4-bf10cd78-f19552bb_553fe9bb000000040000.db [15/Jan/2016:05:21:46 +0000] - _cl5PositionCursorForReplay (agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389)): Consumer RUV: [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replicageneration} 553fe9bb000000040000 [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 4 ldap://dc1-ipa-dev-van.dev-mydomain.net:389} 553fe9c9000000040000 569882e2000000040000 569881ea [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 3 ldap://dc1-ipa-dev-nvan.dev-mydomain.net:389} 553fe9c4000000030000 5696f872000300030000 00000000 [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 5} 56921205000100050000 56972b38000500050000 5698802b [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 6} 56971a3b000000060000 56974fcf000400060000 56988036 [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 7} 569738e8000200070000 56975902000100070000 5698803b [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 8} 56976262000000080000 5697639a000000080000 56988049 [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 9} 569766ae000000090000 56986c8f000000090000 5698808b [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 10} 56976bc60000000a0000 5698139b0002000a0000 5698807a [15/Jan/2016:05:21:46 +0000] - _cl5PositionCursorForReplay (agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389)): Supplier RUV: [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replicageneration} 553fe9bb000000040000 [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 4 ldap://dc1-ipa-dev-van.dev-mydomain.net:389} 553fe9c9000000040000 569882e2000200040000 569881ea [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 3} 56846eee000300030000 56846eee000300030000 5698802a [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 5} 56972b38000500050000 56972b38000500050000 5698802a [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 6} 56974fcf000400060000 56974fcf000400060000 5698802a [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 7} 56975902000100070000 56975902000100070000 5698802a [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 8} 5697639a000000080000 5697639a000000080000 5698802a [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 9} 56986c8f000000090000 56986c8f000000090000 5698802a [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 10} 5698139b0002000a0000 5698139b0002000a0000 5698802a [15/Jan/2016:05:21:46 +0000] agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389) - clcache_get_buffer: found thread private buffer cache 7ffa2c0746a0 [15/Jan/2016:05:21:46 +0000] agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389) - clcache_get_buffer: _pool is 7ffa5b425660 _pool->pl_busy_lists is 7ffa2c075c30 _pool->pl_busy_lists->bl_buffers is 7ffa2c0746a0 [15/Jan/2016:05:21:46 +0000] agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389) - session start: anchorcsn=569882e2000000040000 [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - changelog program - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): CSN 569882e2000000040000 found, position set for replay [15/Jan/2016:05:21:46 +0000] agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389) - load=1 rec=1 csn=569882e2000200040000 [15/Jan/2016:05:21:46 +0000] agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389) - clcache_load_buffer: rc=-30988 [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - windows sync - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): No more updates to send (cl5GetNextOperationToReplay) [15/Jan/2016:05:21:46 +0000] agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389) - session end: state=5 load=1 sent=1 skipped=0 skipped_new_rid=0 skipped_csn_gt_cons_maxcsn=0 skipped_up_to_date=0 skipped_csn_gt_ruv=0 skipped_csn_covered=0 [15/Jan/2016:05:21:46 +0000] - Calling dirsync search request plugin [15/Jan/2016:05:21:46 +0000] - Sending dirsync search request [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - windows sync - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): Beginning linger on the connection [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - windows sync - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): State: sending_updates -> wait_for_changes [15/Jan/2016:05:21:47 +0000] - _csngen_adjust_local_time: gen state before 569882e20004:1452835306:0:248 [15/Jan/2016:05:21:47 +0000] - _csngen_adjust_local_time: gen state after 569882e30000:1452835307:0:248 [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - ruv_add_csn_inprogress: successfully inserted csn 569882e3000000040000 into pending list [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - Purged state information from entry fqdn=zk1-msg-mbsnap1-nva.dev-mydomain.net,cn=computers,cn=accounts,dc=dev-mydomain,dc=net up to CSN 568f4862000200040000 [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - changelog program - _cl5GetDBFileByReplicaName: found DB object 7ffa5b17b8f0 for database /var/lib/dirsrv/slapd-DEV-mydomain-NET/cldb/e054c085-ede211e4-bf10cd78-f19552bb_553fe9bb000000040000.db [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - changelog program - _cl5GetDBFileByReplicaName: found DB object 7ffa5b17b8f0 for database /var/lib/dirsrv/slapd-DEV-mydomain-NET/cldb/e054c085-ede211e4-bf10cd78-f19552bb_553fe9bb000000040000.db [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - ruv_update_ruv: successfully committed csn 569882e3000000040000 [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - windows sync - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): State: wait_for_changes -> wait_for_changes [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - windows sync - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): State: wait_for_changes -> ready_to_acquire_replica [15/Jan/2016:05:21:47 +0000] - acquire_replica, supplier RUV: [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - supplier: {replicageneration} 553fe9bb000000040000 [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - supplier: {replica 4 ldap://dc1-ipa-dev-van.dev-mydomain.net:389} 553fe9c9000000040000 569882e3000000040000 569881eb [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - supplier: {replica 3} 56846eee000300030000 56846eee000300030000 5698802a [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - supplier: {replica 5} 56972b38000500050000 56972b38000500050000 5698802a [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - supplier: {replica 6} 56974fcf000400060000 56974fcf000400060000 5698802a [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - supplier: {replica 7} 56975902000100070000 56975902000100070000 5698802a [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - supplier: {replica 8} 5697639a000000080000 5697639a000000080000 5698802a [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - supplier: {replica 9} 56986c8f000000090000 56986c8f000000090000 5698802a [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - supplier: {replica 10} 5698139b0002000a0000 5698139b0002000a0000 5698802a [15/Jan/2016:05:21:47 +0000] - acquire_replica, consumer RUV: [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - consumer: {replicageneration} 553fe9bb000000040000 [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - consumer: {replica 4 ldap://dc1-ipa-dev-van.dev-mydomain.net:389} 553fe9c9000000040000 569882e2000200040000 569881ea [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - consumer: {replica 3 ldap://dc1-ipa-dev-nvan.dev-mydomain.net:389} 553fe9c4000000030000 5696f872000300030000 00000000 [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - consumer: {replica 5} 56921205000100050000 56972b38000500050000 5698802b ^C [root@dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ipa-replica-manage list-ruv unable to decode: {replica 7} 56975902000100070000 56975902000100070000 unable to decode: {replica 10} 5698139b0002000a0000 5698139b0002000a0000 unable to decode: {replica 5} 56972b38000500050000 56972b38000500050000 unable to decode: {replica 8} 5697639a000000080000 5697639a000000080000 unable to decode: {replica 6} 56974fcf000400060000 56974fcf000400060000 unable to decode: {replica 3} 56846eee000300030000 56846eee000300030000 unable to decode: {replica 9} 56986c8f000000090000 56986c8f000000090000 dc1-ipa-dev-van.dev-mydomain.net:389: 4 [root@dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ipa-replica-manage clean-ruv 7 unable to decode: {replica 7} 56975902000100070000 56975902000100070000 unable to decode: {replica 10} 5698139b0002000a0000 5698139b0002000a0000 unable to decode: {replica 5} 56972b38000500050000 56972b38000500050000 unable to decode: {replica 8} 5697639a000000080000 5697639a000000080000 unable to decode: {replica 6} 56974fcf000400060000 56974fcf000400060000 unable to decode: {replica 3} 56846eee000300030000 56846eee000300030000 unable to decode: {replica 9} 56986c8f000000090000 56986c8f000000090000 Replica ID 7 not found [root@dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ipa-replica-manage list-ruv unable to decode: {replica 7} 56975902000100070000 56975902000100070000 unable to decode: {replica 10} 5698139b0002000a0000 5698139b0002000a0000 unable to decode: {replica 5} 56972b38000500050000 56972b38000500050000 unable to decode: {replica 8} 5697639a000000080000 5697639a000000080000 unable to decode: {replica 6} 56974fcf000400060000 56974fcf000400060000 unable to decode: {replica 3} 56846eee000300030000 56846eee000300030000 unable to decode: {replica 9} 56986c8f000000090000 56986c8f000000090000 dc1-ipa-dev-van.dev-mydomain.net:389: 4 [root@dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ldapmodify -D "cn=directory manager" -W -a Enter LDAP Password: ldap_bind: Invalid credentials (49) [root@dc1-ipa-dev-van slapd-DEV-mydomain-NET]# objectclass: extensibleObject -bash: objectclass:: command not found [root@dc1-ipa-dev-van slapd-DEV-mydomain-NET]# replica-base-dn: dc=dev-mydomain,dc=net -bash: replica-base-dn:: command not found [root@dc1-ipa-dev-van slapd-DEV-mydomain-NET]# replica-id: 7 -bash: replica-id:: command not found [root@dc1-ipa-dev-van slapd-DEV-mydomain-NET]# cn: clean 7MZKXswIqn3arBMw1xzLl -bash: cn:: command not found [root@dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ldapmodify -D "cn=directory manager" -W -a Enter LDAP Password: dn: cn=clean 7, cn=cleanallruv, cn=tasks, cn=config objectclass: extensibleObject replica-base-dn: dc=dev-mydomain,dc=net replica-id: 7 cn: clean 7 adding new entry "cn=clean 7, cn=cleanallruv, cn=tasks, cn=config" [root@dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ipa-replica-manage list-ruv unable to decode: {replica 5} 56972b38000500050000 56972b38000500050000 unable to decode: {replica 8} 5697639a000000080000 5697639a000000080000 unable to decode: {replica 6} 56974fcf000400060000 56974fcf000400060000 unable to decode: {replica 3} 56846eee000300030000 56846eee000300030000 unable to decode: {replica 9} 56986c8f000000090000 56986c8f000000090000 unable to decode: {replica 10} 5698139b0002000a0000 5698139b0002000a0000 dc1-ipa-dev-van.dev-mydomain.net:389: 4 [root@dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ldapmodify -D "cn=directory manager" -W -a Enter LDAP Password: dn: cn=clean 5, cn=cleanallruv, cn=tasks, cn=config objectclass: extensibleObject replica-base-dn: dc=dev-mydomain,dc=net replica-id: 5 cn: clean 5 adding new entry "cn=clean 5, cn=cleanallruv, cn=tasks, cn=config" dn: cn=clean 8, cn=cleanallruv, cn=tasks, cn=config objectclass: extensibleObject replica-base-dn: dc=dev-mydomain,dc=net replica-id: 8 cn: clean 8 adding new entry "cn=clean 8, cn=cleanallruv, cn=tasks, cn=config" dn: cn=clean 6, cn=cleanallruv, cn=tasks, cn=config objectclass: extensibleObject replica-base-dn: dc=dev-mydomain,dc=net replica-id: 6 cn: clean 6 adding new entry "cn=clean 6, cn=cleanallruv, cn=tasks, cn=config" dn: cn=clean 3, cn=cleanallruv, cn=tasks, cn=config objectclass: extensibleObject replica-base-dn: dc=dev-mydomain,dc=net replica-id: 3 cn: clean 3 adding new entry "cn=clean 3, cn=cleanallruv, cn=tasks, cn=config" dn: cn=clean 9, cn=cleanallruv, cn=tasks, cn=config objectclass: extensibleObject replica-base-dn: dc=dev-mydomain,dc=net replica-id: 9 cn: clean 9 adding new entry "cn=clean 9, cn=cleanallruv, cn=tasks, cn=config" dn: cn=clean 10, cn=cleanallruv, cn=tasks, cn=config objectclass: extensibleObject replica-base-dn: dc=dev-mydomain,dc=net replica-id: 10 cn: clean 10 [root@dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ldapsearch -D "cn=directory manager" -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" nscpentrywsi Enter LDAP Password: # extended LDIF # # LDAPv3 # base <o=ipaca> with scope subtree # filter: (&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff)) # requesting: nscpentrywsi # # replica, o\3Dipaca, mapping tree, config dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nscpentrywsi: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nscpentrywsi: objectClass: top nscpentrywsi: objectClass: nsDS5Replica nscpentrywsi: objectClass: extensibleobject nscpentrywsi: nsDS5ReplicaRoot: o=ipaca nscpentrywsi: nsDS5ReplicaType: 3 nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc1- ipa-dev-nvan.dev-mydomain.net-pki-tomcat,ou=csusers,cn=config nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc2- ipa-dev-nvan.dev-mydomain.net-pki-tomcat,ou=csusers,cn=config nscpentrywsi: cn: replica nscpentrywsi: nsDS5ReplicaId: 96 nscpentrywsi: nsDS5Flags: 1 nscpentrywsi: creatorsName: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: modifiersName: cn=Multimaster Replication Plugin,cn=plugins,cn=c onfig nscpentrywsi: createTimestamp: 20160114034427Z nscpentrywsi: modifyTimestamp: 20160115060020Z nscpentrywsi: nsState:: YAAAAAAAAADXiphWAAAAAAAAAAAAAAAAAgAAAAAAAAABAAAAAAAAAA == nscpentrywsi: nsDS5ReplicaName: 0c97968e-ba7111e5-b1f1cd78-f19552bb nscpentrywsi: nsds50ruv: {replicageneration} 5697199b000000600000 nscpentrywsi: nsds50ruv: {replica 96 ldap://dc1-ipa-dev-van.dev-mydomain.ne t:389} 569719a0000000600000 56988ad9000000600000 nscpentrywsi: nsds50ruv: {replica 97 ldap://dc1-ipa-dev-nvan.dev-mydomain.n et:389} 569719a4000000610000 569719e6001100610000 nscpentrywsi: nsds50ruv: {replica 91} 569738790004005b0000 569738790004005b000 0 nscpentrywsi: nsds50ruv: {replica 86} 5697620b000500560000 5697620b00050056000 0 nscpentrywsi: nsruvReplicaLastModified: {replica 96 ldap://dc1-ipa-dev-van.dev -mydomain.net:389} 56988ad7 nscpentrywsi: nsruvReplicaLastModified: {replica 97 ldap://dc1-ipa-dev-nvan.de v-mydomain.net:389} 00000000 nscpentrywsi: nsruvReplicaLastModified: {replica 91} 5698802a nscpentrywsi: nsruvReplicaLastModified: {replica 86} 5698802a nscpentrywsi: nsds5ReplicaChangeCount: 908 nscpentrywsi: nsds5replicareapactive: 0 # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 [root@dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ldapmodify -x -D "cn=directory manager" -W <<EOF dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config changetype: modify replace: nsds5task nsds5task: CLEANRUV91 EOF Enter LDAP Password: modifying entry "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config" [root@dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ldapmodify -x -D "cn=directory manager" -W <<EOF dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config changetype: modify replace: nsds5task nsds5task: CLEANRUV86 EOF Enter LDAP Password: modifying entry "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config" [root@dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ldapsearch -D "cn=directory manager" -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" nscpentrywsi Enter LDAP Password: # extended LDIF # # LDAPv3 # base <o=ipaca> with scope subtree # filter: (&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff)) # requesting: nscpentrywsi # # replica, o\3Dipaca, mapping tree, config dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nscpentrywsi: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nscpentrywsi: objectClass: top nscpentrywsi: objectClass: nsDS5Replica nscpentrywsi: objectClass: extensibleobject nscpentrywsi: nsDS5ReplicaRoot: o=ipaca nscpentrywsi: nsDS5ReplicaType: 3 nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc1- ipa-dev-nvan.dev-mydomain.net-pki-tomcat,ou=csusers,cn=config nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc2- ipa-dev-nvan.dev-mydomain.net-pki-tomcat,ou=csusers,cn=config nscpentrywsi: cn: replica nscpentrywsi: nsDS5ReplicaId: 96 nscpentrywsi: nsDS5Flags: 1 nscpentrywsi: creatorsName: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: modifiersName: cn=Multimaster Replication Plugin,cn=plugins,cn=c onfig nscpentrywsi: createTimestamp: 20160114034427Z nscpentrywsi: modifyTimestamp: 20160115061052Z nscpentrywsi: nsState:: YAAAAAAAAADXiphWAAAAAAAAAAAAAAAAAgAAAAAAAAABAAAAAAAAAA == nscpentrywsi: nsDS5ReplicaName: 0c97968e-ba7111e5-b1f1cd78-f19552bb nscpentrywsi: nsds50ruv: {replicageneration} 5697199b000000600000 nscpentrywsi: nsds50ruv: {replica 96 ldap://dc1-ipa-dev-van.dev-mydomain.ne t:389} 569719a0000000600000 56988ad9000000600000 nscpentrywsi: nsds50ruv: {replica 97 ldap://dc1-ipa-dev-nvan.dev-mydomain.n et:389} 569719a4000000610000 569719e6001100610000 nscpentrywsi: nsruvReplicaLastModified: {replica 96 ldap://dc1-ipa-dev-van.dev -mydomain.net:389} 56988ad7 nscpentrywsi: nsruvReplicaLastModified: {replica 97 ldap://dc1-ipa-dev-nvan.de v-mydomain.net:389} 00000000 nscpentrywsi: nsds5ReplicaChangeCount: 430 nscpentrywsi: nsds5replicareapactive: 0 # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 [root@dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ldapsearch -xLLL -D "cn=directory manager" -W -b dc=dev-mydomain,dc=net \ '(&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))' ldapmodify -D "cn=directory manager" -W -a dn: cn=clean 7, cn=cleanallruv, cn=tasks, cn=config objectclass: extensibleObject replica-base-dn: dc=dev-mydomain,dc=net replica-id: 7 cn: clean 7 ldapmodify -D "cn=directory manager" -W -a dn: cn=clean 5, cn=cleanallruv, cn=tasks, cn=config objectclass: extensibleObject replica-base-dn: dc=dev-mydomain,dc=net replica-id: 5 cn: clean 5 dn: cn=clean 8, cn=cleanallruv, cn=tasks, cn=config objectclass: extensibleObject replica-base-dn: dc=dev-mydomain,dc=net replica-id: 8 cn: clean 8 dn: cn=clean 6, cn=cleanallruv, cn=tasks, cn=config objectclass: extensibleObject replica-base-dn: dc=dev-mydomain,dc=net replica-id: 6 cn: clean 6 dn: cn=clean 3, cn=cleanallruv, cn=tasks, cn=config objectclass: extensibleObject replica-base-dn: dc=dev-mydomain,dc=net replica-id: 3 cn: clean 3 dn: cn=clean 9, cn=cleanallruv, cn=tasks, cn=config objectclass: extensibleObject replica-base-dn: dc=dev-mydomain,dc=net replica-id: 9 cn: clean 9 dn: cn=clean 10, cn=cleanallruv, cn=tasks, cn=config objectclass: extensibleObject replica-base-dn: dc=dev-mydomain,dc=net replica-id: 10 cn: clean 10 dn: cn=clean 86, cn=cleanallruv, cn=tasks, cn=config objectclass: extensibleObject replica-base-dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config replica-id: 86 cn: clean 86 -----Original Message----- From: freeipa-users-boun...@redhat.com [mailto:freeipa-users-boun...@redhat.com] On Behalf Of Nathan Peters Sent: January-14-16 8:25 PM To: Rob Crittenden; Ludwig Krispenz; freeipa-users@redhat.com Subject: Re: [Freeipa-users] Replication failing on FreeIPA 4.2.0 And the saga continues... In my latest round of trying to fix this, I've attempted to remove the replicas again, this time ensuring to use the --force and --cleanup flags to try to remove the data. As you can see from the output below, it seems like every possible error that could happen did. Some examples : Ruvs needed to be manually cleaned. Ldapsearch reveals that nothing at all has been deleted in the ruv section, and I still have 6 duplicates somehow ipa : ERROR Instance removal failed. ipa : ERROR Failed to remove DS instance. You may need to remove instance data manually SASL failures while removing or trying to get replication agreements At this point I think I may need to manually clean all the old data, but I'm not even sure where to start. Also... When dc1 is alone with no replicas, why does he have a ruv for himself... does he need one ? And... isn't there supposed to be some kind of clean-all-ruv task or is that not in 4.2.0 but only a later version ? -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project