Re: [Freeipa-users] nsds5ReplConflict / Replication issue!
On 05/06/2016 03:29 PM, Devin Acosta wrote: I am running the latest FreeIPA on CentOS 7.2. I noticed I had a “nsds5ReplConflict” with an item, i tried to follow the webpage to rename and delete but that failed. Is this the page you looked at: https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/10/html/Administration_Guide/Managing_Replication-Solving_Common_Replication_Conflicts.html If it is the same process, what exactly failed? Thanks, Mark I then tried to have ipa1-i2x reload from ipa01-aws instance, now now it seems to have gone maybe worse? can you please advise how to get back to a healthy system. I initially added a system account as recommended so i could have say like Jira/Confluence do User searches against IDM. [dacosta@ipa1-i2x ~]$ ldapsearch -x -D "cn=directory manager" -w ‘password' -b "dc=rsinc,dc=local" "nsds5ReplConflict=*" \* nsds5ReplConflict # extended LDIF # # LDAPv3 # base
Re: [Freeipa-users] nsds5ReplConflict / Replication issue!
I did try to resync idm1-i2x from ipa01-aws, probably was a bad idea.. Is there any way to basically have it resync and get a fresh copy from the other nodes that are ok? Well it initially started when I noticed errors in the logs about having a conflict on a record. So i was trying to get that record cleaned up. I then though oh maybe I should just have it reload everything from another server, and i wonder if now that's why the box is just giving strange results. i had ipa1-i2x.rsinc.local reload from ipa01-aws.rsinc.local, you can see the output of the commands below about replication status. I can still log into ipa1-i2x.rsinc.local, [dacosta@ipa1-i2x ~]$ ipa-replica-manage -v list ipa02-aws.rsinc.local ipa: WARNING: session memcached servers not running ipa01-aws.rsinc.local: replica last init status: None last init ended: 1970-01-01 00:00:00+00:00 last update status: 0 Replica acquired successfully: Incremental update started last update ended: 1970-01-01 00:00:00+00:00 [dacosta@ipa1-i2x ~]$ ipa-replica-manage -v list ipa01-aws.rsinc.local ipa: WARNING: session memcached servers not running ipa02-aws.rsinc.local: replica last init status: None last init ended: 1970-01-01 00:00:00+00:00 last update status: 0 Replica acquired successfully: Incremental update succeeded last update ended: 2016-05-06 19:47:26+00:00 ipa1-i2x.rsinc.local: replica last init status: 0 Total update succeeded last init ended: 2016-05-06 18:46:29+00:00 last update status: 0 Replica acquired successfully: Incremental update succeeded last update ended: 2016-05-06 19:46:59+00:00 [dacosta@ipa1-i2x ~]$ ipa-replica-manage -v list ipa1-i2x.rsinc.local ipa: WARNING: session memcached servers not running ipa01-aws.rsinc.local: replica last init status: None last init ended: 1970-01-01 00:00:00+00:00 last update status: 1 Can't acquire busy replica last update ended: 1970-01-01 00:00:00+00:00 I do have these errors on (idm1-i2x) in the errors: [06/May/2016:18:48:46 +] NSMMReplicationPlugin - ruv_compare_ruv: RUV [changelog max RUV] does not contain element [{replica 4 ldap://ipa01-aws.rsinc.local:389} 56e2f9e70004 572ce68100020004] which is present in RUV [database RUV] [06/May/2016:18:48:46 +] NSMMReplicationPlugin - replica_check_for_data_reload: Warning: for replica dc=rsinc,dc=local there were some differences between the changelog max RUV and the database RUV. If there are obsolete elements in the database RUV, you should remove them using the CLEANALLRUV task. If they are not obsolete, you should check their status to see why there are no changes from those servers in the changelog. [06/May/2016:18:48:46 +] NSMMReplicationPlugin - ruv_compare_ruv: RUV [changelog max RUV] does not contain element [{replica 91 ldap://ipa1-i2x.rsinc.local:389} 56f02d3b005b 56f02d67005b] which is present in RUV [database RUV] [06/May/2016:18:48:46 +] NSMMReplicationPlugin - replica_check_for_data_reload: Warning: for replica o=ipaca there were some differences between the changelog max RUV and the database RUV. If there are obsolete elements in the database RUV, you should remove them using the CLEANALLRUV task. If they are not obsolete, you should check their status to see why there are no changes from those servers in the changelog. [06/May/2016:18:48:46 +] set_krb5_creds - Could not get initial credentials for principal [ldap/ipa1-i2x.rsinc.local@RSINC.LOCAL] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328324 (Generic error (see e-text)) [06/May/2016:18:48:46 +] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available)) errno 0 (Success) [06/May/2016:18:48:46 +] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error -2 (Local error) [06/May/2016:18:48:46 +] NSMMReplicationPlugin - agmt="cn=meToipa01-aws.rsinc.local" (ipa01-aws:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available)) [06/May/2016:18:48:46 +] - slapd started. Listening on All Interfaces port 389 for LDAP requests [06/May/2016:18:48:46 +] - Listening on All Interfaces port 636 for LDAPS requests [06/May/2016:18:48:46 +] - Listening on /var/run/slapd-RSINC-LOCAL.socket for LDAPI requests [06/May/2016:18:48:50 +] NSMMReplicationPlugin - agmt="cn=meToipa01-aws.rsinc.local" (ipa01-aws:389): Replication bind with GSSAPI auth resumed [06/May/2016:18:49:18 +] - Retry count exceeded in delete [06/May/2016:18:49:18 +] DSRetroclPlugin - delete_changerecord: could not delete change record 436145 (rc: 51) Thanks
Re: [Freeipa-users] nsds5ReplConflict / Replication issue!
Please keep freeipa-users in loop Well indeed something bad is happening with replication, did you tried reinitialize replica? Maybe guys from DS will know what is happening. Martin On 06.05.2016 21:51, Devin Acosta wrote: Martin, Well it initially started when I noticed errors in the logs about having a conflict on a record. So i was trying to get that record cleaned up. I then though oh maybe I should just have it reload everything from another server, and i wonder if now that's why the box is just giving strange results. i had ipa1-i2x.rsinc.local reload from ipa01-aws.rsinc.local, you can see the output of the commands below about replication status. I can still log into ipa1-i2x.rsinc.local, [dacosta@ipa1-i2x ~]$ ipa-replica-manage -v list ipa02-aws.rsinc.local ipa: WARNING: session memcached servers not running ipa01-aws.rsinc.local: replica last init status: None last init ended: 1970-01-01 00:00:00+00:00 last update status: 0 Replica acquired successfully: Incremental update started last update ended: 1970-01-01 00:00:00+00:00 [dacosta@ipa1-i2x ~]$ ipa-replica-manage -v list ipa01-aws.rsinc.local ipa: WARNING: session memcached servers not running ipa02-aws.rsinc.local: replica last init status: None last init ended: 1970-01-01 00:00:00+00:00 last update status: 0 Replica acquired successfully: Incremental update succeeded last update ended: 2016-05-06 19:47:26+00:00 ipa1-i2x.rsinc.local: replica last init status: 0 Total update succeeded last init ended: 2016-05-06 18:46:29+00:00 last update status: 0 Replica acquired successfully: Incremental update succeeded last update ended: 2016-05-06 19:46:59+00:00 [dacosta@ipa1-i2x ~]$ ipa-replica-manage -v list ipa1-i2x.rsinc.local ipa: WARNING: session memcached servers not running ipa01-aws.rsinc.local: replica last init status: None last init ended: 1970-01-01 00:00:00+00:00 last update status: 1 Can't acquire busy replica last update ended: 1970-01-01 00:00:00+00:00 I do have these errors on (idm1-i2x) in the errors: [06/May/2016:18:48:46 +] NSMMReplicationPlugin - ruv_compare_ruv: RUV [changelog max RUV] does not contain element [{replica 4 ldap://ipa01-aws.rsinc.local:389} 56e2f9e70004 572ce68100020004] which is present in RUV [database RUV] [06/May/2016:18:48:46 +] NSMMReplicationPlugin - replica_check_for_data_reload: Warning: for replica dc=rsinc,dc=local there were some differences between the changelog max RUV and the database RUV. If there are obsolete elements in the database RUV, you should remove them using the CLEANALLRUV task. If they are not obsolete, you should check their status to see why there are no changes from those servers in the changelog. [06/May/2016:18:48:46 +] NSMMReplicationPlugin - ruv_compare_ruv: RUV [changelog max RUV] does not contain element [{replica 91 ldap://ipa1-i2x.rsinc.local:389} 56f02d3b005b 56f02d67005b] which is present in RUV [database RUV] [06/May/2016:18:48:46 +] NSMMReplicationPlugin - replica_check_for_data_reload: Warning: for replica o=ipaca there were some differences between the changelog max RUV and the database RUV. If there are obsolete elements in the database RUV, you should remove them using the CLEANALLRUV task. If they are not obsolete, you should check their status to see why there are no changes from those servers in the changelog. [06/May/2016:18:48:46 +] set_krb5_creds - Could not get initial credentials for principal [ldap/ipa1-i2x.rsinc.local@RSINC.LOCAL] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328324 (Generic error (see e-text)) [06/May/2016:18:48:46 +] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available)) errno 0 (Success) [06/May/2016:18:48:46 +] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error -2 (Local error) [06/May/2016:18:48:46 +] NSMMReplicationPlugin - agmt="cn=meToipa01-aws.rsinc.local" (ipa01-aws:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available)) [06/May/2016:18:48:46 +] - slapd started. Listening on All Interfaces port 389 for LDAP requests [06/May/2016:18:48:46 +] - Listening on All Interfaces port 636 for LDAPS requests [06/May/2016:18:48:46 +] - Listening on /var/run/slapd-RSINC-LOCAL.socket for LDAPI requests [06/May/2016:18:48:50 +] NSMMReplicationPlugin - agmt="cn=meToipa01-aws.rsinc.local" (ipa01-aws:389): Replication bind with GSSAPI auth resumed [06/May/2016:18:49:18 +] - Retry count exceeded in delete [06/May/2016:18:49:18 +] DSRetroclPlugin -
Re: [Freeipa-users] nsds5ReplConflict / Replication issue!
On 06.05.2016 21:29, Devin Acosta wrote: I am running the latest FreeIPA on CentOS 7.2. I noticed I had a “nsds5ReplConflict” with an item, i tried to follow the webpage to rename and delete but that failed. I then tried to have ipa1-i2x reload from ipa01-aws instance, now now it seems to have gone maybe worse? can you please advise how to get back to a healthy system. I initially added a system account as recommended so i could have say like Jira/Confluence do User searches against IDM. [dacosta@ipa1-i2x ~]$ ldapsearch -x -D "cn=directory manager" -w ‘password' -b "dc=rsinc,dc=local" "nsds5ReplConflict=*" \* nsds5ReplConflict # extended LDIF # # LDAPv3 # base