Re: [Freeipa-users] Replication stopped working
On 09/04/2014 05:11 PM, Guillermo Fuentes wrote: Hello list, We’re running FreeIPA with a master and 3 replicas. The replication stopped working and currently we’re adding resources only to the master. This is the environment we have: m1: OS: CentOS release 6.5 FreeIPA: 3.0.0-37 CA: pki-ca-9.0.3 # ipa-replica-manage list -v `hostname` m2.example.com: replica last init status: None last init ended: None last update status: 49 - LDAP error: Invalid credentials last update ended: None m3.example.com: replica last init status: None last init ended: None last update status: 0 Replica acquired successfully: Incremental update succeeded last update ended: 2014-09-04 14:28:44+00:00 m4.example.com: replica last init status: None last init ended: None last update status: -2 - LDAP error: Local error last update ended: None m2: OS: CentOS release 6.5 FreeIPA: 3.0.0-37 # ipa-replica-manage list -v `hostname` m1.example.com: replica last init status: None last init ended: None last update status: -1 Incremental update has failed and requires administrator actionLDAP error: Can't contact LDAP server last update ended: 2014-09-03 22:53:21+00:00 m3: OS: CentOS release 6.5 FreeIPA: 3.0.0-37 # ipa-replica-manage list -v `hostname` m1.example.com: replica last init status: None last init ended: None last update status: 0 Replica acquired successfully: Incremental update succeeded last update ended: 2014-09-04 14:31:51+00:00 m4: OS: CentOS release 6.5 FreeIPA: 3.3.3-28 # ipa-replica-manage list -v `hostname` m1.example.com: replica last init status: None last init ended: None last update status: 49 Unable to acquire replicaLDAP error: Invalid credentials last update ended: None Note that although m3 reports “Incremental update succeeded”, users created on m1 are not replicated to m3, and users created on m3 are not replicated back to m1. We’ve tried different things including re-initializing m2. Can somebody point me in the right direction to get replication going again? Thanks in advance! Guillermo Hello, I think we would need more troubleshooting information that are available in /var/log/dirsrv/slapd-EXAMPLE-COM/errors, especially on m2, m3, m4. Few pointers what I would try myself: 1) Check that all masters have time synced (difference in matter of seconds is OK) 2) Check that DNS is all right - all replicas can resolve master's forward and reverse address. Master can resolve all replicas forward and reverse address. This is common source of replication/Kerberos errors (http://www.freeipa.org/page/Troubleshooting#Kerberos_does_not_work) The error Can't contact LDAP server may point to DNS issues. 3) Check that you can do plain ldapsearch from replica to master. Ideally even authenticated with keytab from /etc/dirsrv/ds.keytab HTH, Martin -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go To http://freeipa.org for more info on the project
Re: [Freeipa-users] Replication stopped working
Update: m2 and m3 are now in sync! After making sure ldapsearch was working both ways (m1=m2 and m1=m3) using the server's keytabs (/etc/dirsrv/ds.keytab) for getting the ticket, I re-initialize both replicas and they were able to get updated: @m2 # ipa-replica-manage re-initialize --from m1.example.com @m3 # ipa-replica-manage re-initialize --from m1.example.com Thanks so much for your hint Martin! On Fri, Sep 5, 2014 at 12:43 PM, Guillermo Fuentes guillermo.fuen...@modernizingmedicine.com wrote: Hi Martin, Attached are m2.log, m3.log and m4.log files. 1) All masters are time synced with same NTP server pool. 2) DNS is fine. Forward and reverse lookup. 3) ldapsearch: m1 to m2 and m3 work: kinit -k -t /etc/dirsrv/ds.keytab ldap/`hostname` # getting ticket on m1 ldapsearch -Y GSSAPI -H ldaps://m2.example.com -b dc=example,dc=com uid=testuser ldapsearch -Y GSSAPI -H ldaps://m3.example.com -b dc=example,dc=com uid=testuser m1 to m4 fails: # ldapsearch -Y GSSAPI -H ldaps://m4.example.com -b dc=example,dc=com uid=testuser SASL/GSSAPI authentication started ldap_sasl_interactive_bind_s: Local error (-2) additional info: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (KDC returned error string: FINDING_SERVER_KEY) m2 to m1, and m3 to m1 work fine: kinit -k -t /etc/dirsrv/ds.keytab ldap/`hostname` ldapsearch -Y GSSAPI -H ldaps://m1.example.com -b dc=example,dc=com uid=testuser m4 to m1 fails: # ldapsearch -Y GSSAPI -H ldaps://m1.example.com -b dc=example,dc=com uid=testuser SASL/GSSAPI authentication started ldap_sasl_interactive_bind_s: Invalid credentials (49) additional info: SASL(-14): authorization failure: security flags do not match required m2 and m3 are at the same state now where connections between them and m1 are fine but the updates won't happen logging the following on m1 (/var/log/dirsrv/slapd-EXAMPLE-COM/errors) for both: [05/Sep/2014:12:30:49 -0400] NSMMReplicationPlugin - agmt=cn=meTom3.example.com (m3:389): replay_update: Sending modify operation (dn=uid=testuser,cn=users,cn=accounts,dc=example,dc=com csn=53d66ecb0004) [05/Sep/2014:12:30:49 -0400] - repl5_inc_result_threadmain: read result for message_id 0 [05/Sep/2014:12:30:49 -0400] NSMMReplicationPlugin - agmt=cn=meTom3.example.com (m3:389): replay_update: modifys operation (dn=uid=testuser,cn=users,cn=accounts,dc=example,dc=com csn=53d66ecb0004) not sent - empty [05/Sep/2014:12:30:49 -0400] NSMMReplicationPlugin - agmt=cn=meTom3.example.com (m3:389): replay_update: Consumer successfully sent operation with csn 53d66ecb0004 [05/Sep/2014:12:30:49 -0400] NSMMReplicationPlugin - agmt=cn=meTom3.example.com (m3:389): Skipping update operation with no message_id (uniqueid 04b0b435-5ef311e3-9c91ec9f-6cd72e64, CSN 53d66ecb0004): [05/Sep/2014:12:30:49 -0400] agmt=cn=meTom3.example.com (m3:389) - load=1 rec=38 csn=53d66ecb00020004 [05/Sep/2014:12:30:49 -0400] NSMMReplicationPlugin - agmt=cn=meTom3.example.com (m3:389): replay_update: Sending modify operation (dn=uid=testuser,cn=users,cn=accounts,dc=example,dc=com csn=53d66ecb00020004) [05/Sep/2014:12:30:49 -0400] NSMMReplicationPlugin - agmt=cn=meTom3.example.com (m3:389): replay_update: modifys operation (dn=uid=testuser,cn=users,cn=accounts,dc=example,dc=com csn=53d66ecb00020004) not sent - empty [05/Sep/2014:12:30:49 -0400] NSMMReplicationPlugin - agmt=cn=meTom3.example.com (m3:389): replay_update: Consumer successfully sent operation with csn 53d66ecb00020004 [05/Sep/2014:12:30:49 -0400] NSMMReplicationPlugin - agmt=cn=meTom3.example.com (m3:389): Skipping update operation with no message_id (uniqueid 04b0b435-5ef311e3-9c91ec9f-6cd72e64, CSN 53d66ecb00020004): [05/Sep/2014:12:30:49 -0400] agmt=cn=meTom3.example.com (m3:389) - load=1 rec=39 csn=53d66ecc00010004 [05/Sep/2014:12:30:49 -0400] NSMMReplicationPlugin - agmt=cn=meTom3.example.com (m3:389): replay_update: Sending modify operation (dn=uid=testuser,cn=users,cn=accounts,dc=example,dc=com csn=53d66ecc00010004) [05/Sep/2014:12:30:49 -0400] NSMMReplicationPlugin - agmt=cn=meTom3.example.com (m3:389): replay_update: modifys operation (dn=uid=testuser,cn=users,cn=accounts,dc=example,dc=com csn=53d66ecc00010004) not sent - empty [05/Sep/2014:12:30:49 -0400] NSMMReplicationPlugin - agmt=cn=meTom3.example.com (m3:389): replay_update: Consumer successfully sent operation with csn 53d66ecc00010004 [05/Sep/2014:12:30:49 -0400] NSMMReplicationPlugin - agmt=cn=meTom3.example.com (m3:389): Skipping update operation with no message_id (uniqueid 04b0b435-5ef311e3-9c91ec9f-6cd72e64, CSN 53d66ecc00010004): [05/Sep/2014:12:30:49 -0400] NSMMReplicationPlugin - agmt=cn=meTom3.example.com (m3:389): No more updates to send (cl5GetNextOperationToReplay) [05/Sep/2014:12:30:49 -0400] - repl5_inc_waitfor_async_results: 0
Re: [Freeipa-users] Replication stopped working
Good to hear Guillermo, I am glad you are back up and running. I am just curious, what as the root cause of your replication errors in the end? I did not catch that from the thread. Is it something we can fix in FreeIPA or is it just a configuration error? Thanks, Martin On 09/05/2014 08:06 PM, Guillermo Fuentes wrote: Update: m2 and m3 are now in sync! After making sure ldapsearch was working both ways (m1=m2 and m1=m3) using the server's keytabs (/etc/dirsrv/ds.keytab) for getting the ticket, I re-initialize both replicas and they were able to get updated: @m2 # ipa-replica-manage re-initialize --from m1.example.com @m3 # ipa-replica-manage re-initialize --from m1.example.com Thanks so much for your hint Martin! -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go To http://freeipa.org for more info on the project
Re: [Freeipa-users] Replication stopped working
Hi Martin, That's a good question! We're not sure what was the root cause of the replication errors. When we realized the replication wasn't happening, we had recently updated FreeIPA from 3.0.0-36 to 3.0.0-37 (on CentOS 6.5) and we had shutdown m1 and m2 in order to do a snapshot of the VMs. We've been doing that for several months and never had a problem. Note that m3 wasn't shutdown and the replication stopped for it as well. The configuration wasn't change so I don't think it was a configuration problem. I did have to get a new ldap service keytab for the m2 replica (/etc/dirsrv/ds.keytab) but not for m3. I'll do more research on what happened and report back if I find anything relevant. Thanks again, Guillermo On Fri, Sep 5, 2014 at 4:22 PM, Martin Kosek mko...@redhat.com wrote: Good to hear Guillermo, I am glad you are back up and running. I am just curious, what as the root cause of your replication errors in the end? I did not catch that from the thread. Is it something we can fix in FreeIPA or is it just a configuration error? Thanks, Martin On 09/05/2014 08:06 PM, Guillermo Fuentes wrote: Update: m2 and m3 are now in sync! After making sure ldapsearch was working both ways (m1=m2 and m1=m3) using the server's keytabs (/etc/dirsrv/ds.keytab) for getting the ticket, I re-initialize both replicas and they were able to get updated: @m2 # ipa-replica-manage re-initialize --from m1.example.com @m3 # ipa-replica-manage re-initialize --from m1.example.com Thanks so much for your hint Martin! -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go To http://freeipa.org for more info on the project
[Freeipa-users] Replication stopped working
Hello list, We’re running FreeIPA with a master and 3 replicas. The replication stopped working and currently we’re adding resources only to the master. This is the environment we have: m1: OS: CentOS release 6.5 FreeIPA: 3.0.0-37 CA: pki-ca-9.0.3 # ipa-replica-manage list -v `hostname` m2.example.com: replica last init status: None last init ended: None last update status: 49 - LDAP error: Invalid credentials last update ended: None m3.example.com: replica last init status: None last init ended: None last update status: 0 Replica acquired successfully: Incremental update succeeded last update ended: 2014-09-04 14:28:44+00:00 m4.example.com: replica last init status: None last init ended: None last update status: -2 - LDAP error: Local error last update ended: None m2: OS: CentOS release 6.5 FreeIPA: 3.0.0-37 # ipa-replica-manage list -v `hostname` m1.example.com: replica last init status: None last init ended: None last update status: -1 Incremental update has failed and requires administrator actionLDAP error: Can't contact LDAP server last update ended: 2014-09-03 22:53:21+00:00 m3: OS: CentOS release 6.5 FreeIPA: 3.0.0-37 # ipa-replica-manage list -v `hostname` m1.example.com: replica last init status: None last init ended: None last update status: 0 Replica acquired successfully: Incremental update succeeded last update ended: 2014-09-04 14:31:51+00:00 m4: OS: CentOS release 6.5 FreeIPA: 3.3.3-28 # ipa-replica-manage list -v `hostname` m1.example.com: replica last init status: None last init ended: None last update status: 49 Unable to acquire replicaLDAP error: Invalid credentials last update ended: None Note that although m3 reports “Incremental update succeeded”, users created on m1 are not replicated to m3, and users created on m3 are not replicated back to m1. We’ve tried different things including re-initializing m2. Can somebody point me in the right direction to get replication going again? Thanks in advance! Guillermo -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go To http://freeipa.org for more info on the project
Re: [Freeipa-users] Replication stopped working
I should add that we already tried everything at https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/9.0/html/Administration_Guide/Managing_Replication-Solving_Common_Replication_Conflicts.html On Thu, Sep 4, 2014 at 11:11 AM, Guillermo Fuentes guillermo.fuen...@modernizingmedicine.com wrote: Hello list, We’re running FreeIPA with a master and 3 replicas. The replication stopped working and currently we’re adding resources only to the master. This is the environment we have: m1: OS: CentOS release 6.5 FreeIPA: 3.0.0-37 CA: pki-ca-9.0.3 # ipa-replica-manage list -v `hostname` m2.example.com: replica last init status: None last init ended: None last update status: 49 - LDAP error: Invalid credentials last update ended: None m3.example.com: replica last init status: None last init ended: None last update status: 0 Replica acquired successfully: Incremental update succeeded last update ended: 2014-09-04 14:28:44+00:00 m4.example.com: replica last init status: None last init ended: None last update status: -2 - LDAP error: Local error last update ended: None m2: OS: CentOS release 6.5 FreeIPA: 3.0.0-37 # ipa-replica-manage list -v `hostname` m1.example.com: replica last init status: None last init ended: None last update status: -1 Incremental update has failed and requires administrator actionLDAP error: Can't contact LDAP server last update ended: 2014-09-03 22:53:21+00:00 m3: OS: CentOS release 6.5 FreeIPA: 3.0.0-37 # ipa-replica-manage list -v `hostname` m1.example.com: replica last init status: None last init ended: None last update status: 0 Replica acquired successfully: Incremental update succeeded last update ended: 2014-09-04 14:31:51+00:00 m4: OS: CentOS release 6.5 FreeIPA: 3.3.3-28 # ipa-replica-manage list -v `hostname` m1.example.com: replica last init status: None last init ended: None last update status: 49 Unable to acquire replicaLDAP error: Invalid credentials last update ended: None Note that although m3 reports “Incremental update succeeded”, users created on m1 are not replicated to m3, and users created on m3 are not replicated back to m1. We’ve tried different things including re-initializing m2. Can somebody point me in the right direction to get replication going again? Thanks in advance! Guillermo -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go To http://freeipa.org for more info on the project -- Cheers, Fredy Sanchez IT Manager @ Modernizing Medicine 561-880-2998 x237 fredy.sanc...@modmed.com Need IT support? Visit https://mmit.zendesk.com - - -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go To http://freeipa.org for more info on the project
Re: [Freeipa-users] Replication stopped working
sudo ipa-replica-conncheck --replica for all replicas comes back with ... The following UDP ports could not be verified as open: 88, 464 This can happen if they are already bound to an application and ipa-replica-conncheck cannot attach own UDP responder. Connection from master to replica is OK. ipa-replica-manage -v list $REPLICA fails w/ Failed to get data from 'REPLICA': Invalid credentials SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context The common error is: nsds5replicaLastUpdateStatus: -2 - LDAP error: Local error On Thu, Sep 4, 2014 at 11:21 AM, Fredy Sanchez fredy.sanc...@modmed.com wrote: I should add that we already tried everything at https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/9.0/html/Administration_Guide/Managing_Replication-Solving_Common_Replication_Conflicts.html On Thu, Sep 4, 2014 at 11:11 AM, Guillermo Fuentes guillermo.fuen...@modernizingmedicine.com wrote: Hello list, We’re running FreeIPA with a master and 3 replicas. The replication stopped working and currently we’re adding resources only to the master. This is the environment we have: m1: OS: CentOS release 6.5 FreeIPA: 3.0.0-37 CA: pki-ca-9.0.3 # ipa-replica-manage list -v `hostname` m2.example.com: replica last init status: None last init ended: None last update status: 49 - LDAP error: Invalid credentials last update ended: None m3.example.com: replica last init status: None last init ended: None last update status: 0 Replica acquired successfully: Incremental update succeeded last update ended: 2014-09-04 14:28:44+00:00 m4.example.com: replica last init status: None last init ended: None last update status: -2 - LDAP error: Local error last update ended: None m2: OS: CentOS release 6.5 FreeIPA: 3.0.0-37 # ipa-replica-manage list -v `hostname` m1.example.com: replica last init status: None last init ended: None last update status: -1 Incremental update has failed and requires administrator actionLDAP error: Can't contact LDAP server last update ended: 2014-09-03 22:53:21+00:00 m3: OS: CentOS release 6.5 FreeIPA: 3.0.0-37 # ipa-replica-manage list -v `hostname` m1.example.com: replica last init status: None last init ended: None last update status: 0 Replica acquired successfully: Incremental update succeeded last update ended: 2014-09-04 14:31:51+00:00 m4: OS: CentOS release 6.5 FreeIPA: 3.3.3-28 # ipa-replica-manage list -v `hostname` m1.example.com: replica last init status: None last init ended: None last update status: 49 Unable to acquire replicaLDAP error: Invalid credentials last update ended: None Note that although m3 reports “Incremental update succeeded”, users created on m1 are not replicated to m3, and users created on m3 are not replicated back to m1. We’ve tried different things including re-initializing m2. Can somebody point me in the right direction to get replication going again? Thanks in advance! Guillermo -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go To http://freeipa.org for more info on the project -- Cheers, Fredy Sanchez IT Manager @ Modernizing Medicine 561-880-2998 x237 fredy.sanc...@modmed.com Need IT support? Visit https://mmit.zendesk.com - - -- Cheers, Fredy Sanchez IT Manager @ Modernizing Medicine 561-880-2998 x237 fredy.sanc...@modmed.com Need IT support? Visit https://mmit.zendesk.com - - -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go To http://freeipa.org for more info on the project
