Thanks Flo, Outlook still hates inline replies, so I'll put it all on top: > [server1]# ldapsearch -Y GSSAPI -h server2.ipa.test -b "" -s base This one fails
> [server1]# klist -kte /etc/dirsrv/ds.keytab > [server1]# kvno ldap/server1.ipa.t...@ipa.test Both of these have kvno 1 > [server2]# kadmin.local getprinc ldap/server.ipa.t...@ipa.test Interesing, this one also has knvo1! Screen output for the lot: [server1]# klist -kte /etc/dirsrv/ds.keytab Keytab name: FILE:/etc/dirsrv/ds.keytab KVNO Timestamp Principal ---- ----------------- -------------------------------------------------------- 1 04/30/24 11:01:59 ldap/server1.ipa.t...@ipa.test (aes256-cts-hmac-sha1-96) 1 04/30/24 11:01:59 ldap/server1.ipa.t...@ipa.test (aes128-cts-hmac-sha1-96) [server1]# kvno ldap/server1.ipa.t...@ipa.test ldap/server1.ipa.t...@ipa.test: kvno = 1 [server2]# kadmin.local getprinc ldap/server1.ipa.t...@ipa.test Principal: ldap/server1.ipa.t...@ipa.test Expiration date: [never] Last password change: Tue Apr 30 09:01:59 UTC 2024 Password expiration date: [never] Maximum ticket life: 1 day 00:00:00 Maximum renewable life: 7 days 00:00:00 Last modified: Tue Apr 30 09:01:59 UTC 2024 (ldap/server1.ipa.t...@ipa.test) Last successful authentication: Fri May 30 12:46:46 UTC 2025 Last failed authentication: [never] Failed password attempts: 0 Number of keys: 2 Key: vno 1, aes256-cts-hmac-sha1-96:special Key: vno 1, aes128-cts-hmac-sha1-96:special MKey: vno 1 Attributes: REQUIRES_PRE_AUTH Policy: [none] Many thanks, Peter ________________________________________ From: Florence Blanc-Renaud <f...@redhat.com> Sent: Monday, 16 June 2025 10:01 To: FreeIPA users list Cc: Kroon PC, Peter Subject: Re: [Freeipa-users] Re: Replication issues U ontvangt niet vaak e-mail van f...@redhat.com. Ontdek waarom dit belangrijk is<https://aka.ms/LearnAboutSenderIdentification> Hi, On Mon, Jun 16, 2025 at 9:10 AM Kroon PC, Peter via FreeIPA-users <freeipa-users@lists.fedorahosted.org<mailto:freeipa-users@lists.fedorahosted.org>> wrote: Hi Mark, thanks for chipping in. Does anyone else have any suggestions before I wipe and reinstall the replica entirely? You need to check if the credentials used for the replication are working. On the first server: [server1]# kinit -kt /etc/dirsrv/ds.keytab ldap/server1.ipa.test [server1]# ldapsearch -Y GSSAPI -h server1.ipa.test -b "" -s base [server1]# ldapsearch -Y GSSAPI -h server2.ipa.test -b "" -s base The first LDAP search authenticates to the local LDAP server with the credentials from ds keytab, the 2nd LDAP search authenticates to the other master. Both should succeed. If one fails, you need to check the kvno: [server1]# klist -kte /etc/dirsrv/ds.keytab [server1]# kvno ldap/server1.ipa.t...@ipa.test The klist command outputs the kvno stored in the ds.keytab file. It has to match the kvno returned by the 2nd command (the kvno as seen by the kerberos server). If they don't match, it means that the key has been re-generated but there is an inconsistency between the one stored in the ds.keytab file and the one stored by the kerberos server. Also check that you get the same value for the KVNO on the other master [server2]# kadmin.local getprinc ldap/server.ipa.t...@ipa.test The above command directly checks in the kerberos server. The repair steps depend on which key is the most recent one, and where it is stored. Please provide the kvnos from all the commands and we'll try to guide you. flo Peter ________________________________________ From: Mark Reynolds <marey...@redhat.com<mailto:marey...@redhat.com>> Sent: Friday, 6 June 2025 17:32 To: FreeIPA users list Cc: Kroon PC, Peter Subject: Re: [Freeipa-users] Re: Replication issues [You don't often get email from marey...@redhat.com<mailto:marey...@redhat.com>. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] On 6/6/25 11:01 AM, Kroon PC, Peter via FreeIPA-users wrote: > Hi Mark, > > thanks for the quick reply. > Server-B has the following in the access log (twice, actually): > ``` > [06/Jun/2025:14:55:33.573567003 +0000] conn=8984 fd=372 slot=372 connection > from 192.168.12.58 to 192.168.13.128 > [06/Jun/2025:14:55:33.613280772 +0000] conn=8984 op=0 BIND dn="" method=sasl > version=3 mech=GSSAPI > [06/Jun/2025:14:55:33.614444924 +0000] conn=8984 op=0 RESULT err=49 tag=97 > nentries=0 wtime=0.000406209 optime=0.001176425 etime=0.001580230 - > SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context > [06/Jun/2025:14:55:33.696712718 +0000] conn=8984 op=1 UNBIND > [06/Jun/2025:14:55:33.696753716 +0000] conn=8984 op=1 fd=372 Disconnect - > Cleanly Closed Connection - U1 > ``` > I don't fully understand why it's trying GSSAPI after I entered the directory > manager password. You authenticated as Directory Manager to the CLI, but it invokes the replication agreement credentials to do the replication initialization behind the scenes. > I'd assume it'd use simple bind. > In the security log there's the following, nothing about the failed gssapi > bind. > ``` > { "date": "[06\/Jun\/2025:14:55:00.776221689 +0000] ", "utc_time": > "1749221700.776221689", "event": "BIND_SUCCESS", "dn": "cn=directory > manager", "bind_method": "SIMPLE", "root_dn": true, "client_ip": > "192.168.12.58", "server_ip": "192.168.13.128", "ldap_version": 3, "conn_id": > 8983, "op_id": 0, "msg": "" } > ``` This is a different connection, but the other failed bind should have been logged. Might have a bug somewhere :-( > > I'm about to leave the office for a long weekend, so I probably won't respond > again before Tuesday. So you "might" need to do an offline initialization to fix this, but I suspect there is an official way the IPA team recommends to fix this (sorry not as familiar with IPA as I am DS). So maybe someone from the core IPA team and provide their recommendation. Mark > > Peter > > ________________________________________ > From: Mark Reynolds <marey...@redhat.com<mailto:marey...@redhat.com>> > Sent: Friday, 6 June 2025 15:26 > To: FreeIPA users list > Cc: Kroon PC, Peter > Subject: Re: [Freeipa-users] Replication issues > > [You don't often get email from > marey...@redhat.com<mailto:marey...@redhat.com>. Learn why this is important > at https://aka.ms/LearnAboutSenderIdentification ] > > Hi Peter, > > So the credentials the replication agreement is using are not valid (for > whatever reason). Please check the directory server access log for > "err=49" and you should see the bind DN and more details on why the > authentication failed. > > HTH, > > Mark > > On 6/6/25 6:56 AM, Kroon PC, Peter via FreeIPA-users wrote: >> Hello world, >> >> I have 3 IPA servers that are supposed to all replicate with each other. For >> one server this stopped working. On all servers I have ipa-server >> 4.12.2-14.el9_6 on Rocky Linux 9.6. >> I'll call my servers A, B, and C. Server A cannot replicate with neither >> server B nor C. B and C can replicate eachother without issues. My first >> suspicion was a firewall, but I've already confirmed that that's not the >> issue. >> Here's the sanitized CLI outputs, all from server A: >> ``` >> $ ipa-healthcheck >> { >> "source": "ipahealthcheck.ds.replication", >> "check": "ReplicationCheck", >> "result": "CRITICAL", >> "uuid": "9c96a61a-d917-44de-907a-d5c7a4df667e", >> "when": "20250606095130Z", >> "duration": "1.072988", >> "kw": { >> "key": "DSREPLLE0001", >> "items": [ >> "Replication", >> "Agreement" >> ], >> "msg": "The replication agreement >> (server-A.example.com-to-server-C.example.com<http://server-a.example.com-to-server-c.example.com/>) >> under \"dc=example,dc=com\" is not in synchronization." >> } >> }, >> { >> "source": "ipahealthcheck.ds.replication", >> "check": "ReplicationCheck", >> "result": "CRITICAL", >> "uuid": "6c29d38e-a216-414e-b47a-d9302f38da67", >> "when": "20250606095131Z", >> "duration": "1.073047", >> "kw": { >> "key": "DSREPLLE0001", >> "items": [ >> "Replication", >> "Agreement" >> ], >> "msg": "The replication agreement >> (metoserver-B.example.com<http://metoserver-b.example.com/>) under >> \"dc=example,dc=com\" is not in synchronization." >> } >> }, >> { >> "source": "ipahealthcheck.ds.replication", >> "check": "ReplicationCheck", >> "result": "CRITICAL", >> "uuid": "445f686b-27d9-47e0-aaa8-bb8d5f3416d4", >> "when": "20250606095131Z", >> "duration": "1.073054", >> "kw": { >> "key": "DSREPLLE0001", >> "items": [ >> "Replication", >> "Agreement" >> ], >> "msg": "The replication agreement >> (catoserver-B.example.com<http://catoserver-b.example.com/>) under >> \"o=ipaca\" is not in synchronization." >> } >> }, >> { >> "source": "ipahealthcheck.ds.replication", >> "check": "ReplicationCheck", >> "result": "CRITICAL", >> "uuid": "bb6fb4b1-54fc-45dc-bb33-c8c3562f5765", >> "when": "20250606095131Z", >> "duration": "1.073058", >> "kw": { >> "key": "DSREPLLE0001", >> "items": [ >> "Replication", >> "Agreement" >> ], >> "msg": "The replication agreement >> (server-A.example.com-to-server-C.example.com<http://server-a.example.com-to-server-c.example.com/>) >> under \"o=ipaca\" is not in synchronization." >> } >> } >> ``` >> ``` >> $ ipa-replica-manage list >> ipa-replica-manage list >> Directory Manager password: >> >> server-A.example.com<http://server-a.example.com/>: master >> server-B.example.com<http://server-b.example.com/>: master >> server-C.example.com<http://server-c.example.com/>: master >> ``` >> Ok, things are broken, let's try force-sync. >> ``` >> $ ipa-replica-manage force-sync --from >> server-B.example.com<http://server-b.example.com/> --verbose --debug >> <snip imports> >> ipa: DEBUG: found 1 A records for >> server-A.example.com<http://server-a.example.com/>.: 192.168.12.58 >> ipa: DEBUG: The DNS response does not contain an answer to the question: >> server-A.example.com<http://server-a.example.com/>. IN AAAA >> Directory Manager password: >> >> ipa: DEBUG: Created connection context.ldap2_139632322538896 >> ipa: DEBUG: found 1 A records for >> server-A.example.com<http://server-a.example.com/>.: 192.168.12.58 >> ipa: DEBUG: The DNS response does not contain an answer to the question: >> server-A.example.com<http://server-a.example.com/>. IN AAAA >> ipa: DEBUG: found 1 A records for >> server-B.example.com<http://server-b.example.com/>.: 192.168.13.128 >> ipa: DEBUG: The DNS response does not contain an answer to the question: >> server-B.example.com<http://server-b.example.com/>. IN AAAA >> ipa: DEBUG: retrieving schema for SchemaCache >> url=ldaps://server-A.example.com:636<http://server-a.example.com:636/> >> conn=<ldap.ldapobject.SimpleLDAPObject object at 0x7efeaefd2460> >> ipa: DEBUG: Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' >> ipa: DEBUG: Loading Index file from >> '/var/lib/ipa/sysrestore/sysrestore.index' >> ipa: DEBUG: retrieving schema for SchemaCache >> url=ldapi://%2Frun%2Fslapd-BIN-BIOINF-NL.socket >> conn=<ldap.ldapobject.SimpleLDAPObject object at 0x7efeaefd2610> >> ipa: DEBUG: retrieving schema for SchemaCache >> url=ldaps://server-B.example.com:636<http://server-b.example.com:636/> >> conn=<ldap.ldapobject.SimpleLDAPObject object at 0x7efeae1ee1f0> >> ipa: INFO: Setting agreement >> cn=meToserver-A.example.com<http://metoserver-a.example.com/>,cn=replica,cn=dc\=example\,dc\=com,cn=mapping >> tree,cn=config schedule to 2358-2359 0 to force synch >> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >> cn=meToserver-A.example.com<http://metoserver-a.example.com/>,cn=replica,cn=dc\=example\,dc\=com,cn=mapping >> tree,cn=config >> ipa: INFO: Replication Update in progress: False: status: Error (49) Problem >> connecting to replica - LDAP error: Invalid credentials (connection error): >> start: 19700101000000: end: 19700101000000 >> ipa: DEBUG: Destroyed connection context.ldap2_139632322538896 >> ``` >> Maybe I can re-initialize? >> ``` >> $ ipa-replica-manage re-initialize --from >> server-B.example.com<http://server-b.example.com/> --verbose --debug >> <snip import> >> <snip DNS responses, same as above> >> Directory Manager password: >> ipa: DEBUG: Created connection context.ldap2_140512488991280 >> ipa: DEBUG: retrieving schema for SchemaCache >> url=ldaps://server-A.example.com:636<http://server-a.example.com:636/> >> conn=<ldap.ldapobject.SimpleLDAPObject object at 0x7fcb9c310dc0> >> ipa: DEBUG: retrieving schema for SchemaCache >> url=ldaps://server-B.example.com:636<http://server-b.example.com:636/> >> conn=<ldap.ldapobject.SimpleLDAPObject object at 0x7fcb9c1e10a0> >> ipa: INFO: Setting agreement >> cn=meToserver-A.example.com<http://metoserver-a.example.com/>,cn=replica,cn=dc\=example\,dc\=com,cn=mapping >> tree,cn=config schedule to 2358-2359 0 to force synch >> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >> cn=meToserver-A.example.com<http://metoserver-a.example.com/>,cn=replica,cn=dc\=example\,dc\=com,cn=mapping >> tree,cn=config >> Update in progress, 15 seconds elapsed >> [ldaps://server-B.example.com:636<http://server-b.example.com:636/>] >> reports: Update failed! Status: [Error (49) - LDAP error: Invalid >> credentials - no response received] >> ipa: DEBUG: Destroyed connection context.ldap2_140512488991280 >> ``` >> Is it a firewall or password issue? No: >> ``` >> $ ldapwhoami -H ldaps://server-B.example.com<http://server-b.example.com/> >> -D 'cn=directory manager' -W >> Enter LDAP Password: >> dn: cn=directory manager >> ``` >> From server-B: >> ``` >> server-B $ ldapwhoami -H >> ldaps://server-A.example.com<http://server-a.example.com/> -D 'cn=directory >> manager' -W >> Enter LDAP Password: >> dn: cn=directory manager >> ``` >> I also checked that `ipa-replica-manage list-ruv` produces the same output >> for all 3 servers. >> Does anyone have any more ideas I could explore? >> >> Many thanks in advance! >> Peter > -- > Identity Management Development Team > -- Identity Management Development Team -- _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org<mailto:freeipa-users@lists.fedorahosted.org> To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org<mailto:freeipa-users-le...@lists.fedorahosted.org> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/<https://docs.fedoraproject.org/en-US/project/code-of-conduct/> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines<https://fedoraproject.org/wiki/Mailing_list_guidelines> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org<https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue<https://pagure.io/fedora-infrastructure/new_issue> -- _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
