Hi, On Mon, Jun 16, 2025 at 10:37 AM Kroon PC, Peter <p.c.kr...@pl.hanze.nl> wrote:
> Thanks Flo, > Outlook still hates inline replies, so I'll put it all on top: > > > [server1]# ldapsearch -Y GSSAPI -h server2.ipa.test -b "" -s base > This one fails > What is the error/output? Do you see a corresponding log on server2 in /var/log/dirsrv/slapd-XXX/access ? As the access log is buffered, it may not appear immediately but the lines should look like this: [16/Jun/2025:14:19:40.196675396 +0000] conn=4308 fd=91 slot=91 connection from 10.0.144.249 to 10.0.147.13 [16/Jun/2025:14:19:40.235073103 +0000] conn=4308 op=0 BIND dn="" method=sasl version=3 mech=GSSAPI [16/Jun/2025:14:19:40.238083749 +0000] conn=4308 op=0 RESULT err=14 tag=97 nentries=0 wtime=0.000239213 optime=0.003014757 etime=0.003253237, SASL bind in progress [16/Jun/2025:14:19:40.242142681 +0000] conn=4308 op=1 BIND dn="" method=sasl version=3 mech=GSSAPI [16/Jun/2025:14:19:40.243578790 +0000] conn=4308 op=1 RESULT err=14 tag=97 nentries=0 wtime=0.000043381 optime=0.001435699 etime=0.001478388, SASL bind in progress [16/Jun/2025:14:19:40.244527413 +0000] conn=4308 op=2 BIND dn="" method=sasl version=3 mech=GSSAPI [16/Jun/2025:14:19:40.245078046 +0000] conn=4308 op=2 RESULT err=0 tag=97 nentries=0 wtime=0.000030979 optime=0.000556235 etime=0.000586594 dn="krbprincipalname=ldap/server1.ipa.t...@ipa.test ,cn=services,cn=accounts,dc=ipa,dc=test" [16/Jun/2025:14:19:40.245470809 +0000] conn=4308 op=3 SRCH base="" scope=0 filter="(objectClass=*)" attrs=ALL [16/Jun/2025:14:19:40.246272148 +0000] conn=4308 op=3 RESULT err=0 tag=101 nentries=1 wtime=0.000101444 optime=0.000801741 etime=0.000902112 [16/Jun/2025:14:19:40.247713690 +0000] conn=4308 op=4 UNBIND If you don't see the first lines, it means the connection is not established, maybe a DNS issue or firewall issue. If you see the BIND lines with a corresponding err=49, it means the server is not able to associate the kerberos principal name with a LDAP entry. Can you check the content of this ldap entry on both server1 and server2 with this command: ipa service-show ldap/server1.ipa.t...@ipa.test --all --raw The value for krbprincipalname should be ldap/server1.ipa.t...@ipa.test flo > > > [server1]# klist -kte /etc/dirsrv/ds.keytab > > [server1]# kvno ldap/server1.ipa.t...@ipa.test > Both of these have kvno 1 > > > [server2]# kadmin.local getprinc ldap/server.ipa.t...@ipa.test > Interesing, this one also has knvo1! > > Screen output for the lot: > [server1]# klist -kte /etc/dirsrv/ds.keytab > Keytab name: FILE:/etc/dirsrv/ds.keytab > KVNO Timestamp Principal > ---- ----------------- > -------------------------------------------------------- > 1 04/30/24 11:01:59 ldap/server1.ipa.t...@ipa.test > (aes256-cts-hmac-sha1-96) > 1 04/30/24 11:01:59 ldap/server1.ipa.t...@ipa.test > (aes128-cts-hmac-sha1-96) > [server1]# kvno ldap/server1.ipa.t...@ipa.test > ldap/server1.ipa.t...@ipa.test: kvno = 1 > > [server2]# kadmin.local getprinc ldap/server1.ipa.t...@ipa.test > Principal: ldap/server1.ipa.t...@ipa.test > Expiration date: [never] > Last password change: Tue Apr 30 09:01:59 UTC 2024 > Password expiration date: [never] > Maximum ticket life: 1 day 00:00:00 > Maximum renewable life: 7 days 00:00:00 > Last modified: Tue Apr 30 09:01:59 UTC 2024 (ldap/server1.ipa.t...@ipa.test > ) > Last successful authentication: Fri May 30 12:46:46 UTC 2025 > Last failed authentication: [never] > Failed password attempts: 0 > Number of keys: 2 > Key: vno 1, aes256-cts-hmac-sha1-96:special > Key: vno 1, aes128-cts-hmac-sha1-96:special > MKey: vno 1 > Attributes: REQUIRES_PRE_AUTH > Policy: [none] > > Many thanks, > Peter > ________________________________________ > From: Florence Blanc-Renaud <f...@redhat.com> > Sent: Monday, 16 June 2025 10:01 > To: FreeIPA users list > Cc: Kroon PC, Peter > Subject: Re: [Freeipa-users] Re: Replication issues > > U ontvangt niet vaak e-mail van f...@redhat.com. Ontdek waarom dit > belangrijk is<https://aka.ms/LearnAboutSenderIdentification> > Hi, > > On Mon, Jun 16, 2025 at 9:10 AM Kroon PC, Peter via FreeIPA-users < > freeipa-users@lists.fedorahosted.org<mailto: > freeipa-users@lists.fedorahosted.org>> wrote: > Hi Mark, > > thanks for chipping in. > Does anyone else have any suggestions before I wipe and reinstall the > replica entirely? > > You need to check if the credentials used for the replication are working. > > On the first server: > [server1]# kinit -kt /etc/dirsrv/ds.keytab ldap/server1.ipa.test > [server1]# ldapsearch -Y GSSAPI -h server1.ipa.test -b "" -s base > [server1]# ldapsearch -Y GSSAPI -h server2.ipa.test -b "" -s base > > The first LDAP search authenticates to the local LDAP server with the > credentials from ds keytab, the 2nd LDAP search authenticates to the other > master. Both should succeed. > If one fails, you need to check the kvno: > [server1]# klist -kte /etc/dirsrv/ds.keytab > [server1]# kvno ldap/server1.ipa.t...@ipa.test > The klist command outputs the kvno stored in the ds.keytab file. It has to > match the kvno returned by the 2nd command (the kvno as seen by the > kerberos server). If they don't match, it means that the key has been > re-generated but there is an inconsistency between the one stored in the > ds.keytab file and the one stored by the kerberos server. > > Also check that you get the same value for the KVNO on the other master > [server2]# kadmin.local getprinc ldap/server.ipa.t...@ipa.test > The above command directly checks in the kerberos server. > > The repair steps depend on which key is the most recent one, and where it > is stored. Please provide the kvnos from all the commands and we'll try to > guide you. > flo > > Peter > > ________________________________________ > From: Mark Reynolds <marey...@redhat.com<mailto:marey...@redhat.com>> > Sent: Friday, 6 June 2025 17:32 > To: FreeIPA users list > Cc: Kroon PC, Peter > Subject: Re: [Freeipa-users] Re: Replication issues > > [You don't often get email from marey...@redhat.com<mailto: > marey...@redhat.com>. Learn why this is important at > https://aka.ms/LearnAboutSenderIdentification ] > > On 6/6/25 11:01 AM, Kroon PC, Peter via FreeIPA-users wrote: > > Hi Mark, > > > > thanks for the quick reply. > > Server-B has the following in the access log (twice, actually): > > ``` > > [06/Jun/2025:14:55:33.573567003 +0000] conn=8984 fd=372 slot=372 > connection from 192.168.12.58 to 192.168.13.128 > > [06/Jun/2025:14:55:33.613280772 +0000] conn=8984 op=0 BIND dn="" > method=sasl version=3 mech=GSSAPI > > [06/Jun/2025:14:55:33.614444924 +0000] conn=8984 op=0 RESULT err=49 > tag=97 nentries=0 wtime=0.000406209 optime=0.001176425 etime=0.001580230 - > SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context > > [06/Jun/2025:14:55:33.696712718 +0000] conn=8984 op=1 UNBIND > > [06/Jun/2025:14:55:33.696753716 +0000] conn=8984 op=1 fd=372 Disconnect > - Cleanly Closed Connection - U1 > > ``` > > I don't fully understand why it's trying GSSAPI after I entered the > directory manager password. > You authenticated as Directory Manager to the CLI, but it invokes the > replication agreement credentials to do the replication initialization > behind the scenes. > > I'd assume it'd use simple bind. > > In the security log there's the following, nothing about the failed > gssapi bind. > > ``` > > { "date": "[06\/Jun\/2025:14:55:00.776221689 +0000] ", "utc_time": > "1749221700.776221689", "event": "BIND_SUCCESS", "dn": "cn=directory > manager", "bind_method": "SIMPLE", "root_dn": true, "client_ip": > "192.168.12.58", "server_ip": "192.168.13.128", "ldap_version": 3, > "conn_id": 8983, "op_id": 0, "msg": "" } > > ``` > This is a different connection, but the other failed bind should have > been logged. Might have a bug somewhere :-( > > > > I'm about to leave the office for a long weekend, so I probably won't > respond again before Tuesday. > > So you "might" need to do an offline initialization to fix this, but I > suspect there is an official way the IPA team recommends to fix this > (sorry not as familiar with IPA as I am DS). So maybe someone from the > core IPA team and provide their recommendation. > > Mark > > > > > Peter > > > > ________________________________________ > > From: Mark Reynolds <marey...@redhat.com<mailto:marey...@redhat.com>> > > Sent: Friday, 6 June 2025 15:26 > > To: FreeIPA users list > > Cc: Kroon PC, Peter > > Subject: Re: [Freeipa-users] Replication issues > > > > [You don't often get email from marey...@redhat.com<mailto: > marey...@redhat.com>. Learn why this is important at > https://aka.ms/LearnAboutSenderIdentification ] > > > > Hi Peter, > > > > So the credentials the replication agreement is using are not valid (for > > whatever reason). Please check the directory server access log for > > "err=49" and you should see the bind DN and more details on why the > > authentication failed. > > > > HTH, > > > > Mark > > > > On 6/6/25 6:56 AM, Kroon PC, Peter via FreeIPA-users wrote: > >> Hello world, > >> > >> I have 3 IPA servers that are supposed to all replicate with each > other. For one server this stopped working. On all servers I have > ipa-server 4.12.2-14.el9_6 on Rocky Linux 9.6. > >> I'll call my servers A, B, and C. Server A cannot replicate with > neither server B nor C. B and C can replicate eachother without issues. My > first suspicion was a firewall, but I've already confirmed that that's not > the issue. > >> Here's the sanitized CLI outputs, all from server A: > >> ``` > >> $ ipa-healthcheck > >> { > >> "source": "ipahealthcheck.ds.replication", > >> "check": "ReplicationCheck", > >> "result": "CRITICAL", > >> "uuid": "9c96a61a-d917-44de-907a-d5c7a4df667e", > >> "when": "20250606095130Z", > >> "duration": "1.072988", > >> "kw": { > >> "key": "DSREPLLE0001", > >> "items": [ > >> "Replication", > >> "Agreement" > >> ], > >> "msg": "The replication agreement ( > server-A.example.com-to-server-C.example.com< > http://server-a.example.com-to-server-c.example.com/>) under > \"dc=example,dc=com\" is not in synchronization." > >> } > >> }, > >> { > >> "source": "ipahealthcheck.ds.replication", > >> "check": "ReplicationCheck", > >> "result": "CRITICAL", > >> "uuid": "6c29d38e-a216-414e-b47a-d9302f38da67", > >> "when": "20250606095131Z", > >> "duration": "1.073047", > >> "kw": { > >> "key": "DSREPLLE0001", > >> "items": [ > >> "Replication", > >> "Agreement" > >> ], > >> "msg": "The replication agreement (metoserver-B.example.com< > http://metoserver-b.example.com/>) under \"dc=example,dc=com\" is not in > synchronization." > >> } > >> }, > >> { > >> "source": "ipahealthcheck.ds.replication", > >> "check": "ReplicationCheck", > >> "result": "CRITICAL", > >> "uuid": "445f686b-27d9-47e0-aaa8-bb8d5f3416d4", > >> "when": "20250606095131Z", > >> "duration": "1.073054", > >> "kw": { > >> "key": "DSREPLLE0001", > >> "items": [ > >> "Replication", > >> "Agreement" > >> ], > >> "msg": "The replication agreement (catoserver-B.example.com< > http://catoserver-b.example.com/>) under \"o=ipaca\" is not in > synchronization." > >> } > >> }, > >> { > >> "source": "ipahealthcheck.ds.replication", > >> "check": "ReplicationCheck", > >> "result": "CRITICAL", > >> "uuid": "bb6fb4b1-54fc-45dc-bb33-c8c3562f5765", > >> "when": "20250606095131Z", > >> "duration": "1.073058", > >> "kw": { > >> "key": "DSREPLLE0001", > >> "items": [ > >> "Replication", > >> "Agreement" > >> ], > >> "msg": "The replication agreement ( > server-A.example.com-to-server-C.example.com< > http://server-a.example.com-to-server-c.example.com/>) under \"o=ipaca\" > is not in synchronization." > >> } > >> } > >> ``` > >> ``` > >> $ ipa-replica-manage list > >> ipa-replica-manage list > >> Directory Manager password: > >> > >> server-A.example.com<http://server-a.example.com/>: master > >> server-B.example.com<http://server-b.example.com/>: master > >> server-C.example.com<http://server-c.example.com/>: master > >> ``` > >> Ok, things are broken, let's try force-sync. > >> ``` > >> $ ipa-replica-manage force-sync --from server-B.example.com< > http://server-b.example.com/> --verbose --debug > >> <snip imports> > >> ipa: DEBUG: found 1 A records for server-A.example.com< > http://server-a.example.com/>.: 192.168.12.58 > >> ipa: DEBUG: The DNS response does not contain an answer to the > question: server-A.example.com<http://server-a.example.com/>. IN AAAA > >> Directory Manager password: > >> > >> ipa: DEBUG: Created connection context.ldap2_139632322538896 > >> ipa: DEBUG: found 1 A records for server-A.example.com< > http://server-a.example.com/>.: 192.168.12.58 > >> ipa: DEBUG: The DNS response does not contain an answer to the > question: server-A.example.com<http://server-a.example.com/>. IN AAAA > >> ipa: DEBUG: found 1 A records for server-B.example.com< > http://server-b.example.com/>.: 192.168.13.128 > >> ipa: DEBUG: The DNS response does not contain an answer to the > question: server-B.example.com<http://server-b.example.com/>. IN AAAA > >> ipa: DEBUG: retrieving schema for SchemaCache url=ldaps:// > server-A.example.com:636<http://server-a.example.com:636/> > conn=<ldap.ldapobject.SimpleLDAPObject object at 0x7efeaefd2460> > >> ipa: DEBUG: Loading StateFile from > '/var/lib/ipa/sysrestore/sysrestore.state' > >> ipa: DEBUG: Loading Index file from > '/var/lib/ipa/sysrestore/sysrestore.index' > >> ipa: DEBUG: retrieving schema for SchemaCache > url=ldapi://%2Frun%2Fslapd-BIN-BIOINF-NL.socket > conn=<ldap.ldapobject.SimpleLDAPObject object at 0x7efeaefd2610> > >> ipa: DEBUG: retrieving schema for SchemaCache url=ldaps:// > server-B.example.com:636<http://server-b.example.com:636/> > conn=<ldap.ldapobject.SimpleLDAPObject object at 0x7efeae1ee1f0> > >> ipa: INFO: Setting agreement cn=meToserver-A.example.com< > http://metoserver-a.example.com/>,cn=replica,cn=dc\=example\,dc\=com,cn=mapping > tree,cn=config schedule to 2358-2359 0 to force synch > >> ipa: INFO: Deleting schedule 2358-2359 0 from agreement cn= > meToserver-A.example.com<http://metoserver-a.example.com/>,cn=replica,cn=dc\=example\,dc\=com,cn=mapping > tree,cn=config > >> ipa: INFO: Replication Update in progress: False: status: Error (49) > Problem connecting to replica - LDAP error: Invalid credentials (connection > error): start: 19700101000000: end: 19700101000000 > >> ipa: DEBUG: Destroyed connection context.ldap2_139632322538896 > >> ``` > >> Maybe I can re-initialize? > >> ``` > >> $ ipa-replica-manage re-initialize --from server-B.example.com< > http://server-b.example.com/> --verbose --debug > >> <snip import> > >> <snip DNS responses, same as above> > >> Directory Manager password: > >> ipa: DEBUG: Created connection context.ldap2_140512488991280 > >> ipa: DEBUG: retrieving schema for SchemaCache url=ldaps:// > server-A.example.com:636<http://server-a.example.com:636/> > conn=<ldap.ldapobject.SimpleLDAPObject object at 0x7fcb9c310dc0> > >> ipa: DEBUG: retrieving schema for SchemaCache url=ldaps:// > server-B.example.com:636<http://server-b.example.com:636/> > conn=<ldap.ldapobject.SimpleLDAPObject object at 0x7fcb9c1e10a0> > >> ipa: INFO: Setting agreement cn=meToserver-A.example.com< > http://metoserver-a.example.com/>,cn=replica,cn=dc\=example\,dc\=com,cn=mapping > tree,cn=config schedule to 2358-2359 0 to force synch > >> ipa: INFO: Deleting schedule 2358-2359 0 from agreement cn= > meToserver-A.example.com<http://metoserver-a.example.com/>,cn=replica,cn=dc\=example\,dc\=com,cn=mapping > tree,cn=config > >> Update in progress, 15 seconds elapsed > >> [ldaps://server-B.example.com:636<http://server-b.example.com:636/>] > reports: Update failed! Status: [Error (49) - LDAP error: Invalid > credentials - no response received] > >> ipa: DEBUG: Destroyed connection context.ldap2_140512488991280 > >> ``` > >> Is it a firewall or password issue? No: > >> ``` > >> $ ldapwhoami -H ldaps://server-B.example.com< > http://server-b.example.com/> -D 'cn=directory manager' -W > >> Enter LDAP Password: > >> dn: cn=directory manager > >> ``` > >> From server-B: > >> ``` > >> server-B $ ldapwhoami -H ldaps://server-A.example.com< > http://server-a.example.com/> -D 'cn=directory manager' -W > >> Enter LDAP Password: > >> dn: cn=directory manager > >> ``` > >> I also checked that `ipa-replica-manage list-ruv` produces the same > output for all 3 servers. > >> Does anyone have any more ideas I could explore? > >> > >> Many thanks in advance! > >> Peter > > -- > > Identity Management Development Team > > > -- > Identity Management Development Team > > -- > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org<mailto: > freeipa-users@lists.fedorahosted.org> > To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org > <mailto:freeipa-users-le...@lists.fedorahosted.org> > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/< > https://docs.fedoraproject.org/en-US/project/code-of-conduct/> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines< > https://fedoraproject.org/wiki/Mailing_list_guidelines> > List Archives: > https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org > < > https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org > > > Do not reply to spam, report it: > https://pagure.io/fedora-infrastructure/new_issue< > https://pagure.io/fedora-infrastructure/new_issue> >
-- _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
