So, some good news. I tried this again today and it works, i turned on the httpd debugging as suggested on the master and it worked. O_O
The only thing i have changed outside of the work we were doing here was that i noticed we had a lot of ghost PKI servers yesterday in `pki securitydomain-show` So i followed https://access.redhat.com/solutions/6615411 and removed them. If this fixed it i don't know, how it could affect it again i dont know, but it was the only other change we made. I shall test again on Tuesday to see if we are still ok. thanks for your continued help on this. have a great weekend. Nick. On Fri, 26 May 2023 at 09:35, Florence Blanc-Renaud <[email protected]> wrote: > Hi, > > in your logs we see the following: > > 2023-05-25T07:52:07Z DEBUG trying https://ipa008.ad.companyx.fm/ipa/json > 2023-05-25T07:52:07Z > <https://ipa008.ad.companyx.fm/ipa/json%0D2023-05-25T07:52:07Z> DEBUG New > HTTP connection (ipa008.ad.companyx.fm) > 2023-05-25T07:52:07Z DEBUG received Set-Cookie (<class > 'list'>)'['ipa_session=MagBearerToken=YcNKsa8%2bZ%2fVQ8AqCubb5p7Jupgb6nvya2wgS8lYw%2fZKoGctAmsRoh%2boJM7vQNfInsBxx%2fEuf%2fqVlfnKYfWCwwcbU0VH66cM8MqnJhEfKLAgFnNIyG91v%2bmDrsrdCijEWBX8p2PJmqVLIfmvYl2L2ke9FeVPEKnPrq18w1DmZSDDSwV5IOp5JZ71EgTjPoz39Y7mR0LCZ5cXbJ2T93%2b0ZjbcdEsoatFdzkjS383%2f2pLhpRjoTMXCmht%2fXYHjMy47O;path=/ipa;httponly;secure;']' > 2023-05-25T07:52:07Z DEBUG storing cookie > 'ipa_session=MagBearerToken=YcNKsa8%2bZ%2fVQ8AqCubb5p7Jupgb6nvya2wgS8lYw%2fZKoGctAmsRoh%2boJM7vQNfInsBxx%2fEuf%2fqVlfnKYfWCwwcbU0VH66cM8MqnJhEfKLAgFnNIyG91v%2bmDrsrdCijEWBX8p2PJmqVLIfmvYl2L2ke9FeVPEKnPrq18w1DmZSDDSwV5IOp5JZ71EgTjPoz39Y7mR0LCZ5cXbJ2T93%2b0ZjbcdEsoatFdzkjS383%2f2pLhpRjoTMXCmht%2fXYHjMy47O;' > for principal host/[email protected] > 2023-05-25T07:52:07Z INFO Connection to > https://ipa008.ad.companyx.fm/ipa/json failed with Insufficient access: > SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the > credentials were unavailable or inaccessible (Credential cache is empty) > 2023-05-25T07:52:07Z DEBUG trying https://ipa011.ad.companyx.fm/ipa/json > > > This shows that in a first pass, the replica tries to connect to the > master ipa008 in order to do RPC calls. This corresponds to this piece of > code > https://github.com/freeipa/freeipa/blob/master/install/tools/ipa-replica-conncheck.in#L560-L567 > where the api.Backend.rpcclient.connect() method internally tries to reach > one of the IPA servers (for the details, see > https://github.com/freeipa/freeipa/blob/master/ipalib/rpc.py#L1002, the > available servers are tried sequentially in the *for url in urls:* loop). > In your case, it fails on ipa008 and continues with ipa011 (the replica). > The call to the api server_conncheck is then executed on the local replica > and this explains the error: we don't want to check connection from local > to local, we want to check from ipa008 to ipa011. > > The piece of code that needs investigation is why does the connection to > ipa008 fail with GSSAPI error. The debug must be set on the existing > master, in order to see why the authentication fails there. > > HTH, > flo > > On Thu, May 25, 2023 at 3:56 PM Nicholas Cross <[email protected]> > wrote: > >> Hi Rob, >> >> Thanks for the tip, it came in handy with what i was working on (the >> python code). >> >> Here is the error_log on the new server (ipa011) >> >> [Thu May 25 13:45:38.197358 2023] [mpm_event:notice] [pid 55597:tid >> 55597] AH00492: caught SIGWINCH, shutting down gracefully >> [Thu May 25 13:45:40.008302 2023] [suexec:notice] [pid 55907:tid 55907] >> AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) >> [Thu May 25 13:45:40.087736 2023] [lbmethod_heartbeat:notice] [pid >> 55907:tid 55907] AH02282: No slotmem from mod_heartmonitor >> [Thu May 25 13:45:40.094676 2023] [mpm_event:notice] [pid 55907:tid >> 55907] AH00489: Apache/2.4.53 (AlmaLinux) OpenSSL/3.0.7 >> mod_auth_gssapi/1.6.3 mod_wsgi/4.7.1 Python/3.9 configured -- resuming >> normal operations >> [Thu May 25 13:45:40.094699 2023] [core:notice] [pid 55907:tid 55907] >> AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' >> [Thu May 25 13:45:47.384449 2023] [wsgi:error] [pid 55912:tid 56171] >> [remote 10.32.225.7:37118] ipa: INFO: [jsonserver_kerb] >> [email protected]: ping(): SUCCESS >> [Thu May 25 13:45:47.414961 2023] [wsgi:error] [pid 55914:tid 56174] >> [remote 10.32.225.7:37118] ipa: INFO: [jsonserver_kerb] >> [email protected]: ping/1(version='2.251'): SUCCESS >> [Thu May 25 13:45:47.445318 2023] [wsgi:error] [pid 55913:tid 56177] >> [remote 10.32.225.7:37118] ipa: INFO: [jsonserver_kerb] >> [email protected]: server_conncheck('ipa008.ad.companyx.fm', >> 'ipa011.ad.companyx.fm', version='2.162'): ValidationError >> >> I tweaked the code a little to do some old skool debugging... >> here: >> https://github.com/freeipa/freeipa/blob/master/ipaserver/plugins/server.py#L927 >> >> This enabled me to dump the vars in question to why the error was raised: >> >> ERROR: Remote master check failed with following error message(s): >> invalid 'cn': xx must be "ipa011.ad.companyx.fm" keys:(' >> ipa008.ad.companyx.fm', 'ipa011.ad.companyx.fm') keys[-2]: >> ipa008.ad.companyx.fm api.env.host:ipa011.ad.companyx.fm >> >> So: >> keys:('ipa008.ad.companyx.fm', 'ipa011.ad.companyx.fm') >> keys[-2]:ipa008.ad.companyx.fm >> api.env.host:ipa011.ad.companyx.fm >> >> As the code states: >> >> if keys[-2] != api.env.host: >> raise errors.ValidationError( >> name='cn', error=_("must be \"%s\"") % api.env.host) >> >> and we can see that in fact ipa008.ad.companyx.fm != >> ipa011.ad.companyx.fm >> >> So, that's about as far as i have gotten so far. >> >> Do we think the keys swapped around for some reason? >> >> thanks, >> Nick >> >> On Thu, 25 May 2023 at 13:52, Rob Crittenden <[email protected]> wrote: >> >>> Florence Blanc-Renaud via FreeIPA-users wrote: >>> > Hi, >>> > >>> > On Thu, May 25, 2023 at 9:53 AM Nicholas Cross <[email protected] >>> > <mailto:[email protected]>> wrote: >>> > >>> > Hi, thanks for the reply. >>> > >>> > This was the process. >>> > 1. prep >>> > 2. test connectivity - works ok >>> > 3. install replica (successfully) - no DNS no CA at this point >>> > 4. test connectivity - fails. >>> > >>> > If i replace step 4 for the following: >>> > >>> > 4. install DNS (successfully) >>> > 5. install CA (fails - with the "invalid 'cn': must be" error) >>> > >>> > If we can ignore the step 4 where the comms check fails and >>> > concentrate on step 5 where the CA fails. I am happy to do so. >>> > But in step 5. it is the comms check that the `ipa-ca-install` >>> > command runs that also fails as far as i can see (no expert here >>> > myself, though my company thinks so LOL) >>> > >>> > Successful DNS install logs http://sprunge.us/7WCtpW >>> > >>> > Failed CA install logs http://sprunge.us/hTVLzL >>> > >>> > I believe this error needs to be investigated: >>> > >>> > 2023-05-25T07:52:07Z INFO Connection to >>> https://ipa008.ad.companyx.fm/ipa/json failed with Insufficient access: >>> SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or >>> the credentials were unavailable or inaccessible (Credential cache is empty) >>> > >>> > We can see that the process is able to connect to the master and get a >>> > cookie but later on GSSAPI auth fails. >>> > >>> > Are there any corresponding logs in the master in >>> > /var/log/httpd/error_log? You can enable debug logs on the master with >>> > the following procedure: >>> > - create /etc/ipa/server.conf: >>> > [global] >>> > debug=True >>> > >>> > then restart apache with systemctl restart httpd. >>> > You can also run ipa-healthcheck on the master/the replica to find any >>> > config issue. >>> >>> It would also be interesting to see the entries around the >>> 'server_conncheck' call in /var/log/httpd/error_log to see what >>> arguments were passed in (on ipa011) since you're enabling debug mode >>> anyway. >>> >>> rob >>> >>> > >>> > flo >>> > >>> > [root@ipa011 ~]# ipa-ca-install >>> > Directory Manager (existing master) password: >>> > >>> > Run connection check to master >>> > >>> > Your system may be partly configured. >>> > Run /usr/sbin/ipa-server-install --uninstall to clean up. >>> > >>> > Connection check failed! >>> > See /var/log/ipareplica-conncheck.log for more information. >>> > If the check results are not valid it can be skipped with >>> > --skip-conncheck parameter. >>> > >>> > >>> > >>> > thanks, >>> > Nick >>> > >>> > On Thu, 25 May 2023 at 08:18, Florence Blanc-Renaud < >>> [email protected] >>> > <mailto:[email protected]>> wrote: >>> > >>> > Hi, >>> > >>> > On Wed, May 24, 2023 at 3:29 PM Nicholas Cross >>> > <[email protected] <mailto:[email protected]>> >>> wrote: >>> > >>> > Hi Flo (and other helpful people on this list), >>> > >>> > After fixing the SID/PAC issue, i am back to looking as to >>> > why the ipa-replica-conncheck fails when installing the CA >>> > to a (working) replica. >>> > >>> > I ran your suggested commands and they look fine now. >>> > >>> > [root@ipa011 ~]# kdestroy >>> > kdestroy: No credentials cache found while destroying cache >>> > >>> > [root@ipa011 ~]# kinit -V -kt /etc/krb5.keytab >>> > host/[email protected] >>> > <mailto:[email protected]> >>> > Using default cache: 0 >>> > Using principal: host/[email protected] >>> > <mailto:[email protected]> >>> > Using keytab: /etc/krb5.keytab >>> > Authenticated to Kerberos v5 >>> > >>> > [root@ipa011 ~]# kvno >>> > HTTP/[email protected] >>> > <mailto:[email protected]> >>> > HTTP/[email protected] >>> > <mailto:[email protected]>: kvno = 1 >>> > >>> > I can run the following BEFORE i install the replica and it >>> > works 100%, it is part of our pre-reqs in our docs to make >>> > sure comms work. >>> > >>> > ipa-replica-conncheck --master=ipa008.ad.companyx.fm >>> > <http://ipa008.ad.companyx.fm> --auto-master-check >>> > --realm=AD.companyx.FM <http://AD.companyx.FM> >>> > --principal=nicholas.cross >>> > >>> > After i install the replica with ipa-replica-install. (no >>> > DNS, no CA) then running the above ipa-replica-conncheck >>> > command fails and we see the errors as described. >>> > >>> > >>> > ipa-replica-conncheck is intended to be run *before* replica >>> > installation, as stated in the man page: >>> > ipa-replica-conncheck - Check a replica-master network >>> > connection before installation >>> > >>> > The reason is that it opens server sockets listening on the >>> > tested ports, for the master to try to reach the replica. If >>> the >>> > replica is already installed, the ports are already in use by >>> > the ldap server, the apache web server etc... >>> > >>> > I understand that you tried to run ipa-replica-conncheck to >>> > ensure that ipa-dns-install or ipa-ca-install won't hit any >>> > issue. Did you actually try to install the DNS service and the >>> > CA service on the replica? If you see errors in those steps we >>> > can help troubleshoot, based on the produced logs, but >>> > ipa-replica-conncheck won't be of any use here. >>> > >>> > flo >>> > >>> > >>> > Some logs: >>> > 1. before ipa-replica-install - conn check works. >>> > http://sprunge.us/4yrInD >>> > 2. after successful ipa-replica-install - but conn check >>> fails >>> > http://sprunge.us/gvqEle >>> > >>> > new replica versions >>> > [root@ipa011 ~]# rpm -qa | grep ipa | sort >>> > almalinux-logos-ipa-90.5.1-1.1.el9.noarch >>> > ipa-client-4.10.1-6.el9.x86_64 >>> > ipa-client-common-4.10.1-6.el9.noarch >>> > ipa-common-4.10.1-6.el9.noarch >>> > ipa-healthcheck-0.12-1.el9.noarch >>> > ipa-healthcheck-core-0.12-1.el9.noarch >>> > ipa-selinux-4.9.8-7.el9_0.noarch >>> > ipa-server-4.10.1-6.el9.x86_64 >>> > ipa-server-common-4.10.1-6.el9.noarch >>> > ipa-server-dns-4.10.1-6.el9.noarch >>> > libipa_hbac-2.8.2-2.el9.x86_64 >>> > python3-ipaclient-4.10.1-6.el9.noarch >>> > python3-ipalib-4.10.1-6.el9.noarch >>> > python3-ipaserver-4.10.1-6.el9.noarch >>> > python3-libipa_hbac-2.8.2-2.el9.x86_64 >>> > sssd-ipa-2.8.2-2.el9.x86_64 >>> > >>> > current master versions >>> > [nicholas.cross@ipa008 ~]$ rpm -qa | grep ipa | sort >>> > almalinux-logos-ipa-90.5.1-1.1.el9.noarch >>> > ipa-client-4.10.0-8.el9_1.x86_64 >>> > ipa-client-common-4.10.0-8.el9_1.noarch >>> > ipa-common-4.10.0-8.el9_1.noarch >>> > ipa-healthcheck-0.9-9.el9.noarch >>> > ipa-healthcheck-core-0.9-9.el9.noarch >>> > ipa-selinux-4.9.8-7.el9_0.noarch >>> > ipa-server-4.10.0-8.el9_1.x86_64 >>> > ipa-server-common-4.10.0-8.el9_1.noarch >>> > ipa-server-dns-4.10.0-8.el9_1.noarch >>> > libipa_hbac-2.7.3-4.el9_1.3.x86_64 >>> > python3-ipaclient-4.10.0-8.el9_1.noarch >>> > python3-ipalib-4.10.0-8.el9_1.noarch >>> > python3-ipaserver-4.10.0-8.el9_1.noarch >>> > python3-libipa_hbac-2.7.3-4.el9_1.3.x86_64 >>> > sssd-ipa-2.7.3-4.el9_1.3.x86_64 >>> > >>> > thanks, >>> > Nick >>> > >>> > On Tue, 23 May 2023 at 10:01, Florence Blanc-Renaud >>> > <[email protected] <mailto:[email protected]>> wrote: >>> > >>> > Hi, >>> > >>> > the replica-conncheck error means that a call to >>> > server_conncheck reached the wrong server. >>> > ipa-replica-conncheck performs multiple checks: >>> > - first from the replica to the existing master (here >>> we >>> > seem to be good) >>> > - then from the existing master to the replica, by >>> doing >>> > a call to the XMLRPC api server_conncheck on the >>> master. >>> > If the connection from replica to master fails, another >>> > server is tried (in this case, the replica launches >>> > server_conncheck on itself), but there is a security >>> > that ensures the right server is handling the call. >>> > The logs shows that the connection fails because of >>> SASL >>> > auth failure: >>> > 2023-05-22T18:14:03Z INFO Connection to >>> > https://ipa010.ad.companyx.fm/ipa/json failed with >>> > Insufficient access: SASL(-1): generic failure: GSSAPI >>> > Error: No credentials were supplied, or the credentials >>> > were unavailable or inaccessible (Credential cache is >>> empty) >>> > >>> > Are you able to do kinit -kt /etc/krb5.keytab >>> > host/<replicafqdn>@<REALM> on the replica? And then >>> kvno >>> > HTTP/<serverfqdn>@<REALM> ? >>> > flo >>> > >>> > On Tue, May 23, 2023 at 9:55 AM Nicholas Cross via >>> > FreeIPA-users <[email protected] >>> > <mailto:[email protected]>> wrote: >>> > >>> > That was the /var/log/ipareplica-conncheck.log log >>> file >>> > >>> > it does looks like a DNs issue, but im not sure >>> where. >>> > >>> > dns resolves the host fine on the host >>> > >>> > [root@ipa011 ~]# host ipa011 >>> > ipa011.ad.companyx.fm < >>> http://ipa011.ad.companyx.fm> >>> > has address 10.32.225.7 >>> > >>> > [root@ipa011 ~]# grep ipa /etc/ipa/default.conf >>> > host = ipa011.ad.companyx.fm >>> > <http://ipa011.ad.companyx.fm> >>> > xmlrpc_uri = https://ipa011.ad.companyx.fm/ipa/xml >>> > ca_host = ipa010.ad.companyx.fm >>> > <http://ipa010.ad.companyx.fm> >>> > >>> > it's odd as i run the connection check before the >>> > start of the install, to check ports and routes. >>> > it works fine. >>> > replica install works. >>> > dns install works. >>> > just the ca installer comes back with this error. >>> > >>> > As an additional test i added the dns record for >>> > this host into IPA before the install. Normally we >>> > don't need to, but just as a test, but it made no >>> > difference. >>> > >>> > >>> > We do have new DNS forwarders on the network - >>> these >>> > are in front of the IPA servers. They are there >>> > just take the load from the k8s clusters away from >>> > IPA DNS. >>> > Would the CA install break if the DNS lookups are >>> > "proxied" by the DNS forwarders? >>> > All DNS tests i can think of work via the >>> > forwarders. The IPA clients (100s) are all fine >>> > with them. >>> > >>> > I will update the client to ignore the forwarders, >>> > but if you can think of anything else to try? >>> > >>> > thanks, Nick >>> > _______________________________________________ >>> > FreeIPA-users mailing list -- >>> > [email protected] >>> > <mailto:[email protected]> >>> > To unsubscribe send an email to >>> > [email protected] >>> > <mailto:[email protected] >>> > >>> > Fedora Code of Conduct: >>> > >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>> > List Guidelines: >>> > >>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>> > List Archives: >>> > >>> https://lists.fedorahosted.org/archives/list/[email protected] >>> > Do not reply to spam, report it: >>> > https://pagure.io/fedora-infrastructure/new_issue >>> > >>> > >>> > _______________________________________________ >>> > FreeIPA-users mailing list -- [email protected] >>> > To unsubscribe send an email to >>> [email protected] >>> > Fedora Code of Conduct: >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>> > List Guidelines: >>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>> > List Archives: >>> https://lists.fedorahosted.org/archives/list/[email protected] >>> > Do not reply to spam, report it: >>> https://pagure.io/fedora-infrastructure/new_issue >>> > >>> >>>
_______________________________________________ FreeIPA-users mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/[email protected] Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
