Hi, I turned on the debugging, but it showed nothing at all when i ran the conn-check command again. :(
Interestingly this turn up in the http errors log when restarted it (but before i ran conn-check) [Thu May 25 09:58:25.521316 2023] [wsgi:error] [pid 2226275:tid 2226275] ipa: ERROR: Failed to pre-populate LDAP schema cache: uri=ldapi://%2Frun%2Fslapd-AD-companyx-FM.socket: Unable to retrieve LDAP schema: Inappropriate authentication: Anonymous access is not allowed. Is this just a warning? We do turn off full anonymous access for Security reasons. Also, when running conn-check, I get the same error if i run the conn-check command against my new server ipa011 from any other current IPA server eg. from ipa008 to ipa011, or ipa004 to ipa011. But not if i run from a current server to another current server. eg. ipa008 to ipa004 this works fine. So it seems the issue is just this newly installed IPA server. # # from ipa011 to ipa008 # Check RPC connection to remote master Connection to https://ipa008.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty) Connection to https://ipa010.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty) Connection to https://ipa004dc.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty) Connection to https://ipa006dc.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty) Execute check on remote master ERROR: Remote master check failed with following error message(s): invalid 'cn': must be "ipa011.ad.companyx.fm" timed out waiting for input: auto-logout # # from ipa008 to ipa011 # Check RPC connection to remote master Connection to https://ipa011.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty) Execute check on remote master ERROR: Remote master check failed with following error message(s): invalid 'cn': must be "ipa008.ad.companyx.fm" # # from ipa008 to ipa010 # Check RPC connection to remote master Execute check on remote master Check connection from master to remote replica 'ipa008.ad.companyx.fm': Directory Service: Unsecure port (389): OK Directory Service: Secure port (636): OK Kerberos KDC: TCP (88): OK Failed to connect to port 88 udp on 10.32.218.203 Kerberos KDC: UDP (88): WARNING Kerberos Kpasswd: TCP (464): OK Failed to connect to port 464 udp on 10.32.218.203 Kerberos Kpasswd: UDP (464): WARNING HTTP Server: Unsecure port (80): OK HTTP Server: Secure port (443): OK The following UDP ports could not be verified as open: 88, 464 This can happen if they are already bound to an application and ipa-replica-conncheck cannot attach own UDP responder. Connection from master to replica is OK. Currently, i am trying to find the python code that spits out the error and then maybe to tweak it a little to throw some more debug out. maybe here? https://github.com/freeipa/freeipa/blob/master/ipaserver/plugins/server.py#L927 thanks, Nick On Thu, 25 May 2023 at 10:13, Florence Blanc-Renaud <[email protected]> wrote: > Hi, > > On Thu, May 25, 2023 at 9:53 AM Nicholas Cross <[email protected]> > wrote: > >> Hi, thanks for the reply. >> >> This was the process. >> 1. prep >> 2. test connectivity - works ok >> 3. install replica (successfully) - no DNS no CA at this point >> 4. test connectivity - fails. >> >> If i replace step 4 for the following: >> >> 4. install DNS (successfully) >> 5. install CA (fails - with the "invalid 'cn': must be" error) >> >> If we can ignore the step 4 where the comms check fails and concentrate >> on step 5 where the CA fails. I am happy to do so. >> But in step 5. it is the comms check that the `ipa-ca-install` command >> runs that also fails as far as i can see (no expert here myself, though my >> company thinks so LOL) >> >> Successful DNS install logs http://sprunge.us/7WCtpW >> >> Failed CA install logs http://sprunge.us/hTVLzL >> >> I believe this error needs to be investigated: > > 2023-05-25T07:52:07Z INFO Connection to > https://ipa008.ad.companyx.fm/ipa/json failed with Insufficient access: > SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the > credentials were unavailable or inaccessible (Credential cache is empty) > > We can see that the process is able to connect to the master and get a > cookie but later on GSSAPI auth fails. > > Are there any corresponding logs in the master in > /var/log/httpd/error_log? You can enable debug logs on the master with the > following procedure: > - create /etc/ipa/server.conf: > [global] > debug=True > > then restart apache with systemctl restart httpd. > You can also run ipa-healthcheck on the master/the replica to find any > config issue. > > flo > > [root@ipa011 ~]# ipa-ca-install >> Directory Manager (existing master) password: >> >> Run connection check to master >> >> Your system may be partly configured. >> Run /usr/sbin/ipa-server-install --uninstall to clean up. >> >> Connection check failed! >> See /var/log/ipareplica-conncheck.log for more information. >> If the check results are not valid it can be skipped with >> --skip-conncheck parameter. >> >> >> >> thanks, >> Nick >> >> On Thu, 25 May 2023 at 08:18, Florence Blanc-Renaud <[email protected]> >> wrote: >> >>> Hi, >>> >>> On Wed, May 24, 2023 at 3:29 PM Nicholas Cross <[email protected]> >>> wrote: >>> >>>> Hi Flo (and other helpful people on this list), >>>> >>>> After fixing the SID/PAC issue, i am back to looking as to why the >>>> ipa-replica-conncheck fails when installing the CA to a (working) replica. >>>> >>>> I ran your suggested commands and they look fine now. >>>> >>>> [root@ipa011 ~]# kdestroy >>>> kdestroy: No credentials cache found while destroying cache >>>> >>>> [root@ipa011 ~]# kinit -V -kt /etc/krb5.keytab host/ >>>> [email protected] >>>> Using default cache: 0 >>>> Using principal: host/[email protected] >>>> Using keytab: /etc/krb5.keytab >>>> Authenticated to Kerberos v5 >>>> >>>> [root@ipa011 ~]# kvno HTTP/[email protected] >>>> HTTP/[email protected]: kvno = 1 >>>> >>>> I can run the following BEFORE i install the replica and it works 100%, >>>> it is part of our pre-reqs in our docs to make sure comms work. >>>> >>>> ipa-replica-conncheck --master=ipa008.ad.companyx.fm >>>> --auto-master-check --realm=AD.companyx.FM --principal=nicholas.cross >>>> >>>> After i install the replica with ipa-replica-install. (no DNS, no CA) >>>> then running the above ipa-replica-conncheck command fails and we see the >>>> errors as described. >>>> >>> >>> ipa-replica-conncheck is intended to be run *before* replica >>> installation, as stated in the man page: >>> ipa-replica-conncheck - Check a replica-master network connection >>> before installation >>> >>> The reason is that it opens server sockets listening on the tested >>> ports, for the master to try to reach the replica. If the replica is >>> already installed, the ports are already in use by the ldap server, the >>> apache web server etc... >>> >>> I understand that you tried to run ipa-replica-conncheck to ensure that >>> ipa-dns-install or ipa-ca-install won't hit any issue. Did you actually try >>> to install the DNS service and the CA service on the replica? If you see >>> errors in those steps we can help troubleshoot, based on the produced logs, >>> but ipa-replica-conncheck won't be of any use here. >>> >>> flo >>> >>>> >>>> Some logs: >>>> 1. before ipa-replica-install - conn check works. >>>> http://sprunge.us/4yrInD >>>> 2. after successful ipa-replica-install - but conn check fails >>>> http://sprunge.us/gvqEle >>>> >>>> new replica versions >>>> [root@ipa011 ~]# rpm -qa | grep ipa | sort >>>> almalinux-logos-ipa-90.5.1-1.1.el9.noarch >>>> ipa-client-4.10.1-6.el9.x86_64 >>>> ipa-client-common-4.10.1-6.el9.noarch >>>> ipa-common-4.10.1-6.el9.noarch >>>> ipa-healthcheck-0.12-1.el9.noarch >>>> ipa-healthcheck-core-0.12-1.el9.noarch >>>> ipa-selinux-4.9.8-7.el9_0.noarch >>>> ipa-server-4.10.1-6.el9.x86_64 >>>> ipa-server-common-4.10.1-6.el9.noarch >>>> ipa-server-dns-4.10.1-6.el9.noarch >>>> libipa_hbac-2.8.2-2.el9.x86_64 >>>> python3-ipaclient-4.10.1-6.el9.noarch >>>> python3-ipalib-4.10.1-6.el9.noarch >>>> python3-ipaserver-4.10.1-6.el9.noarch >>>> python3-libipa_hbac-2.8.2-2.el9.x86_64 >>>> sssd-ipa-2.8.2-2.el9.x86_64 >>>> >>>> current master versions >>>> [nicholas.cross@ipa008 ~]$ rpm -qa | grep ipa | sort >>>> almalinux-logos-ipa-90.5.1-1.1.el9.noarch >>>> ipa-client-4.10.0-8.el9_1.x86_64 >>>> ipa-client-common-4.10.0-8.el9_1.noarch >>>> ipa-common-4.10.0-8.el9_1.noarch >>>> ipa-healthcheck-0.9-9.el9.noarch >>>> ipa-healthcheck-core-0.9-9.el9.noarch >>>> ipa-selinux-4.9.8-7.el9_0.noarch >>>> ipa-server-4.10.0-8.el9_1.x86_64 >>>> ipa-server-common-4.10.0-8.el9_1.noarch >>>> ipa-server-dns-4.10.0-8.el9_1.noarch >>>> libipa_hbac-2.7.3-4.el9_1.3.x86_64 >>>> python3-ipaclient-4.10.0-8.el9_1.noarch >>>> python3-ipalib-4.10.0-8.el9_1.noarch >>>> python3-ipaserver-4.10.0-8.el9_1.noarch >>>> python3-libipa_hbac-2.7.3-4.el9_1.3.x86_64 >>>> sssd-ipa-2.7.3-4.el9_1.3.x86_64 >>>> >>>> thanks, >>>> Nick >>>> >>>> On Tue, 23 May 2023 at 10:01, Florence Blanc-Renaud <[email protected]> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> the replica-conncheck error means that a call to server_conncheck >>>>> reached the wrong server. ipa-replica-conncheck performs multiple checks: >>>>> - first from the replica to the existing master (here we seem to be >>>>> good) >>>>> - then from the existing master to the replica, by doing a call to the >>>>> XMLRPC api server_conncheck on the master. If the connection from replica >>>>> to master fails, another server is tried (in this case, the replica >>>>> launches server_conncheck on itself), but there is a security that ensures >>>>> the right server is handling the call. >>>>> The logs shows that the connection fails because of SASL auth failure: >>>>> 2023-05-22T18:14:03Z INFO Connection to >>>>> https://ipa010.ad.companyx.fm/ipa/json failed with Insufficient >>>>> access: SASL(-1): generic failure: GSSAPI Error: No credentials were >>>>> supplied, or the credentials were unavailable or inaccessible (Credential >>>>> cache is empty) >>>>> >>>>> Are you able to do kinit -kt /etc/krb5.keytab >>>>> host/<replicafqdn>@<REALM> on the replica? And then kvno >>>>> HTTP/<serverfqdn>@<REALM> ? >>>>> flo >>>>> >>>>> On Tue, May 23, 2023 at 9:55 AM Nicholas Cross via FreeIPA-users < >>>>> [email protected]> wrote: >>>>> >>>>>> That was the /var/log/ipareplica-conncheck.log log file >>>>>> >>>>>> it does looks like a DNs issue, but im not sure where. >>>>>> >>>>>> dns resolves the host fine on the host >>>>>> >>>>>> [root@ipa011 ~]# host ipa011 >>>>>> ipa011.ad.companyx.fm has address 10.32.225.7 >>>>>> >>>>>> [root@ipa011 ~]# grep ipa /etc/ipa/default.conf >>>>>> host = ipa011.ad.companyx.fm >>>>>> xmlrpc_uri = https://ipa011.ad.companyx.fm/ipa/xml >>>>>> ca_host = ipa010.ad.companyx.fm >>>>>> >>>>>> it's odd as i run the connection check before the start of the >>>>>> install, to check ports and routes. it works fine. >>>>>> replica install works. >>>>>> dns install works. >>>>>> just the ca installer comes back with this error. >>>>>> >>>>>> As an additional test i added the dns record for this host into IPA >>>>>> before the install. Normally we don't need to, but just as a test, but >>>>>> it >>>>>> made no difference. >>>>>> >>>>>> >>>>>> We do have new DNS forwarders on the network - these are in front of >>>>>> the IPA servers. They are there just take the load from the k8s >>>>>> clusters >>>>>> away from IPA DNS. >>>>>> Would the CA install break if the DNS lookups are "proxied" by the >>>>>> DNS forwarders? >>>>>> All DNS tests i can think of work via the forwarders. The IPA >>>>>> clients (100s) are all fine with them. >>>>>> >>>>>> I will update the client to ignore the forwarders, but if you can >>>>>> think of anything else to try? >>>>>> >>>>>> thanks, Nick >>>>>> _______________________________________________ >>>>>> FreeIPA-users mailing list -- [email protected] >>>>>> To unsubscribe send an email to >>>>>> [email protected] >>>>>> Fedora Code of Conduct: >>>>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>>>>> List Guidelines: >>>>>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>>>>> List Archives: >>>>>> https://lists.fedorahosted.org/archives/list/[email protected] >>>>>> Do not reply to spam, report it: >>>>>> https://pagure.io/fedora-infrastructure/new_issue >>>>>> >>>>>
_______________________________________________ FreeIPA-users mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/[email protected] Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
