[Freeipa-users] freeipa-client joins keep failing : Cannot find KDC for realm
Hello all. First want to thank everyone for all the hard work going into continually making this platform a better and better offering. I'm running into some challenges though in joining clients to a relatively fresh install for a client. I have a pair of replicating IPA nodes that are responding on all ports and services as expected. If I make manual connections to the nodes from clients, I am able to talk successfully via the various services (LDAP, KRB, DNS, NTP). My trouble comes when trying to join clients to the IPA servers. If I run the following: = ipa-client-install -p admin --mkhomedir --hostname=`hostname` -d = The client looks up all the name records correctly, prompts for the admin credentials, then starts exchanging certs, making https calls, and so on, but never completes successfully in joining the client. I keep getting the dreaded "Client uninstall complete." whenever the client-install completes. Parsing through the /var/log/ipaclient-install.log, I see what I believe to be the culprit component of the join process: = ...[output truncated]... 2018-01-15T21:55:24Z DEBUG Writing Kerberos configuration to /etc/krb5.conf: 2018-01-15T21:55:24Z DEBUG #File modified by ipa-client-install includedir /var/lib/sss/pubconf/krb5.include.d/ [libdefaults] default_realm = IPA.XYZ.COM dns_lookup_realm = true dns_lookup_kdc = true rdns = false ticket_lifetime = 24h forwardable = true udp_preference_limit = 0 default_ccache_name = KEYRING:persistent:%{uid} [realms] IPA.XYZ.COM = { pkinit_anchors = FILE:/etc/ipa/ca.crt } [domain_realm] .xyz.com = IPA.XYZ.COM xyz.com = IPA.XYZ.COM 2018-01-15T21:55:24Z INFO Configured /etc/krb5.conf for IPA realm IPA.XYZ.COM 2018-01-15T21:55:24Z DEBUG Starting external process 2018-01-15T21:55:24Z DEBUG args=keyctl search @s user ipa_session_cookie:host/sfca-do-1.xyz@ipa.xyz.com 2018-01-15T21:55:24Z DEBUG Process finished, return code=1 2018-01-15T21:55:24Z DEBUG stdout= 2018-01-15T21:55:24Z DEBUG stderr=keyctl_search: Required key not available 2018-01-15T21:55:24Z DEBUG Starting external process 2018-01-15T21:55:24Z DEBUG args=/usr/bin/certutil -d /tmp/tmpoXIXYU -N -f /tmp/tmpfNulOs 2018-01-15T21:55:24Z DEBUG Process finished, return code=0 2018-01-15T21:55:24Z DEBUG stdout= 2018-01-15T21:55:24Z DEBUG stderr= 2018-01-15T21:55:24Z DEBUG Starting external process 2018-01-15T21:55:24Z DEBUG args=/usr/bin/certutil -d /tmp/tmpoXIXYU -A -n CA certificate 1 -t C,, 2018-01-15T21:55:24Z DEBUG Process finished, return code=0 2018-01-15T21:55:24Z DEBUG stdout= 2018-01-15T21:55:24Z DEBUG stderr= 2018-01-15T21:55:24Z DEBUG Starting external process 2018-01-15T21:55:24Z DEBUG args=/usr/bin/certutil -d /tmp/tmpoXIXYU -A -n CA certificate 2 -t C,, 2018-01-15T21:55:24Z DEBUG Process finished, return code=0 2018-01-15T21:55:24Z DEBUG stdout= 2018-01-15T21:55:24Z DEBUG stderr= 2018-01-15T21:55:24Z DEBUG Starting external process 2018-01-15T21:55:24Z DEBUG args=/usr/bin/certutil -d /tmp/tmpoXIXYU -A -n CA certificate 3 -t C,, 2018-01-15T21:55:24Z DEBUG Process finished, return code=0 2018-01-15T21:55:24Z DEBUG stdout= 2018-01-15T21:55:24Z DEBUG stderr= 2018-01-15T21:55:24Z DEBUG Starting external process 2018-01-15T21:55:24Z DEBUG args=/usr/bin/certutil -d /tmp/tmpoXIXYU -A -n CA certificate 4 -t C,, 2018-01-15T21:55:24Z DEBUG Process finished, return code=0 2018-01-15T21:55:24Z DEBUG stdout= 2018-01-15T21:55:24Z DEBUG stderr= 2018-01-15T21:55:24Z DEBUG Starting external process 2018-01-15T21:55:24Z DEBUG args=/usr/bin/certutil -d /tmp/tmpoXIXYU -A -n CA certificate 5 -t C,, 2018-01-15T21:55:24Z DEBUG Process finished, return code=0 2018-01-15T21:55:24Z DEBUG stdout= 2018-01-15T21:55:24Z DEBUG stderr= 2018-01-15T21:55:24Z DEBUG failed to find session_cookie in persistent storage for principal 'host/sfca-do-1.xyz@ipa.xyz.com' 2018-01-15T21:55:24Z INFO trying https://sfca-do-4.ipa.xyz.com/ipa/json 2018-01-15T21:55:24Z DEBUG Created connection context.rpcclient_140336485358096 2018-01-15T21:55:24Z DEBUG Try RPC connection 2018-01-15T21:55:24Z INFO Forwarding 'ping' to json server 'https://sfca-do-4.ipa.xyz.com/ipa/json' 2018-01-15T21:55:24Z DEBUG Destroyed connection context.rpcclient_140336485358096 2018-01-15T21:55:24Z INFO Cannot connect to the server due to Kerberos error: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (2529639066): Cannot find KDC for realm "IPA.XYZ.COM". Trying with delegate=True 2018-01-15T21:55:24Z INFO trying https://sfca-do-4.ipa.xyz.com/ipa/json 2018-01-15T21:55:24Z DEBUG Created connection context.rpcclient_140336485358096 2018-01-15T21:55:24Z DEBUG Try RPC connection 2018-01-15T21:55:24Z INFO Forwarding 'ping' to json server 'https://sfca-do-4.ipa.xyz.com/ipa/json' 2018-01-15T21:55:24Z WARNING Second connect with delegate=True also failed: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (2529639066): Cannot find
[Freeipa-users] Re: freeipa-client joins keep failing : Cannot find KDC for realm
Yes - I am redacting just the 2nd level domain name portion from any logs. -Chris On 1/17/18 8:27 AM, Robbie Harwood wrote: > Chris Moodywrites: > >> Thanks for taking a look gents. Ask and ye shall receive. :) >> >> -Chris >> >> ===[ CLI output ]== >> root@sfca-do-1:~# ipa-client-install -p admin --mkhomedir >> --hostname=`hostname` >> Discovery was successful! >> Client hostname: sfca-do-1.xyz.com >> Realm: IPA.xyz.COM > Are you redacting this? Because that "xyz" shouldn't be lowercased. > Realm names are all-caps (because some implementations are > case-sensitive and others are not). > > Thanks, > --Robbie smime.p7s Description: S/MIME Cryptographic Signature ___ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org
[Freeipa-users] Re: freeipa-client joins keep failing : Cannot find KDC for realm
Affirmative, it is all caps in the logs. I can re-send the log with the redactions case sensitive if that's helpful. My apologies for causing confusion via my obfuscation. -Chris On 1/17/18 12:36 PM, Robbie Harwood wrote: > Chris Moodywrites: > >> On 1/17/18 8:27 AM, Robbie Harwood wrote: >>> Chris Moody writes: >>> Thanks for taking a look gents. Ask and ye shall receive. :) -Chris ===[ CLI output ]== root@sfca-do-1:~# ipa-client-install -p admin --mkhomedir --hostname=`hostname` Discovery was successful! Client hostname: sfca-do-1.xyz.com Realm: IPA.xyz.COM >>> Are you redacting this? Because that "xyz" shouldn't be lowercased. >>> Realm names are all-caps (because some implementations are >>> case-sensitive and others are not). >> Yes - I am redacting just the 2nd level domain name portion from any logs. > Can you confirm that it's all-caps in the actual logs? > > Thanks, > --Robbie smime.p7s Description: S/MIME Cryptographic Signature ___ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org
[Freeipa-users] Re: freeipa-client joins keep failing : Cannot find KDC for realm
My reply with the log output is pending moderator approval. -Chris On 1/16/18 1:11 PM, Rob Crittenden wrote: > Robbie Harwood via FreeIPA-users wrote: >> Chris Moody via FreeIPA-users <freeipa-users@lists.fedorahosted.org> >> writes: >> >>> 2018-01-15T21:55:24Z INFO Configured /etc/krb5.conf for IPA realm >>> IPA.XYZ.COM >>> 2018-01-15T21:55:24Z DEBUG Starting external process >>> 2018-01-15T21:55:24Z DEBUG args=keyctl search @s user >>> ipa_session_cookie:host/sfca-do-1.xyz@ipa.xyz.com >>> 2018-01-15T21:55:24Z DEBUG Process finished, return code=1 >>> 2018-01-15T21:55:24Z DEBUG stdout= >>> 2018-01-15T21:55:24Z DEBUG stderr=keyctl_search: Required key not available >> I'm not familiar with what IPA's trying to do here, but this looks like >> a problem? Can someone else comment? > This is perfectly normal. IPA stores the session cookie in the kernel > keyring. Given this is a new install there is no cookie to find. > >>> I have tried manually setting /etc/krb5.conf to the contents that get> >>> generated & display during the verbose client-install process (as seen >>> above), that manually spell out the KDC details, and am able to run a >>> 'kinit admin' just fine from the CLI on the client, so kerberos DOES >>> function from the client. It talks to the KDC beautifully and >>> authenticates just fine... so I'm not sure how the client-install >>> process is getting confused/lost when trying to find/contact the KDC. >> Someone else who knows more than me: how is the install different than a >> normal kinit? > I think we'd need to see the full ipaclient-install.log. > > rob smime.p7s Description: S/MIME Cryptographic Signature ___ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org
[Freeipa-users] Re: freeipa-client joins keep failing : Cannot find KDC for realm
Server: = [root@sfca-do-4 ~]# ipa --version VERSION: 4.4.4, API_VERSION: 2.215 [root@sfca-do-4 ~]# cat /etc/fedora-release Fedora release 25 (Twenty Five) Client Node: = root@sfca-do-1:~# ipa-client-install --version 4.3.1 root@sfca-do-1:~# cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=16.04 I should also mention that my Ubuntu 14.04 nodes cannot join either, and they have different freeipa-client versions in their repos and are throwing some different log data if that's of any possible help. The only system that's been able to ipa-client-install join is the IPA replication mate which is running the same rev of Fedora and ipa-client/server. Some more background, these servers for this client were recently built and configured to use letsencrypt certificates so they can provide public and ssl-accepted interfaces to users that this client services. Not sure if certificates and CAs could perhaps be playing into a client-join (since I see no complaint about them in the install logs on this client), but wanted to mention it anyway just in case there's some reason that letsencrypt issued certs are perhaps factoring in. Other clients I service have successfully used similar setups to what I'm trying to build currently, but were running on the 3.x services of IPA. This is my first pass at standing up functioning 4.x IPA servers. Other replies inline. On 1/17/18 2:36 PM, Rob Crittenden via FreeIPA-users wrote: > Chris Moody wrote: >> Thanks for taking a look gents. Ask and ye shall receive. :) >> > What version of IPA is this and what platform? > > Before an install can you ensure that there is nothing in > /etc/krb5.conf.d/ (except may be crypto-policies)? There is no /etc/krb5.conf.d/ dir on the client node. I have tried with both the system defaults in the /etc/krb5.conf file as well as with the contents generated/output by the ipa-client-install command as I mentioned initially if that's the component you're questioning. > > Same with /var/lib/sss/pubconf/krb5.include.d/ On client node: root@sfca-do-1:~# ls -l /var/lib/sss/pubconf/krb5.include.d/ total 0 > > Might also be interesting to try to force a specific master by adding > --server to the install line, just to see. > > I'm guessing the client is old as it doesn't appear to support the > newer-style ipa-getkeytab: Hmm... This client is fully updated/upgraded for any packages installed via the Ubuntu repos. Is the client version 4.3.1 not recent? I can manually add a different repo or pull source if need be to get whichever client version you think might help. > > 2018-01-17T02:11:50Z DEBUG args=/usr/sbin/ipa-join -s > sfca-do-4.ipa.xyz.com -b dc=ipa,dc=xyz,dc=com -h sfca-do-1.xyz.com > 2018-01-17T02:11:51Z DEBUG Process finished, return code=0 > 2018-01-17T02:11:51Z DEBUG stdout= > 2018-01-17T02:11:51Z DEBUG stderr=Failed to parse result: Failed to > decode GetKeytab Control. > > Retrying with pre-4.0 keytab retrieval method... > Keytab successfully retrieved and stored in: /etc/krb5.keytab > Certificate subject base is: O=IPA.xyz.COM > > 2018-01-17T02:11:51Z INFO Enrolled in IPA realm IPA.xyz.COM > > It does look like it enrolls ok and gets a keytab. > > Note too that just about this it is able to get a TGT for the admin user > via kinit: > > 2018-01-17T02:11:50Z DEBUG args=/usr/bin/kinit ad...@ipa.xyz.com -c > /tmp/krbccCNSUmS/ccache > > The only difference between Kerberos usage between the enrollment and > the rest is that during enrollment a fixed KDC is defined in the > temporary krb5.conf: > > includedir /var/lib/sss/pubconf/krb5.include.d/ > > [libdefaults] > default_realm = IPA.xyz.COM > dns_lookup_realm = false > dns_lookup_kdc = false > rdns = false > ticket_lifetime = 24h > forwardable = true > udp_preference_limit = 0 > default_ccache_name = KEYRING:persistent:%{uid} > > > [realms] > IPA.xyz.COM = { > kdc = sfca-do-4.ipa.xyz.com:88 > master_kdc = sfca-do-4.ipa.xyz.com:88 > admin_server = sfca-do-4.ipa.xyz.com:749 > default_domain = xyz.com > pkinit_anchors = FILE:/etc/ipa/ca.crt > > } > > [domain_realm] > .xyz.com = IPA.xyz.COM > xyz.com = IPA.xyz.COM > > > It is failing trying to autodiscover things later: > > includedir /var/lib/sss/pubconf/krb5.include.d/ > > [libdefaults] > default_realm = IPA.xyz.COM > dns_lookup_realm = true > dns_lookup_kdc = true > rdns = false > ticket_lifetime = 24h > forwardable = true > udp_preference_limit = 0 > default_ccache_name = KEYRING:persistent:%{uid} > > > [realms] > IPA.xyz.COM = { > pkinit_anchors = FILE:/etc/ipa/ca.crt > > } > > > [domain_realm] > .xyz.com = IPA.xyz.COM > xyz.com = IPA.xyz.COM > > Discovery appears to be working as expected: > > 2018-01-17T02:11:41Z DEBUG Search DNS for TXT record of _kerberos.xyz.com > 2018-01-17T02:11:41Z DEBUG DNS record found: "IPA.xyz.COM" > 2018-01-17T02:11:41Z DEBUG Search DNS for SRV record of > _kerberos._udp.xyz.com > 2018-01-17T02:11:41Z DEBUG DNS
[Freeipa-users] Re: freeipa-client joins keep failing : Cannot find KDC for realm
That being said, just tried again on an ubuntu 14.04 node with these same CLI params, and it failed, but the logs are complaining about "SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been marked as not trusted by the user", which never was reported in the ubuntu 16 system's logs. Seems to mirror the bug/issue noted here: https://pagure.io/freeipa/issue/7072 I am still curious why one has to explicitly call out '--server' for the Ubuntu 16 system to join. I can also start a different thread for the Ubuntu 14 system debugging if need be, or can just continue here - your call. -Chris On 1/17/18 6:10 PM, Chris Moody via FreeIPA-users wrote: > Just attempted the '--server' option you mention, as well as the > '--domain' value that the parameter requires, and it actually SUCCEEDED > in joining! > > I received "Client configuration complete." via the ipa-client-install > command and was just able to successfully login to this node with a user > in IPA. > > Which is wonderful news however I'm still now wondering what > component might be failing or portion of autodiscovery perhaps > missing/b0rk3d that's necessitating the --server param to be explicitly > called. > > -Chris > > > On 1/17/18 5:30 PM, Chris Moody via FreeIPA-users wrote: >>> Might also be interesting to try to force a specific master by adding >>> --server to the install line, just to see. >>> >>> I'm guessing the client is old as it doesn't appear to support the >>> newer-style ipa-getkeytab: > > > > ___ > FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org > To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org smime.p7s Description: S/MIME Cryptographic Signature ___ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org
[Freeipa-users] Re: freeipa-client joins keep failing : Cannot find KDC for realm
Just attempted the '--server' option you mention, as well as the '--domain' value that the parameter requires, and it actually SUCCEEDED in joining! I received "Client configuration complete." via the ipa-client-install command and was just able to successfully login to this node with a user in IPA. Which is wonderful news however I'm still now wondering what component might be failing or portion of autodiscovery perhaps missing/b0rk3d that's necessitating the --server param to be explicitly called. -Chris On 1/17/18 5:30 PM, Chris Moody via FreeIPA-users wrote: > >> Might also be interesting to try to force a specific master by adding >> --server to the install line, just to see. >> >> I'm guessing the client is old as it doesn't appear to support the >> newer-style ipa-getkeytab: > smime.p7s Description: S/MIME Cryptographic Signature ___ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org
[Freeipa-users] Brand new server install fails - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate
Trying to stand up a brand new IPA Server install on a brand new VM. I am lightly obfuscating some strings out of respect for the client so their domain-name will say 'DOMAIN' in my email. == ~# cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=19.10 DISTRIB_CODENAME=eoan DISTRIB_DESCRIPTION="Ubuntu 19.10" == ~# ipa --version VERSION: 4.8.1, API_VERSION: 2.233 == Having built a number of IPA Servers for various entities in the past, I've already got the requisite setup/prep stuff configured. - DNS Resolution in functioning forward/reverse - /etc/hosts is set correctly to point to the public IPv4 and IPv6 interface IPs. - hostname is set to fqdn. - time is current and sync'd before any IPA commands are run Issuing the following command to kick off the ipa-server-install process: == ipa-server-install --allow-zone-overlap -v -d --setup-dns --mkhomedir --auto-reverse -p X -a Y --forwarder=2604:ZZZ::AAA -n ipa.DOMAIN.com -r IPA.DOMAIN.COM --hostname=`hostname` --ntp-pool=pool.ntp.org == The server install process proceeds and succeeds up to the point: == [6/7]: creating replica keys [7/7]: configuring ipa-dnskeysyncd to start on boot Starting external process = Which is kicking off: = 2020-04-15T20:15:46Z DEBUG args=['/usr/sbin/ipa-client-install', '--on-master', '--unattended', '--domain', 'ipa.DOMAIN.com', '--server', 'sfca-do-ipa-1.ipa.DOMAIN.com', '--realm', 'IPA.DOMAIN.COM', '--hostname', 'sfca-do-ipa-1.ipa.DOMAIN.com', '--no-ntp', '--mkhomedir'] = The client setup portion fails every single time with the following error: = 2020-04-15T20:15:48Z ERROR cannot connect to 'https://sfca-do-ipa-1.ipa.DOMAIN.com/ipa/json': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076) = I've done some searching to see how other people have dealt with python throwing the CERTIFICATE_VERIFY_FAILED error, but nothing seems to make any difference in telling the ipa-client-install to respect the locally issued IPA Certs that are read during the setup process. Since some threads mention it helping, I've ensured the python-certifi package is installed and up to date. I've tried toggling between the version of python being used [the system default of python2.7 or python3.7]. Even though it should not make any difference, since the client is reading an IPA generated cert and complaining, but I've also rebuilt the /etc/ssl/certs store since some threads have mentioned this error having some relations [update-ca-certificates -f -v]. Any thoughts on how to get past the ipa-client-install section failing on this? This server setup is -so- close to being complete. Cheers, -Chris signature.asc Description: OpenPGP digital signature ___ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
[Freeipa-users] Re: FreeIPA w. letsencrypt for HTTPS/LDAP failing to communicate with itself
Apologies for the belated response - took me a bit to verify across all clients. When I installed the LE certs on each replica/server, I performed the following: =(the privkey & fullchain files provided by LE)= ipa-server-certinstall -w -d privkey.pem fullchain.pem & /usr/sbin/ipa-certupdate = I have verified via 'openssl s_client -connect' that both https & ldaps are serving the proper LE certificates across all IPA servers. I have also now iterated across every ipa-client installation and run 'ipa-certupdate' as well. All the /etc/ssl/certs/ca-certificates.crt files on each and every system have the LE certs and are current. Unfortunately, the 'Actions->New Certificate' process I mentioned is still giving me identical behavior. I followed the exact steps in the NewCert dialogue from one of the validated-current ipa-clients: = # Create a certificate database or use an existing one. To create a new database: |# certutil -N -d | # Create a CSR with subject /CN=,O=/, for example: |# certutil -R -d -a -g -s 'CN=chris,O=IPA.REDACTED.COM'| = IPA Error 907: NetworkError cannot connect to 'https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login': [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2508) = -Chris On 6/12/21 3:55 AM, Florence Renaud wrote: Hi, when the let's encrypt certificates were installed, did you run ipa-cacert-manage install on one of the nodes + ipa-certupdate on _all the IPA machines_? It's important to run ipa-certupdate on all the server/replicas/clients in order to install the CA everywhere. flo On Sat, Jun 12, 2021 at 2:19 AM Chris Moody via FreeIPA-users <mailto:freeipa-users@lists.fedorahosted.org>> wrote: Hello folks. Hopefully I'm just missing something face-palm level obvious, but I am running into some trouble when interfacing with my CA functionality on an IPA server cluster. My attempts at scouring all my saved prior-comms from the mailing-list as well as several search-engines are not enchanting me with much clue. It appears that my need for the LetsEncrypt certs for the user-facing Web-UI and LDAPs components are causing IPA to dis-trust itself. === 4-node cluster - Ubuntu19.10 (all nodes currently fully updated/patched via the official Ubuntu repos) === ipa --version VERSION: 4.8.1, API_VERSION: 2.233 === running letsencrypt certificates successfully for HTTPs & LDAPs connectivity === These 4-nodes are all happily running and replicating betwixt each other. LDAPs is functioning great and many linux systems are able to all join as freeipa-clients. Users and groups are replicating and being used elegantly for many LDAP-based authentication/authorization needs. Overall, for these nodes, life is good. Where I'm running into trouble is in finally wanting to leverage certificate issuance on a per-user basis. End goal is integrating things like yubikeys, user-cert auth, and so on. In the UI, when I enter a user's account and select Actions->New Certificate, I am able to successfully issue the couple prompted 'certutil' commands to generate the user's CSR. I then paste in the contents of the CSR and hit 'Issue' and run into the following error: == IPA Error 907: NetworkError cannot connect to 'https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login <https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login>': [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2508) == As I then start digging into cli-mode to attempt to understand where things are unhappy, I run into similar troubles with the server attempting to talk to itself and not being very happy about it. == chris@REDACTED-1:~$ ipa ca-find 1 CA matched Name: ipa Description: IPA CA Authority ID: 8acca54b-64d7-44bf-b8f7-59316213cfb6 Subject DN: CN=Certificate Authority,O=IPA.REDACTED.COM <http://IPA.REDACTED.COM> Issuer DN: CN=Certificate Authority,O=IPA.REDACTED.COM <http://IPA.REDACTED.COM> Number of entries returned 1 chris@REDACTED-1:~$ ipa ca-show Name: ipa ipa: ERROR: cannot connect to 'https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login <https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login>': [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2508) == Verifying with 'openssl s_client' returns the valid and non-expired LE cert-chain. == chris@REDACTED-1:~$ openssl s_client REDACTED-1.ipa.REDACTED.com:443 <http://REDACTED-1.ipa.REDACTED.com:443> CONNECTED(0003) depth=2 O = Digital Signature
[Freeipa-users] Re: FreeIPA w. letsencrypt for HTTPS/LDAP failing to communicate with itself
Just found some additional possible clues in the apache error.log = [Tue Jun 15 17:11:34.636290 2021] [:warn] [pid 31831:tid 139703600768768] [client 2001:470:8af9:255::10:47920] failed to set perms (3140) on file (/run/ipa/ccaches/ch...@ipa.node-nine.com)!, referer: https://REDACTED-1.ipa.REDACTED.com/ipa/ui/ [Tue Jun 15 17:11:34.674975 2021] [ssl:error] [pid 31830:tid 139703550412544] [client 2604:a880:2:d1::36:4001:58500] AH02039: Certificate Verification: Error (19): self signed certificate in certificate chain [Tue Jun 15 17:11:34.675088 2021] [ssl:error] [pid 31830:tid 139703550412544] [client 2604:a880:2:d1::36:4001:58500] AH02261: Re-negotiation handshake failed [Tue Jun 15 17:11:34.675111 2021] [ssl:error] [pid 31830:tid 139703550412544] SSL Library Error: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed [Tue Jun 15 17:11:34.675884 2021] [wsgi:error] [pid 31828:tid 139703722125056] [remote 2001:470:8af9:255::10:47920] ipa: INFO: [jsonserver_session] ch...@ipa.redacted.com: cert_request('-BEGIN NEW CERTIFICATE REQUEST-\\nMIIEcTCCAlkCAQAwLDEaMBgGA1UEChMRSVBBLk5PREUtTklORS5DT00xDjAMBgNV\\nBAMTBWNocmlzMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA8QR2JbFR\\nE7S7/XAk9V1bNEvvXxB65MpLnIu6gLnfXr8yz0xcTjvEDmKAukS7u+GhwnVfQvwS\\nRjlexS5leIZfMFU59R5K6ubzM5e3Qy07xQU40+8jLS2o3SUo7CvY+TcgTmfJWiK4\\nlWlG70LcMy6VbBVf+nQNoenryPDK2Y94kG3ydcKjGLZsaJ69s1OaSXxZY4esEhk/\\nFJs+odjMQRguS5sbUmfpSy3FTJsvuy4Sge2X5HsHOCKM2bWMcDqkRGI428toET0i\\n+FIXlvroqJQ7OL/RmwVcYOFMwMWN5Ne3g0ECZUrOWRNGt+TfMmcUxfDaw1kl/QqA\\nIYtv0xBszOC9ECSAak9pUuhabn616jI0teZE1+4trXc1ZLwBKiF+Ev2F7wFLdoPK\\nTyikms8lzQ/GLj9c61NieXsDRJLU3ZMjhQkcfbKKp5R4fp6UPUuJ8MdDzNDiLyuB\\nkgyRLRmF7NBFGvdEV9javdvAuSQz/Wa2ReGRWhI4hj8OgYZaFrNdWbgIRSZUDy3f\\ngDt8NoS7ouoD8vGQ93OvvaUfqgMhs57qZbSZNTgoLKcDFwGyT4oqfA25wzQhzlXU\\nV+q6JLpyz2J+d2CI4B/7/Udh44YT9LiLn9y84QNBaD3qJWrLuYrZYk9hb5HIuJf7\\nXDSh7zHmSed1BrMn/BnzKpxwl201P/Oy/5cCAwEAAaAAMA0GCSqGSIb3DQEBCwUA\\nA4ICAQBhFLpWtZXDhpnmfRu1tkfVQEuGMhG0MdIC1NXAUMs0EoBXKA0/Ak3xCN9c\\nbbvZtBBOktqIV5bxV+d4X0RHLGYh2NGezWHS5gkA92xzmw9gyUKHRQpDmIV9IicH\\nUCSFwL5xO5DFUC2XXshHFgHqpi3kYxCvfcX+wvrmJwjxsbv7raMVsZlN/6w2Z4/p\\nS1VYakUZnXXBl8x3TbK4yZ+mvRX6RjQOU8oCvZMAGns/IgjTL++4O59XThV7VaAa\\nIromeDGnS9klR/hx7BBy7UureQT/aIBpFczjaU5qPBJxkrFi7K+f3vdms9PZOfKv\\n4eowQhanJwoxhkSE11sVmrQQ9+pmrTXHLgO8IFRWAonnM0La31WQ+uBD9vF8iAGS\\neMk+CvEGV6u2UwZph4P6t/KCCGT89BMnJHTRYmyMulx3Ko4jJDMV9v3nm/Ls75ti\\abcdefghijklmnophPfch6I7wJEp+t7egdgrd5jbj4m4lDOT9v9msknlOXoUu\\nibGzaylvac6xhtggDVdz/OIQS7l0jNZE0t0w5ZgEP2fkUlCP6ZpBBL7hloBIzv28...REDACTED\\n-END NEW CERTIFICATE REQUEST-', cacn='ipa', principal='chris', version='2.233'): NetworkError = -Chris On 6/15/21 5:09 PM, Chris Moody via FreeIPA-users wrote: Apologies for the belated response - took me a bit to verify across all clients. When I installed the LE certs on each replica/server, I performed the following: =(the privkey & fullchain files provided by LE)= ipa-server-certinstall -w -d privkey.pem fullchain.pem & /usr/sbin/ipa-certupdate = I have verified via 'openssl s_client -connect' that both https & ldaps are serving the proper LE certificates across all IPA servers. I have also now iterated across every ipa-client installation and run 'ipa-certupdate' as well. All the /etc/ssl/certs/ca-certificates.crt files on each and every system have the LE certs and are current. Unfortunately, the 'Actions->New Certificate' process I mentioned is still giving me identical behavior. I followed the exact steps in the NewCert dialogue from one of the validated-current ipa-clients: = # Create a certificate database or use an existing one. To create a new database: |# certutil -N -d | # Create a CSR with subject /CN=,O=/, for example: |# certutil -R -d -a -g -s 'CN=chris,O=IPA.REDACTED.COM'| = IPA Error 907: NetworkError cannot connect to 'https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login': [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2508) = -Chris On 6/12/21 3:55 AM, Florence Renaud wrote: Hi, when the let's encrypt certificates were installed, did you run ipa-cacert-manage install on one of the nodes + ipa-certupdate on _all the IPA machines_? It's important to run ipa-certupdate on all the server/replicas/clients in order to install the CA everywhere. flo On Sat, Jun 12, 2021 at 2:19 AM Chris Moody via FreeIPA-users <mailto:freeipa-users@lists.fedorahosted.org>> wrote: Hello folks. Hopefully I'm just missing something face-palm level obvious, but I am running into some trouble when interfacing with my CA functionality on an IPA server cluster. My attempts at scouring all my saved prior-comms from the mailing-list as well as several search-engines are not enchanting me with much clue. It appears that my need for the LetsEncrypt certs for the user-facing Web-UI and LDAPs components are caus
[Freeipa-users] FreeIPA w. letsencrypt for HTTPS/LDAP failing to communicate with itself
Hello folks. Hopefully I'm just missing something face-palm level obvious, but I am running into some trouble when interfacing with my CA functionality on an IPA server cluster. My attempts at scouring all my saved prior-comms from the mailing-list as well as several search-engines are not enchanting me with much clue. It appears that my need for the LetsEncrypt certs for the user-facing Web-UI and LDAPs components are causing IPA to dis-trust itself. === 4-node cluster - Ubuntu19.10 (all nodes currently fully updated/patched via the official Ubuntu repos) === ipa --version VERSION: 4.8.1, API_VERSION: 2.233 === running letsencrypt certificates successfully for HTTPs & LDAPs connectivity === These 4-nodes are all happily running and replicating betwixt each other. LDAPs is functioning great and many linux systems are able to all join as freeipa-clients. Users and groups are replicating and being used elegantly for many LDAP-based authentication/authorization needs. Overall, for these nodes, life is good. Where I'm running into trouble is in finally wanting to leverage certificate issuance on a per-user basis. End goal is integrating things like yubikeys, user-cert auth, and so on. In the UI, when I enter a user's account and select Actions->New Certificate, I am able to successfully issue the couple prompted 'certutil' commands to generate the user's CSR. I then paste in the contents of the CSR and hit 'Issue' and run into the following error: == IPA Error 907: NetworkError cannot connect to 'https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login': [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2508) == As I then start digging into cli-mode to attempt to understand where things are unhappy, I run into similar troubles with the server attempting to talk to itself and not being very happy about it. == chris@REDACTED-1:~$ ipa ca-find 1 CA matched Name: ipa Description: IPA CA Authority ID: 8acca54b-64d7-44bf-b8f7-59316213cfb6 Subject DN: CN=Certificate Authority,O=IPA.REDACTED.COM Issuer DN: CN=Certificate Authority,O=IPA.REDACTED.COM Number of entries returned 1 chris@REDACTED-1:~$ ipa ca-show Name: ipa ipa: ERROR: cannot connect to 'https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login': [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2508) == Verifying with 'openssl s_client' returns the valid and non-expired LE cert-chain. == chris@REDACTED-1:~$ openssl s_client REDACTED-1.ipa.REDACTED.com:443 CONNECTED(0003) depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3 verify return:1 depth=1 C = US, O = Let's Encrypt, CN = R3 verify return:1 depth=0 CN = REDACTED-1.ipa.REDACTED.com verify return:1 --- Certificate chain 0 s:CN = REDACTED-1.ipa.REDACTED.com i:C = US, O = Let's Encrypt, CN = R3 1 s:C = US, O = Let's Encrypt, CN = R3 i:O = Digital Signature Trust Co., CN = DST Root CA X3 --- .. --- SSL handshake has read 3046 bytes and written 413 bytes Verification: OK --- New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384 Server public key is 2048 bit Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 0 (ok) .. == Can anyone please hit me with some clue-bat as to where I can read to understand how to get IPA to love itself? I'm suspecting it's likely some certificate inclusion/exception that I need to add so that API calls and the ipa command itself will actually respect the LE cert-chain? Any hints would be greatly appreciated. Thanks, -Chris -- Node-Nine, Inc. ch...@node-nine.com 619.354.6463 OpenPGP_signature Description: OpenPGP digital signature ___ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
[Freeipa-users] Re: FreeIPA w. letsencrypt for HTTPS/LDAP failing to communicate with itself
dns: REDACTED-1.ipa.REDACTED.com key usage: digitalSignature,keyEncipherment,dataEncipherment eku: id-kp-serverAuth pre-save command: /usr/lib/ipa/certmonger/stop_pkicad post-save command: /usr/lib/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca" track: yes auto-renew: yes Request ID '20200416204854': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/lib/ipa/certs/kdc.key' certificate: type=FILE,location='/var/lib/ipa/certs/kdc.crt' CA: IPA issuer: CN=Certificate Authority,O=IPA.REDACTED.COM subject: CN=REDACTED-1.ipa.REDACTED.com,O=IPA.REDACTED.COM expires: 2022-04-17 13:48:54 PDT principal name: krbtgt/ipa.redacted@ipa.redacted.com key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-pkinit-KPKdc pre-save command: post-save command: /usr/lib/ipa/certmonger/renew_kdc_cert track: yes auto-renew: yes Request ID '20200416205655': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-kra',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-kra',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=IPA.REDACTED.COM subject: CN=KRA Audit,O=IPA.REDACTED.COM expires: 2022-04-06 13:53:14 PDT key usage: digitalSignature,nonRepudiation pre-save command: /usr/lib/ipa/certmonger/stop_pkicad post-save command: /usr/lib/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-kra" track: yes auto-renew: yes Request ID '20200416205656': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert cert-pki-kra',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert cert-pki-kra',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=IPA.REDACTED.COM subject: CN=KRA Transport Certificate,O=IPA.REDACTED.COM expires: 2022-04-06 13:53:13 PDT key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-clientAuth pre-save command: /usr/lib/ipa/certmonger/stop_pkicad post-save command: /usr/lib/ipa/certmonger/renew_ca_cert "transportCert cert-pki-kra" track: yes auto-renew: yes Request ID '20200416205657': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=IPA.REDACTED.COM subject: CN=KRA Storage Certificate,O=IPA.REDACTED.COM expires: 2022-04-06 13:53:13 PDT key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-clientAuth pre-save command: /usr/lib/ipa/certmonger/stop_pkicad post-save command: /usr/lib/ipa/certmonger/renew_ca_cert "storageCert cert-pki-kra" track: yes auto-renew: yes = Where else should I be looking to try and understand/debug why the server is rejecting itś own connection to itself? From my (albeit limited) understanding thus far, all the requisite components are present and accounted for, no? Do my apache logs of the following give any hints to anyone as to what isn´t being trusted? = [Tue Jun 15 17:11:34.674975 2021] [ssl:error] [pid 31830:tid 139703550412544] [client 2604:XXX::36:4001:58500] AH02039: Certificate Verification: Error (19): self signed certificate in certificate chain [Tue Jun 15 17:11:34.675088 2021] [ssl:error] [pid 31830:tid 139703550412544] [client 2604:XXX::36:4001:58500] AH02261: Re-negotiation handshake failed [Tue Jun 15 17:11:34.675111 2021] [ssl:error] [pid 31830:tid 139703550412544] SSL Library Error: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed = -Chris On 6/16/21 3:32 PM, Chris Moody via FreeIPA-users wrote: That was kinda my belief thus far as well that the hosts were not trusting themselves - not 100% sure how things got here though. I have a hunch it might be related to the initial deployment and the prior admin using an outdated method to install/manage/renew the LE-certificates. = # ipa-cacert-manage list IPA.REDACTED.COM IPA CA IPA.REDACTED.COM IPA CA DSTRootCAX3 letsencryptx3 isrgrootx1 lets-encrypt-r3-cross-signed The ipa-cacert-manage command was successful = How would I go about forcing re-installation of the host's own CA certificate to ensure it's trust? Also, since these nodes are not runn
[Freeipa-users] Re: FreeIPA w. letsencrypt for HTTPS/LDAP failing to communicate with itself
That was kinda my belief thus far as well that the hosts were not trusting themselves - not 100% sure how things got here though. I have a hunch it might be related to the initial deployment and the prior admin using an outdated method to install/manage/renew the LE-certificates. = # ipa-cacert-manage list IPA.XXXYYY.COM IPA CA IPA.XXXYYY.COM IPA CA DSTRootCAX3 letsencryptx3 isrgrootx1 lets-encrypt-r3-cross-signed The ipa-cacert-manage command was successful = How would I go about forcing re-installation of the host's own CA certificate to ensure it's trust? Also, since these nodes are not running on an RPM-based distro, the typical cert-store locations I have seen on other systems are not in the same location(s) so I'm not sure totally sure every location to point certutil to be able to examine each cert-store in depth as well (if that might help diagnose further). I ask because I believe these were initially built and then had the "https://github.com/freeipa/freeipa-letsencrypt/; project used to initially deploy the LE-certs - prior to the ipa-cacert-manage command being the official path toward installing/managing these external certificates. I know because this code had been git-pulled onto these nodes, but it obviously doesn't work properly since this git project manipulates the paths below directly instead of managing via the ipa-cacert-manage command. ex> /etc/httpd/alias/ (<=== not on these systems) /etc/pki/pki-tomcat/alias/ /etc/ipa/nssdb/ /etc/dirsrv/slapd-IPA-XXXYYY-COM/ Checking that project's git page now though, I see their readme now mentions /var/lib/ipa/certs/, where I just noticed cacert.pem. = /var/lib/ipa/certs# openssl x509 -text -noout -in cacert.pem Certificate: Data: Version: 3 (0x2) Serial Number: 1 (0x1) Signature Algorithm: sha256WithRSAEncryption Issuer: O = IPA.XXXYYY.COM, CN = Certificate Authority Validity Not Before: Apr 16 20:45:46 2020 GMT Not After : Apr 16 20:45:46 2040 GMT Subject: O = IPA.XXXYYY.COM, CN = Certificate Authority ... = so I believe I might have just located the IPA CA cert in case I need to re-install it. To the following question, I have the following LE-related certs installed. And yes, I did run into issues a couple months back when LE moved to the new certs on their end so had to import the new authority certs to get the LE host certs to update & import. The LE certificates are functioning and verify for both slapd and apache/tomcat. = DSTRootCAX3.pem LetsEncryptAuthorityX3.pem isrgrootx1.pem lets-encrypt-r3-cross-signed.pem = Thank you all so much for the assistance through all this. -Chris On 6/16/21 1:26 PM, Rob Crittenden via FreeIPA-users wrote: The error suggests that your IPA server doesn't trust its own CA certificate. Does ipa-cacert-manage list include the IPA CA? BTW the new certificate steps are unrelated. This affects all CA requests. rob Chris Moody via FreeIPA-users wrote: Just found some additional possible clues in the apache error.log = [Tue Jun 15 17:11:34.636290 2021] [:warn] [pid 31831:tid 139703600768768] [client 2001:470:8af9:255::10:47920] failed to set perms (3140) on file (/run/ipa/ccaches/ch...@ipa.node-nine.com)!, referer: https://REDACTED-1.ipa.REDACTED.com/ipa/ui/ [Tue Jun 15 17:11:34.674975 2021] [ssl:error] [pid 31830:tid 139703550412544] [client 2604:a880:2:d1::36:4001:58500] AH02039: Certificate Verification: Error (19): self signed certificate in certificate chain [Tue Jun 15 17:11:34.675088 2021] [ssl:error] [pid 31830:tid 139703550412544] [client 2604:a880:2:d1::36:4001:58500] AH02261: Re-negotiation handshake failed [Tue Jun 15 17:11:34.675111 2021] [ssl:error] [pid 31830:tid 139703550412544] SSL Library Error: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed [Tue Jun 15 17:11:34.675884 2021] [wsgi:error] [pid 31828:tid 139703722125056] [remote 2001:470:8af9:255::10:47920] ipa: INFO: [jsonserver_session] ch...@ipa.redacted.com: cert_request('-BEGIN NEW CERTIFICATE REQUEST-\\nMIIEcTCCAlkCAQAwLDEaMBgGA1UEChMRSVBBLk5PREUtTklORS5DT00xDjAMBgNV\\nBAMTBWNocmlzMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA8QR2JbFR\\nE7S7/XAk9V1bNEvvXxB65MpLnIu6gLnfXr8yz0xcTjvEDmKAukS7u+GhwnVfQvwS\\nRjlexS5leIZfMFU59R5K6ubzM5e3Qy07xQU40+8jLS2o3SUo7CvY+TcgTmfJWiK4\\nlWlG70LcMy6VbBVf+nQNoenryPDK2Y94kG3ydcKjGLZsaJ69s1OaSXxZY4esEhk/\\nFJs+odjMQRguS5sbUmfpSy3FTJsvuy4Sge2X5HsHOCKM2bWMcDqkRGI428toET0i\\n+FIXlvroqJQ7OL/RmwVcYOFMwMWN5Ne3g0ECZUrOWRNGt+TfMmcUxfDaw1kl/QqA\\nIYtv0xBszOC9ECSAak9pUuhabn616jI0teZE1+4trXc1ZLwBKiF+Ev2F7wFLdoPK\\nTyikms8lzQ/GLj9c61NieXsDRJLU3ZMjhQkcfbKKp5R4fp6UPUuJ8MdDzNDiLyuB\\nkgyRLRmF7NBFGvdEV9javdvAuSQz/Wa2ReGRWhI4hj8OgYZaFrNdWbgIRSZUDy3f\\ngDt8NoS7ouoD8vGQ93OvvaUfqgMhs57qZbSZNTgoLKcDFwGyT4oqfA25wzQhzlXU\\nV+q6JLpyz2J+d2CI4B/7/Udh44YT9LiLn9y84QNBaD3qJWrLuYrZYk9hb5HIuJf7\\nXDSh7zHmSed1BrMn/