Hi Florence,
On Fri, 21 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
On 12/21/18 11:58 AM, dbischof--- via FreeIPA-users wrote:
thank you very much for your help.
On Fri, 21 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
On 12/20/18 6:52 PM, dbischof--- via FreeIPA-users wrote:
On Thu, 20 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
On 12/20/18 4:22 PM, dbischof--- via FreeIPA-users wrote:
my IPA system consists of 2 masters with their own self-signed CAs,
one of
them being the certificate renewal master (ipa1). The system has
been
running for years and has been migrated from an IPA 3 system.
Since a while, the Web UI logins on ipa1 don't work anymore ("Login
failed
due to an unknown reason.").
Web UI logins on the other server (ipa2) work and everything else is
working fine, too, ipactl status reports all services running.
On login attempt:
--- httpd log
[...]
[:error] [pid 15551] [remote 141.51.X.X:0] mod_wsgi (pid=15551):
Exception
occurred processing WSGI script '/usr/share/ipa/wsgi.py'.
[...]
[:error] [pid 15551] [remote 141.51.X.X:0] CalledProcessError:
Command
'/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_15551 -X
X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X
X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem'
returned
non-zero exit status 1
---
--- krb5kdc.log
[...]
Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): AS_REQ (8
etypes
{18 17 20 19 16 23 25 26}) 141.51.X.Y: NEEDED_PREAUTH:
WELLKNOWN/anonym...@example.com for krbtgt/example....@example.com,
Additional pre-authentication required
Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): closing down
fd
11
Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): AS_REQ (8
etypes
{18 17 20 19 16 23 25 26}) 141.51.X.Y: KDC_RETURN_PADATA:
WELLKNOWN/anonym...@example.com for krbtgt/example....@example.com,
Failed
to verify own certificate (depth 0): certificate has expired
Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): closing down
fd
11
---
--- ipa-checkcerts.py
IPA version 4.5.4-10.el7.centos.3
Check CA status
Check tracking
Check NSS trust
Check dates
Checking certificates in CS.cfg
Comparing certificates to requests in LDAP
Checking RA certificate
Checking authorities
Checking host keytab
Validating certificates
Checking renewal master
End-to-end cert API test
Checking permissions and ownership
Failures:
Unable to find request for serial 268304391
Unable to find request for serial 268304394
Unable to find request for serial 268304393
Unable to find request for serial 268304392
Subject O=EXAMPLE.COM,CN=ipa2.example.com and template subject
CN=ipa1.example.com,O=EXAMPLE.COM do not match for serial 57
---
--- ipa pkinit-status --all
-----------------
2 servers matched
-----------------
Server name: ipa2.example.com
PKINIT status: enabled
Server name: ipa1.example.com
PKINIT status: enabled
----------------------------
Number of entries returned 2
----------------------------
To my understanding, proper certificate exchange between my two
servers
ceased working at some point. How do i track this down and fix it?
your issue looks similar to ticket #6792 [1]. Can you check the result
of
upgrade in /var/log/ipaupgrade.log?
Also check the output of
$ ipa-pkinit-manage status
and if the files /var/lib/ipa-client/pki/kdc-ca-bundle.pem and
/var/lib/ipa-client/pki/ca-bundle.pem exist, with -rw-r--r--
permissions.
Regarding the certificates, does getcert list show expired certs?
flo
[1] https://pagure.io/freeipa/issue/6792
---
$ ipa-pkinit-manage status
PKINIT is enabled
---
There are no expired certificates, kdc-ca-bundle.pem and ca-bundle.pem
exist with proper permissions, but I found something in ipaupgrade.log:
---
2018-09-12T13:37:18Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias
-L
-f /etc/httpd/alias/pwdfile.txt
2018-09-12T13:37:19Z DEBUG Process finished, return code=0
2018-09-12T13:37:19Z DEBUG stdout=
Certificate Nickname Trust
Attributes
SSL,S/MIME,JAR/XPI
Server-Cert u,u,u
EXAMPLE.COM IPA CA CT,C,C
2018-09-12T13:37:19Z DEBUG stderr=
2018-09-12T13:37:19Z DEBUG Starting external process
2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias
-L
-n EXAMPLE.COM IPA CA -a -f /etc/httpd/alias/pwdfile.txt
2018-09-12T13:37:19Z DEBUG Process finished, return code=0
2018-09-12T13:37:19Z DEBUG stdout=
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
2018-09-12T13:37:19Z DEBUG stderr=
2018-09-12T13:37:19Z DEBUG Executing upgrade plugin:
update_ra_cert_store
2018-09-12T13:37:19Z DEBUG raw: update_ra_cert_store
2018-09-12T13:37:19Z DEBUG raw: ca_is_enabled(version=u'2.228')
2018-09-12T13:37:19Z DEBUG ca_is_enabled(version=u'2.228')
2018-09-12T13:37:19Z DEBUG Starting external process
2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias
-L
-n ipaCert -a -f /etc/httpd/alias/pwdfile.txt
2018-09-12T13:37:19Z DEBUG Process finished, return code=255
2018-09-12T13:37:19Z DEBUG stdout=
2018-09-12T13:37:19Z DEBUG stderr=certutil: Could not find cert:
ipaCert
: PR_FILE_NOT_FOUND_ERROR: File not found
[...]
---
this error can be ignored in most of the cases. The upgrade is trying to
move ipaCert (cert+key) from the NSS database /etc/httpd/alias to the
files /var/lib/ipa/ra-agent.pem and /var/lib/ipa/ra-agent.key. So if the
upgrade is run a second time, he won't find ipaCert in the NSS database.
To be sure, you can check if /var/lib/ipa/ra-agent.{pem|key} are present
and contain a certificate with Subject CN=IPA RA,O=DOMAIN.COM. The files
must be readable by root and ipaapi group, and must contain the same cert
as the other masters.
checked this, is ok.
What is the content of /var/lib/ipa-client/pki/kdc-ca-bundle.pem and
ca-bundle.pem? both must contain IPA CA certificate.
True on both servers.
What are the permissions of /var/kerberos/krb5kdc/kdc.crt? It needs to be
readable by everyone. And what is the content of this cert? It should be
issued by IPA CA.
Permissions are ok, contents:
--- ipa1
$ openssl x509 -in /var/kerberos/krb5kdc/kdc.crt -text -noout -fingerprint
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1 (0x1)
Signature Algorithm: sha256WithRSAEncryption
Issuer: O=EXAMPLE.COM, CN=ipa1.example.com
Validity
Not Before: Nov 28 12:43:05 2017 GMT
Not After : Nov 28 12:43:05 2018 GMT
Subject: O=EXAMPLE.COM, CN=ipa1.example.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
[...]
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Alternative Name:
othername:<unsupported>, othername:<unsupported>
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Key Identifier:
86:52:EC:A1:C3:FB:EC:CC:6D:F2:09:E7:64:88:D1:80:F4:71:81:AE
1.3.6.1.4.1.311.20.2:
.".K.D.C.s._.P.K.I.N.I.T._.C.e.r.t.s
[...]
---
--- ipa2
$ openssl x509 -in /var/kerberos/krb5kdc/kdc.crt -text -noout -fingerprint
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 805240833 (0x2fff0001)
Signature Algorithm: sha256WithRSAEncryption
Issuer: O=EXAMPLE.COM, CN=Certificate Authority
Validity
Not Before: Jan 18 13:04:17 2018 GMT
Not After : Jan 19 13:04:17 2020 GMT
Subject: O=EXAMPLE.COM, CN=ipa2.example.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
[...]
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Authority Key Identifier:
keyid:4B:BA:AA:46:F1:29:E4:43:8B:DC:30:B4:90:3E:66:72:DD:F6:C7:FB
Authority Information Access:
OCSP - URI:http://ipa-ca.example.com/ca/ocsp
X509v3 Key Usage: critical
Digital Signature, Non Repudiation, Key Encipherment,
Data Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, 1.3.6.1.5.2.3.5
X509v3 CRL Distribution Points:
Full Name:
URI:http://ipa-ca.example.com/ipa/crl/MasterCRL.bin
CRL Issuer:
DirName: O = ipaca, CN = Certificate Authority
X509v3 Subject Key Identifier:
96:C3:94:70:7E:46:77:DB:91:F8:DF:D6:27:FE:73:0A:45:F3:78:F3
X509v3 Subject Alternative Name:
othername:<unsupported>, othername:<unsupported>
[...]
---
Additional info: I have DNS separate from IPA, but i (hopefully) made
proper records as IPA would have done it. In particular, i made an A
record "ipa-ca" that has IPs of both ipa1 and ipa2 - hope, this is not the
root cause of my problems, since DNS is not under my control.
Since /var/kerberos/krb5kdc/kdc.crt on ipa1 appears to be not issued by
IPA CA, might this be the actual problem?
The cert is expired on ipa1, which is the real root cause.
ipa-pkinit-manage status is reporting that PKINIT is enabled but this
does not match the actual configuration. This is a known issue [1].
If the host ipa1 is running a CA instance, then you can delete
/var/kerberos/krb5kdc/kdc.{key,crt} and run ipa-pkinit-manage enable to
re-generate the KDC cert. Note: if the host doesn't run a CA instance,
then this won't work because of the issue [2].
To check which hosts are running a CA instance, you can use
# ipa server-role-find --role 'CA server'
[1] https://pagure.io/freeipa/issue/7200
[2] https://pagure.io/freeipa/issue/7795
removed the expired certificates, re-enabled PKINIT, certificates were
regenerated, Web UI logins working again. Thank you very much.
About the errors spotted by ipa-checkcerts.py, what are the certificates
with errors reported? You can find them with ipa cert-show <serial>.
---
$ipa cert-show 57
Issuing CA: ipa
Certificate: [...]
Subject: CN=ipa1.example.com,O=EXAMPLE.COM
Issuer: CN=Certificate Authority,O=EXAMPLE.COM
Not Before: Tue Nov 28 12:42:07 2017 UTC
Not After: Mon Nov 18 12:42:07 2019 UTC
Serial number: 57
Serial number (hex): 0x39
Revoked: False
$ ipa cert-show 268304391
Issuing CA: ipa
Certificate: [...]
Subject: CN=IPA RA,O=EXAMPLE.COM
Issuer: CN=Certificate Authority,O=EXAMPLE.COM
Not Before: Wed Jul 11 14:26:35 2018 UTC
Not After: Tue Jun 30 14:26:35 2020 UTC
Serial number: 268304391
Serial number (hex): 0xFFE0007
Revoked: False
$ ipa cert-show 268304392
Issuing CA: ipa
Certificate: [...]
Subject: CN=CA Subsystem,O=EXAMPLE.COM
Issuer: CN=Certificate Authority,O=EXAMPLE.COM
Not Before: Wed Jul 11 14:26:45 2018 UTC
Not After: Tue Jun 30 14:26:45 2020 UTC
Serial number: 268304392
Serial number (hex): 0xFFE0008
Revoked: False
$ ipa cert-show 268304393
Issuing CA: ipa
Certificate: [...]
Subject: CN=OCSP Subsystem,O=EXAMPLE.COM
Issuer: CN=Certificate Authority,O=EXAMPLE.COM
Not Before: Wed Jul 11 14:27:05 2018 UTC
Not After: Tue Jun 30 14:27:05 2020 UTC
Serial number: 268304393
Serial number (hex): 0xFFE0009
Revoked: False
$ ipa cert-show 268304394
Issuing CA: ipa
Certificate: [...]
Subject: CN=CA Audit,O=EXAMPLE.COM
Issuer: CN=Certificate Authority,O=EXAMPLE.COM
Not Before: Wed Jul 11 14:27:25 2018 UTC
Not After: Tue Jun 30 14:27:25 2020 UTC
Serial number: 268304394
Serial number (hex): 0xFFE000A
Revoked: False
---
ipa cert-show on server ipa2 fails with "ipa: ERROR: Certificate operation
cannot be completed: EXCEPTION (Invalid Credential.)" btw.
The errors related to "Unable to find request for serial xxx" mean that the
cert is tracked by certmonger, but there is no corresponding LDAP entry
cn=xxx,ou=ca,ou=requests,o=ipaca on the local LDAP server.
Is the replication still working between your two masters?
"ipa cert-show" broken on ipa2 often points to an out-of-date certificate in
/var/lib/ipa/ra-agent.{pem|key}. The content should be the same as on the
renewal master, and also the same as in the entry uid=ipara,ou=People,o=ipaca
(which should be replicated and identical on all the masters, except if you
have replication issues).
Replication appeared to be working, however, I did server maintenance
today, ipa1 and ipa2 got upgraded and rebooted. ipa1 works fine, ipa2 does
not come up properly:
--- ipa2
$ ipactl status
Directory Service: RUNNING
krb5kdc Service: STOPPED
kadmin Service: STOPPED
httpd Service: RUNNING
ipa-custodia Service: STOPPED
ntpd Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: STOPPED
ipa: INFO: The ipactl command was successful
---
--- ipa1
$ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial
Serial Number: 268304391 (0xffe0007)
$ ldapsearch -x -b uid=ipara,ou=People,o=ipaca
[...]
description: 2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA
RA,O=EXAMPLE.COM
---
Looks good.
--- ipa2
$ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial
Serial Number: 268304391 (0xffe0007)
$ ldapsearch -x -b uid=ipara,ou=People,o=ipaca
[...]
description: 2;44;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM
---
Looks bad.
--- /var/log/ipaupgrade.log
[...]
2018-12-28T10:55:53Z DEBUG The ipa-server-upgrade command failed, exception:
RemoteRetrieveError: Failed to authenticate to CA REST API
---
--- /var/log/dirsrv/slapd-EXAMPLE-COM/errors
[...]
[28/Dec/2018:11:55:55.559648469 +0100] - ERR - set_krb5_creds - Could not get
initial credentials for principal [ldap/ipa2.example....@example.com] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested
realm)
---
--- ipa-checkcerts.py
[...]
ipa: ERROR: ra.get_certificate(): EXCEPTION (Invalid Credential.)
ra.get_certificate(): EXCEPTION (Invalid Credential.)
ipa: INFO: Checking permissions and ownership
Checking permissions and ownership
ipa: INFO: Failures:
Failures:
ipa: INFO: Unable to find request for serial 268304391
Unable to find request for serial 268304391
ipa: INFO: Unable to find request for serial 268304394
Unable to find request for serial 268304394
ipa: INFO: Unable to find request for serial 268304393
Unable to find request for serial 268304393
ipa: INFO: Unable to find request for serial 268304392
Unable to find request for serial 268304392
ipa: INFO: Unable to find request for serial 268304390
Unable to find request for serial 268304390
ipa: INFO: RA agent description does not match 2;44;CN=Certificate
Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM in LDAP and
2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM
expected
ipa: INFO: RA agent certificate not found in LDAP RA agent certificate not
found in LDAP
[...]
---
The root cause appears to be the wrong IPA RA certificate in ipa2's LDAP,
right? I guess this has to fixed by manually importing the proper
certificate using ldapmodify, similar to the procedure described in [1]?
[1]
https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/
Mit freundlichen Gruessen/With best regards,
--Daniel.
_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
