On 15/01/2018 20:07, Rob Crittenden via FreeIPA-users wrote:
Roderick Johnstone via FreeIPA-users wrote:
On 15/01/2018 16:06, Rob Crittenden via FreeIPA-users wrote:
Roderick Johnstone via FreeIPA-users wrote:
Hi
Our freeipa certificates need to be renewed due to passing their expiry
dates.
While some certificates have renewed ok, the ipaCert and
auditSigningCert are renewing but the new certificates have the wrong
Subject.
Environment is:
serverA (CRL, first, master) RHEL 7.3, ipa 4.4
serverB (replica) RHEL 7.3, ipa 4.4
serverC (replica) RHEL 7.4, ipa 4.5
Once there are renewed certificates with the wrong Subject present,
there are various problems with renewing the remaining certificates,
which I think might be related to the bad Subject:
1) When just ipaCert has the wrong subject no further renewals happen
2) When auditSigningCert has the wrong subject the ipa pki-tomcatd
service will not start and no further renewals happen.
I've been round the following loop many times on ServerA, our first
master:
1) Restore good certificates from backup
2) Put the clock back to a time when certificates are all valid
3) Resubmit certificates for renewal
Each time the ipaCert renews it has the same wrong Subject. The wrong
Subject includes the host name of one of our ipa client systems.
Each time the auditSigningCert renews it has the same wrong Subject but
a different subject to the ipaCert. The wrong Subject in this case
includes the host name of a system which has never been an ipa client,
but might have been added and removed with ipa host-add and ipa host-del
for testing something, a while ago.
As far as I can see, the "cert_subject" is set correctly in the file
/var/lib/certmonger/<request id> until the point at which the
certificate is actually renewed.
I'd be very grateful for some pointers as to which configuration options
and logs to check through to resolve this problem on our production
system.
If its of any relevance we did change which server is the first master
some time ago.
I'd pull the CSR out of dogtag (CS.cfg) and/or certmonger to see what
the subject is.
I'm not seeing any obvious CSR fields in the
/etc/pki/pki-tomcat/ca/CS.cfg file.
foo.bar.certreq=
Thanks for the hint (and for your input in general).
I'm seeing only the following four certreq entries in CS.cfg, nothing
obvious for ca.audit_signing:
$ pwd
/etc/pki/pki-tomcat/ca
$ ls -l CS.cfg
-rw-rw---- 1 pkiuser pkiuser 82417 Oct 24 12:00 CS.cfg
$ grep certreq CS.cfg | awk -F= '{print $1}'
ca.ocsp_signing.certreq
ca.signing.certreq
ca.sslserver.certreq
ca.subsystem.certreq
Interestingly the CS.cfg file was written at just the time I have been
putting the clock back to for certificate renewal purposes.
If I look at a backup of the CS.cfg file that I have I see all certreq
as expected:
$ pwd
/var/tmp/rmj/etc/pki/pki-tomcat/ca
$ ls -l CS.cfg
-rw-rw---- 1 pkiuser pkiuser 83015 Aug 18 22:43 CS.cfg
$ grep certreq CS.cfg | awk -F= '{print $1}'
ca.audit_signing.certreq
ca.ocsp_signing.certreq
ca.signing.certreq
ca.sslserver.certreq
ca.subsystem.certreq
Here are the results of checking the CSRs in both the cetmonger requests
and CS.cfg locations in the live system:
The CSRs in the CS.cfg file show:
ca.ocsp_signing.certreq
Subject: O=<my domain in caps>, CN=OCSP Subsystem
ca.signing.certreq
Subject: O=<my domain in caps>, CN=Certificate Authority
ca.sslserver.certreq
Subject: O=<my domain in caps>, CN=<hostname of first master>
ca.subsystem.certreq
Subject: O=<my domain in caps>, CN=CA Subsystem
The CSRs in certmonger requests show:
auditSigningCert_certmongerrequest_csr.lis
Subject: O=<my domain in caps>, CN=CA Audit
ipaCert_certmongerrequest_csr.lis
Subject: O=<my domain in caps>, CN=IPA RA
ocspSigningCert_certmongerrequest_csr.lis
Subject: O=<my domain in caps>, CN=OCSP Subsystem
Server-Cert_certmongerrequest_csr.lis
Subject: O=<my domain in caps>, CN=<hostname of first master>
subsystemCert_certmongerrequest_csr.lis
Subject: O=<my domain in caps>, CN=CA Subsystem
So those look ok, except that there is no entry for
ca.audit_signing.certreq in the live CS.cfg file.
The auditSigningCert in the backup of the CS.cfg looks to be correct:
Subject: O=<my domain in caps>, CN=CA Audit
The getcert list output that shows the bad subject for current
auditSigningCert:
$ getcert list | egrep "certificate:|subject|expires"
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB'
subject: CN=<hostname of system never a freeipa client>,O=<my domain in
caps>
expires: 2019-10-25 11:19:50 UTC
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS Certificate DB'
subject: CN=OCSP Subsystem,O=<my domain in caps>
expires: 2017-10-25 12:24:01 UTC
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB'
subject: CN=CA Subsystem,O=<my domain in caps>
expires: 2017-10-25 12:24:02 UTC
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
cert-pki-ca',token='NSS Certificate DB'
subject: CN=Certificate Authority,O=<my domain in caps>
expires: 2035-11-05 12:24:00 UTC
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB'
subject: CN=<hostname of first master>,O=<my domain in caps>
expires: 2017-10-26 12:22:11 UTC
certificate:
type=NSSDB,location='/etc/dirsrv/slapd-AST-CAM-AC-UK',nickname='Server-Cert',token='NSS
Certificate DB'
subject: CN=<hostname of first master>,O=<my domain in caps>
expires: 2019-10-23 23:04:22 UTC
certificate:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
Certificate DB'
subject: CN=<hostname of first master>,O=<my domain in caps>
expires: 2019-10-25 08:42:33 UTC
certificate:
type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS
Certificate DB'
subject: CN=IPA RA,O=<my domain in caps>
expires: 2017-10-25 12:24:23 UTC
Originally I was working on the theory that the bad Subject in the
auditSigningCert was what was blocking the server from issuing any more
certificates, but I have now tried a few things and done quite a bit of
digging and found some other issues, so I'd like to recap what I have
done and what the situation is.
Everything I'm looking at is on the first master server. I also have
certificates which have expired on the replicas but I understand that I
need to fix the first master first.
I restored certificate databases in /etc/httpd/aliases and
/etc/pki/pki-tomcat/alias from a good backup, showing correct subjects
for all certificates, but with various certificates expiring between 23
Oct 2017 and 5 Nov 2017.
I managed to renew the /etc/httpd/alias Server-Cert ok.
The /etc/httpd/alias ipaCert renewed with the bad subject:
CN=<hostname of a freeipa client>,O=<my domain in caps>
I replaced the ipaCert with the good backup, expiring 25 Oct 2017.
I put the clock back to 24 Oct 2017 12:00:00 when all the certificates
were valid.
I restarted certmonger and the auditSigningCert was renewed but with
subject: CN=<hostname of system that was never a freeipa client>,O=<my
domain in caps>
At this point the freeipa pki-tomcat service will not start.
In fact it seems that the pki-tomcat will not start at all or produce
any logs until I do:
pki-server subsystem-enable ca
(it had become disabled).
Now when I check the /var/log/pki/pki-tomcatd/ca/selftest.log file I see:
0.localhost-startStop-1 - [24/Oct/2017:12:00:37 BST] [20] [1]
SystemCertsVerification: system certs verification failure: Certificate
auditSigningCert cert-pki-ca is invalid: Invalid certificate: (-8101)
Certificate type not approved for application.
0.localhost-startStop-1 - [24/Oct/2017:12:00:37 BST] [20] [1]
SelfTestSubsystem: The CRITICAL self test plugin called
selftests.container.instance.SystemCertsVerification running at startup
FAILED!
Can you elaborate a bit on what "Certificate type not approved for
application" might mean in this context please?
When I check the certificate keys in the database
/etc/pki/pki-tomcat/alias I see:
$ certutil -K -d .
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private
Key and Certificate Services"
Enter Password or Pin for "NSS Certificate DB":
< 0> rsa hex-numbers (orphan)
< 1> rsa hex-numbers caSigningCert cert-pki-ca
< 2> rsa hex-numbers ocspSigningCert cert-pki-ca
< 3> rsa hex-numbers subsystemCert cert-pki-ca
< 4> rsa hex-numbers Server-Cert cert-pki-ca
So something has gone wrong here too as there is no entry for
auditSigningCert and there is an orphan key.
I've checked the backup I have of the /etc/pki/pki-tomcat/alias database
files and it shows more correctly that there is a key for auditSigningCert:
$ pwd
/var/tmp/rmj/etc/pki/pki-tomcat/alias
$ certutil -K -d .
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private
Key and Certificate Services"
Enter Password or Pin for "NSS Certificate DB":
< 0> rsa hex-numbers (orphan)
< 1> rsa hex-numbers caSigningCert cert-pki-ca
< 2> rsa hex-numbers auditSigningCert cert-pki-ca
< 3> rsa hex-numbers ocspSigningCert cert-pki-ca
< 4> rsa hex-numbers subsystemCert cert-pki-ca
< 5> rsa hex-numbers Server-Cert cert-pki-ca
So it looks like something in the renewal process has removed the key
for auditSigningCert.
So, I'm not sure whether the bad subject in the renewed auditSigningCert
certificate is causing all the breakage or whether thats a symptom of
something else thats wrong.
Any idea which of these is more likely?
The CSR in the certmonger requests file for the auditSigningCert seems
to be showing with the correct Subject. This is different from the bad
subject showing in the requests file field:
cert_subject=
The value of cert_subject comes from the issued certificate.
and the Subject which is showing in the 'getcert list' output (which is
the same as that in the cert_subject= field.>
I'm not quite sure what this all means.
It is displayed from the data within the tracked certmonger request.
Thanks for the info.
certmonger logs to syslog so you can check there or you can stop the
process and run it manually with: certmonger -n -d 9 2>&1 | tee
certmonger.log
That will provide a lot of debugging output that may show what is going on.
So, what I'm planning now is:
1) restore the /etc/pki/pki-tomcatd/alias database files from backup
2) restore the CS.cfg file from backup
3) make sure the clock is at a time when all certificates are valid
4) Stop certmonger systemd service
5) Start cermonger with the command you gave above
6) run ipactl start and check pki-tomcatd starts ok
7) resubmit the auditSigningCert with:
getcert resubmit id <request>
8) Check the certmonger.log
Does that sound like a reasonable way forward at this point?
Anything else worth checking at this time?
Roderick
rob
_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org
_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org
