On Thu, 15 Jun 2017, Rob Crittenden wrote:

Rob Foehl wrote:
Can I at least get a yes or no on whether external CA certificate
renewal has ever been tested when that certificate is nearing expiration?

Yes. I tested this with IPA v3.0. Did it break in between? Possible.

As I pointed out certmonger is unaware of the certificate chain and
focuses only on the cert not-after date and resubmits the CSR to the CA
that issued the certificate originally.

Thanks for the reply.

certmonger not knowing about the chain is understandable, as is the resubmission of each tracked cert to the existing CA. Doing this results in a pile of certs that expire relatively quickly, being tied to the old CA, but that's also not surprising -- the surprise is that it only did that once, and has since appeared to ignore them all, even after the CA was renewed manually and the newly-issued-but-short-lived certs tied to the old CA expired.

I just duplicated last week's result using an earlier snapshot of the
same VM and a renewed CA cert with a 3-day validity.  certmonger ignored
every other cert that it already renewed once with the original CA;
whole system is hosed after the original cert expires.  It's probably
possible to recover by manually replacing every certificate, but I
haven't had time to try that.

certmonger checks at days 28, 7, 3, 2 and 1 before expiration by default
for certificate expiration so it should have looked at the certs at
least two times, three depending on timing (and really, it's seconds
before expiration). Did you let the system sit for 3 days before things
died? Was anything logged to syslog? Moving time forward a day at a time
is insufficient to test this without restarting certmonger.

I let the original VM snapshot run for a month straight, renewing the IPA CA by hand after the first round of certmonger-initiated renewals with 14 days til expiration and on the second attempt after expiration. The first attempt used another 30-day cert, the second used a 3-day and was allowed to run straight through. No time jumps while the VM is running, and all snapshots with the VM powered off, so it always booted with an accurate clock.

certmonger never logged anything after the first renewal cycle on either attempt. A 'getcert list' on the long-running VM shows all of the tracked certificates with an expiration date of 2017-06-24, which matches the lifetime of the renewed CA cert, but none of the services attempting to load or use them are happy.

Now that I poke around some more, here's a wrinkle:

# getcert list -d /etc/httpd/alias -n Server-Cert
Number of certificates and requests being tracked: 8.
Request ID '20170508063315':
        status: MONITORING
        stuck: no
        key pair storage: 
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
Certificate DB'
        CA: IPA
        issuer: CN=Certificate Authority,O=EXAMPLE.COM
        subject: CN=ipa1.example.com,O=EXAMPLE.COM
        expires: 2017-06-24 04:32:24 UTC
        dns: ipa1.example.com
        principal name: HTTP/ipa1.example....@example.com
        key usage: 
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/restart_httpd
        track: yes
        auto-renew: yes

So certmonger thinks it renewed that, and indeed that certificate is now tied to the new IPA CA lifetime *and* was renewed a week after the original IPA CA renewal on May 24 (even though this was never logged):

# certutil -L -n Server-Cert -d /etc/httpd/alias
        Version: 3 (0x2)
        Serial Number: 19 (0x13)
        Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
        Issuer: "CN=Certificate Authority,O=EXAMPLE.COM"
            Not Before: Wed May 31 14:52:53 2017
            Not After : Sat Jun 24 04:32:24 2017
        Subject: "CN=ipa1.example.com,O=EXAMPLE.COM"

But httpd still refuses to start with that NSSDB, and this appears to be why:

# certutil -L -n Signing-Cert -d /etc/httpd/alias
        Version: 3 (0x2)
        Serial Number: 9 (0x9)
        Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
        Issuer: "CN=Certificate Authority,O=EXAMPLE.COM"
            Not Before: Mon May 08 06:33:16 2017
            Not After : Wed Jun 07 06:25:53 2017
        Subject: "CN=Object Signing Cert,O=EXAMPLE.COM"

Does certmonger know how to replace the entire certificate chain in the respective store(s)?

(The third certificate in there, ipaCert / CN=IPA RA, has the same dates as the Server-Cert above.)

Even in a worst-case scenario, where all the certs expire, it is a
fairly straightforward process to get the services back up by going back
in time, renewing the IPA CA then restarting certmonger to renew the
service certificates.

Is it perfect? No. A search of the users forum should make that
apparent. It has been difficult to reproduce the failures because it's
difficult to simulate by moving time around. Several years ago I left
VMs running for months to try to simulate failures and it always worked
for me.

I haven't tried kicking the clock around yet... The second attempt booted from a month-old snapshot and immediately blew itself up; renewing the CA cert and restarting certmonger (really, the whole VM) didn't change anything.

Note too that there is a difference between certmonger and the renewals.
certmonger renews certs but there are helpers that need to fire off to
update information within IPA as well and to distribute updated
certificates to replicas. These scripts were updated significantly since
I wrote them to be much more robust in terms of reliability and logging.

Consider uses of "certmonger" above to include these... Another wrinkle, discovered early on, was broken SELinux policy that prevented certmonger from running any of them. That was (apparently) fixed by a later selinux-policy-targeted package release, but I haven't tried the whole process from a bare install since. The second test with the 3-day lifetime on the IPA CA renewal should've been okay here. I can try again with a fresh install and relatively short IPA CA cert lifetimes, say 4 days per renewal if that'll be sufficient to provoke this a bit faster.

I'm still worried about the missing "phase 2" when it comes to distributing a new external CA certificate -- the CA I have expires in 3 years, and it'd be nice to know whether I'm shooting myself in the foot if I try signing the for-real IPA CA with it now.

FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org

Reply via email to