Roderick Johnstone via FreeIPA-users wrote:
> On 24/01/2018 15:22, Rob Crittenden via FreeIPA-users wrote:
>> Roderick Johnstone via FreeIPA-users wrote:
>>> On 23/01/2018 14:34, Rob Crittenden via FreeIPA-users wrote:
>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>> On 15/01/2018 20:07, Rob Crittenden via FreeIPA-users wrote:
>>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>>> On 15/01/2018 16:06, Rob Crittenden via FreeIPA-users wrote:
>>>>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>> Our freeipa certificates need to be renewed due to passing their
>>>>>>>>> expiry
>>>>>>>>> dates.
>>>>>>>>>
>>>>>>>>> While some certificates have renewed ok, the ipaCert and
>>>>>>>>> auditSigningCert are renewing but the new certificates have the
>>>>>>>>> wrong
>>>>>>>>> Subject.
>>>>>>>>>
>>>>>>>>> Environment is:
>>>>>>>>> serverA (CRL, first, master) RHEL 7.3, ipa 4.4
>>>>>>>>> serverB (replica) RHEL 7.3, ipa 4.4
>>>>>>>>> serverC (replica) RHEL 7.4, ipa 4.5
>>>>>>>>>
>>>>>>>>> Once there are renewed certificates with the wrong Subject
>>>>>>>>> present,
>>>>>>>>> there are various problems with renewing the remaining
>>>>>>>>> certificates,
>>>>>>>>> which I think might be related to the bad Subject:
>>>>>>>>>
>>>>>>>>> 1) When just ipaCert has the wrong subject no further renewals
>>>>>>>>> happen
>>>>>>>>>
>>>>>>>>> 2) When auditSigningCert has the wrong subject the ipa pki-tomcatd
>>>>>>>>> service will not start and no further renewals happen.
>>>>>>>>>
>>>>>>>>> I've been round the following loop many times on ServerA, our
>>>>>>>>> first
>>>>>>>>> master:
>>>>>>>>>
>>>>>>>>> 1) Restore good certificates from backup
>>>>>>>>> 2) Put the clock back to a time when certificates are all valid
>>>>>>>>> 3) Resubmit certificates for renewal
>>>>>>>>>
>>>>>>>>> Each time the ipaCert renews it has the same wrong Subject. The
>>>>>>>>> wrong
>>>>>>>>> Subject includes the host name of one of our ipa client systems.
>>>>>>>>>
>>>>>>>>> Each time the auditSigningCert renews it has the same wrong
>>>>>>>>> Subject
>>>>>>>>> but
>>>>>>>>> a different subject to the ipaCert. The wrong Subject in this case
>>>>>>>>> includes the host name of a system which has never been an ipa
>>>>>>>>> client,
>>>>>>>>> but might have been added and removed with ipa host-add and ipa
>>>>>>>>> host-del
>>>>>>>>> for testing something, a while ago.
>>>>>>>>>
>>>>>>>>> As far as I can see, the "cert_subject" is set correctly in the
>>>>>>>>> file
>>>>>>>>> /var/lib/certmonger/<request id> until the point at which the
>>>>>>>>> certificate is actually renewed.
>>>>>>>>>
>>>>>>>>> I'd be very grateful for some pointers as to which configuration
>>>>>>>>> options
>>>>>>>>> and logs to check through to resolve this problem on our
>>>>>>>>> production
>>>>>>>>> system.
>>>>>>>>>
>>>>>>>>> If its of any relevance we did change which server is the first
>>>>>>>>> master
>>>>>>>>> some time ago.
>>>>>>>>
>>>>>>>> I'd pull the CSR out of dogtag (CS.cfg) and/or certmonger to see
>>>>>>>> what
>>>>>>>> the subject is.
>>>>>>>
>>>>>>> I'm not seeing any obvious CSR fields in the
>>>>>>> /etc/pki/pki-tomcat/ca/CS.cfg file.
>>>>>>
>>>>>> foo.bar.certreq=
>>>>>>
>>>>>>> The CSR in the certmonger requests file for the auditSigningCert
>>>>>>> seems
>>>>>>> to be showing with the correct Subject. This is different from
>>>>>>> the bad
>>>>>>> subject showing in the requests file field:
>>>>>>> cert_subject=
>>>>>>
>>>>>> The value of cert_subject comes from the issued certificate.
>>>>>>
>>>>>>> and the Subject which is showing in the 'getcert list' output
>>>>>>> (which is
>>>>>>> the same as that in the cert_subject= field.>
>>>>>>> I'm not quite sure what this all means.
>>>>>>
>>>>>> It is displayed from the data within the tracked certmonger request.
>>>>>>
>>>>>> certmonger logs to syslog so you can check there or you can stop the
>>>>>> process and run it manually with: certmonger -n -d 9 2>&1 | tee
>>>>>> certmonger.log
>>>>>>
>>>>>> That will provide a lot of debugging output that may show what is
>>>>>> going on.
>>>>>
>>>>> I've restored certificate databases from backup and put the clock back
>>>>> to a time when certificates are valid and renewed the ocspSigining
>>>>> certificate with:
>>>>> getcert resubmit -N "CN=OCSP Subsystem,O=<REALM>" -i 20161124081302
>>>>>
>>>>> (I've previously tried without the -N with similar results)
>>>>>
>>>>> What I am seeing in the certmonger logs is:
>>>>>
>>>>>
>>>>> 2017-10-23 00:05:28 [438] Located the key 'ocspSigningCert
>>>>> cert-pki-ca'.
>>>>> 2017-10-23 00:05:28 [438] Converted private key 'ocspSigningCert
>>>>> cert-pki-ca' to public key.
>>>>> 2017-10-23 00:05:28 [439] Located the certificate "ocspSigningCert
>>>>> cert-pki-ca".
>>>>> 2017-10-23 00:05:28 [440] 0x1d Certificate named "ocspSigningCert
>>>>> cert-pki-ca" in token "NSS Certificate DB" in database
>>>>> "/etc/pki/pki-tomcat/alias" will not be valid after 20171025122401.
>>>>> 2017-10-23 00:05:28 [442] Located the key 'ocspSigningCert
>>>>> cert-pki-ca'.
>>>>> 2017-10-23 00:05:28 [442] Converted private key 'ocspSigningCert
>>>>> cert-pki-ca' to public key.
>>>>> 2017-10-23 00:05:28 [443] Located the certificate "ocspSigningCert
>>>>> cert-pki-ca".
>>>>> 2017-10-23 00:05:28 [444] Located the key 'ocspSigningCert
>>>>> cert-pki-ca'.
>>>>> 2017-10-23 00:05:28 [444] Converted private key 'ocspSigningCert
>>>>> cert-pki-ca' to public key.
>>>>> 2017-10-23 00:05:39 [581] Found a certificate with the same
>>>>> nickname but
>>>>> different subject, removing certificate "ocspSigningCert cert-pki-ca"
>>>>> with subject "CN=OCSP Subsystem,O=<REALM>".
>>>>> 2017-10-23 00:05:39 [581] Imported certificate "ocspSigningCert
>>>>> cert-pki-ca", got nickname "ocspSigningCert cert-pki-ca".
>>>>> 2017-10-23 00:05:39 [583] Located the certificate "ocspSigningCert
>>>>> cert-pki-ca".
>>>>> 2017-10-23 00:05:39 [48576] Adding hook
>>>>> "/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert
>>>>> cert-pki-ca"" (0).
>>>>> 2017-10-23 00:10:43 [942] 0x1d Certificate named "ocspSigningCert
>>>>> cert-pki-ca" in token "NSS Certificate DB" in database
>>>>> "/etc/pki/pki-tomcat/alias" issued by CA and saved.
>>>>>
>>>>> I now have a date valid ocspSigningCertificate, but with the wrong
>>>>> subject, and a broken certificate system which will no longer start.
>>>>>
>>>>> ipactl status
>>>>> ...
>>>>> pki-tomcatd Service: STOPPED
>>>>>
>>>>> I can't renew other expired certificates.
>>>>>
>>>>> I also note that there is now no key for ocspSigningCertificate as
>>>>> shown
>>>>> by:
>>>>> certutil -K -d /etc/pki/pki-tomcat/alias
>>>>>
>>>>> I wonder if this is because the certificate subject changed? There
>>>>> was a
>>>>> key before the certificate renewed.
>>>>>
>>>>> The ca debug logs are showing:
>>>>>
>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: Found cert by nickname:
>>>>> 'ocspSigningCert cert-pki-ca' with serial number: 268370108
>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: converted to
>>>>> x509CertImpl
>>>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: SigningUnit:
>>>>> Certificate
>>>>> object not found
>>>>> Certificate object not found
>>>>>       at com.netscape.ca.SigningUnit.init(SigningUnit.java:184)
>>>>>
>>>>> Any help in repairing my broken ipa system will be much appreciated.
>>>>
>>>> Sorry for the delay. I think my previous reading of the certmonger
>>>> csrgen code was wrong.
>>>>
>>>> IIRC in your certmonger entry the cert_subject has the hostname value
>>>> right? Do you also have cert_subject_der?
>>>>
>>>> You can decode that by:
>>>>
>>>> 1. Running a hex-to-bin, something hacky like this in python:
>>>>
>>>> import binascii
>>>>
>>>> hex_string = "<hex value>"
>>>>
>>>> binary_string = binascii.unhexlify(hex_string)
>>>>
>>>> fd = open("out", "wb")
>>>> fd.write(binary_string)
>>>> fd.close()
>>>>
>>>> 2. Run: openssl asn1parse -in out -inform der
>>>>
>>>> I'm going to assume this also has the hostname encoded in it.
>>>
>>> Hi Again
>>>
>>> Yes, correct. The cert_subject_der does have the bad CN with the
>>> hostname encoded in it.
>>>
>>>>
>>>> Can you try this:
>>>>
>>>> 1. Make a backup of the request file (just in case) > 2. Remove
>>>> cert_subject_der
>>>> 3. Modify cert_subject to be CN=OCSP Subsystem,O=<YOUR_REALM>
>>>> 3. Try the renewal again
>>>
>>> I ran though all of this, checking that the request file was still what
>>> we wanted after restarting certmonger with verbose logging, restoring
>>> all the databases, fixing permissions, checking keys etc., and ran the
>>> getcert resubmit.
>>>
>>> It still generates a certificate with the bad CN including a hostname.
>>>
>>> Then the pki-tomcat server fails to start again.
>>>
>>> Other things I have checked are:
>>> 1) The csr= field in the (modified) request file has the correct
>>> subject.
>>>
>>> 2) The dogtag ca debug log file is showing:
>>> processRenewal: origNotAfter =Wed Oct 25 13:24:01 BST 2017
>>> processRenewal: orig subj dn =CN=OCSP Subsystem,O=AST.CAM.AC.UK
>>> ...
>>> RenewalSubmitter: renewal original profileId=caIPAserviceCert
>>> RenewalSubmitter: for renewal, original authenticator raCertAuth found
>>> CertProcessor: No authenticator credentials required
>>> processRenewal: set original Inputs into profile Context
>>> RenewalSubmitter: setInputsIntoContext() getting input name=
>>> cert_request_type
>>> RenewalSubmitter: setInputsIntoContext() setting value in ctx:pkcs10
>>> RenewalSubmitter: setInputsIntoContext() getting input name=
>>> cert_request
>>> RenewalSubmitter: setInputsIntoContext() setting value in ctx:<hex
>>> encoded csr>
>>> ...
>>> Finish parsePKCS10 - CN=<HOSTNAME>,O=<REALM>
>>>
>>> The hex encoded csr in the last line decodes to have the bad subject
>>> where the hostname is one of our other ipa servers.
>>>
>>> The certificate always getthe same hostname in the bad subject cn.
>>>
>>> Do you have any other ideas of things to check to find out what's
>>> generating the bad subject?
>>
>> The order certmonger uses for the subject is:
>>
>> 1. cert_subject_der
>> 2. If there is no DER, try to encode cert_subject
>> 3. If that fails try again a different way
>> 4. If it fails yet again use CN=localhost
>>
>> It is baffling that it would pick a hostname much less one that
>> certmonger shouldn't even know about.
> 
> Indeed, and if I try to renew other certificates for some of them it
> chooses other host names that should not be known about. Each
> certificate seems to get the same bad hostname each time I try to renew.
> 
>>
>> I assume the CSR found in the dogtag logs matches the csr value in the
>> certmonger request?
> 
> No, its different. The issued certificate matches the csr seen in dogtag
> which makes sense. But the csr in the dogtag logs has the bad subject.
> The csr in the request directory file has the good subject.
> 
>>
>> Can you share the certmonger request file privately?
> 
> Yes, certainly.
> 
> Thanks for your continued help on this.

Let's try this. We'll drop the current request and create a new one.

Stop tomcat, restore the old cert database, start tomcat, then:

# getcert stop-tracking -i <request id>
# getcert start-tracking -c dogtag-ipa-ca-renew-agent -n
"ocspSigningCert cert-pki-ca" -d /etc/pki/pki-tomcat/alias -P <pin> -B
/usr/libexec/ipa/certmonger/stop_pkicad -C
'/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"'

Then try resubmitting the request.

rob
_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org

Reply via email to