Roderick Johnstone via FreeIPA-users wrote:
> On 23/01/2018 14:34, Rob Crittenden via FreeIPA-users wrote:
>> Roderick Johnstone via FreeIPA-users wrote:
>>> On 15/01/2018 20:07, Rob Crittenden via FreeIPA-users wrote:
>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>> On 15/01/2018 16:06, Rob Crittenden via FreeIPA-users wrote:
>>>>>> Roderick Johnstone via FreeIPA-users wrote:
>>>>>>> Hi
>>>>>>>
>>>>>>> Our freeipa certificates need to be renewed due to passing their
>>>>>>> expiry
>>>>>>> dates.
>>>>>>>
>>>>>>> While some certificates have renewed ok, the ipaCert and
>>>>>>> auditSigningCert are renewing but the new certificates have the
>>>>>>> wrong
>>>>>>> Subject.
>>>>>>>
>>>>>>> Environment is:
>>>>>>> serverA (CRL, first, master) RHEL 7.3, ipa 4.4
>>>>>>> serverB (replica) RHEL 7.3, ipa 4.4
>>>>>>> serverC (replica) RHEL 7.4, ipa 4.5
>>>>>>>
>>>>>>> Once there are renewed certificates with the wrong Subject present,
>>>>>>> there are various problems with renewing the remaining certificates,
>>>>>>> which I think might be related to the bad Subject:
>>>>>>>
>>>>>>> 1) When just ipaCert has the wrong subject no further renewals
>>>>>>> happen
>>>>>>>
>>>>>>> 2) When auditSigningCert has the wrong subject the ipa pki-tomcatd
>>>>>>> service will not start and no further renewals happen.
>>>>>>>
>>>>>>> I've been round the following loop many times on ServerA, our first
>>>>>>> master:
>>>>>>>
>>>>>>> 1) Restore good certificates from backup
>>>>>>> 2) Put the clock back to a time when certificates are all valid
>>>>>>> 3) Resubmit certificates for renewal
>>>>>>>
>>>>>>> Each time the ipaCert renews it has the same wrong Subject. The
>>>>>>> wrong
>>>>>>> Subject includes the host name of one of our ipa client systems.
>>>>>>>
>>>>>>> Each time the auditSigningCert renews it has the same wrong Subject
>>>>>>> but
>>>>>>> a different subject to the ipaCert. The wrong Subject in this case
>>>>>>> includes the host name of a system which has never been an ipa
>>>>>>> client,
>>>>>>> but might have been added and removed with ipa host-add and ipa
>>>>>>> host-del
>>>>>>> for testing something, a while ago.
>>>>>>>
>>>>>>> As far as I can see, the "cert_subject" is set correctly in the file
>>>>>>> /var/lib/certmonger/<request id> until the point at which the
>>>>>>> certificate is actually renewed.
>>>>>>>
>>>>>>> I'd be very grateful for some pointers as to which configuration
>>>>>>> options
>>>>>>> and logs to check through to resolve this problem on our production
>>>>>>> system.
>>>>>>>
>>>>>>> If its of any relevance we did change which server is the first
>>>>>>> master
>>>>>>> some time ago.
>>>>>>
>>>>>> I'd pull the CSR out of dogtag (CS.cfg) and/or certmonger to see what
>>>>>> the subject is.
>>>>>
>>>>> I'm not seeing any obvious CSR fields in the
>>>>> /etc/pki/pki-tomcat/ca/CS.cfg file.
>>>>
>>>> foo.bar.certreq=
>>>>
>>>>> The CSR in the certmonger requests file for the auditSigningCert seems
>>>>> to be showing with the correct Subject. This is different from the bad
>>>>> subject showing in the requests file field:
>>>>> cert_subject=
>>>>
>>>> The value of cert_subject comes from the issued certificate.
>>>>
>>>>> and the Subject which is showing in the 'getcert list' output
>>>>> (which is
>>>>> the same as that in the cert_subject= field.>
>>>>> I'm not quite sure what this all means.
>>>>
>>>> It is displayed from the data within the tracked certmonger request.
>>>>
>>>> certmonger logs to syslog so you can check there or you can stop the
>>>> process and run it manually with: certmonger -n -d 9 2>&1 | tee
>>>> certmonger.log
>>>>
>>>> That will provide a lot of debugging output that may show what is
>>>> going on.
>>>
>>> I've restored certificate databases from backup and put the clock back
>>> to a time when certificates are valid and renewed the ocspSigining
>>> certificate with:
>>> getcert resubmit -N "CN=OCSP Subsystem,O=<REALM>" -i 20161124081302
>>>
>>> (I've previously tried without the -N with similar results)
>>>
>>> What I am seeing in the certmonger logs is:
>>>
>>>
>>> 2017-10-23 00:05:28 [438] Located the key 'ocspSigningCert cert-pki-ca'.
>>> 2017-10-23 00:05:28 [438] Converted private key 'ocspSigningCert
>>> cert-pki-ca' to public key.
>>> 2017-10-23 00:05:28 [439] Located the certificate "ocspSigningCert
>>> cert-pki-ca".
>>> 2017-10-23 00:05:28 [440] 0x1d Certificate named "ocspSigningCert
>>> cert-pki-ca" in token "NSS Certificate DB" in database
>>> "/etc/pki/pki-tomcat/alias" will not be valid after 20171025122401.
>>> 2017-10-23 00:05:28 [442] Located the key 'ocspSigningCert cert-pki-ca'.
>>> 2017-10-23 00:05:28 [442] Converted private key 'ocspSigningCert
>>> cert-pki-ca' to public key.
>>> 2017-10-23 00:05:28 [443] Located the certificate "ocspSigningCert
>>> cert-pki-ca".
>>> 2017-10-23 00:05:28 [444] Located the key 'ocspSigningCert cert-pki-ca'.
>>> 2017-10-23 00:05:28 [444] Converted private key 'ocspSigningCert
>>> cert-pki-ca' to public key.
>>> 2017-10-23 00:05:39 [581] Found a certificate with the same nickname but
>>> different subject, removing certificate "ocspSigningCert cert-pki-ca"
>>> with subject "CN=OCSP Subsystem,O=<REALM>".
>>> 2017-10-23 00:05:39 [581] Imported certificate "ocspSigningCert
>>> cert-pki-ca", got nickname "ocspSigningCert cert-pki-ca".
>>> 2017-10-23 00:05:39 [583] Located the certificate "ocspSigningCert
>>> cert-pki-ca".
>>> 2017-10-23 00:05:39 [48576] Adding hook
>>> "/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert
>>> cert-pki-ca"" (0).
>>> 2017-10-23 00:10:43 [942] 0x1d Certificate named "ocspSigningCert
>>> cert-pki-ca" in token "NSS Certificate DB" in database
>>> "/etc/pki/pki-tomcat/alias" issued by CA and saved.
>>>
>>> I now have a date valid ocspSigningCertificate, but with the wrong
>>> subject, and a broken certificate system which will no longer start.
>>>
>>> ipactl status
>>> ...
>>> pki-tomcatd Service: STOPPED
>>>
>>> I can't renew other expired certificates.
>>>
>>> I also note that there is now no key for ocspSigningCertificate as shown
>>> by:
>>> certutil -K -d /etc/pki/pki-tomcat/alias
>>>
>>> I wonder if this is because the certificate subject changed? There was a
>>> key before the certificate renewed.
>>>
>>> The ca debug logs are showing:
>>>
>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: Found cert by nickname:
>>> 'ocspSigningCert cert-pki-ca' with serial number: 268370108
>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: converted to x509CertImpl
>>> [23/Oct/2017:00:55:47][localhost-startStop-1]: SigningUnit: Certificate
>>> object not found
>>> Certificate object not found
>>>      at com.netscape.ca.SigningUnit.init(SigningUnit.java:184)
>>>
>>> Any help in repairing my broken ipa system will be much appreciated.
>>
>> Sorry for the delay. I think my previous reading of the certmonger
>> csrgen code was wrong.
>>
>> IIRC in your certmonger entry the cert_subject has the hostname value
>> right? Do you also have cert_subject_der?
>>
>> You can decode that by:
>>
>> 1. Running a hex-to-bin, something hacky like this in python:
>>
>> import binascii
>>
>> hex_string = "<hex value>"
>>
>> binary_string = binascii.unhexlify(hex_string)
>>
>> fd = open("out", "wb")
>> fd.write(binary_string)
>> fd.close()
>>
>> 2. Run: openssl asn1parse -in out -inform der
>>
>> I'm going to assume this also has the hostname encoded in it.
> 
> Hi Again
> 
> Yes, correct. The cert_subject_der does have the bad CN with the
> hostname encoded in it.
> 
>>
>> Can you try this:
>>
>> 1. Make a backup of the request file (just in case) > 2. Remove
>> cert_subject_der
>> 3. Modify cert_subject to be CN=OCSP Subsystem,O=<YOUR_REALM>
>> 3. Try the renewal again
> 
> I ran though all of this, checking that the request file was still what
> we wanted after restarting certmonger with verbose logging, restoring
> all the databases, fixing permissions, checking keys etc., and ran the
> getcert resubmit.
> 
> It still generates a certificate with the bad CN including a hostname.
> 
> Then the pki-tomcat server fails to start again.
> 
> Other things I have checked are:
> 1) The csr= field in the (modified) request file has the correct subject.
> 
> 2) The dogtag ca debug log file is showing:
> processRenewal: origNotAfter =Wed Oct 25 13:24:01 BST 2017
> processRenewal: orig subj dn =CN=OCSP Subsystem,O=AST.CAM.AC.UK
> ...
> RenewalSubmitter: renewal original profileId=caIPAserviceCert
> RenewalSubmitter: for renewal, original authenticator raCertAuth found
> CertProcessor: No authenticator credentials required
> processRenewal: set original Inputs into profile Context
> RenewalSubmitter: setInputsIntoContext() getting input name=
> cert_request_type
> RenewalSubmitter: setInputsIntoContext() setting value in ctx:pkcs10
> RenewalSubmitter: setInputsIntoContext() getting input name= cert_request
> RenewalSubmitter: setInputsIntoContext() setting value in ctx:<hex
> encoded csr>
> ...
> Finish parsePKCS10 - CN=<HOSTNAME>,O=<REALM>
> 
> The hex encoded csr in the last line decodes to have the bad subject
> where the hostname is one of our other ipa servers.
> 
> The certificate always getthe same hostname in the bad subject cn.
> 
> Do you have any other ideas of things to check to find out what's
> generating the bad subject?

The order certmonger uses for the subject is:

1. cert_subject_der
2. If there is no DER, try to encode cert_subject
3. If that fails try again a different way
4. If it fails yet again use CN=localhost

It is baffling that it would pick a hostname much less one that
certmonger shouldn't even know about.

I assume the CSR found in the dogtag logs matches the csr value in the
certmonger request?

Can you share the certmonger request file privately?

rob
_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org

Reply via email to