[Freeipa-users] Re: Replica not renewing IPA certificates
Hello Roderick, Would you care to confirm that you indeed ran "getcert resubmit" on the replica (the non-renewal master)? I'm in the same situation as you were, and I'm reluctant to run commands that could potentially make things worse. -- Kees On 31-01-2020 16:04, Roderick Johnstone via FreeIPA-users wrote: > On 31/01/2020 13:25, Florence Blanc-Renaud wrote: >> On 1/31/20 2:03 PM, Roderick Johnstone via FreeIPA-users wrote: >>> Hi >>> >>> This is freeipa (ipa-server-4.6.5-11.el7_7.3.x86_64) on RHEL7 with >>> freeipa's own internal CA. >>> >>> One of my ipa server replicas (host3) has not renewed its IPA system >>> certificates and is now showing >>> ca-error: Invalid cookie: u'' >>> in the 'getcert list' output for certificates: >>> "auditSigningCert cert-pki-ca", "ocspSigningCert cert-pki-ca", >>> "subsystemCert cert-pki-ca", and the >>> certificate in the file /var/lib/ipa/ra-agent.pem >>> >>> As far as I can see, the sequence of events has been as follows: >>> >>> host3 noticed the certificates needed renewing at 30 Jan 2020 05:37 >>> and certmonger initiated a renewal. >>> >>> The state of those certificates went from MONITORING to CA_WORKING but >>> the certificates were not renewed. >>> >>> The CA renewal master (host1) noticed its same set of certificates >>> (plus "Server-Cert cert-pki-ca") needed renewing at 30 Jan 2020 07:28 >>> and renewed them successfully. >>> >>> Another replica (host2) noticed that its certificates needed renewing >>> at 30 Jan 2020 07:32 and renewed them successfully. >>> >>> At 30 Jan 13:37 on host3 the certificates needing to be renewed went >>> from CA_WORKING back to MONITORING, but 'getcert list' now shows them >>> with: >>> ca-error: Invalid cookie: u'' >>> and they still haven't renewed. >>> >> Hi >> the 'Invalid cookie' error message is an issue already tracked in ticket >> 8164 Renewed certs are not picked up by IPA CAs [1]. >> >> When a replica tries to renew a cert before the renewal master, the >> output of getcert list should be 'CA-WORKING' and certmonger should >> retry 8 hours laters (see the code in [2]). >> >> Since you are hitting the issue 8164, you can manually force the renewal >> on the replica (once the CA renewal master has actually renewed the >> cert) with getcert resubmit. >> >> HTH, >> flo > > Hi Flo > > Thank you very much! The getcert resubmit has successfully renewed all > the certificates in need of renewal. > > The comments from Rob on the commit to fix this issue are very helpful > in understanding what is happening too. > > Roderick > >> >> [1] https://pagure.io/freeipa/issue/8164 >> [2] >> https://pagure.io/freeipa/blob/b5b9efeb57c010443c33c6f14f831abdbd804e78/f/install/certmonger/dogtag-ipa-ca-renew-agent-submit.in#_370 >> >> >>> I haven't seen certmonger attempt to try the renewal again on host3 >>> (nothing from certmonger in /var/log/messages since 30 Jan 13:37). >>> >>> While I could try a getcert resubmit on host3 to force it to try >>> again, I'd like to know if what I am seeing is the expected behaviour >>> when a replica tried to renew certificates before the renewal master. >>> >>> How long should I have to wait till certmonger on host3 tries again? - >>> I couldn't find any reference to how often certmonger tries the renewal. >>> >>> Rob Crittenden's freeipa-healthcheck script is now showing the >>> following for host3: >>> >>> ERROR: ipahealthcheck.ipa.certs.IPARAAgent: RA agent description does >>> not match 2;16;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA >>> RA,O=EXAMPLE.COM in LDAP and 2;7;CN=Certificate >>> Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM expected >>> ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20180926040924: >>> Request for certificate failed, Certificate operation cannot be >>> completed: EXCEPTION (Invalid Credential.) >>> ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20180926040920: >>> Request for certificate failed, Certificate operation cannot be >>> completed: EXCEPTION (Invalid Credential.) >>> ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20180926040921: >>> Request for certificate failed, Certificate operation cannot be >>> completed: EXCEPTION (Invalid Credential.) >>> ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20180926040922: >>> Request for certificate failed, Certificate operation cannot be >>> completed: EXCEPTION (Invalid Credential.) >>> ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20180926040923: >>> Request for certificate failed, Certificate operation cannot be >>> completed: EXCEPTION (Invalid Credential.) >>> ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20180926040925: >>> Request for certificate failed, Certificate operation cannot be >>> completed: EXCEPTION (Invalid Credential.) >>> ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20180926040927: >>> Request for certificate failed, Certificate operation cannot be >>> completed: EXCEPTION (Invalid Credential.) >>> ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20180926040926: >>> Request
[Freeipa-users] Re: Replica not renewing IPA certificates
On 31/01/2020 13:25, Florence Blanc-Renaud wrote: On 1/31/20 2:03 PM, Roderick Johnstone via FreeIPA-users wrote: Hi This is freeipa (ipa-server-4.6.5-11.el7_7.3.x86_64) on RHEL7 with freeipa's own internal CA. One of my ipa server replicas (host3) has not renewed its IPA system certificates and is now showing ca-error: Invalid cookie: u'' in the 'getcert list' output for certificates: "auditSigningCert cert-pki-ca", "ocspSigningCert cert-pki-ca", "subsystemCert cert-pki-ca", and the certificate in the file /var/lib/ipa/ra-agent.pem As far as I can see, the sequence of events has been as follows: host3 noticed the certificates needed renewing at 30 Jan 2020 05:37 and certmonger initiated a renewal. The state of those certificates went from MONITORING to CA_WORKING but the certificates were not renewed. The CA renewal master (host1) noticed its same set of certificates (plus "Server-Cert cert-pki-ca") needed renewing at 30 Jan 2020 07:28 and renewed them successfully. Another replica (host2) noticed that its certificates needed renewing at 30 Jan 2020 07:32 and renewed them successfully. At 30 Jan 13:37 on host3 the certificates needing to be renewed went from CA_WORKING back to MONITORING, but 'getcert list' now shows them with: ca-error: Invalid cookie: u'' and they still haven't renewed. Hi the 'Invalid cookie' error message is an issue already tracked in ticket 8164 Renewed certs are not picked up by IPA CAs [1]. When a replica tries to renew a cert before the renewal master, the output of getcert list should be 'CA-WORKING' and certmonger should retry 8 hours laters (see the code in [2]). Since you are hitting the issue 8164, you can manually force the renewal on the replica (once the CA renewal master has actually renewed the cert) with getcert resubmit. HTH, flo Hi Flo Thank you very much! The getcert resubmit has successfully renewed all the certificates in need of renewal. The comments from Rob on the commit to fix this issue are very helpful in understanding what is happening too. Roderick [1] https://pagure.io/freeipa/issue/8164 [2] https://pagure.io/freeipa/blob/b5b9efeb57c010443c33c6f14f831abdbd804e78/f/install/certmonger/dogtag-ipa-ca-renew-agent-submit.in#_370 I haven't seen certmonger attempt to try the renewal again on host3 (nothing from certmonger in /var/log/messages since 30 Jan 13:37). While I could try a getcert resubmit on host3 to force it to try again, I'd like to know if what I am seeing is the expected behaviour when a replica tried to renew certificates before the renewal master. How long should I have to wait till certmonger on host3 tries again? - I couldn't find any reference to how often certmonger tries the renewal. Rob Crittenden's freeipa-healthcheck script is now showing the following for host3: ERROR: ipahealthcheck.ipa.certs.IPARAAgent: RA agent description does not match 2;16;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM in LDAP and 2;7;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM expected ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20180926040924: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20180926040920: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20180926040921: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20180926040922: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20180926040923: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20180926040925: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20180926040927: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20180926040926: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20180831064406: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.dogtag.ca.DogtagCertsConnectivityCheck: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) Each of host1, host2 and host3 are showing serial number 16 in ldap using: ldapsearch -D "cn=directory manager" -W -b uid=ipara,ou=people,o=ipaca
[Freeipa-users] Re: Replica not renewing IPA certificates
On 1/31/20 2:03 PM, Roderick Johnstone via FreeIPA-users wrote: Hi This is freeipa (ipa-server-4.6.5-11.el7_7.3.x86_64) on RHEL7 with freeipa's own internal CA. One of my ipa server replicas (host3) has not renewed its IPA system certificates and is now showing ca-error: Invalid cookie: u'' in the 'getcert list' output for certificates: "auditSigningCert cert-pki-ca", "ocspSigningCert cert-pki-ca", "subsystemCert cert-pki-ca", and the certificate in the file /var/lib/ipa/ra-agent.pem As far as I can see, the sequence of events has been as follows: host3 noticed the certificates needed renewing at 30 Jan 2020 05:37 and certmonger initiated a renewal. The state of those certificates went from MONITORING to CA_WORKING but the certificates were not renewed. The CA renewal master (host1) noticed its same set of certificates (plus "Server-Cert cert-pki-ca") needed renewing at 30 Jan 2020 07:28 and renewed them successfully. Another replica (host2) noticed that its certificates needed renewing at 30 Jan 2020 07:32 and renewed them successfully. At 30 Jan 13:37 on host3 the certificates needing to be renewed went from CA_WORKING back to MONITORING, but 'getcert list' now shows them with: ca-error: Invalid cookie: u'' and they still haven't renewed. Hi the 'Invalid cookie' error message is an issue already tracked in ticket 8164 Renewed certs are not picked up by IPA CAs [1]. When a replica tries to renew a cert before the renewal master, the output of getcert list should be 'CA-WORKING' and certmonger should retry 8 hours laters (see the code in [2]). Since you are hitting the issue 8164, you can manually force the renewal on the replica (once the CA renewal master has actually renewed the cert) with getcert resubmit. HTH, flo [1] https://pagure.io/freeipa/issue/8164 [2] https://pagure.io/freeipa/blob/b5b9efeb57c010443c33c6f14f831abdbd804e78/f/install/certmonger/dogtag-ipa-ca-renew-agent-submit.in#_370 I haven't seen certmonger attempt to try the renewal again on host3 (nothing from certmonger in /var/log/messages since 30 Jan 13:37). While I could try a getcert resubmit on host3 to force it to try again, I'd like to know if what I am seeing is the expected behaviour when a replica tried to renew certificates before the renewal master. How long should I have to wait till certmonger on host3 tries again? - I couldn't find any reference to how often certmonger tries the renewal. Rob Crittenden's freeipa-healthcheck script is now showing the following for host3: ERROR: ipahealthcheck.ipa.certs.IPARAAgent: RA agent description does not match 2;16;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM in LDAP and 2;7;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM expected ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20180926040924: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20180926040920: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20180926040921: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20180926040922: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20180926040923: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20180926040925: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20180926040927: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20180926040926: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20180831064406: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.dogtag.ca.DogtagCertsConnectivityCheck: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) Each of host1, host2 and host3 are showing serial number 16 in ldap using: ldapsearch -D "cn=directory manager" -W -b uid=ipara,ou=people,o=ipaca description At this stage I'm not sure whether this will resolve itself when certmonger tries to renew certificates again or whether I need to be more proactive. I'm happy to supply more logs as necessary. Thanks Roderick ___ FreeIPA-users mailing list --
