This looks like the root cause: ERROR: ipahealthcheck.ipa.certs.IPARAAgent: RA agent description does not match 2;186;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM in LDAP and 2;66;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM expected
It looks like an updated RA certificate was issued and was not picked up by the mother server(s). Off the top of my head this could be: - replication of the CA data has a problem: ipa-csreplica-manage list -v `hostname` - The updated certificate wasn't published to cn=certificates,cn=ipa,cn=etc,$SUFFIX - certmonger isn't picking up the renewal for some reason. The journal may hold clues. - something I'm forgetting I'd start with the first two. rob Kathy Zhu via FreeIPA-users wrote: > I just found this post about the same or similar issue: > > https://lists.fedoraproject.org/archives/list/[email protected]/thread/DFEMDNWSCE4FDDFRDUCZYYIIOIUC3RFD/ > > One detail I missed - this happens on all IPA servers BUT the renewal > IPA server. I will go through ^ post to see if that applies to our > situation. > > Thanks. > > Kathy. > > > On Wed, Apr 13, 2022 at 10:21 AM Kathy Zhu wrote: > > Hi team, > > > ipa-healthcheck has been a great tool for us. I run it weekly on all > IPA servers via cron. This week ipa-healthcheck reported errors on > all IPA servers. > > > Take IPA server ipa2 as an example for the investigation: > > > > [root@ipa2 ~]# ipa-healthcheck --failures-only --output-type=human > > ra.get_certificate(): EXCEPTION (Invalid Credential.) > > ra.get_certificate(): EXCEPTION (Invalid Credential.) > > ra.get_certificate(): EXCEPTION (Invalid Credential.) > > ra.get_certificate(): EXCEPTION (Invalid Credential.) > > ra.get_certificate(): EXCEPTION (Invalid Credential.) > > ra.get_certificate(): EXCEPTION (Invalid Credential.) > > ra.get_certificate(): EXCEPTION (Invalid Credential.) > > ra.get_certificate(): EXCEPTION (Invalid Credential.) > > ra.get_certificate(): EXCEPTION (Invalid Credential.) > > ra.get_certificate(): EXCEPTION (Invalid Credential.) > > WARNING: > ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210040: > Request > id 20190425210040 expires in 27 days > > WARNING: > ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210052: > Request > id 20190425210052 expires in 27 days > > WARNING: > ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210053: > Request > id 20190425210053 expires in 27 days > > WARNING: > ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210054: > Request > id 20190425210054 expires in 27 days > > WARNING: > ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210040: > Request id 20190425210040 expires in 27 days > > WARNING: > ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210052: > Request id 20190425210052 expires in 27 days > > WARNING: > ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210053: > Request id 20190425210053 expires in 27 days > > WARNING: > ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210054: > Request id 20190425210054 expires in 27 days > > ERROR: ipahealthcheck.ipa.certs.IPARAAgent: RA agent description > does not match 2;186;CN=Certificate Authority,O=EXAMPLE.COM > <http://EXAMPLE.COM>;CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM> in > LDAP and 2;66;CN=Certificate Authority,O=EXAMPLE.COM > <http://EXAMPLE.COM>;CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM> > expected > > ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210040: > Request for certificate failed, Certificate operation cannot be > completed: EXCEPTION (Invalid Credential.) > > ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210052: > Request for certificate failed, Certificate operation cannot be > completed: EXCEPTION (Invalid Credential.) > > ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210053: > Request for certificate failed, Certificate operation cannot be > completed: EXCEPTION (Invalid Credential.) > > ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210054: > Request for certificate failed, Certificate operation cannot be > completed: EXCEPTION (Invalid Credential.) > > ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210055: > Request for certificate failed, Certificate operation cannot be > completed: EXCEPTION (Invalid Credential.) > > ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210056: > Request for certificate failed, Certificate operation cannot be > completed: EXCEPTION (Invalid Credential.) > > ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425205849: > Request for certificate failed, Certificate operation cannot be > completed: EXCEPTION (Invalid Credential.) > > ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425205831: > Request for certificate failed, Certificate operation cannot be > completed: EXCEPTION (Invalid Credential.) > > ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210120: > Request for certificate failed, Certificate operation cannot be > completed: EXCEPTION (Invalid Credential.) > > ERROR: ipahealthcheck.dogtag.ca.DogtagCertsConnectivityCheck: > Request for certificate failed, Certificate operation cannot be > completed: EXCEPTION (Invalid Credential.) > > [root@ipa2 ~]# > > > > The list of certs: > > > [root@ipa2 ~]# getcert list > > Number of certificates and requests being tracked: 9. > > Request ID '20190425205831': > > status: MONITORING > > stuck: no > > key pair storage: > > type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/dirsrv/slapd-EXAMPLE-COM/pwdfile.txt' > > certificate: > > type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS > Certificate DB' > > CA: IPA > > issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> > > subject: CN=ipa2.example.com <http://ipa2.example.com>,O=EXAMPLE.COM > <http://EXAMPLE.COM> > > expires: 2023-03-29 21:37:22 UTC > > dns: ipa2.example.com <http://ipa2.example.com> > > principal name: ldap/[email protected] > <mailto:[email protected]> > > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > eku: id-kp-serverAuth,id-kp-clientAuth > > pre-save command: > > post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv > EXAMPLE-COM > > track: yes > > auto-renew: yes > > Request ID '20190425205849': > > status: MONITORING > > stuck: no > > key pair storage: > type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' > > certificate: > type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS > Certificate DB' > > CA: IPA > > issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> > > subject: CN=ipa2.example.com <http://ipa2.example.com>,O=EXAMPLE.COM > <http://EXAMPLE.COM> > > expires: 2023-03-29 21:37:46 UTC > > dns: ipa2.example.com <http://ipa2.example.com> > > principal name: HTTP/[email protected] > <mailto:[email protected]> > > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > eku: id-kp-serverAuth,id-kp-clientAuth > > pre-save command: > > post-save command: /usr/libexec/ipa/certmonger/restart_httpd > > track: yes > > auto-renew: yes > > Request ID '20190425210040': > > status: MONITORING > > stuck: no > > key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key' > > certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' > > CA: dogtag-ipa-ca-renew-agent > > issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> > > subject: CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM> > > expires: 2022-05-11 03:40:55 UTC > > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > eku: id-kp-serverAuth,id-kp-clientAuth > > pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre > > post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert > > track: yes > > auto-renew: yes > > Request ID '20190425210052': > > status: MONITORING > > stuck: no > > key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert > cert-pki-ca',token='NSS Certificate DB',pin set > > certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert > cert-pki-ca',token='NSS Certificate DB' > > CA: dogtag-ipa-ca-renew-agent > > issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> > > subject: CN=CA Audit,O=EXAMPLE.COM <http://EXAMPLE.COM> > > expires: 2022-05-11 03:40:05 UTC > > key usage: digitalSignature,nonRepudiation > > pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > > post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert > "auditSigningCert cert-pki-ca" > > track: yes > > auto-renew: yes > > Request ID '20190425210053': > > status: MONITORING > > stuck: no > > key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert > cert-pki-ca',token='NSS Certificate DB',pin set > > certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert > cert-pki-ca',token='NSS Certificate DB' > > CA: dogtag-ipa-ca-renew-agent > > issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> > > subject: CN=OCSP Subsystem,O=EXAMPLE.COM <http://EXAMPLE.COM> > > expires: 2022-05-11 03:40:25 UTC > > eku: id-kp-OCSPSigning > > pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > > post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert > "ocspSigningCert cert-pki-ca" > > track: yes > > auto-renew: yes > > Request ID '20190425210054': > > status: MONITORING > > stuck: no > > key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert > cert-pki-ca',token='NSS Certificate DB',pin set > > certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert > cert-pki-ca',token='NSS Certificate DB' > > CA: dogtag-ipa-ca-renew-agent > > issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> > > subject: CN=CA Subsystem,O=EXAMPLE.COM <http://EXAMPLE.COM> > > expires: 2022-05-11 03:40:05 UTC > > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > eku: id-kp-serverAuth,id-kp-clientAuth > > pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > > post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert > "subsystemCert cert-pki-ca" > > track: yes > > auto-renew: yes > > Request ID '20190425210055': > > status: MONITORING > > stuck: no > > key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert > cert-pki-ca',token='NSS Certificate DB',pin set > > certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert > cert-pki-ca',token='NSS Certificate DB' > > CA: dogtag-ipa-ca-renew-agent > > issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> > > subject: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> > > expires: 2038-06-28 21:19:45 UTC > > key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign > > pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > > post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert > "caSigningCert cert-pki-ca" > > track: yes > > auto-renew: yes > > Request ID '20190425210056': > > status: MONITORING > > stuck: no > > key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert > cert-pki-ca',token='NSS > Certificate DB',pin set > > certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert > cert-pki-ca',token='NSS > Certificate DB' > > CA: dogtag-ipa-ca-renew-agent > > issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> > > subject: CN=ipa2.example.com <http://ipa2.example.com>,O=EXAMPLE.COM > <http://EXAMPLE.COM> > > expires: 2023-03-07 22:37:22 UTC > > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection > > pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > > post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert > "Server-Cert cert-pki-ca" > > track: yes > > auto-renew: yes > > Request ID '20190425210120': > > status: MONITORING > > stuck: no > > key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' > > certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' > > CA: IPA > > issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> > > subject: CN=ipa2.example.com <http://ipa2.example.com>,O=EXAMPLE.COM > <http://EXAMPLE.COM> > > expires: 2023-03-29 21:37:52 UTC > > principal name: krbtgt/[email protected] > <mailto:[email protected]> > > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > eku: id-kp-serverAuth,id-pkinit-KPKdc > > pre-save command: > > post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert > > track: yes > > auto-renew: yes > > [root@ipa2 ~]# > > > > > There are 4 certs which expire on 2022-05-11 which match "expires in > 27 days". Take 20190425210040 as an example, we have: > > > > > WARNING: > ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210040: > Request > id 20190425210040 expires in 27 days > > WARNING: > ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210040: > Request id 20190425210040 expires in 27 days > > ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210040: > Request for certificate failed, Certificate operation cannot be > completed: EXCEPTION (Invalid Credential.) > > > Request ID '20190425210040': > > status: MONITORING > > stuck: no > > key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key' > > certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' > > CA: dogtag-ipa-ca-renew-agent > > issuer: CN=Certificate Authority,O=EXAMPLE.COM > <http://EXAMPLE.COM> > > subject: CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM> > > expires: 2022-05-11 03:40:55 UTC > > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > eku: id-kp-serverAuth,id-kp-clientAuth > > pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre > > post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert > > track: yes > > auto-renew: yes > > > > I was able to manually renew it: > > > > [root@ipa2 ~]# ipa-getcert resubmit -i '20190425210040' > > Resubmitting "20190425210040" to "dogtag-ipa-ca-renew-agent". > > [root@ipa2 ~]# > > > > After renew, it "expires: 2024-04-02 06:09:32 UTC": > > > > Request ID '20190425210040': > > status: MONITORING > > stuck: no > > key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key' > > certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' > > CA: dogtag-ipa-ca-renew-agent > > issuer: CN=Certificate Authority,O=EXAMPLE.COM > <http://EXAMPLE.COM> > > subject: CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM> > > expires: 2024-04-02 06:09:32 UTC > > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > eku: id-kp-serverAuth,id-kp-clientAuth > > pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre > > post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert > > track: yes > > auto-renew: yes > > > > How to fix the issue reported by ipa-healthcheck? And what is this > issue about? > > > All IPA servers are at same level: > > > CentOS Linux release 7.9.2009 (Core) > > ipa-*server*.x86_64 4.6.8-5.el7.centos.7 > > *slapi-nis*.x86_64 0.56.5-3.el7_9 > > *389-ds-base*.x86_64 1.3.10.2-12.el7_9 > > *389-ds-base*-libs.x86_64 1.3.10.2-12.el7_9 > > > Many thanks! > > > Kathy. > > > _______________________________________________ > FreeIPA-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedorahosted.org/archives/list/[email protected] > Do not reply to spam on the list, report it: > https://pagure.io/fedora-infrastructure > _______________________________________________ FreeIPA-users mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/[email protected] Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
