Hi, Thanks for the help so far! I've actually run `ipa-cert-fix` on both nodes, it says everything is ok on both nodes. When I run it with verbose mode, it spits out the command it's running and the certificate it got, for example:
``` ipapython.ipautil: DEBUG: args=['/usr/bin/certutil', '-d', 'sql:/etc/pki/pki-tomcat/alias', '-L', '-n', 'storageCert cert-pki-kra', '-a', '-f', '/etc/pki/pki-tomcat/alias/pwdfile.txt'] ``` If I then take that cert and ask what `openssl x509 -text -noout` thinks about it, it tells me that it's valid from 2020-06-29 to 2022-06-29. Strangely, though, when I ask `getcert list`, it shows that the certificate: ``` certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS Certificate DB' ``` expires on 2020-06-27. It's almost as if this node's certificate has _already_ been renewed but certmonger (I think) doesn't know about it, which might be why it's having trouble renewing it. Here's what the two nodes say about replication: >From node one: ``` ipa-two.mydomain.org last update status: Error (0) Replica acquired successfully: Incremental update succeeded last update ended: 2020-07-06 17:46:17+00:00 ``` >From node two: ``` ipa-one.gaea.mythicnet.org last update status: Error (0) Replica acquired successfully: Incremental update succeeded last update ended: 2020-07-06 17:46:17+00:00 ``` I suppose this might be a good time to mention that this is a simple two node multi-master setup. Finally, I'm not sure if I'm doing this correctly, but to make absolutely sure about which node is the renewal master, I ran this on both nodes: ``` ldapsearch -H ldap://ipa-one.gaea.mythicnet.org -D 'cn=Directory Manager' -W -b 'cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org' '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn ldapsearch -H ldap://ipa-two.gaea.mythicnet.org -D 'cn=Directory Manager' -W -b 'cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org' '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn ``` The result for both is: ``` dn: cn=CA,cn=ipa-one.gaea.mythicnet.org ,cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org ``` So it looks like the renewal master is the one having this problem. Ilya Kogan w: github.com/ikogan e: [email protected] <http://twitter.com/ilkogan> <https://www.linkedin.com/in/ilyakogan/> On Mon, Jul 6, 2020 at 1:25 PM Rob Crittenden <[email protected]> wrote: > Florence Blanc-Renaud via FreeIPA-users wrote: > > On 7/5/20 2:23 AM, Ilya Kogan via FreeIPA-users wrote: > >> Hi, > >> > >> I seem to be facing a similar issue with one of my KRAs. My KRA > >> certificates were, for some reason, not automatically renewed when > >> they expired last month. Using `ipa-cert-fix` correctly fixed them on > >> _one_ host. On the other, they seem to be stuck in the renewal state > >> and `ipa-cert-fix` claims there's nothing to do: > >> > >> ``` > >> Request ID '20191031183458': > >> status: MONITORING > >> ca-error: Server at > >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit"; replied: > >> Missing credential: sessionID > >> stuck: no > >> key pair storage: > >> > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert > >> cert-pki-kra',token='NSS Certificate DB',pin set > >> certificate: > >> > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert > >> cert-pki-kra',token='NSS Certificate DB' > >> CA: dogtag-ipa-ca-renew-agent > >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG > >> <http://MYDOMAIN.ORG> > >> subject: CN=KRA Audit,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG> > >> expires: 2020-06-27 01:54:34 EDT > >> key usage: digitalSignature,nonRepudiation > >> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > >> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert > >> "auditSigningCert cert-pki-kra" > >> track: yes > >> auto-renew: yes > >> Request ID '20191031183459': > >> status: MONITORING > >> ca-error: Server at > >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit"; replied: > >> Missing credential: sessionID > >> stuck: no > >> key pair storage: > >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert > cert-pki-kra',token='NSS > >> Certificate DB',pin set > >> certificate: > >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert > cert-pki-kra',token='NSS > >> Certificate DB' > >> CA: dogtag-ipa-ca-renew-agent > >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG > >> <http://MYDOMAIN.ORG> > >> subject: CN=KRA Transport Certificate,O=MYDOMAIN.ORG > >> <http://MYDOMAIN.ORG> > >> expires: 2020-06-27 01:54:30 EDT > >> key usage: > >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > >> eku: id-kp-clientAuth > >> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > >> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert > >> "transportCert cert-pki-kra" > >> track: yes > >> auto-renew: yes > >> Request ID '20191031183500': > >> status: MONITORING > >> ca-error: Server at > >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit"; replied: > >> Missing credential: sessionID > >> stuck: no > >> key pair storage: > >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert > >> cert-pki-kra',token='NSS Certificate DB',pin set > >> certificate: > >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert > >> cert-pki-kra',token='NSS Certificate DB' > >> CA: dogtag-ipa-ca-renew-agent > >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG > >> <http://MYDOMAIN.ORG> > >> subject: CN=KRA Storage Certificate,O=MYDOMAIN.ORG > >> <http://MYDOMAIN.ORG> > >> expires: 2020-06-27 01:54:32 EDT > >> key usage: > >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > >> eku: id-kp-clientAuth > >> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > >> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert > >> "storageCert cert-pki-kra" > >> track: yes > >> auto-renew: yes > >> ``` > >> > >> Here are the sequence of events that seem to have led to this: > >> > >> 1. Install FreeIPA Master many years ago and continue to upgrade it > >> from time to time. > >> 2. Install FreeIPA Replica a few years after and continue to upgrade > >> it from time to time. > >> 3. Allow the certificates to expire on both nodes. > >> 4. Attempt to patch the replica via `yum upgrade` on the second node. > >> 5. Notice after reboot that `pki-tomcatd` is having trouble and > >> discover certificate issues. > >> 5. Issue `ipa-cert-fix`, reboot again, and notice that things are > >> working. Try and create a key in the vault. > >> 6. Attempt to patch the master via `yum upgrade` on the first node. > >> 7. Notice after reboot that everything seems to be ok. Try and create > >> a key in the vault. > >> 8. Notice a few days later that renewal seems to be broken on the > >> first node. > >> > >> At this point `ipa-cert-fix` just shows that everything is fine. If I > >> run it with -v, and then check the "storageCert cert-pki-kra" > >> certificate with `openssl x509 -text -in`, I'm shown: > > > > Hi, > > just double-checking, but did you run ipa-cert-fix on the replica that > > was repaired in step 5? If that's the case, it's normal that > > ipa-cert-fix does not see any issue as it's running only locally and > > does not attempt to repair remote nodes. > > > > You will need to login to the node with expired certs and run > > ipa-cert-fix there. > > I'd also look to see which one is the renewal master. That is the one > that should renew the cert. I'm too curious why the renewal raised an > error (as if it actually tried to renew) rather than either go into > CA_WORKING or pick up the updated cert. > > I'd also make sure that replication is working. On each master: > > # ipa-csreplica-manage list -v `hostname` > > rob > > > > > HTH, > > flo > > > >> > >> Validity > >> Not Before: Jun 29 00:52:33 2020 GMT > >> Not After : Jun 19 00:52:33 2022 GMT > >> > >> On the second known, `getcert list` shows correct expirations for > >> those certificates: > >> > >> Request ID '20191206005909': > >> status: MONITORING > >> stuck: no > >> key pair storage: > >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert > >> cert-pki-kra',token='NSS Certificate DB',pin set > >> certificate: > >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert > >> cert-pki-kra',token='NSS Certificate DB' > >> CA: dogtag-ipa-ca-renew-agent > >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG > >> <http://MYDOMAIN.ORG> > >> subject: CN=KRA Storage Certificate,O=MYDOMAIN.ORG > >> <http://MYDOMAIN.ORG> > >> expires: 2022-06-18 20:52:33 EDT > >> key usage: > >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > >> eku: id-kp-clientAuth > >> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > >> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert > >> "storageCert cert-pki-kra" > >> track: yes > >> auto-renew: yes > >> > >> It seems like _something_, perhaps `ipa-cert-fix` somehow renewed > >> these certificates but...outside of certmonger? Is this some other > >> version of https://bugzilla.redhat.com/show_bug.cgi?id=1788907? The > >> certificates are not in CA_WORKING though, they're in MONITORING. > >> > >> What can I do to get myself out of this state as it seems like I'm in > >> a "this could explode at any moment" situation? > >> > >> This is on Fedora 30 with IP version: > >> > >> Last metadata expiration check: 0:23:05 ago on Sat 04 Jul 2020 > >> 07:59:16 PM EDT. > >> Installed Packages > >> Name : certmonger > >> Version : 0.79.9 > >> Release : 1.fc30 > >> Architecture : x86_64 > >> Size : 3.4 M > >> Source : certmonger-0.79.9-1.fc30.src.rpm > >> Repository : @System > >> From repo : updates > >> > >> .. snip .. > >> > >> Name : freeipa-server > >> Version : 4.8.3 > >> Release : 1.fc30 > >> Architecture : x86_64 > >> Size : 1.3 M > >> Source : freeipa-4.8.3-1.fc30.src.rpm > >> Repository : @System > >> From repo : updates > >> > >> .. snip .. > >> > >> Thanks! > >> > >> > >> Ilya Kogan > >> w: github.com/ikogan <http://github.com/ikogan> e: > >> [email protected] <mailto:[email protected]> > >> <http://twitter.com/ilkogan> <https://www.linkedin.com/in/ilyakogan/> > >> > >> > >> _______________________________________________ > >> FreeIPA-users mailing list -- [email protected] > >> To unsubscribe send an email to > >> [email protected] > >> Fedora Code of Conduct: > >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > >> List Archives: > >> > https://lists.fedorahosted.org/archives/list/[email protected] > >> > >> > > _______________________________________________ > > FreeIPA-users mailing list -- [email protected] > > To unsubscribe send an email to > [email protected] > > Fedora Code of Conduct: > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: > > > https://lists.fedorahosted.org/archives/list/[email protected] > > > >
_______________________________________________ FreeIPA-users mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/[email protected]
