Jernej Jakob via FreeIPA-users wrote: > I've been trying to debug this for the last couple of days. I can't > find what's wrong. I found that another client whose cert also expired > on 2023-06-07 was in the same SUBMITTING state. The same exact > conditions. Same exact OS, Ubuntu 20.04 LTS. certmonger package is > up-to-date. > > I increased certmonger debug level to 6.
I'd suggest 3. 6 will include a lot of unnecessary dbus noise. > I also changed in /var/lib/certmonger/cas/20141205060547-3 > ca_external_helper=/usr/lib/certmonger/dogtag-ipa-renew-agent-submit > to > ca_external_helper=/usr/lib/certmonger/dogtag-ipa-renew-agent-submit -v That is the wrong certmonger CA. You want the IPA CA helper. This one is for renewing the CA certificates itself. And yes CA here is an overloaded term. > The logs are attached. > > What I find weird is that on line 180: > GET "http://redacted.example.com:9180/ca/ee/ca/profileList?xml=true"; > this is the client's address, not the ipa-ca server's. Perhaps check the values in /etc/ipa/default.conf. > I found these that may or may not be related > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=955530 > https://salsa.debian.org/freeipa-team/certmonger/-/blob/debian/0.79.9-2/debian/patches/Revert-Don-t-close-STDERR-when-submitting-request.patch > https://bugs.launchpad.net/ubuntu/+source/certmonger/+bug/1875227 > > Any ideas on what to do next? > > On Fri, 16 Jun 2023 15:38:46 +0200 > Jernej Jakob <[email protected]> (by way of Jernej Jakob > <[email protected]>) wrote: > >> Hi, I have a client whose host certificate expired on 2023-06-07. >> Today I logged into the FreeIPA webui and went to the certificates page >> which was very slow to load. I had this problem before when there was >> one host (a different one) stuck in a certificate request loop, so I >> immediately suspected the same thing happened again. Sure enough, there >> are >3000 certificates for this host listed in IPA. >> Running 'getcert list' on the host shows: >> >> Number of certificates and requests being tracked: 1. >> Request ID '20190703221417': >> status: SUBMITTING >> stuck: no >> key pair storage: >> type=NSSDB,location='/etc/ipa/nssdb',nickname='Local IPA host',token='NSS >> Certificate DB',pinfile='/etc/ipa/nssdb/pwdfile.txt' >> certificate: type=NSSDB,location='/etc/ipa/nssdb',nickname='Local >> IPA host',token='NSS Certificate DB' >> CA: IPA >> issuer: CN=Certificate Authority,O=EXAMPLE.COM >> subject: CN=redacted.example.com,O=EXAMPLE.com >> expires: 2023-06-07 00:14:30 CEST >> dns: redacted.example.com >> principal name: host/[email protected] >> key usage: >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >> eku: id-kp-serverAuth,id-kp-clientAuth >> pre-save command: >> post-save command: >> track: yes >> auto-renew: yes >> >> I stopped certmonger to stop the loop. Then checked if the problem was >> the same as on that other host some time ago, but it was not. >> I saw one error in syslog (this turned out to not be the issue): >> >> "Server at https://ipa1.example.com/ipa/xml failed request, will retry: -504 >> (libcurl failed to execute the HTTP POST transaction, explaining: Could not >> resolve host: ipa1.example.com)." >> >> That is a server that has been shut down. It was the first server I >> installed IPA onto, which had a replica ipa2. That was on CentOS 7. >> This year I migrated to Rocky 8 by adding two new clients, ipa3 and >> ipa4, following the official procedure by promoting each client to a >> replica, set ipa3 as CA renewal master, and so on. Then after that was >> done I removed the old servers from replication, shut down IPA services, >> unenrolled them and updated all SRV records in DNS. I forgot about >> A/AAAA ipa-ca, which was left pointing at two addresses, that of ipa1 >> and ipa3. >> >> It's weird becase it can obviously contact and obtain a certificate >> from ipa3, whose logs contain these successful issuances. It could >> contact it on 2023-06-07, when the ipa-ca record was still pointing to >> both the ipa1 and ipa3 servers. >> On 2023-06-09 I dist-upgraded both ipa3/4 servers, at which time I also >> found the incorrect ipa-ca A/AAAA records (coincidentally as I was >> configuring a smart card CA certificate), so I corrected it. But the >> failing certmonger client still kept failing which makes me believe >> it was not the fault of the incorrect ipa-ca records. >> >> I searched around on the "redacted.example.com" for ipa1. It's not in >> /var/lib/certmonger/requests/20190703221417 (the failing request) >> but it is in /etc/ipa/default.conf: >> >> server = ipa1.example.com >> xmlrpc_uri = https://ipa1.example.com/ipa/xml >> >> I changed these values to ipa3 now, started certmonger and resubmitted >> the request. Unfortunately no change. That libcurl error for ipa1 is >> gone now: >> >> Jun 16 15:28:01 redacted certmonger[2091940]: 2023-06-16 15:28:01 [2091940] >> Token is named "NSS Generic Crypto Services", not "NSS Certificate DB", >> skipping. >> Jun 16 15:28:01 redacted certmonger[2091940]: 2023-06-16 15:28:01 [2091940] >> Unable to initialize NSS. >> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 [2091941] >> Setting "CERTMONGER_REQ_SUBJECT" to "CN=redacted.example.com,O=EXAMPLE.COM" >> for child. >> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 [2091941] >> Setting "CERTMONGER_REQ_PRINCIPAL" to >> "host/[email protected]" for child. >> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 [2091941] >> Setting "CERTMONGER_OPERATION" to "SUBMIT" for child. >> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 [2091941] >> Setting "CERTMONGER_CSR" to "-----BEGIN NEW CERTIFICATE REQUEST----- >> Jun 16 15:28:01 redacted certmonger[2091941]: ...redacted..." for child. >> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 [2091941] >> Setting "CERTMONGER_SPKAC" to ...redacted... >> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 [2091941] >> Setting "CERTMONGER_SPKI" to ...redacted... >> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 [2091941] >> Setting "CERTMONGER_LOCAL_CA_DIR" to "/var/lib/certmonger/local" for child. >> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 [2091941] >> Setting "CERTMONGER_KEY_TYPE" to "RSA" for child. >> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 [2091941] >> Setting "CERTMONGER_CA_NICKNAME" to "IPA" for child. >> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 [2091941] >> Setting "CERTMONGER_CERTIFICATE" to "-----BEGIN CERTIFICATE----- >> Jun 16 15:28:01 redacted certmonger[2091941]: ...redacted..." for child. >> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 [2091941] >> Redirecting stdin and stderr to /dev/null, leaving stdout open for child >> "/usr/lib/certmonger/ipa-submit". >> >> At this point it just keeps repeating the same over and over again. >> >> I would appreciate any help. Thanks. > > > _______________________________________________ > FreeIPA-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedorahosted.org/archives/list/[email protected] > Do not reply to spam, report it: > https://pagure.io/fedora-infrastructure/new_issue > _______________________________________________ FreeIPA-users mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/[email protected] Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
