Hi, On Wed, Jun 28, 2023 at 4:45 PM Rob Crittenden via FreeIPA-users < [email protected]> wrote:
> Jernej Jakob via FreeIPA-users wrote: > > I've been trying to debug this for the last couple of days. I can't > > find what's wrong. I found that another client whose cert also expired > > on 2023-06-07 was in the same SUBMITTING state. The same exact > > conditions. Same exact OS, Ubuntu 20.04 LTS. certmonger package is > > up-to-date. > > > > I increased certmonger debug level to 6. > > I'd suggest 3. 6 will include a lot of unnecessary dbus noise. > > > I also changed in /var/lib/certmonger/cas/20141205060547-3 > > ca_external_helper=/usr/lib/certmonger/dogtag-ipa-renew-agent-submit > > to > > ca_external_helper=/usr/lib/certmonger/dogtag-ipa-renew-agent-submit -v > > That is the wrong certmonger CA. You want the IPA CA helper. This one is > for renewing the CA certificates itself. And yes CA here is an > overloaded term. > > > The logs are attached. > > > > What I find weird is that on line 180: > > GET "http://redacted.example.com:9180/ca/ee/ca/profileList?xml=true"; > > this is the client's address, not the ipa-ca server's. > > Can you check the output of "getcert list-cas -c IPA"? What is the content of /etc/certmonger/certmonger.conf? Does /etc/ipa/default.conf contain a line with dogtag_version = <some value> ? flo > Perhaps check the values in /etc/ipa/default.conf. > > > I found these that may or may not be related > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=955530 > > > https://salsa.debian.org/freeipa-team/certmonger/-/blob/debian/0.79.9-2/debian/patches/Revert-Don-t-close-STDERR-when-submitting-request.patch > > https://bugs.launchpad.net/ubuntu/+source/certmonger/+bug/1875227 > > > > Any ideas on what to do next? > > > > On Fri, 16 Jun 2023 15:38:46 +0200 > > Jernej Jakob <[email protected]> (by way of Jernej Jakob > > <[email protected]>) wrote: > > > >> Hi, I have a client whose host certificate expired on 2023-06-07. > >> Today I logged into the FreeIPA webui and went to the certificates page > >> which was very slow to load. I had this problem before when there was > >> one host (a different one) stuck in a certificate request loop, so I > >> immediately suspected the same thing happened again. Sure enough, there > >> are >3000 certificates for this host listed in IPA. > >> Running 'getcert list' on the host shows: > >> > >> Number of certificates and requests being tracked: 1. > >> Request ID '20190703221417': > >> status: SUBMITTING > >> stuck: no > >> key pair storage: > type=NSSDB,location='/etc/ipa/nssdb',nickname='Local IPA host',token='NSS > Certificate DB',pinfile='/etc/ipa/nssdb/pwdfile.txt' > >> certificate: > type=NSSDB,location='/etc/ipa/nssdb',nickname='Local IPA host',token='NSS > Certificate DB' > >> CA: IPA > >> issuer: CN=Certificate Authority,O=EXAMPLE.COM > >> subject: CN=redacted.example.com,O=EXAMPLE.com > >> expires: 2023-06-07 00:14:30 CEST > >> dns: redacted.example.com > >> principal name: host/[email protected] > >> key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > >> eku: id-kp-serverAuth,id-kp-clientAuth > >> pre-save command: > >> post-save command: > >> track: yes > >> auto-renew: yes > >> > >> I stopped certmonger to stop the loop. Then checked if the problem was > >> the same as on that other host some time ago, but it was not. > >> I saw one error in syslog (this turned out to not be the issue): > >> > >> "Server at https://ipa1.example.com/ipa/xml failed request, will > retry: -504 (libcurl failed to execute the HTTP POST transaction, > explaining: Could not resolve host: ipa1.example.com)." > >> > >> That is a server that has been shut down. It was the first server I > >> installed IPA onto, which had a replica ipa2. That was on CentOS 7. > >> This year I migrated to Rocky 8 by adding two new clients, ipa3 and > >> ipa4, following the official procedure by promoting each client to a > >> replica, set ipa3 as CA renewal master, and so on. Then after that was > >> done I removed the old servers from replication, shut down IPA services, > >> unenrolled them and updated all SRV records in DNS. I forgot about > >> A/AAAA ipa-ca, which was left pointing at two addresses, that of ipa1 > >> and ipa3. > >> > >> It's weird becase it can obviously contact and obtain a certificate > >> from ipa3, whose logs contain these successful issuances. It could > >> contact it on 2023-06-07, when the ipa-ca record was still pointing to > >> both the ipa1 and ipa3 servers. > >> On 2023-06-09 I dist-upgraded both ipa3/4 servers, at which time I also > >> found the incorrect ipa-ca A/AAAA records (coincidentally as I was > >> configuring a smart card CA certificate), so I corrected it. But the > >> failing certmonger client still kept failing which makes me believe > >> it was not the fault of the incorrect ipa-ca records. > >> > >> I searched around on the "redacted.example.com" for ipa1. It's not in > >> /var/lib/certmonger/requests/20190703221417 (the failing request) > >> but it is in /etc/ipa/default.conf: > >> > >> server = ipa1.example.com > >> xmlrpc_uri = https://ipa1.example.com/ipa/xml > >> > >> I changed these values to ipa3 now, started certmonger and resubmitted > >> the request. Unfortunately no change. That libcurl error for ipa1 is > >> gone now: > >> > >> Jun 16 15:28:01 redacted certmonger[2091940]: 2023-06-16 15:28:01 > [2091940] Token is named "NSS Generic Crypto Services", not "NSS > Certificate DB", skipping. > >> Jun 16 15:28:01 redacted certmonger[2091940]: 2023-06-16 15:28:01 > [2091940] Unable to initialize NSS. > >> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 > [2091941] Setting "CERTMONGER_REQ_SUBJECT" to "CN=redacted.example.com,O= > EXAMPLE.COM" for child. > >> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 > [2091941] Setting "CERTMONGER_REQ_PRINCIPAL" to "host/ > [email protected]" for child. > >> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 > [2091941] Setting "CERTMONGER_OPERATION" to "SUBMIT" for child. > >> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 > [2091941] Setting "CERTMONGER_CSR" to "-----BEGIN NEW CERTIFICATE > REQUEST----- > >> Jun 16 15:28:01 redacted certmonger[2091941]: ...redacted..." for child. > >> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 > [2091941] Setting "CERTMONGER_SPKAC" to ...redacted... > >> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 > [2091941] Setting "CERTMONGER_SPKI" to ...redacted... > >> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 > [2091941] Setting "CERTMONGER_LOCAL_CA_DIR" to "/var/lib/certmonger/local" > for child. > >> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 > [2091941] Setting "CERTMONGER_KEY_TYPE" to "RSA" for child. > >> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 > [2091941] Setting "CERTMONGER_CA_NICKNAME" to "IPA" for child. > >> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 > [2091941] Setting "CERTMONGER_CERTIFICATE" to "-----BEGIN CERTIFICATE----- > >> Jun 16 15:28:01 redacted certmonger[2091941]: ...redacted..." for child. > >> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 > [2091941] Redirecting stdin and stderr to /dev/null, leaving stdout open > for child "/usr/lib/certmonger/ipa-submit". > >> > >> At this point it just keeps repeating the same over and over again. > >> > >> I would appreciate any help. Thanks. > > > > > > _______________________________________________ > > FreeIPA-users mailing list -- [email protected] > > To unsubscribe send an email to > [email protected] > > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: > https://lists.fedorahosted.org/archives/list/[email protected] > > Do not reply to spam, report it: > https://pagure.io/fedora-infrastructure/new_issue > > > _______________________________________________ > FreeIPA-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedorahosted.org/archives/list/[email protected] > Do not reply to spam, report it: > https://pagure.io/fedora-infrastructure/new_issue >
_______________________________________________ FreeIPA-users mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/[email protected] Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
