[Freeipa-users] Re: pki-tomcat won't start + expired certificates
Hi! Here is the output of ipa-cert-fix on the original instance: ``` The following certificates will be renewed: Dogtag sslserver certificate: Subject: CN=ipa.DOMAIN.COM,O=DOMAIN.COM Serial: 3 Expires: 2024-03-19 20:36:25 Dogtag subsystem certificate: Subject: CN=CA Subsystem,O=DOMAIN.COM Serial: 4 Expires: 2024-03-19 20:36:27 Dogtag ca_ocsp_signing certificate: Subject: CN=OCSP Subsystem,O=DOMAIN.COM Serial: 2 Expires: 2024-03-19 20:36:24 Dogtag ca_audit_signing certificate: Subject: CN=CA Audit,O=DOMAIN.COM Serial: 5 Expires: 2024-03-19 20:36:30 IPA IPA RA certificate: Subject: CN=IPA RA,O=DOMAIN.COM Serial: 7 Expires: 2024-03-19 20:38:19 IPA KDC certificate: Subject: CN=ipa.DOMAIN.COM,O=DOMAIN.COM Serial: 10 Expires: 2024-03-30 20:40:27 Enter "yes" to proceed: yes Proceeding. CalledProcessError(Command ['pki-server', 'cert-fix', '--ldapi-socket', '/run/slapd-DOMAIN-COM.socket', '--agent-uid', 'ipara', '--cert', 'sslserver' , '--cert', 'subsystem', '--cert', 'ca_ocsp_signing', '--cert', 'ca_audit_signing', '--extra-cert', '7', '--extra-cert', '10'] returned non-zero exit stat us 1: "INFO: Loading instance: pki-tomcat\nINFO: Loading global Tomcat config: /etc/tomcat/tomcat.conf\nINFO: Loading PKI Tomcat config: /usr/share/pki/et c/tomcat.conf\nINFO: Loading instance Tomcat config: /etc/pki/pki-tomcat/tomcat.conf\nINFO: Loading password config: /etc/pki/pki-tomcat/password.conf\nIN FO: Loading subsystem config: /var/lib/pki/pki-tomcat/ca/conf/CS.cfg\nINFO: Loading subsystem registry: /var/lib/pki/pki-tomcat/ca/conf/registry.cfg\nINFO : Loading instance registry: /etc/sysconfig/pki/tomcat/pki-tomcat/pki-tomcat\nINFO: Fixing the following system certs: ['sslserver', 'subsystem', 'ca_ocsp _signing', 'ca_audit_signing']\nINFO: Renewing the following additional certs: ['7', '10']\nINFO: Stopping the instance to proceed with system cert renewa l\nINFO: Configuring LDAP connection for CA\nINFO: Setting pkidbuser password via ldappasswd\nSASL/EXTERNAL authentication started\nSASL username: gidNumb er=0+uidNumber=0,cn=peercred,cn=external,cn=auth\nSASL SSF: 0\nINFO: Storing subsystem config: /var/lib/pki/pki-tomcat/ca/conf/CS.cfg\nINFO: Storing regis try config: /var/lib/pki/pki-tomcat/ca/conf/registry.cfg\nINFO: Storing subsystem config: /var/lib/pki/pki-tomcat/ca/conf/CS.cfg\nINFO: Storing registry c onfig: /var/lib/pki/pki-tomcat/ca/conf/registry.cfg\nINFO: Selftests disabled for subsystems: ca\nSASL/EXTERNAL authentication started\nSASL username: gid Number=0+uidNumber=0,cn=peercred,cn=external,cn=auth\nSASL SSF: 0\nINFO: Resetting password for uid=ipara,ou=people,o=ipaca\nSASL/EXTERNAL authentication started\nSASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth\nSASL SSF: 0\nINFO: Creating a temporary sslserver cert\nINFO: Getting ssl server cert info from CS.cfg\nINFO: Getting sslserver cert info from NSS database\nINFO: Trying to create a new temp cert for sslserver.\nINFO: Generate t emp SSL certificate\nINFO: Getting sslserver cert info from CS.cfg\nINFO: Getting sslserver cert info from NSS database\nINFO: CSR for sslserver has been written to /tmp/tmpydx011j8/sslserver.csr\nINFO: Getting signing cert info from CS.cfg\nINFO: Getting signing cert info from NSS database\nINFO: CA cert w ritten to /tmp/tmpydx011j8/ca_certificate.crt\nINFO: AKI: 0x7A0D23C6A1283EB899A0E5A4EFA3F92042F7F6D0\nINFO: Storing subsystem config: /var/lib/pki/pki-tom cat/ca/conf/CS.cfg\nINFO: Storing registry config: /var/lib/pki/pki-tomcat/ca/conf/registry.cfg\nINFO: Selftests enabled for subsystems: ca\nINFO: Restori ng LDAP connection for CA\nINFO: Storing subsystem config: /var/lib/pki/pki-tomcat/ca/conf/CS.cfg\nINFO: Storing registry config: /var/lib/pki/pki-tomcat/ ca/conf/registry.cfg\nERROR: Failed to generate CA-signed temp SSL certificate. RC: 255\n") The ipa-cert-fix command failed. ``` > If you have a backup of the previous http/ldap certs you can put them back > in place. Unfortunately, I don't have these anymore. However, I tried the approach I described above on a copy of the data in another container, managed to install temporary certs/CA for the ldap/httpd servers, pki-tomcat seems to be able to establish the connection to the LDAP but crashes at the following error. `Certificate not found: caSigningCert cert-pki-ca` Not sure what else needs to be fixed. On this copy, with the hacked temporary certs, if I run `ipa-cert-fix` I get the same error as on the original instance. If I run the `pki-server cert-fix` command that crashes, but removing `--cert sslserver`, it goes a bit further but is still blocked by `pki-tomcat` not being able to start. Thanks for all the help. -- ___ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org Fedora Code of Conduct:
[Freeipa-users] Re: pki-tomcat won't start + expired certificates
Bonjour Florence, Thanks for your help. I am using the docker image `freeipa/freeipa-server:fedora-34-4.9.6`, I guess the dependencies are correct as this is all bundled in the container, (though there might exists config mismatched if ipa upgrades failed containers updates). Se-linux is disabled on host and in the container. I made progress by fixing the missing instanceRoot parameter in the config file. Now I think I am stuck in a deadlock, because of letsencrypt certificates used for httpd/ldap (installed with ipa-cacert-manage) . The certificated managed by freeipa is expired, but the letsencrypt one have renewed and there is no overlap of their period of validity. - If I set back the date to when the freeipa certs are valid, pki connection to the ldap fails, as the letsencrypt one is not yet valid. error is `SEVERE: Unable to create socket: org.mozilla.jss.ssl.SSLSocketException: org.mozilla.jss.ssl.SSLSocketException: SSL_ForceHandshake failed: (-8181) Peer's Certificate has expired.` I think the message says expired for not-yet-valid certs too. - If I use the current time, it is not possible to start the pki-server as the certs are expired. ( at least that's my guess, error is :`netscape.ldap.LDAPException: Authentication failed (48)` not much more details) I was thinking about trying to: - set the date to when the freeipa managed certs were still valid. - manually generate a certificate/key from the CA (not sure how exactly, though) - copy these certificate and key in the httpd and ldap config folder at the right place. - try to spin-up the pki-tomcat, hoping that it works. - then hope that it auto-renews certs or manually trigger the renewal. - move the date back to today, maybe by increments that cover the certs validity, and trigger certs renewal at each increment. Would that make sense? Do you see any more sensible/simpler way? Many thanks! Basile -- ___ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
[Freeipa-users] pki-tomcat won't start + expired certificates
Hi freeipa experts. I have been using freeipa for the past 5 years running in a docker container, no replicas. currently on VERSION: 4.9.6, API_VERSION: 2.245 I have the following issue, not sure what caused this: pki-tomcat service is not starting, and it is no longer possible to login through the web-ui. Auth through ldap (some websites) and through sssd on linux servers is still working, kerberos tickets are generated when logging with password or when running kinit, so critical operations are still possible. The messages in `systemctl status pki-tomcatd@pki-tomcat.service` are ``` Apr 12 13:50:33 ipa.domain.com ipa-pki-wait-running[17869]: ipa-pki-wait-running: Request failed unexpectedly, 404 Client Error: for url: http://ipa.domain.com:8080/ca/admin/ca/getStatus Apr 12 13:50:34 ipa.domain.com systemd[1]: pki-tomcatd@pki-tomcat.service: start-post operation timed out. Terminating. Apr 12 13:50:34 ipa.domain.com systemd[1]: pki-tomcatd@pki-tomcat.service: Control process exited, code=killed, status=15/TERM Apr 12 13:50:34 ipa.domain.com systemd[1]: pki-tomcatd@pki-tomcat.service: Failed with result 'timeout'. Apr 12 13:50:34 ipa.domain.com systemd[1]: Failed to start PKI Tomcat Server pki-tomcat. ``` journalctl give other errors (filtered what seems relevant). ``` Apr 12 13:49:05 ipa.domain.com server[17868]: WARNING: Problem with JAR file [/usr/share/pki/server/common/lib/commons-collections.jar], exists: [false], canRead: [false] Apr 12 13:49:07 ipa.domain.com java[17868]: usr/lib/api/apiutil.c Could not open /run/lock/opencryptoki/LCK..APIlock Apr 12 13:49:18 ipa.domain.com server[17868]: SEVERE: Context [/acme] startup failed due to previous errors ``` `/var/log/pki/pki-tomcat/pki/debug.2024-04-12.log` contains the following errors ``` 2024-04-12 15:01:12 [main] SEVERE: Exception initializing random number generator using provider [Mozilla-JSS] java.security.NoSuchProviderException: no such provider: Mozilla-JSS at java.base/sun.security.jca.GetInstance.getService(GetInstance.java:83) at java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:206) ``` `/var/log/pki/pki-tomcat/ca/debug.2024-04-12.log` contains the following type of errors ``` 2024-04-12 00:17:37 [main] SEVERE: Unable to start CA engine: Property instanceRoot missing value Property instanceRoot missing value at com.netscape.cmscore.base.PropConfigStore.getString(PropConfigStore.java:297) at com.netscape.cmscore.apps.EngineConfig.getInstanceDir(EngineConfig.java:55) at com.netscape.cmscore.apps.CMSEngine.loadConfig(CMSEngine.java:233) at com.netscape.cmscore.apps.CMSEngine.start(CMSEngine.java:1025) 2024-04-12 17:49:21 [main] SEVERE: Exception sending context initialized event to listener instance of class [org.dogtagpki.server.ca.CAEngine] java.lang.RuntimeException: Unable to start CA engine: Property instanceRoot missing value at com.netscape.cmscore.apps.CMSEngine.contextInitialized(CMSEngine.java:1672) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4768) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5230) ``` `getcert list` reports all entries except the caCACert as expired. I tried pretty much everything I could find on the internet (though most of the threads I found were never resolved). Tried ipa-cert-fix. Tried ipa-restoring a backup in a new container, same problem occurs. My guess is that an upgrade years back did break the certificate auto-renewal and went undetected, and now everything is expired it's failing. If you have any ideas of what to check/try I would be very grateful as I am losing my sanity here. Also, I am a bit scared of breaking what is currently working (ldap+sssd) and critical to our operations, so if anything can be tested on a copy of the data in a container that would be great. Thanks! -- ___ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
