Hi all, I'm having serious issues with our FreeIPA setup and I need some direction.
Our FreeIPA setup had two master-replicas. Late last month one of the hypervisors at OVH died, they replaced hardware but the server is having issues so hasn't come up yet. So for all matters, one master-replica is dead. The original master was configured with letsencrypt-freeipa which failed to renew certificates. There are around 10 clients connected to it, and several services authenticate against it. One for example is Gitlab, but I am still able to login to Gitlab. Another example we have a number of pfSense routers that also use LDAP auth and that always fails we had to fallback to the local admin user. One of the most critical services is the DNS. When DNS goes down, everything goes down, including email. This is currently one of the most critical services. ipactl always fails. I have to manually start the services using systemctl, like `systemctl start {named-pkcs11,httpd,ipa-custodia,ipa-dnskeysyncd,ipa-ods-exporter,ods-enforcerd,krb5kdc,kadmin}` getcert list returns 7 certificates, all MONITORING, none expired. # getcert list -d /etc/httpd/alias -n ipaCert No request found that matched arguments. I can run ldap commands on the cli. ALL ipa commands fail: # ipa userlist ipa: ERROR: cannot connect to 'any of the configured servers': https://main.domain.io/ipa/json, https://secondary.domain.io/ipa/json # certutil -L -d /etc/httpd/alias Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI DSTRootCAX3 C,, CN=main.domain.io u,u,u letsencryptx3 C,, letsencryptx3 C,, ISRGRootCAX1 C,, DOMAIN.IO IPA CA CT,C, the ipa-cert-fix command with increased verbosity: ``` ipapython.ipautil: DEBUG: stderr= ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/pki/pki-tomcat/alias -L -n transportCert cert-pki-kra -a -f /etc/pki/pki-tomcat/alias/pwdfile.txt ipapython.ipautil: DEBUG: Process finished, return code=255 ipapython.ipautil: DEBUG: stdout= ipapython.ipautil: DEBUG: stderr=certutil: Could not find cert: transportCert cert-pki-kra : PR_FILE_NOT_FOUND_ERROR: File not found ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/pki/pki-tomcat/alias -L -n storageCert cert-pki-kra -a -f /etc/pki/pki-tomcat/alias/pwdfile.txt ipapython.ipautil: DEBUG: Process finished, return code=255 ipapython.ipautil: DEBUG: stdout= ipapython.ipautil: DEBUG: stderr=certutil: Could not find cert: storageCert cert-pki-kra : PR_FILE_NOT_FOUND_ERROR: File not found ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/pki/pki-tomcat/alias -L -n auditSigningCert cert-pki-kra -a -f /etc/pki/pki-tomcat/alias/pwdfile.txt ipapython.ipautil: DEBUG: Process finished, return code=255 ipapython.ipautil: DEBUG: stdout= ipapython.ipautil: DEBUG: stderr=certutil: Could not find cert: auditSigningCert cert-pki-kra : PR_FILE_NOT_FOUND_ERROR: File not found ipapython.ipautil: DEBUG: Starting external process ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/httpd/alias -L -n Server-Cert -a -f /etc/httpd/alias/pwdfile.txt ipapython.ipautil: DEBUG: Process finished, return code=255 ipapython.ipautil: DEBUG: stdout= ipapython.ipautil: DEBUG: stderr=certutil: Could not find cert: Server-Cert : PR_FILE_NOT_FOUND_ERROR: File not found ipapython.admintool: DEBUG: File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_cert_fix.py", line 100, in run certs, extra_certs = expired_certs(now) File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_cert_fix.py", line 142, in expired_certs return expired_dogtag_certs(now), expired_ipa_certs(now) File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_cert_fix.py", line 191, in expired_ipa_certs cert = db.get_cert('Server-Cert') File "/usr/lib/python2.7/site-packages/ipapython/certdb.py", line 744, in get_cert raise RuntimeError("Failed to get %s" % nickname) ipapython.admintool: DEBUG: The ipa-cert-fix command failed, exception: RuntimeError: Failed to get Server-Cert ipapython.admintool: ERROR: Failed to get Server-Cert ipapython.admintool: ERROR: The ipa-cert-fix command failed. ``` I thought this command was to fix the certificates, so I don't get it why it fails if one certificate is missing. But anyway, can someone PLEASE give me some help I'm not great with certificates and I'm not being able to fix this. If there's a way of creating a new master from start and then importing the data would be nice, but looking at ipa-backup/restore it clearly says it has to be the same server. _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org