> If tomcat fails to start then you need to stop and figure out why. Any further poking at certificates won't yield anything useful without a working CA.

So at all times I should make sure `pki-tomcatd` is working. Well it is often failing with: `ipa-pki-wait-running: Request failed unexpectedly, 404 Client Error:  for url: http://ipa1.my.lab:8080/ca/admin/ca/getStatus`

Presumably this is meant to be fixed by `ipa-cacert-manage renew`, but as indicated it fails with `ERROR: CA
certificate chain in is incomplete`, specifically the traceback:
```
ipapython.admintool: DEBUG:   File "/usr/lib/python3.11/site-packages/ipapython/admintool.py", line 180, in execute
    return_value = self.run()
                   ^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ipaserver/install/ipa_cacert_manage.py", line 138, in run
    return self.renew()
           ^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ipaserver/install/ipa_cacert_manage.py", line 201, in renew
    return self.renew_external_step_2(ca, cert)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ipaserver/install/ipa_cacert_manage.py", line 277, in renew_external_step_2
    cert_file, ca_file = installutils.load_external_cert(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ipaserver/install/installutils.py", line 1194, in load_external_cert
    raise ScriptError(
```
(disregard the slightly offset line numbers, I've added some debugging to confirm that in the loop at [1] the trust chain has indeed only 1 item and it is not self-signed).

One thing suspicious is that the root CA is not set to `CT,C,C`:
```
ipapython.ipautil: DEBUG: args=['/usr/bin/certutil', '-d', 'sql:/tmp/tmprpz4cur1', '-M', '-n', 'CN=FreeIPA Root Certificate Authority,O=My Lab', '-t', 'C,,', '-f', '/tmp/tmprpz4cur1/pwdfile.txt'] ipapython.ipautil: DEBUG: args=['/usr/bin/certutil', '-d', 'sql:/tmp/tmprpz4cur1', '-M', '-n', 'CN=My Lab Root CA,O=My Lab', '-t', 'C,,', '-f', '/tmp/tmprpz4cur1/pwdfile.txt']
```

Do you think it is worth delving more into this issue and finding a way to recover `ipa-cacert-manage renew`?

> Manually add how?

Using `certutil -A -d` on the `nssdb` and individual files like in `/etc/httpd`. I get the locations as reported in `getcert list`.

> The KDC isn't running.

The KDC is running, but the `kdc.crt` certificates are busted.

[1] https://pagure.io/freeipa/blob/master/f/ipaserver/install/installutils.py#_1187

On 2023/09/25 21:06, Rob Crittenden wrote:
Cristian Le via FreeIPA-users wrote:
Ok, let me walk through some of the specific errors, and I will also
censor out some of the output since this is going to the public
mail-list as well.

Starting from the beginning.
- I have set the date to `1 month` before certificate expired with `sudo
date`
- I ran `ipactl restart --force` and `pki-tomcatd` failed
If tomcat fails to start then you need to stop and figure out why. Any
further poking at certificates won't yield anything useful without a
working CA.

- I try to run `ipa-cacert-manage renew` with `--external-ca` and
`--external-cert-file`, with the CSR signed with `openssl x509 -req
-copy_extensions=copyall`, but this one failed because of `ERROR: CA
certificate chain in is incomplete: missing certificate with subject`,
even though I passed both the root and new CA
What failed?

Then I have tried to manually add the new CA, but then, that did not
work. So moving on to the next layer, when I run to `sudo getcert list`,
the certificates I get:
Manually add how?

```
Request ID '20220810070559':
         status: CA_UNREACHABLE
         ca-error: Error setting up ccache for "host" service on client
using default keytab: Cannot contact any KDC for requested realm.
The KDC isn't running.

         stuck: no
         key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
         certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
         CA: IPA
         issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
         subject: CN=ipa1.my.lab,O=My Lab
         issued: 2023-09-20 14:21:40 UTC
         expires: 2025-09-20 14:21:40 UTC
Request ID '20230920203122':
         status: MONITORING
         stuck: no
         key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
         certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB'
         CA: dogtag-ipa-ca-renew-agent
         issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
         subject: CN=CA Audit,O=My Lab
         issued: 2023-09-20 14:21:35 UTC
         expires: 2025-09-09 14:21:35 UTC
Request ID '20230920203123':
         status: MONITORING
         stuck: no
         key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert 
cert-pki-ca',token='NSS
Certificate DB',pin set
         certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert 
cert-pki-ca',token='NSS
Certificate DB'
         CA: dogtag-ipa-ca-renew-agent
         issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
         subject: CN=OCSP Subsystem,O=My Lab
         issued: 2023-09-20 14:21:33 UTC
         expires: 2025-09-09 14:21:33 UTC
Request ID '20230920203125':
         status: MONITORING
         stuck: no
         key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB',pin set
         certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB'
         CA: dogtag-ipa-ca-renew-agent
         issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
         subject: CN=CA Subsystem,O=My Lab
         issued: 2023-09-20 14:21:31 UTC
         expires: 2025-09-09 14:21:31 UTC
Request ID '20230920203128':
         status: MONITORING
         stuck: no
         key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB',pin set
         certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB'
         CA: dogtag-ipa-ca-renew-agent
         issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
         subject: CN=ipa1.my.lab,O=My Lab
         issued: 2023-09-20 11:29:21 UTC
         expires: 2023-12-20 11:29:21 UTC
Request ID '20230920203130':
         status: MONITORING
         stuck: no
         key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
         certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
         CA: dogtag-ipa-ca-renew-agent
         issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
         subject: CN=IPA RA,O=My Lab
         issued: 2023-09-20 14:21:36 UTC
         expires: 2025-09-09 14:21:36 UTC
Request ID '20230920203131':
         status: CA_UNREACHABLE
         ca-error: Error setting up ccache for "host" service on client
using default keytab: Cannot contact any KDC for requested realm.
         stuck: no
         key pair storage:
type=NSSDB,location='/etc/dirsrv/slapd-MY-LAB',nickname='Server-Cert',token='NSS
Certificate DB',pinfile='/etc/dirsrv/slapd-MY-LAB/pwdfile.txt'
         certificate:
type=NSSDB,location='/etc/dirsrv/slapd-MY-LAB',nickname='Server-Cert',token='NSS
Certificate DB'
         CA: IPA
         issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
         subject: CN=ipa1.my.lab,O=My Lab
         issued: 2023-09-20 14:21:39 UTC
         expires: 2025-09-20 14:21:39 UTC
Request ID '20230920203134':
         status: CA_UNREACHABLE
         stuck: no
         key pair storage:
type=FILE,location='/var/lib/ipa/private/httpd.key',pinfile='/var/lib/ipa/passwds/ipa1.my.lab-443-RSA'
         certificate: type=FILE,location='/var/lib/ipa/certs/httpd.crt'
         CA: IPA
         issuer: CN=My Lab Root CA,O=My Lab
         subject: CN=ipa1.my.lab,O=My Lab
         issued: 2023-07-10 04:31:16 UTC
         expires: 2023-08-09 04:31:16 UTC
Request ID '20230510042436':
         status: MONITORING
         ca-error: Updated certificate not available
         stuck: no
         key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
         certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
cert-pki-ca',token='NSS Certificate DB'
         CA: dogtag-ipa-ca-renew-agent
         issuer: CN=My Lab Root CA,O=My Lab
         subject: CN=FreeIPA Root Certificate Authority,O=My Lab
         issued: 2022-08-10 04:29:31 UTC
         expires: 2023-08-10 04:29:31 UTC
```
Note that even the `MONITORING` ones are unusable due to the
non-overlapping date. So when I do something like `ipa-certupdate` I get:
`The ipa-certupdate command failed, exception: DatabaseError: Connect
error: error:0A000086:SSL routines::certificate verify failed (unable to
get issuer certificate)` when trying to use `ipaldap.py`.

It is a mess, that's why I think it is better to have a way to simply
re-bootstrap all certificates.
There isn't really a good way to start over.

rob

On 2023/09/22 15:57, Florence Blanc-Renaud wrote:
Hi,

On Fri, Sep 22, 2023 at 12:36 PM Cristian Le <[email protected]> wrote:

     Hi Florence,

     Thanks for the feedback, let me clarify the situation on the
     certificates:
     - External CA is still valid and it is a self-signed certificate
     that we use for other services. So we can manually sign any
     service certificates to get them back up and running
     - IPA CA is expired, let's say Aug/10
     - I have managed to import a renewed IPA CA and ran `ipa-cert-fix`
     (and also seemed to have run `ipa-certupdate`) on a current date,
     let's say Sep/20. But not all services were recovered and now
     there is no overlap between earliest date in service certificates
     and the original IPA CA

Which are the not-recovered services? Can you provide the output of
"getcert list" at the current date?
flo

     - I have run a backup, but also did some system upgrades to get
     the `ipa-cacert-manage prune` command, but when I've tried to
     recover it, I've found that the backup was not there.

      > you can still find the original certificates in the LDAP
     database (below ou=certificateRepository,ou=ca,o=ipaca) but it
     requires a bit of searching. You would need to restore the expired
     certificates, go back in time and force the renewal.

     I suspect we cannot do this all within LDAP right? If we get back
     the expired certificates, how do we restore them in each service?
     `httpd` is straightforward, and I guess `nssdb` should be doable,
     assuming the same key is used, but is there another database type
     where the certificates are located? Are all the certificates
     tracked by `getcert list`? Is it safe to assume that after running
     something like `ipa-cert-fix`, they are using the same private key?

     Some symptoms in the current setup:
     - When we are forward in time, `pki-tomcatd` is able to run, but
     then I can't do any `ipa-cert-fix` or `ipa-cacert-manage renew`.
     From what I've read, all of these commands (or at least
     `ipa-cacert-manage renew`) must be done backwards in time.
     - When we are backwards in time `pki-tomcatd` is unable to run,
     failing to access `:8080/ca/admin/ca/getStatus`. This then blocks
     various other services to be run. But about `ipactl restart`, only
     `pki-tomcatd` service is actually failing (and ipa service itself
     of course).

     I have navigated to `ou=certificateRepository,ou=ca,o=ipaca` and
     indeed there are still a bunch of certificates there in linear
     order. What are the services I should look for in there? I am
     using Apache Directory Studio and I can download the
     `userCertificate`. Should I just run `certutil -A` with those
     values with corresponding `subjectName`?

     BTW, I want to document this process on the website, should I make
     a PR on the github repo or is there somewhere else?

     Kind regards,
     Cristian

     On 2023/09/22 9:00, Florence Blanc-Renaud wrote:
     Hi,

     On Thu, Sep 21, 2023 at 5:04 PM Cristian Le via FreeIPA-users
     <[email protected]> wrote:

         I have tried my luck around with all the helpers: `pki-server
         cert-fix`, `ipa-cacert-manage`, `ipa-certupdate`, etc. but
         each one is failing on me for multiple reasons.
         - `ipa-cacert-manage` Cannot update the CA with
         `--external-cert-file` because the root ca is not detected to
         be in the trust list

     This command is useful if you need to trust a new external CA or
     renew IPA CA. Is your IPA CA expired?

         - `ipa-cert-fix` Was run without overlapping validity time,
         and the certificate were re-created, so now it is not
         recoverable, neither back in time, nor in current time

     It is recommended to do a backup before running ipa-cert-fix. If
     you didn't, and want to try the back-in-time method, you can
     still find the original certificates in the LDAP database (below
     ou=certificateRepository,ou=ca,o=ipaca) but it requires a bit of
     searching. You would need to restore the expired certificates, go
     back in time and force the renewal.

         - `pki-tomcat` is failing

     What is the current situation? Which certs are expired (getcert
     list)? If you start the services with "ipactl start
     --ignore-service-failures", is pki-tomcat the only service failing?
     flo

         It is quite a mess and I would like to ask for some guidance
         on how one could recover manually from  such dependency issues:
         - Is it possible to do a `ipa-server-install` and keep the
         user data?

         - If I sign all of the service's certificates manually, what
         are all of the manual steps needed to get the services back
         up so that the helpers can be run.
           - I've tried to install the CA certificate in the nssdb
         database, ldap, and /etc/ipa/ca.crt. Are there other locations?
           - I've recreated an httpd certificate signed by the root,
         but I can't figure how to do the same with the ones located
         in the nssdb database, i.e. to recreate a csr with the same
         data as one of the certificates there
         - What is the order of services that should be updated. My
         understanding is CA -> `certutil`'s CA -> httpd + slapd +
         pki-tomcat (not sure where the last one is or how to edit it)
         -> `ipa-certupdate`
         _______________________________________________
         FreeIPA-users mailing list --
         [email protected]
         To unsubscribe send an email to
         [email protected]
         Fedora Code of Conduct:
         https://docs.fedoraproject.org/en-US/project/code-of-conduct/
         List Guidelines:
         https://fedoraproject.org/wiki/Mailing_list_guidelines
         List Archives:
         
https://lists.fedorahosted.org/archives/list/[email protected]
         Do not reply to spam, report it:
         https://pagure.io/fedora-infrastructure/new_issue

_______________________________________________
FreeIPA-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedorahosted.org/archives/list/[email protected]
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

_______________________________________________
FreeIPA-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedorahosted.org/archives/list/[email protected]
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Reply via email to