Thanks for the input rob - First: I found the cause for the frontend weirdness - I realised that some later issued certs were issued based on a accidentally issued, renewed CA cert that at some point I deleted because I thought it was never actually used - it turns out it was. The front-end json is based on `ipa cert-find` which seems it will only show certs properly with a complete and valid cert chain. So I reissued all certs based on the now missing intermediate cert (those in `getcert list').
But this turned out not to be the cause and unfortunately did not change the failure on upgrade from f35->f36 in any way. I still got the same error: $ less /var/log/ipaupgrade.log [...] 2024-06-25T23:36:50Z DEBUG Profile 'caECDualCert' is already in LDAP and enabled; skipping 2024-06-25T23:36:50Z INFO Migrating profile 'caECFullCMCSharedTokenCert' 2024-06-25T23:36:50Z DEBUG request GET https://ipa.mydomain.tld:8443/ca/rest/account/login 2024-06-25T23:36:50Z DEBUG request body '' 2024-06-25T23:36:51Z DEBUG response status 404 2024-06-25T23:36:51Z DEBUG response headers Content-Type: text/html;charset=utf-8 Content-Language: en Content-Length: 784 Date: Tue, 25 Jun 2024 23:36:51 GMT 2024-06-25T23:36:51Z DEBUG response body (decoded): b'<!doctype html><html lang="en"><head><title>HTTP Status 404 \xe2\x80\x93 Not Found</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 404 \xe2\x80\x93 Not Found</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Message</b> The requested resource [/ca/rest/account/login] is not available</p><p><b>Description</b> The origin server did not find a current representation for the target resource or is not willing to disclose that one exists.</p><hr class="line" /><h3>Apache Tomcat/9.0.71</h3></body></html>' 2024-06-25T23:36:51Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. 2024-06-25T23:36:51Z DEBUG File "/usr/lib/python3.10/site-packages/ipapython/admintool.py", line 180, in execute return_value = self.run() File "/usr/lib/python3.10/site-packages/ipaserver/install/ipa_server_upgrade.py", line 54, in run server.upgrade() File "/usr/lib/python3.10/site-packages/ipaserver/install/server/upgrade.py", line 2058, in upgrade upgrade_configuration() File "/usr/lib/python3.10/site-packages/ipaserver/install/server/upgrade.py", line 1911, in upgrade_configuration ca_enable_ldap_profile_subsystem(ca) File "/usr/lib/python3.10/site-packages/ipaserver/install/server/upgrade.py", line 458, in ca_enable_ldap_profile_subsystem cainstance.migrate_profiles_to_ldap() File "/usr/lib/python3.10/site-packages/ipaserver/install/cainstance.py", line 2111, in migrate_profiles_to_ldap _create_dogtag_profile(profile_id, profile_data, overwrite=False) File "/usr/lib/python3.10/site-packages/ipaserver/install/cainstance.py", line 2165, in _create_dogtag_profile with api.Backend.ra_certprofile as profile_api: File "/usr/lib/python3.10/site-packages/ipaserver/plugins/dogtag.py", line 1211, in __enter__ raise errors.RemoteRetrieveError(reason=_('Failed to authenticate to CA REST API')) 2024-06-25T23:36:51Z DEBUG The ipa-server-upgrade command failed, exception: RemoteRetrieveError: Failed to authenticate to CA REST API 2024-06-25T23:36:51Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details: RemoteRetrieveError: Failed to authenticate to CA REST API 2024-06-25T23:36:51Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information Which is still the exact failure mode as mentioned in the original post. in /var/log/pki/pki-tomcat/ca/debug.<date>.log I see the following exception: 2024-07-07 14:49:29 [main] FINEST: Getting internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca 2024-07-07 14:49:29 [main] FINEST: Property tcp.keepAlive not found 2024-07-07 14:49:29 [main] FINEST: Getting tcp.keepAlive=true 2024-07-07 14:49:29 [main] FINE: TCP Keep-Alive: true 2024-07-07 14:49:29 [main] FINE: LdapBoundConnection: Connecting to ipa.mydomain.tld:636 with client cert auth 2024-07-07 14:49:29 [main] FINE: ldapconn/PKISocketFactory.makeSSLSocket: begins 2024-07-07 14:49:29 [main] FINE: SignedAuditLogger: event CLIENT_ACCESS_SESSION_ESTABLISH 2024-07-07 14:49:29 [main] SEVERE: Unable to create socket: java.net.ConnectException: Connection refused java.net.ConnectException: Connection refused [... traceback ...] 2024-07-07 14:49:29 [main] SEVERE: LdapBoundConnFactory: Unable to connect to LDAP server: Unable to create socket: java.net.ConnectException: Connection refused netscape.ldap.LDAPException: Unable to create socket: java.net.ConnectException: Connection refused (-1) [... traceback ...] 2024-07-07 14:49:29 [main] SEVERE: Unable to start CA engine: Unable to connect to LDAP server: Unable to create socket: java.net.ConnectException: Connection refused Unable to connect to LDAP server: Unable to create socket: java.net.ConnectException: Connection refused [... traceback ...] 2024-07-07 14:49:29 [main] INFO: Shutting down CA subsystem 2024-07-07 14:49:29 [main] SEVERE: Exception sending context initialized event to listener instance of class [org.dogtagpki.server.ca.CAEngine] java.lang.NullPointerException: Cannot invoke "com.netscape.cmscore.authentication.AuthSubsystem.shutdown()" because "this.authSubsystem" is null [... traceback ...] 2024-07-07 14:49:29 [main] INFO: Shutting down CA subsystem 2024-07-07 14:49:29 [main] SEVERE: Exception sending context destroyed event to listener instance of class [org.dogtagpki.server.ca.CAEngine] java.lang.NullPointerException: Cannot invoke "com.netscape.cmscore.authentication.AuthSubsystem.shutdown()" because "this.authSubsystem" is null [... traceback ...] So as I understand it for some reason after the upgrade the CA subsystem is unable to start because the cert auth to ldap is failing. Which was working fine in f35 - even with no reissued certs. And it fails in f36 - even with reissued certs. However, I did not explicitly revoke the old certs (so they are still in LDAP as valid). Then I followed https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/ I also checked that /etc/pki/pki-tomcat/alias contains the correct renewed cert for 'subsystemCert cert-pki-ca', which should be the one used for ldap auth, which it was. However, I was unable to extract e.g. private keys: $ grep internal /var/lib/pki/pki-tomcat/conf/password.conf | cut -d= -f2 > /tmp/pwdfile.txt $ certutil -K -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -f /tmp/pwdfile.txt certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" certutil: problem listing keys: SEC_ERROR_INVALID_ARGS: security library: invalid arguments. But it seems this was not actually important. While searching for this particular error I found https://pagure.io/freeipa/issue/9381 via https://lists.pagure.io/archives/list/[email protected]/thread/BQ5JOYFNM6O3CASSJUYWWC3QGJSMSI64/ Adding the missing drop-in fixed it, it was yet another race condition. Best regards Johannes On Tue, 25 Jun 2024 at 03:50, Rob Crittenden <[email protected]> wrote: > Johannes Falke via FreeIPA-users wrote: > > Okay, I double checked the timestamps and read out the certs from the > > pki-tomcat alias NSSDB and realised that this is not the case, it was > > last modified exactly when the new certs were issued and it is perfectly > > in sync with with the certs in LDAP. > > > > Then I'm still stumped why my freeipa cert list looks like: > > https://i.imgur.com/JjgoWTy.png in the web frontend - anyone have an > idea? > > That could be only a small view of things. If you look at the certs in > the db for the serial numbers you can check the individually in IPA. > > I don't use the UI but from the cli: ipa cert-show <serial-number> > > The output will include status. > > If you provide the upgrade log, or at least the last bit which > demonstrates the failure, that could be enlightening. > > rob > > > > > > > > > > > > > > > On Mon, 24 Jun 2024 at 16:07, Johannes Falke <[email protected] > > <mailto:[email protected]>> wrote: > > > > Dear Alexander, > > > > On Wednesday, 21 December 2022 10:10 a.m., Martin (Lists) > > < > https://lists.fedorahosted.org/archives/users/233826406258280183065251682616278220915/ > > > > wrote: > > > > Hallo all > > I have a strange issue with one of my ipa servers. after an > > upgrade from > > fedora 35 to fedora 37 the ipa-server-upgrade failed on the > > pki-tomcat > > part. The ipaupgrade.log says: > > 2022-12-21T15:27:52Z INFO Migrating profile > > 'caECFullCMCSharedTokenCert' > > 2022-12-21T15:27:52Z DEBUG request GET > > https://ipa1.server.org:8443/ca/rest/account/login > > 2022-12-21T15:27:52Z DEBUG request body '' > > 2022-12-21T15:27:52Z DEBUG response status 404 > > 2022-12-21T15:27:52Z DEBUG response headers Content-Type: > > text/html;charset=utf-8 > > Content-Language: de > > Content-Length: 795 > > Date: Wed, 21 Dec 2022 15:27:52 GMT > > 2022-12-21T15:27:52Z DEBUG response body (decoded): b'<!doctype > > html><html lang="de"><head><title>HTTP Status 404 > > \xe2\x80\x93 nicht > > gefunden</title><style > > type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, > > h2, h3, > > b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 > > {font-size:16px; > > } h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line > > > > {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP > > Status 40 > > 4 \xe2\x80\x93 nicht gefunden</h1><hr class="line" > > /><p><b>Type</b> > > Status Report</p><p><b>Message</b> The requested resource > > [/ca/rest/account > > /login] is not available</p><p><b>Beschreibung</b> The > > origin server > > did not find a current representation for the target resource or > > is not > > willing to > > disclose that one exists.</p><hr class="line" /><h3>Apache > > Tomcat/9.0.68</h3></body></html>' > > 2022-12-21T15:27:52Z ERROR IPA server upgrade failed: Inspect > > /var/log/ipaupgrade.log and run command ipa-server-upgrade > manually. > > 2022-12-21T15:27:52Z DEBUG File > > "/usr/lib/python3.11/site-packages/ipapython/admintool.py", line > > 180, in > > execute > > return_value = self.run() > > ^^^^^^^^^^ > > File > > > "/usr/lib/python3.11/site-packages/ipaserver/install/ipa_server_upgrade.py", > > > > line 54, in run server.upgrade() > > File > > > "/usr/lib/python3.11/site-packages/ipaserver/install/server/upgrade.py", > > > > line 2061, in upgrade upgrade_configuration() > > File > > > "/usr/lib/python3.11/site-packages/ipaserver/install/server/upgrade.py", > > > > line 1914, in upgrade_configuration > > ca_enable_ldap_profile_subsystem(ca) > > File > > > "/usr/lib/python3.11/site-packages/ipaserver/install/server/upgrade.py", > > > > line 458, in ca_enable_ldap_profile_subsystem > > cainstance.migrate_profiles_to_ldap() > > File > > > "/usr/lib/python3.11/site-packages/ipaserver/install/cainstance.py", > > > > line 2155, in migrate_profiles_to_ldap > > _create_dogtag_profile(profile_id, profile_data, overwrite=False) > > File > > > "/usr/lib/python3.11/site-packages/ipaserver/install/cainstance.py", > > > > line 2209, in _create_dogtag_profile with > > api.Backend.ra_certprofile > > as profile_api: > > File > > "/usr/lib/python3.11/site-packages/ipaserver/plugins/dogtag.py", > > line 1211, in __enter__ raise > > errors.RemoteRetrieveError(reason=_('Failed to authenticate to > > CA REST > > API')) > > 2022-12-21T15:27:52Z DEBUG The ipa-server-upgrade command failed, > > exception: RemoteRetrieveError: Failed to authenticate to CA > > REST API > > The catalina logfile says: > > 21-Dec-2022 16:27:26.946 SCHWERWIEGEND [main] > > org.apache.catalina.core.StandardContext.startInternal One or > more > > listeners failed to start. Full details will be found in the > > appropriate > > container log file > > 21-Dec-2022 16:27:26.948 SCHWERWIEGEND [main] > > org.apache.catalina.core.StandardContext.startInternal Context > > [/ca] > > startup failed due to previous errors > > the CA debug log file says: > > 2022-12-21 16:27:26 [main] FINE: LdapBoundConnection: Connecting > to > > ipa1.server.org:636 <http://ipa1.server.org:636> with client > > cert auth > > 2022-12-21 16:27:26 [main] FINE: > > ldapconn/PKISocketFactory.makeSSLSocket: begins > > 2022-12-21 16:27:26 [main] FINE: SignedAuditLogger: event > > CLIENT_ACCESS_SESSION_ESTABLISH > > 2022-12-21 16:27:26 [main] SEVERE: Unable to create socket: > > java.net.ConnectException: Verbindungsaufbau abgelehnt > > with many java traceback errors following. directory server is > > running > > at this time and there is no connection reported at the given > time. > > ipa-healthceck does not give anny errors or warnings. > > Re-starting the > > pki-tomcat server manually afterwards ist working fine and does > > not give > > any errors. starting ipa in force mode gives no errors as well. > > What can > > I do? > > Regards > > Martin > > > > > > FWIW, I used the hyperkitty web reply link, which gives a direct > > mailto link and no option to add anything, and unfortunately didn't > > realize the context would be missing. Sorry about that (and for the > > double reply spam). > > I was replying to > > > https://lists.fedorahosted.org/archives/list/[email protected]/message/5PC566RTR2XDCSV5MYLM7QJZBXART535/ > > (also quoted at the start of this e-mail this time). > > > > I see the exact same error as above upgrading either F35->F36 or > > F35->F37 (and I also tried 37->38 on top in case it is somehow > > resolved by some later patch). > > I am currently on F35/freeipa-server 4.9.11. And ipa-healthcheck > > finds no issues (besides missing URI records). It's a single master > > server. > > > > The second error mentioned later in the thread > > (55-pbacmemberof.update) for me does in fact show up in an earlier > > upgrade to reach Fedora 35/freeipa-server 4.9.11. However, the > > upgrade returned exit 0 anyway on F34->F35. And the error does not > > occur again on the later attempted updates F35->F36 or F35->F37. So > > I think it's probably unrelated. > > > > On F35 everything *appears* to be working fine, no expired certs, > > except for the fact that the freeipa web interface seems to be > showing > > 1) a number of expired certs (old ones) > > 2) a number of greyed-out certs without name (and higher serial # - > > the actual renewed ones) > > getcert list shows all certs as normal (I assume it grabs them from > > LDAP) - but I see that /etc/pki/pki-tomcat/alias NSSDB last > > modification date is from before their issuing - so it seems it was > > not updated properly for some reason. > > > > I suspect this might be related to the upgrade failure I am seeing. > > Is there some easy way to refresh the pki-tomcat alias NSSDB without > > reissuing certs from those stored in LDAP? > > > > > > Best wishes, > > Johannes > > > > On Mon, 24 Jun 2024 at 08:44, Alexander Bokovoy <[email protected] > > <mailto:[email protected]>> wrote: > > > > On Няд, 23 чэр 2024, Johannes Falke via FreeIPA-users wrote: > > > How did you actually manage to resolve this issue? I'm > > seeing the same > > > thing trying to upgrade either f35->f37 or f35->f36 (and NO > > ldap errors). > > > On f35, freeipa says it's healthy. > > > > > > Just a reminder: this is a mailing list, not a forum. If you are > > using > > lists.fedorahosted.org <http://lists.fedorahosted.org> web > > interface, make sure to check the box to > > quote the email you are answering to because otherwise this > message > > appears without a reference to the thread you are trying to > > reply to. > > > > As such, this email contains no reference to existing discussion > > of the > > same topic and it is literally impossible to identify what do > > you mean > > by 'resolve this issue'. > > > > Also, lists.fedorahosted.org <http://lists.fedorahosted.org> web > > interface gives you a way to search the > > list archives and see the whole discussion there. It should help > in > > searching for successful resolutions of many problems. If you > still > > cannot find a solution, please make sure to include enough > > details in > > your new email thread to provide information that will allow list > > members to help you. > > > > > > > > > > -- > > / Alexander Bokovoy > > Sr. Principal Software Engineer > > Security / Identity Management Engineering > > Red Hat Limited, Finland > > > > > > -- > > _______________________________________________ > > FreeIPA-users mailing list -- [email protected] > > To unsubscribe send an email to > [email protected] > > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: > https://lists.fedorahosted.org/archives/list/[email protected] > > Do not reply to spam, report it: > https://pagure.io/fedora-infrastructure/new_issue > > > >
-- _______________________________________________ FreeIPA-users mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/[email protected] Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
