[Freeipa-users] Re: CentOS 7 FreeIPA upgrade, 4.5 to 4.6.8: certmonger hanger

Mon, 05 Feb 2024 06:16:49 -0800

Melissa,

I’ve been following your thread here as I also have a 4.5.x system that _really_ needs to be updated, but I’m nervous to touch it since it “works”.

How you testing this without breaking the instance? Im nervous I have issues like you are experiencing and it breaks everything. I would like to test and validate that the upgrade works before going all in. Care to share your process?

-Kevin

On Feb 5, 2024, at 7:21 AM, Melissa Ferreira da Silva Boiko via FreeIPA-users <[email protected]> wrote:


Thanks for the suggestion!

I spun a new CentOS 7 image with 7.9.2009 / FreeIPA 4.6.8 (which involved setting up the incus server to cgroups v1).  Then I tried creating a replica from the 4.5.  It again broke on pki-tomcatd, but with a somewhat baffling error that I didn't know what to do about:

      [3/30]: creating ACIs for admin
      [4/30]: creating installation admin user
      [5/30]: configuring certificate server instance
      [error] IOError: [Errno 13] Permission denied: '/tmp/tmpRJcwYV'
    Your system may be partly configured.
    Run /usr/sbin/ipa-server-install --uninstall to clean up.

This appears to be related to
Which apparently was fixed on FreeIPA 4.7.

I couldn't find a release with 4.7 so I tried again with a VM with CentOS 8-Stream, which is supposed to have 4.9.  Only yum/dnf couldn't find the ipa-server package at all; they'd list ipa-client, but not server.  I'm not familiar with RPM-based systems so it took a lot of digging (including trying my hand at Rocky Linux and localinstall from manual RPM downloads, which led to problems with dependencies) until I found out I had to add module_hotfixes=1 in the appstream.repo file.

With FreeIPA 4.9 once again the CA setup failed, with a new error:
  [28/30]: importing IPA certificate profiles
Lookup failed: Preferred host vm-ipa-half-3.intra.viaboxxsystems.de does not provide CA.
Lookup failed: Preferred host vm-ipa-half-3.intra.viaboxxsystems.de does not provide CA.
  [error] RemoteRetrieveError: Failed to authenticate to CA REST API
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

There were some hard-to-read HTTP errors on the install log so I was already getting ready to give up and try to create a replica from Docker or something.  But just as a hail-mary I run the uninstall scripts, rebooted, and tried again, and to my surprise this time pki-tomcatd install worked!  I got a warning then on the KDC step:

Configuring Kerberos KDC (krb5kdc)
  [1/1]: installing X509 Certificate for PKINIT
PKINIT certificate request failed: Certificate issuance failed (CA_UNREACHABLE: Server at https://vm-ipa-half-3.intra.vi
aboxxsystems.de/ipa/json failed request, will retry: 4035 (Request failed with status 500: Non-2xx response from CA REST
 API: 500. ).)
Failed to configure PKINIT
Full PKINIT configuration did not succeed
The setup will only install bits essential to the server functionality
You can enable PKINIT after the setup completed using 'ipa-pkinit-manage'

But ipa-replica-install finished with status: successful for the first time, and "ipa-pkinit-manage enable" worked.  I now have a FreeIPA 4.9.13 replica with CA; hopefully 4.11 will replicate from that without issues and then I can promote it to primary CA.

---

In summary, these are the issues I found so far trying to upgrade from FreeIPA 4.5.0:

Upgrade FreeIPA 4.5 / CentOS7 to the last CentOS 7 / FreeIPA 4.6: breaks due to certmonger timeout.
Replicate to current FreeIPA 4.11 / Fedora 39: breaks due to hash incompatibility.
Replicate to FreeIPA 4.6 / fresh CentOS 7: breaks due to systemd /tmp permissions error.
Replicate to FreeIPA 4.7: couldn't find lxc containers with this version.
Replicate to Freeipa 4.9 / CentOS 8-Stream: had to downgrade incus host to cgroups v1; first couple attempts failed, but eventually it worked.

[]s

Am Do., 1. Feb. 2024 um 17:07 Uhr schrieb Rob Crittenden <[email protected]>:
Melissa Ferreira da Silva Boiko via FreeIPA-users wrote:
> Hello all.
>
> I'm trying to replace an ancient FreeIPA 4.5.0 master (and primary CA
> master) on CentOS 7.4.  I am having problems trying to make replicas
> with FreeIPA 4.11, and past threads suggest the errors are due to
> incompatibility of password hash algorithms, which are supposed to be
> fixed on the older releases rather than the newer.
>
> Therefore I'm trying to upgrade the old server to the current version in
> the CentOS 7 repos, 4.6.8, to try to create fresh replicas from there. 
> But I'm having issues with the certmonger systemd service hanging, and
> breaking ipa-server-upgrade--whether I update the whole CentOS to
> 7.9.2009, or just ipa-server and its dependencies, the result is the same.
>
> This is where ipa-server-upgrade breaks:
>
>         [Verifying that root certificate is published]
>         [Migrate CRL publish directory]
>         CRL tree already moved
>         [Verifying that CA proxy configuration is correct]
>         IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and
> run command ipa-server-upgrade manually.
>         Unexpected error - see /var/log/ipaupgrade.log for details:
>         CalledProcessError: Command '/bin/systemctl start
> certmonger.service' returned non-zero exit status 1
>         The ipa-server-upgrade command failed. See
> /var/log/ipaupgrade.log for more information
>
> This is because certmonger.service hangs until timeout.  That happens
> when starting the service manually, too.  Logs for certmonger.service
> are not informative:
>
>         -- Subject: Unit certmonger.service has begun start-up
>         -- Unit certmonger.service has begun starting up.
>         Jan 31 14:38:59 vm-ipa-1.intra.viaboxxsystems.de
> <http://vm-ipa-1.intra.viaboxxsystems.de> systemd[1]: certmonger.service
> start operation timed out. Terminating.
>         Jan 31 14:40:29 vm-ipa-1.intra.viaboxxsystems.de
> <http://vm-ipa-1.intra.viaboxxsystems.de> systemd[1]: certmonger.service
> stop-sigterm timed out. Killing.
>         Jan 31 14:40:29 vm-ipa-1.intra.viaboxxsystems.de
> <http://vm-ipa-1.intra.viaboxxsystems.de> systemd[1]:
> certmonger.service: main process exited, code=killed, status=9/KILL
>         -- Subject: Unit certmonger.service has failed
>         -- Unit certmonger.service has failed.
>         Jan 31 14:40:29 vm-ipa-1.intra.viaboxxsystems.de
> <http://vm-ipa-1.intra.viaboxxsystems.de> systemd[1]: Unit
> certmonger.service entered failed state.
>         Jan 31 14:40:29 vm-ipa-1.intra.viaboxxsystems.de
> <http://vm-ipa-1.intra.viaboxxsystems.de> systemd[1]: certmonger.service
> failed.
>         [email protected]
> <mailto:[email protected]>[lxc](e:0,1s)(j:0) ~
>
> Running `certmonger -S -n -d 9` seems to run ok.  The only difference in
> the systemd service file is, I think, whatever it is that the BusName
> setting does.  dbus is running seemingly without issue, nothing on
> logs.  Restarting dbus.service doesn't help.
>
> The machine is an LXC container with 4GiB RAM, which doesn't come close
> to being exhausted when trying to restart certmonger.  No OOM in logs.
>
> I saw this thread about certmonger problems with ulimit in containers:
> https://bugzilla.redhat.com/show_bug.cgi?id=1656519
> But the suggested workaround (make sure ulimit -n is the same in
> container and host) doesn't apply because it's already the same for us.
>
> How should I proceed from here?

Why not create a new CentOS replica from the current one. Then use that
to upgrade to 8 and then 9?

rob

--
_______________________________________________
FreeIPA-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/[email protected]
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
--
_______________________________________________
FreeIPA-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedorahosted.org/archives/list/[email protected]
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Reply via email to