>>> >>> In my case it will be used to "adopt" the services, but after that what is >>> its intended usage? The "orchestrator" should be used to add/remove new >>> hosts and to upgrade the cluster. >> Cephadm *is* the orchestrator. The nomenclature may seem misleading. >> There’s part that runs on each node as well as a Manager module, and CLI >> commands. > > Understand. My question was around the "cephadm" command, the one > documentation says that has to be installed on all the nodes to start the > "adoption" process. I'm not sure if this command is needed only for that > process or is an integral part of the cephadm/orchestrator.
Both. >>> I'm asking this question because my upgrade path consists in multiple >>> consecutive upgrades and I see that for the latest version of Quincy >>> (17.2.9) and for newer Ceph releases there is no cephadm "package" for el8 >> Check the compatibility matrix in the docs. > > Mhmm... I suspect that I'm entering in some edge case here (end exiting the > subject of this thread). Edge cases keep this list spicy. > I have a cluster running Ceph Octopus 15.2.17 on EL8 (almost all the nodes, > some nodes are EL7 but can be reinstalled/removed). Yep, you’ll want to bump those to EL8 as a first step. Depending on your setup you may be able to do that without having to recreate the OSDs. Any OSDs that are still Filestore or with bluestore_min_alloc_size > 4KiB will want to be redeployed at some point though, which given what I’ve been reading lately should probably be done on Pacific, Reef, or Quincy. Unless you have coarse-IU QLC. > My goal is to upgrade it AND to embrace cephadm as orchestrator. As well it should — it will take multiple steps to get there. > My plan is/was: > > 1. Adopt cephadm (with octopus containers) > 2. Upgrade to Quincy. > 3. Upgrade to Squid. > > But now looking to the various compatibility matrices I'm start to worry that > this is not possible: > > - Podman on EL8 is version 4, but from the table > https://docs.ceph.com/en/squid/cephadm/compatibility/#cephadm-compatibility-with-podman > podman will not run with Octopus. I think updating to Pacific or Quincy before messing with containerization is the way to go. Pacific and Quincy are expected, vis-a-vis that table, to work with Podman 3+. Note this section in the Quincy release notes: Users should expect to see the el8 rpm subdirectory empty and the “dnf” commands are expected to fail with 17.2.9. They can choose to use 17.2.9 RPM packages for centos 8/el8 provided by CERN as a community member or stay at 17.2.7 following instructions from https://docs.ceph.com/en/latest/install/get-packages/#rhel, the ceph.repo file should point to https://download.ceph.com/rpm-17.2.7/el8 instead of https://download.ceph.com/rpm-quincy/el8 > - There is no mention of EL8 in > https://docs.ceph.com/en/squid/start/os-recommendations/ You’ll want to be on EL9 before going to Squid. > I could skip the adoption with Octopus containers and go directly with > adoption+upgrade to Quincy. I do recommend minimizing the changes you make in one step. Maybe something like: * Update EL7 hosts to EL8 * Update Ceph packages to Quincy * Update hosts to EL9 * Adopt OSDs * Update Ceph to Squid > But I'll end up with in any case with most hosts running EL8 which I cannot > find in any matrices. EL8 is an ex-parrot. > So the best solution will be to upgrade all the machines to EL9 and > concurrently upgrade Ceph to Quincy. See above re one thing at a time. Trust me on this. > For the MON and MGR shouldn't be too much of a problem (it should be possible > to add a new monitor with newer ceph release to an existing cluster), but how > to reinstall the OSD servers without data movement? This varies depending on your deployment, so I hesitate to make an authoritative statement. Depending on whether one has RHEL, CentOS*, and/or Rocky, the options for in-situ OS upgrades vary. Are you using LUKS-encrypted OSDs aka dmcrypt? If you can update to 8 then 9 in-situ, then the OSDs should just work. Worst case you manually activate them after the OS update. One node at a time. If you can’t update in-situ, or want to change your partitioning (get rid of swap, stop balkanizing the boot volume, etc), then you can have your OS provisioning system protect the OSD devices. You don’t have boot and OSDs on the same devices, I hope? Kickstart: ignoredisk, clearpart —drives, etc. Don’t assume that the drive enumeration and usage is the same on every system. Set the `noout` flag, reimage the OS, install the Ceph packages, reboot. The OSDs may come right up. If they don’t, try activating them. Lather, rinse, repeat. Unset noout. Worst case, wipe one system entirely and deploy new OSDs, unset noout, once recovery completes, move to the next. > In theory it should suffice to put the cluster in 'noout', reinstall the host > without touching the OSD disks and then recreate the OSDs. Is this possible? I’ve done it, when circumstances required moving from Ubuntu Precise to RHEL 7 in-situ. It’s easier if you’re using ceph-volume than with ceph-disk, but possible either way. > > Thank you for your time! > Iztok > > > > _______________________________________________ > ceph-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
