Hi Steve, I saw that you already opened a ticket for the issue (thanks):
https://tracker.ceph.com/issues/76185. This problem is basically related to the env (FIPS is enabled). A PR with a candidate fix is under-review: https://github.com/ceph/ceph/pull/68638. We will backport this change to Tentacle. Best, Redo. On Mon, Apr 27, 2026 at 5:02 PM Barber, Steve W. (Fed) < [email protected]> wrote: > Hi Redo, > > The spec was very simple: > > service_type: mgmt-gateway > placement: > #label: mgmt > hosts: > - alfred > - cyclops > - mask > spec: > virtual_ip: (redacted)/24 > port: 443 > > As far as active mgr logs, at this point I’m not seeing anything unusual > in the logs from the active mgr host. If I attempt to remove the service > again (it was already theoretically removed) I get: > > $ ceph orch rm mgmt-gateway > Invalid service 'mgmt-gateway'. Use 'ceph orch ls' to list available > services. > > That generated these lines in the log: > > ... > Apr 27 10:35:03 alfred bash[3568828]: debug 2026-04-27T14:35:03.592+0000 > 7f21e7e80640 0 log_channel(audit) log [DBG] : from=‘mgr.42485063 (alfred’s > ip):0/3890111191' entity='mgr.alfred.afvnok' cmd={"prefix": "osd blocklist > ls", "format": "json"} : dispatch > Apr 27 10:35:04 alfred bash[3568828]: audit > 2026-04-27T14:35:03.598161+0000 mon.alfred (mon.2) 48315 : audit [DBG] > from='mgr.42485063 (alfred’s ip):0/3890111191' entity='mgr.alfred.afvnok' > cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch > Apr 27 10:35:04 alfred bash[3568828]: audit > 2026-04-27T14:35:03.598161+0000 mon.alfred (mon.2) 48315 : audit [DBG] > from='mgr.42485063 (alfred’s ip):0/3890111191' entity='mgr.alfred.afvnok' > cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch > Apr 27 10:35:06 alfred bash[3566093]: debug 2026-04-27T14:35:06.596+0000 > 7f4324bbe640 0 log_channel(audit) log [DBG] : from=‘client.42505047 -' > entity='client.admin' cmd=[{"prefix": "orch rm", "service_name": > "mgmt-gateway", "target": ["mon-mgr", ""]}]: dispatch > Apr 27 10:35:07 alfred bash[3568828]: audit > 2026-04-27T14:35:06.601835+0000 mgr.alfred.afvnok (mgr.42485063) 258116 : > audit [DBG] from='client.42505047 -' entity='client.admin' cmd=[{"prefix": > "orch rm", "service_name": "mgmt-gateway", "target": ["mon-mgr", ""]}]: > dispatch > Apr 27 10:35:07 alfred bash[3568828]: audit > 2026-04-27T14:35:06.601835+0000 mgr.alfred.afvnok (mgr.42485063) 258116 : > audit [DBG] from='client.42505047 -' entity='client.admin' cmd=[{"prefix": > "orch rm", "service_name": "mgmt-gateway", "target": ["mon-mgr", ""]}]: > dispatch > Apr 27 10:35:07 alfred bash[3568828]: cephadm > 2026-04-27T14:35:06.602299+0000 mgr.alfred.afvnok (mgr.42485063) 258117 : > cephadm [INF] Remove service mgmt-gateway > Apr 27 10:35:07 alfred bash[3568828]: cephadm > 2026-04-27T14:35:06.602299+0000 mgr.alfred.afvnok (mgr.42485063) 258117 : > cephadm [INF] Remove service mgmt.-gateway > Apr 27 10:35:18 alfred bash[3568828]: debug 2026-04-27T14:35:18.596+0000 > 7f21e7e80640 0 log_channel(audit) log [DBG] : from='mgr.42485063 (alfred’s > ip):0/3890111191' entity='mgr.alfred.afvnok' cmd={"prefix": "osd blocklist > ls", "format": "json"} : dispatch > Apr 27 10:35:19 alfred bash[3568828]: audit > 2026-04-27T14:35:18.598798+0000 mon.alfred (mon.2) 48316 : audit [DBG] > from='mgr.42485063 (alfred’s ip):0/3890111191' entity='mgr.alfred.afvnok' > cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch > Apr 27 10:35:19 alfred bash[3568828]: audit > 2026-04-27T14:35:18.598798+0000 mon.alfred (mon.2) 48316 : audit [DBG] > from='mgr.42485063 (alfred’s ip):0/3890111191' entity='mgr.alfred.afvnok' > cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch > ... > > But otherwise everything else looks pretty normal. > > This is still with 2 of the 3 mon/mgr nodes running the mgmt-gateway > service: > $ ceph orch ps | egrep 'mgr|mon|mgmt' > mgmt-gateway.cyclops cyclops running (6d) 5d ago > 6d 6188k - 1.26.1 c2348906c6ff 68ce6d58327a > mgmt-gateway.mask mask running (6d) 5d ago > 6d 5980k - 1.26.1 c2348906c6ff 329705c23aad > mgr.alfred.afvnok alfred *:8443,9283,8765 running (2w) 5d ago > 7M 564M - 20.2.1 fb63cba66eea 875883d08efc > mgr.cyclops.nifeym cyclops *:8443,9283,8765 running (2w) 5d ago > 7M 463M - 20.2.1 fb63cba66eea 676ec1753348 > mgr.mask.yfbdpf mask *:8443,9283,8765 running (2w) 5d ago > 7M 486M - 20.2.1 fb63cba66eea 6a1bdd37997c > mon.alfred alfred running (2w) 5d ago > 7M 525M 2048M 20.2.1 fb63cba66eea ce5bbb75d650 > mon.cyclops cyclops running (2w) 5d ago > 7M 571M 2048M 20.2.1 fb63cba66eea e675cf6714e4 > mon.mask mask running (2w) 5d ago > 7M 510M 2048M 20.2.1 fb63cba66eea 9a034ab6f132 > > And I still have the same health error. > > Let me know if there’s anything more specific you want me to show. > > Steve > > *From: *Redouane Kachach <[email protected]> > *Date: *Friday, April 24, 2026 at 4:32 AM > *To: *Barber, Steve W. (Fed) <[email protected]> > *Cc: *Ceph Users <[email protected]> > *Subject: *[EXTERNAL] Re: [ceph-users] Attempt to enable mgmt-gateway on > v20.2.1 failed, and now cephadm module is broken > > Hi Steve, > > Can you plz share the spec you used to configure the mgmt-gateway and the > active mgr logs? > > Best, > Redo. > > On Thu, Apr 23, 2026 at 6:56 PM Barber, Steve W. (Fed) via ceph-users < > [email protected]> wrote: > > I have a pre-production cluster running on Ubuntu 22.04 hosts using > cephadm for orchestration. I recently upgraded from 19.2.3 to 20.2.1, which > went completely smoothly. We are a government facility so we are required > to enable FIPS mode in the OS. I believe this last fact may be the cause of > the following: > > I wanted to enable mgmt-gateway so I made a service definition file to > enable it and applied it. It appeared to work, but then “ceph health > detail” showed: > > HEALTH_ERR Module 'cephadm' has failed: [digital envelope routines] > unsupported > [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: [digital envelope > routines] unsupported > Module 'cephadm' has failed: [digital envelope routines] unsupported > > The mgmt-gateway service was started on all 3 nodes however. > > $ ceph orch ps --service_name mgmt-gateway > NAME HOST PORTS STATUS REFRESHED AGE MEM > USE MEM LIM VERSION IMAGE ID CONTAINER ID > mgmt-gateway.alfred alfred running (4m) 3m ago 4m > 4492k - 1.26.1 c2348906c6ff 9343ff0e1ada > mgmt-gateway.cyclops cyclops running (3m) 3m ago 3m > 4492k - 1.26.1 c2348906c6ff 68ce6d58327a > mgmt-gateway.mask mask running (4m) 3m ago 4m > 4452k - 1.26.1 c2348906c6ff 329705c23aad > > I then attempted to remove the service definition with ceph orch rm, but > it only succeeded in removing it on 1 of the 3 nodes. The process is still > running on the other 2, and the warning is still present. “ceph mgr fail && > ceph orch rm mgmt-gateway” momentarily clears the warnings/error, but it > comes right back, and the process is still present on 2 nodes (as shown by > “ceph orch ps”). > > $ ceph mgr fail && ceph orch rm mgmt-gateway > Removed service mgmt-gateway > > $ ceph orch ps --service_name mgmt-gateway > NAME HOST PORTS STATUS REFRESHED AGE MEM > USE MEM LIM VERSION IMAGE ID CONTAINER ID > mgmt-gateway.cyclops cyclops running (15h) 15h ago 15h > 4492k - 1.26.1 c2348906c6ff 68ce6d58327a > mgmt-gateway.mask mask running (15h) 15h ago 15h > 4448k - 1.26.1 c2348906c6ff 329705c23aad > > $ ceph health detail > HEALTH_ERR Module 'cephadm' has failed: [digital envelope routines] > unsupported > [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: [digital envelope > routines] unsupported > Module 'cephadm' has failed: [digital envelope routines] unsupported > > Despite the 2 processes running, the configured service IP does not > respond to a ping or requests from curl. > > Since we were getting ready to move this cluster into production I decided > to look for help here rather than proceed experimenting further and risk > mucking it up worse. How can I (a) diagnose what’s actually going wrong; > (b) recover the cephadm module to get back to normal with no warnings; and > less importantly, (c) is there a way to get mgmt-gateway working in this > environment? A little googling and AI querying suggests it may be an issue > with certs/ciphers so maybe I can provide my own? (Yes I know I could set > up my own proxy; I was just trying to use the built-in tools where > possible.) > > Thanks in advance. > > > _______________________________________________ > ceph-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > > _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
