Hi Roman, what ceph version are you on? Also, when you ran the restart command originally, did you get a message about scheduling the restarts or no output?
On Tue, Nov 23, 2021 at 6:04 AM Roman Steinhart <[email protected]> wrote: > Hi all, > > while digging down another issue I had with the managers I restarted them > using "ceph orch restart mgr". > After running that command the main manager is now in a restart loop. > The only way for me to stop this is running "ceph orch pause", as soon as I > do "ceph orch resume" the loop starts again. > > Has anyone a suggestion on how I can remove that stuck "restart" job? > "ceph orch cancel" does not work, it returns "This Orchestrator does not > support `orch cancel`" > > While googling for this issue it seems I'm not the first one having that > issue: > > https://lists.ceph.io/hyperkitty/list/[email protected]/thread/IRC4ZPSSZYELWPU5D2FHKWJ2VU7IP3JG/ > > https://lists.ceph.io/hyperkitty/list/[email protected]/message/WZUWZGVYPFII5HTUTW6OAQXIZ6VNT2E2/ > > See the logs below: > user@ceph1:~# journalctl -fu > [email protected] --since "1 day > ago" -g "Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a" > Nov 22 15:31:38 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 15:31:51 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:31:51 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:34:49 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 15:35:00 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:35:00 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:38:39 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 15:38:50 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:38:50 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:43:35 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 15:43:46 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:43:46 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:50:19 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 15:50:30 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:50:30 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:58:26 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 15:58:37 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:58:37 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:10:16 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 16:10:28 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:10:28 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:17:17 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 16:17:29 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:17:29 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:40:52 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 16:41:07 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:41:07 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:45:34 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 16:45:47 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:45:47 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:49:34 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 16:49:39 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:49:39 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:51:47 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 16:51:51 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:51:51 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:54:02 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 16:54:08 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:54:08 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:57:37 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 16:57:41 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:57:41 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:59:58 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 17:00:02 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 17:00:02 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 17:02:42 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 17:02:45 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 17:02:45 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 17:04:40 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 17:04:43 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 17:04:43 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 17:07:28 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 17:07:35 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 17:07:35 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 17:10:50 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 17:10:54 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 17:10:54 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > > Thanks in advance > ~ Roman > _______________________________________________ > ceph-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > > _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
