Something else to try might be to delete the .mgr pool and let it be recreated.
> On Jun 28, 2026, at 9:31 AM, Eugen Block via ceph-users <[email protected]> > wrote: > > Hi, > > I don’t have a response to all your questions, but will try to answer a few. > > Zitat von Chris Palmer via ceph-users <[email protected]>: > >> Due to a number of power and heat events, I had to power down a small ceph >> cluster last night. There was only time to do a operating system shutdown of >> all nodes. >> >> When the systems rebooted a few hours later, all was ok except that all 3 >> MGRs would abort and restart until they reached the restart limit. This left >> the system with no MGRs and no orchestration. All OSD, MDS, RGW & monitoring >> subsystems were working as normally as could be expected without an MGR. >> >> Analysis of the log files showed each MGR aborting with the assertion below. >> Every one was in the telemetry thread. I disabled telemetry and the MGRs >> could be restarted normally. Questions: >> >> * What caused the assertion error? Was it related to the total shutdown? >> * Aborting the entire MGR seems overkill. Disabling the telemetry >> module and raising a warning seems more proportionate. >> * When running with no active MGR the client IO stats in ceph -s were >> frozen (at plausible non-zero values). Is that expected with no >> active MGR? > > Yes, OSDs report their stats to the MGR which is responsible for displaying > those in tje status. > >> * With no active MGR a large number of deep-scrubs started - roughly 3 >> times the maximum number we ever see were actively deep-scrubbing, >> with none in the scrub wait state, and response times were horrible. >> Is this expected with no MGR to control it properly? I had to >> temporarily disable scrubs and deep-scrubs as a workaround. > > I see the same after restarting an entire cluster (mostly in small lab > environments), I don’t think that has to do with no active mgr. Although mgr > and osd have each a deep-scrub interval, the OSDs are responsible for > scheduling. So this is most likely just how OSDs work after a cluster reboot. > >> I haven't yet tried 20.2.2, but I couldn't see anything obvious in the >> release notes. (Although there were about 3 PRs that I thought might have >> had the potential to upset exception handling...). >> >> Thanks, Chris >> >> >> { >> "crash_id": >> "2026-06-26T01:46:00.265207Z_b0579062-d65c-4a51-8dbc-d9837209aa80", >> "timestamp": "2026-06-26T01:46:00.265207Z", >> "process_name": "ceph-mgr", >> "entity_name": "mgr.ceph2", >> "ceph_version": "20.2.1", >> "utsname_hostname": "ceph2", >> "utsname_sysname": "Linux", >> "utsname_release": "7.0.12-201.fc44.x86_64", >> "utsname_version": "#1 SMP PREEMPT_DYNAMIC Thu Jun 11 01:30:16 UTC 2026", >> "utsname_machine": "x86_64", >> "os_name": "CentOS Stream", >> "os_id": "centos", >> "os_version_id": "9", >> "os_version": "9", >> "assert_condition": "cursor != root", >> "assert_func": "virtual void PyFormatter::close_section()", >> "assert_file": "/ceph/rpmbuild/BUILD/ceph-20.2.1/src/mgr/PyFormatter.h", >> "assert_line": 84, >> "assert_thread_name": "telemetry", >> "assert_msg": "/ceph/rpmbuild/BUILD/ceph-20.2.1/src/mgr/PyFormatter.h: In >> function 'virtual void PyFormatter::close_section()' thread 7f752186e640 >> time >> 2026-06-26T01:46:00.263380+0000\n/ceph/rpmbuild/BUILD/ceph-20.2.1/src/mgr/PyFormatter.h: >> 84: FAILED ceph_assert(cursor != root)\n", >> "backtrace": [ >> "/lib64/libc.so.6(+0x3fc30) [0x7f75a9eb8c30]", >> "/lib64/libc.so.6(+0x8d05c) [0x7f75a9f0605c]", >> "raise()", >> "abort()", >> "(ceph::__ceph_assert_fail(char const*, char const*, int, char >> const*)+0x18e) [0x7f75aa9ed95b]", >> "/usr/bin/ceph-mgr(+0x6f7a5) [0x55ac462367a5]", >> "(ActivePyModules::get_perf_schema_python(std::__cxx11::basic_string<char, >> std::char_traits<char>, std::allocator<char> > const&, >> std::__cxx11::basic_string<char, std::char_traits<char>, >> std::allocator<char> > const&)+0x15a3) [0x55ac462d3143]", >> "/usr/bin/ceph-mgr(+0x12ba12) [0x55ac462f2a12]", >> "/lib64/libpython3.9.so.1.0(+0x15b4b7) [0x7f75ab5e34b7]", >> "_PyEval_EvalFrameDefault()", >> "/lib64/libpython3.9.so.1.0(+0x14169b) [0x7f75ab5c969b]", >> "_PyEval_EvalFrameDefault()", >> "/lib64/libpython3.9.so.1.0(+0x12ec69) [0x7f75ab5b6c69]", >> "_PyFunction_Vectorcall()", >> "_PyEval_EvalFrameDefault()", >> "/lib64/libpython3.9.so.1.0(+0x12ec69) [0x7f75ab5b6c69]", >> "_PyFunction_Vectorcall()", >> "_PyEval_EvalFrameDefault()", >> "/lib64/libpython3.9.so.1.0(+0x12ec69) [0x7f75ab5b6c69]", >> "_PyFunction_Vectorcall()", >> "_PyEval_EvalFrameDefault()", >> "/lib64/libpython3.9.so.1.0(+0x12ec69) [0x7f75ab5b6c69]", >> "_PyFunction_Vectorcall()", >> "_PyEval_EvalFrameDefault()", >> "/lib64/libpython3.9.so.1.0(+0x14169b) [0x7f75ab5c969b]", >> "/lib64/libpython3.9.so.1.0(+0x14fa85) [0x7f75ab5d7a85]", >> "/lib64/libpython3.9.so.1.0(+0x138725) [0x7f75ab5c0725]", >> "_PyObject_CallMethod_SizeT()", >> "(PyModuleRunner::serve()+0x66) [0x55ac463759e6]", >> "(PyModuleRunner::PyModuleRunnerThread::entry()+0x139) [0x55ac463761e9]", >> "/lib64/libc.so.6(+0x8b31a) [0x7f75a9f0431a]" >> ] >> } >> _______________________________________________ >> ceph-users mailing list -- [email protected] >> To unsubscribe send an email to [email protected] > > > _______________________________________________ > ceph-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
