Hi,
I don’t have a response to all your questions, but will try to answer a few.
Zitat von Chris Palmer via ceph-users <[email protected]>:
Due to a number of power and heat events, I had to power down a
small ceph cluster last night. There was only time to do a operating
system shutdown of all nodes.
When the systems rebooted a few hours later, all was ok except that
all 3 MGRs would abort and restart until they reached the restart
limit. This left the system with no MGRs and no orchestration. All
OSD, MDS, RGW & monitoring subsystems were working as normally as
could be expected without an MGR.
Analysis of the log files showed each MGR aborting with the
assertion below. Every one was in the telemetry thread. I disabled
telemetry and the MGRs could be restarted normally. Questions:
* What caused the assertion error? Was it related to the total shutdown?
* Aborting the entire MGR seems overkill. Disabling the telemetry
module and raising a warning seems more proportionate.
* When running with no active MGR the client IO stats in ceph -s were
frozen (at plausible non-zero values). Is that expected with no
active MGR?
Yes, OSDs report their stats to the MGR which is responsible for
displaying those in tje status.
* With no active MGR a large number of deep-scrubs started - roughly 3
times the maximum number we ever see were actively deep-scrubbing,
with none in the scrub wait state, and response times were horrible.
Is this expected with no MGR to control it properly? I had to
temporarily disable scrubs and deep-scrubs as a workaround.
I see the same after restarting an entire cluster (mostly in small lab
environments), I don’t think that has to do with no active mgr.
Although mgr and osd have each a deep-scrub interval, the OSDs are
responsible for scheduling. So this is most likely just how OSDs work
after a cluster reboot.
I haven't yet tried 20.2.2, but I couldn't see anything obvious in
the release notes. (Although there were about 3 PRs that I thought
might have had the potential to upset exception handling...).
Thanks, Chris
{
"crash_id":
"2026-06-26T01:46:00.265207Z_b0579062-d65c-4a51-8dbc-d9837209aa80",
"timestamp": "2026-06-26T01:46:00.265207Z",
"process_name": "ceph-mgr",
"entity_name": "mgr.ceph2",
"ceph_version": "20.2.1",
"utsname_hostname": "ceph2",
"utsname_sysname": "Linux",
"utsname_release": "7.0.12-201.fc44.x86_64",
"utsname_version": "#1 SMP PREEMPT_DYNAMIC Thu Jun 11 01:30:16 UTC 2026",
"utsname_machine": "x86_64",
"os_name": "CentOS Stream",
"os_id": "centos",
"os_version_id": "9",
"os_version": "9",
"assert_condition": "cursor != root",
"assert_func": "virtual void PyFormatter::close_section()",
"assert_file": "/ceph/rpmbuild/BUILD/ceph-20.2.1/src/mgr/PyFormatter.h",
"assert_line": 84,
"assert_thread_name": "telemetry",
"assert_msg":
"/ceph/rpmbuild/BUILD/ceph-20.2.1/src/mgr/PyFormatter.h: In function
'virtual void PyFormatter::close_section()' thread 7f752186e640 time
2026-06-26T01:46:00.263380+0000\n/ceph/rpmbuild/BUILD/ceph-20.2.1/src/mgr/PyFormatter.h: 84: FAILED ceph_assert(cursor !=
root)\n",
"backtrace": [
"/lib64/libc.so.6(+0x3fc30) [0x7f75a9eb8c30]",
"/lib64/libc.so.6(+0x8d05c) [0x7f75a9f0605c]",
"raise()",
"abort()",
"(ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x18e) [0x7f75aa9ed95b]",
"/usr/bin/ceph-mgr(+0x6f7a5) [0x55ac462367a5]",
"(ActivePyModules::get_perf_schema_python(std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&,
std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > const&)+0x15a3) [0x55ac462d3143]",
"/usr/bin/ceph-mgr(+0x12ba12) [0x55ac462f2a12]",
"/lib64/libpython3.9.so.1.0(+0x15b4b7) [0x7f75ab5e34b7]",
"_PyEval_EvalFrameDefault()",
"/lib64/libpython3.9.so.1.0(+0x14169b) [0x7f75ab5c969b]",
"_PyEval_EvalFrameDefault()",
"/lib64/libpython3.9.so.1.0(+0x12ec69) [0x7f75ab5b6c69]",
"_PyFunction_Vectorcall()",
"_PyEval_EvalFrameDefault()",
"/lib64/libpython3.9.so.1.0(+0x12ec69) [0x7f75ab5b6c69]",
"_PyFunction_Vectorcall()",
"_PyEval_EvalFrameDefault()",
"/lib64/libpython3.9.so.1.0(+0x12ec69) [0x7f75ab5b6c69]",
"_PyFunction_Vectorcall()",
"_PyEval_EvalFrameDefault()",
"/lib64/libpython3.9.so.1.0(+0x12ec69) [0x7f75ab5b6c69]",
"_PyFunction_Vectorcall()",
"_PyEval_EvalFrameDefault()",
"/lib64/libpython3.9.so.1.0(+0x14169b) [0x7f75ab5c969b]",
"/lib64/libpython3.9.so.1.0(+0x14fa85) [0x7f75ab5d7a85]",
"/lib64/libpython3.9.so.1.0(+0x138725) [0x7f75ab5c0725]",
"_PyObject_CallMethod_SizeT()",
"(PyModuleRunner::serve()+0x66) [0x55ac463759e6]",
"(PyModuleRunner::PyModuleRunnerThread::entry()+0x139) [0x55ac463761e9]",
"/lib64/libc.so.6(+0x8b31a) [0x7f75a9f0431a]"
]
}
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
