Hi,
some comments inline.
Zitat von Iztok Gregori via ceph-users <[email protected]>:
Hi Eugen!
Thanks for the support, my response is inline
On 14/04/26 13:03, Eugen Block via ceph-users wrote:
Hi,
I don't have a solution (yet?), but a couple of points to clarify.
- I assume your cluster isn't managed by cephadm, correct?
Correct! It's an old cluster initialy deployed with ceph-deploy in
the Hammer days and upgraded since then. Right now is manually
managed (but one of the targets at the end of the upgrade is to
switch to cephadm).
- Can you share the output of 'ceph mon dump'?
Sure!
# ceph mon dump
epoch 18
fsid 5bed1bf6-123f-4597-aaaa-d77931228548
last_changed 2026-04-08T20:23:59.781295+0200
created 2018-01-09T10:20:36.351967+0100
min_mon_release 15 (octopus)
election_strategy: 1
0: [v2:10.10.10.33:3300/0,v1:10.10.10.33:6789/0] mon.node3
1: [v2:10.10.10.34:3300/0,v1:10.10.10.34:6789/0] mon.node4
2: [v2:10.10.10.35:3300/0,v1:10.10.10.35:6789/0] mon.node5
dumped monmap epoch 18
So your cluster currently has quorum with 2 out of 3 MONs on Quincy
already, correct? Restarting the mon on node3 shouldn't be dangerous,
I think. You'd still have quorum with node4 and node5, even if node3
wouldn't be able to join. If it doesn't join as Octopus, you could
still upgrade it and retry with Quincy.
What I'm wondering about, why did you remove the MONs you wanted to
upgrade from Ceph? Is that recommended somewhere? Maybe you need to
run 'ceph mon add <name> <addr>' as well to let node1 or node2 join
the monmap again? I'm not sure, it has been a long time doing this
kind of stuff. ;-) But I don't think I ever removed a MON before
upgrading it, not even when reinstalling the OS.
Just for completeness sake, could you also share 'ceph -s' and 'ceph
versions'?
- There were a couple of reports back at O/P versions about MONs
not joining the quorum. Sometimes it worked after hours, sometimes
it didn't, and I'm still not aware of a real solution. One of our
customers has been facing severe MON sync delay, sometimes taking
around 40 minutes until a MON joined quorum. It turned out to be
caused by some paxos default settings, requring the MONs to do a
full sync instead of a diff sync. This has to do with millions of
snapshots they have in their cluster. What made it worse was the
MON's filesystems were on HDDs, increasing the sync time as well.
We didn't wait for a long time before stopping the new MON a regain
"control" of the cluster, maybe 10 minutes max...
- Inspecting debug logs on the syncing MON could help finding the
root cause.
With "syncing MON" you mean the MON which is just added to the
cluster, right? Collecting this kind of logs is on the to-do list,
but blocked because the "log problem" mentioned bellow.
Yes, that's what I meant. The missing logs are a problem, of course.
It does sound familiar, but I don't have a good idea right now except
for looking at directory and file permissions, apparmor/selinux
things, dmesg, stuff like that.
Some more comments inline.
[cut]
We taught, among other things, to:
- Restart the ceph-mon@node1 (centos7+ceph15.2.16) daemon. It is
the only daemon which wasn't restarted and when queried ("ceph
daemon mon.node1 mon_status") has in "extra_probe_peers" the IP
addresses of the original removed nodes, which are the same IPs as
the nodes we want to add. We aren't sure if this prevent the
ceph-mon to join, because we don't understand when/what
"extra_probe_peers" is doing.
According to the output above node1 is already out of the cluster,
are you referring to node3?
- Remove the node1 from the cluster, set min_mon_release to Quincy
and then try to add the new monitors.
Yes, sorry for the misunderstanding, I was referring to node3 in
both instances.
We are unsure about both approaches because if they don't work we
will have a "unstable" cluster with only 2 monitors and no much
space to operate.
To "complicate" the matter we notice that on node4 and node5 the
ceph- mon is not logging anymore to "/var/log/ceph/ceph*.log" even
tough a "mon_cluster_log_to_file: true". If I search for open FD
there is nothing in /var/log/ceph for the running ceph-mon...
Is there something in journalctl?
We increased the 'debug_mon' to 20 ('ceph daemon mon.node4 config
set debug_mon 20') and we check it with 'ceph daemon mon.node4
config show' but the output of 'journalctl -u [email protected]` is
not more verbose then before, the last line is from 2:00 this
morning and it's related to sudo:
apr 14 02:35:40 node4 sudo[881890]: ceph : PWD=/ ; USER=root ;
COMMAND=/usr/sbin/smartctl -x --json=o /dev/md127
Iztok
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
