So... According to this: https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#changing-a-monitor-s-ip-address-advanced-method
I should be able to extract the monmap and get it inject again to create a new monitor. But I get an error when tried to inject: ceph-mon -i pve01 --inject-monmap monmap 2025-08-20T16:18:28.654-0300 7278e2496cc0 -1 unable to read magic from mon data Here's the monmap content: monmaptool --print monmap monmaptool: monmap file monmap epoch 3 fsid 28833d73-fb7e-48b6-bc0b-4ff95be9a7b7 last_changed 2025-08-20T15:07:35.677005-0300 created 2025-08-20T15:05:43.413542-0300 min_mon_release 19 (squid) election_strategy: 1 0: [v2:172.16.100.110:3300/0,v1:172.16.100.110:6789/0] mon.pve01 I tried to correct the user and group to all the content of the folder /var/lib/ceph/mon/ceph-pve01/store.db to ceph:ceph, but as soon as I execute the inject command, I got the error about unable to read magic from mon data. If I list those files: ls -l /var/lib/ceph/mon/ceph-pve01/store.db total 6392 -rw-r--r-- 1 ceph ceph 2325712 Aug 20 16:12 000008.sst -rw-r--r-- 1 ceph ceph 0 Aug 20 16:12 000009.log -rw-r--r-- 1 ceph ceph 0 Aug 20 16:16 000013.log -rw-r--r-- 1 ceph ceph 0 Aug 20 16:16 000017.log -rw-r--r-- 1 ceph ceph 0 Aug 20 16:16 000021.log -rw-r--r-- 1 ceph ceph 0 Aug 20 16:18 000025.log -rw-r--r-- 1 root root 0 Aug 20 16:18 000029.log -rw-r--r-- 1 root root 16 Aug 20 16:18 CURRENT -rw-r--r-- 1 ceph ceph 36 Aug 20 16:11 IDENTITY -rw-r--r-- 1 ceph ceph 0 Aug 20 16:11 LOCK -rw-r--r-- 1 root root 184 Aug 20 16:18 MANIFEST-000030 -rw-r--r-- 1 ceph ceph 7021 Aug 20 16:16 OPTIONS-000028 -rw-r--r-- 1 root root 7021 Aug 20 16:18 OPTIONS-000032 Em qua., 20 de ago. de 2025 às 15:45, Gilberto Ferreira < gilberto.nune...@gmail.com> escreveu: > >>>> Yes, I was just mid-draft of a response explaining the single steps. > oh... that would be nice... > > > > > > > Em qua., 20 de ago. de 2025 às 15:41, Eugen Block <ebl...@nde.ag> > escreveu: > >> Yes, I was just mid-draft of a response explaining the single steps. >> ;-) I'll ping Zac tomorrow to update the docs to include the actual >> list of hosts. >> >> Zitat von Gilberto Ferreira <gilberto.nune...@gmail.com>: >> >> > Well... Ends up I need to adapt the original script to this: >> > >> > #!/bin/bash >> > >> > ms=/root/monstore >> > host=/root/hosts -------- here there's a list of my 3 nodes, one per >> line >> > mkdir $ms >> > >> > # collect the cluster map from stopped OSDs >> > for host in $(cat /root/hosts); do >> > rsync -avz $ms/. root@$host:$ms.remote >> > rm -rf $ms >> > ssh root@$host <<EOF >> > for osd in /var/lib/ceph/osd/ceph-*; do >> > ceph-objectstore-tool --data-path \$osd --no-mon-config --op >> > update-mon-db --mon-store-path $ms.remote >> > done >> > EOF >> > rsync -avz root@$host:$ms.remote/. $ms >> > done >> > >> > Now on node1 I have the following folder: >> > >> > ls monstore/store.db/ >> > 000008.sst 000013.sst 000018.sst 000023.sst 000028.sst 000029.log >> > CURRENT IDENTITY LOCK MANIFEST-000030 OPTIONS-000027 >> OPTIONS-000032 >> > >> > I believe I am getting closer... >> > >> > >> > >> > >> > Em qua., 20 de ago. de 2025 às 15:19, Gilberto Ferreira < >> > gilberto.nune...@gmail.com> escreveu: >> > >> >> So right now I have this: >> >> pve01:~# find mon-store/ >> >> mon-store/ >> >> mon-store/ceph-0 >> >> mon-store/ceph-0/kv_backend >> >> mon-store/ceph-0/store.db >> >> mon-store/ceph-0/store.db/LOCK >> >> mon-store/ceph-0/store.db/IDENTITY >> >> mon-store/ceph-0/store.db/CURRENT >> >> mon-store/ceph-0/store.db/000004.log >> >> mon-store/ceph-0/store.db/MANIFEST-000005 >> >> mon-store/ceph-0/store.db/OPTIONS-000007 >> >> mon-store/ceph-1 >> >> mon-store/ceph-1/kv_backend >> >> mon-store/ceph-1/store.db >> >> mon-store/ceph-1/store.db/LOCK >> >> mon-store/ceph-1/store.db/IDENTITY >> >> mon-store/ceph-1/store.db/CURRENT >> >> mon-store/ceph-1/store.db/000004.log >> >> mon-store/ceph-1/store.db/MANIFEST-000005 >> >> mon-store/ceph-1/store.db/OPTIONS-000007 >> >> >> >> pve02:~# find mon-store/ >> >> mon-store/ >> >> mon-store/ceph-2 >> >> mon-store/ceph-2/kv_backend >> >> mon-store/ceph-2/store.db >> >> mon-store/ceph-2/store.db/LOCK >> >> mon-store/ceph-2/store.db/IDENTITY >> >> mon-store/ceph-2/store.db/CURRENT >> >> mon-store/ceph-2/store.db/000004.log >> >> mon-store/ceph-2/store.db/MANIFEST-000005 >> >> mon-store/ceph-2/store.db/OPTIONS-000007 >> >> mon-store/ceph-3 >> >> mon-store/ceph-3/kv_backend >> >> mon-store/ceph-3/store.db >> >> mon-store/ceph-3/store.db/LOCK >> >> mon-store/ceph-3/store.db/IDENTITY >> >> mon-store/ceph-3/store.db/CURRENT >> >> mon-store/ceph-3/store.db/000004.log >> >> mon-store/ceph-3/store.db/MANIFEST-000005 >> >> mon-store/ceph-3/store.db/OPTIONS-000007 >> >> >> >> pve03:~# find mon-store/ >> >> mon-store/ >> >> mon-store/ceph-4 >> >> mon-store/ceph-4/kv_backend >> >> mon-store/ceph-4/store.db >> >> mon-store/ceph-4/store.db/LOCK >> >> mon-store/ceph-4/store.db/IDENTITY >> >> mon-store/ceph-4/store.db/CURRENT >> >> mon-store/ceph-4/store.db/000004.log >> >> mon-store/ceph-4/store.db/MANIFEST-000005 >> >> mon-store/ceph-4/store.db/OPTIONS-000007 >> >> mon-store/ceph-5 >> >> mon-store/ceph-5/kv_backend >> >> mon-store/ceph-5/store.db >> >> mon-store/ceph-5/store.db/LOCK >> >> mon-store/ceph-5/store.db/IDENTITY >> >> mon-store/ceph-5/store.db/CURRENT >> >> mon-store/ceph-5/store.db/000004.log >> >> mon-store/ceph-5/store.db/MANIFEST-000005 >> >> mon-store/ceph-5/store.db/OPTIONS-000007 >> >> >> >> >> >> >> >> --- >> >> >> >> >> >> >> >> >> >> >> >> >> >> Em qua., 20 de ago. de 2025 às 15:15, Gilberto Ferreira < >> >> gilberto.nune...@gmail.com> escreveu: >> >> >> >>> Ok... >> >>> I am doing it again. >> >>> I have 2 osd per node. >> >>> Do I need to create multiple folder for each osd? >> >>> Like >> >>> node1: >> >>> mon-store/ceph-osd0 >> >>> mon-store/ceph-osd1 >> >>> node2: >> >>> mon-store/ceph-osd2 >> >>> mon-store/ceph-osd3 >> >>> node3: >> >>> mon-store/ceph-osd3 >> >>> mon-store/ceph-osd3 >> >>> >> >>> And than rsynced everything to one node let's say: >> >>> nove1: /root/mon-store? >> >>> >> >>> Which one I should use in order to restore or recreate the mon? >> >>> >> >>> Sorry for so many questions. >> >>> I am trying to understand the whole process, so bare with me. >> >>> >> >>> Thanks for your patience. >> >>> >> >>> >> >>> >> >>> --- >> >>> >> >>> >> >>> Gilberto Nunes Ferreira >> >>> +55 (47) 99676-7530 - Whatsapp / Telegram >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> Em qua., 20 de ago. de 2025 às 14:35, Eugen Block <ebl...@nde.ag> >> >>> escreveu: >> >>> >> >>>> I feel like there's still a misunderstanding here. >> >>>> >> >>>> The mentioned procedure is: >> >>>> >> >>>> ms=/root/mon-store >> >>>> mkdir $ms >> >>>> >> >>>> # collect the cluster map from stopped OSDs >> >>>> for host in $hosts; do >> >>>> rsync -avz $ms/. user@$host:$ms.remote >> >>>> rm -rf $ms >> >>>> ssh user@$host <<EOF >> >>>> for osd in /var/lib/ceph/osd/ceph-*; do >> >>>> ceph-objectstore-tool --data-path \$osd --no-mon-config --op >> >>>> update-mon-db --mon-store-path $ms.remote >> >>>> done >> >>>> EOF >> >>>> rsync -avz user@$host:$ms.remote/. $ms >> >>>> done >> >>>> >> >>>> >> >>>> It collects the clustermap on each host, querying each OSD, but then >> >>>> it "merges" it into one store, the local $ms store. That is used then >> >>>> to start up the first monitor. So however you do this, make sure you >> >>>> have all the clustermaps in one store. Did you stop the newly created >> >>>> mon first? And I don't care about the ceph-mon.target, that's always >> >>>> on to ensure the MON starts automatically after boot. >> >>>> >> >>>> Can you clarify that you really have all the clustermaps in one >> store? >> >>>> If not, you'll need to repeat the steps. In theory the steps should >> >>>> work exactly as they're described. >> >>>> >> >>>> Zitat von Gilberto Ferreira <gilberto.nune...@gmail.com>: >> >>>> >> >>>> > That's strange. >> >>>> > Now I have only the ceph-mon.target available: >> >>>> > >> >>>> > systemctl status ceph-mon.target >> >>>> > ● ceph-mon.target - ceph target allowing to start/stop all >> ceph-mon@ >> >>>> .service >> >>>> > instances at once >> >>>> > Loaded: loaded (/usr/lib/systemd/system/ceph-mon.target; >> enabled; >> >>>> > preset: enabled) >> >>>> > Active: active since Wed 2025-08-20 14:07:12 -03; 1min 47s ago >> >>>> > Invocation: 1fcbb21af715460294bd6d8549557ed9 >> >>>> > >> >>>> > Notice: journal has been rotated since unit was started, output >> may be >> >>>> > incomplete. >> >>>> > >> >>>> >>>> And you did rebuild the store from all OSDs as I mentioned, >> correct? >> >>>> > Yes... >> >>>> > Like that: >> >>>> > >> >>>> > ceph-volume lvm activate --all >> >>>> > mkdir /root/mon-store >> >>>> > ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 >> >>>> --no-mon-config >> >>>> > --op update-mon-db --mon-store-path mon-store/ >> >>>> > ceph-monstore-tool mon-store/ rebuild -- --keyring >> >>>> > /etc/pve/priv/ceph.client.admin.keyring --mon-ids pve01 pve02 pve03 >> >>>> > mv /var/lib/ceph/mon/ceph-pve01/store.db/ >> >>>> > /var/lib/ceph/mon/ceph-pve01/store.db-bkp >> >>>> > cp -rf mon-store/store.db/ /var/lib/ceph/mon/ceph-pve01/ >> >>>> > chown -R ceph:ceph /var/lib/ceph/mon/ceph-pve01/store.db >> >>>> > >> >>>> > On each node. >> >>>> > --- >> >>>> > >> >>>> > >> >>>> > Gilberto Nunes Ferreira >> >>>> > +55 (47) 99676-7530 - Whatsapp / Telegram >> >>>> > >> >>>> > >> >>>> > >> >>>> > >> >>>> > >> >>>> > >> >>>> > Em qua., 20 de ago. de 2025 às 13:49, Eugen Block <ebl...@nde.ag> >> >>>> escreveu: >> >>>> > >> >>>> >> What does the monitor log? Does it at least start successfully? >> And >> >>>> >> you did rebuild the store from all OSDs as I mentioned, correct? >> >>>> >> >> >>>> >> Zitat von Gilberto Ferreira <gilberto.nune...@gmail.com>: >> >>>> >> >> >>>> >> > Hi again... >> >>>> >> > I have reinstall all Proxmox nodes and install ceph on each >> node. >> >>>> >> > Create the mons and mgr on eatch node. >> >>>> >> > I have issue the command ceph-volume lvm activate --all, on each >> >>>> node, in >> >>>> >> > order bring up the /var/lib/ceph/osd/<node> >> >>>> >> > After that I ran this commands: >> >>>> >> > ceph-volume lvm activate --all >> >>>> >> > mkdir /root/mon-store >> >>>> >> > ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 >> >>>> >> --no-mon-config >> >>>> >> > --op update-mon-db --mon-store-path mon-store/ >> >>>> >> > ceph-monstore-tool mon-store/ rebuild -- --keyring >> >>>> >> > /etc/pve/priv/ceph.client.admin.keyring --mon-ids pve01 pve02 >> pve03 >> >>>> >> > mv /var/lib/ceph/mon/ceph-pve01/store.db/ >> >>>> >> > /var/lib/ceph/mon/ceph-pve01/store.db-bkp >> >>>> >> > cp -rf mon-store/store.db/ /var/lib/ceph/mon/ceph-pve01/ >> >>>> >> > chown -R ceph:ceph /var/lib/ceph/mon/ceph-pve01/store.db >> >>>> >> > >> >>>> >> > But now I got nothing! >> >>>> >> > No monitor, no manager, no osd, none! >> >>>> >> > >> >>>> >> > Perhaps somebody point me what I did wrong. >> >>>> >> > >> >>>> >> > Thanks >> >>>> >> > >> >>>> >> > Em qua., 20 de ago. de 2025 às 11:32, Gilberto Ferreira < >> >>>> >> > gilberto.nune...@gmail.com> escreveu: >> >>>> >> > >> >>>> >> >> I can see the content of the mentioned folders just after >> issue the >> >>>> >> >> command ceph-volume.... >> >>>> >> >> Thanks anyway. >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> Em qua., 20 de ago. de 2025 às 11:26, Eugen Block < >> ebl...@nde.ag> >> >>>> >> >> escreveu: >> >>>> >> >> >> >>>> >> >>> I assume you're right. Do you see the OSD contents in >> >>>> >> >>> /var/lib/ceph/osd/ceph-pve01 after activating? >> >>>> >> >>> And remember to collect the clustermap from all OSDs for this >> >>>> >> >>> procedure to succeed. >> >>>> >> >>> >> >>>> >> >>> Zitat von Gilberto Ferreira <gilberto.nune...@gmail.com>: >> >>>> >> >>> >> >>>> >> >>> > I see... >> >>>> >> >>> > >> >>>> >> >>> > But I had another problem. >> >>>> >> >>> > The script from (0) indicate that should be exist a >> >>>> /var/lib/ceph/osd >> >>>> >> >>> > folder, like: >> >>>> >> >>> > /var/lib/ceph/osd/ceph-pve01 >> >>>> >> >>> > /var/lib/ceph/osd/ceph-pve02 >> >>>> >> >>> > and so on. >> >>>> >> >>> > >> >>>> >> >>> > But this folder appears only if I run ceph-volume lvm >> activate >> >>>> --all. >> >>>> >> >>> > So my question is: when I should run this command: after or >> >>>> before >> >>>> >> use >> >>>> >> >>> the >> >>>> >> >>> > script? >> >>>> >> >>> > I think I need to run ceph-volume lvm activate --all, right? >> >>>> >> >>> > Just to clarify. >> >>>> >> >>> > >> >>>> >> >>> > Thanks >> >>>> >> >>> > >> >>>> >> >>> > Em qua., 20 de ago. de 2025 às 11:08, Eugen Block < >> >>>> ebl...@nde.ag> >> >>>> >> >>> escreveu: >> >>>> >> >>> > >> >>>> >> >>> >> Yes, you need a monitor. The mgr is not required and can be >> >>>> deployed >> >>>> >> >>> >> later. After you created the monitor, replace the mon store >> >>>> contents >> >>>> >> >>> >> by the collected clustermaps from the mentioned procedure. >> >>>> Keep the >> >>>> >> >>> >> ownerships of the directories/files in mind. If the monitor >> >>>> starts >> >>>> >> >>> >> successfully (with the original FSID), you can try to start >> >>>> one of >> >>>> >> the >> >>>> >> >>> >> OSDs. If that works, start the rest of them, wait for the >> >>>> peering >> >>>> >> >>> >> storm to settle, create two more monitors and two mgr >> daemons. >> >>>> >> >>> >> >> >>>> >> >>> >> Note that if you lose the mon store and you had a CephFS, >> >>>> you'll >> >>>> >> need >> >>>> >> >>> >> to recreate that from the existing pools. >> >>>> >> >>> >> >> >>>> >> >>> >> Zitat von Gilberto Ferreira <gilberto.nune...@gmail.com>: >> >>>> >> >>> >> >> >>>> >> >>> >> > Hi >> >>>> >> >>> >> > >> >>>> >> >>> >> > Do I need to create any mon and/or mgr in the new ceph >> >>>> cluster? >> >>>> >> >>> >> > >> >>>> >> >>> >> > >> >>>> >> >>> >> > >> >>>> >> >>> >> > Em seg., 18 de ago. de 2025 às 13:03, Eugen Block < >> >>>> ebl...@nde.ag> >> >>>> >> >>> >> escreveu: >> >>>> >> >>> >> > >> >>>> >> >>> >> >> Hi, >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> this sounds like you created a new cluster (new fsid), >> the >> >>>> OSDs >> >>>> >> >>> still >> >>>> >> >>> >> >> have the previous fsid configured. I'd rather recommend >> to >> >>>> follow >> >>>> >> >>> this >> >>>> >> >>> >> >> procedure [0] to restore the mon store utilizing OSDs >> >>>> rather than >> >>>> >> >>> >> >> trying to manipulate otherwise intact OSDs to fit into >> the >> >>>> "new" >> >>>> >> >>> >> >> cluster. That way you'll have "your" cluster back. I >> don't >> >>>> know >> >>>> >> if >> >>>> >> >>> >> >> there are any specifics to using proxmox, though. But >> the >> >>>> >> mentioned >> >>>> >> >>> >> >> procedure seems to work just fine, I've read multiple >> >>>> reports on >> >>>> >> >>> this >> >>>> >> >>> >> >> list. Luckily, I haven't had to use it myself. >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> Regards, >> >>>> >> >>> >> >> Eugen >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> [0] >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >>>> >> >> >>>> >> https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/#recovery-using-osds >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> Zitat von Gilberto Ferreira <gilberto.nune...@gmail.com >> >: >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> > Hi >> >>>> >> >>> >> >> > >> >>>> >> >>> >> >> > I have 3 nodes Proxmox Cluster with CEPH, and after a >> >>>> crash, I >> >>>> >> >>> have to >> >>>> >> >>> >> >> > reinstall Proxmox from scratch, along with Ceph. >> >>>> >> >>> >> >> > OSD are intact. >> >>>> >> >>> >> >> > I already did ceph-volume lvm activate --all and the >> OSD >> >>>> >> appears >> >>>> >> >>> with >> >>>> >> >>> >> >> > ceph-volum lvm list and I got a folder with the name >> of >> >>>> the OSD >> >>>> >> >>> under >> >>>> >> >>> >> >> > /var/lib/ceph/osd. >> >>>> >> >>> >> >> > However is not appear in ceph osd tree or ceph -s or >> even >> >>>> in >> >>>> >> the >> >>>> >> >>> web >> >>>> >> >>> >> gui. >> >>>> >> >>> >> >> > Is there any way to re-add this OSD to Proxmox CEPH? >> >>>> >> >>> >> >> > >> >>>> >> >>> >> >> > Thanks a lot for any help. >> >>>> >> >>> >> >> > >> >>>> >> >>> >> >> > >> >>>> >> >>> >> >> > Best Regards >> >>>> >> >>> >> >> > --- >> >>>> >> >>> >> >> > Gilbert >> >>>> >> >>> >> >> > _______________________________________________ >> >>>> >> >>> >> >> > ceph-users mailing list -- ceph-users@ceph.io >> >>>> >> >>> >> >> > To unsubscribe send an email to >> ceph-users-le...@ceph.io >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> _______________________________________________ >> >>>> >> >>> >> >> ceph-users mailing list -- ceph-users@ceph.io >> >>>> >> >>> >> >> To unsubscribe send an email to >> ceph-users-le...@ceph.io >> >>>> >> >>> >> >> >> >>>> >> >>> >> > _______________________________________________ >> >>>> >> >>> >> > ceph-users mailing list -- ceph-users@ceph.io >> >>>> >> >>> >> > To unsubscribe send an email to ceph-users-le...@ceph.io >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >>> >> _______________________________________________ >> >>>> >> >>> >> ceph-users mailing list -- ceph-users@ceph.io >> >>>> >> >>> >> To unsubscribe send an email to ceph-users-le...@ceph.io >> >>>> >> >>> >> >> >>>> >> >>> > _______________________________________________ >> >>>> >> >>> > ceph-users mailing list -- ceph-users@ceph.io >> >>>> >> >>> > To unsubscribe send an email to ceph-users-le...@ceph.io >> >>>> >> >>> >> >>>> >> >>> >> >>>> >> >>> _______________________________________________ >> >>>> >> >>> ceph-users mailing list -- ceph-users@ceph.io >> >>>> >> >>> To unsubscribe send an email to ceph-users-le...@ceph.io >> >>>> >> >>> >> >>>> >> >> >> >>>> >> > _______________________________________________ >> >>>> >> > ceph-users mailing list -- ceph-users@ceph.io >> >>>> >> > To unsubscribe send an email to ceph-users-le...@ceph.io >> >>>> >> >> >>>> >> >> >>>> >> _______________________________________________ >> >>>> >> ceph-users mailing list -- ceph-users@ceph.io >> >>>> >> To unsubscribe send an email to ceph-users-le...@ceph.io >> >>>> >> >> >>>> > _______________________________________________ >> >>>> > ceph-users mailing list -- ceph-users@ceph.io >> >>>> > To unsubscribe send an email to ceph-users-le...@ceph.io >> >>>> >> >>>> >> >>>> _______________________________________________ >> >>>> ceph-users mailing list -- ceph-users@ceph.io >> >>>> To unsubscribe send an email to ceph-users-le...@ceph.io >> >>>> >> >>> >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users@ceph.io >> > To unsubscribe send an email to ceph-users-le...@ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-le...@ceph.io >> > _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
