>>>> Yes, I was just mid-draft of a response explaining the single steps. oh... that would be nice...
Em qua., 20 de ago. de 2025 às 15:41, Eugen Block <ebl...@nde.ag> escreveu: > Yes, I was just mid-draft of a response explaining the single steps. > ;-) I'll ping Zac tomorrow to update the docs to include the actual > list of hosts. > > Zitat von Gilberto Ferreira <gilberto.nune...@gmail.com>: > > > Well... Ends up I need to adapt the original script to this: > > > > #!/bin/bash > > > > ms=/root/monstore > > host=/root/hosts -------- here there's a list of my 3 nodes, one per > line > > mkdir $ms > > > > # collect the cluster map from stopped OSDs > > for host in $(cat /root/hosts); do > > rsync -avz $ms/. root@$host:$ms.remote > > rm -rf $ms > > ssh root@$host <<EOF > > for osd in /var/lib/ceph/osd/ceph-*; do > > ceph-objectstore-tool --data-path \$osd --no-mon-config --op > > update-mon-db --mon-store-path $ms.remote > > done > > EOF > > rsync -avz root@$host:$ms.remote/. $ms > > done > > > > Now on node1 I have the following folder: > > > > ls monstore/store.db/ > > 000008.sst 000013.sst 000018.sst 000023.sst 000028.sst 000029.log > > CURRENT IDENTITY LOCK MANIFEST-000030 OPTIONS-000027 OPTIONS-000032 > > > > I believe I am getting closer... > > > > > > > > > > Em qua., 20 de ago. de 2025 às 15:19, Gilberto Ferreira < > > gilberto.nune...@gmail.com> escreveu: > > > >> So right now I have this: > >> pve01:~# find mon-store/ > >> mon-store/ > >> mon-store/ceph-0 > >> mon-store/ceph-0/kv_backend > >> mon-store/ceph-0/store.db > >> mon-store/ceph-0/store.db/LOCK > >> mon-store/ceph-0/store.db/IDENTITY > >> mon-store/ceph-0/store.db/CURRENT > >> mon-store/ceph-0/store.db/000004.log > >> mon-store/ceph-0/store.db/MANIFEST-000005 > >> mon-store/ceph-0/store.db/OPTIONS-000007 > >> mon-store/ceph-1 > >> mon-store/ceph-1/kv_backend > >> mon-store/ceph-1/store.db > >> mon-store/ceph-1/store.db/LOCK > >> mon-store/ceph-1/store.db/IDENTITY > >> mon-store/ceph-1/store.db/CURRENT > >> mon-store/ceph-1/store.db/000004.log > >> mon-store/ceph-1/store.db/MANIFEST-000005 > >> mon-store/ceph-1/store.db/OPTIONS-000007 > >> > >> pve02:~# find mon-store/ > >> mon-store/ > >> mon-store/ceph-2 > >> mon-store/ceph-2/kv_backend > >> mon-store/ceph-2/store.db > >> mon-store/ceph-2/store.db/LOCK > >> mon-store/ceph-2/store.db/IDENTITY > >> mon-store/ceph-2/store.db/CURRENT > >> mon-store/ceph-2/store.db/000004.log > >> mon-store/ceph-2/store.db/MANIFEST-000005 > >> mon-store/ceph-2/store.db/OPTIONS-000007 > >> mon-store/ceph-3 > >> mon-store/ceph-3/kv_backend > >> mon-store/ceph-3/store.db > >> mon-store/ceph-3/store.db/LOCK > >> mon-store/ceph-3/store.db/IDENTITY > >> mon-store/ceph-3/store.db/CURRENT > >> mon-store/ceph-3/store.db/000004.log > >> mon-store/ceph-3/store.db/MANIFEST-000005 > >> mon-store/ceph-3/store.db/OPTIONS-000007 > >> > >> pve03:~# find mon-store/ > >> mon-store/ > >> mon-store/ceph-4 > >> mon-store/ceph-4/kv_backend > >> mon-store/ceph-4/store.db > >> mon-store/ceph-4/store.db/LOCK > >> mon-store/ceph-4/store.db/IDENTITY > >> mon-store/ceph-4/store.db/CURRENT > >> mon-store/ceph-4/store.db/000004.log > >> mon-store/ceph-4/store.db/MANIFEST-000005 > >> mon-store/ceph-4/store.db/OPTIONS-000007 > >> mon-store/ceph-5 > >> mon-store/ceph-5/kv_backend > >> mon-store/ceph-5/store.db > >> mon-store/ceph-5/store.db/LOCK > >> mon-store/ceph-5/store.db/IDENTITY > >> mon-store/ceph-5/store.db/CURRENT > >> mon-store/ceph-5/store.db/000004.log > >> mon-store/ceph-5/store.db/MANIFEST-000005 > >> mon-store/ceph-5/store.db/OPTIONS-000007 > >> > >> > >> > >> --- > >> > >> > >> > >> > >> > >> > >> Em qua., 20 de ago. de 2025 às 15:15, Gilberto Ferreira < > >> gilberto.nune...@gmail.com> escreveu: > >> > >>> Ok... > >>> I am doing it again. > >>> I have 2 osd per node. > >>> Do I need to create multiple folder for each osd? > >>> Like > >>> node1: > >>> mon-store/ceph-osd0 > >>> mon-store/ceph-osd1 > >>> node2: > >>> mon-store/ceph-osd2 > >>> mon-store/ceph-osd3 > >>> node3: > >>> mon-store/ceph-osd3 > >>> mon-store/ceph-osd3 > >>> > >>> And than rsynced everything to one node let's say: > >>> nove1: /root/mon-store? > >>> > >>> Which one I should use in order to restore or recreate the mon? > >>> > >>> Sorry for so many questions. > >>> I am trying to understand the whole process, so bare with me. > >>> > >>> Thanks for your patience. > >>> > >>> > >>> > >>> --- > >>> > >>> > >>> Gilberto Nunes Ferreira > >>> +55 (47) 99676-7530 - Whatsapp / Telegram > >>> > >>> > >>> > >>> > >>> > >>> > >>> Em qua., 20 de ago. de 2025 às 14:35, Eugen Block <ebl...@nde.ag> > >>> escreveu: > >>> > >>>> I feel like there's still a misunderstanding here. > >>>> > >>>> The mentioned procedure is: > >>>> > >>>> ms=/root/mon-store > >>>> mkdir $ms > >>>> > >>>> # collect the cluster map from stopped OSDs > >>>> for host in $hosts; do > >>>> rsync -avz $ms/. user@$host:$ms.remote > >>>> rm -rf $ms > >>>> ssh user@$host <<EOF > >>>> for osd in /var/lib/ceph/osd/ceph-*; do > >>>> ceph-objectstore-tool --data-path \$osd --no-mon-config --op > >>>> update-mon-db --mon-store-path $ms.remote > >>>> done > >>>> EOF > >>>> rsync -avz user@$host:$ms.remote/. $ms > >>>> done > >>>> > >>>> > >>>> It collects the clustermap on each host, querying each OSD, but then > >>>> it "merges" it into one store, the local $ms store. That is used then > >>>> to start up the first monitor. So however you do this, make sure you > >>>> have all the clustermaps in one store. Did you stop the newly created > >>>> mon first? And I don't care about the ceph-mon.target, that's always > >>>> on to ensure the MON starts automatically after boot. > >>>> > >>>> Can you clarify that you really have all the clustermaps in one store? > >>>> If not, you'll need to repeat the steps. In theory the steps should > >>>> work exactly as they're described. > >>>> > >>>> Zitat von Gilberto Ferreira <gilberto.nune...@gmail.com>: > >>>> > >>>> > That's strange. > >>>> > Now I have only the ceph-mon.target available: > >>>> > > >>>> > systemctl status ceph-mon.target > >>>> > ● ceph-mon.target - ceph target allowing to start/stop all ceph-mon@ > >>>> .service > >>>> > instances at once > >>>> > Loaded: loaded (/usr/lib/systemd/system/ceph-mon.target; > enabled; > >>>> > preset: enabled) > >>>> > Active: active since Wed 2025-08-20 14:07:12 -03; 1min 47s ago > >>>> > Invocation: 1fcbb21af715460294bd6d8549557ed9 > >>>> > > >>>> > Notice: journal has been rotated since unit was started, output may > be > >>>> > incomplete. > >>>> > > >>>> >>>> And you did rebuild the store from all OSDs as I mentioned, > correct? > >>>> > Yes... > >>>> > Like that: > >>>> > > >>>> > ceph-volume lvm activate --all > >>>> > mkdir /root/mon-store > >>>> > ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 > >>>> --no-mon-config > >>>> > --op update-mon-db --mon-store-path mon-store/ > >>>> > ceph-monstore-tool mon-store/ rebuild -- --keyring > >>>> > /etc/pve/priv/ceph.client.admin.keyring --mon-ids pve01 pve02 pve03 > >>>> > mv /var/lib/ceph/mon/ceph-pve01/store.db/ > >>>> > /var/lib/ceph/mon/ceph-pve01/store.db-bkp > >>>> > cp -rf mon-store/store.db/ /var/lib/ceph/mon/ceph-pve01/ > >>>> > chown -R ceph:ceph /var/lib/ceph/mon/ceph-pve01/store.db > >>>> > > >>>> > On each node. > >>>> > --- > >>>> > > >>>> > > >>>> > Gilberto Nunes Ferreira > >>>> > +55 (47) 99676-7530 - Whatsapp / Telegram > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > Em qua., 20 de ago. de 2025 às 13:49, Eugen Block <ebl...@nde.ag> > >>>> escreveu: > >>>> > > >>>> >> What does the monitor log? Does it at least start successfully? And > >>>> >> you did rebuild the store from all OSDs as I mentioned, correct? > >>>> >> > >>>> >> Zitat von Gilberto Ferreira <gilberto.nune...@gmail.com>: > >>>> >> > >>>> >> > Hi again... > >>>> >> > I have reinstall all Proxmox nodes and install ceph on each node. > >>>> >> > Create the mons and mgr on eatch node. > >>>> >> > I have issue the command ceph-volume lvm activate --all, on each > >>>> node, in > >>>> >> > order bring up the /var/lib/ceph/osd/<node> > >>>> >> > After that I ran this commands: > >>>> >> > ceph-volume lvm activate --all > >>>> >> > mkdir /root/mon-store > >>>> >> > ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 > >>>> >> --no-mon-config > >>>> >> > --op update-mon-db --mon-store-path mon-store/ > >>>> >> > ceph-monstore-tool mon-store/ rebuild -- --keyring > >>>> >> > /etc/pve/priv/ceph.client.admin.keyring --mon-ids pve01 pve02 > pve03 > >>>> >> > mv /var/lib/ceph/mon/ceph-pve01/store.db/ > >>>> >> > /var/lib/ceph/mon/ceph-pve01/store.db-bkp > >>>> >> > cp -rf mon-store/store.db/ /var/lib/ceph/mon/ceph-pve01/ > >>>> >> > chown -R ceph:ceph /var/lib/ceph/mon/ceph-pve01/store.db > >>>> >> > > >>>> >> > But now I got nothing! > >>>> >> > No monitor, no manager, no osd, none! > >>>> >> > > >>>> >> > Perhaps somebody point me what I did wrong. > >>>> >> > > >>>> >> > Thanks > >>>> >> > > >>>> >> > Em qua., 20 de ago. de 2025 às 11:32, Gilberto Ferreira < > >>>> >> > gilberto.nune...@gmail.com> escreveu: > >>>> >> > > >>>> >> >> I can see the content of the mentioned folders just after issue > the > >>>> >> >> command ceph-volume.... > >>>> >> >> Thanks anyway. > >>>> >> >> > >>>> >> >> > >>>> >> >> > >>>> >> >> Em qua., 20 de ago. de 2025 às 11:26, Eugen Block < > ebl...@nde.ag> > >>>> >> >> escreveu: > >>>> >> >> > >>>> >> >>> I assume you're right. Do you see the OSD contents in > >>>> >> >>> /var/lib/ceph/osd/ceph-pve01 after activating? > >>>> >> >>> And remember to collect the clustermap from all OSDs for this > >>>> >> >>> procedure to succeed. > >>>> >> >>> > >>>> >> >>> Zitat von Gilberto Ferreira <gilberto.nune...@gmail.com>: > >>>> >> >>> > >>>> >> >>> > I see... > >>>> >> >>> > > >>>> >> >>> > But I had another problem. > >>>> >> >>> > The script from (0) indicate that should be exist a > >>>> /var/lib/ceph/osd > >>>> >> >>> > folder, like: > >>>> >> >>> > /var/lib/ceph/osd/ceph-pve01 > >>>> >> >>> > /var/lib/ceph/osd/ceph-pve02 > >>>> >> >>> > and so on. > >>>> >> >>> > > >>>> >> >>> > But this folder appears only if I run ceph-volume lvm > activate > >>>> --all. > >>>> >> >>> > So my question is: when I should run this command: after or > >>>> before > >>>> >> use > >>>> >> >>> the > >>>> >> >>> > script? > >>>> >> >>> > I think I need to run ceph-volume lvm activate --all, right? > >>>> >> >>> > Just to clarify. > >>>> >> >>> > > >>>> >> >>> > Thanks > >>>> >> >>> > > >>>> >> >>> > Em qua., 20 de ago. de 2025 às 11:08, Eugen Block < > >>>> ebl...@nde.ag> > >>>> >> >>> escreveu: > >>>> >> >>> > > >>>> >> >>> >> Yes, you need a monitor. The mgr is not required and can be > >>>> deployed > >>>> >> >>> >> later. After you created the monitor, replace the mon store > >>>> contents > >>>> >> >>> >> by the collected clustermaps from the mentioned procedure. > >>>> Keep the > >>>> >> >>> >> ownerships of the directories/files in mind. If the monitor > >>>> starts > >>>> >> >>> >> successfully (with the original FSID), you can try to start > >>>> one of > >>>> >> the > >>>> >> >>> >> OSDs. If that works, start the rest of them, wait for the > >>>> peering > >>>> >> >>> >> storm to settle, create two more monitors and two mgr > daemons. > >>>> >> >>> >> > >>>> >> >>> >> Note that if you lose the mon store and you had a CephFS, > >>>> you'll > >>>> >> need > >>>> >> >>> >> to recreate that from the existing pools. > >>>> >> >>> >> > >>>> >> >>> >> Zitat von Gilberto Ferreira <gilberto.nune...@gmail.com>: > >>>> >> >>> >> > >>>> >> >>> >> > Hi > >>>> >> >>> >> > > >>>> >> >>> >> > Do I need to create any mon and/or mgr in the new ceph > >>>> cluster? > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > Em seg., 18 de ago. de 2025 às 13:03, Eugen Block < > >>>> ebl...@nde.ag> > >>>> >> >>> >> escreveu: > >>>> >> >>> >> > > >>>> >> >>> >> >> Hi, > >>>> >> >>> >> >> > >>>> >> >>> >> >> this sounds like you created a new cluster (new fsid), > the > >>>> OSDs > >>>> >> >>> still > >>>> >> >>> >> >> have the previous fsid configured. I'd rather recommend > to > >>>> follow > >>>> >> >>> this > >>>> >> >>> >> >> procedure [0] to restore the mon store utilizing OSDs > >>>> rather than > >>>> >> >>> >> >> trying to manipulate otherwise intact OSDs to fit into > the > >>>> "new" > >>>> >> >>> >> >> cluster. That way you'll have "your" cluster back. I > don't > >>>> know > >>>> >> if > >>>> >> >>> >> >> there are any specifics to using proxmox, though. But the > >>>> >> mentioned > >>>> >> >>> >> >> procedure seems to work just fine, I've read multiple > >>>> reports on > >>>> >> >>> this > >>>> >> >>> >> >> list. Luckily, I haven't had to use it myself. > >>>> >> >>> >> >> > >>>> >> >>> >> >> Regards, > >>>> >> >>> >> >> Eugen > >>>> >> >>> >> >> > >>>> >> >>> >> >> [0] > >>>> >> >>> >> >> > >>>> >> >>> >> >> > >>>> >> >>> >> > >>>> >> >>> > >>>> >> > >>>> > https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/#recovery-using-osds > >>>> >> >>> >> >> > >>>> >> >>> >> >> Zitat von Gilberto Ferreira <gilberto.nune...@gmail.com > >: > >>>> >> >>> >> >> > >>>> >> >>> >> >> > Hi > >>>> >> >>> >> >> > > >>>> >> >>> >> >> > I have 3 nodes Proxmox Cluster with CEPH, and after a > >>>> crash, I > >>>> >> >>> have to > >>>> >> >>> >> >> > reinstall Proxmox from scratch, along with Ceph. > >>>> >> >>> >> >> > OSD are intact. > >>>> >> >>> >> >> > I already did ceph-volume lvm activate --all and the > OSD > >>>> >> appears > >>>> >> >>> with > >>>> >> >>> >> >> > ceph-volum lvm list and I got a folder with the name of > >>>> the OSD > >>>> >> >>> under > >>>> >> >>> >> >> > /var/lib/ceph/osd. > >>>> >> >>> >> >> > However is not appear in ceph osd tree or ceph -s or > even > >>>> in > >>>> >> the > >>>> >> >>> web > >>>> >> >>> >> gui. > >>>> >> >>> >> >> > Is there any way to re-add this OSD to Proxmox CEPH? > >>>> >> >>> >> >> > > >>>> >> >>> >> >> > Thanks a lot for any help. > >>>> >> >>> >> >> > > >>>> >> >>> >> >> > > >>>> >> >>> >> >> > Best Regards > >>>> >> >>> >> >> > --- > >>>> >> >>> >> >> > Gilbert > >>>> >> >>> >> >> > _______________________________________________ > >>>> >> >>> >> >> > ceph-users mailing list -- ceph-users@ceph.io > >>>> >> >>> >> >> > To unsubscribe send an email to > ceph-users-le...@ceph.io > >>>> >> >>> >> >> > >>>> >> >>> >> >> > >>>> >> >>> >> >> _______________________________________________ > >>>> >> >>> >> >> ceph-users mailing list -- ceph-users@ceph.io > >>>> >> >>> >> >> To unsubscribe send an email to ceph-users-le...@ceph.io > >>>> >> >>> >> >> > >>>> >> >>> >> > _______________________________________________ > >>>> >> >>> >> > ceph-users mailing list -- ceph-users@ceph.io > >>>> >> >>> >> > To unsubscribe send an email to ceph-users-le...@ceph.io > >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> _______________________________________________ > >>>> >> >>> >> ceph-users mailing list -- ceph-users@ceph.io > >>>> >> >>> >> To unsubscribe send an email to ceph-users-le...@ceph.io > >>>> >> >>> >> > >>>> >> >>> > _______________________________________________ > >>>> >> >>> > ceph-users mailing list -- ceph-users@ceph.io > >>>> >> >>> > To unsubscribe send an email to ceph-users-le...@ceph.io > >>>> >> >>> > >>>> >> >>> > >>>> >> >>> _______________________________________________ > >>>> >> >>> ceph-users mailing list -- ceph-users@ceph.io > >>>> >> >>> To unsubscribe send an email to ceph-users-le...@ceph.io > >>>> >> >>> > >>>> >> >> > >>>> >> > _______________________________________________ > >>>> >> > ceph-users mailing list -- ceph-users@ceph.io > >>>> >> > To unsubscribe send an email to ceph-users-le...@ceph.io > >>>> >> > >>>> >> > >>>> >> _______________________________________________ > >>>> >> ceph-users mailing list -- ceph-users@ceph.io > >>>> >> To unsubscribe send an email to ceph-users-le...@ceph.io > >>>> >> > >>>> > _______________________________________________ > >>>> > ceph-users mailing list -- ceph-users@ceph.io > >>>> > To unsubscribe send an email to ceph-users-le...@ceph.io > >>>> > >>>> > >>>> _______________________________________________ > >>>> ceph-users mailing list -- ceph-users@ceph.io > >>>> To unsubscribe send an email to ceph-users-le...@ceph.io > >>>> > >>> > > _______________________________________________ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-le...@ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io