>>>> Yes, I was just mid-draft of a response explaining the single steps.
oh... that would be nice...






Em qua., 20 de ago. de 2025 às 15:41, Eugen Block <ebl...@nde.ag> escreveu:

> Yes, I was just mid-draft of a response explaining the single steps.
> ;-) I'll ping Zac tomorrow to update the docs to include the actual
> list of hosts.
>
> Zitat von Gilberto Ferreira <gilberto.nune...@gmail.com>:
>
> > Well... Ends up I need to adapt the original script to this:
> >
> > #!/bin/bash
> >
> > ms=/root/monstore
> > host=/root/hosts   -------- here there's a list of my 3 nodes, one per
> line
> > mkdir $ms
> >
> > # collect the cluster map from stopped OSDs
> > for host in $(cat /root/hosts); do
> >   rsync -avz $ms/. root@$host:$ms.remote
> >   rm -rf $ms
> >   ssh root@$host <<EOF
> >     for osd in /var/lib/ceph/osd/ceph-*; do
> >       ceph-objectstore-tool --data-path \$osd --no-mon-config --op
> > update-mon-db --mon-store-path $ms.remote
> >     done
> > EOF
> >   rsync -avz root@$host:$ms.remote/. $ms
> > done
> >
> > Now on node1 I have the following folder:
> >
> > ls monstore/store.db/
> > 000008.sst  000013.sst  000018.sst  000023.sst  000028.sst  000029.log
> >  CURRENT  IDENTITY  LOCK  MANIFEST-000030  OPTIONS-000027  OPTIONS-000032
> >
> > I believe I am getting closer...
> >
> >
> >
> >
> > Em qua., 20 de ago. de 2025 às 15:19, Gilberto Ferreira <
> > gilberto.nune...@gmail.com> escreveu:
> >
> >> So right now I have this:
> >> pve01:~# find mon-store/
> >> mon-store/
> >> mon-store/ceph-0
> >> mon-store/ceph-0/kv_backend
> >> mon-store/ceph-0/store.db
> >> mon-store/ceph-0/store.db/LOCK
> >> mon-store/ceph-0/store.db/IDENTITY
> >> mon-store/ceph-0/store.db/CURRENT
> >> mon-store/ceph-0/store.db/000004.log
> >> mon-store/ceph-0/store.db/MANIFEST-000005
> >> mon-store/ceph-0/store.db/OPTIONS-000007
> >> mon-store/ceph-1
> >> mon-store/ceph-1/kv_backend
> >> mon-store/ceph-1/store.db
> >> mon-store/ceph-1/store.db/LOCK
> >> mon-store/ceph-1/store.db/IDENTITY
> >> mon-store/ceph-1/store.db/CURRENT
> >> mon-store/ceph-1/store.db/000004.log
> >> mon-store/ceph-1/store.db/MANIFEST-000005
> >> mon-store/ceph-1/store.db/OPTIONS-000007
> >>
> >> pve02:~# find mon-store/
> >> mon-store/
> >> mon-store/ceph-2
> >> mon-store/ceph-2/kv_backend
> >> mon-store/ceph-2/store.db
> >> mon-store/ceph-2/store.db/LOCK
> >> mon-store/ceph-2/store.db/IDENTITY
> >> mon-store/ceph-2/store.db/CURRENT
> >> mon-store/ceph-2/store.db/000004.log
> >> mon-store/ceph-2/store.db/MANIFEST-000005
> >> mon-store/ceph-2/store.db/OPTIONS-000007
> >> mon-store/ceph-3
> >> mon-store/ceph-3/kv_backend
> >> mon-store/ceph-3/store.db
> >> mon-store/ceph-3/store.db/LOCK
> >> mon-store/ceph-3/store.db/IDENTITY
> >> mon-store/ceph-3/store.db/CURRENT
> >> mon-store/ceph-3/store.db/000004.log
> >> mon-store/ceph-3/store.db/MANIFEST-000005
> >> mon-store/ceph-3/store.db/OPTIONS-000007
> >>
> >> pve03:~# find mon-store/
> >> mon-store/
> >> mon-store/ceph-4
> >> mon-store/ceph-4/kv_backend
> >> mon-store/ceph-4/store.db
> >> mon-store/ceph-4/store.db/LOCK
> >> mon-store/ceph-4/store.db/IDENTITY
> >> mon-store/ceph-4/store.db/CURRENT
> >> mon-store/ceph-4/store.db/000004.log
> >> mon-store/ceph-4/store.db/MANIFEST-000005
> >> mon-store/ceph-4/store.db/OPTIONS-000007
> >> mon-store/ceph-5
> >> mon-store/ceph-5/kv_backend
> >> mon-store/ceph-5/store.db
> >> mon-store/ceph-5/store.db/LOCK
> >> mon-store/ceph-5/store.db/IDENTITY
> >> mon-store/ceph-5/store.db/CURRENT
> >> mon-store/ceph-5/store.db/000004.log
> >> mon-store/ceph-5/store.db/MANIFEST-000005
> >> mon-store/ceph-5/store.db/OPTIONS-000007
> >>
> >>
> >>
> >> ---
> >>
> >>
> >>
> >>
> >>
> >>
> >> Em qua., 20 de ago. de 2025 às 15:15, Gilberto Ferreira <
> >> gilberto.nune...@gmail.com> escreveu:
> >>
> >>> Ok...
> >>> I am doing it again.
> >>> I have 2 osd per node.
> >>> Do I need to create multiple folder for each osd?
> >>> Like
> >>> node1:
> >>> mon-store/ceph-osd0
> >>> mon-store/ceph-osd1
> >>> node2:
> >>> mon-store/ceph-osd2
> >>> mon-store/ceph-osd3
> >>> node3:
> >>> mon-store/ceph-osd3
> >>> mon-store/ceph-osd3
> >>>
> >>> And than rsynced everything to one node let's say:
> >>> nove1: /root/mon-store?
> >>>
> >>> Which one I should use in order to restore or recreate the mon?
> >>>
> >>> Sorry for so many questions.
> >>> I am trying to understand the whole process, so bare with me.
> >>>
> >>> Thanks for your patience.
> >>>
> >>>
> >>>
> >>> ---
> >>>
> >>>
> >>> Gilberto Nunes Ferreira
> >>> +55 (47) 99676-7530 - Whatsapp / Telegram
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> Em qua., 20 de ago. de 2025 às 14:35, Eugen Block <ebl...@nde.ag>
> >>> escreveu:
> >>>
> >>>> I feel like there's still a misunderstanding here.
> >>>>
> >>>> The mentioned procedure is:
> >>>>
> >>>> ms=/root/mon-store
> >>>> mkdir $ms
> >>>>
> >>>> # collect the cluster map from stopped OSDs
> >>>> for host in $hosts; do
> >>>>    rsync -avz $ms/. user@$host:$ms.remote
> >>>>    rm -rf $ms
> >>>>    ssh user@$host <<EOF
> >>>>      for osd in /var/lib/ceph/osd/ceph-*; do
> >>>>        ceph-objectstore-tool --data-path \$osd --no-mon-config --op
> >>>> update-mon-db --mon-store-path $ms.remote
> >>>>      done
> >>>> EOF
> >>>>    rsync -avz user@$host:$ms.remote/. $ms
> >>>> done
> >>>>
> >>>>
> >>>> It collects the clustermap on each host, querying each OSD, but then
> >>>> it "merges" it into one store, the local $ms store. That is used then
> >>>> to start up the first monitor. So however you do this, make sure you
> >>>> have all the clustermaps in one store. Did you stop the newly created
> >>>> mon first? And I don't care about the ceph-mon.target, that's always
> >>>> on to ensure the MON starts automatically after boot.
> >>>>
> >>>> Can you clarify that you really have all the clustermaps in one store?
> >>>> If not, you'll need to repeat the steps. In theory the steps should
> >>>> work exactly as they're described.
> >>>>
> >>>> Zitat von Gilberto Ferreira <gilberto.nune...@gmail.com>:
> >>>>
> >>>> > That's strange.
> >>>> > Now I have only the ceph-mon.target available:
> >>>> >
> >>>> > systemctl status ceph-mon.target
> >>>> > ● ceph-mon.target - ceph target allowing to start/stop all ceph-mon@
> >>>> .service
> >>>> > instances at once
> >>>> >      Loaded: loaded (/usr/lib/systemd/system/ceph-mon.target;
> enabled;
> >>>> > preset: enabled)
> >>>> >      Active: active since Wed 2025-08-20 14:07:12 -03; 1min 47s ago
> >>>> >  Invocation: 1fcbb21af715460294bd6d8549557ed9
> >>>> >
> >>>> > Notice: journal has been rotated since unit was started, output may
> be
> >>>> > incomplete.
> >>>> >
> >>>> >>>> And you did rebuild the store from all OSDs as I mentioned,
> correct?
> >>>> > Yes...
> >>>> > Like that:
> >>>> >
> >>>> > ceph-volume lvm activate --all
> >>>> > mkdir /root/mon-store
> >>>> > ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0
> >>>> --no-mon-config
> >>>> > --op update-mon-db --mon-store-path mon-store/
> >>>> > ceph-monstore-tool mon-store/ rebuild -- --keyring
> >>>> > /etc/pve/priv/ceph.client.admin.keyring --mon-ids pve01 pve02 pve03
> >>>> > mv /var/lib/ceph/mon/ceph-pve01/store.db/
> >>>> > /var/lib/ceph/mon/ceph-pve01/store.db-bkp
> >>>> > cp -rf mon-store/store.db/ /var/lib/ceph/mon/ceph-pve01/
> >>>> > chown -R ceph:ceph /var/lib/ceph/mon/ceph-pve01/store.db
> >>>> >
> >>>> > On each node.
> >>>> > ---
> >>>> >
> >>>> >
> >>>> > Gilberto Nunes Ferreira
> >>>> > +55 (47) 99676-7530 - Whatsapp / Telegram
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > Em qua., 20 de ago. de 2025 às 13:49, Eugen Block <ebl...@nde.ag>
> >>>> escreveu:
> >>>> >
> >>>> >> What does the monitor log? Does it at least start successfully? And
> >>>> >> you did rebuild the store from all OSDs as I mentioned, correct?
> >>>> >>
> >>>> >> Zitat von Gilberto Ferreira <gilberto.nune...@gmail.com>:
> >>>> >>
> >>>> >> > Hi again...
> >>>> >> > I have reinstall all Proxmox nodes and install ceph on each node.
> >>>> >> > Create the mons and mgr on eatch node.
> >>>> >> > I have issue the command ceph-volume lvm activate --all, on each
> >>>> node, in
> >>>> >> > order bring up the /var/lib/ceph/osd/<node>
> >>>> >> > After that I ran this commands:
> >>>> >> > ceph-volume lvm activate --all
> >>>> >> > mkdir /root/mon-store
> >>>> >> > ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0
> >>>> >> --no-mon-config
> >>>> >> > --op update-mon-db --mon-store-path mon-store/
> >>>> >> > ceph-monstore-tool mon-store/ rebuild -- --keyring
> >>>> >> > /etc/pve/priv/ceph.client.admin.keyring --mon-ids pve01 pve02
> pve03
> >>>> >> > mv /var/lib/ceph/mon/ceph-pve01/store.db/
> >>>> >> > /var/lib/ceph/mon/ceph-pve01/store.db-bkp
> >>>> >> > cp -rf mon-store/store.db/ /var/lib/ceph/mon/ceph-pve01/
> >>>> >> > chown -R ceph:ceph /var/lib/ceph/mon/ceph-pve01/store.db
> >>>> >> >
> >>>> >> > But now I got nothing!
> >>>> >> > No monitor, no manager, no osd, none!
> >>>> >> >
> >>>> >> > Perhaps somebody point me what I did wrong.
> >>>> >> >
> >>>> >> > Thanks
> >>>> >> >
> >>>> >> > Em qua., 20 de ago. de 2025 às 11:32, Gilberto Ferreira <
> >>>> >> > gilberto.nune...@gmail.com> escreveu:
> >>>> >> >
> >>>> >> >> I can see the content of the mentioned folders just after issue
> the
> >>>> >> >> command ceph-volume....
> >>>> >> >> Thanks anyway.
> >>>> >> >>
> >>>> >> >>
> >>>> >> >>
> >>>> >> >> Em qua., 20 de ago. de 2025 às 11:26, Eugen Block <
> ebl...@nde.ag>
> >>>> >> >> escreveu:
> >>>> >> >>
> >>>> >> >>> I assume you're right. Do you see the OSD contents in
> >>>> >> >>> /var/lib/ceph/osd/ceph-pve01 after activating?
> >>>> >> >>> And remember to collect the clustermap from all OSDs for this
> >>>> >> >>> procedure to succeed.
> >>>> >> >>>
> >>>> >> >>> Zitat von Gilberto Ferreira <gilberto.nune...@gmail.com>:
> >>>> >> >>>
> >>>> >> >>> > I see...
> >>>> >> >>> >
> >>>> >> >>> > But I had another problem.
> >>>> >> >>> > The script from (0) indicate that should be exist a
> >>>> /var/lib/ceph/osd
> >>>> >> >>> > folder, like:
> >>>> >> >>> > /var/lib/ceph/osd/ceph-pve01
> >>>> >> >>> > /var/lib/ceph/osd/ceph-pve02
> >>>> >> >>> > and so on.
> >>>> >> >>> >
> >>>> >> >>> > But this folder appears only if I run ceph-volume lvm
> activate
> >>>> --all.
> >>>> >> >>> > So my question is: when I should run this command: after or
> >>>> before
> >>>> >> use
> >>>> >> >>> the
> >>>> >> >>> > script?
> >>>> >> >>> > I think I need to run ceph-volume lvm activate --all, right?
> >>>> >> >>> > Just to clarify.
> >>>> >> >>> >
> >>>> >> >>> > Thanks
> >>>> >> >>> >
> >>>> >> >>> > Em qua., 20 de ago. de 2025 às 11:08, Eugen Block <
> >>>> ebl...@nde.ag>
> >>>> >> >>> escreveu:
> >>>> >> >>> >
> >>>> >> >>> >> Yes, you need a monitor. The mgr is not required and can be
> >>>> deployed
> >>>> >> >>> >> later. After you created the monitor, replace the mon store
> >>>> contents
> >>>> >> >>> >> by the collected clustermaps from the mentioned procedure.
> >>>> Keep the
> >>>> >> >>> >> ownerships of the directories/files in mind. If the monitor
> >>>> starts
> >>>> >> >>> >> successfully (with the original FSID), you can try to start
> >>>> one of
> >>>> >> the
> >>>> >> >>> >> OSDs. If that works, start the rest of them, wait for the
> >>>> peering
> >>>> >> >>> >> storm to settle, create two more monitors and two mgr
> daemons.
> >>>> >> >>> >>
> >>>> >> >>> >> Note that if you lose the mon store and you had a CephFS,
> >>>> you'll
> >>>> >> need
> >>>> >> >>> >> to recreate that from the existing pools.
> >>>> >> >>> >>
> >>>> >> >>> >> Zitat von Gilberto Ferreira <gilberto.nune...@gmail.com>:
> >>>> >> >>> >>
> >>>> >> >>> >> > Hi
> >>>> >> >>> >> >
> >>>> >> >>> >> > Do I need to create any mon and/or mgr in the new ceph
> >>>> cluster?
> >>>> >> >>> >> >
> >>>> >> >>> >> >
> >>>> >> >>> >> >
> >>>> >> >>> >> > Em seg., 18 de ago. de 2025 às 13:03, Eugen Block <
> >>>> ebl...@nde.ag>
> >>>> >> >>> >> escreveu:
> >>>> >> >>> >> >
> >>>> >> >>> >> >> Hi,
> >>>> >> >>> >> >>
> >>>> >> >>> >> >> this sounds like you created a new cluster (new fsid),
> the
> >>>> OSDs
> >>>> >> >>> still
> >>>> >> >>> >> >> have the previous fsid configured. I'd rather recommend
> to
> >>>> follow
> >>>> >> >>> this
> >>>> >> >>> >> >> procedure [0] to restore the mon store utilizing OSDs
> >>>> rather than
> >>>> >> >>> >> >> trying to manipulate otherwise intact OSDs to fit into
> the
> >>>> "new"
> >>>> >> >>> >> >> cluster. That way you'll have "your" cluster back. I
> don't
> >>>> know
> >>>> >> if
> >>>> >> >>> >> >> there are any specifics to using proxmox, though. But the
> >>>> >> mentioned
> >>>> >> >>> >> >> procedure seems to work just fine, I've read multiple
> >>>> reports on
> >>>> >> >>> this
> >>>> >> >>> >> >> list. Luckily, I haven't had to use it myself.
> >>>> >> >>> >> >>
> >>>> >> >>> >> >> Regards,
> >>>> >> >>> >> >> Eugen
> >>>> >> >>> >> >>
> >>>> >> >>> >> >> [0]
> >>>> >> >>> >> >>
> >>>> >> >>> >> >>
> >>>> >> >>> >>
> >>>> >> >>>
> >>>> >>
> >>>>
> https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/#recovery-using-osds
> >>>> >> >>> >> >>
> >>>> >> >>> >> >> Zitat von Gilberto Ferreira <gilberto.nune...@gmail.com
> >:
> >>>> >> >>> >> >>
> >>>> >> >>> >> >> > Hi
> >>>> >> >>> >> >> >
> >>>> >> >>> >> >> > I have 3 nodes Proxmox Cluster with CEPH, and after a
> >>>> crash, I
> >>>> >> >>> have to
> >>>> >> >>> >> >> > reinstall Proxmox from scratch, along with Ceph.
> >>>> >> >>> >> >> > OSD are intact.
> >>>> >> >>> >> >> > I already did ceph-volume lvm activate --all and the
> OSD
> >>>> >> appears
> >>>> >> >>> with
> >>>> >> >>> >> >> > ceph-volum lvm list and I got a folder with the name of
> >>>> the OSD
> >>>> >> >>> under
> >>>> >> >>> >> >> > /var/lib/ceph/osd.
> >>>> >> >>> >> >> > However is not appear in ceph osd tree or ceph -s or
> even
> >>>> in
> >>>> >> the
> >>>> >> >>> web
> >>>> >> >>> >> gui.
> >>>> >> >>> >> >> > Is there any way to re-add this OSD to Proxmox CEPH?
> >>>> >> >>> >> >> >
> >>>> >> >>> >> >> > Thanks a lot for any help.
> >>>> >> >>> >> >> >
> >>>> >> >>> >> >> >
> >>>> >> >>> >> >> > Best Regards
> >>>> >> >>> >> >> > ---
> >>>> >> >>> >> >> > Gilbert
> >>>> >> >>> >> >> > _______________________________________________
> >>>> >> >>> >> >> > ceph-users mailing list -- ceph-users@ceph.io
> >>>> >> >>> >> >> > To unsubscribe send an email to
> ceph-users-le...@ceph.io
> >>>> >> >>> >> >>
> >>>> >> >>> >> >>
> >>>> >> >>> >> >> _______________________________________________
> >>>> >> >>> >> >> ceph-users mailing list -- ceph-users@ceph.io
> >>>> >> >>> >> >> To unsubscribe send an email to ceph-users-le...@ceph.io
> >>>> >> >>> >> >>
> >>>> >> >>> >> > _______________________________________________
> >>>> >> >>> >> > ceph-users mailing list -- ceph-users@ceph.io
> >>>> >> >>> >> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >>>> >> >>> >>
> >>>> >> >>> >>
> >>>> >> >>> >> _______________________________________________
> >>>> >> >>> >> ceph-users mailing list -- ceph-users@ceph.io
> >>>> >> >>> >> To unsubscribe send an email to ceph-users-le...@ceph.io
> >>>> >> >>> >>
> >>>> >> >>> > _______________________________________________
> >>>> >> >>> > ceph-users mailing list -- ceph-users@ceph.io
> >>>> >> >>> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >>>> >> >>>
> >>>> >> >>>
> >>>> >> >>> _______________________________________________
> >>>> >> >>> ceph-users mailing list -- ceph-users@ceph.io
> >>>> >> >>> To unsubscribe send an email to ceph-users-le...@ceph.io
> >>>> >> >>>
> >>>> >> >>
> >>>> >> > _______________________________________________
> >>>> >> > ceph-users mailing list -- ceph-users@ceph.io
> >>>> >> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >>>> >>
> >>>> >>
> >>>> >> _______________________________________________
> >>>> >> ceph-users mailing list -- ceph-users@ceph.io
> >>>> >> To unsubscribe send an email to ceph-users-le...@ceph.io
> >>>> >>
> >>>> > _______________________________________________
> >>>> > ceph-users mailing list -- ceph-users@ceph.io
> >>>> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> ceph-users mailing list -- ceph-users@ceph.io
> >>>> To unsubscribe send an email to ceph-users-le...@ceph.io
> >>>>
> >>>
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

Reply via email to