On Fri, Nov 21, 2025 at 5:07 PM Adrian Sevcenco <[email protected]>
wrote:
> Thank you so much everyone for helping me view and understand the bigger
> picture.
> I do realize (or did realize) that the container model AND cephadm have a
> goal
> to both isolate and remove any kind of middle layer between developers (or
> developing party)
> and any other build process/repository, system dependencies or other 3rd
> party tools
> used for configuration (as ansible is)
> This create a short circuit from developers to end users which of course
> increase
> stability through fast response and independence of underlying environment
> ..
>
> So, understanding this, now i can ask about potential improvements to
> cephadm ..
>
> 1. first of my pain problem is the assumption of lacking of ssh
> communication between hosts.
> while there is --skip-ssh for bootstrap, it would be so much easier if
> there could be some
> environment variables that could steer and configure the behavior of the
> whole cluster
> without having to remember what i choose for the cluster.
> it would be really useful if all bootstrap options could be automatically
> read from
> CEPHENV_<name of option without -- and s/-/_/>
> so i.e. --skip-ssh would firstly be read/checked for as CEPHENV_skip_ssh
> this would allow to automatically "seed" the main parameters of cluster to
> all involved
> nodes with something like ansible :)
>
> there are some other global options that are usually managed by other tools
> and have nothing to do with cephadm like --skip-firewalld
>
[Redo] A John has indicated, as we have so many options, maybe going with a
config file here (which you can then build with your favourite config tool)
can be more helpful.
>
> 2. the second pain point is that the cephadm does not act remotely ..
> it could be really useful to add a --host argument and to act through ssh
> on the remote host
>
> 3. documentation : explanation of arguments is missing
> for example for cephadm deploy
> --name NAME where NAME should be type.id
> i get the type which maybe i can get a list, but what is id?
> an index, number? if it's an index, cannot cephadm enumerate it
> automatically and just put it?
> i.e i want to deploy alert mananager that would be the 6th service
> in cluster, so automatically the name could be alertmanager.6 ?
>
> [Redo] I'll try to answer 2) and 3). For 2 the answer is that in general
the user doesn't have to ssh to the hosts to do orchestration.
That's something that cephadm orchestrator takes care of automatically.
Please correct me if I'm wrong, but I think there's some confusion
going on here. In fact, cephadm architecture has two different layers and
different components:
1) User-facing layer
- The user runs cephadm bootstrap once to create the first cluster.
- After that you mostly interact with the cluster via "ceph orch
commands (or the dashboard)" --> normally this run by the cephadm mgr-module
2) Internal / orchestrator layer (this normally is executed by the
cephadm binary)
The mgr’s cephadm orchestrator module calls cephadm on hosts to do
the real work: such as cephadm deploy,
cephadm rm-daemon, etc.These commands are primarily intended for
the orchestrator, not for day-to-day human.
So normally, once the user bootstrapped the cluster (and added all the
nodes), he doesn't need to ssh into hosts anymore. He just needs
to have a copy of the config conf and key (stored
in /var/lib/ceph/<fsid>/config/ during the bootstrap) and then run the
cephadm shell as:
> cephadm shell --fsid <fsid> -c my-ceph.conf -k my-keyring
If the ceph.conf or the keyring exists on the localhost, or the monitor is
running, cephadm will infer automatically the config) so you just needs to
run
> cephadm shell
This will create a shell from where you can run "ceph orch" commands, and
any other ceph command. The orch commands should be good enough to
orchestrate the whole cluster. There's no need to ssh into any host.
Docs:
https://docs.ceph.com/en/latest/cephadm/
https://docs.ceph.com/en/latest/cephadm/services/
Plz, let me know if this helps to answer your questions, otherwise just let
me know which operations you can't run following the above procedure.
Best,
Redo.
> --config and --config-json can be inferred that they are expecting a
> filepath
> but what about --keyring and --key ?
> some kind of example and structure of names should be shown or linked
> what content/information is present in the name of the key?
> is it arbitrary?
>
> Thank you for help and info!!
> Adrian
>
>
> -------- Original Message --------
> Subject: [ceph-users] cephadm without containers
> From: Anthony D'Atri
> To: Marc
> Date: 11/21/2025, 5:03:27 PM
> >
> >
> >> On Nov 21, 2025, at 9:55 AM, Marc <[email protected]> wrote:
> >>
> >>>
> >>>>
> >>>> Package based Ceph deployments, while popular, are not a good choice
> >>>> in general. The very simple reason is that it makes upgrades more
> >>>> dangerous: you can unintentionally upgrade services in the wrong order
> >>>> due to failovers.
> >>>
> >>> Or when a node crashes / reboots during an upgrade. This has happened
> >>> to me.
> >>>
> >>>
> >>
> >> Hmmm, that is not really nice to read that ceph is so picky that
> everything can go wrong with just a minor update on a single node.
> >
> > Then if you really want to stick with package installs, deploy every
> service on a separate node. Your DC runneth over.
> >
> >> I am not sure if that is a good direction for development.
> >
> > Hence the increasing motivations for container installations, which are
> by nature immune from this dynamic.
> >
> >> I would expect ceph to be more robust.
> >
> > It's not a function of Ceph, but robustness is exactly the goal.
>
> _______________________________________________
> ceph-users mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
>
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
