[ceph-users] lvm / udev regression, was Re: Re: v20.2.2 Tentacle released

Wed, 17 Jun 2026 09:18:06 -0700

Confirming Reto's report and adding the root cause, verified against the diff.  That regression hit me as well.  Basically, using the lvm system for anything else on the same host that uses ceph wedges.  For example, booting the host that uses ceph... 

Detail:
The regression is PR #67989 ("ceph-volume: include LVM mapper devices in get_devices()"). It removed this block from get_devices() in ceph_volume/util/disk.py: 

   python

   # If the mapper device is a logical volume it gets excluded

   try:
        if UdevData(diskname).is_lvm:
            continue
        except RuntimeError:
            logger.debug("get_devices(): device {} couldn't be
   found.".format(diskname))
            continue
That block did two things, both now gone: it skipped LVM devices, and — decisively — it caught the RuntimeError that UdevData.__init__ raises when a device path has no udev node.
RAID-internal sub-LVs (<lv>_rmeta_N, <lv>_rimage_N) have no standalone udev node. Post-PR, get_devices() calls UdevData() on every dm node including those sub-LVs, the RuntimeError is uncaught, and the whole inventory aborts — hence CEPHADM_REFRESH_FAILED on any host with an LVM RAID1 (or RAID5/6/10) LV.
The PR's added test (test_lvm_device_is_included) only covers an LV with a resolvable udev node; the raising path is untested.
Trigger condition: present on v20.2.2; absent on v20.2.1 (where the exclude-and-catch block is still in place — verified here, inventory clean on a host with multiple RAID1 LVs).
Any host whose OS/boot mirror uses LVM RAID1 is affected; the run aborts on the first RAID-internal sub-LV enumerated. LVM filter/global_filter is not a workaround — those act on PV device paths, not LV names, so a reject of *_rmeta_*/*_rimage_* leaves the sub-LVs fully visible to ceph-volume's /sys/block walk.
Suggested fix: restore handling for LVs whose UdevData can't resolve a udev node — re-add the except RuntimeError: continue, or skip LVs whose LVM role is RAID-internal (the [bracketed], private,raid,metadata/image LVs) during enumeration. Reto's "only enumerate public-role LVs" framing is the same idea.
Workaround until fixed: stay on v20.2.1 for hosts with RAID-backed OS mirrors. 

On 6/17/26 05:07, Reto Gysi via ceph-users wrote:

Hi

I've noticed the following bug/regression with Tentacle release v20.2.2
Ceph health checks are failing because there are existing raid mirror
volumes on the node hosts, and ceph-volume is trying to read the "raid
internal" logical volumes

[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices
    host zephir `cephadm ceph-volume` failed: cephadm exited with an error
code: 1, stderr: Inferring config
/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/mon.zephir/config
Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host
--stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume
--privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/cep
h/ceph:v20.2.2 -e NODE_NAME=zephir -e CEPH_VOLUME_SKIP_RESTORECON=yes -e
CEPH_VOLUME_DEBUG=1 -v
/var/run/ceph/27923302-87a5-11ec-ac5b-976d21a49941:/var/run/ceph:z -v
/var/log/ceph/27923302-87a5-11e
c-ac5b-976d21a49941:/var/log/ceph:z -v
/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/crash:/var/lib/ceph/crash:z
-v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v
/run/udev:/run/ude
v -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v
/:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v
/tmp/ceph-tmpe1ewv2jz:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v20.2.2
inventory
--format=json-pretty --with-lsm --filter-for-batch
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr  stderr: blkid: error: /dev/debian/var_rmeta_0: No
such file or directory
/usr/bin/podman: stderr Traceback (most recent call last):
/usr/bin/podman: stderr   File "/usr/sbin/ceph-volume", line 33, in <module>
/usr/bin/podman: stderr     sys.exit(load_entry_point('ceph-volume==1.0.0',
'console_scripts', 'ceph-volume')())
/usr/bin/podman: stderr   File
"/usr/lib/python3.9/site-packages/ceph_volume/main.py", line 62, in __init__
/usr/bin/podman: stderr     self.main(self.argv)
/usr/bin/podman: stderr   File
"/usr/lib/python3.9/site-packages/ceph_volume/decorators.py", line 59, in
newfunc
/usr/bin/podman: stderr     return f(*a, **kw)
/usr/bin/podman: stderr   File
"/usr/lib/python3.9/site-packages/ceph_volume/main.py", line 174, in main
/usr/bin/podman: stderr     terminal.dispatch(self.mapper, subcommand_args)
/usr/bin/podman: stderr   File
"/usr/lib/python3.9/site-packages/ceph_volume/terminal.py", line 194, in
dispatch
/usr/bin/podman: stderr     instance.main()
/usr/bin/podman: stderr   File
"/usr/lib/python3.9/site-packages/ceph_volume/inventory/main.py", line 58,
in main
/usr/bin/podman: stderr
     self.format_report(Devices(filter_for_batch=self.args.filter_for_batch,
/usr/bin/podman: stderr   File
"/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 43, in
__init__
/usr/bin/podman: stderr     self._devices = [Device(k,
/usr/bin/podman: stderr   File
"/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 43, in
<listcomp>
/usr/bin/podman: stderr     self._devices = [Device(k,
/usr/bin/podman: stderr   File
"/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 156, in
__init__
/usr/bin/podman: stderr     self.device_id = self._get_device_id()
/usr/bin/podman: stderr   File
"/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 322, in
_get_device_id
/usr/bin/podman: stderr     udev_data = disk.UdevData(self.path)
/usr/bin/podman: stderr   File
"/usr/lib/python3.9/site-packages/ceph_volume/util/disk.py", line 1355, in
__init__
/usr/bin/podman: stderr     raise RuntimeError(f'{path} not found.')
/usr/bin/podman: stderr RuntimeError: /dev/debian/var_rmeta_0 not found.
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File
"/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py",
line 5296, in <module>
    main()
    ~~~~^^
  File
"/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py",
line 5284, in main
    r = ctx.func(ctx)
  File
"/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py",
line 397, in _infer_config
    return func(ctx)
  File
"/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py",
line 312, in _infer_fsid
    return func(ctx)
  File
"/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py",
line 425, in _infer_image
    return func(ctx)
  File
"/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py",
line 299, in _validate_fsid
    return func(ctx)
  File
"/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py",
line 3231, in command_ceph_volume
    out, err, code = call_throws(ctx, c.run_cmd(),
verbosity=CallVerbosity.QUIET_UNLESS_ERROR)

~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File
"/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/cephadmlib/call_wrappers.py",
line 310, in call_throws
    raise RuntimeError('Failed command: %s' % ' '.join(command))
RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host
--stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume
--privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io
/ceph/ceph:v20.2.2 -e NODE_NAME=zephir -e CEPH_VOLUME_SKIP_RESTORECON=yes
-e CEPH_VOLUME_DEBUG=1 -v
/var/run/ceph/27923302-87a5-11ec-ac5b-976d21a49941:/var/run/ceph:z -v
/var/log/ceph/27923302-87a5
-11ec-ac5b-976d21a49941:/var/log/ceph:z -v
/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/crash:/var/lib/ceph/crash:z
-v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run
/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v
/:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v
/tmp/ceph-tmpe1ewv2jz:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v20.2.2 inven
tory --format=json-pretty --with-lsm --filter-for-batch


lvs output example:
root@zephir:~# lvs -a -o +role -o +devices |head -1; lvs -a -o +role -o
+devices| grep var
  LV                                             VG
                                        Attr       LSize    Pool Origin
                         Data%  Meta%  Move Log Cpy%Sync Convert Role
                               Devices

  var                                            debian
                                    rwi-aor--- <139.70g
                                                            100.00
           public
                               var_rimage_0(0),var_rimage_1(0)

  var_backup                                     debian
                                    -wi-ao----  120.00g

      public

                               /dev/mapper/sdc1_crypt(23040)

  [var_rimage_0]                                 debian
                                    iwi-aor--- <139.70g

      private,ra
id,image                       /dev/mapper/nvme0n1p3_crypt(71525)

  [var_rimage_1]                                 debian
                                    iwi-aor--- <139.70g

      private,ra
id,image                       /dev/mapper/nvme1n1p1_crypt(214578)

  [var_rmeta_0]                                  debian
                                    ewi-aor---    4.00m

      private,ra
id,metadata                    /dev/mapper/nvme0n1p3_crypt(0)

  [var_rmeta_1]                                  debian
                                    ewi-aor---    4.00m

      private,ra
id,metadata                    /dev/mapper/nvme1n1p1_crypt(214577)

  var-lib-ceph                                   debian-4k
                                 rwi-aor---   40.00g
                                                            100.00
           public
                               var-lib-ceph_rimage_0(0),var-lib-ceph_rimage_1(0)

  [var-lib-ceph_rimage_0]                        debian-4k
                                 iwi-aor---   40.00g

      private,ra
id,image                       /dev/mapper/nvme3n1_crypt(55050)

  [var-lib-ceph_rimage_1]                        debian-4k
                                 iwi-aor---   40.00g

      private,ra
id,image                       /dev/mapper/nvme4n1_crypt(786433)

  [var-lib-ceph_rmeta_0]                         debian-4k
                                 ewi-aor---    4.00m

      private,ra
id,metadata                    /dev/mapper/nvme3n1_crypt(55049)

  [var-lib-ceph_rmeta_1]                         debian-4k
                                 ewi-aor---    4.00m

      private,ra
id,metadata                    /dev/mapper/nvme4n1_crypt(786432)

root@zephir:~#

Maybe ceph-volume should only try to read the logical volumes with Role
'public' and not the lv raid internal ones.
Those volumes are used to host the standard OS installation directories
(/,/var,etc) on mirrored volumes.



ceph-volume
-----------

* Inventory Scanning: Enhanced inventory discovery logic to
automatically skip RAM disk devices (/dev/ram*).

To unsubscribe send an email [email protected]



Thanks & Regards

Reto
_______________________________________________
ceph-users mailing list [email protected]
To unsubscribe send an email [email protected]

_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]

Reply via email to