------- Comment From [email protected] 2016-04-04 11:42 EDT-------
Details about the 'property blacklisting' problem.
---
Debugging this in the initramfs (w/ break=top,pre-multipath,post-
multipath).
This LVM PV should be on a multipath device, but is on an individual
path.
(initramfs) lvm pvdisplay
...
Found duplicate PV SNTOUh7JC3dzwS0BfzNf14bqpw1u58Uh: using /dev/sdao3 not
/dev/sdl3
Using duplicate PV /dev/sdao3 without holders, ignoring /dev/sdl3
--- Physical volume ---
PV Name /dev/sdao3
...
The udev rule which removes the partition nodes from invididual paths is
(56-dm-mpath-lvm.rules):
PROGRAM=="/sbin/multipath -c $devnode", \
ENV{DM_MULTIPATH_DEVICE_PATH}="1", \
ENV{ID_FS_TYPE}="mpath_member", \
RUN+="/usr/bin/partx -d --nr 1-1024 $devnode", \
GOTO="end_mpath"
This indicates 'multipath -c' stated the path was not a valid path,
but running it manually tells otherwise:
(initramfs) multipath -c /dev/sdao
/dev/sdao is a valid multipath device path
(initramfs) multipath -c /dev/sdl
/dev/sdl is a valid multipath device path
By increasing verbosity to 'multipath -v3' and logging the debug output of udev,
we find the 'property blacklist' is behind it:
starting '/sbin/multipath -c -v3 /dev/sdao'
...
'/sbin/multipath -c -v3 /dev/sdao'(out) 'Mar 30 14:48:46 | sdao: blacklisted,
udev property missing'
'/sbin/multipath -c -v3 /dev/sdao'(out) 'Mar 30 14:48:46 | /dev/sdao: failed to
get wwid'
...
Process '/sbin/multipath -c -v3 /dev/sdao' failed with exit code 1.
Comparing the 'multipath -v3 -c' output with running it manually afterward,
the properties are indeed present:
(initramfs) multipath -c -v3 /dev/sdao
...
Mar 30 16:21:17 | sdao: udev property ID_WWN whitelisted
Mar 30 16:21:17 | scope limited to 36005076307ffd1540000000000000115
/dev/sdao is a valid multipath device path
(initramfs) udevadm info -q property /dev/sdao | grep ^ID_WWN
ID_WWN=0x6005076307ffd1540000000000000115
ID_WWN_WITH_EXTENSION=0x6005076307ffd1540000000000000115
So, it turns out the proberties are not yet available during the udev
rule execution of the PROGRAM='multipath -c <path>' rule.
It happens because the properties are only written to the udev database
after the udev rules processing, and before the 'RUN' rules are
executed.
For example, this is the comparison of the 'udevadm info' command on PROGRAM
and RUN rules.
The latter can read a lot more of udev properties.
(initramfs) cat /udev.sda.udevadm-info.program
P: /devices/vio/71000002/host0/target0:0:0/0:0:0:0/block/sda
N: sda
E: DEVNAME=/dev/sda
E: DEVPATH=/devices/vio/71000002/host0/target0:0:0/0:0:0:0/block/sda
E: DEVTYPE=disk
E: MAJOR=8
E: MINOR=0
E: SUBSYSTEM=block
(initramfs) cat /udev.sda.udevadm-info.run
P: /devices/vio/71000002/host0/target0:0:0/0:0:0:0/block/sda
N: sda
S: disk/by-id/scsi-0QEMU_QEMU_HARDDISK_xenialroot
E: DEVLINKS=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_xenialroot
E: DEVNAME=/dev/sda
E: DEVPATH=/devices/vio/71000002/host0/target0:0:0/0:0:0:0/block/sda
E: DEVTYPE=disk
E: DM_MULTIPATH_DEVICE_PATH=1
E: ID_BUS=scsi
E: ID_FS_TYPE=mpath_member
E: ID_MODEL=QEMU_HARDDISK
E: ID_MODEL_ENC=QEMU\x20HARDDISK\x20\x20\x20
...
E: SCSI_IDENT_SERIAL=xenialroot
...
E: SCSI_VENDOR=QEMU
E: SCSI_VENDOR_ENC=QEMU\x20\x20\x20\x20
E: SUBSYSTEM=block
E: USEC_INITIALIZED=9582006
This is some of call chain in the systemd/src/udev/ and-friends dirs:
worker_spawn()
/* PROGRAM */
udev_event_execute_rules()
/* case TK_M_PROGRAM: { */
udev_event_spawn()
spawn_exec(event, cmd, argv, udev_device_get_properties_envp(event->dev),
udev_device_update_db
device_update_db
ORDERED_HASHMAP_FOREACH_KEY(value, property, device->properties_db, i)
fprintf(f, "E:%s=%s\n", property, value);
/* RUN */
udev_event_execute_run()
udev_event_spawn()
spawn_exec(event, cmd, argv, udev_device_get_properties_envp(event->dev),
This 2 upstream commits (present with the merge) are relevant to this issue:
- [1] multipath: implement option '-u' for uevents
- [2] multipath: check for running daemon when called with '-u'
Basically, '-u' addresses exactly this issue (commit message):
multipath: implement option '-u' for uevents
When calling 'multipath' from an uevent udev will not have
all information for that device, as it's being written into
the database _after_ the event has been processed.
This patch implements an option '-u' which uses the information
from the program environment when checking the device.
So, let's change the udev rule from 'multipath -c' to 'multipath -u'.
This requires starting multipathd before udev rules are processed
(-u checks for multipathd running) - so init-top starts multipathd now.
In order to confirm it's working, it's a bit tricky - 'multipath -u -v3'
logs to the system logger rather than stdio, so I modified the multipath
initramfs hook to copy the rsyslogd stuff into the initramfs, and
manually started it on break=top.
@ /usr/share/initramfs/hooks/multipath
mkdir -p $DESTDIR/var/log
mkdir -p $DESTDIR/usr/sbin
copy_exec /usr/sbin/rsyslogd /usr/sbin
cp -rp /usr/lib/rsyslog $DESTDIR/usr/lib
cp -rp /etc/rsyslog* $DESTDIR/etc
...
(initramfs) rsyslogd
Verifying the 'multipath -u -v3' messages on /var/log/syslog,
we can confirm the udev properties are now correctly found,
and the paths are considered valid:
(initramfs) grep 'multipath: sda:' /var/log/syslog
...
Apr 4 15:37:59 (none) multipath: sda: udev property SCSI_IDENT_LUN_VENDOR
whitelisted
(initramfs) grep 'multipath: sdb:' /var/log/syslog
...
Apr 4 15:37:59 (none) multipath: sdb: udev property SCSI_IDENT_LUN_VENDOR
whitelisted
(initramfs) grep 'multipath: sdc:' /var/log/syslog
...
Apr 4 15:37:59 (none) multipath: sdc: udev property SCSI_IDENT_LUN_VENDOR
whitelisted
And the partition nodes for sda and sdb (paths of a multipath device) are
removed,
and those for sdc (not from a multipath device) are not -- which match
'multipath -c':
(initramfs) ls -1 /dev/sd* | sort
/dev/sda
/dev/sdb
/dev/sdc
/dev/sdc1
/dev/sdc2
/dev/sdc3
(initramfs) multipath -c /dev/sda
/dev/sda is a valid multipath device path
(initramfs) multipath -c /dev/sdb
/dev/sdb is a valid multipath device path
(initramfs) multipath -c /dev/sdc
/dev/sdc is not a valid multipath device path
(initramfs) multipath -l
mpatha (0QEMU_QEMU_HARDDISK_xenialroot) dm-0 QEMU,QEMU HARDDISK
size=32G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=0 status=active
| `- 0:0:0:0 sda 8:0 active undef running
`-+- policy='round-robin 0' prio=0 status=enabled
`- 0:0:1:0 sdb 8:16 active undef running
(initramfs)
Links:
[1]
http://git.opensvc.com/gitweb.cgi?p=multipath-tools/.git;a=commit;h=98cbca1773eec5d740bb0c441b27383c75a30449
[2]
http://git.opensvc.com/gitweb.cgi?p=multipath-tools/.git;a=commit;h=77febddfdce8a7d746227c09066e5f507b53d2fb
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1565817
Title:
multipath: LVM physical volumes on individual paths
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1565817/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
