During an array controller failover on an iSCSI host, multipathd briefly put a path to one LUN into the map of another LUN, and I/O meant for the first LUN landed on the second. See the v1 cover letter for the full sequence.
Changes in v3, after Martin's review: - No SG_IO. The re-add check now compares the udev WWID with one built from the VPD page 0x83 data the kernel caches in sysfs, so it does no I/O on the device and can't hang. The new get_sysfs_wwid() in libmultipath shares get_vpd_uid() and get_uid()'s WWID clean-up with the existing sysfs fallback, so both produce the same string. - The uev_update_path() check that ignored a changed udev WWID is gone. With cached data it would be dangerous: if a LUN is really remapped and the kernel hasn't rescanned yet, sysfs still shows the old WWID, and ignoring the change would keep a path to one LUN in another LUN's map. Removing a path from its map is always safe; the add is the step that needs checking. - check_path_wwid_change() is back to its upstream form. Unchanged from v2: the check runs only on PATH_ADD_REINIT, i.e. only for the re-add at the end of uev_update_path(), and only for paths where can_recheck_wwid() is true. Ordinary adds never run it. On mixing WWID sources: Martin is right that multipath-tools should not use different methods in different places. But on RHEL 9 the udev WWID for sd devices doesn't come from sysfs. The RHEL (and Fedora) sg3_utils package installs upstream's 55-scsi-sg3_id.rules as 61-scsi-sg3_id.rules, with the comment "need to run after 60-persistent-storage.rules". So 60-persistent-storage.rules runs first, and its "scsi_id --export --whitelisted -d $devnode" sets ID_SERIAL; the sysfs-based sg3 rules then leave ID_SERIAL alone. scsi_id sends INQUIRY over SG_IO (strace on the host shows ioctl(SG_IO) for the standard INQUIRY and EVPD pages 0x00, 0x80 and 0x83 on every run). On the host where we hit this, the bad WWID came from that SG_IO INQUIRY inside udev, not from cached kernel data. Where sg3_utils' rules run first, as on openSUSE, which installs them as 55-scsi-sg3_id.rules, ID_SERIAL is built from the same sysfs page this check reads. With everything at defaults, which can_recheck_wwid() requires, the designator selection is the same. On that RHEL 9 host, ID_SERIAL equals "3" + the NAA designator decoded from sysfs vpd_pg83 on all 18 Pure paths. So the check doesn't introduce another way of computing the WWID. It compares udev's value with the data udev is meant to derive it from. What happens on a mismatch: the path is left out of its map and udev is retriggered, up to retrigger_tries times. If udev and sysfs still disagree after that, the path goes to INIT_FAILED, and the checker later re-adds it through the normal path, which doesn't run the check. So a persistent disagreement, for example a device where udev rules pick a different designator, only delays the add (by roughly retrigger_tries * retrigger_delay) and doesn't lose the path. A transient bad value like ours is caught, because the retriggered udev event returns the right WWID. In our case, the kernel's cached page 0x83 for the path came from the original scan, so unless the kernel rescanned the device during the failover (we have no sign that it did, but didn't capture sysfs at the time), it still held the correct WWID when the path was re-added, and v3 would have refused the add. Brian Bunker (1): multipathd: check re-added paths' WWIDs against sysfs libmultipath/discovery.c | 42 ++++++++++++++++++++++--------- libmultipath/discovery.h | 1 + libmultipath/libmultipath.version | 5 ++++ multipathd/cli_handlers.c | 2 +- multipathd/main.c | 39 ++++++++++++++++++++-------- multipathd/main.h | 12 ++++++++- 6 files changed, 76 insertions(+), 25 deletions(-) -- 2.56.0
