On Mon, 2026-11-09 at 10:55 -0800, Brian Bunker wrote: > During an array controller failover on an iSCSI host, multipathd > briefly put a path to one LUN into the map of another LUN, and I/O > meant for the first LUN landed on the second. See the v1 cover letter > for the full sequence. > > Changes in v3, after Martin's review: > > - No SG_IO. The re-add check now compares the udev WWID with one > built > from the VPD page 0x83 data the kernel caches in sysfs, so it does > no > I/O on the device and can't hang. The new get_sysfs_wwid() in > libmultipath shares get_vpd_uid() and get_uid()'s WWID clean-up > with > the existing sysfs fallback, so both produce the same string.
I am confused. Does this really fix the problem in your use case? You reported that the udev-derived WWID was false in some uevents, and you said that, as the test was running under RHEL9, udev had obtained the WWID from scsi_id, which uses SG_IO. While my previous argument that SG_IO can block is valid, it's clear that *if* SG_IO works, it's a better approximation for the REAL device WWID than the kernel's cached value in sysfs, which might be stale. It makes only limited sense to me that reading a cached sysfs value in multipathd would be more reliable than what happened during udev rule processing. Also, talking about consistency, on a RHEL system it would actually be more consistent to use SG_IO than to read the WWID from sysfs. I'd like to go back to your original problem description for a moment. > 1. A change uevent for sdq (verify3) carried verify1's WWID. The next > uevent 4.5 s later carried the correct one. We don't know whether > the array answered the INQUIRY (VPD 0x83, run by scsi_id) wrongly > during the takeover or the bad value arose elsewhere in the udev > path; we have no packet capture. "Elsewhere in the udev path" makes little sense to me. As we discussed, udev on RHEL9 uses a VPD 83 INQUIRY, sent by scsi_id via SG_IO. The WWID that multipathd reads from the uevent represents (part of) what the storage had sent to scsi_id. If multipathd uses SG_IO and obtains a different value, then either the storage must have answered one of the two INQUIRYs wrongly, or something must have changed in the meantime. (Of course we could have a blatant bug in multipathd that mixes up the WWIDs in some corner case. But if that's the case, your patch wouldn't help). IMO, if we can't trust the reply of the storage array, all bets are off. Why would the reply that multipathd obtains directly be more trustworthy than what scsi_id had got? (Under SUSE you could see this differently, because SUSE reads from sysfs, but you observed the issue on RHEL). You wrote that you'd seen this problem in an "array controller failover" scenario while some test script was generating uevent storms. I'd like to understand better what happens in this concrete failover scenario. One possible case for a WWID change is that a a Linux SCSI disk is removed, and another disk is added later, taking over the previous device's dev_t. In this case we'd expect "remove" and "add" uevents. With the uevent storm ongoing, these events may be delayed. So multipathd may see a change event for the old device while the kernel had already added the new one. This makes some sense, but unfortunately it doesn't match your description: > A change uevent for sdq (verify3) carried verify1's WWID. The next > uevent 4.5 s later carried the correct one. This sounds as if the WWID of sdq was that of "verify3", then "verify1", and then "verify3" again. In my scenario above, sdq might change from "verify3" to "verify1", but it wouldn't change back again, unless there was another array failover in a very short time span. Question about your test: Apparently the problem is reproducible, otherwise you wouldn't be able to verify your patch. But you said "we have no packet capture". If the test is actually reproducible, it should be possible to create a capture and verify that the storage is not the culprit. That would be very important IMO, because like I said above, if this can happen, all bets are off. I would really appreciate if we could clarify these open questions. Back to the patch set: IMO if you want to determine the *real* WWID, you _must_ use SG_IO. I am contradicting my previous statement here, I know. I still hold that SG_IO may block and that I'd rather avoid using it in multipathd. But again, if you want to be certain (ignoring the fact that the storage may provide false replies), SG_IO is you best bet. With this in mind, I suggest that you go back to v2 and SG_IO. Sorry for having led you on the wrong track previously. Because I am generally sceptical of the use of SG_IO in multipathd in its current form, and because the the overall idea is very similar to the "recheck_wwid" feature, I further suggest that you make this code path conditional on "recheck_wwid". In the long run, we may want to redesign our WWID detection. We do have a general framework for asynchronous jobs now. Thus we actually could run asynchronous VPD inquries. This would eliminate the problem of multipathd getting stuck on a hanging INQUIRY. But this needs more thought and experiments. Regards Martin -- Dr. Martin Wilck <[email protected]> SUSE Software Solutions Germany GmbH, Frankenstr. 146, 90461 Nürnberg, Germany Geschäftsführer: Stefan Gaiser, Jochen Jaser, Abhinav Puri (HRB 36809,AG Nürnberg)
