On 11/9/26 20:19, Laurence Oberman wrote:
I have been testing v9 (your scsi-multipath-v7.4-v9 branch, on
7.4-rc1) on real hardware
Good to know.
and found two problems with ALUA arrays,
one in scsi_dh_alua and one in the series itself.
Setup:
- HP P2000 G3 FC array, implicit ALUA, two port groups (one per
controller)
- 2 FC HBAs, 2 target ports each, 30 LUNs with 4 paths per LUN
- scsi_mod.multipath=on, dm-multipath not in use
- Under dm-multipath, the same LUNs show port group 1 at prio 50
(active/optimized) and port group 0 at prio 10
(active/non-optimized)
1. sdev->access_state is wrong after boot (scsi_dh_alua, not this
series)
After boot, every path read active/optimized in
/sys/class/scsi_device/H:C:T:L/device/access_state, while sg_rtpg
reports port group 0 as active/non-optimized. No
"alua: port group XX state ..." message was logged at all.
In this series, ALUA state really has been ignored. It was felt that
driving ALUA state and sdev->access_state by device handler ALUA driver
should should not be relied on for Native SCSI multipath. There had been
considerable discussion on this on list. I already have the work done
for ALUA support from some time ago and can be posted once this series
is accepted.
Since 50759b881e1d ("scsi: device_handler: alua: Call
scsi_device_put() from non-atomic context"), alua_rtpg_queue() only
queues an RTPG if sdev->handler_data is set, but alua_bus_attach()
sets handler_data after alua_initialize() has tried to queue the
initial RTPG. That RTPG is dropped, and access_state stays at its
zeroed default (active/optimized) until something calls
scsi_dh_activate(). dm-multipath hides this through pg_init; native
multipath never calls activate, so the series sees every path as
optimized.
I have sent a separate fix for this to linux-scsi:
https://lore.kernel.org/linux-scsi/20261109201120.1067815-1-
[email protected]/
It is independent of your series, but without it, native multipath
on ALUA arrays has no correct path state.
Ack
2. ANO paths are ranked the same as AO paths
With the ALUA state corrected (by writing "activate" to dh_state, or
with the fix above), I/O still went to active/non-optimized paths.
scsi_multipath_sdev_init() sets mpath_device->access_state to
MPATH_STATE_OPTIMIZED and nothing updates it, while
__mpath_find_path(), mpath_round_robin_path() and
mpath_queue_depth_path() rank paths by that field. Only
scsi_mpath_is_optimized()/is_disabled() read sdev->access_state.
With numa, the first path in dev_list wins for every node, and
dev_list order follows sd probe completion, so for about a third of
the LUNs the current path was an ANO path. Example, LUN sdn, reads
completed per path for a 2000 x 1M O_DIRECT dd (:0/:3 AO, :1/:2 ANO
at the time):
sdn:0 0sdn:1 2000sdn:2 0sdn:3 0
Every read went through the non-owning controller. From the code, for
these LUNs the numa recheck of is_optimized() fails on every I/O and
__mpath_find_path() picks the same ANO path again.
With the change below (a template callback so the selectors use the
live state), plus the scsi_dh_alua fix, all 30 LUNs pick an AO
current path at boot, and on LUN sdl (:0/:2 AO, :1/:3 ANO):
numa 2000 0 0 0
round-robin 1000 0 1000 0
queue-depth 18039 0 13961 0 (8 parallel dd, 64k)
No I/O reached the ANO paths under any policy. This is a test patch to
show the problem, not a polished change - feel free to fold it in or
do it differently.
I'll check it. Sounds like a real problem.
As you may know, this code is really a refactoring of the NVMe multipath
code. The ambition was to have NVMe use libmultipath eventually, so it
would be good for any change to work for both NVMe and SCSI.
3. Smaller points
- numa with equal NUMA distance uses one path per LUN, so with two AO
paths through two HBAs, half the optimized bandwidth is idle (sdl:2
above under numa). That matches NVMe, but users coming from
dm-multipath service-time will notice. Would round-robin or
queue-depth be a better default for SCSI?
Maybe, but round-robin may not be always ideal considering only a bio
mode of operation is supported and no merging like dm-mpath RQ mode.
- There are no Documentation/ABI entries for the new sysfs attributes
(scsi_mpath_device vpd_id/iopolicy/multipath, delayed_removal_secs,
mpath_dev, mpath_numa_nodes, mpath_queue_depth), and the
scsi_mod.multipath and multipath_iopolicy parameters are not in
kernel-parameters.txt. I have a draft ABI file if useful.
Cool, I would suggest to post if and when this is merged. I would rather
not post another version of this series. It is just "experimental".
- There is no userspace tool yet to show heads, paths and ALUA state
(the equivalent of multipath -ll or nvme list-subsys). I used a
small sysfs script for this testing. Ben, is native SCSI multipath
support in multipath-tools planned?
Well, the whole point is that we don't need userspace tools. However, I
appreciate that there is no general SCSI tooling which could help here.
I know that Himanshu M did work on a SCSI version of nvme-cli tool
previously. Martin may know the status of that.
- CONFIG_SCSI_MULTIPATH/CONFIG_LIBMULTIPATH are only enabled in your
branch's x86_64_defconfig, so building from a distro config leaves
them off, and scsi_mod.multipath= is silently ignored. Maybe worth a
note in the cover letter.
I thought that the kernel warns of unrecognised params (which would
happen for a relevant module disabled).
I have not tested failover or path loss yet; I will follow up with
that and a Tested-by once done.
ok.
Thanks
