** Changed in: ubuntu-z-systems
Status: In Progress => Fix Committed
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1874056
Title:
[UBUNTU 20.04] s390x/pci: enumerate pci functions per physical adapter
Status in Ubuntu on IBM z Systems:
Fix Committed
Status in linux package in Ubuntu:
In Progress
Status in linux source package in Focal:
Fix Committed
Bug description:
SRU Justification:
==================
[Impact]
* Mellanox CX5 port multi-pathing is broken on s390x due to non-
standard topology of PCI IDs (phys. and virtual)
* The Mellanox Connect-X 5 PCI driver (mlx5) implements multi-path
that can be used to combine multiple networking ports to improve
performance and reliability.
* For that purpose, the mlx5 driver combines PCI functions based on
topology information (the function number) as determined by their PCI
ID.
* Currently the Linux on Z PCI bus does not reflect PCI topology
information in the PCI ID. As a result, the mlx5 multi-path function
is broken and cannot be activated.
[Fix]
* Backport 1: https://launchpadlibrarian.net/479699471/0001-s390-pci-
Improve-handling-of-unset-UID.patch
* Backport 2: https://launchpadlibrarian.net/479699482/0002-s390-pci-
embedding-hotplug_slot-in-zdev.patch
* Backport 3: https://launchpadlibrarian.net/479699492/0003-s390-pci-
Expose-new-port-attribute-for-PCIe-function.patch
* Backport 4: https://launchpadlibrarian.net/479699497/0004-s390-pci-
adaptation-of-iommu-to-multifunction.patch
* Backport 5: https://launchpadlibrarian.net/479700706/0005-s390-pci-
define-kernel-parameters-for-PCI-multifunct.patch
* Backport 6: https://launchpadlibrarian.net/479700712/0006-s390-pci-
define-RID-and-RID-available.patch
* Backport 7: https://launchpadlibrarian.net/479700739/0007-s390-pci-
create-zPCI-bus.patch
* Backport 8: https://launchpadlibrarian.net/479700769/0008-s390-pci-
adapt-events-for-zbus.patch
* Backport 9: https://launchpadlibrarian.net/479700786/0009-s390-pci-
Handling-multifunctions.patch
* Backport 10: https://launchpadlibrarian.net/479700794/0010-s390-pci-
Do-not-disable-PF-when-VFs-exist.patch
* Backport 11: https://launchpadlibrarian.net/479700798/0011-s390-pci-
Documentation-for-zPCI.patch
* Backport 12: https://launchpadlibrarian.net/479700799/0012-s390-pci-
removes-wrong-PCI-multifunction-assignment.patch
[Test Case]
* Prepare an IBM z13 or LinuxONE III (or newer) system with two or
more RoCE Express PCI 2(.1) adapters.
* Assign the adapters (and it's virtual functions) to an LPAR.
* Verify whether the physical and virtual functions are grouped in
arbitrary order or in consecutive order - physical first (for example
with lspci -t ...)
[Regression Potential]
* The regression potential can be considered as moderate, since:
* It is purely s390x specific code (arch/s390/*
drivers/iommu/s390-iommu.c and drivers/pci/hotplug/s390_pci_hpc.c -
and some doc adjustments, too).
* It largely affects zPCI, the s390x specific PCI code layer.
* PCI cards available for s390x are optional cards (RoCE and zEDC) and
not very wide-spread.
* The situation described above affects the RoCE adapters only
(Mellanox based).
* The patches are also upstream accepted and available via linux-next,
but to apply them to focal kernel 5.4 the above backports are needed.
* However, the code is modified by several patches (12), hence there
is a chance to break zPCI with them.
* For upfront testing a PPA got created with a focal (master-next)
kernel that incl. all the above patches.
__________
Today, the enumeration of PCI functions on s390x does not reflect
which functions belongs to which physical adapter.
Layout of a PCI function address on Linux:
0000:00:00.0
<root complex>:<bus>:<device>.<function>
On s390x, each function is presented as individual root complex today,
e.g.:
PCHID 0100 VF1 0000:00:00.0
PCHID 0100 VF23 0001:00:00.0
PCHID 0200 VF1 0002:00:00.0
OCHID 0100 VF17 0003:00:00.0
On other platforms, the addresses correctly reflect the actual HW
configuration. Some device drivers (mlx5 for Mellanox adapters) group
functions of one physical adapter by checking which PCI functions have
identical values for <root complex>:<bus>:<device>. We need to use the
same enumeration scheme to achieve this functionality on s390x.
In this case, the two physical functions of a Mellanox adapter need to
get function number 0 and 1, and all virtual functions need to use the
same <root complex>:<bus> numbers with function/device numbers
counting up.
Required result (example with 4 VFs per PF):
PCHID 0100 PF 0 0000:00:00.0
PCHID 0100 PF 1 0000:00:00.1
PCHID 0100 PF 0 VF 0 0000:00:00.2
PCHID 0100 PF 0 VF 1 0000:00:00.3
PCHID 0100 PF 0 VF 2 0000:00:00.4
PCHID 0100 PF 0 VF 3 0000:00:00.5
PCHID 0100 PF 1 VF 0 0000:00:00.6
PCHID 0100 PF 1 VF 1 0000:00:00.7
PCHID 0100 PF 1 VF 2 0000:00:00.8
PCHID 0100 PF 1 VF 3 0000:00:00.9
PCHID 0200 PF 0 0001:00:00.0
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-z-systems/+bug/1874056/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
