Re: [pci PATCH v8 0/4] Add support for unmanaged SR-IOV
On Tue, Apr 24, 2018 at 2:51 PM, Bjorn Helgaas wrote: > On Sat, Apr 21, 2018 at 05:22:27PM -0700, Alexander Duyck wrote: >> On Sat, Apr 21, 2018 at 1:34 PM, Bjorn Helgaas wrote: > >> > For example, I'm not sure what you mean by "devices where the PF is >> > not capable of managing VF resources." >> > >> > It *sounds* like you're saying the hardware works differently on some >> > devices, but I don't think that's what you mean. I think you're >> > saying something about which drivers are used for the PF and the VF. >> >> That is sort of what I am saying. >> >> So for example with ixgbe there is functionality which is controlled >> in the MMIO space of the PF that affects the functionality of the VFs >> that are generated on the device. The PF has to rearrange the >> resources such as queues and interrupts on the device before it can >> enable SR-IOV, and it could alter those later to limit what the VF is >> capable of doing. >> >> The model I am dealing with via this patch set has a PF that is not >> much different than the VFs other than the fact that it has some >> extended configuration space bits in place for SR-IOV, ARI, ACS, and >> whatever other bits are needed in order to support spawning isolated >> VFs. > > OK, thanks for the explanation, I think I understand what's going on > now, correct me if I'm mistaken. I added a hint about "PF" for Randy, > too. > > These are on pci/virtualization for v4.18. I reviewed them and all of the changes to patches 1 & 2 both below, and in the tree look good to me. Thanks for taking care of all this. - Alex > commit 8effc395c209 > Author: Alexander Duyck > Date: Sat Apr 21 15:23:09 2018 -0500 > > PCI/IOV: Add pci_sriov_configure_simple() > > SR-IOV (Single Root I/O Virtualization) is an optional PCIe capability > (see > PCIe r4.0, sec 9). A PCIe Function with the SR-IOV capability is referred > to as a PF (Physical Function). If SR-IOV is enabled on the PF, several > VFs (Virtual Functions) may be created. The VFs can be individually > assigned to virtual machines, which allows them to share a single hardware > device while being isolated from each other. > > Some SR-IOV devices have resources such as queues and interrupts that must > be set up in the PF before enabling the VFs, so they require a PF driver > to > do that. > > Other SR-IOV devices don't require any PF setup before enabling VFs. Add > a > pci_sriov_configure_simple() interface so PF drivers for such devices can > use it without repeating the VF-enabling code. > > Tested-by: Mark Rustad > Signed-off-by: Alexander Duyck > [bhelgaas: changelog, comment] > Signed-off-by: Bjorn Helgaas > Reviewed-by: Greg Rose > Reviewed-by: Christoph Hellwig :wq > > commit a8ccf8a3 > Author: Alexander Duyck > Date: Tue Apr 24 16:47:16 2018 -0500 > > PCI/IOV: Add pci-pf-stub driver for PFs that only enable VFs > > Some SR-IOV PF devices provide no functionality other than acting as a > means of enabling VFs. For these devices, we want to enable the VFs and > assign them to guest virtual machines, but there's no need to have a > driver > for the PF itself. > > Add a new pci-pf-stub driver to claim those PF devices and provide the > generic VF enable functionality. An administrator can use the sysfs > "sriov_numvfs" file to enable VFs, then assign them to guests. > > For now I only have one example ID provided by Amazon in terms of devices > that require this functionality. The general idea is that in the future > we > will see other devices added as vendors come up with devices where the PF > is more or less just a lightweight shim used to allocate VFs. > > Signed-off-by: Alexander Duyck > [bhelgaas: changelog] > Signed-off-by: Bjorn Helgaas > Reviewed-by: Greg Rose > Reviewed-by: Christoph Hellwig > > commit 115ddc491922 > Author: Alexander Duyck > Date: Tue Apr 24 16:47:22 2018 -0500 > > net: ena: Use pci_sriov_configure_simple() to enable VFs > > Instead of implementing our own version of a SR-IOV configuration stub in > the ena driver, use the existing pci_sriov_configure_simple() function. > > Signed-off-by: Alexander Duyck > Signed-off-by: Bjorn Helgaas > Reviewed-by: Greg Rose > Reviewed-by: Christoph Hellwig > > commit 74d986abc20b > Author: Alexander Duyck > Date: Tue Apr 24 16:47:27 2018 -0500 > > nvme-pci: Use pci_sriov_configure_simple() to enable VFs > > Instead of implementing our own version of a SR-IOV configuration stub in > the nvme driver, use the existing pci_sriov_configure_simple() function. > > Signed-off-by: Alexander Duyck > Signed-off-by: Bjorn Helgaas > Reviewed-by: Christoph Hellwig
Re: [pci PATCH v8 0/4] Add support for unmanaged SR-IOV
On Sat, Apr 21, 2018 at 05:22:27PM -0700, Alexander Duyck wrote: > On Sat, Apr 21, 2018 at 1:34 PM, Bjorn Helgaas wrote: > > For example, I'm not sure what you mean by "devices where the PF is > > not capable of managing VF resources." > > > > It *sounds* like you're saying the hardware works differently on some > > devices, but I don't think that's what you mean. I think you're > > saying something about which drivers are used for the PF and the VF. > > That is sort of what I am saying. > > So for example with ixgbe there is functionality which is controlled > in the MMIO space of the PF that affects the functionality of the VFs > that are generated on the device. The PF has to rearrange the > resources such as queues and interrupts on the device before it can > enable SR-IOV, and it could alter those later to limit what the VF is > capable of doing. > > The model I am dealing with via this patch set has a PF that is not > much different than the VFs other than the fact that it has some > extended configuration space bits in place for SR-IOV, ARI, ACS, and > whatever other bits are needed in order to support spawning isolated > VFs. OK, thanks for the explanation, I think I understand what's going on now, correct me if I'm mistaken. I added a hint about "PF" for Randy, too. These are on pci/virtualization for v4.18. commit 8effc395c209 Author: Alexander Duyck Date: Sat Apr 21 15:23:09 2018 -0500 PCI/IOV: Add pci_sriov_configure_simple() SR-IOV (Single Root I/O Virtualization) is an optional PCIe capability (see PCIe r4.0, sec 9). A PCIe Function with the SR-IOV capability is referred to as a PF (Physical Function). If SR-IOV is enabled on the PF, several VFs (Virtual Functions) may be created. The VFs can be individually assigned to virtual machines, which allows them to share a single hardware device while being isolated from each other. Some SR-IOV devices have resources such as queues and interrupts that must be set up in the PF before enabling the VFs, so they require a PF driver to do that. Other SR-IOV devices don't require any PF setup before enabling VFs. Add a pci_sriov_configure_simple() interface so PF drivers for such devices can use it without repeating the VF-enabling code. Tested-by: Mark Rustad Signed-off-by: Alexander Duyck [bhelgaas: changelog, comment] Signed-off-by: Bjorn Helgaas Reviewed-by: Greg Rose Reviewed-by: Christoph Hellwig :wq commit a8ccf8a3 Author: Alexander Duyck Date: Tue Apr 24 16:47:16 2018 -0500 PCI/IOV: Add pci-pf-stub driver for PFs that only enable VFs Some SR-IOV PF devices provide no functionality other than acting as a means of enabling VFs. For these devices, we want to enable the VFs and assign them to guest virtual machines, but there's no need to have a driver for the PF itself. Add a new pci-pf-stub driver to claim those PF devices and provide the generic VF enable functionality. An administrator can use the sysfs "sriov_numvfs" file to enable VFs, then assign them to guests. For now I only have one example ID provided by Amazon in terms of devices that require this functionality. The general idea is that in the future we will see other devices added as vendors come up with devices where the PF is more or less just a lightweight shim used to allocate VFs. Signed-off-by: Alexander Duyck [bhelgaas: changelog] Signed-off-by: Bjorn Helgaas Reviewed-by: Greg Rose Reviewed-by: Christoph Hellwig commit 115ddc491922 Author: Alexander Duyck Date: Tue Apr 24 16:47:22 2018 -0500 net: ena: Use pci_sriov_configure_simple() to enable VFs Instead of implementing our own version of a SR-IOV configuration stub in the ena driver, use the existing pci_sriov_configure_simple() function. Signed-off-by: Alexander Duyck Signed-off-by: Bjorn Helgaas Reviewed-by: Greg Rose Reviewed-by: Christoph Hellwig commit 74d986abc20b Author: Alexander Duyck Date: Tue Apr 24 16:47:27 2018 -0500 nvme-pci: Use pci_sriov_configure_simple() to enable VFs Instead of implementing our own version of a SR-IOV configuration stub in the nvme driver, use the existing pci_sriov_configure_simple() function. Signed-off-by: Alexander Duyck Signed-off-by: Bjorn Helgaas Reviewed-by: Christoph Hellwig
Re: [pci PATCH v8 0/4] Add support for unmanaged SR-IOV
On Mon, Apr 23, 2018 at 8:21 AM, Don Dutile wrote: > On 04/21/2018 04:34 PM, Bjorn Helgaas wrote: >> >> On Fri, Apr 20, 2018 at 12:28:08PM -0400, Alexander Duyck wrote: >>> >>> This series is meant to add support for SR-IOV on devices when the VFs >>> are >>> not managed by the kernel. Examples of recent patches attempting to do >>> this >>> include: >>> virto - https://patchwork.kernel.org/patch/10241225/ >>> pci-stub - https://patchwork.kernel.org/patch/10109935/ >>> vfio - https://patchwork.kernel.org/patch/10103353/ >>> uio - https://patchwork.kernel.org/patch/9974031/ >>> >>> Since this is quickly blowing up into a multi-driver problem it is >>> probably >>> best to implement this solution as generically as possible. >>> >>> This series is an attempt to do that. What we do with this patch set is >>> provide a generic framework to enable SR-IOV in the case that the PF >>> driver >>> doesn't support managing the VFs itself. >>> >>> I based my patch set originally on the patch by Mark Rustad but there >>> isn't >>> much left after going through and cleaning out the bits that were no >>> longer >>> needed, and after incorporating the feedback from David Miller. At this >>> point >>> the only items to be fully reused was his patch description which is now >>> present in patch 3 of the set. >>> >>> This solution is limited in scope to just adding support for devices that >>> provide no functionality for SR-IOV other than allocating the VFs by >>> calling pci_enable_sriov. Previous sets had included patches for VFIO, >>> but >>> for now I am dropping that as the scope of that work is larger then I >>> think I can take on at this time. >>> >>> v2: Reduced scope back to just virtio_pci and vfio-pci >>> Broke into 3 patch set from single patch >>> Changed autoprobe behavior to always set when num_vfs is set >>> non-zero >>> v3: Updated Documentation to clarify when sriov_unmanaged_autoprobe is >>> used >>> Wrapped vfio_pci_sriov_configure to fix build errors w/o SR-IOV in >>> kernel >>> v4: Dropped vfio-pci patch >>> Added ena and nvme to drivers now using >>> pci_sriov_configure_unmanaged >>> Dropped pci_disable_sriov call in virtio_pci to be consistent with >>> ena >>> v5: Dropped sriov_unmanaged_autoprobe and pci_sriov_conifgure_unmanaged >>> Added new patch that enables pci_sriov_configure_simple >>> Updated drivers to use pci_sriov_configure_simple >>> v6: Defined pci_sriov_configure_simple as NULL when SR-IOV is not enabled >>> Updated drivers to drop "#ifdef" checks for IOV >>> Added pci-pf-stub as place for PF-only drivers to add support >>> v7: Dropped pci_id table explanation from pci-pf-stub driver >>> Updated pci_sriov_configure_simple to drop need for err value >>> Fixed comment explaining why pci_sriov_configure_simple is NULL >>> v8: Dropped virtio from the set, support to be added later after TC >>> approval >>> >>> Cc: Mark Rustad >>> Cc: Maximilian Heyne >>> Cc: Liang-Min Wang >>> Cc: David Woodhouse >>> >>> --- >>> >>> Alexander Duyck (4): >>>pci: Add pci_sriov_configure_simple for PFs that don't manage VF >>> resources >>>ena: Migrate over to unmanaged SR-IOV support >>>nvme: Migrate over to unmanaged SR-IOV support >>>pci-pf-stub: Add PF driver stub for PFs that function only to >>> enable VFs >>> >>> >>> drivers/net/ethernet/amazon/ena/ena_netdev.c | 28 - >>> drivers/nvme/host/pci.c | 20 -- >>> drivers/pci/Kconfig | 12 ++ >>> drivers/pci/Makefile |2 + >>> drivers/pci/iov.c| 31 +++ >>> drivers/pci/pci-pf-stub.c| 54 >>> ++ >>> include/linux/pci.h |3 + >>> include/linux/pci_ids.h |2 + >>> 8 files changed, 106 insertions(+), 46 deletions(-) >>> create mode 100644 drivers/pci/pci-pf-stub.c >> >> >> I tentatively applied these to pci/virtualization-review. >> >> The code changes look fine, but I want to flesh out the changelogs a >> little bit before merging them. >> >> For example, I'm not sure what you mean by "devices where the PF is >> not capable of managing VF resources." >> > I agree w/Bjorn's assessment of the changelog. > The VF's are (minimally) assigned via the pf-stub driver, so they are > 'managed by the kernel'. > The security model is the same as the existing one, which was the issue we > resolved in the previous set(s) of patches. > > I am hoping that something like vfio will be used to deal with the VF > ownership > and the reset mechanisms during assignement & de-assignment to 'guests' > (qemu-kvm, DPDK, or whatever user-process), > so the known, existing security model(s) is(are) maintained as well. > If so, it'd be good to add such verbage somewhere (as 0/n is not kept in > anything but possibly Bjorn's patchwork, or whateve
Re: [pci PATCH v8 0/4] Add support for unmanaged SR-IOV
On 04/21/2018 04:34 PM, Bjorn Helgaas wrote: On Fri, Apr 20, 2018 at 12:28:08PM -0400, Alexander Duyck wrote: This series is meant to add support for SR-IOV on devices when the VFs are not managed by the kernel. Examples of recent patches attempting to do this include: virto - https://patchwork.kernel.org/patch/10241225/ pci-stub - https://patchwork.kernel.org/patch/10109935/ vfio - https://patchwork.kernel.org/patch/10103353/ uio - https://patchwork.kernel.org/patch/9974031/ Since this is quickly blowing up into a multi-driver problem it is probably best to implement this solution as generically as possible. This series is an attempt to do that. What we do with this patch set is provide a generic framework to enable SR-IOV in the case that the PF driver doesn't support managing the VFs itself. I based my patch set originally on the patch by Mark Rustad but there isn't much left after going through and cleaning out the bits that were no longer needed, and after incorporating the feedback from David Miller. At this point the only items to be fully reused was his patch description which is now present in patch 3 of the set. This solution is limited in scope to just adding support for devices that provide no functionality for SR-IOV other than allocating the VFs by calling pci_enable_sriov. Previous sets had included patches for VFIO, but for now I am dropping that as the scope of that work is larger then I think I can take on at this time. v2: Reduced scope back to just virtio_pci and vfio-pci Broke into 3 patch set from single patch Changed autoprobe behavior to always set when num_vfs is set non-zero v3: Updated Documentation to clarify when sriov_unmanaged_autoprobe is used Wrapped vfio_pci_sriov_configure to fix build errors w/o SR-IOV in kernel v4: Dropped vfio-pci patch Added ena and nvme to drivers now using pci_sriov_configure_unmanaged Dropped pci_disable_sriov call in virtio_pci to be consistent with ena v5: Dropped sriov_unmanaged_autoprobe and pci_sriov_conifgure_unmanaged Added new patch that enables pci_sriov_configure_simple Updated drivers to use pci_sriov_configure_simple v6: Defined pci_sriov_configure_simple as NULL when SR-IOV is not enabled Updated drivers to drop "#ifdef" checks for IOV Added pci-pf-stub as place for PF-only drivers to add support v7: Dropped pci_id table explanation from pci-pf-stub driver Updated pci_sriov_configure_simple to drop need for err value Fixed comment explaining why pci_sriov_configure_simple is NULL v8: Dropped virtio from the set, support to be added later after TC approval Cc: Mark Rustad Cc: Maximilian Heyne Cc: Liang-Min Wang Cc: David Woodhouse --- Alexander Duyck (4): pci: Add pci_sriov_configure_simple for PFs that don't manage VF resources ena: Migrate over to unmanaged SR-IOV support nvme: Migrate over to unmanaged SR-IOV support pci-pf-stub: Add PF driver stub for PFs that function only to enable VFs drivers/net/ethernet/amazon/ena/ena_netdev.c | 28 - drivers/nvme/host/pci.c | 20 -- drivers/pci/Kconfig | 12 ++ drivers/pci/Makefile |2 + drivers/pci/iov.c| 31 +++ drivers/pci/pci-pf-stub.c| 54 ++ include/linux/pci.h |3 + include/linux/pci_ids.h |2 + 8 files changed, 106 insertions(+), 46 deletions(-) create mode 100644 drivers/pci/pci-pf-stub.c I tentatively applied these to pci/virtualization-review. The code changes look fine, but I want to flesh out the changelogs a little bit before merging them. For example, I'm not sure what you mean by "devices where the PF is not capable of managing VF resources." I agree w/Bjorn's assessment of the changelog. The VF's are (minimally) assigned via the pf-stub driver, so they are 'managed by the kernel'. The security model is the same as the existing one, which was the issue we resolved in the previous set(s) of patches. I am hoping that something like vfio will be used to deal with the VF ownership and the reset mechanisms during assignement & de-assignment to 'guests' (qemu-kvm, DPDK, or whatever user-process), so the known, existing security model(s) is(are) maintained as well. If so, it'd be good to add such verbage somewhere (as 0/n is not kept in anything but possibly Bjorn's patchwork, or whatever patch mgmt tool he uses, and future reference would be good to have) say, an update to Documentation/PCI/pci-iov-howto.txt. So... the 'unmanaged SR-IOV' Subject, IMO, is not a valid Subject for the patch series any longer. No objections to the patch series, as Bjorn noted, just the commit log(s)/nomenclature of what is really being done. The expectation of VF enablement via the PF was born out of the fairly complicated, and unique
Re: [pci PATCH v8 0/4] Add support for unmanaged SR-IOV
On Sat, Apr 21, 2018 at 1:34 PM, Bjorn Helgaas wrote: > On Fri, Apr 20, 2018 at 12:28:08PM -0400, Alexander Duyck wrote: >> This series is meant to add support for SR-IOV on devices when the VFs are >> not managed by the kernel. Examples of recent patches attempting to do this >> include: >> virto - https://patchwork.kernel.org/patch/10241225/ >> pci-stub - https://patchwork.kernel.org/patch/10109935/ >> vfio - https://patchwork.kernel.org/patch/10103353/ >> uio - https://patchwork.kernel.org/patch/9974031/ >> >> Since this is quickly blowing up into a multi-driver problem it is probably >> best to implement this solution as generically as possible. >> >> This series is an attempt to do that. What we do with this patch set is >> provide a generic framework to enable SR-IOV in the case that the PF driver >> doesn't support managing the VFs itself. >> >> I based my patch set originally on the patch by Mark Rustad but there isn't >> much left after going through and cleaning out the bits that were no longer >> needed, and after incorporating the feedback from David Miller. At this point >> the only items to be fully reused was his patch description which is now >> present in patch 3 of the set. >> >> This solution is limited in scope to just adding support for devices that >> provide no functionality for SR-IOV other than allocating the VFs by >> calling pci_enable_sriov. Previous sets had included patches for VFIO, but >> for now I am dropping that as the scope of that work is larger then I >> think I can take on at this time. >> >> v2: Reduced scope back to just virtio_pci and vfio-pci >> Broke into 3 patch set from single patch >> Changed autoprobe behavior to always set when num_vfs is set non-zero >> v3: Updated Documentation to clarify when sriov_unmanaged_autoprobe is used >> Wrapped vfio_pci_sriov_configure to fix build errors w/o SR-IOV in kernel >> v4: Dropped vfio-pci patch >> Added ena and nvme to drivers now using pci_sriov_configure_unmanaged >> Dropped pci_disable_sriov call in virtio_pci to be consistent with ena >> v5: Dropped sriov_unmanaged_autoprobe and pci_sriov_conifgure_unmanaged >> Added new patch that enables pci_sriov_configure_simple >> Updated drivers to use pci_sriov_configure_simple >> v6: Defined pci_sriov_configure_simple as NULL when SR-IOV is not enabled >> Updated drivers to drop "#ifdef" checks for IOV >> Added pci-pf-stub as place for PF-only drivers to add support >> v7: Dropped pci_id table explanation from pci-pf-stub driver >> Updated pci_sriov_configure_simple to drop need for err value >> Fixed comment explaining why pci_sriov_configure_simple is NULL >> v8: Dropped virtio from the set, support to be added later after TC approval >> >> Cc: Mark Rustad >> Cc: Maximilian Heyne >> Cc: Liang-Min Wang >> Cc: David Woodhouse >> >> --- >> >> Alexander Duyck (4): >> pci: Add pci_sriov_configure_simple for PFs that don't manage VF >> resources >> ena: Migrate over to unmanaged SR-IOV support >> nvme: Migrate over to unmanaged SR-IOV support >> pci-pf-stub: Add PF driver stub for PFs that function only to enable >> VFs >> >> >> drivers/net/ethernet/amazon/ena/ena_netdev.c | 28 - >> drivers/nvme/host/pci.c | 20 -- >> drivers/pci/Kconfig | 12 ++ >> drivers/pci/Makefile |2 + >> drivers/pci/iov.c| 31 +++ >> drivers/pci/pci-pf-stub.c| 54 >> ++ >> include/linux/pci.h |3 + >> include/linux/pci_ids.h |2 + >> 8 files changed, 106 insertions(+), 46 deletions(-) >> create mode 100644 drivers/pci/pci-pf-stub.c > > I tentatively applied these to pci/virtualization-review. > > The code changes look fine, but I want to flesh out the changelogs a > little bit before merging them. Thanks. > For example, I'm not sure what you mean by "devices where the PF is > not capable of managing VF resources." > > It *sounds* like you're saying the hardware works differently on some > devices, but I don't think that's what you mean. I think you're > saying something about which drivers are used for the PF and the VF. That is sort of what I am saying. So for example with ixgbe there is functionality which is controlled in the MMIO space of the PF that affects the functionality of the VFs that are generated on the device. The PF has to rearrange the resources such as queues and interrupts on the device before it can enable SR-IOV, and it could alter those later to limit what the VF is capable of doing. The model I am dealing with via this patch set has a PF that is not much different than the VFs other than the fact that it has some extended configuration space bits in place for SR-IOV, ARI, ACS, and whatever other bits are needed in order to support spawning isolated VFs. >
Re: [pci PATCH v8 0/4] Add support for unmanaged SR-IOV
On Fri, Apr 20, 2018 at 12:28:08PM -0400, Alexander Duyck wrote: > This series is meant to add support for SR-IOV on devices when the VFs are > not managed by the kernel. Examples of recent patches attempting to do this > include: > virto - https://patchwork.kernel.org/patch/10241225/ > pci-stub - https://patchwork.kernel.org/patch/10109935/ > vfio - https://patchwork.kernel.org/patch/10103353/ > uio - https://patchwork.kernel.org/patch/9974031/ > > Since this is quickly blowing up into a multi-driver problem it is probably > best to implement this solution as generically as possible. > > This series is an attempt to do that. What we do with this patch set is > provide a generic framework to enable SR-IOV in the case that the PF driver > doesn't support managing the VFs itself. > > I based my patch set originally on the patch by Mark Rustad but there isn't > much left after going through and cleaning out the bits that were no longer > needed, and after incorporating the feedback from David Miller. At this point > the only items to be fully reused was his patch description which is now > present in patch 3 of the set. > > This solution is limited in scope to just adding support for devices that > provide no functionality for SR-IOV other than allocating the VFs by > calling pci_enable_sriov. Previous sets had included patches for VFIO, but > for now I am dropping that as the scope of that work is larger then I > think I can take on at this time. > > v2: Reduced scope back to just virtio_pci and vfio-pci > Broke into 3 patch set from single patch > Changed autoprobe behavior to always set when num_vfs is set non-zero > v3: Updated Documentation to clarify when sriov_unmanaged_autoprobe is used > Wrapped vfio_pci_sriov_configure to fix build errors w/o SR-IOV in kernel > v4: Dropped vfio-pci patch > Added ena and nvme to drivers now using pci_sriov_configure_unmanaged > Dropped pci_disable_sriov call in virtio_pci to be consistent with ena > v5: Dropped sriov_unmanaged_autoprobe and pci_sriov_conifgure_unmanaged > Added new patch that enables pci_sriov_configure_simple > Updated drivers to use pci_sriov_configure_simple > v6: Defined pci_sriov_configure_simple as NULL when SR-IOV is not enabled > Updated drivers to drop "#ifdef" checks for IOV > Added pci-pf-stub as place for PF-only drivers to add support > v7: Dropped pci_id table explanation from pci-pf-stub driver > Updated pci_sriov_configure_simple to drop need for err value > Fixed comment explaining why pci_sriov_configure_simple is NULL > v8: Dropped virtio from the set, support to be added later after TC approval > > Cc: Mark Rustad > Cc: Maximilian Heyne > Cc: Liang-Min Wang > Cc: David Woodhouse > > --- > > Alexander Duyck (4): > pci: Add pci_sriov_configure_simple for PFs that don't manage VF > resources > ena: Migrate over to unmanaged SR-IOV support > nvme: Migrate over to unmanaged SR-IOV support > pci-pf-stub: Add PF driver stub for PFs that function only to enable VFs > > > drivers/net/ethernet/amazon/ena/ena_netdev.c | 28 - > drivers/nvme/host/pci.c | 20 -- > drivers/pci/Kconfig | 12 ++ > drivers/pci/Makefile |2 + > drivers/pci/iov.c| 31 +++ > drivers/pci/pci-pf-stub.c| 54 > ++ > include/linux/pci.h |3 + > include/linux/pci_ids.h |2 + > 8 files changed, 106 insertions(+), 46 deletions(-) > create mode 100644 drivers/pci/pci-pf-stub.c I tentatively applied these to pci/virtualization-review. The code changes look fine, but I want to flesh out the changelogs a little bit before merging them. For example, I'm not sure what you mean by "devices where the PF is not capable of managing VF resources." It *sounds* like you're saying the hardware works differently on some devices, but I don't think that's what you mean. I think you're saying something about which drivers are used for the PF and the VF. I think a trivial example of how this will be used might help. I assume this involves a virtualization scenario where the host uses the PF to enable several VFs, but the host doesn't use the PF for much else. Then you assign the VFs to guests, and drivers in the guest OSes use the VFs. Since .sriov_configure() is only used by sriov_numvfs_store(), I assume the usage model involves writing to the sysfs sriov_numvfs attribute to enable the VFs, then assigning them to guests? Bjorn
Re: [pci PATCH v8 0/4] Add support for unmanaged SR-IOV
On 04/20/18 13:01, Alexander Duyck wrote: > On Fri, Apr 20, 2018 at 10:23 AM, Randy Dunlap wrote: >> On 04/20/18 09:28, Alexander Duyck wrote: >>> This series is meant to add support for SR-IOV on devices when the VFs are >>> not managed by the kernel. Examples of recent patches attempting to do this >>> include: >>> virto - https://patchwork.kernel.org/patch/10241225/ >>> pci-stub - https://patchwork.kernel.org/patch/10109935/ >>> vfio - https://patchwork.kernel.org/patch/10103353/ >>> uio - https://patchwork.kernel.org/patch/9974031/ >> >> Hi, >> >> Somewhere in this patch series it would be nice to tell us what the heck >> a "PF" is. :) >> >> Thanks. > > Sorry, I was kind of operating on the assumption of everyone > understanding SR-IOV nomenclature. Yes, I understood that. :) > A "PF" is a PCIe Physical Function. When you bring up a PCIe device > that supports SR-IOV it is the device that is there to begin with. > > A "VF" is a PCIe Virtual Function. You could think of as a logical > device that is spawned from the physical function using a combination > of hardware configuration via the SR-IOV block in the PCIe extended > configuration space and kernel/driver features. > > There are also a number of online resources you could use to research > SR-IOV further. Hope that helps to clarify some of this. > > Thanks. Thank you. -- ~Randy
Re: [pci PATCH v8 0/4] Add support for unmanaged SR-IOV
On Fri, Apr 20, 2018 at 10:23 AM, Randy Dunlap wrote: > On 04/20/18 09:28, Alexander Duyck wrote: >> This series is meant to add support for SR-IOV on devices when the VFs are >> not managed by the kernel. Examples of recent patches attempting to do this >> include: >> virto - https://patchwork.kernel.org/patch/10241225/ >> pci-stub - https://patchwork.kernel.org/patch/10109935/ >> vfio - https://patchwork.kernel.org/patch/10103353/ >> uio - https://patchwork.kernel.org/patch/9974031/ > > Hi, > > Somewhere in this patch series it would be nice to tell us what the heck > a "PF" is. :) > > Thanks. Sorry, I was kind of operating on the assumption of everyone understanding SR-IOV nomenclature. A "PF" is a PCIe Physical Function. When you bring up a PCIe device that supports SR-IOV it is the device that is there to begin with. A "VF" is a PCIe Virtual Function. You could think of as a logical device that is spawned from the physical function using a combination of hardware configuration via the SR-IOV block in the PCIe extended configuration space and kernel/driver features. There are also a number of online resources you could use to research SR-IOV further. Hope that helps to clarify some of this. Thanks. Alex
Re: [pci PATCH v8 0/4] Add support for unmanaged SR-IOV
On 04/20/18 09:28, Alexander Duyck wrote: > This series is meant to add support for SR-IOV on devices when the VFs are > not managed by the kernel. Examples of recent patches attempting to do this > include: > virto - https://patchwork.kernel.org/patch/10241225/ > pci-stub - https://patchwork.kernel.org/patch/10109935/ > vfio - https://patchwork.kernel.org/patch/10103353/ > uio - https://patchwork.kernel.org/patch/9974031/ Hi, Somewhere in this patch series it would be nice to tell us what the heck a "PF" is. :) Thanks. > Since this is quickly blowing up into a multi-driver problem it is probably > best to implement this solution as generically as possible. > > This series is an attempt to do that. What we do with this patch set is > provide a generic framework to enable SR-IOV in the case that the PF driver > doesn't support managing the VFs itself. > > I based my patch set originally on the patch by Mark Rustad but there isn't > much left after going through and cleaning out the bits that were no longer > needed, and after incorporating the feedback from David Miller. At this point > the only items to be fully reused was his patch description which is now > present in patch 3 of the set. > > This solution is limited in scope to just adding support for devices that > provide no functionality for SR-IOV other than allocating the VFs by > calling pci_enable_sriov. Previous sets had included patches for VFIO, but > for now I am dropping that as the scope of that work is larger then I > think I can take on at this time. > > v2: Reduced scope back to just virtio_pci and vfio-pci > Broke into 3 patch set from single patch > Changed autoprobe behavior to always set when num_vfs is set non-zero > v3: Updated Documentation to clarify when sriov_unmanaged_autoprobe is used > Wrapped vfio_pci_sriov_configure to fix build errors w/o SR-IOV in kernel > v4: Dropped vfio-pci patch > Added ena and nvme to drivers now using pci_sriov_configure_unmanaged > Dropped pci_disable_sriov call in virtio_pci to be consistent with ena > v5: Dropped sriov_unmanaged_autoprobe and pci_sriov_conifgure_unmanaged > Added new patch that enables pci_sriov_configure_simple > Updated drivers to use pci_sriov_configure_simple > v6: Defined pci_sriov_configure_simple as NULL when SR-IOV is not enabled > Updated drivers to drop "#ifdef" checks for IOV > Added pci-pf-stub as place for PF-only drivers to add support > v7: Dropped pci_id table explanation from pci-pf-stub driver > Updated pci_sriov_configure_simple to drop need for err value > Fixed comment explaining why pci_sriov_configure_simple is NULL > v8: Dropped virtio from the set, support to be added later after TC approval > > Cc: Mark Rustad > Cc: Maximilian Heyne > Cc: Liang-Min Wang > Cc: David Woodhouse > > --- > > Alexander Duyck (4): > pci: Add pci_sriov_configure_simple for PFs that don't manage VF > resources > ena: Migrate over to unmanaged SR-IOV support > nvme: Migrate over to unmanaged SR-IOV support > pci-pf-stub: Add PF driver stub for PFs that function only to enable VFs > > > drivers/net/ethernet/amazon/ena/ena_netdev.c | 28 - > drivers/nvme/host/pci.c | 20 -- > drivers/pci/Kconfig | 12 ++ > drivers/pci/Makefile |2 + > drivers/pci/iov.c| 31 +++ > drivers/pci/pci-pf-stub.c| 54 > ++ > include/linux/pci.h |3 + > include/linux/pci_ids.h |2 + > 8 files changed, 106 insertions(+), 46 deletions(-) > create mode 100644 drivers/pci/pci-pf-stub.c > > -- > -- ~Randy
[pci PATCH v8 0/4] Add support for unmanaged SR-IOV
This series is meant to add support for SR-IOV on devices when the VFs are not managed by the kernel. Examples of recent patches attempting to do this include: virto - https://patchwork.kernel.org/patch/10241225/ pci-stub - https://patchwork.kernel.org/patch/10109935/ vfio - https://patchwork.kernel.org/patch/10103353/ uio - https://patchwork.kernel.org/patch/9974031/ Since this is quickly blowing up into a multi-driver problem it is probably best to implement this solution as generically as possible. This series is an attempt to do that. What we do with this patch set is provide a generic framework to enable SR-IOV in the case that the PF driver doesn't support managing the VFs itself. I based my patch set originally on the patch by Mark Rustad but there isn't much left after going through and cleaning out the bits that were no longer needed, and after incorporating the feedback from David Miller. At this point the only items to be fully reused was his patch description which is now present in patch 3 of the set. This solution is limited in scope to just adding support for devices that provide no functionality for SR-IOV other than allocating the VFs by calling pci_enable_sriov. Previous sets had included patches for VFIO, but for now I am dropping that as the scope of that work is larger then I think I can take on at this time. v2: Reduced scope back to just virtio_pci and vfio-pci Broke into 3 patch set from single patch Changed autoprobe behavior to always set when num_vfs is set non-zero v3: Updated Documentation to clarify when sriov_unmanaged_autoprobe is used Wrapped vfio_pci_sriov_configure to fix build errors w/o SR-IOV in kernel v4: Dropped vfio-pci patch Added ena and nvme to drivers now using pci_sriov_configure_unmanaged Dropped pci_disable_sriov call in virtio_pci to be consistent with ena v5: Dropped sriov_unmanaged_autoprobe and pci_sriov_conifgure_unmanaged Added new patch that enables pci_sriov_configure_simple Updated drivers to use pci_sriov_configure_simple v6: Defined pci_sriov_configure_simple as NULL when SR-IOV is not enabled Updated drivers to drop "#ifdef" checks for IOV Added pci-pf-stub as place for PF-only drivers to add support v7: Dropped pci_id table explanation from pci-pf-stub driver Updated pci_sriov_configure_simple to drop need for err value Fixed comment explaining why pci_sriov_configure_simple is NULL v8: Dropped virtio from the set, support to be added later after TC approval Cc: Mark Rustad Cc: Maximilian Heyne Cc: Liang-Min Wang Cc: David Woodhouse --- Alexander Duyck (4): pci: Add pci_sriov_configure_simple for PFs that don't manage VF resources ena: Migrate over to unmanaged SR-IOV support nvme: Migrate over to unmanaged SR-IOV support pci-pf-stub: Add PF driver stub for PFs that function only to enable VFs drivers/net/ethernet/amazon/ena/ena_netdev.c | 28 - drivers/nvme/host/pci.c | 20 -- drivers/pci/Kconfig | 12 ++ drivers/pci/Makefile |2 + drivers/pci/iov.c| 31 +++ drivers/pci/pci-pf-stub.c| 54 ++ include/linux/pci.h |3 + include/linux/pci_ids.h |2 + 8 files changed, 106 insertions(+), 46 deletions(-) create mode 100644 drivers/pci/pci-pf-stub.c --
