Re: [PATCH v1 2/2] vfio/pci: Emulate PASID/PRI capability for VFs
Hi Alex Going through the PCIe Spec, there seems a lot of such capabilities that are different between PF and VF. Some that make sense and some don't. On Sun, Apr 12, 2020 at 08:10:43PM -0700, Raj, Ashok wrote: > > > > > I agree though, I don't know why the SIG would preclude implementing > > per VF control of these features. Thanks, > > For e.g. VF doesn't have I/O and Mem space enables, but has BME Interrupt Status Correctable Error Reporting Almost all of Device Control Register. So it seems like there is a ton of them we have to deal with today for VF's. How do we manage to emulate them without any support for them in VF's? > > Cheers, > Ashok ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v1 2/2] vfio/pci: Emulate PASID/PRI capability for VFs
On Wed, Apr 08, 2020 at 10:19:40AM -0600, Alex Williamson wrote: > On Tue, 7 Apr 2020 21:00:21 -0700 > "Raj, Ashok" wrote: > > > Hi Alex > > > > + Bjorn > > + Don > > > FWIW I can't understand why PCI SIG went different ways with ATS, > > where its enumerated on PF and VF. But for PASID and PRI its only > > in PF. > > > > I'm checking with our internal SIG reps to followup on that. > > > > On Tue, Apr 07, 2020 at 09:58:01AM -0600, Alex Williamson wrote: > > > > Is there vendor guarantee that hidden registers will locate at the > > > > same offset between PF and VF config space? > > > > > > I'm not sure if the spec really precludes hidden registers, but the > > > fact that these registers are explicitly outside of the capability > > > chain implies they're only intended for device specific use, so I'd say > > > there are no guarantees about anything related to these registers. > > > > As you had suggested in the other thread, we could consider > > using the same offset as in PF, but even that's a better guess > > still not reliable. > > > > The other option is to maybe extend driver ops in the PF to expose > > where the offsets should be. Sort of adding the quirk in the > > implementation. > > > > I'm not sure how prevalent are PASID and PRI in VF devices. If SIG is > > resisting > > making VF's first class citizen, we might ask them to add some verbiage > > to suggest leave the same offsets as PF open to help emulation software. > > Even if we know where to expose these capabilities on the VF, it's not > clear to me how we can actually virtualize the capability itself. If > the spec defines, for example, an enable bit as r/w then software that > interacts with that register expects the bit is settable. There's no > protocol for "try to set the bit and re-read it to see if the hardware > accepted it". Therefore a capability with a fixed enable bit > representing the state of the PF, not settable by the VF, is > disingenuous to the spec. I think we are all in violent agreement. A lot of times the pci spec gets defined several years ahead of real products and no one remembers the justification on why they restricted things the way they did. Maybe someone early product wasn't quite exposing these features to the VF and hence the spec is bug compatible :-) > > If what we're trying to do is expose that PASID and PRI are enabled on > the PF to a VF driver, maybe duplicating the PF capabilities on the VF > without the ability to control it is not the right approach. Maybe we As long as the capability enable is only provided when the PF has enabled the feature. Then it seems the hardware seems to do the right thing. Assume we expose PASID/PRI only when PF has enabled it. It will be the case since the PF driver needs to exist, and IOMMU would have set the PASID/PRI/ATS on PF. If the emulation is purely spoofing the capability. Once vIOMMU driver enables PASID, the context entries for the VF are completely independent from the PF context entries. vIOMMU would enable PASID, and we just spoof the PASID capability. If vIOMMU or guest for some reason does disable_pasid(), then the vIOMMU driver can disaable PASID on the VF context entries. So the VF although the capability is blanket enabled on PF, IOMMU gaurantees the transactions are blocked. In the interim, it seems like the intent of the virtual capability can be honored via help from the IOMMU for the controlling aspect.. Did i miss anything? > need new capabilities exposing these as slave features that cannot be > controlled? We could define our own vendor capability for this, but of > course we have both the where to put it in config space issue, as well > as the issue of trying to push an ad-hoc standard. vfio could expose > these as device features rather than emulating capabilities, but that > still leaves a big gap between vfio in the hypervisor and the driver in > the guest VM. That might still help push the responsibility and policy > for how to expose it to the VM as a userspace problem though. I think this is a good long term solution, but if the vIOMMU implenentations can carry us for the time being, we can probably defer them unless we are stuck. > > I agree though, I don't know why the SIG would preclude implementing > per VF control of these features. Thanks, > Cheers, Ashok ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH 1/1] iommu/vt-d: use DMA domain for real DMA devices and subdevices
Hi Daniel, On 2020/4/13 10:25, Daniel Drake wrote: On Fri, Apr 10, 2020 at 9:22 AM Lu Baolu wrote: This is caused by the fragile private domain implementation. We are in process of removing it by enhancing the iommu subsystem with per-group default domain. https://www.spinics.net/lists/iommu/msg42976.html So ultimately VMD subdevices should have their own per-device iommu data and support per-device dma ops. Interesting. There's also this patchset you posted: [PATCH 00/19] [PULL REQUEST] iommu/vt-d: patches for v5.7 https://lists.linuxfoundation.org/pipermail/iommu/2020-April/042967.html (to be pushed out to 5.8) Both are trying to solve a same problem. I have sync'ed with Joerg. This patch set will be replaced with Joerg's proposal due to a race concern between domain switching and driver binding. I will rebase all vt-d patches in this set on top of Joerg's change. Best regards, baolu In there you have: iommu/vt-d: Don't force 32bit devices to uses DMA domain which seems to clash with the approach being explored in this thread. And: iommu/vt-d: Apply per-device dma_ops This effectively solves the trip point that caused me to open these discussions, where intel_map_page() -> iommu_need_mapping() would incorrectly determine that a intel-iommu DMA mapping was needed for a PCI subdevice running in identity mode. After this patch, a PCI subdevice in identity mode uses the default system dma_ops and completely avoids intel-iommu. So that solves the issues I was looking at. Jon, you might want to check if the problems you see are likewise solved for you by these patches. I didn't try Joerg's iommu group rework yet as it conflicts with those patches above. Daniel ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH 1/1] iommu/vt-d: use DMA domain for real DMA devices and subdevices
On Fri, Apr 10, 2020 at 9:22 AM Lu Baolu wrote: > This is caused by the fragile private domain implementation. We are in > process of removing it by enhancing the iommu subsystem with per-group > default domain. > > https://www.spinics.net/lists/iommu/msg42976.html > > So ultimately VMD subdevices should have their own per-device iommu data > and support per-device dma ops. Interesting. There's also this patchset you posted: [PATCH 00/19] [PULL REQUEST] iommu/vt-d: patches for v5.7 https://lists.linuxfoundation.org/pipermail/iommu/2020-April/042967.html (to be pushed out to 5.8) In there you have: > iommu/vt-d: Don't force 32bit devices to uses DMA domain which seems to clash with the approach being explored in this thread. And: > iommu/vt-d: Apply per-device dma_ops This effectively solves the trip point that caused me to open these discussions, where intel_map_page() -> iommu_need_mapping() would incorrectly determine that a intel-iommu DMA mapping was needed for a PCI subdevice running in identity mode. After this patch, a PCI subdevice in identity mode uses the default system dma_ops and completely avoids intel-iommu. So that solves the issues I was looking at. Jon, you might want to check if the problems you see are likewise solved for you by these patches. I didn't try Joerg's iommu group rework yet as it conflicts with those patches above. Daniel ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH 1/1] iommu/vt-d: use DMA domain for real DMA devices and subdevices
On Sun, Apr 12, 2020 at 11:50:09AM +0800, Daniel Drake wrote: > > different addressing capabilities and some require translation. Instead we > > can > > put the real DMA dev and any subdevices on the DMA domain. This change > > assigns > > subdevices to the DMA domain, and moves the real DMA device to the DMA > > domain > > if necessary. > > Have you tested this with the real DMA device in identity mode? > It is not quite working for me. (Again, I'm not using VMD here, but > have looked closely and believe we're working under the same > constraints) So if you are not using VMD how does this matter for upstream? ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
