Re: [PATCH v8 3/9] Documentation/x86: Add documentation for SVA (Shared Virtual Addressing)
On Thu, Sep 17, 2020 at 07:30:41PM +0200, Borislav Petkov wrote: > On Thu, Sep 17, 2020 at 10:22:39AM -0700, Raj, Ashok wrote: > > s/translation again/translation > > Ok, last one. Now stop looking at that text because you'll find more. > > :-))) Thank you very much for taking care of the series, Boris! I tested the tip:x86/pasid branch and everything works fine by my tests. I'll enable more tests for the branch. -Fenghua ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v8 3/9] Documentation/x86: Add documentation for SVA (Shared Virtual Addressing)
On Thu, Sep 17, 2020 at 10:22:39AM -0700, Raj, Ashok wrote: > s/translation again/translation Ok, last one. Now stop looking at that text because you'll find more. :-))) -- Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v8 3/9] Documentation/x86: Add documentation for SVA (Shared Virtual Addressing)
Thanks Boris. multiple "again" makes it funny again :-) On Thu, Sep 17, 2020 at 07:18:49PM +0200, Borislav Petkov wrote: > On Thu, Sep 17, 2020 at 07:56:09AM -0700, Raj, Ashok wrote: > > Just tweaked it a bit: > > > > "When ATS lookup fails for a virtual address, device should use PRI in > > order to request the virtual address to be paged into the CPU page tables. > > The device must use ATS again in order the fetch the translation again s/translation again/translation > > before use" > > Thanks, amended. > ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v8 3/9] Documentation/x86: Add documentation for SVA (Shared Virtual Addressing)
On Thu, Sep 17, 2020 at 07:56:09AM -0700, Raj, Ashok wrote: > Just tweaked it a bit: > > "When ATS lookup fails for a virtual address, device should use PRI in > order to request the virtual address to be paged into the CPU page tables. > The device must use ATS again in order the fetch the translation again > before use" Thanks, amended. -- Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v8 3/9] Documentation/x86: Add documentation for SVA (Shared Virtual Addressing)
Hi Boris, On Thu, Sep 17, 2020 at 09:53:38AM +0200, Borislav Petkov wrote: > On Tue, Sep 15, 2020 at 09:30:07AM -0700, Fenghua Yu wrote: > > +Background > > +== > > + > > +Shared Virtual Addressing (SVA) allows the processor and device to use the > > +same virtual addresses avoiding the need for software to translate virtual > > +addresses to physical addresses. SVA is what PCIe calls Shared Virtual > > +Memory (SVM). > > + > > +In addition to the convenience of using application virtual addresses > > +by the device, it also doesn't require pinning pages for DMA. > > +PCIe Address Translation Services (ATS) along with Page Request Interface > > +(PRI) allow devices to function much the same way as the CPU handling > > +application page-faults. For more information please refer to the PCIe > > +specification Chapter 10: ATS Specification. > > + > > +Use of SVA requires IOMMU support in the platform. IOMMU also is required > > +to support PCIe features ATS and PRI. ATS allows devices to cache > > +translations for virtual addresses. The IOMMU driver uses the > > mmu_notifier() > > +support to keep the device TLB cache and the CPU cache in sync. PRI allows > > +the device to request paging the virtual address by using the CPU page > > tables > > +before accessing the address. > > That still reads funny, the "the device to request paging the virtual > address" part. Do you mean that per chance here: > > "Before the device can access that address, the device uses the PRI in > order to request the virtual address to be paged in into the CPU page > tables." > Agree, this reads a bit funny. Just tweaked it a bit: "When ATS lookup fails for a virtual address, device should use PRI in order to request the virtual address to be paged into the CPU page tables. The device must use ATS again in order the fetch the translation again before use" Cheers, Ashok ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v8 3/9] Documentation/x86: Add documentation for SVA (Shared Virtual Addressing)
On Tue, Sep 15, 2020 at 09:30:07AM -0700, Fenghua Yu wrote: > +Background > +== > + > +Shared Virtual Addressing (SVA) allows the processor and device to use the > +same virtual addresses avoiding the need for software to translate virtual > +addresses to physical addresses. SVA is what PCIe calls Shared Virtual > +Memory (SVM). > + > +In addition to the convenience of using application virtual addresses > +by the device, it also doesn't require pinning pages for DMA. > +PCIe Address Translation Services (ATS) along with Page Request Interface > +(PRI) allow devices to function much the same way as the CPU handling > +application page-faults. For more information please refer to the PCIe > +specification Chapter 10: ATS Specification. > + > +Use of SVA requires IOMMU support in the platform. IOMMU also is required > +to support PCIe features ATS and PRI. ATS allows devices to cache > +translations for virtual addresses. The IOMMU driver uses the mmu_notifier() > +support to keep the device TLB cache and the CPU cache in sync. PRI allows > +the device to request paging the virtual address by using the CPU page tables > +before accessing the address. That still reads funny, the "the device to request paging the virtual address" part. Do you mean that per chance here: "Before the device can access that address, the device uses the PRI in order to request the virtual address to be paged in into the CPU page tables." ? -- Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
[PATCH v8 3/9] Documentation/x86: Add documentation for SVA (Shared Virtual Addressing)
From: Ashok Raj ENQCMD and Data Streaming Accelerator (DSA) and all of their associated features are a complicated stack with lots of interconnected pieces. This documentation provides a big picture overview for all of the features. Signed-off-by: Ashok Raj Co-developed-by: Fenghua Yu Signed-off-by: Fenghua Yu Reviewed-by: Tony Luck --- v8: - Address all of comments from Boris and Randy. v7: - Change the doc for updating PASID by IPI and context switch (Andy). v3: - Replace deprecated intel_svm_bind_mm() by iommu_sva_bind_mm() (Baolu) - Fix a couple of typos (Baolu) v2: - Fix the doc format and add the doc in toctree (Thomas) - Modify the doc for better description (Thomas, Tony, Dave) Documentation/x86/index.rst | 1 + Documentation/x86/sva.rst | 256 2 files changed, 257 insertions(+) create mode 100644 Documentation/x86/sva.rst diff --git a/Documentation/x86/index.rst b/Documentation/x86/index.rst index 265d9e9a093b..e5d5ff096685 100644 --- a/Documentation/x86/index.rst +++ b/Documentation/x86/index.rst @@ -30,3 +30,4 @@ x86-specific Documentation usb-legacy-support i386/index x86_64/index + sva diff --git a/Documentation/x86/sva.rst b/Documentation/x86/sva.rst new file mode 100644 index ..a1f008ef7dad --- /dev/null +++ b/Documentation/x86/sva.rst @@ -0,0 +1,256 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=== +Shared Virtual Addressing (SVA) with ENQCMD +=== + +Background +== + +Shared Virtual Addressing (SVA) allows the processor and device to use the +same virtual addresses avoiding the need for software to translate virtual +addresses to physical addresses. SVA is what PCIe calls Shared Virtual +Memory (SVM). + +In addition to the convenience of using application virtual addresses +by the device, it also doesn't require pinning pages for DMA. +PCIe Address Translation Services (ATS) along with Page Request Interface +(PRI) allow devices to function much the same way as the CPU handling +application page-faults. For more information please refer to the PCIe +specification Chapter 10: ATS Specification. + +Use of SVA requires IOMMU support in the platform. IOMMU also is required +to support PCIe features ATS and PRI. ATS allows devices to cache +translations for virtual addresses. The IOMMU driver uses the mmu_notifier() +support to keep the device TLB cache and the CPU cache in sync. PRI allows +the device to request paging the virtual address by using the CPU page tables +before accessing the address. + + +Shared Hardware Workqueues +== + +Unlike Single Root I/O Virtualization (SR-IOV), Scalable IOV (SIOV) permits +the use of Shared Work Queues (SWQ) by both applications and Virtual +Machines (VM's). This allows better hardware utilization vs. hard +partitioning resources that could result in under utilization. In order to +allow the hardware to distinguish the context for which work is being +executed in the hardware by SWQ interface, SIOV uses Process Address Space +ID (PASID), which is a 20-bit number defined by the PCIe SIG. + +PASID value is encoded in all transactions from the device. This allows the +IOMMU to track I/O on a per-PASID granularity in addition to using the PCIe +Resource Identifier (RID) which is the Bus/Device/Function. + + +ENQCMD +== + +ENQCMD is a new instruction on Intel platforms that atomically submits a +work descriptor to a device. The descriptor includes the operation to be +performed, virtual addresses of all parameters, virtual address of a completion +record, and the PASID (process address space ID) of the current process. + +ENQCMD works with non-posted semantics and carries a status back if the +command was accepted by hardware. This allows the submitter to know if the +submission needs to be retried or other device specific mechanisms to +implement fairness or ensure forward progress should be provided. + +ENQCMD is the glue that ensures applications can directly submit commands +to the hardware and also permits hardware to be aware of application context +to perform I/O operations via use of PASID. + +Process Address Space Tagging += + +A new thread-scoped MSR (IA32_PASID) provides the connection between +user processes and the rest of the hardware. When an application first +accesses an SVA-capable device this MSR is initialized with a newly +allocated PASID. The driver for the device calls an IOMMU-specific API +that sets up the routing for DMA and page-requests. + +For example, the Intel Data Streaming Accelerator (DSA) uses +iommu_sva_bind_device(), which will do the following: + +- Allocate the PASID, and program the process page-table (%cr3 register) in the + PASID context entries. +- Register for mmu_notifier() to track any page-table invalidations to keep + the device TLB in sync. For example, when a page-table entry is invalidated,
