Re: [Qemu-devel] [RFC Design Doc v3] Enable Shared Virtual Memory feature in pass-through scenarios
On Wed, Mar 01, 2017 at 04:09:38PM -0500, Konrad Rzeszutek Wilk wrote: > .snip.. > > > > No. SVM is purely about sharing CPU address space with device. Command > > submission is still through kernel driver which controls rings (with SVM > > then > > you can put VA into those commands). There are other vendor specific > > features to enable direct user space submission which is orthogonal to SVM. > > Apologies for my ignorance but how is this beneficial? As in > currently you would put in bus addresses on the ring, but now you > can put VA addresses. > > The obvious benefit I see is that you omit the DMA ops which means there is > less of 'lookup' (VA->bus address) in software - but I would have thought this > would be negligible performance impact? And now the IOMMU alongside with > the CPU would do this lookup. > > Or are there some other improvements in this? Other benefits include, - Application can simply pass its pointers to the SVM capable devices. which means no memory registration overhead to get IO Virtual Addresses. - No need to pin memory for DMA, since the devices can handle faults and can request pages to be paged-in on demand. > > > > > Thanks > > Kevin > ___ > iommu mailing list > io...@lists.linux-foundation.org > https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [Qemu-devel] [RFC Design Doc v3] Enable Shared Virtual Memory feature in pass-through scenarios
.snip.. > > > Shared Virtual Memory feature in pass-through scenarios is actually SVM > > > virtualization. It is to let application programs(running in guest)share > > > their > > > virtual address with assigned device(e.g. graphics processors or > > > accelerators). > > > > I think I am missing something obvious, but the current way that DRM > > works is that the kernel sets up its VA addresses for the GPU and it uses > > that for its ring. It also setups an user level mapping for the GPU if the > > application (Xorg) really wants it - but most of the time the kernel is > > in charge of poking at the ring, and the memory that is shared with the > > Xorg is normal RAM allocated via alloc_pages (see > > drivers/gpu/drm/ttm/ttm_page_alloc_dma.c > > and drivers/gpu/drm/ttm/ttm_page_alloc.c). > > > > So are talking about the guest applications having access to the > > ring of the GPU? > > No. SVM is purely about sharing CPU address space with device. Command > submission is still through kernel driver which controls rings (with SVM then > you can put VA into those commands). There are other vendor specific > features to enable direct user space submission which is orthogonal to SVM. Apologies for my ignorance but how is this beneficial? As in currently you would put in bus addresses on the ring, but now you can put VA addresses. The obvious benefit I see is that you omit the DMA ops which means there is less of 'lookup' (VA->bus address) in software - but I would have thought this would be negligible performance impact? And now the IOMMU alongside with the CPU would do this lookup. Or are there some other improvements in this? > > Thanks > Kevin
Re: [Qemu-devel] [RFC Design Doc v3] Enable Shared Virtual Memory feature in pass-through scenarios
> From: Konrad Rzeszutek Wilk [mailto:konrad.w...@oracle.com] > Sent: Wednesday, March 01, 2017 6:07 AM > > On Wed, Nov 30, 2016 at 08:49:24AM +, Liu, Yi L wrote: > > What's changed from v2: > > a) Detailed feature description > > b) refine description in "Address translation in virtual SVM" > > b) "Terms" is added > > > > Content > > === > > 1. Feature description > > 2. Why use it? > > 3. How to enable it > > 4. How to test > > 5. Terms > > > > Details > > === > > 1. Feature description > > Shared virtual memory(SVM) is to let application program share its virtual > > address with SVM capable devices. > > > > Shared virtual memory details: > > a) SVM feature requires ATS/PRQ/PASID support on both device side and > > IOMMU side. > > b) SVM capable devices could send DMA requests with PASID, the address > > in the request would be a virtual address within a program's virtual address > > space. > > c) IOMMU would use first level page table to translate the address in the > > request. > > d) First level page table is a HVA->HPA mapping on bare metal. > > > > Shared Virtual Memory feature in pass-through scenarios is actually SVM > > virtualization. It is to let application programs(running in guest)share > > their > > virtual address with assigned device(e.g. graphics processors or > > accelerators). > > I think I am missing something obvious, but the current way that DRM > works is that the kernel sets up its VA addresses for the GPU and it uses > that for its ring. It also setups an user level mapping for the GPU if the > application (Xorg) really wants it - but most of the time the kernel is > in charge of poking at the ring, and the memory that is shared with the > Xorg is normal RAM allocated via alloc_pages (see > drivers/gpu/drm/ttm/ttm_page_alloc_dma.c > and drivers/gpu/drm/ttm/ttm_page_alloc.c). > > So are talking about the guest applications having access to the > ring of the GPU? No. SVM is purely about sharing CPU address space with device. Command submission is still through kernel driver which controls rings (with SVM then you can put VA into those commands). There are other vendor specific features to enable direct user space submission which is orthogonal to SVM. Thanks Kevin
Re: [Qemu-devel] [RFC Design Doc v3] Enable Shared Virtual Memory feature in pass-through scenarios
On Wed, Nov 30, 2016 at 08:49:24AM +, Liu, Yi L wrote: > What's changed from v2: > a) Detailed feature description > b) refine description in "Address translation in virtual SVM" > b) "Terms" is added > > Content > === > 1. Feature description > 2. Why use it? > 3. How to enable it > 4. How to test > 5. Terms > > Details > === > 1. Feature description > Shared virtual memory(SVM) is to let application program share its virtual > address with SVM capable devices. > > Shared virtual memory details: > a) SVM feature requires ATS/PRQ/PASID support on both device side and > IOMMU side. > b) SVM capable devices could send DMA requests with PASID, the address > in the request would be a virtual address within a program's virtual address > space. > c) IOMMU would use first level page table to translate the address in the > request. > d) First level page table is a HVA->HPA mapping on bare metal. > > Shared Virtual Memory feature in pass-through scenarios is actually SVM > virtualization. It is to let application programs(running in guest)share their > virtual address with assigned device(e.g. graphics processors or > accelerators). I think I am missing something obvious, but the current way that DRM works is that the kernel sets up its VA addresses for the GPU and it uses that for its ring. It also setups an user level mapping for the GPU if the application (Xorg) really wants it - but most of the time the kernel is in charge of poking at the ring, and the memory that is shared with the Xorg is normal RAM allocated via alloc_pages (see drivers/gpu/drm/ttm/ttm_page_alloc_dma.c and drivers/gpu/drm/ttm/ttm_page_alloc.c). So are talking about the guest applications having access to the ring of the GPU?
[Qemu-devel] [RFC Design Doc v3] Enable Shared Virtual Memory feature in pass-through scenarios
What's changed from v2: a) Detailed feature description b) refine description in "Address translation in virtual SVM" b) "Terms" is added Content === 1. Feature description 2. Why use it? 3. How to enable it 4. How to test 5. Terms Details === 1. Feature description Shared virtual memory(SVM) is to let application program share its virtual address with SVM capable devices. Shared virtual memory details: a) SVM feature requires ATS/PRQ/PASID support on both device side and IOMMU side. b) SVM capable devices could send DMA requests with PASID, the address in the request would be a virtual address within a program's virtual address space. c) IOMMU would use first level page table to translate the address in the request. d) First level page table is a HVA->HPA mapping on bare metal. Shared Virtual Memory feature in pass-through scenarios is actually SVM virtualization. It is to let application programs(running in guest)share their virtual address with assigned device(e.g. graphics processors or accelerators). In virtualization, SVM would be: a) Require a vIOMMU exposed to guest b) Assigned SVM capable device could send DMA requests with PASID, the address in the request would be a virtual address within a guest program's virtual address space(GVA). c) Physical IOMMU needs to do GVA->GPA->HPA translation. Nested mode would be enabled, first level page table would achieve GVA->GPA mapping, while second level page table would achieve GPA->HPA translation. For more SVM detail, you may want refer to section 2.5.1.1 of Intel VT-d spec and section 5.6 of OpenCL spec. For details about SVM address translation, pls refer to section 3 of Intel VT-d spec. It's also welcomed to discuss directly in this thread. Link to related specs: http://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/vt-directed-io-spec.pdf https://www.khronos.org/registry/cl/specs/opencl-2.0.pdf 2. Why use it? It is common to pass-through devices to guest and expect to achieve as much similar performance as it is on host. With this feature enabled, the application programs in guest would be able to share data-structures with assigned devices without unnecessary overheads. 3. How to enable it As mentioned above, SVM virtualization requires a vIOMMU exposed to guest. Since there is an existing IOMMU emulator in host user space(QEMU), it is more acceptable to extend the IOMMU emulator to support SVM for assigned devices. So far, the vIOMMU exposed to guest is only for emulated devices. In this design, it would focus on virtual SVM for assigned devices. Virtual IOVA and virtual interrupt remapping will not be included here. The enabling work would include the following items. a) IOMMU Register Access Emulation Already existed in QEMU, need some extensions to support SVM. e.g. support page request service related registers(PQA_REG). b) vIOMMU Capability Report SVM related capabilities(PASID,PRS,DT,PT,ECS etc.) in ex-capability register and cache mode, DWD, DRD in capability register. c) QI Handling Emulation Already existed in QEMU, need to shadow the QIs related to assigned devices to physical IOMMU. i. ex-context entry cache invalidation(nested mode setting, guest PASID table pointer shadowing) ii. 1st level translation cache invalidation iii.Response for recoverable faults d) Address translation in virtual SVM In virtualization, for requests with PASID from assigned device, the address translation would be subjected to first level page table and then second level page table, which is named nested mode. Extended context mode should be supported on hardware. DMA remapping in SVM virtualization would be: i. For requests with PASID, the related extended context entry should have the NESTE bit set. ii. Guest PASID table pointer should be shadowed to host IOMMU driver. The PASID table pointer field in extended context entry would be a GPA as nested mode is on. First level page table would be maintained by guest IOMMU driver. Second level page table would be maintained by host IOMMU driver. e) Recoverable Address Translation Faults Handling Emulation It is serviced by page request when device support PRS. For assigned devices, host IOMMU driver would get page requests from pIOMMU. Here, we need a mechanism to drain the page requests from devices which are assigned to a guest. In this design it would be done through VFIO. Page request descriptors would be propagated to user space and then exposed to guest IOMMU driver. This requires following support: i. a mechanism to notify vIOMMU emulator to fetch PRQ descriptor ii. a notify framework in QEMU to signal the PRQ descriptor fetching when notified by pIOMMU f) Non-Recoverable Address Translation Handling Emulation The non-recoverable fault propagation is similar to recoverable faults. In this design it would propagate fault data to user
