RE: [PATCH v11 00/13] SMMUv3 Nested Stage Setup (IOMMU part)
Hi Eric, First, many thanks for the respin. I will go through all of these(iommu/vfio/Qemu) and will do a thorough verification/tests on our hardware. > -Original Message- > From: Auger Eric [mailto:eric.au...@redhat.com] > Sent: 17 November 2020 08:40 > To: Shameerali Kolothum Thodi ; > Zhangfei Gao ; eric.auger@gmail.com; > io...@lists.linux-foundation.org; linux-ker...@vger.kernel.org; > k...@vger.kernel.org; kvmarm@lists.cs.columbia.edu; w...@kernel.org; > j...@8bytes.org; m...@kernel.org; robin.mur...@arm.com > Cc: jean-phili...@linaro.org; alex.william...@redhat.com; > jacob.jun@linux.intel.com; yi.l@intel.com; peter.mayd...@linaro.org; > t...@semihalf.com; bbhush...@marvell.com > Subject: Re: [PATCH v11 00/13] SMMUv3 Nested Stage Setup (IOMMU part) > > Hi Shameer, > > On 5/13/20 5:57 PM, Shameerali Kolothum Thodi wrote: > > Hi Eric, > > > >> -Original Message- > >> From: Auger Eric [mailto:eric.au...@redhat.com] > >> Sent: 13 May 2020 14:29 > >> To: Shameerali Kolothum Thodi ; > >> Zhangfei Gao ; eric.auger@gmail.com; > >> io...@lists.linux-foundation.org; linux-ker...@vger.kernel.org; > >> k...@vger.kernel.org; kvmarm@lists.cs.columbia.edu; w...@kernel.org; > >> j...@8bytes.org; m...@kernel.org; robin.mur...@arm.com > >> Cc: jean-phili...@linaro.org; alex.william...@redhat.com; > >> jacob.jun....@linux.intel.com; yi.l....@intel.com; > peter.mayd...@linaro.org; > >> t...@semihalf.com; bbhush...@marvell.com > >> Subject: Re: [PATCH v11 00/13] SMMUv3 Nested Stage Setup (IOMMU part) > >> > > [...] > > > >>>>> Yes that's normal this series is not meant to support vSVM at this > >>>>> stage. > >>>>> > >>>>> I intend to add the missing pieces during the next weeks. > >>>> > >>>> Thanks for that. I have made an attempt to add the vSVA based on > >>>> your v10 + JPBs sva patches. The host kernel and Qemu changes can > >>>> be found here[1][2]. > >>>> > >>>> This basically adds multiple pasid support on top of your changes. > >>>> I have done some basic sanity testing and we have some initial success > >>>> with the zip vf dev on our D06 platform. Please note that the STALL event > is > >>>> not yet supported though, but works fine if we mlock() guest usr mem. > >>> > >>> I have added STALL support for our vSVA prototype and it seems to be > >>> working(on our hardware). I have updated the kernel and qemu branches > >> with > >>> the same[1][2]. I should warn you though that these are prototype code > and I > >> am pretty > >>> much re-using the VFIO_IOMMU_SET_PASID_TABLE interface for almost > >> everything. > >>> But thought of sharing, in case if it is useful somehow!. > >> > >> Thank you again for sharing the POC. I looked at the kernel and QEMU > >> branches. > >> > >> Here are some preliminary comments: > >> - "arm-smmu-v3: Reset S2TTB while switching back from nested stage": > as > >> you mentionned S2TTB reset now is featured in v11 > > > > Yes. > > > >> - "arm-smmu-v3: Add support for multiple pasid in nested mode": I could > >> easily integrate this into my series. Update the iommu api first and > >> pass multiple CD info in a separate patch > > > > Ok. > in v12, I added > [PATCH v12 14/15] iommu/smmuv3: Accept configs with more than one > context descriptor > > I don't think you need to add s1cdmax addition as we already have > pasid_bits which should do the job. Ok. > >> - "arm-smmu-v3: Add support to Invalidate CD": CD invalidation should be > >> cascaded to host through the PASID cache invalidation uapi (no pb you > >> warned us for the POC you simply used VFIO_IOMMU_SET_PASID_TABLE). I > >> think I should add this support in my original series although it does > >> not seem to trigger any issue up to now. > > > > Agree. Cache invalidation uapi is a better interface for this. Also I don’t > > think > > this matters for non-vsva cases as Guest kernel table/CD(pasid 0) will never > > get invalidated. > in v12 I added [PATCH v12 15/15] iommu/smmuv3: Add PASID cache > invalidation per PASID. I have not tested it though. Ok. Will verify this. > >> - "arm-smmu-v3: Remove duplication of fault propagation". I understand > >> the transcode is done somewhere else with SVA but we
Re: [PATCH v11 00/13] SMMUv3 Nested Stage Setup (IOMMU part)
Hi Shameer, On 5/13/20 5:57 PM, Shameerali Kolothum Thodi wrote: > Hi Eric, > >> -Original Message- >> From: Auger Eric [mailto:eric.au...@redhat.com] >> Sent: 13 May 2020 14:29 >> To: Shameerali Kolothum Thodi ; >> Zhangfei Gao ; eric.auger@gmail.com; >> io...@lists.linux-foundation.org; linux-ker...@vger.kernel.org; >> k...@vger.kernel.org; kvmarm@lists.cs.columbia.edu; w...@kernel.org; >> j...@8bytes.org; m...@kernel.org; robin.mur...@arm.com >> Cc: jean-phili...@linaro.org; alex.william...@redhat.com; >> jacob.jun@linux.intel.com; yi.l@intel.com; peter.mayd...@linaro.org; >> t...@semihalf.com; bbhush...@marvell.com >> Subject: Re: [PATCH v11 00/13] SMMUv3 Nested Stage Setup (IOMMU part) >> > [...] > >>>>> Yes that's normal this series is not meant to support vSVM at this stage. >>>>> >>>>> I intend to add the missing pieces during the next weeks. >>>> >>>> Thanks for that. I have made an attempt to add the vSVA based on >>>> your v10 + JPBs sva patches. The host kernel and Qemu changes can >>>> be found here[1][2]. >>>> >>>> This basically adds multiple pasid support on top of your changes. >>>> I have done some basic sanity testing and we have some initial success >>>> with the zip vf dev on our D06 platform. Please note that the STALL event >>>> is >>>> not yet supported though, but works fine if we mlock() guest usr mem. >>> >>> I have added STALL support for our vSVA prototype and it seems to be >>> working(on our hardware). I have updated the kernel and qemu branches >> with >>> the same[1][2]. I should warn you though that these are prototype code and I >> am pretty >>> much re-using the VFIO_IOMMU_SET_PASID_TABLE interface for almost >> everything. >>> But thought of sharing, in case if it is useful somehow!. >> >> Thank you again for sharing the POC. I looked at the kernel and QEMU >> branches. >> >> Here are some preliminary comments: >> - "arm-smmu-v3: Reset S2TTB while switching back from nested stage": as >> you mentionned S2TTB reset now is featured in v11 > > Yes. > >> - "arm-smmu-v3: Add support for multiple pasid in nested mode": I could >> easily integrate this into my series. Update the iommu api first and >> pass multiple CD info in a separate patch > > Ok. in v12, I added [PATCH v12 14/15] iommu/smmuv3: Accept configs with more than one context descriptor I don't think you need to add s1cdmax addition as we already have pasid_bits which should do the job. >> - "arm-smmu-v3: Add support to Invalidate CD": CD invalidation should be >> cascaded to host through the PASID cache invalidation uapi (no pb you >> warned us for the POC you simply used VFIO_IOMMU_SET_PASID_TABLE). I >> think I should add this support in my original series although it does >> not seem to trigger any issue up to now. > > Agree. Cache invalidation uapi is a better interface for this. Also I don’t > think > this matters for non-vsva cases as Guest kernel table/CD(pasid 0) will never > get invalidated. in v12 I added [PATCH v12 15/15] iommu/smmuv3: Add PASID cache invalidation per PASID. I have not tested it though. > >> - "arm-smmu-v3: Remove duplication of fault propagation". I understand >> the transcode is done somewhere else with SVA but we still need to do it >> if a single CD is used, right? I will review the SVA code to better >> understand. Since I have rebase on 5.10-rc4 you will still have this duplication to handle. > > Hmm..not sure. Need to take another look to see whether we need a special > handling for single CD or not. > >> - for the STALL response injection I would tend to use a new VFIO region >> for responses. At the moment there is a single VFIO region for reporting >> the fault. in v12 I added a new VFIO region to inject your fault. This was tested with dummy event injection, this should work properly. If we clearly identify all the public dependencies needed for vSVA/ARM I can help you respinning on top of them Thanks Eric > > Sure. That will be much cleaner and probably improve the context switch > latency. Another thing I noted with STALL is that pasid_valid flag needs to be > taken care in the SVA kernel path. > > "iommu: Remove pasid validity check for STALL model page response msg" > Not sure this one is a proper way to handle this. > >> On QEMU side: >> - I am currently working on 3.2 range invalidation support which is >> needed for DPDK/VFIO >> - While at it I will look at how to incrementally introduce some of the >> features you need in this series. > > Ok. > > Thanks for taking a look at the POC. > > Cheers, > Shameer > ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
RE: [PATCH v11 00/13] SMMUv3 Nested Stage Setup (IOMMU part)
Hi Eric, > -Original Message- > From: Auger Eric [mailto:eric.au...@redhat.com] > Sent: 13 May 2020 14:29 > To: Shameerali Kolothum Thodi ; > Zhangfei Gao ; eric.auger@gmail.com; > io...@lists.linux-foundation.org; linux-ker...@vger.kernel.org; > k...@vger.kernel.org; kvmarm@lists.cs.columbia.edu; w...@kernel.org; > j...@8bytes.org; m...@kernel.org; robin.mur...@arm.com > Cc: jean-phili...@linaro.org; alex.william...@redhat.com; > jacob.jun@linux.intel.com; yi.l@intel.com; peter.mayd...@linaro.org; > t...@semihalf.com; bbhush...@marvell.com > Subject: Re: [PATCH v11 00/13] SMMUv3 Nested Stage Setup (IOMMU part) > [...] > >>> Yes that's normal this series is not meant to support vSVM at this stage. > >>> > >>> I intend to add the missing pieces during the next weeks. > >> > >> Thanks for that. I have made an attempt to add the vSVA based on > >> your v10 + JPBs sva patches. The host kernel and Qemu changes can > >> be found here[1][2]. > >> > >> This basically adds multiple pasid support on top of your changes. > >> I have done some basic sanity testing and we have some initial success > >> with the zip vf dev on our D06 platform. Please note that the STALL event > >> is > >> not yet supported though, but works fine if we mlock() guest usr mem. > > > > I have added STALL support for our vSVA prototype and it seems to be > > working(on our hardware). I have updated the kernel and qemu branches > with > > the same[1][2]. I should warn you though that these are prototype code and I > am pretty > > much re-using the VFIO_IOMMU_SET_PASID_TABLE interface for almost > everything. > > But thought of sharing, in case if it is useful somehow!. > > Thank you again for sharing the POC. I looked at the kernel and QEMU > branches. > > Here are some preliminary comments: > - "arm-smmu-v3: Reset S2TTB while switching back from nested stage": as > you mentionned S2TTB reset now is featured in v11 Yes. > - "arm-smmu-v3: Add support for multiple pasid in nested mode": I could > easily integrate this into my series. Update the iommu api first and > pass multiple CD info in a separate patch Ok. > - "arm-smmu-v3: Add support to Invalidate CD": CD invalidation should be > cascaded to host through the PASID cache invalidation uapi (no pb you > warned us for the POC you simply used VFIO_IOMMU_SET_PASID_TABLE). I > think I should add this support in my original series although it does > not seem to trigger any issue up to now. Agree. Cache invalidation uapi is a better interface for this. Also I don’t think this matters for non-vsva cases as Guest kernel table/CD(pasid 0) will never get invalidated. > - "arm-smmu-v3: Remove duplication of fault propagation". I understand > the transcode is done somewhere else with SVA but we still need to do it > if a single CD is used, right? I will review the SVA code to better > understand. Hmm..not sure. Need to take another look to see whether we need a special handling for single CD or not. > - for the STALL response injection I would tend to use a new VFIO region > for responses. At the moment there is a single VFIO region for reporting > the fault. Sure. That will be much cleaner and probably improve the context switch latency. Another thing I noted with STALL is that pasid_valid flag needs to be taken care in the SVA kernel path. "iommu: Remove pasid validity check for STALL model page response msg" Not sure this one is a proper way to handle this. > On QEMU side: > - I am currently working on 3.2 range invalidation support which is > needed for DPDK/VFIO > - While at it I will look at how to incrementally introduce some of the > features you need in this series. Ok. Thanks for taking a look at the POC. Cheers, Shameer ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v11 00/13] SMMUv3 Nested Stage Setup (IOMMU part)
Hi Shameer, On 5/7/20 8:59 AM, Shameerali Kolothum Thodi wrote: > Hi Eric, > >> -Original Message- >> From: Shameerali Kolothum Thodi >> Sent: 30 April 2020 10:38 >> To: 'Auger Eric' ; Zhangfei Gao >> ; eric.auger@gmail.com; >> io...@lists.linux-foundation.org; linux-ker...@vger.kernel.org; >> k...@vger.kernel.org; kvmarm@lists.cs.columbia.edu; w...@kernel.org; >> j...@8bytes.org; m...@kernel.org; robin.mur...@arm.com >> Cc: jean-phili...@linaro.org; alex.william...@redhat.com; >> jacob.jun@linux.intel.com; yi.l@intel.com; peter.mayd...@linaro.org; >> t...@semihalf.com; bbhush...@marvell.com >> Subject: RE: [PATCH v11 00/13] SMMUv3 Nested Stage Setup (IOMMU part) >> >> Hi Eric, >> >>> -Original Message- >>> From: Auger Eric [mailto:eric.au...@redhat.com] >>> Sent: 16 April 2020 08:45 >>> To: Zhangfei Gao ; eric.auger@gmail.com; >>> io...@lists.linux-foundation.org; linux-ker...@vger.kernel.org; >>> k...@vger.kernel.org; kvmarm@lists.cs.columbia.edu; w...@kernel.org; >>> j...@8bytes.org; m...@kernel.org; robin.mur...@arm.com >>> Cc: jean-phili...@linaro.org; Shameerali Kolothum Thodi >>> ; alex.william...@redhat.com; >>> jacob.jun@linux.intel.com; yi.l@intel.com; peter.mayd...@linaro.org; >>> t...@semihalf.com; bbhush...@marvell.com >>> Subject: Re: [PATCH v11 00/13] SMMUv3 Nested Stage Setup (IOMMU part) >>> >>> Hi Zhangfei, >>> >>> On 4/16/20 6:25 AM, Zhangfei Gao wrote: >>>> >>>> >>>> On 2020/4/14 下午11:05, Eric Auger wrote: >>>>> This version fixes an issue observed by Shameer on an SMMU 3.2, >>>>> when moving from dual stage config to stage 1 only config. >>>>> The 2 high 64b of the STE now get reset. Otherwise, leaving the >>>>> S2TTB set may cause a C_BAD_STE error. >>>>> >>>>> This series can be found at: >>>>> https://github.com/eauger/linux/tree/v5.6-2stage-v11_10.1 >>>>> (including the VFIO part) >>>>> The QEMU fellow series still can be found at: >>>>> https://github.com/eauger/qemu/tree/v4.2.0-2stage-rfcv6 >>>>> >>>>> Users have expressed interest in that work and tested v9/v10: >>>>> - https://patchwork.kernel.org/cover/11039995/#23012381 >>>>> - https://patchwork.kernel.org/cover/11039995/#23197235 >>>>> >>>>> Background: >>>>> >>>>> This series brings the IOMMU part of HW nested paging support >>>>> in the SMMUv3. The VFIO part is submitted separately. >>>>> >>>>> The IOMMU API is extended to support 2 new API functionalities: >>>>> 1) pass the guest stage 1 configuration >>>>> 2) pass stage 1 MSI bindings >>>>> >>>>> Then those capabilities gets implemented in the SMMUv3 driver. >>>>> >>>>> The virtualizer passes information through the VFIO user API >>>>> which cascades them to the iommu subsystem. This allows the guest >>>>> to own stage 1 tables and context descriptors (so-called PASID >>>>> table) while the host owns stage 2 tables and main configuration >>>>> structures (STE). >>>>> >>>>> >>>> >>>> Thanks Eric >>>> >>>> Tested v11 on Hisilicon kunpeng920 board via hardware zip accelerator. >>>> 1. no-sva works, where guest app directly use physical address via ioctl. >>> Thank you for the testing. Glad it works for you. >>>> 2. vSVA still not work, same as v10, >>> Yes that's normal this series is not meant to support vSVM at this stage. >>> >>> I intend to add the missing pieces during the next weeks. >> >> Thanks for that. I have made an attempt to add the vSVA based on >> your v10 + JPBs sva patches. The host kernel and Qemu changes can >> be found here[1][2]. >> >> This basically adds multiple pasid support on top of your changes. >> I have done some basic sanity testing and we have some initial success >> with the zip vf dev on our D06 platform. Please note that the STALL event is >> not yet supported though, but works fine if we mlock() guest usr mem. > > I have added STALL support for our vSVA prototype and it seems to be > working(on our hardware). I have updated the kernel and qemu branches with > the same[1][2]. I should warn you though that these are prototype code and I > am pretty > much
Re: [PATCH v11 00/13] SMMUv3 Nested Stage Setup (IOMMU part)
Hi Shameer, On 5/7/20 8:59 AM, Shameerali Kolothum Thodi wrote: > Hi Eric, > >> -Original Message- >> From: Shameerali Kolothum Thodi >> Sent: 30 April 2020 10:38 >> To: 'Auger Eric' ; Zhangfei Gao >> ; eric.auger@gmail.com; >> io...@lists.linux-foundation.org; linux-ker...@vger.kernel.org; >> k...@vger.kernel.org; kvmarm@lists.cs.columbia.edu; w...@kernel.org; >> j...@8bytes.org; m...@kernel.org; robin.mur...@arm.com >> Cc: jean-phili...@linaro.org; alex.william...@redhat.com; >> jacob.jun@linux.intel.com; yi.l@intel.com; peter.mayd...@linaro.org; >> t...@semihalf.com; bbhush...@marvell.com >> Subject: RE: [PATCH v11 00/13] SMMUv3 Nested Stage Setup (IOMMU part) >> >> Hi Eric, >> >>> -Original Message- >>> From: Auger Eric [mailto:eric.au...@redhat.com] >>> Sent: 16 April 2020 08:45 >>> To: Zhangfei Gao ; eric.auger@gmail.com; >>> io...@lists.linux-foundation.org; linux-ker...@vger.kernel.org; >>> k...@vger.kernel.org; kvmarm@lists.cs.columbia.edu; w...@kernel.org; >>> j...@8bytes.org; m...@kernel.org; robin.mur...@arm.com >>> Cc: jean-phili...@linaro.org; Shameerali Kolothum Thodi >>> ; alex.william...@redhat.com; >>> jacob.jun@linux.intel.com; yi.l@intel.com; peter.mayd...@linaro.org; >>> t...@semihalf.com; bbhush...@marvell.com >>> Subject: Re: [PATCH v11 00/13] SMMUv3 Nested Stage Setup (IOMMU part) >>> >>> Hi Zhangfei, >>> >>> On 4/16/20 6:25 AM, Zhangfei Gao wrote: >>>> >>>> >>>> On 2020/4/14 下午11:05, Eric Auger wrote: >>>>> This version fixes an issue observed by Shameer on an SMMU 3.2, >>>>> when moving from dual stage config to stage 1 only config. >>>>> The 2 high 64b of the STE now get reset. Otherwise, leaving the >>>>> S2TTB set may cause a C_BAD_STE error. >>>>> >>>>> This series can be found at: >>>>> https://github.com/eauger/linux/tree/v5.6-2stage-v11_10.1 >>>>> (including the VFIO part) >>>>> The QEMU fellow series still can be found at: >>>>> https://github.com/eauger/qemu/tree/v4.2.0-2stage-rfcv6 >>>>> >>>>> Users have expressed interest in that work and tested v9/v10: >>>>> - https://patchwork.kernel.org/cover/11039995/#23012381 >>>>> - https://patchwork.kernel.org/cover/11039995/#23197235 >>>>> >>>>> Background: >>>>> >>>>> This series brings the IOMMU part of HW nested paging support >>>>> in the SMMUv3. The VFIO part is submitted separately. >>>>> >>>>> The IOMMU API is extended to support 2 new API functionalities: >>>>> 1) pass the guest stage 1 configuration >>>>> 2) pass stage 1 MSI bindings >>>>> >>>>> Then those capabilities gets implemented in the SMMUv3 driver. >>>>> >>>>> The virtualizer passes information through the VFIO user API >>>>> which cascades them to the iommu subsystem. This allows the guest >>>>> to own stage 1 tables and context descriptors (so-called PASID >>>>> table) while the host owns stage 2 tables and main configuration >>>>> structures (STE). >>>>> >>>>> >>>> >>>> Thanks Eric >>>> >>>> Tested v11 on Hisilicon kunpeng920 board via hardware zip accelerator. >>>> 1. no-sva works, where guest app directly use physical address via ioctl. >>> Thank you for the testing. Glad it works for you. >>>> 2. vSVA still not work, same as v10, >>> Yes that's normal this series is not meant to support vSVM at this stage. >>> >>> I intend to add the missing pieces during the next weeks. >> >> Thanks for that. I have made an attempt to add the vSVA based on >> your v10 + JPBs sva patches. The host kernel and Qemu changes can >> be found here[1][2]. >> >> This basically adds multiple pasid support on top of your changes. >> I have done some basic sanity testing and we have some initial success >> with the zip vf dev on our D06 platform. Please note that the STALL event is >> not yet supported though, but works fine if we mlock() guest usr mem. > > I have added STALL support for our vSVA prototype and it seems to be > working(on our hardware). I have updated the kernel and qemu branches with > the same[1][2]. I should warn you though that these are prototype code and I > am pretty > much re-using the VFIO_IOMMU_SET_PASID_TABLE interface for almost everything. > But thought of sharing, in case if it is useful somehow!. Thank you very much for your work. I intend to look at your additions by beginning of next week. Best Regards Eric > > Thanks, > Shameer > > [1]https://github.com/hisilicon/kernel-dev/commits/vsva-prototype-host-v1 > > [2]https://github.com/hisilicon/qemu/tree/v4.2.0-2stage-rfcv6-vsva-prototype-v1 > ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
RE: [PATCH v11 00/13] SMMUv3 Nested Stage Setup (IOMMU part)
Hi Eric, > -Original Message- > From: Shameerali Kolothum Thodi > Sent: 30 April 2020 10:38 > To: 'Auger Eric' ; Zhangfei Gao > ; eric.auger@gmail.com; > io...@lists.linux-foundation.org; linux-ker...@vger.kernel.org; > k...@vger.kernel.org; kvmarm@lists.cs.columbia.edu; w...@kernel.org; > j...@8bytes.org; m...@kernel.org; robin.mur...@arm.com > Cc: jean-phili...@linaro.org; alex.william...@redhat.com; > jacob.jun@linux.intel.com; yi.l@intel.com; peter.mayd...@linaro.org; > t...@semihalf.com; bbhush...@marvell.com > Subject: RE: [PATCH v11 00/13] SMMUv3 Nested Stage Setup (IOMMU part) > > Hi Eric, > > > -Original Message- > > From: Auger Eric [mailto:eric.au...@redhat.com] > > Sent: 16 April 2020 08:45 > > To: Zhangfei Gao ; eric.auger@gmail.com; > > io...@lists.linux-foundation.org; linux-ker...@vger.kernel.org; > > k...@vger.kernel.org; kvmarm@lists.cs.columbia.edu; w...@kernel.org; > > j...@8bytes.org; m...@kernel.org; robin.mur...@arm.com > > Cc: jean-phili...@linaro.org; Shameerali Kolothum Thodi > > ; alex.william...@redhat.com; > > jacob.jun@linux.intel.com; yi.l@intel.com; peter.mayd...@linaro.org; > > t...@semihalf.com; bbhush...@marvell.com > > Subject: Re: [PATCH v11 00/13] SMMUv3 Nested Stage Setup (IOMMU part) > > > > Hi Zhangfei, > > > > On 4/16/20 6:25 AM, Zhangfei Gao wrote: > > > > > > > > > On 2020/4/14 下午11:05, Eric Auger wrote: > > >> This version fixes an issue observed by Shameer on an SMMU 3.2, > > >> when moving from dual stage config to stage 1 only config. > > >> The 2 high 64b of the STE now get reset. Otherwise, leaving the > > >> S2TTB set may cause a C_BAD_STE error. > > >> > > >> This series can be found at: > > >> https://github.com/eauger/linux/tree/v5.6-2stage-v11_10.1 > > >> (including the VFIO part) > > >> The QEMU fellow series still can be found at: > > >> https://github.com/eauger/qemu/tree/v4.2.0-2stage-rfcv6 > > >> > > >> Users have expressed interest in that work and tested v9/v10: > > >> - https://patchwork.kernel.org/cover/11039995/#23012381 > > >> - https://patchwork.kernel.org/cover/11039995/#23197235 > > >> > > >> Background: > > >> > > >> This series brings the IOMMU part of HW nested paging support > > >> in the SMMUv3. The VFIO part is submitted separately. > > >> > > >> The IOMMU API is extended to support 2 new API functionalities: > > >> 1) pass the guest stage 1 configuration > > >> 2) pass stage 1 MSI bindings > > >> > > >> Then those capabilities gets implemented in the SMMUv3 driver. > > >> > > >> The virtualizer passes information through the VFIO user API > > >> which cascades them to the iommu subsystem. This allows the guest > > >> to own stage 1 tables and context descriptors (so-called PASID > > >> table) while the host owns stage 2 tables and main configuration > > >> structures (STE). > > >> > > >> > > > > > > Thanks Eric > > > > > > Tested v11 on Hisilicon kunpeng920 board via hardware zip accelerator. > > > 1. no-sva works, where guest app directly use physical address via ioctl. > > Thank you for the testing. Glad it works for you. > > > 2. vSVA still not work, same as v10, > > Yes that's normal this series is not meant to support vSVM at this stage. > > > > I intend to add the missing pieces during the next weeks. > > Thanks for that. I have made an attempt to add the vSVA based on > your v10 + JPBs sva patches. The host kernel and Qemu changes can > be found here[1][2]. > > This basically adds multiple pasid support on top of your changes. > I have done some basic sanity testing and we have some initial success > with the zip vf dev on our D06 platform. Please note that the STALL event is > not yet supported though, but works fine if we mlock() guest usr mem. I have added STALL support for our vSVA prototype and it seems to be working(on our hardware). I have updated the kernel and qemu branches with the same[1][2]. I should warn you though that these are prototype code and I am pretty much re-using the VFIO_IOMMU_SET_PASID_TABLE interface for almost everything. But thought of sharing, in case if it is useful somehow!. Thanks, Shameer [1]https://github.com/hisilicon/kernel-dev/commits/vsva-prototype-host-v1 [2]https://github.com/hisilicon/qemu/tree/v4.2.0-2stage-rfcv6-vsva-prototype-v1 ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
RE: [PATCH v11 00/13] SMMUv3 Nested Stage Setup (IOMMU part)
Hi Eric, > -Original Message- > From: Auger Eric [mailto:eric.au...@redhat.com] > Sent: 16 April 2020 08:45 > To: Zhangfei Gao ; eric.auger@gmail.com; > io...@lists.linux-foundation.org; linux-ker...@vger.kernel.org; > k...@vger.kernel.org; kvmarm@lists.cs.columbia.edu; w...@kernel.org; > j...@8bytes.org; m...@kernel.org; robin.mur...@arm.com > Cc: jean-phili...@linaro.org; Shameerali Kolothum Thodi > ; alex.william...@redhat.com; > jacob.jun@linux.intel.com; yi.l@intel.com; peter.mayd...@linaro.org; > t...@semihalf.com; bbhush...@marvell.com > Subject: Re: [PATCH v11 00/13] SMMUv3 Nested Stage Setup (IOMMU part) > > Hi Zhangfei, > > On 4/16/20 6:25 AM, Zhangfei Gao wrote: > > > > > > On 2020/4/14 下午11:05, Eric Auger wrote: > >> This version fixes an issue observed by Shameer on an SMMU 3.2, > >> when moving from dual stage config to stage 1 only config. > >> The 2 high 64b of the STE now get reset. Otherwise, leaving the > >> S2TTB set may cause a C_BAD_STE error. > >> > >> This series can be found at: > >> https://github.com/eauger/linux/tree/v5.6-2stage-v11_10.1 > >> (including the VFIO part) > >> The QEMU fellow series still can be found at: > >> https://github.com/eauger/qemu/tree/v4.2.0-2stage-rfcv6 > >> > >> Users have expressed interest in that work and tested v9/v10: > >> - https://patchwork.kernel.org/cover/11039995/#23012381 > >> - https://patchwork.kernel.org/cover/11039995/#23197235 > >> > >> Background: > >> > >> This series brings the IOMMU part of HW nested paging support > >> in the SMMUv3. The VFIO part is submitted separately. > >> > >> The IOMMU API is extended to support 2 new API functionalities: > >> 1) pass the guest stage 1 configuration > >> 2) pass stage 1 MSI bindings > >> > >> Then those capabilities gets implemented in the SMMUv3 driver. > >> > >> The virtualizer passes information through the VFIO user API > >> which cascades them to the iommu subsystem. This allows the guest > >> to own stage 1 tables and context descriptors (so-called PASID > >> table) while the host owns stage 2 tables and main configuration > >> structures (STE). > >> > >> > > > > Thanks Eric > > > > Tested v11 on Hisilicon kunpeng920 board via hardware zip accelerator. > > 1. no-sva works, where guest app directly use physical address via ioctl. > Thank you for the testing. Glad it works for you. > > 2. vSVA still not work, same as v10, > Yes that's normal this series is not meant to support vSVM at this stage. > > I intend to add the missing pieces during the next weeks. Thanks for that. I have made an attempt to add the vSVA based on your v10 + JPBs sva patches. The host kernel and Qemu changes can be found here[1][2]. This basically adds multiple pasid support on top of your changes. I have done some basic sanity testing and we have some initial success with the zip vf dev on our D06 platform. Please note that the STALL event is not yet supported though, but works fine if we mlock() guest usr mem. I also noted that Intel patches for vSVA has couple of changes in the vfio interfaces and hope there will be a convergence soon. Please let me know your plans of a respin of this series and see whether incorporating the changes for multiple pasid make sense or not for now. Thanks, Shameer [1]https://github.com/hisilicon/qemu/tree/v4.2.0-2stage-rfcv6-vsva-prototype-v1 [2]https://github.com/hisilicon/kernel-dev/tree/vsva-prototype-host-v1 > Thanks > > Eric > > 3. the v10 issue reported by Shameer has been solved, first start qemu > > with iommu=smmuv3, then start qemu without iommu=smmuv3 > > 4. no-sva also works without iommu=smmuv3 > > > > Test details in https://docs.qq.com/doc/DRU5oR1NtUERseFNL > > > > Thanks > > ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v11 00/13] SMMUv3 Nested Stage Setup (IOMMU part)
On 2020/4/14 下午11:05, Eric Auger wrote: This version fixes an issue observed by Shameer on an SMMU 3.2, when moving from dual stage config to stage 1 only config. The 2 high 64b of the STE now get reset. Otherwise, leaving the S2TTB set may cause a C_BAD_STE error. This series can be found at: https://github.com/eauger/linux/tree/v5.6-2stage-v11_10.1 (including the VFIO part) The QEMU fellow series still can be found at: https://github.com/eauger/qemu/tree/v4.2.0-2stage-rfcv6 Users have expressed interest in that work and tested v9/v10: - https://patchwork.kernel.org/cover/11039995/#23012381 - https://patchwork.kernel.org/cover/11039995/#23197235 Background: This series brings the IOMMU part of HW nested paging support in the SMMUv3. The VFIO part is submitted separately. The IOMMU API is extended to support 2 new API functionalities: 1) pass the guest stage 1 configuration 2) pass stage 1 MSI bindings Then those capabilities gets implemented in the SMMUv3 driver. The virtualizer passes information through the VFIO user API which cascades them to the iommu subsystem. This allows the guest to own stage 1 tables and context descriptors (so-called PASID table) while the host owns stage 2 tables and main configuration structures (STE). Thanks Eric Tested v11 on Hisilicon kunpeng920 board via hardware zip accelerator. 1. no-sva works, where guest app directly use physical address via ioctl. 2. vSVA still not work, same as v10, 3. the v10 issue reported by Shameer has been solved, first start qemu with iommu=smmuv3, then start qemu without iommu=smmuv3 4. no-sva also works without iommu=smmuv3 Test details in https://docs.qq.com/doc/DRU5oR1NtUERseFNL Thanks ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v11 00/13] SMMUv3 Nested Stage Setup (IOMMU part)
Hi Zhangfei, On 4/16/20 6:25 AM, Zhangfei Gao wrote: > > > On 2020/4/14 下午11:05, Eric Auger wrote: >> This version fixes an issue observed by Shameer on an SMMU 3.2, >> when moving from dual stage config to stage 1 only config. >> The 2 high 64b of the STE now get reset. Otherwise, leaving the >> S2TTB set may cause a C_BAD_STE error. >> >> This series can be found at: >> https://github.com/eauger/linux/tree/v5.6-2stage-v11_10.1 >> (including the VFIO part) >> The QEMU fellow series still can be found at: >> https://github.com/eauger/qemu/tree/v4.2.0-2stage-rfcv6 >> >> Users have expressed interest in that work and tested v9/v10: >> - https://patchwork.kernel.org/cover/11039995/#23012381 >> - https://patchwork.kernel.org/cover/11039995/#23197235 >> >> Background: >> >> This series brings the IOMMU part of HW nested paging support >> in the SMMUv3. The VFIO part is submitted separately. >> >> The IOMMU API is extended to support 2 new API functionalities: >> 1) pass the guest stage 1 configuration >> 2) pass stage 1 MSI bindings >> >> Then those capabilities gets implemented in the SMMUv3 driver. >> >> The virtualizer passes information through the VFIO user API >> which cascades them to the iommu subsystem. This allows the guest >> to own stage 1 tables and context descriptors (so-called PASID >> table) while the host owns stage 2 tables and main configuration >> structures (STE). >> >> > > Thanks Eric > > Tested v11 on Hisilicon kunpeng920 board via hardware zip accelerator. > 1. no-sva works, where guest app directly use physical address via ioctl. Thank you for the testing. Glad it works for you. > 2. vSVA still not work, same as v10, Yes that's normal this series is not meant to support vSVM at this stage. I intend to add the missing pieces during the next weeks. Thanks Eric > 3. the v10 issue reported by Shameer has been solved, first start qemu > with iommu=smmuv3, then start qemu without iommu=smmuv3 > 4. no-sva also works without iommu=smmuv3 > > Test details in https://docs.qq.com/doc/DRU5oR1NtUERseFNL > > Thanks > ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
