Re: Re: [PATCH v9 00/11] SMMUv3 Nested Stage Setup (VFIO part)
Hi, Eric On 2019/11/20 下午6:18, Auger Eric wrote: This series brings the VFIO part of HW nested paging support in the SMMUv3. The series depends on: [PATCH v9 00/14] SMMUv3 Nested Stage Setup (IOMMU part) (https://www.spinics.net/lists/kernel/msg3187714.html) 3 new IOCTLs are introduced that allow the userspace to 1) pass the guest stage 1 configuration 2) pass stage 1 MSI bindings 3) invalidate stage 1 related caches They map onto the related new IOMMU API functions. We introduce the capability to register specific interrupt indexes (see [1]). A new DMA_FAULT interrupt index allows to register an eventfd to be signaled whenever a stage 1 related fault is detected at physical level. Also a specific region allows to expose the fault records to the user space. Best Regards Eric This series can be found at: https://github.com/eauger/linux/tree/v5.3.0-rc0-2stage-v9 I think you have already tested on ThunderX2, but as a formality, for the whole series: Tested-by: Tomasz Nowicki qemu: https://github.com/eauger/qemu/tree/v4.1.0-rc0-2stage-rfcv5 kernel: https://github.com/eauger/linux/tree/v5.3.0-rc0-2stage-v9 + Shameer's fix patch In my test I assigned Intel 82574L NIC and perform iperf tests. Thank you for your testing efforts. Other folks from Marvell claimed this to be important feature so I asked them to review and speak up on mailing list. That's nice to read that! So it is time for me to rebase both the iommu and vfio parts. I will submit something quickly. Then I would encourage the review efforts to focus first on the iommu part. vSVA feature is also very important to us, it will be great if vSVA can be supported in guest world. We just submitted uacce for accelerator, which will be supporting SVA on host, thanks to Jean's effort. https://lkml.org/lkml/2020/2/11/54 However, supporting vSVA in guest is also a key component for accelerator. Looking forward this going to be happen. Any respin, I will be very happy to test. Thanks ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v9 00/11] SMMUv3 Nested Stage Setup (VFIO part)
Hi Zhangfei, On 3/3/20 1:57 PM, zhangfei wrote: > Hi, Eric > > On 2019/11/20 下午6:18, Auger Eric wrote: >> This series brings the VFIO part of HW nested paging support in the SMMUv3. The series depends on: [PATCH v9 00/14] SMMUv3 Nested Stage Setup (IOMMU part) (https://www.spinics.net/lists/kernel/msg3187714.html) 3 new IOCTLs are introduced that allow the userspace to 1) pass the guest stage 1 configuration 2) pass stage 1 MSI bindings 3) invalidate stage 1 related caches They map onto the related new IOMMU API functions. We introduce the capability to register specific interrupt indexes (see [1]). A new DMA_FAULT interrupt index allows to register an eventfd to be signaled whenever a stage 1 related fault is detected at physical level. Also a specific region allows to expose the fault records to the user space. Best Regards Eric This series can be found at: https://github.com/eauger/linux/tree/v5.3.0-rc0-2stage-v9 >>> I think you have already tested on ThunderX2, but as a formality, for >>> the whole series: >>> >>> Tested-by: Tomasz Nowicki >>> qemu: https://github.com/eauger/qemu/tree/v4.1.0-rc0-2stage-rfcv5 >>> kernel: https://github.com/eauger/linux/tree/v5.3.0-rc0-2stage-v9 + >>> Shameer's fix patch >>> >>> In my test I assigned Intel 82574L NIC and perform iperf tests. >> Thank you for your testing efforts. >>> Other folks from Marvell claimed this to be important feature so I asked >>> them to review and speak up on mailing list. >> That's nice to read that! So it is time for me to rebase both the iommu >> and vfio parts. I will submit something quickly. Then I would encourage >> the review efforts to focus first on the iommu part. >> >> > vSVA feature is also very important to us, it will be great if vSVA can > be supported in guest world. > > We just submitted uacce for accelerator, which will be supporting SVA on > host, thanks to Jean's effort. > > https://lkml.org/lkml/2020/2/11/54 > > > However, supporting vSVA in guest is also a key component for accelerator. > > Looking forward this going to be happen. > > > Any respin, I will be very happy to test. OK. Based on your interest and Marvell's interest too, I will respin both iommu & vfio series. Thanks Eric > > > Thanks > > > > ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v9 00/11] SMMUv3 Nested Stage Setup (VFIO part)
Hi Tomasz, On 11/20/19 9:15 AM, Tomasz Nowicki wrote: > Hi Eric, > > On 11.07.2019 15:56, Eric Auger wrote: >> This series brings the VFIO part of HW nested paging support >> in the SMMUv3. >> >> The series depends on: >> [PATCH v9 00/14] SMMUv3 Nested Stage Setup (IOMMU part) >> (https://www.spinics.net/lists/kernel/msg3187714.html) >> >> 3 new IOCTLs are introduced that allow the userspace to >> 1) pass the guest stage 1 configuration >> 2) pass stage 1 MSI bindings >> 3) invalidate stage 1 related caches >> >> They map onto the related new IOMMU API functions. >> >> We introduce the capability to register specific interrupt >> indexes (see [1]). A new DMA_FAULT interrupt index allows to register >> an eventfd to be signaled whenever a stage 1 related fault >> is detected at physical level. Also a specific region allows >> to expose the fault records to the user space. >> >> Best Regards >> >> Eric >> >> This series can be found at: >> https://github.com/eauger/linux/tree/v5.3.0-rc0-2stage-v9 > > I think you have already tested on ThunderX2, but as a formality, for > the whole series: > > Tested-by: Tomasz Nowicki > qemu: https://github.com/eauger/qemu/tree/v4.1.0-rc0-2stage-rfcv5 > kernel: https://github.com/eauger/linux/tree/v5.3.0-rc0-2stage-v9 + > Shameer's fix patch > > In my test I assigned Intel 82574L NIC and perform iperf tests. Thank you for your testing efforts. > > Other folks from Marvell claimed this to be important feature so I asked > them to review and speak up on mailing list. That's nice to read that! So it is time for me to rebase both the iommu and vfio parts. I will submit something quickly. Then I would encourage the review efforts to focus first on the iommu part. Thanks Eric > > Thanks, > Tomasz > ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v9 00/11] SMMUv3 Nested Stage Setup (VFIO part)
Hi Eric, On 11.07.2019 15:56, Eric Auger wrote: > This series brings the VFIO part of HW nested paging support > in the SMMUv3. > > The series depends on: > [PATCH v9 00/14] SMMUv3 Nested Stage Setup (IOMMU part) > (https://www.spinics.net/lists/kernel/msg3187714.html) > > 3 new IOCTLs are introduced that allow the userspace to > 1) pass the guest stage 1 configuration > 2) pass stage 1 MSI bindings > 3) invalidate stage 1 related caches > > They map onto the related new IOMMU API functions. > > We introduce the capability to register specific interrupt > indexes (see [1]). A new DMA_FAULT interrupt index allows to register > an eventfd to be signaled whenever a stage 1 related fault > is detected at physical level. Also a specific region allows > to expose the fault records to the user space. > > Best Regards > > Eric > > This series can be found at: > https://github.com/eauger/linux/tree/v5.3.0-rc0-2stage-v9 I think you have already tested on ThunderX2, but as a formality, for the whole series: Tested-by: Tomasz Nowicki qemu: https://github.com/eauger/qemu/tree/v4.1.0-rc0-2stage-rfcv5 kernel: https://github.com/eauger/linux/tree/v5.3.0-rc0-2stage-v9 + Shameer's fix patch In my test I assigned Intel 82574L NIC and perform iperf tests. Other folks from Marvell claimed this to be important feature so I asked them to review and speak up on mailing list. Thanks, Tomasz ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
RE: [PATCH v9 00/11] SMMUv3 Nested Stage Setup (VFIO part)
Hi Eric, > -Original Message- > From: Auger Eric [mailto:eric.au...@redhat.com] > Sent: 12 November 2019 20:35 > To: Shameerali Kolothum Thodi ; > eric.auger@gmail.com; io...@lists.linux-foundation.org; > linux-ker...@vger.kernel.org; k...@vger.kernel.org; > kvmarm@lists.cs.columbia.edu; j...@8bytes.org; > alex.william...@redhat.com; jacob.jun@linux.intel.com; > yi.l@intel.com; jean-philippe.bruc...@arm.com; will.dea...@arm.com; > robin.mur...@arm.com > Cc: kevin.t...@intel.com; vincent.ste...@arm.com; ashok@intel.com; > marc.zyng...@arm.com; tina.zh...@intel.com; Linuxarm > ; xuwei (O) > Subject: Re: [PATCH v9 00/11] SMMUv3 Nested Stage Setup (VFIO part) > > Hi Shameer, > [..] > > > > I just noted that CMDQ_OP_TLBI_NH_VA is missing the vmid filed which > seems > > to be the cause for single IOVA TLBI not working properly. > > > > I had this fix in arm-smmuv3.c, > > > > @@ -947,6 +947,7 @@ static int arm_smmu_cmdq_build_cmd(u64 *cmd, > struct arm_smmu_cmdq_ent *ent) > > cmd[1] |= FIELD_PREP(CMDQ_CFGI_1_RANGE, 31); > > break; > > case CMDQ_OP_TLBI_NH_VA: > > + cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_VMID, ent->tlbi.vmid); > Damn, I did not see that! That's it. ASID invalidation fills this field > indeed. You may post an independent patch for that. Sure. Just did that. " iommu/arm-smmu-v3: Populate VMID field for CMDQ_OP_TLBI_NH_VA" > cmd[0] |= > FIELD_PREP(CMDQ_TLBI_0_ASID, ent->tlbi.asid); > > cmd[1] |= FIELD_PREP(CMDQ_TLBI_1_LEAF, ent->tlbi.leaf); > > cmd[1] |= ent->tlbi.addr & CMDQ_TLBI_1_VA_MASK; > > > > > > With this, your original qemu branch is working. > > > > root@ubuntu:~# iperf -c 10.202.225.185 > > > > Client connecting to 10.202.225.185, TCP port 5001 TCP window size: 85.0 > KByte (default) > > > > [ 3] local 10.202.225.169 port 44894 connected with 10.202.225.185 port > 5001 > > [ ID] Interval Transfer Bandwidth > > [ 3] 0.0-10.0 sec 3.21 GBytes 2.76 Gbits/sec > > > > Could you please check this... > > > > I also have a rebase of your patches on top of 5.4-rc5. This has some > optimizations > > From Will such as batched TLBI inv. Please find it here, > > > > https://github.com/hisilicon/kernel-dev/tree/private-vSMMUv3-v9-v5.4-rc5 > > > > This gives me a better performance with iperf, > > > > root@ubuntu:~# iperf -c 10.202.225.185 > > > > Client connecting to 10.202.225.185, TCP port 5001 TCP window size: 85.0 > KByte (default) > > > > [ 3] local 10.202.225.169 port 55450 connected with 10.202.225.185 port > 5001 > > [ ID] Interval Transfer Bandwidth > > [ 3] 0.0-10.0 sec 4.91 GBytes 4.22 Gbits/sec root@ubuntu:~# > > > > If possible please check this branch as well. > > To be honest I don't really know what to do with this work. Despite the > efforts, this has suffered from a lack of traction in the community. My > last attempt to explain the use cases, upon Will's request at Plumber, > has not received any comment (https://lkml.org/lkml/2019/9/20/104). > > I think I will post a rebased version with your fix, as a matter to get > a clean snapshot. Thanks. That makes sense. If you think this work is useful for your projects, > please let it know on the ML. Right. While SVA use case is definitely the one we are very much interested, I will check within our team the priority for use case 1(native drivers in Guest) you mentioned in the above link. Cheers, Shameer > Thank you again! > > Eric > > > > Thanks, > > Shameer > > > >> Thanks, > >> Shameer > >> > >> > >>> Thanks > >>> > >>> Eric > >>>> > >>>> Cheers, > >>>> Shameer > >>>> > >>>>> Thanks > >>>>> > >>>>> Eric > >>>>> > >>>>> > >>>>> > >>>>>> > >>>>>> Please let me know. > >>>>>> > >>>>>> Thanks, > >>>>>> Shameer > >>>>>> > >>>>>>> Best Regards > >>>>>>> > >>>>>>> Eric > >>>>>>> > >
Re: [PATCH v9 00/11] SMMUv3 Nested Stage Setup (VFIO part)
Hi Shameer,
On 11/12/19 6:56 PM, Shameerali Kolothum Thodi wrote:
> Hi Eric,
>
>> -Original Message-
>> From: Shameerali Kolothum Thodi
>> Sent: 12 November 2019 14:21
>> To: 'Auger Eric' ; eric.auger@gmail.com;
>> io...@lists.linux-foundation.org; linux-ker...@vger.kernel.org;
>> k...@vger.kernel.org; kvmarm@lists.cs.columbia.edu; j...@8bytes.org;
>> alex.william...@redhat.com; jacob.jun@linux.intel.com;
>> yi.l@intel.com; jean-philippe.bruc...@arm.com; will.dea...@arm.com;
>> robin.mur...@arm.com
>> Cc: kevin.t...@intel.com; vincent.ste...@arm.com; ashok@intel.com;
>> marc.zyng...@arm.com; tina.zh...@intel.com; Linuxarm
>> ; xuwei (O)
>> Subject: RE: [PATCH v9 00/11] SMMUv3 Nested Stage Setup (VFIO part)
>>
> [...]
>>>>>> I am trying to get this running on one of our platform that has smmuv3
>> dual
>>>>>> stage support. I am seeing some issues with this when an ixgbe vf dev is
>>>>>> made pass-through and is behind a vSMMUv3 in Guest.
>>>>>>
>>>>>> Kernel used : https://github.com/eauger/linux/tree/v5.3.0-rc0-2stage-v9
>>>>>> Qemu: https://github.com/eauger/qemu/tree/v4.1.0-rc0-2stage-rfcv5
>>>>>>
>>>>>> And this is my Qemu cmd line,
>>>>>>
>>>>>> ./qemu-system-aarch64
>>>>>> -machine virt,kernel_irqchip=on,gic-version=3,iommu=smmuv3 -cpu host
>> \
>>>>>> -kernel Image \
>>>>>> -drive if=none,file=ubuntu,id=fs \
>>>>>> -device virtio-blk-device,drive=fs \
>>>>>> -device vfio-pci,host=:01:10.1 \
>>>>>> -bios QEMU_EFI.fd \
>>>>>> -net none \
>>>>>> -m 4G \
>>>>>> -nographic -D -d -enable-kvm \
>>>>>> -append "console=ttyAMA0 root=/dev/vda rw acpi=force"
>>>>>>
>>>>>> The basic ping from Guest works fine,
>>>>>> root@ubuntu:~# ping 10.202.225.185
>>>>>> PING 10.202.225.185 (10.202.225.185) 56(84) bytes of data.
>>>>>> 64 bytes from 10.202.225.185: icmp_seq=2 ttl=64 time=0.207 ms
>>>>>> 64 bytes from 10.202.225.185: icmp_seq=3 ttl=64 time=0.203 ms
>>>>>> ...
>>>>>>
>>>>>> But if I increase ping packet size,
>>>>>>
>>>>>> root@ubuntu:~# ping -s 1024 10.202.225.185
>>>>>> PING 10.202.225.185 (10.202.225.185) 1024(1052) bytes of data.
>>>>>> 1032 bytes from 10.202.225.185: icmp_seq=22 ttl=64 time=0.292 ms
>>>>>> 1032 bytes from 10.202.225.185: icmp_seq=23 ttl=64 time=0.207 ms
>>>>>> From 10.202.225.169 icmp_seq=66 Destination Host Unreachable
>>>>>> From 10.202.225.169 icmp_seq=67 Destination Host Unreachable
>>>>>> From 10.202.225.169 icmp_seq=68 Destination Host Unreachable
>>>>>> From 10.202.225.169 icmp_seq=69 Destination Host Unreachable
>>>>>>
>>>>>> And from Host kernel I get,
>>>>>> [ 819.970742] ixgbe :01:00.1 enp1s0f1: 3 Spoofed packets
>> detected
>>>>>> [ 824.002707] ixgbe :01:00.1 enp1s0f1: 1 Spoofed packets
>> detected
>>>>>> [ 828.034683] ixgbe :01:00.1 enp1s0f1: 1 Spoofed packets
>> detected
>>>>>> [ 830.050673] ixgbe :01:00.1 enp1s0f1: 4 Spoofed packets
>> detected
>>>>>> [ 832.066659] ixgbe :01:00.1 enp1s0f1: 1 Spoofed packets
>> detected
>>>>>> [ 834.082640] ixgbe :01:00.1 enp1s0f1: 3 Spoofed packets
>> detected
>>>>>>
>>>>>> Also noted that iperf cannot work as it fails to establish the connection
>>> with
>>>>> iperf
>>>>>> server.
>>>>>>
>>>>>> Please find attached the trace logs(vfio*, smmuv3*) from Qemu for your
>>>>> reference.
>>>>>> I haven't debugged this further yet and thought of checking with you if
>> this
>>> is
>>>>>> something you have seen already or not. Or maybe I am missing
>> something
>>>>> here?
>>>>>
>>>>> Please can you try to edit and modify hw/vfio/common.c, function
>>>>> vfio_iommu_unmap_notify
>>>>>
>>>>>
>>>>> /*
>>>>> if (size <= 0x1) {
>>>>> ustruct.info.cache = IOMMU_CACHE_INV_TYPE_IOTLB;
>>&
RE: [PATCH v9 00/11] SMMUv3 Nested Stage Setup (VFIO part)
Hi Eric,
> -Original Message-
> From: Shameerali Kolothum Thodi
> Sent: 12 November 2019 14:21
> To: 'Auger Eric' ; eric.auger@gmail.com;
> io...@lists.linux-foundation.org; linux-ker...@vger.kernel.org;
> k...@vger.kernel.org; kvmarm@lists.cs.columbia.edu; j...@8bytes.org;
> alex.william...@redhat.com; jacob.jun@linux.intel.com;
> yi.l@intel.com; jean-philippe.bruc...@arm.com; will.dea...@arm.com;
> robin.mur...@arm.com
> Cc: kevin.t...@intel.com; vincent.ste...@arm.com; ashok@intel.com;
> marc.zyng...@arm.com; tina.zh...@intel.com; Linuxarm
> ; xuwei (O)
> Subject: RE: [PATCH v9 00/11] SMMUv3 Nested Stage Setup (VFIO part)
>
[...]
> > >>> I am trying to get this running on one of our platform that has smmuv3
> dual
> > >>> stage support. I am seeing some issues with this when an ixgbe vf dev is
> > >>> made pass-through and is behind a vSMMUv3 in Guest.
> > >>>
> > >>> Kernel used : https://github.com/eauger/linux/tree/v5.3.0-rc0-2stage-v9
> > >>> Qemu: https://github.com/eauger/qemu/tree/v4.1.0-rc0-2stage-rfcv5
> > >>>
> > >>> And this is my Qemu cmd line,
> > >>>
> > >>> ./qemu-system-aarch64
> > >>> -machine virt,kernel_irqchip=on,gic-version=3,iommu=smmuv3 -cpu host
> \
> > >>> -kernel Image \
> > >>> -drive if=none,file=ubuntu,id=fs \
> > >>> -device virtio-blk-device,drive=fs \
> > >>> -device vfio-pci,host=:01:10.1 \
> > >>> -bios QEMU_EFI.fd \
> > >>> -net none \
> > >>> -m 4G \
> > >>> -nographic -D -d -enable-kvm \
> > >>> -append "console=ttyAMA0 root=/dev/vda rw acpi=force"
> > >>>
> > >>> The basic ping from Guest works fine,
> > >>> root@ubuntu:~# ping 10.202.225.185
> > >>> PING 10.202.225.185 (10.202.225.185) 56(84) bytes of data.
> > >>> 64 bytes from 10.202.225.185: icmp_seq=2 ttl=64 time=0.207 ms
> > >>> 64 bytes from 10.202.225.185: icmp_seq=3 ttl=64 time=0.203 ms
> > >>> ...
> > >>>
> > >>> But if I increase ping packet size,
> > >>>
> > >>> root@ubuntu:~# ping -s 1024 10.202.225.185
> > >>> PING 10.202.225.185 (10.202.225.185) 1024(1052) bytes of data.
> > >>> 1032 bytes from 10.202.225.185: icmp_seq=22 ttl=64 time=0.292 ms
> > >>> 1032 bytes from 10.202.225.185: icmp_seq=23 ttl=64 time=0.207 ms
> > >>> From 10.202.225.169 icmp_seq=66 Destination Host Unreachable
> > >>> From 10.202.225.169 icmp_seq=67 Destination Host Unreachable
> > >>> From 10.202.225.169 icmp_seq=68 Destination Host Unreachable
> > >>> From 10.202.225.169 icmp_seq=69 Destination Host Unreachable
> > >>>
> > >>> And from Host kernel I get,
> > >>> [ 819.970742] ixgbe :01:00.1 enp1s0f1: 3 Spoofed packets
> detected
> > >>> [ 824.002707] ixgbe :01:00.1 enp1s0f1: 1 Spoofed packets
> detected
> > >>> [ 828.034683] ixgbe :01:00.1 enp1s0f1: 1 Spoofed packets
> detected
> > >>> [ 830.050673] ixgbe :01:00.1 enp1s0f1: 4 Spoofed packets
> detected
> > >>> [ 832.066659] ixgbe :01:00.1 enp1s0f1: 1 Spoofed packets
> detected
> > >>> [ 834.082640] ixgbe :01:00.1 enp1s0f1: 3 Spoofed packets
> detected
> > >>>
> > >>> Also noted that iperf cannot work as it fails to establish the
> > >>> connection
> > with
> > >> iperf
> > >>> server.
> > >>>
> > >>> Please find attached the trace logs(vfio*, smmuv3*) from Qemu for your
> > >> reference.
> > >>> I haven't debugged this further yet and thought of checking with you if
> this
> > is
> > >>> something you have seen already or not. Or maybe I am missing
> something
> > >> here?
> > >>
> > >> Please can you try to edit and modify hw/vfio/common.c, function
> > >> vfio_iommu_unmap_notify
> > >>
> > >>
> > >> /*
> > >> if (size <= 0x1) {
> > >> ustruct.info.cache = IOMMU_CACHE_INV_TYPE_IOTLB;
> > >> ustruct.info.granularity = IOMMU_INV_GRANU_ADDR;
> > >> ustruct.info.addr_info.flags =
> > IOMMU_INV_ADDR_FLAGS_ARCHID;
> > >> if (iotlb->leaf) {
> > >> ustruct.info.addr_info.flags |=
> > >&
RE: [PATCH v9 00/11] SMMUv3 Nested Stage Setup (VFIO part)
> -Original Message- > From: Auger Eric [mailto:eric.au...@redhat.com] > Sent: 12 November 2019 13:22 > To: Shameerali Kolothum Thodi ; > eric.auger@gmail.com; io...@lists.linux-foundation.org; > linux-ker...@vger.kernel.org; k...@vger.kernel.org; > kvmarm@lists.cs.columbia.edu; j...@8bytes.org; > alex.william...@redhat.com; jacob.jun@linux.intel.com; > yi.l@intel.com; jean-philippe.bruc...@arm.com; will.dea...@arm.com; > robin.mur...@arm.com > Cc: kevin.t...@intel.com; vincent.ste...@arm.com; ashok@intel.com; > marc.zyng...@arm.com; tina.zh...@intel.com; Linuxarm > ; xuwei (O) > Subject: Re: [PATCH v9 00/11] SMMUv3 Nested Stage Setup (VFIO part) > > Hi Shameer, > > On 11/12/19 2:06 PM, Shameerali Kolothum Thodi wrote: > > Hi Eric, > > > >> -Original Message- > >> From: Auger Eric [mailto:eric.au...@redhat.com] > >> Sent: 12 November 2019 11:29 > >> To: Shameerali Kolothum Thodi ; > >> eric.auger@gmail.com; io...@lists.linux-foundation.org; > >> linux-ker...@vger.kernel.org; k...@vger.kernel.org; > >> kvmarm@lists.cs.columbia.edu; j...@8bytes.org; > >> alex.william...@redhat.com; jacob.jun@linux.intel.com; > >> yi.l@intel.com; jean-philippe.bruc...@arm.com; will.dea...@arm.com; > >> robin.mur...@arm.com > >> Cc: kevin.t...@intel.com; vincent.ste...@arm.com; ashok@intel.com; > >> marc.zyng...@arm.com; tina.zh...@intel.com; Linuxarm > >> ; xuwei (O) > >> Subject: Re: [PATCH v9 00/11] SMMUv3 Nested Stage Setup (VFIO part) > >> > >> Hi Shameer, > >> On 11/12/19 12:08 PM, Shameerali Kolothum Thodi wrote: > >>> Hi Eric, > >>> > >>>> -Original Message- > >>>> From: kvmarm-boun...@lists.cs.columbia.edu > >>>> [mailto:kvmarm-boun...@lists.cs.columbia.edu] On Behalf Of Eric Auger > >>>> Sent: 11 July 2019 14:56 > >>>> To: eric.auger@gmail.com; eric.au...@redhat.com; > >>>> io...@lists.linux-foundation.org; linux-ker...@vger.kernel.org; > >>>> k...@vger.kernel.org; kvmarm@lists.cs.columbia.edu; j...@8bytes.org; > >>>> alex.william...@redhat.com; jacob.jun@linux.intel.com; > >>>> yi.l@intel.com; jean-philippe.bruc...@arm.com; > will.dea...@arm.com; > >>>> robin.mur...@arm.com > >>>> Cc: kevin.t...@intel.com; vincent.ste...@arm.com; > ashok@intel.com; > >>>> marc.zyng...@arm.com; tina.zh...@intel.com > >>>> Subject: [PATCH v9 00/11] SMMUv3 Nested Stage Setup (VFIO part) > >>>> > >>>> This series brings the VFIO part of HW nested paging support > >>>> in the SMMUv3. > >>>> > >>>> The series depends on: > >>>> [PATCH v9 00/14] SMMUv3 Nested Stage Setup (IOMMU part) > >>>> (https://www.spinics.net/lists/kernel/msg3187714.html) > >>>> > >>>> 3 new IOCTLs are introduced that allow the userspace to > >>>> 1) pass the guest stage 1 configuration > >>>> 2) pass stage 1 MSI bindings > >>>> 3) invalidate stage 1 related caches > >>>> > >>>> They map onto the related new IOMMU API functions. > >>>> > >>>> We introduce the capability to register specific interrupt > >>>> indexes (see [1]). A new DMA_FAULT interrupt index allows to register > >>>> an eventfd to be signaled whenever a stage 1 related fault > >>>> is detected at physical level. Also a specific region allows > >>>> to expose the fault records to the user space. > >>> > >>> I am trying to get this running on one of our platform that has smmuv3 > >>> dual > >>> stage support. I am seeing some issues with this when an ixgbe vf dev is > >>> made pass-through and is behind a vSMMUv3 in Guest. > >>> > >>> Kernel used : https://github.com/eauger/linux/tree/v5.3.0-rc0-2stage-v9 > >>> Qemu: https://github.com/eauger/qemu/tree/v4.1.0-rc0-2stage-rfcv5 > >>> > >>> And this is my Qemu cmd line, > >>> > >>> ./qemu-system-aarch64 > >>> -machine virt,kernel_irqchip=on,gic-version=3,iommu=smmuv3 -cpu host \ > >>> -kernel Image \ > >>> -drive if=none,file=ubuntu,id=fs \ > >>> -device virtio-blk-device,drive=fs \ > >>> -device vfio-pci,host=:01:10.1 \ > >>> -bios QEMU_EFI.fd \ > >>> -net none \ > >>> -m 4G \ > >>> -nographic
Re: [PATCH v9 00/11] SMMUv3 Nested Stage Setup (VFIO part)
Hi Shameer, On 11/12/19 2:06 PM, Shameerali Kolothum Thodi wrote: > Hi Eric, > >> -Original Message- >> From: Auger Eric [mailto:eric.au...@redhat.com] >> Sent: 12 November 2019 11:29 >> To: Shameerali Kolothum Thodi ; >> eric.auger@gmail.com; io...@lists.linux-foundation.org; >> linux-ker...@vger.kernel.org; k...@vger.kernel.org; >> kvmarm@lists.cs.columbia.edu; j...@8bytes.org; >> alex.william...@redhat.com; jacob.jun@linux.intel.com; >> yi.l@intel.com; jean-philippe.bruc...@arm.com; will.dea...@arm.com; >> robin.mur...@arm.com >> Cc: kevin.t...@intel.com; vincent.ste...@arm.com; ashok@intel.com; >> marc.zyng...@arm.com; tina.zh...@intel.com; Linuxarm >> ; xuwei (O) >> Subject: Re: [PATCH v9 00/11] SMMUv3 Nested Stage Setup (VFIO part) >> >> Hi Shameer, >> On 11/12/19 12:08 PM, Shameerali Kolothum Thodi wrote: >>> Hi Eric, >>> >>>> -Original Message- >>>> From: kvmarm-boun...@lists.cs.columbia.edu >>>> [mailto:kvmarm-boun...@lists.cs.columbia.edu] On Behalf Of Eric Auger >>>> Sent: 11 July 2019 14:56 >>>> To: eric.auger@gmail.com; eric.au...@redhat.com; >>>> io...@lists.linux-foundation.org; linux-ker...@vger.kernel.org; >>>> k...@vger.kernel.org; kvmarm@lists.cs.columbia.edu; j...@8bytes.org; >>>> alex.william...@redhat.com; jacob.jun@linux.intel.com; >>>> yi.l@intel.com; jean-philippe.bruc...@arm.com; will.dea...@arm.com; >>>> robin.mur...@arm.com >>>> Cc: kevin.t...@intel.com; vincent.ste...@arm.com; ashok@intel.com; >>>> marc.zyng...@arm.com; tina.zh...@intel.com >>>> Subject: [PATCH v9 00/11] SMMUv3 Nested Stage Setup (VFIO part) >>>> >>>> This series brings the VFIO part of HW nested paging support >>>> in the SMMUv3. >>>> >>>> The series depends on: >>>> [PATCH v9 00/14] SMMUv3 Nested Stage Setup (IOMMU part) >>>> (https://www.spinics.net/lists/kernel/msg3187714.html) >>>> >>>> 3 new IOCTLs are introduced that allow the userspace to >>>> 1) pass the guest stage 1 configuration >>>> 2) pass stage 1 MSI bindings >>>> 3) invalidate stage 1 related caches >>>> >>>> They map onto the related new IOMMU API functions. >>>> >>>> We introduce the capability to register specific interrupt >>>> indexes (see [1]). A new DMA_FAULT interrupt index allows to register >>>> an eventfd to be signaled whenever a stage 1 related fault >>>> is detected at physical level. Also a specific region allows >>>> to expose the fault records to the user space. >>> >>> I am trying to get this running on one of our platform that has smmuv3 dual >>> stage support. I am seeing some issues with this when an ixgbe vf dev is >>> made pass-through and is behind a vSMMUv3 in Guest. >>> >>> Kernel used : https://github.com/eauger/linux/tree/v5.3.0-rc0-2stage-v9 >>> Qemu: https://github.com/eauger/qemu/tree/v4.1.0-rc0-2stage-rfcv5 >>> >>> And this is my Qemu cmd line, >>> >>> ./qemu-system-aarch64 >>> -machine virt,kernel_irqchip=on,gic-version=3,iommu=smmuv3 -cpu host \ >>> -kernel Image \ >>> -drive if=none,file=ubuntu,id=fs \ >>> -device virtio-blk-device,drive=fs \ >>> -device vfio-pci,host=:01:10.1 \ >>> -bios QEMU_EFI.fd \ >>> -net none \ >>> -m 4G \ >>> -nographic -D -d -enable-kvm \ >>> -append "console=ttyAMA0 root=/dev/vda rw acpi=force" >>> >>> The basic ping from Guest works fine, >>> root@ubuntu:~# ping 10.202.225.185 >>> PING 10.202.225.185 (10.202.225.185) 56(84) bytes of data. >>> 64 bytes from 10.202.225.185: icmp_seq=2 ttl=64 time=0.207 ms >>> 64 bytes from 10.202.225.185: icmp_seq=3 ttl=64 time=0.203 ms >>> ... >>> >>> But if I increase ping packet size, >>> >>> root@ubuntu:~# ping -s 1024 10.202.225.185 >>> PING 10.202.225.185 (10.202.225.185) 1024(1052) bytes of data. >>> 1032 bytes from 10.202.225.185: icmp_seq=22 ttl=64 time=0.292 ms >>> 1032 bytes from 10.202.225.185: icmp_seq=23 ttl=64 time=0.207 ms >>> From 10.202.225.169 icmp_seq=66 Destination Host Unreachable >>> From 10.202.225.169 icmp_seq=67 Destination Host Unreachable >>> From 10.202.225.169 icmp_seq=68 Destination Host Unreachable >>> From 10.202.225.169 icmp_seq=69 Destination Host Unreachable >>> >>> And from
RE: [PATCH v9 00/11] SMMUv3 Nested Stage Setup (VFIO part)
Hi Eric, > -Original Message- > From: Auger Eric [mailto:eric.au...@redhat.com] > Sent: 12 November 2019 11:29 > To: Shameerali Kolothum Thodi ; > eric.auger@gmail.com; io...@lists.linux-foundation.org; > linux-ker...@vger.kernel.org; k...@vger.kernel.org; > kvmarm@lists.cs.columbia.edu; j...@8bytes.org; > alex.william...@redhat.com; jacob.jun@linux.intel.com; > yi.l@intel.com; jean-philippe.bruc...@arm.com; will.dea...@arm.com; > robin.mur...@arm.com > Cc: kevin.t...@intel.com; vincent.ste...@arm.com; ashok@intel.com; > marc.zyng...@arm.com; tina.zh...@intel.com; Linuxarm > ; xuwei (O) > Subject: Re: [PATCH v9 00/11] SMMUv3 Nested Stage Setup (VFIO part) > > Hi Shameer, > On 11/12/19 12:08 PM, Shameerali Kolothum Thodi wrote: > > Hi Eric, > > > >> -Original Message- > >> From: kvmarm-boun...@lists.cs.columbia.edu > >> [mailto:kvmarm-boun...@lists.cs.columbia.edu] On Behalf Of Eric Auger > >> Sent: 11 July 2019 14:56 > >> To: eric.auger@gmail.com; eric.au...@redhat.com; > >> io...@lists.linux-foundation.org; linux-ker...@vger.kernel.org; > >> k...@vger.kernel.org; kvmarm@lists.cs.columbia.edu; j...@8bytes.org; > >> alex.william...@redhat.com; jacob.jun@linux.intel.com; > >> yi.l@intel.com; jean-philippe.bruc...@arm.com; will.dea...@arm.com; > >> robin.mur...@arm.com > >> Cc: kevin.t...@intel.com; vincent.ste...@arm.com; ashok@intel.com; > >> marc.zyng...@arm.com; tina.zh...@intel.com > >> Subject: [PATCH v9 00/11] SMMUv3 Nested Stage Setup (VFIO part) > >> > >> This series brings the VFIO part of HW nested paging support > >> in the SMMUv3. > >> > >> The series depends on: > >> [PATCH v9 00/14] SMMUv3 Nested Stage Setup (IOMMU part) > >> (https://www.spinics.net/lists/kernel/msg3187714.html) > >> > >> 3 new IOCTLs are introduced that allow the userspace to > >> 1) pass the guest stage 1 configuration > >> 2) pass stage 1 MSI bindings > >> 3) invalidate stage 1 related caches > >> > >> They map onto the related new IOMMU API functions. > >> > >> We introduce the capability to register specific interrupt > >> indexes (see [1]). A new DMA_FAULT interrupt index allows to register > >> an eventfd to be signaled whenever a stage 1 related fault > >> is detected at physical level. Also a specific region allows > >> to expose the fault records to the user space. > > > > I am trying to get this running on one of our platform that has smmuv3 dual > > stage support. I am seeing some issues with this when an ixgbe vf dev is > > made pass-through and is behind a vSMMUv3 in Guest. > > > > Kernel used : https://github.com/eauger/linux/tree/v5.3.0-rc0-2stage-v9 > > Qemu: https://github.com/eauger/qemu/tree/v4.1.0-rc0-2stage-rfcv5 > > > > And this is my Qemu cmd line, > > > > ./qemu-system-aarch64 > > -machine virt,kernel_irqchip=on,gic-version=3,iommu=smmuv3 -cpu host \ > > -kernel Image \ > > -drive if=none,file=ubuntu,id=fs \ > > -device virtio-blk-device,drive=fs \ > > -device vfio-pci,host=:01:10.1 \ > > -bios QEMU_EFI.fd \ > > -net none \ > > -m 4G \ > > -nographic -D -d -enable-kvm \ > > -append "console=ttyAMA0 root=/dev/vda rw acpi=force" > > > > The basic ping from Guest works fine, > > root@ubuntu:~# ping 10.202.225.185 > > PING 10.202.225.185 (10.202.225.185) 56(84) bytes of data. > > 64 bytes from 10.202.225.185: icmp_seq=2 ttl=64 time=0.207 ms > > 64 bytes from 10.202.225.185: icmp_seq=3 ttl=64 time=0.203 ms > > ... > > > > But if I increase ping packet size, > > > > root@ubuntu:~# ping -s 1024 10.202.225.185 > > PING 10.202.225.185 (10.202.225.185) 1024(1052) bytes of data. > > 1032 bytes from 10.202.225.185: icmp_seq=22 ttl=64 time=0.292 ms > > 1032 bytes from 10.202.225.185: icmp_seq=23 ttl=64 time=0.207 ms > > From 10.202.225.169 icmp_seq=66 Destination Host Unreachable > > From 10.202.225.169 icmp_seq=67 Destination Host Unreachable > > From 10.202.225.169 icmp_seq=68 Destination Host Unreachable > > From 10.202.225.169 icmp_seq=69 Destination Host Unreachable > > > > And from Host kernel I get, > > [ 819.970742] ixgbe :01:00.1 enp1s0f1: 3 Spoofed packets detected > > [ 824.002707] ixgbe :01:00.1 enp1s0f1: 1 Spoofed packets detected > > [ 828.034683] ixgbe :01:00.1 enp1s0f1: 1 Spoofed packets detected > > [ 830.050673] ixgbe :01:00.1 enp1s0f1: 4 Spoofed packets detected > > [ 832.06
Re: [PATCH v9 00/11] SMMUv3 Nested Stage Setup (VFIO part)
Hi Shameer,
On 11/12/19 12:08 PM, Shameerali Kolothum Thodi wrote:
> Hi Eric,
>
>> -Original Message-
>> From: kvmarm-boun...@lists.cs.columbia.edu
>> [mailto:kvmarm-boun...@lists.cs.columbia.edu] On Behalf Of Eric Auger
>> Sent: 11 July 2019 14:56
>> To: eric.auger@gmail.com; eric.au...@redhat.com;
>> io...@lists.linux-foundation.org; linux-ker...@vger.kernel.org;
>> k...@vger.kernel.org; kvmarm@lists.cs.columbia.edu; j...@8bytes.org;
>> alex.william...@redhat.com; jacob.jun@linux.intel.com;
>> yi.l@intel.com; jean-philippe.bruc...@arm.com; will.dea...@arm.com;
>> robin.mur...@arm.com
>> Cc: kevin.t...@intel.com; vincent.ste...@arm.com; ashok@intel.com;
>> marc.zyng...@arm.com; tina.zh...@intel.com
>> Subject: [PATCH v9 00/11] SMMUv3 Nested Stage Setup (VFIO part)
>>
>> This series brings the VFIO part of HW nested paging support
>> in the SMMUv3.
>>
>> The series depends on:
>> [PATCH v9 00/14] SMMUv3 Nested Stage Setup (IOMMU part)
>> (https://www.spinics.net/lists/kernel/msg3187714.html)
>>
>> 3 new IOCTLs are introduced that allow the userspace to
>> 1) pass the guest stage 1 configuration
>> 2) pass stage 1 MSI bindings
>> 3) invalidate stage 1 related caches
>>
>> They map onto the related new IOMMU API functions.
>>
>> We introduce the capability to register specific interrupt
>> indexes (see [1]). A new DMA_FAULT interrupt index allows to register
>> an eventfd to be signaled whenever a stage 1 related fault
>> is detected at physical level. Also a specific region allows
>> to expose the fault records to the user space.
>
> I am trying to get this running on one of our platform that has smmuv3 dual
> stage support. I am seeing some issues with this when an ixgbe vf dev is
> made pass-through and is behind a vSMMUv3 in Guest.
>
> Kernel used : https://github.com/eauger/linux/tree/v5.3.0-rc0-2stage-v9
> Qemu: https://github.com/eauger/qemu/tree/v4.1.0-rc0-2stage-rfcv5
>
> And this is my Qemu cmd line,
>
> ./qemu-system-aarch64
> -machine virt,kernel_irqchip=on,gic-version=3,iommu=smmuv3 -cpu host \
> -kernel Image \
> -drive if=none,file=ubuntu,id=fs \
> -device virtio-blk-device,drive=fs \
> -device vfio-pci,host=:01:10.1 \
> -bios QEMU_EFI.fd \
> -net none \
> -m 4G \
> -nographic -D -d -enable-kvm \
> -append "console=ttyAMA0 root=/dev/vda rw acpi=force"
>
> The basic ping from Guest works fine,
> root@ubuntu:~# ping 10.202.225.185
> PING 10.202.225.185 (10.202.225.185) 56(84) bytes of data.
> 64 bytes from 10.202.225.185: icmp_seq=2 ttl=64 time=0.207 ms
> 64 bytes from 10.202.225.185: icmp_seq=3 ttl=64 time=0.203 ms
> ...
>
> But if I increase ping packet size,
>
> root@ubuntu:~# ping -s 1024 10.202.225.185
> PING 10.202.225.185 (10.202.225.185) 1024(1052) bytes of data.
> 1032 bytes from 10.202.225.185: icmp_seq=22 ttl=64 time=0.292 ms
> 1032 bytes from 10.202.225.185: icmp_seq=23 ttl=64 time=0.207 ms
> From 10.202.225.169 icmp_seq=66 Destination Host Unreachable
> From 10.202.225.169 icmp_seq=67 Destination Host Unreachable
> From 10.202.225.169 icmp_seq=68 Destination Host Unreachable
> From 10.202.225.169 icmp_seq=69 Destination Host Unreachable
>
> And from Host kernel I get,
> [ 819.970742] ixgbe :01:00.1 enp1s0f1: 3 Spoofed packets detected
> [ 824.002707] ixgbe :01:00.1 enp1s0f1: 1 Spoofed packets detected
> [ 828.034683] ixgbe :01:00.1 enp1s0f1: 1 Spoofed packets detected
> [ 830.050673] ixgbe :01:00.1 enp1s0f1: 4 Spoofed packets detected
> [ 832.066659] ixgbe :01:00.1 enp1s0f1: 1 Spoofed packets detected
> [ 834.082640] ixgbe :01:00.1 enp1s0f1: 3 Spoofed packets detected
>
> Also noted that iperf cannot work as it fails to establish the connection
> with iperf
> server.
>
> Please find attached the trace logs(vfio*, smmuv3*) from Qemu for your
> reference.
> I haven't debugged this further yet and thought of checking with you if this
> is
> something you have seen already or not. Or maybe I am missing something here?
Please can you try to edit and modify hw/vfio/common.c, function
vfio_iommu_unmap_notify
/*
if (size <= 0x1) {
ustruct.info.cache = IOMMU_CACHE_INV_TYPE_IOTLB;
ustruct.info.granularity = IOMMU_INV_GRANU_ADDR;
ustruct.info.addr_info.flags = IOMMU_INV_ADDR_FLAGS_ARCHID;
if (iotlb->leaf) {
ustruct.info.addr_info.flags |= IOMMU_INV_ADDR_FLAGS_LEAF;
}
ustruct.info.addr_info.archid = iotlb->arch_id;
ustruct.info.addr_info.addr = start;
ustruct.info.addr_info.granule_size = size;
ustruct.info.addr_info.nb_granules = 1;
trace_vfio_iommu_addr_inv_iotlb(iotlb->arch_id, start, size, 1,
iotlb->leaf);
} else {
*/
ustruct.info.cache = IOMMU_CACHE_INV_TYPE_IOTLB;
ustruct.info.granularity = IOMMU_INV_GRANU_PASID;
ustruct.info.pasid_info.archid = iotlb->arch_id;
ustruct.info.pasid_info.flags = IOMMU_INV_PASID_FLAGS_ARCHID;
RE: [PATCH v9 00/11] SMMUv3 Nested Stage Setup (VFIO part)
Hi Eric, > -Original Message- > From: kvmarm-boun...@lists.cs.columbia.edu > [mailto:kvmarm-boun...@lists.cs.columbia.edu] On Behalf Of Eric Auger > Sent: 11 July 2019 14:56 > To: eric.auger@gmail.com; eric.au...@redhat.com; > io...@lists.linux-foundation.org; linux-ker...@vger.kernel.org; > k...@vger.kernel.org; kvmarm@lists.cs.columbia.edu; j...@8bytes.org; > alex.william...@redhat.com; jacob.jun@linux.intel.com; > yi.l@intel.com; jean-philippe.bruc...@arm.com; will.dea...@arm.com; > robin.mur...@arm.com > Cc: kevin.t...@intel.com; vincent.ste...@arm.com; ashok@intel.com; > marc.zyng...@arm.com; tina.zh...@intel.com > Subject: [PATCH v9 00/11] SMMUv3 Nested Stage Setup (VFIO part) > > This series brings the VFIO part of HW nested paging support > in the SMMUv3. > > The series depends on: > [PATCH v9 00/14] SMMUv3 Nested Stage Setup (IOMMU part) > (https://www.spinics.net/lists/kernel/msg3187714.html) > > 3 new IOCTLs are introduced that allow the userspace to > 1) pass the guest stage 1 configuration > 2) pass stage 1 MSI bindings > 3) invalidate stage 1 related caches > > They map onto the related new IOMMU API functions. > > We introduce the capability to register specific interrupt > indexes (see [1]). A new DMA_FAULT interrupt index allows to register > an eventfd to be signaled whenever a stage 1 related fault > is detected at physical level. Also a specific region allows > to expose the fault records to the user space. I am trying to get this running on one of our platform that has smmuv3 dual stage support. I am seeing some issues with this when an ixgbe vf dev is made pass-through and is behind a vSMMUv3 in Guest. Kernel used : https://github.com/eauger/linux/tree/v5.3.0-rc0-2stage-v9 Qemu: https://github.com/eauger/qemu/tree/v4.1.0-rc0-2stage-rfcv5 And this is my Qemu cmd line, ./qemu-system-aarch64 -machine virt,kernel_irqchip=on,gic-version=3,iommu=smmuv3 -cpu host \ -kernel Image \ -drive if=none,file=ubuntu,id=fs \ -device virtio-blk-device,drive=fs \ -device vfio-pci,host=:01:10.1 \ -bios QEMU_EFI.fd \ -net none \ -m 4G \ -nographic -D -d -enable-kvm \ -append "console=ttyAMA0 root=/dev/vda rw acpi=force" The basic ping from Guest works fine, root@ubuntu:~# ping 10.202.225.185 PING 10.202.225.185 (10.202.225.185) 56(84) bytes of data. 64 bytes from 10.202.225.185: icmp_seq=2 ttl=64 time=0.207 ms 64 bytes from 10.202.225.185: icmp_seq=3 ttl=64 time=0.203 ms ... But if I increase ping packet size, root@ubuntu:~# ping -s 1024 10.202.225.185 PING 10.202.225.185 (10.202.225.185) 1024(1052) bytes of data. 1032 bytes from 10.202.225.185: icmp_seq=22 ttl=64 time=0.292 ms 1032 bytes from 10.202.225.185: icmp_seq=23 ttl=64 time=0.207 ms >From 10.202.225.169 icmp_seq=66 Destination Host Unreachable >From 10.202.225.169 icmp_seq=67 Destination Host Unreachable >From 10.202.225.169 icmp_seq=68 Destination Host Unreachable >From 10.202.225.169 icmp_seq=69 Destination Host Unreachable And from Host kernel I get, [ 819.970742] ixgbe :01:00.1 enp1s0f1: 3 Spoofed packets detected [ 824.002707] ixgbe :01:00.1 enp1s0f1: 1 Spoofed packets detected [ 828.034683] ixgbe :01:00.1 enp1s0f1: 1 Spoofed packets detected [ 830.050673] ixgbe :01:00.1 enp1s0f1: 4 Spoofed packets detected [ 832.066659] ixgbe :01:00.1 enp1s0f1: 1 Spoofed packets detected [ 834.082640] ixgbe :01:00.1 enp1s0f1: 3 Spoofed packets detected Also noted that iperf cannot work as it fails to establish the connection with iperf server. Please find attached the trace logs(vfio*, smmuv3*) from Qemu for your reference. I haven't debugged this further yet and thought of checking with you if this is something you have seen already or not. Or maybe I am missing something here? Please let me know. Thanks, Shameer > Best Regards > > Eric > > This series can be found at: > https://github.com/eauger/linux/tree/v5.3.0-rc0-2stage-v9 > > It series includes Tina's patch steming from > [1] "[RFC PATCH v2 1/3] vfio: Use capability chains to handle device > specific irq" plus patches originally contributed by Yi. > > History: > > v8 -> v9: > - introduce specific irq framework > - single fault region > - iommu_unregister_device_fault_handler failure case not handled > yet. > > v7 -> v8: > - rebase on top of v5.2-rc1 and especially > 8be39a1a04c1 iommu/arm-smmu-v3: Add a master->domain pointer > - dynamic alloc of s1_cfg/s2_cfg > - __arm_smmu_tlb_inv_asid/s1_range_nosync > - check there is no HW MSI regions > - asid invalidation using pasid extended struct (change in the uapi) > - add s1_live/s2_live checks > - move check about support of nested stages in domain finalise > - fixes in error reporting according to the discussion with Robin > - reordered the patches to have first iommu/smmuv3 patches and then > VFIO patches > > v6 -> v7: > - removed device handle from bind/unbind_guest_msi > - added "iommu/smmuv3: Nested mode single MSI doorbell per domain > enforcement" >
Re: [PATCH v9 00/11] SMMUv3 Nested Stage Setup (VFIO part)
On 2019/7/11 下午9:56, Eric Auger wrote: This series brings the VFIO part of HW nested paging support in the SMMUv3. The series depends on: [PATCH v9 00/14] SMMUv3 Nested Stage Setup (IOMMU part) (https://www.spinics.net/lists/kernel/msg3187714.html) 3 new IOCTLs are introduced that allow the userspace to 1) pass the guest stage 1 configuration 2) pass stage 1 MSI bindings 3) invalidate stage 1 related caches They map onto the related new IOMMU API functions. We introduce the capability to register specific interrupt indexes (see [1]). A new DMA_FAULT interrupt index allows to register an eventfd to be signaled whenever a stage 1 related fault is detected at physical level. Also a specific region allows to expose the fault records to the user space. Best Regards Eric This series can be found at: https://github.com/eauger/linux/tree/v5.3.0-rc0-2stage-v9 It series includes Tina's patch steming from [1] "[RFC PATCH v2 1/3] vfio: Use capability chains to handle device specific irq" plus patches originally contributed by Yi. Thanks Eric. Have tested vfio mode in qemu on Hisilicon arm64 platform, using gIOVA qemu command: -machine virt,gic_version=3,iommu=smmuv3 -device vfio-pci,host=:75:00.1 Tested-by: Zhangfei Gao qemu: https://github.com/eauger/qemu/tree/v4.1.0-rc0-2stage-rfcv5 kernel: https://github.com/eauger/linux/tree/v5.3.0-rc0-2stage-v9 ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
