Re: [RFC 13/20] iommu: Extend iommu_at[de]tach_device() for multiple devices group

2021-10-25 Thread Jason Gunthorpe via iommu
On Tue, Oct 26, 2021 at 12:16:43AM +1100, David Gibson wrote: > If you attach devices A and B (both in group X) to IOAS 1, then detach > device A, what happens? Do you detach both devices? Or do you have a > counter so you have to detach as many time as you attached? I would refcount it since

Re: [RFC 10/20] iommu/iommufd: Add IOMMU_DEVICE_GET_INFO

2021-10-25 Thread Jason Gunthorpe via iommu
On Fri, Oct 22, 2021 at 03:08:06AM +, Tian, Kevin wrote: > > I have no idea what security model makes sense for wbinvd, that is the > > major question you have to answer. > > wbinvd flushes the entire cache in local cpu. It's more a performance > isolation problem but nothing can prevent it

Re: [RFC 10/20] iommu/iommufd: Add IOMMU_DEVICE_GET_INFO

2021-10-25 Thread Jason Gunthorpe via iommu
On Fri, Oct 22, 2021 at 08:49:03AM +0100, Jean-Philippe Brucker wrote: > On Thu, Oct 21, 2021 at 08:22:23PM -0300, Jason Gunthorpe wrote: > > On Thu, Oct 21, 2021 at 03:58:02PM +0100, Jean-Philippe Brucker wrote: > > > On Thu, Oct 21, 2021 at 02:26:00AM +, Tian, Kevin wro

Re: [RFC 02/20] vfio: Add device class for /dev/vfio/devices

2021-10-25 Thread Jason Gunthorpe via iommu
On Mon, Oct 25, 2021 at 06:28:09AM +, Liu, Yi L wrote: >thanks for the guiding. will also refer to your vfio_group_cdev series. > >Need to double confirm here. Not quite following on the kfree. Is >this kfree to free the vfio_device structure? But now the >vfio_device pointer

Re: [RFC 13/20] iommu: Extend iommu_at[de]tach_device() for multiple devices group

2021-10-25 Thread Jason Gunthorpe via iommu
On Mon, Oct 25, 2021 at 04:14:56PM +1100, David Gibson wrote: > On Mon, Oct 18, 2021 at 01:32:38PM -0300, Jason Gunthorpe wrote: > > On Mon, Oct 18, 2021 at 02:57:12PM +1100, David Gibson wrote: > > > > > The first user might read this. Subsequent users are likely

Re: [RFC 10/20] iommu/iommufd: Add IOMMU_DEVICE_GET_INFO

2021-10-21 Thread Jason Gunthorpe via iommu
On Thu, Oct 21, 2021 at 02:26:00AM +, Tian, Kevin wrote: > But in reality only Intel integrated GPUs have this special no-snoop > trick (fixed knowledge), with a dedicated IOMMU which doesn't > support enforce-snoop format at all. In this case there is no choice > that the user can further

Re: [RFC 10/20] iommu/iommufd: Add IOMMU_DEVICE_GET_INFO

2021-10-21 Thread Jason Gunthorpe via iommu
On Thu, Oct 21, 2021 at 03:58:02PM +0100, Jean-Philippe Brucker wrote: > On Thu, Oct 21, 2021 at 02:26:00AM +, Tian, Kevin wrote: > > > I'll leave it to Jean to confirm. If only coherent DMA can be used in > > > the guest on other platforms, suppose VFIO should not blindly set > > >

Re: [RFC 01/20] iommu/iommufd: Add /dev/iommu core

2021-10-19 Thread Jason Gunthorpe via iommu
On Tue, Oct 19, 2021 at 10:11:34AM -0700, Jacob Pan wrote: > Hi Jason, > > On Tue, 19 Oct 2021 13:57:47 -0300, Jason Gunthorpe wrote: > > > On Tue, Oct 19, 2021 at 09:57:34AM -0700, Jacob Pan wrote: > > > Hi Jason, > > > > > > On Fri, 15 Oct 2021

Re: [RFC 01/20] iommu/iommufd: Add /dev/iommu core

2021-10-19 Thread Jason Gunthorpe via iommu
On Tue, Oct 19, 2021 at 09:57:34AM -0700, Jacob Pan wrote: > Hi Jason, > > On Fri, 15 Oct 2021 08:18:07 -0300, Jason Gunthorpe wrote: > > > On Fri, Oct 15, 2021 at 09:18:06AM +, Liu, Yi L wrote: > > > > > > Acquire from the xarray is > > &

Re: [RFC 11/20] iommu/iommufd: Add IOMMU_IOASID_ALLOC/FREE

2021-10-18 Thread Jason Gunthorpe via iommu
On Mon, Oct 18, 2021 at 02:50:54PM +1100, David Gibson wrote: > Hrm... which makes me think... if we allow this for the common > kernel-managed case, do we even need to have capcity in the high-level > interface for reporting IO holes? If the kernel can choose a non-zero > base, it could just

Re: [RFC 13/20] iommu: Extend iommu_at[de]tach_device() for multiple devices group

2021-10-18 Thread Jason Gunthorpe via iommu
On Mon, Oct 18, 2021 at 02:57:12PM +1100, David Gibson wrote: > The first user might read this. Subsequent users are likely to just > copy paste examples from earlier things without fully understanding > them. In general documenting restrictions somewhere is never as > effective as making those

Re: [RFC 01/20] iommu/iommufd: Add /dev/iommu core

2021-10-15 Thread Jason Gunthorpe via iommu
On Fri, Oct 15, 2021 at 09:18:06AM +, Liu, Yi L wrote: > > Acquire from the xarray is > >rcu_lock() > >ioas = xa_load() > >if (ioas) > > if (down_read_trylock(>destroying_lock)) > > all good suggestions, will refine accordingly. Here destroying_lock is a > rw_semaphore.

Re: [RFC 06/20] iommu: Add iommu_device_init[exit]_user_dma interfaces

2021-10-15 Thread Jason Gunthorpe via iommu
On Fri, Oct 15, 2021 at 01:29:16AM +, Tian, Kevin wrote: > Hi, Jason, > > > From: Jason Gunthorpe > > Sent: Wednesday, September 29, 2021 8:59 PM > > > > On Wed, Sep 29, 2021 at 12:38:35AM +, Tian, Kevin wrote: > > > > > /* If set the

Re: [RFC 10/20] iommu/iommufd: Add IOMMU_DEVICE_GET_INFO

2021-10-14 Thread Jason Gunthorpe via iommu
On Thu, Oct 14, 2021 at 09:11:58AM +, Tian, Kevin wrote: > But in both cases cache maintenance instructions are available from > guest p.o.v and no coherency semantics would be violated. You've described how Intel's solution papers over the problem. In part wbinvd is defined to restore CPU

Re: [RFC 11/20] iommu/iommufd: Add IOMMU_IOASID_ALLOC/FREE

2021-10-14 Thread Jason Gunthorpe via iommu
On Thu, Oct 14, 2021 at 03:33:21PM +1100, da...@gibson.dropbear.id.au wrote: > > If the HW can attach multiple non-overlapping IOAS's to the same > > device then the HW is routing to the correct IOAS by using the address > > bits. This is not much different from the prior discussion we had > >

Re: [RFC 11/20] iommu/iommufd: Add IOMMU_IOASID_ALLOC/FREE

2021-10-14 Thread Jason Gunthorpe via iommu
On Thu, Oct 14, 2021 at 03:53:33PM +1100, David Gibson wrote: > > My feeling is that qemu should be dealing with the host != target > > case, not the kernel. > > > > The kernel's job should be to expose the IOMMU HW it has, with all > > features accessible, to userspace. > > See... to me this

Re: [RFC 11/20] iommu/iommufd: Add IOMMU_IOASID_ALLOC/FREE

2021-10-11 Thread Jason Gunthorpe via iommu
On Mon, Oct 11, 2021 at 09:49:57AM +0100, Jean-Philippe Brucker wrote: > Seems like we don't need the negotiation part? The host kernel > communicates available IOVA ranges to userspace including holes (patch > 17), and userspace can check that the ranges it needs are within the IOVA > space

Re: [RFC 11/20] iommu/iommufd: Add IOMMU_IOASID_ALLOC/FREE

2021-10-11 Thread Jason Gunthorpe via iommu
On Mon, Oct 11, 2021 at 05:02:01PM +1100, David Gibson wrote: > > This means we cannot define an input that has a magic HW specific > > value. > > I'm not entirely sure what you mean by that. I mean if you make a general property 'foo' that userspace must specify correctly then your API isn't

Re: [RFC 11/20] iommu/iommufd: Add IOMMU_IOASID_ALLOC/FREE

2021-10-11 Thread Jason Gunthorpe via iommu
On Mon, Oct 11, 2021 at 04:37:38PM +1100, da...@gibson.dropbear.id.au wrote: > > PASID support will already require that a device can be multi-bound to > > many IOAS's, couldn't PPC do the same with the windows? > > I don't see how that would make sense. The device has no awareness of > multiple

Re: [RFC 0/7] Support in-kernel DMA with PASID and SVA

2021-10-07 Thread Jason Gunthorpe via iommu
On Thu, Oct 07, 2021 at 12:11:27PM -0700, Jacob Pan wrote: > Hi Barry, > > On Thu, 7 Oct 2021 18:43:33 +1300, Barry Song <21cn...@gmail.com> wrote: > > > > > Security-wise, KVA respects kernel mapping. So permissions are better > > > > enforced than pass-through and identity mapping. > > > > >

Re: [RFC 0/7] Support in-kernel DMA with PASID and SVA

2021-10-07 Thread Jason Gunthorpe via iommu
On Thu, Oct 07, 2021 at 10:50:10AM -0700, Jacob Pan wrote: > On platforms that are DMA snooped, this barrier is not needed. But I think > your point is that once we convert to DMA API, the sync/barrier is covered > by DMA APIs if !dev_is_dma_coherent(dev). Then all archs are good. No.. my point

Re: [RFC 0/7] Support in-kernel DMA with PASID and SVA

2021-10-07 Thread Jason Gunthorpe via iommu
On Fri, Oct 08, 2021 at 12:54:52AM +1300, Barry Song wrote: > On Fri, Oct 8, 2021 at 12:32 AM Jason Gunthorpe wrote: > > > > On Thu, Oct 07, 2021 at 06:43:33PM +1300, Barry Song wrote: > > > > > So do we have a case where devices can directly access the

Re: [RFC 07/20] iommu/iommufd: Add iommufd_[un]bind_device()

2021-10-07 Thread Jason Gunthorpe via iommu
On Thu, Oct 07, 2021 at 12:23:13PM +1100, David Gibson wrote: > On Fri, Oct 01, 2021 at 09:43:22AM -0300, Jason Gunthorpe wrote: > > On Thu, Sep 30, 2021 at 01:10:29PM +1000, David Gibson wrote: > > > On Wed, Sep 29, 2021 at 09:24:57AM -0300, Jason Gunthorpe wrote: > >

Re: [RFC 0/7] Support in-kernel DMA with PASID and SVA

2021-10-07 Thread Jason Gunthorpe via iommu
On Thu, Oct 07, 2021 at 06:43:33PM +1300, Barry Song wrote: > So do we have a case where devices can directly access the kernel's data > structure such as a list/graph/tree with pointers to a kernel virtual address? > then devices don't need to translate the address of pointers in a structure. >

Re: [RFC 0/7] Support in-kernel DMA with PASID and SVA

2021-10-04 Thread Jason Gunthorpe via iommu
On Mon, Oct 04, 2021 at 09:40:03AM -0700, Jacob Pan wrote: > Hi Barry, > > On Sat, 2 Oct 2021 01:45:59 +1300, Barry Song <21cn...@gmail.com> wrote: > > > > > > > > I assume KVA mode can avoid this iotlb flush as the device is using > > > > the page table of the kernel and sharing the whole

Re: [PATCH v3 19/20] PCI/P2PDMA: introduce pci_mmap_p2pmem()

2021-10-04 Thread Jason Gunthorpe
On Mon, Oct 04, 2021 at 03:22:22PM +0200, Christian König wrote: > That use case is completely unrelated to GUP and when this doesn't work we > have quite a problem. My read is that unmap_mapping_range() guarentees the physical TLB hardware is serialized across all CPUs upon return. It also

Re: [PATCH v3 19/20] PCI/P2PDMA: introduce pci_mmap_p2pmem()

2021-10-04 Thread Jason Gunthorpe
On Mon, Oct 04, 2021 at 08:58:35AM +0200, Christian König wrote: > I'm not following this discussion to closely, but try to look into it from > time to time. > > Am 01.10.21 um 19:45 schrieb Jason Gunthorpe: > > On Fri, Oct 01, 2021 at 11:01:49AM -0600, Logan Gunthorpe wrote: &g

Re: [RFC 11/20] iommu/iommufd: Add IOMMU_IOASID_ALLOC/FREE

2021-10-02 Thread Jason Gunthorpe via iommu
On Sat, Oct 02, 2021 at 02:21:38PM +1000, da...@gibson.dropbear.id.au wrote: > > > No. qemu needs to supply *both* the 32-bit and 64-bit range to its > > > guest, and therefore needs to request both from the host. > > > > As I understood your remarks each IOAS can only be one of the formats > >

Re: [PATCH v3 19/20] PCI/P2PDMA: introduce pci_mmap_p2pmem()

2021-10-01 Thread Jason Gunthorpe
On Fri, Oct 01, 2021 at 04:22:28PM -0600, Logan Gunthorpe wrote: > > It would close this issue, however synchronize_rcu() is very slow > > (think > 1second) in some cases and thus cannot be inserted here. > > It shouldn't be *that* slow, at least not the vast majority of the > time... it seems a

Re: [PATCH v3 19/20] PCI/P2PDMA: introduce pci_mmap_p2pmem()

2021-10-01 Thread Jason Gunthorpe
On Fri, Oct 01, 2021 at 02:13:14PM -0600, Logan Gunthorpe wrote: > > > On 2021-10-01 11:45 a.m., Jason Gunthorpe wrote: > >> Before the invalidation, an active flag is cleared to ensure no new > >> mappings can be created while the unmap is proceeding. > >> u

Re: [PATCH v3 19/20] PCI/P2PDMA: introduce pci_mmap_p2pmem()

2021-10-01 Thread Jason Gunthorpe
On Fri, Oct 01, 2021 at 11:01:49AM -0600, Logan Gunthorpe wrote: > In device-dax, the refcount is only used to prevent the device, and > therefore the pages, from going away on device unbind. Pages cannot be > recycled, as you say, as they are mapped linearly within the device. The > address

Re: [PATCH v3 19/20] PCI/P2PDMA: introduce pci_mmap_p2pmem()

2021-10-01 Thread Jason Gunthorpe
On Wed, Sep 29, 2021 at 09:36:52PM -0300, Jason Gunthorpe wrote: > Why would DAX want to do this in the first place?? This means the > address space zap is much more important that just speeding up > destruction, it is essential for correctness since the PTEs are not > holding refcoun

Re: [RFC 07/20] iommu/iommufd: Add iommufd_[un]bind_device()

2021-10-01 Thread Jason Gunthorpe via iommu
On Thu, Sep 30, 2021 at 01:10:29PM +1000, David Gibson wrote: > On Wed, Sep 29, 2021 at 09:24:57AM -0300, Jason Gunthorpe wrote: > 65;6402;1c> On Wed, Sep 29, 2021 at 03:25:54PM +1000, David Gibson wrote: > > > > > > +struct iommufd_device { > > > > +

Re: [RFC 0/7] Support in-kernel DMA with PASID and SVA

2021-10-01 Thread Jason Gunthorpe via iommu
On Sat, Oct 02, 2021 at 01:24:54AM +1300, Barry Song wrote: > I assume KVA mode can avoid this iotlb flush as the device is using > the page table of the kernel and sharing the whole kernel space. But > will users be glad to accept this mode? You can avoid the lock be identity mapping the

Re: [RFC 11/20] iommu/iommufd: Add IOMMU_IOASID_ALLOC/FREE

2021-10-01 Thread Jason Gunthorpe via iommu
On Fri, Oct 01, 2021 at 04:19:22PM +1000, da...@gibson.dropbear.id.au wrote: > On Wed, Sep 22, 2021 at 11:09:11AM -0300, Jason Gunthorpe wrote: > > On Wed, Sep 22, 2021 at 03:40:25AM +, Tian, Kevin wrote: > > > > From: Jason Gunthorpe > > > > Sent: Wedn

Re: [RFC 11/20] iommu/iommufd: Add IOMMU_IOASID_ALLOC/FREE

2021-10-01 Thread Jason Gunthorpe via iommu
On Fri, Oct 01, 2021 at 04:13:58PM +1000, David Gibson wrote: > On Tue, Sep 21, 2021 at 02:44:38PM -0300, Jason Gunthorpe wrote: > > On Sun, Sep 19, 2021 at 02:38:39PM +0800, Liu Yi L wrote: > > > This patch adds IOASID allocation/free interface per iommufd. When > >

Re: [RFC 06/20] iommu: Add iommu_device_init[exit]_user_dma interfaces

2021-09-30 Thread Jason Gunthorpe via iommu
On Thu, Sep 30, 2021 at 01:09:22PM +1000, David Gibson wrote: > > The *admin* the one responsible to understand the groups, not the > > applications. The admin has no idea what a group FD is - they should > > be looking at the sysfs and seeing the iommu_group directories. > > Not just the admin.

Re: [RFC 10/20] iommu/iommufd: Add IOMMU_DEVICE_GET_INFO

2021-09-30 Thread Jason Gunthorpe via iommu
On Thu, Sep 30, 2021 at 09:35:45AM +, Tian, Kevin wrote: > > The Intel functional issue is that Intel blocks the cache maintaince > > ops from the VM and the VM has no way to self-discover that the cache > > maintaince ops don't work. > > the VM doesn't need to know whether the maintenance

Re: [RFC 10/20] iommu/iommufd: Add IOMMU_DEVICE_GET_INFO

2021-09-30 Thread Jason Gunthorpe via iommu
On Thu, Sep 30, 2021 at 08:49:03AM +, Tian, Kevin wrote: > > From: Jason Gunthorpe > > Sent: Thursday, September 23, 2021 8:22 PM > > > > > > These are different things and need different bits. Since the ARM path > > > > has a lot more code supp

Re: [PATCH v3 19/20] PCI/P2PDMA: introduce pci_mmap_p2pmem()

2021-09-29 Thread Jason Gunthorpe
On Wed, Sep 29, 2021 at 05:49:36PM -0600, Logan Gunthorpe wrote: > Some of this seems out of date. Pretty sure the pages are not refcounted > with vmf_insert_mixed() and vmf_insert_mixed() is currently the only way > to use VM_MIXEDMAP mappings. Hum. vmf_insert_mixed() boils down to

Re: [RFC 0/7] Support in-kernel DMA with PASID and SVA

2021-09-29 Thread Jason Gunthorpe via iommu
On Wed, Sep 29, 2021 at 03:57:20PM -0700, Jacob Pan wrote: > Hi Jason, > > On Wed, 29 Sep 2021 16:39:53 -0300, Jason Gunthorpe wrote: > > > On Wed, Sep 29, 2021 at 12:37:19PM -0700, Jacob Pan wrote: > > > > > For #2, it seems we can store

Re: [PATCH v3 4/20] PCI/P2PDMA: introduce helpers for dma_map_sg implementations

2021-09-29 Thread Jason Gunthorpe via iommu
On Wed, Sep 29, 2021 at 05:00:43PM -0600, Logan Gunthorpe wrote: > > > > On 2021-09-29 4:46 p.m., Jason Gunthorpe wrote: > > On Wed, Sep 29, 2021 at 03:30:42PM -0600, Logan Gunthorpe wrote: > >> On 2021-09-28 4:05 p.m., Jason Gunthorpe wrote: > >> No, that

Re: [PATCH v3 00/20] Userspace P2PDMA with O_DIRECT NVMe devices

2021-09-29 Thread Jason Gunthorpe
On Wed, Sep 29, 2021 at 05:28:38PM -0600, Logan Gunthorpe wrote: > > > On 2021-09-29 5:21 p.m., Jason Gunthorpe wrote: > > On Wed, Sep 29, 2021 at 03:50:02PM -0600, Logan Gunthorpe wrote: > >> > >> > >> On 2021-09-28 2:02 p.m., Jason Gunthorpe wrote

Re: [PATCH v3 19/20] PCI/P2PDMA: introduce pci_mmap_p2pmem()

2021-09-29 Thread Jason Gunthorpe
On Wed, Sep 29, 2021 at 05:27:22PM -0600, Logan Gunthorpe wrote: > > finish_fault() should set the pte_devmap - eg by passing the > > PFN_DEV|PFN_MAP somehow through the vma->vm_page_prot to mk_pte() or > > otherwise signaling do_set_pte() that it should set those PTE bits > > when it creates the

Re: [PATCH v3 00/20] Userspace P2PDMA with O_DIRECT NVMe devices

2021-09-29 Thread Jason Gunthorpe
On Wed, Sep 29, 2021 at 03:50:02PM -0600, Logan Gunthorpe wrote: > > > On 2021-09-28 2:02 p.m., Jason Gunthorpe wrote: > > On Thu, Sep 16, 2021 at 05:40:40PM -0600, Logan Gunthorpe wrote: > >> Hi, > >> > >> This patchset continues my work to add usersp

Re: [PATCH v3 19/20] PCI/P2PDMA: introduce pci_mmap_p2pmem()

2021-09-29 Thread Jason Gunthorpe
On Wed, Sep 29, 2021 at 03:42:00PM -0600, Logan Gunthorpe wrote: > The main reason is probably this: if we don't use VM_MIXEDMAP, then we > can't set pte_devmap(). I think that is an API limitation in the fault routines.. finish_fault() should set the pte_devmap - eg by passing the

Re: [PATCH v3 14/20] mm: introduce FOLL_PCI_P2PDMA to gate getting PCI P2PDMA pages

2021-09-29 Thread Jason Gunthorpe
On Wed, Sep 29, 2021 at 03:34:22PM -0600, Logan Gunthorpe wrote: > > > > On 2021-09-28 1:47 p.m., Jason Gunthorpe wrote: > > On Thu, Sep 16, 2021 at 05:40:54PM -0600, Logan Gunthorpe wrote: > >> Callers that expect PCI P2PDMA pages can now set FOLL_PCI_P2PDMA to

Re: [PATCH v3 4/20] PCI/P2PDMA: introduce helpers for dma_map_sg implementations

2021-09-29 Thread Jason Gunthorpe via iommu
On Wed, Sep 29, 2021 at 03:30:42PM -0600, Logan Gunthorpe wrote: > > > > On 2021-09-28 4:05 p.m., Jason Gunthorpe wrote: > > On Thu, Sep 16, 2021 at 05:40:44PM -0600, Logan Gunthorpe wrote: > > > >> +enum pci_p2pdma_map_type > >> +pci_p2pdma_map_

Re: [RFC 0/7] Support in-kernel DMA with PASID and SVA

2021-09-29 Thread Jason Gunthorpe via iommu
On Wed, Sep 29, 2021 at 12:37:19PM -0700, Jacob Pan wrote: > For #2, it seems we can store the kernel PASID in struct device. This will > preserve the DMA API interface while making it PASID capable. Essentially, > each PASID capable device would have two special global PASIDs: > - PASID

Re: [RFC 06/20] iommu: Add iommu_device_init[exit]_user_dma interfaces

2021-09-29 Thread Jason Gunthorpe via iommu
On Wed, Sep 29, 2021 at 12:38:35AM +, Tian, Kevin wrote: > /* If set the driver must call iommu_XX as the first action in probe() or > * before it attempts to do DMA > */ > bool suppress_dma_owner:1; It is not "attempts to do DMA" but more "operates the physical device in any away" Not

Re: [RFC 06/20] iommu: Add iommu_device_init[exit]_user_dma interfaces

2021-09-29 Thread Jason Gunthorpe via iommu
On Wed, Sep 29, 2021 at 04:35:19PM +1000, David Gibson wrote: > Yes, exactly. And with a group interface it's obvious it has to > understand it. With the non-group interface, you can get to this > stage in ignorance of groups. It will even work as long as you are > lucky enough only to try

Re: [RFC 10/20] iommu/iommufd: Add IOMMU_DEVICE_GET_INFO

2021-09-29 Thread Jason Gunthorpe via iommu
On Wed, Sep 29, 2021 at 08:48:28AM +, Tian, Kevin wrote: > ARM: > - set to snoop format if IOMMU_CACHE > - set to nonsnoop format if !IOMMU_CACHE > (in both cases TLP snoop bit is ignored?) Where do you see this? I couldn't even find this functionality in the ARM HW manual?? What I

Re: [RFC 17/20] iommu/iommufd: Report iova range to userspace

2021-09-29 Thread Jason Gunthorpe via iommu
On Wed, Sep 29, 2021 at 01:07:56PM +0100, Jean-Philippe Brucker wrote: > Yes, so the guest knows the size of GPA it can write into the page table. > For Arm SMMU the GPA size is determined by both the SMMU implementation > and the host kernel configuration. But maybe that could also be >

Re: [RFC 08/20] vfio/pci: Add VFIO_DEVICE_BIND_IOMMUFD

2021-09-29 Thread Jason Gunthorpe via iommu
On Wed, Sep 29, 2021 at 06:41:00AM +, Tian, Kevin wrote: > > From: David Gibson > > Sent: Wednesday, September 29, 2021 2:01 PM > > > > On Sun, Sep 19, 2021 at 02:38:36PM +0800, Liu Yi L wrote: > > > This patch adds VFIO_DEVICE_BIND_IOMMUFD for userspace to bind the > > vfio > > > device to

Re: [RFC 07/20] iommu/iommufd: Add iommufd_[un]bind_device()

2021-09-29 Thread Jason Gunthorpe via iommu
On Wed, Sep 29, 2021 at 03:25:54PM +1000, David Gibson wrote: > > +struct iommufd_device { > > + unsigned int id; > > + struct iommufd_ctx *ictx; > > + struct device *dev; /* always be the physical device */ > > + u64 dev_cookie; > > Why do you need both an 'id' and a 'dev_cookie'?

Re: [RFC 03/20] vfio: Add vfio_[un]register_device()

2021-09-29 Thread Jason Gunthorpe via iommu
On Wed, Sep 29, 2021 at 12:46:14PM +1000, da...@gibson.dropbear.id.au wrote: > On Tue, Sep 21, 2021 at 10:00:14PM -0300, Jason Gunthorpe wrote: > > On Wed, Sep 22, 2021 at 12:54:02AM +, Tian, Kevin wrote: > > > > From: Jason Gunthorpe > > > > Sent: Wedne

Re: [RFC 03/20] vfio: Add vfio_[un]register_device()

2021-09-29 Thread Jason Gunthorpe via iommu
On Wed, Sep 29, 2021 at 09:08:25AM +0200, Cornelia Huck wrote: > On Wed, Sep 29 2021, "Tian, Kevin" wrote: > > >> From: David Gibson > >> Sent: Wednesday, September 29, 2021 10:44 AM > >> > >> > One alternative option is to arrange device nodes in sub-directories > >> > based > >> > on the

Re: [PATCH v3 4/20] PCI/P2PDMA: introduce helpers for dma_map_sg implementations

2021-09-28 Thread Jason Gunthorpe via iommu
On Thu, Sep 16, 2021 at 05:40:44PM -0600, Logan Gunthorpe wrote: > +enum pci_p2pdma_map_type > +pci_p2pdma_map_segment(struct pci_p2pdma_map_state *state, struct device > *dev, > +struct scatterlist *sg) > +{ > + if (state->pgmap != sg_page(sg)->pgmap) { > +

Re: [PATCH v3 19/20] PCI/P2PDMA: introduce pci_mmap_p2pmem()

2021-09-28 Thread Jason Gunthorpe
On Thu, Sep 16, 2021 at 05:40:59PM -0600, Logan Gunthorpe wrote: > +static void pci_p2pdma_unmap_mappings(void *data) > +{ > + struct pci_dev *pdev = data; > + struct pci_p2pdma *p2pdma = rcu_dereference_protected(pdev->p2pdma, 1); > + > + p2pdma->active = false; > +

Re: [PATCH v3 00/20] Userspace P2PDMA with O_DIRECT NVMe devices

2021-09-28 Thread Jason Gunthorpe
On Thu, Sep 16, 2021 at 05:40:40PM -0600, Logan Gunthorpe wrote: > Hi, > > This patchset continues my work to add userspace P2PDMA access using > O_DIRECT NVMe devices. My last posting[1] just included the first 13 > patches in this series, but the early P2PDMA cleanup and map_sg error > changes

Re: [PATCH v3 19/20] PCI/P2PDMA: introduce pci_mmap_p2pmem()

2021-09-28 Thread Jason Gunthorpe
On Thu, Sep 16, 2021 at 05:40:59PM -0600, Logan Gunthorpe wrote: > +int pci_mmap_p2pmem(struct pci_dev *pdev, struct vm_area_struct *vma) > +{ > + struct pci_p2pdma_map *pmap; > + struct pci_p2pdma *p2pdma; > + int ret; > + > + /* prevent private mappings from being established */

Re: [PATCH v3 14/20] mm: introduce FOLL_PCI_P2PDMA to gate getting PCI P2PDMA pages

2021-09-28 Thread Jason Gunthorpe
On Thu, Sep 16, 2021 at 05:40:54PM -0600, Logan Gunthorpe wrote: > Callers that expect PCI P2PDMA pages can now set FOLL_PCI_P2PDMA to > allow obtaining P2PDMA pages. If a caller does not set this flag > and tries to map P2PDMA pages it will fail. > > This is implemented by adding a flag and a

Re: [PATCH v3 13/20] PCI/P2PDMA: remove pci_p2pdma_[un]map_sg()

2021-09-28 Thread Jason Gunthorpe
2pdma.c | 65 -- > include/linux/pci-p2pdma.h | 27 > 2 files changed, 92 deletions(-) Reviewed-by: Jason Gunthorpe Good riddance :) Jason ___ iommu mailing list iommu@lists.li

Re: [PATCH v3 12/20] RDMA/rw: use dma_map_sgtable()

2021-09-28 Thread Jason Gunthorpe
atch to do it. But as it is, this looks fine Reviewed-by: Jason Gunthorpe Jason ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v3 11/20] RDMA/core: introduce ib_dma_pci_p2p_dma_supported()

2021-09-28 Thread Jason Gunthorpe
arget/rdma.c | 2 +- > include/rdma/ib_verbs.h| 11 +++ > 2 files changed, 12 insertions(+), 1 deletion(-) Reviewed-by: Jason Gunthorpe > +/* > + * Check if a IB device's underlying DMA mapping supports P2PDMA transfers. > + */ > +static inline b

Re: [PATCH v3 08/20] iommu/dma: support PCI P2PDMA pages in dma-iommu map_sg

2021-09-28 Thread Jason Gunthorpe
t; > Signed-off-by: Logan Gunthorpe > --- > drivers/iommu/dma-iommu.c | 68 +++---- > 1 file changed, 61 insertions(+), 7 deletions(-) Reviewed-by: Jason Gunthorpe Jason ___ iommu mailing list iommu@lists

Re: [PATCH v3 07/20] dma-mapping: add flags to dma_map_ops to indicate PCI P2PDMA support

2021-09-28 Thread Jason Gunthorpe
> include/linux/dma-map-ops.h | 10 ++ > include/linux/dma-mapping.h | 5 + > kernel/dma/mapping.c| 18 ++ > 3 files changed, 33 insertions(+) Reviewed-by: Jason Gunthorpe Jason ___ iommu mailing list

Re: [PATCH v3 06/20] dma-direct: support PCI P2PDMA pages in dma-direct map_sg

2021-09-28 Thread Jason Gunthorpe
On Thu, Sep 16, 2021 at 05:40:46PM -0600, Logan Gunthorpe wrote: > Add PCI P2PDMA support for dma_direct_map_sg() so that it can map > PCI P2PDMA pages directly without a hack in the callers. This allows > for heterogeneous SGLs that contain both P2PDMA and regular pages. > > A P2PDMA page may

Re: [PATCH v3 05/20] dma-mapping: allow EREMOTEIO return code for P2PDMA transfers

2021-09-28 Thread Jason Gunthorpe
--- > kernel/dma/mapping.c | 16 +--- > 1 file changed, 9 insertions(+), 7 deletions(-) Reviewed-by: Jason Gunthorpe Jason ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v3 04/20] PCI/P2PDMA: introduce helpers for dma_map_sg implementations

2021-09-28 Thread Jason Gunthorpe
On Thu, Sep 16, 2021 at 05:40:44PM -0600, Logan Gunthorpe wrote: > Add pci_p2pdma_map_segment() as a helper for simple dma_map_sg() > implementations. It takes an scatterlist segment that must point to a > pci_p2pdma struct page and will map it if the mapping requires a bus > address. > > The

Re: [PATCH v3 03/20] PCI/P2PDMA: make pci_p2pdma_map_type() non-static

2021-09-28 Thread Jason Gunthorpe
uld be used to program the DMA engine. > + */ > + PCI_P2PDMA_MAP_THRU_HOST_BRIDGE, > +}; Reviewed-by: Jason Gunthorpe Jason ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v3 01/20] lib/scatterlist: add flag for indicating P2PDMA segments in an SGL

2021-09-28 Thread Jason Gunthorpe
ssign a given page to an SG entry > @@ -86,13 +103,13 @@ struct sg_append_table { > **/ > static inline void sg_assign_page(struct scatterlist *sg, struct page *page) > { > - unsigned long page_link = sg->page_link & (SG_CHAIN | SG_END); > +

Re: [RFC 06/20] iommu: Add iommu_device_init[exit]_user_dma interfaces

2021-09-28 Thread Jason Gunthorpe via iommu
On Tue, Sep 28, 2021 at 09:35:05PM +0800, Lu Baolu wrote: > Another issue is, when putting a device into user-dma mode, all devices > belonging to the same iommu group shouldn't be bound with a kernel-dma > driver. Kevin's prototype checks this by READ_ONCE(dev->driver). This is > not lock safe as

Re: [RFC 06/20] iommu: Add iommu_device_init[exit]_user_dma interfaces

2021-09-28 Thread Jason Gunthorpe via iommu
On Tue, Sep 28, 2021 at 07:30:41AM +, Tian, Kevin wrote: > > Also, don't call it "hint", there is nothing hinty about this, it has > > definitive functional impacts. > > possibly dma_mode (too broad?) or dma_usage You just need a flag to specify if the driver manages DMA ownership itself,

Re: [RFC 06/20] iommu: Add iommu_device_init[exit]_user_dma interfaces

2021-09-27 Thread Jason Gunthorpe via iommu
On Mon, Sep 27, 2021 at 09:42:58AM +, Tian, Kevin wrote: > +static int iommu_dev_viable(struct device *dev, void *data) > +{ > + enum dma_hint hint = *data; > + struct device_driver *drv = READ_ONCE(dev->driver); > + > + /* no conflict if the new device doesn't do DMA */ > +

Re: [RFC 06/20] iommu: Add iommu_device_init[exit]_user_dma interfaces

2021-09-27 Thread Jason Gunthorpe via iommu
On Mon, Sep 27, 2021 at 01:00:08PM +, Tian, Kevin wrote: > > I think for such a narrow usage you should not change the struct > > device_driver. Just have pci_stub call a function to flip back to user > > mode. > > Here we want to ensure that kernel dma should be blocked > if the group is

Re: [RFC 06/20] iommu: Add iommu_device_init[exit]_user_dma interfaces

2021-09-27 Thread Jason Gunthorpe via iommu
On Mon, Sep 27, 2021 at 09:42:58AM +, Tian, Kevin wrote: > > From: Jason Gunthorpe > > Sent: Wednesday, September 22, 2021 8:40 PM > > > > > > Ie the basic flow would see the driver core doing some: > > > > > > Just double confirm. Is there

Re: [RFC 11/20] iommu/iommufd: Add IOMMU_IOASID_ALLOC/FREE

2021-09-23 Thread Jason Gunthorpe via iommu
On Thu, Sep 23, 2021 at 01:20:55PM +, Tian, Kevin wrote: > > > this is not a flow for mdev. It's also required for pdev on Intel > > > platform, > > > because the pasid table is in HPA space thus must be managed by host > > > kernel. Even no translation we still need the user to provide the

Re: [RFC 11/20] iommu/iommufd: Add IOMMU_IOASID_ALLOC/FREE

2021-09-23 Thread Jason Gunthorpe via iommu
On Thu, Sep 23, 2021 at 12:45:17PM +, Tian, Kevin wrote: > > From: Jason Gunthorpe > > Sent: Thursday, September 23, 2021 8:31 PM > > > > On Thu, Sep 23, 2021 at 12:22:23PM +, Tian, Kevin wrote: > > > > From: Jason Gunthorpe > > >

Re: [RFC 11/20] iommu/iommufd: Add IOMMU_IOASID_ALLOC/FREE

2021-09-23 Thread Jason Gunthorpe via iommu
On Thu, Sep 23, 2021 at 12:22:23PM +, Tian, Kevin wrote: > > From: Jason Gunthorpe > > Sent: Thursday, September 23, 2021 8:07 PM > > > > On Thu, Sep 23, 2021 at 09:14:58AM +, Tian, Kevin wrote: > > > > > currently the type is aimed to different

Re: [RFC 10/20] iommu/iommufd: Add IOMMU_DEVICE_GET_INFO

2021-09-23 Thread Jason Gunthorpe via iommu
On Thu, Sep 23, 2021 at 12:05:29PM +, Tian, Kevin wrote: > > From: Jason Gunthorpe > > Sent: Thursday, September 23, 2021 7:27 PM > > > > On Thu, Sep 23, 2021 at 11:15:24AM +0100, Jean-Philippe Brucker wrote: > > > > > So we can only tell userspace &

Re: [RFC 11/20] iommu/iommufd: Add IOMMU_IOASID_ALLOC/FREE

2021-09-23 Thread Jason Gunthorpe via iommu
On Thu, Sep 23, 2021 at 09:14:58AM +, Tian, Kevin wrote: > currently the type is aimed to differentiate three usages: > > - kernel-managed I/O page table > - user-managed I/O page table > - shared I/O page table (e.g. with mm, or ept) Creating a shared ios is something that should probably

Re: [RFC 03/20] vfio: Add vfio_[un]register_device()

2021-09-23 Thread Jason Gunthorpe via iommu
On Thu, Sep 23, 2021 at 09:25:27AM +0200, Eric Auger wrote: > Hi, > > On 9/22/21 3:00 AM, Jason Gunthorpe wrote: > > On Wed, Sep 22, 2021 at 12:54:02AM +, Tian, Kevin wrote: > >>> From: Jason Gunthorpe > >>> Sent: Wednesday, September 22, 2021 12

Re: [RFC 10/20] iommu/iommufd: Add IOMMU_DEVICE_GET_INFO

2021-09-23 Thread Jason Gunthorpe via iommu
On Thu, Sep 23, 2021 at 03:38:10AM +, Tian, Kevin wrote: > > From: Tian, Kevin > > Sent: Thursday, September 23, 2021 11:11 AM > > > > > > > > The required behavior for iommufd is to have the IOMMU ignore the > > > no-snoop bit so that Intel HW can disable wbinvd. This bit should be > > >

Re: [RFC 10/20] iommu/iommufd: Add IOMMU_DEVICE_GET_INFO

2021-09-23 Thread Jason Gunthorpe via iommu
On Thu, Sep 23, 2021 at 03:10:47AM +, Tian, Kevin wrote: > Disabling wbinvd is one purpose. imo the more important intention > is that iommu vendor uses different PTE formats between snoop and > !snoop. The PTE format for userspace is communicated through the format input, not through

Re: [RFC 10/20] iommu/iommufd: Add IOMMU_DEVICE_GET_INFO

2021-09-23 Thread Jason Gunthorpe via iommu
On Thu, Sep 23, 2021 at 11:15:24AM +0100, Jean-Philippe Brucker wrote: > So we can only tell userspace "No_snoop is not supported" (provided we > even want to allow them to enable No_snoop). Users in control of stage-1 > tables can create non-cacheable mappings through MAIR attributes. My point

Re: [RFC 03/20] vfio: Add vfio_[un]register_device()

2021-09-22 Thread Jason Gunthorpe via iommu
On Wed, Sep 22, 2021 at 02:10:36PM -0600, Alex Williamson wrote: > But why would we create vfio device interface files at all if they > can't work? I'm not really on board with creating a try-and-fail > interface for a mechanism that cannot work for a given device. The > existence of the device

Re: [RFC 03/20] vfio: Add vfio_[un]register_device()

2021-09-22 Thread Jason Gunthorpe via iommu
On Wed, Sep 22, 2021 at 11:45:33PM +, Tian, Kevin wrote: > > From: Alex Williamson > btw I realized another related piece regarding to the new layout that > Jason suggested, which have sys device node include a link to the vfio > devnode: > >

Re: [RFC 10/20] iommu/iommufd: Add IOMMU_DEVICE_GET_INFO

2021-09-22 Thread Jason Gunthorpe via iommu
On Wed, Sep 22, 2021 at 03:24:07PM -0600, Alex Williamson wrote: > On Sun, 19 Sep 2021 14:38:38 +0800 > Liu Yi L wrote: > > > +struct iommu_device_info { > > + __u32 argsz; > > + __u32 flags; > > +#define IOMMU_DEVICE_INFO_ENFORCE_SNOOP(1 << 0) /* IOMMU enforced > > snoop */ > > Is

Re: [RFC 08/20] vfio/pci: Add VFIO_DEVICE_BIND_IOMMUFD

2021-09-22 Thread Jason Gunthorpe via iommu
On Wed, Sep 22, 2021 at 03:01:01PM -0600, Alex Williamson wrote: > On Tue, 21 Sep 2021 14:29:39 -0300 > Jason Gunthorpe wrote: > > > On Sun, Sep 19, 2021 at 02:38:36PM +0800, Liu Yi L wrote: > > > +struct vfio_device_iommu_bind_data { > > > + __u32 argsz; >

Re: [RFC 0/7] Support in-kernel DMA with PASID and SVA

2021-09-22 Thread Jason Gunthorpe via iommu
On Tue, Sep 21, 2021 at 01:29:34PM -0700, Jacob Pan wrote: > Hi Joerg/Jason/Christoph et all, > > The current in-kernel supervisor PASID support is based on the SVM/SVA > machinery in sva-lib. Kernel SVA is achieved by extending a special flag > to indicate the binding of the device and a page

Re: [RFC 01/20] iommu/iommufd: Add /dev/iommu core

2021-09-22 Thread Jason Gunthorpe via iommu
On Wed, Sep 22, 2021 at 01:59:39PM +, Tian, Kevin wrote: > > From: Jason Gunthorpe > > Sent: Wednesday, September 22, 2021 8:41 PM > > > > On Wed, Sep 22, 2021 at 01:51:03AM +, Tian, Kevin wrote: > > > > From: Jason Gunthorpe > > >

Re: [RFC 11/20] iommu/iommufd: Add IOMMU_IOASID_ALLOC/FREE

2021-09-22 Thread Jason Gunthorpe via iommu
On Wed, Sep 22, 2021 at 03:40:25AM +, Tian, Kevin wrote: > > From: Jason Gunthorpe > > Sent: Wednesday, September 22, 2021 1:45 AM > > > > On Sun, Sep 19, 2021 at 02:38:39PM +0800, Liu Yi L wrote: > > > This patch adds IOASID allocation/free interface p

Re: [RFC 11/20] iommu/iommufd: Add IOMMU_IOASID_ALLOC/FREE

2021-09-22 Thread Jason Gunthorpe via iommu
On Wed, Sep 22, 2021 at 12:51:38PM +, Liu, Yi L wrote: > > From: Jason Gunthorpe > > Sent: Wednesday, September 22, 2021 1:45 AM > > > [...] > > > diff --git a/drivers/iommu/iommufd/iommufd.c > > b/drivers/iommu/iommufd/iommufd.c > > > index

Re: [RFC 15/20] vfio/pci: Add VFIO_DEVICE_[DE]ATTACH_IOASID

2021-09-22 Thread Jason Gunthorpe via iommu
On Wed, Sep 22, 2021 at 03:56:18AM +, Tian, Kevin wrote: > > From: Jason Gunthorpe > > Sent: Wednesday, September 22, 2021 2:04 AM > > > > On Sun, Sep 19, 2021 at 02:38:43PM +0800, Liu Yi L wrote: > > > This patch adds interface for userspace to attach

Re: [RFC 14/20] iommu/iommufd: Add iommufd_device_[de]attach_ioasid()

2021-09-22 Thread Jason Gunthorpe via iommu
On Wed, Sep 22, 2021 at 03:53:52AM +, Tian, Kevin wrote: > Actually this was one open we closed in previous design proposal, but > looks you have a different thought now. > > vfio maintains one ioas per container. Devices in the container > can be attached to different domains (e.g. due to

Re: [RFC 12/20] iommu/iommufd: Add IOMMU_CHECK_EXTENSION

2021-09-22 Thread Jason Gunthorpe via iommu
On Wed, Sep 22, 2021 at 03:41:50AM +, Tian, Kevin wrote: > > From: Jason Gunthorpe > > Sent: Wednesday, September 22, 2021 1:47 AM > > > > On Sun, Sep 19, 2021 at 02:38:40PM +0800, Liu Yi L wrote: > > > As aforementioned, userspace should check ext

Re: [RFC 02/20] vfio: Add device class for /dev/vfio/devices

2021-09-22 Thread Jason Gunthorpe via iommu
On Wed, Sep 22, 2021 at 03:22:42AM +, Tian, Kevin wrote: > > From: Tian, Kevin > > Sent: Wednesday, September 22, 2021 9:07 AM > > > > > From: Jason Gunthorpe > > > Sent: Wednesday, September 22, 2021 8:55 AM > > > > > > On Tu

Re: [RFC 10/20] iommu/iommufd: Add IOMMU_DEVICE_GET_INFO

2021-09-22 Thread Jason Gunthorpe via iommu
On Wed, Sep 22, 2021 at 03:30:09AM +, Tian, Kevin wrote: > > From: Jason Gunthorpe > > Sent: Wednesday, September 22, 2021 1:41 AM > > > > On Sun, Sep 19, 2021 at 02:38:38PM +0800, Liu Yi L wrote: > > > After a device is bound to the iommufd, userspace can

Re: [RFC 01/20] iommu/iommufd: Add /dev/iommu core

2021-09-22 Thread Jason Gunthorpe via iommu
On Wed, Sep 22, 2021 at 01:51:03AM +, Tian, Kevin wrote: > > From: Jason Gunthorpe > > Sent: Tuesday, September 21, 2021 11:42 PM > > > > - Delete the iommufd_ctx->lock. Use RCU to protect load, erase/alloc does > >not need locking (order it prope

  1   2   3   4   5   6   >