Re: [PATCH 02/12] pci-p2p: Add sysfs group to display p2pmem stats

On 04/01/18 02:50 PM, Bjorn Helgaas wrote: On Thu, Jan 04, 2018 at 12:01:27PM -0700, Logan Gunthorpe wrote: Attributes display the total amount of P2P memory, the ammount available and whether it is published or not. s/ammount/amount/ (also below) Will fix. I wonder if "p2pdma" would be

Re: [PATCH 06/12] IB/core: Add optional PCI P2P flag to rdma_rw_ctx_[init|destroy]()

On 04/01/18 03:13 PM, Jason Gunthorpe wrote: On Thu, Jan 04, 2018 at 12:52:24PM -0700, Logan Gunthorpe wrote: We tried things like this in an earlier iteration[1] which assumed the SG was homogenous (all P2P or all regular memory). This required serious ugliness to try and ensure SGs were in

Re: [PATCH 01/12] pci-p2p: Support peer to peer memory

On 04/01/18 02:59 PM, Bjorn Helgaas wrote: On Thu, Jan 04, 2018 at 12:01:26PM -0700, Logan Gunthorpe wrote: Some PCI devices may have memory mapped in a BAR space that's intended for use in Peer-to-Peer transactions. In order to enable such transactions the memory must be registered with

Re: [PATCH 04/12] pci-p2p: Clear ACS P2P flags for all client devices

On 04/01/18 05:00 PM, Logan Gunthorpe wrote: > > > On 04/01/18 03:35 PM, Alex Williamson wrote: >> Yep, flipping these ACS bits invalidates any IOMMU groups that depend >> on the isolation of that downstream port and I suspect also any peers >> within the same PCI slot of that port and their

Re: [PATCH 04/12] pci-p2p: Clear ACS P2P flags for all client devices

On 04/01/18 03:35 PM, Alex Williamson wrote: Yep, flipping these ACS bits invalidates any IOMMU groups that depend on the isolation of that downstream port and I suspect also any peers within the same PCI slot of that port and their downstream devices. The entire sub-hierarchy grouping needs

Re: [PATCH 01/12] pci-p2p: Support peer to peer memory

Thanks for the speedy review! On 04/01/18 02:40 PM, Bjorn Helgaas wrote: Run "git log --oneline drivers/pci" and follow the convention. I think it would make sense to add a new tag like "PCI/P2P", although "P2P" has historically also been used in the "PCI-to-PCI bridge" context, so maybe

Re: [PATCH 02/12] pci-p2p: Add sysfs group to display p2pmem stats

On Thu, Jan 04, 2018 at 12:01:27PM -0700, Logan Gunthorpe wrote: > Attributes display the total amount of P2P memory, the ammount available > and whether it is published or not. s/ammount/amount/ (also below) > Signed-off-by: Logan Gunthorpe > --- >

Re: [PATCH 01/12] pci-p2p: Support peer to peer memory

On Thu, Jan 04, 2018 at 12:01:26PM -0700, Logan Gunthorpe wrote: > Some PCI devices may have memory mapped in a BAR space that's > intended for use in Peer-to-Peer transactions. In order to enable > such transactions the memory must be registered with ZONE_DEVICE pages > so it can be used by DMA

Re: [PATCH 04/12] pci-p2p: Clear ACS P2P flags for all client devices

[+cc Alex] On Thu, Jan 04, 2018 at 12:01:29PM -0700, Logan Gunthorpe wrote: > When the ACS P2P flags are set in the downstream port of the switch, > any P2P TLPs will be sent back to the root complex. The whole point of > the P2P work is to have TLPs avoid the RC seeing it may not support > P2P

Re: [PATCH 04/12] pci-p2p: Clear ACS P2P flags for all client devices

On Thu, 4 Jan 2018 15:57:21 -0600 Bjorn Helgaas wrote: > [+cc Alex] > > On Thu, Jan 04, 2018 at 12:01:29PM -0700, Logan Gunthorpe wrote: > > When the ACS P2P flags are set in the downstream port of the switch, > > any P2P TLPs will be sent back to the root complex. The whole

Re: [PATCH 02/12] pci-p2p: Add sysfs group to display p2pmem stats

On Thu, Jan 04, 2018 at 03:50:40PM -0600, Bjorn Helgaas wrote: > > This is similar to /sys/bus/pci/drivers_autoprobe, but > > affects only the VFs associated with a specific PF. > > + > > +What: /sys/bus/pci/devices/.../p2pmem/available > > I wonder if

[PATCH 00/11] Copy Offload in NVMe Fabrics with P2P PCI Memory

Hello, This is a continuation of our work to enable using Peer-to-Peer PCI memory in NVMe fabrics targets. Many thanks go to Christoph Hellwig who provided valuable feedback to get these patches to where they are today. The concept here is to use memory that's exposed on a PCI BAR as data

[PATCH 12/12] nvmet: Optionally use PCI P2P memory

We create a configfs attribute in each nvme-fabrics target port to enable p2p memory use. When enabled, the port will only then use the p2p memory if a p2p memory device can be found which is behind the same switch as the RDMA port and all the block devices in use. If the user enabled it an no

[PATCH 01/12] pci-p2p: Support peer to peer memory

Some PCI devices may have memory mapped in a BAR space that's intended for use in Peer-to-Peer transactions. In order to enable such transactions the memory must be registered with ZONE_DEVICE pages so it can be used by DMA interfaces in existing drivers. A kernel interface is provided so that

Re: [PATCH 08/12] nvme-pci: clean up SMBSZ bit definitions

On 04/01/18 12:01 PM, Logan Gunthorpe wrote: From: Christoph Hellwig Define the bit positions instead of macros using the magic values, and move the expanded helpers to calculate the size and size unit into the implementation C file. Signed-off-by: Christoph Hellwig

Re: [PATCH 07/12] nvme-pci: clean up CMB initialization

On 04/01/18 12:01 PM, Logan Gunthorpe wrote: From: Christoph Hellwig Refactor the call to nvme_map_cmb, and change the conditions for probing for the CMB. First remove the version check as NVMe TPs always apply to earlier versions of the spec as well. Second check for the

[PATCH 05/12] block: Introduce PCI P2P flags for request and request queue

QUEUE_FLAG_PCI_P2P is introduced meaning a driver's request queue supports targeting P2P memory. REQ_PCI_P2P is introduced to indicate a particular bio request is directed to/from PCI P2P memory. A request with this flag is not accepted unless the corresponding queues have the QUEUE_FLAG_PCI_P2P

[PATCH 04/12] pci-p2p: Clear ACS P2P flags for all client devices

When the ACS P2P flags are set in the downstream port of the switch, any P2P TLPs will be sent back to the root complex. The whole point of the P2P work is to have TLPs avoid the RC seeing it may not support P2P transactions or, at best, it will perform poorly. So we clear these flags for all

[PATCH 03/12] pci-p2p: Add PCI p2pmem dma mappings to adjust the bus offset

The DMA address used when mapping PCI P2P memory must be the PCI bus address. Thus, introduce pci_p2pmem_[un]map_sg() to map the correct addresses when using P2P memory. For this, we assume that an SGL passed to these functions contain all p2p memory or no p2p memory. Signed-off-by: Logan

[PATCH 02/12] pci-p2p: Add sysfs group to display p2pmem stats

Attributes display the total amount of P2P memory, the ammount available and whether it is published or not. Signed-off-by: Logan Gunthorpe --- Documentation/ABI/testing/sysfs-bus-pci | 25 drivers/pci/p2p.c | 51

[PATCH 11/12] nvme-pci: Add a quirk for a pseudo CMB

Introduce a quirk to use CMB-like memory on older devices that have an exposed BAR but do not advertise support for using CMBLOC and CMBSIZE. We'd like to use some of these older cards to test P2P memory. Signed-off-by: Logan Gunthorpe --- drivers/nvme/host/nvme.h | 7

[PATCH 09/12] nvme-pci: Use PCI p2pmem subsystem to manage the CMB

Register the CMB buffer as p2pmem and use the appropriate allocation functions to create and destroy the IO SQ. If the CMB supports WDS and RDS, publish it for use as p2p memory by other devices. Signed-off-by: Logan Gunthorpe --- drivers/nvme/host/pci.c | 74

[PATCH 10/12] nvme-pci: Add support for P2P memory in requests

For P2P requests we must use the pci_p2pmem_[un]map_sg() functions instead of the dma_map_sg functions. With that, we can then indicate PCI_P2P support in the request queue. For this, we create an NVME_F_PCI_P2P flag which tells the core to set QUEUE_FLAG_PCI_P2P in the request queue.

[PATCH 06/12] IB/core: Add optional PCI P2P flag to rdma_rw_ctx_[init|destroy]()

In order to use PCI P2P memory pci_p2pmem_[un]map_sg() functions must be called to map the correct DMA address. To do this, we add a flags variable and the RDMA_RW_CTX_FLAG_PCI_P2P flag. When the flag is specified use the appropriate map function. Signed-off-by: Logan Gunthorpe

[PATCH 08/12] nvme-pci: clean up SMBSZ bit definitions

From: Christoph Hellwig Define the bit positions instead of macros using the magic values, and move the expanded helpers to calculate the size and size unit into the implementation C file. Signed-off-by: Christoph Hellwig --- drivers/nvme/host/pci.c | 23

Re: [PATCH 01/12] pci-p2p: Support peer to peer memory

Run "git log --oneline drivers/pci" and follow the convention. I think it would make sense to add a new tag like "PCI/P2P", although "P2P" has historically also been used in the "PCI-to-PCI bridge" context, so maybe there's something less ambiguous. "P2PDMA"? When you add new files, I guess

Re: [PATCH 06/12] IB/core: Add optional PCI P2P flag to rdma_rw_ctx_[init|destroy]()

On 04/01/18 12:22 PM, Jason Gunthorpe wrote: This seems really clunky since we are going to want to do this same logic all over the place. I'd be much happier if dma_map_sg can tell the memory is P2P or not from the scatterlist or dir arguments and not require the callers to have this. We

Re: [PATCH 04/12] pci-p2p: Clear ACS P2P flags for all client devices

On Thu, Jan 04, 2018 at 08:33:00PM -0700, Alex Williamson wrote: > On Thu, 4 Jan 2018 17:00:47 -0700 > Logan Gunthorpe wrote: > > > On 04/01/18 03:35 PM, Alex Williamson wrote: > > > Yep, flipping these ACS bits invalidates any IOMMU groups that depend > > > on the isolation

Re: [PATCH 04/12] pci-p2p: Clear ACS P2P flags for all client devices

On Thu, 4 Jan 2018 17:00:47 -0700 Logan Gunthorpe wrote: > On 04/01/18 03:35 PM, Alex Williamson wrote: > > Yep, flipping these ACS bits invalidates any IOMMU groups that depend > > on the isolation of that downstream port and I suspect also any peers > > within the same PCI

Re: [PATCH v4 04/18] dax: require 'struct page' by default for filesystem dax

Fine with me: Reviewed-by: Christoph Hellwig ___ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: [PATCH v4 00/18] dax: fix dma vs truncate/hole-punch

> So far this solution only targets xfs since it already implements > xfs_break_layouts in all the locations that would need this > synchronization. It applies on top of the vmem_altmap / dev_pagemap > reworks from Christoph. Those got a rebase since your posting of this series, btw.

Re: [PATCH v4 16/18] wait_bit: introduce {wait_on,wake_up}_atomic_one

Looks fine, Reviewed-by: Christoph Hellwig ___ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: [PATCH v4 14/18] ext2: use DEFINE_FSDAX_AOPS

Same comment as for XFS applies here as well. ___ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: [PATCH v4 15/18] mm, fs, dax: use page->mapping to warn if dma collides with truncate

Looks fine, Reviewed-by: Christoph Hellwig ___ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: [PATCH v4 13/18] ext4: use DEFINE_FSDAX_AOPS

Same comment as for XFS applies here as well. ___ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: [PATCH v4 01/18] mm, dax: introduce pfn_t_special()

On Sat, Dec 23, 2017 at 04:56:06PM -0800, Dan Williams wrote: > In support of removing the VM_MIXEDMAP indication from DAX VMAs, > introduce pfn_t_special() for drivers to indicate that _PAGE_SPECIAL > should be used for DAX ptes. This also helps identify drivers like > dccssblk that only want to

Re: [PATCH v4 10/18] mm, dev_pagemap: introduce CONFIG_DEV_PAGEMAP_OPS

This looks fine except for a few nitpicks below: > } > > + dev_pagemap_enable_ops(); > pgmap->type = MEMORY_DEVICE_FS_DAX; > pgmap->page_free = generic_dax_pagefree; > pgmap->data = owner; > @@ -215,6 +216,7 @@ void fs_dax_release(struct dax_device *dax_dev, void >

Re: [PATCH v4 18/18] xfs, dax: wire up dax_flush_dma support via a new xfs_sync_dma helper

On Wed, Jan 03, 2018 at 10:00:27AM +1100, Dave Chinner wrote: > AFAIC, these DMA references are just another external layout > reference that needs to be broken. IOWs, this "sync DMA" complexity > needs to go inside xfs_break_layouts() as it is part of breaking the > external reference to the

Re: [PATCH v4 18/18] xfs, dax: wire up dax_flush_dma support via a new xfs_sync_dma helper

On Wed, Jan 03, 2018 at 06:51:12PM +1100, Dave Chinner wrote: > > "Additionally we call it during the write operation, where aren't > > concerned about exposing unallocated blocks but just want to provide > > basic synchronization between a local writer and pNFS clients. mmap > > writes would

Re: [PATCH v4 15/18] mm, fs, dax: use page->mapping to warn if dma collides with truncate

On Sat 23-12-17 16:57:20, Dan Williams wrote: > Catch cases where truncate encounters pages that are still under active > dma. This warning is a canary for potential data corruption as truncated > blocks could be allocated to a new file while the device is still > perform i/o. > > Here is an

Re: [PATCH v4 09/18] mm, dax: enable filesystems to trigger dev_pagemap ->page_free callbacks

Looks fine, Reviewed-by: Christoph Hellwig ___ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: [PATCH v6 0/2] dax, dm: stop requiring dax for device-mapper

On Wed, Nov 29 2017 at 1:59pm -0500, Dan Williams wrote: > Changes since v5 [1]: > * Make DAX_DRIVER select DAX to simplify the Kconfig dependencies > (Michael) > * Rebase on 4.15-rc1 and add new IS_ENABLED(CONFIG_DAX_DRIVER) checks in > drivers/md/dm-log-writes.c.

Re: [PATCH v4 17/18] mm, fs, dax: dax_flush_dma, handle dma vs block-map-change collisions

On Sat 23-12-17 16:57:31, Dan Williams wrote: > +static struct page *dma_busy_page(void *entry) > +{ > + unsigned long pfn, end_pfn; > + > + for_each_entry_pfn(entry, pfn, end_pfn) { > + struct page *page = pfn_to_page(pfn); > + > + if (page_ref_count(page) > 1) > +

[PATCH 2/2] ext4: test for inline data + DAX corruption

Add a regression test for the following kernel commit: ext4: prevent data corruption with inline data + DAX The test passes either if we don't encounter corruption, or if mounting with DAX + inline data fails. The latter is the way that we prevent this issue in the kernel. Signed-off-by:

[PATCH 1/2] ext4: test for DAX + journaling corruption

Add a regression test for the following kernel commit: ext4: prevent data corruption with journaling + DAX The test passes if either we successfully compare the data between the mmap with journaling turned on and the one with journaling turned off, or if we fail the chattr command to turn on

Re: [PATCH v4 12/18] xfs: use DEFINE_FSDAX_AOPS

On Sat, Dec 23, 2017 at 04:57:04PM -0800, Dan Williams wrote: > In preparation for the dax implementation to start associating dax pages > to inodes via page->mapping, we need to provide a 'struct > address_space_operations' instance for dax. Otherwise, direct-I/O > triggers incorrect page cache

Re: [PATCH v4 17/18] mm, fs, dax: dax_flush_dma, handle dma vs block-map-change collisions

Not pretty, but probably the best we can do for now.. Reviewed-by: Christoph Hellwig ___ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm