Re: [PATCH v2 00/26] InfiniBand Transport (IBTRS) and Network Block Device (IBNBD)

On Fri, May 18, 2018 at 03:03:47PM +0200, Roman Pen wrote: > Hi all, > > This is v2 of series, which introduces IBNBD/IBTRS modules. > > This cover letter is split on three parts: > > 1. Introduction, which almost repeats everything from previous cover >letters. > 2. Changelog. > 3.

Re: [PATCH v3 01/11] PCI/P2PDMA: Support peer-to-peer memory

On Mon, Mar 26, 2018 at 11:30:38AM -0600, Logan Gunthorpe wrote: > > > On 26/03/18 10:41 AM, Jason Gunthorpe wrote: > > On Mon, Mar 26, 2018 at 12:11:38PM +0100, Jonathan Cameron wrote: > >> On Tue, 13 Mar 2018 10:43:55 -0600 > >> Logan Gunthorpe <log...@delt

Re: [PATCH v3 01/11] PCI/P2PDMA: Support peer-to-peer memory

On Mon, Mar 26, 2018 at 12:11:38PM +0100, Jonathan Cameron wrote: > On Tue, 13 Mar 2018 10:43:55 -0600 > Logan Gunthorpe wrote: > > > On 12/03/18 09:28 PM, Sinan Kaya wrote: > > > On 3/12/2018 3:35 PM, Logan Gunthorpe wrote: > > > Regarding the switch business, It is amazing

Re: [PATCH v2 07/10] nvme-pci: Use PCI p2pmem subsystem to manage the CMB

On Mon, Mar 05, 2018 at 01:42:12PM -0700, Keith Busch wrote: > On Mon, Mar 05, 2018 at 01:10:53PM -0700, Jason Gunthorpe wrote: > > So when reading the above mlx code, we see the first wmb() being used > > to ensure that CPU stores to cachable memory are visible to the DM

Re: [PATCH v2 07/10] nvme-pci: Use PCI p2pmem subsystem to manage the CMB

On Mon, Mar 05, 2018 at 09:57:27PM +0200, Sagi Grimberg wrote: > Keith, while we're on this, regardless of cmb, is SQE memcopy and DB update > ordering always guaranteed? > > If you look at mlx4 (rdma device driver) that works exactly the same as > nvme you will find: > -- >

Re: [PATCH v2 10/10] nvmet: Optionally use PCI P2P memory

On Thu, Mar 01, 2018 at 11:57:43PM +, Stephen Bates wrote: > > We don't want to lump these all together without knowing which region > > you're allocating from, right? > > In all seriousness I do agree with you on these Keith in the long > term. We would consider adding property flags for

Re: [PATCH v2 10/10] nvmet: Optionally use PCI P2P memory

On Thu, Mar 01, 2018 at 11:00:51PM +, Stephen Bates wrote: > > Seems like a very subtle and hard to debug performance trap to leave > > for the users, and pretty much the only reason to use P2P is > > performance... So why have such a dangerous interface? > > P2P is about offloading the

Re: [PATCH v2 10/10] nvmet: Optionally use PCI P2P memory

On Thu, Mar 01, 2018 at 12:27:03PM -0700, Logan Gunthorpe wrote: > > > On 01/03/18 11:42 AM, Jason Gunthorpe wrote: > >On Thu, Mar 01, 2018 at 08:35:55PM +0200, Sagi Grimberg wrote: > >This is also why I don't entirely understand why this series has a > >generic allo

Re: [PATCH v2 00/10] Copy Offload in NVMe Fabrics with P2P PCI Memory

On Fri, Mar 02, 2018 at 07:40:15AM +1100, Benjamin Herrenschmidt wrote: > Also we need to be able to hard block MEMREMAP_WB mappings of non-RAM > on ppc64 (maybe via an arch hook as it might depend on the processor > family). Server powerpc cannot do cachable accesses on IO memory > (unless it's

Re: [PATCH v2 10/10] nvmet: Optionally use PCI P2P memory

On Thu, Mar 01, 2018 at 08:35:55PM +0200, Sagi Grimberg wrote: > > >On 01/03/18 04:03 AM, Sagi Grimberg wrote: > >>Can you describe what would be the plan to have it when these devices > >>do come along? I'd say that p2p_dev needs to become a nvmet_ns reference > >>and not from nvmet_ctrl. Then,

Re: [PATCH 03/24] ibtrs: core: lib functions shared between client and server modules

On Tue, Feb 06, 2018 at 01:01:23PM +0100, Roman Penyaev wrote: > >> +static int ibtrs_ib_dev_init(struct ibtrs_ib_dev *d, struct ib_device > >> *dev) > >> +{ > >> + int err; > >> + > >> + d->pd = ib_alloc_pd(dev, IB_PD_UNSAFE_GLOBAL_RKEY); > >> + if (IS_ERR(d->pd)) > >> +

Re: [PATCH 06/12] IB/core: Add optional PCI P2P flag to rdma_rw_ctx_[init|destroy]()

On Tue, Jan 09, 2018 at 05:46:40PM +0100, Christoph Hellwig wrote: > On Mon, Jan 08, 2018 at 12:49:50PM -0700, Jason Gunthorpe wrote: > > Pretty sure P2P capable IOMMU hardware exists. > > > > With SOC's we also have the scenario that an DMA originated from an > > on-

Re: [PATCH 06/12] IB/core: Add optional PCI P2P flag to rdma_rw_ctx_[init|destroy]()

On Mon, Jan 8, 2018 at 11:57 AM, Christoph Hellwig wrote: >> (c) setup/manage any security permissions on mappings >> Which P2P may at some point be concerned with. > > Maybe once root complexes with iommus actually support P2P. But until > then we have a lot more more important

Re: [PATCH 06/12] IB/core: Add optional PCI P2P flag to rdma_rw_ctx_[init|destroy]()

On Mon, Jan 08, 2018 at 07:34:34PM +0100, Christoph Hellwig wrote: > > > > And on that topic, does this scheme work with HFI? > > > > > > No, and I guess we need an opt-out. HFI generally seems to be > > > extremely weird. > > > > This series needs some kind of fix so HFI, QIB, rxe, etc don't

Re: [PATCH 06/12] IB/core: Add optional PCI P2P flag to rdma_rw_ctx_[init|destroy]()

On Mon, Jan 08, 2018 at 11:17:38AM -0700, Logan Gunthorpe wrote: > >>If at all it should be in the dma_map* wrappers, but for that we'd need > >>a good identifier. And it still would not solve the whole fake dma > >>ops issue. > > > >Very long term the IOMMUs under the ops will need to care

Re: [PATCH 06/12] IB/core: Add optional PCI P2P flag to rdma_rw_ctx_[init|destroy]()

On Mon, Jan 08, 2018 at 03:59:01PM +0100, Christoph Hellwig wrote: > On Thu, Jan 04, 2018 at 09:50:31PM -0700, Jason Gunthorpe wrote: > > Well that argument applies equally to the RDMA RW API wrappers around > > the DMA API. I think it is fine if sgl are defined t

Re: [PATCH 06/12] IB/core: Add optional PCI P2P flag to rdma_rw_ctx_[init|destroy]()

On Thu, Jan 04, 2018 at 04:44:00PM -0700, Logan Gunthorpe wrote: > On 04/01/18 03:13 PM, Jason Gunthorpe wrote: > >On Thu, Jan 04, 2018 at 12:52:24PM -0700, Logan Gunthorpe wrote: > >>We tried things like this in an earlier iteration[1] which assumed the SG > >>was

Re: [PATCH 02/12] pci-p2p: Add sysfs group to display p2pmem stats

On Thu, Jan 04, 2018 at 03:50:40PM -0600, Bjorn Helgaas wrote: > > This is similar to /sys/bus/pci/drivers_autoprobe, but > > affects only the VFs associated with a specific PF. > > + > > +What: /sys/bus/pci/devices/.../p2pmem/available > > I wonder if

Re: [PATCH 06/12] IB/core: Add optional PCI P2P flag to rdma_rw_ctx_[init|destroy]()

On Thu, Jan 04, 2018 at 12:52:24PM -0700, Logan Gunthorpe wrote: > >I'd be much happier if dma_map_sg can tell the memory is P2P or not > >from the scatterlist or dir arguments and not require the callers to > >have this. > > We tried things like this in an earlier iteration[1] which assumed the

Re: [PATCH 06/12] IB/core: Add optional PCI P2P flag to rdma_rw_ctx_[init|destroy]()

On Thu, Jan 04, 2018 at 12:01:31PM -0700, Logan Gunthorpe wrote: > > @@ -269,18 +270,24 @@ static int rdma_rw_init_single_wr(struct rdma_rw_ctx > *ctx, struct ib_qp *qp, > * @remote_addr:remote address to read/write (relative to @rkey) > * @rkey:remote key to operate on > * @dir: