> -----Original Message----- > From: Jason Gunthorpe [mailto:jguntho...@obsidianresearch.com] > Sent: Friday, July 24, 2015 2:24 PM > To: Steve Wise > Cc: dledf...@redhat.com; infinip...@intel.com; sa...@mellanox.com; > ogerl...@mellanox.com; r...@mellanox.com; linux- > r...@vger.kernel.org; e...@mellanox.com; target-de...@vger.kernel.org; > linux-...@vger.kernel.org; bfie...@fieldses.org > Subject: Re: [PATCH V6 9/9] isert: Support iWARP transports using FRMRs > > On Fri, Jul 24, 2015 at 01:48:09PM -0500, Steve Wise wrote: > > > The use of FRWR for RDMA READ should be iWarp specific, IB shouldn't > > > pay that overhead. I am expecting to see a cap_rdma_read_rkey or > > > something in here ? > > > > Ok. But cap_rdma_read_rkey() doesn't really describe the > > requirement. The requirement is rkey + REMOTE_WRITE. So it is more > > like rdma_cap_read_requires_remote_write() which is ugly and too > > long (but descriptive)... > > I don't care much what name you pick, just jam something like this in > the description > > If set then RDMA_READ must be performed by mapping the local > buffers through a rkey MR with ACCESS_REMOTE_WRITE enabled. > The rkey of this MR should be passed in as the sg_lists's lkey for > IB_WR_RDMA_READ_WITH_INV. > > FRWR should be used to register the buffer in the send queue, > and the read should be issued using IB_WR_RDMA_READ_WITH_INV (xx > can we just implicitly rely on this? Are there any iWarp cards that > support FRWR but not WITH_INV?) >
No. And iWARP devices must support READ_WITH_INV from my reading of the iWARP verbs spec. I will add all these comments and make use of READ_WITH_INV. > Finally, only a single SGE can be used with RDMA_READ, all scattering > must be accomplished with the MR. > > This quite dramatically changes what is an allowed scatter for the > transfer, IB can support arbitary unaligned S/G lists, while this is > now forced into gapless page aligned elements. > > Your patch takes care of this? And only impacts IB? Did you mean "only impacts iWARP?" Yes the patch takes care of sg->fr page list packing. And it uses the existing isert_map_fr_pagelist() to pack the sg into the fastreg page list. This same routine is used by the IB FRMR/PI reg/unreg routines as well. None of this impacts isert over IB, assuming the suggested change above to only use the new reg/unreg functions for devices that need the iwarp style of read. IE IB devices will either use the existing FRMR/PI methods if they support FRMR/LOCAL_DMA_LKEY/PI (mlx5) or they will use the existing DMA_MR methods (all the rest of the IB devices). IW devices will get my new reg/unreg methods... Steve. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
