[PATCH 4/9] srpt: chain RDMA READ/WRITE requests

2015-11-13 Thread Christoph Hellwig
Remove struct rdma_iu and instead allocate the struct ib_rdma_wr array early and fill out directly. This allows us to chain the WRs, and thus archive both less lock contention on the HCA workqueue as well as much simpler error handling. Signed-off-by: Christoph Hellwig <h...@lst.de> --- d

[PATCH 5/9] srpt: use the new CQ API

2015-11-13 Thread Christoph Hellwig
Signed-off-by: Christoph Hellwig <h...@lst.de> --- drivers/infiniband/ulp/srpt/ib_srpt.c | 327 +- drivers/infiniband/ulp/srpt/ib_srpt.h | 28 +-- 2 files changed, 88 insertions(+), 267 deletions(-) diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/d

[PATCH 7/9] IB/iser: Use a dedicated descriptor for login

2015-11-13 Thread Christoph Hellwig
From: Sagi Grimberg <sa...@mellanox.com> Makes better sense and we'll need it later with CQ abstraction. iser switch login bufs to void Signed-off-by: Sagi Grimberg <sa...@mellanox.com> Signed-off-by: Christoph Hellwig <h...@lst.de> --- drivers/infiniband/ulp/iser/iscs

[PATCH 6/9] srp: use the new CQ API

2015-11-13 Thread Christoph Hellwig
This also moves recv completion handling from hardirq context into softirq context. Signed-off-by: Christoph Hellwig <h...@lst.de> --- drivers/infiniband/ulp/srp/ib_srp.c | 198 ++-- drivers/infiniband/ulp/srp/ib_srp.h | 7 +- 2 files changed, 76 inse

[PATCH 9/9] IB/iser: Convert to CQ abstraction

2015-11-13 Thread Christoph Hellwig
From: Sagi Grimberg <sa...@mellanox.com> Signed-off-by: Sagi Grimberg <sa...@mellanox.com> Signed-off-by: Christoph Hellwig <h...@lst.de> --- drivers/infiniband/ulp/iser/iscsi_iser.h | 68 --- drivers/infiniband/ulp/iser/iser_initiator.c | 142 ++- driv

[PATCH 2/9] IB: add a proper completion queue abstraction

2015-11-13 Thread Christoph Hellwig
sucked too much and converted the iSER initiator to the new API. Signed-off-by: Christoph Hellwig <h...@lst.de> --- drivers/infiniband/Kconfig | 1 + drivers/infiniband/core/Makefile| 2 +- drivers/infiniband/core/cq.c

[PATCH 1/9] move blk_iopoll to limit and make it generally available

2015-11-13 Thread Christoph Hellwig
The new name is irq_poll as iopoll is already taken. Better suggestions welcome. Signed-off-by: Christoph Hellwig <h...@lst.de> --- Documentation/kernel-per-CPU-kthreads.txt | 2 +- block/Makefile| 2 +- block/blk-iopoll.c

add a proper completion queue abstraction

2015-11-13 Thread Christoph Hellwig
This series adds a new RDMA core abstraction that insulated the ULPs from the nitty gritty details of CQ polling. See the individual patches for more details. Note that this series should be applied on top of my "IB: merge struct ib_device_attr into struct ib_device" patch. A git tree is also

[PATCH 3/9] IB: add a helper to safely drain a QP

2015-11-13 Thread Christoph Hellwig
Signed-off-by: Christoph Hellwig <h...@lst.de> --- drivers/infiniband/core/cq.c | 46 include/rdma/ib_verbs.h | 2 ++ 2 files changed, 48 insertions(+) diff --git a/drivers/infiniband/core/cq.c b/drivers/infiniband/core/cq.c index d

Re: srp state in current mainline

2015-11-12 Thread Christoph Hellwig
On Thu, Nov 12, 2015 at 05:12:14PM -0800, Bart Van Assche wrote: > On 11/12/2015 09:59 AM, Christoph Hellwig wrote: > >[ 108.998574] WARNING: CPU: 0 PID: 1258 at kernel/sched/core.c:7389 > >__might_sleep+0xa7/0xb0() > >[ 108.998580] do not call blocking ops when !TASK

Re: srp state in current mainline

2015-11-12 Thread Christoph Hellwig
On Wed, Nov 11, 2015 at 01:07:44PM -0800, Bart Van Assche wrote: > On 11/10/2015 09:15 AM, Christoph Hellwig wrote: > >scsi host3: ib_srp: failed receive status WR flushed (5) for iu > >880313f4ca40 > > Can you also post the logs from the target system from around th

Re: [PATCH RFC 1/3] IB/core: Expose a device attribute for rdma_read access flags

2015-11-11 Thread Christoph Hellwig
On Wed, Nov 11, 2015 at 11:09:03AM +0200, Sagi Grimberg wrote: > It does, but it doesn't look like something we'd want to check for each > IO... Both potential callers already have a access flags variable in object that's assigned to at setup time so I don't see a problem here. -- To unsubscribe

Re: [PATCH RFC 1/3] IB/core: Expose a device attribute for rdma_read access flags

2015-11-11 Thread Christoph Hellwig
On Wed, Nov 11, 2015 at 11:07:14AM +0200, Sagi Grimberg wrote: > > > On 10/11/2015 15:41, Christoph Hellwig wrote: > >FYI, this is the API I'd aim for (only SRP and no HW driver converted > >yet): > > This looks fine, although personally I find scope and direction

Re: srp state in current mainline

2015-11-11 Thread Christoph Hellwig
On Wed, Nov 11, 2015 at 07:35:47AM -0800, Bart Van Assche wrote: > On 11/10/2015 09:15 AM, Christoph Hellwig wrote: > >This is a simply xfstests run using XFS on a remote LIO ramdisk. > > Hello Christoph, > > Which version of the kernel and LIO were installed at the targ

Re: srp state in current mainline

2015-11-11 Thread Christoph Hellwig
On Wed, Nov 11, 2015 at 08:03:46AM -0800, Bart Van Assche wrote: > Hello Christoph, > > The SRP initiator from kernel 4.3 is working fine on my test setup. I will > start a test with Linus' tree and with the following SRP kernel module > parameters: > > # cat /etc/modprobe.d/ib_srp.conf >

Re: [PATCH RFC 2/3] svcrdma: Use device rdma_read_access_flags

2015-11-11 Thread Christoph Hellwig
On Tue, Nov 10, 2015 at 11:25:46AM -0700, Jason Gunthorpe wrote: > I like this, my only comment is we should have a rdma_cap for this > behavior, rdma_cap_needs_rdma_read_mr(pd) or something? Yes, that's better than checking the protocol. > > + if (!(dev->device_cap_flags & > >

Re: [PATCH RFC 1/3] IB/core: Expose a device attribute for rdma_read access flags

2015-11-11 Thread Christoph Hellwig
On Tue, Nov 10, 2015 at 11:36:27AM -0700, Jason Gunthorpe wrote: > > n = ib_map_mr_sg(desc->mr, state->sg, state->sg_nents, > > -dev->mr_page_size); > > +dev->mr_page_size, > > +/* > > + * XXX: add a bool write

Re: [PATCH RFC 2/3] svcrdma: Use device rdma_read_access_flags

2015-11-11 Thread Christoph Hellwig
On Tue, Nov 10, 2015 at 04:55:51PM -0700, Jason Gunthorpe wrote: > IMHO, the break point for when it makes sense to switch from a RDMA > READ chain to a MR is going to be a RDMA core tunable. The performance > curve won't have much to do with the ULP. core/device, a lot of it depends on when we'd

Re: [PATCH RFC 1/3] IB/core: Expose a device attribute for rdma_read access flags

2015-11-11 Thread Christoph Hellwig
On Tue, Nov 10, 2015 at 11:01:56AM -0700, Jason Gunthorpe wrote: > No need to change every driver. > > I'd suggest something like > > unsigned int rdma_cap_rdma_read_mr_flags(const struct ib_pd *pd) > { > if (rdma_protocol_iwarp(pd->device, rdma_start_port(pd->device))) >

srp state in current mainline

2015-11-10 Thread Christoph Hellwig
I've just tried forward porting some work affecting SRP from a 4.1-ish base, and started to run into error ASAP on current Linus' HEAD and also 4.3. In current HEAD memory registrations on the client seem to fail, probably due to the MR rework, but even on 4.3 I run into crazy corruption reports

Re: [PATCH RFC 2/3] svcrdma: Use device rdma_read_access_flags

2015-11-10 Thread Christoph Hellwig
On Tue, Nov 10, 2015 at 01:46:40PM +0200, Sagi Grimberg wrote: > > > On 10/11/2015 13:41, Christoph Hellwig wrote: > >Oh, and while we're at it. Can someone explain why we're even > >using rdma_read_chunk_frmr for IB? It seems to work around the > >fact tat iWarp on

Re: [PATCH RFC 1/3] IB/core: Expose a device attribute for rdma_read access flags

2015-11-10 Thread Christoph Hellwig
On Tue, Nov 10, 2015 at 12:44:13PM +0200, Sagi Grimberg wrote: > Different devices may require different access permissions > for rdma reads e.g. for Infiniband devices, local write access > suffices while iWARP devices require remote write permissions as > well. > > This situation generates

[PATCH] IB: start documenting device capabilities

2015-11-10 Thread Christoph Hellwig
Just IB_DEVICE_LOCAL_DMA_LKEY and IB_DEVICE_MEM_MGT_EXTENSIONS for now as I'm most familar with those. Signed-off-by: Christoph Hellwig <h...@lst.de> --- include/rdma/ib_verbs.h | 18 ++ 1 file changed, 18 insertions(+) diff --git a/include/rdma/ib_verbs.h b/includ

Re: [PATCH RFC 1/3] IB/core: Expose a device attribute for rdma_read access flags

2015-11-10 Thread Christoph Hellwig
On Tue, Nov 10, 2015 at 02:42:44PM +0200, Sagi Grimberg wrote: > >When this approach is used, the upper layer doesn't have to be aware > >at all of the lower layer's details. > > Yea, we could do that too. Any preferences from other people? > I'm pretty indifferent on which way to go... Yes,

Re: [PATCH] IB/mlx: Expose max_fmr to ib_query_device

2015-11-05 Thread Christoph Hellwig
On Thu, Nov 05, 2015 at 12:48:45PM +0200, Yuval Shaia wrote: > On Thu, Oct 29, 2015 at 07:21:45PM +0200, Sagi Grimberg wrote: > > Hi Yuval, > > > > The title prefix should be IB/mlx4: > > > > >Expose max_fmr so it will be available to ULPs. > > >max_fmr is num_mpts minus reserved. What do you

Re: [PATCH] IB/mlx: Expose max_fmr to ib_query_device

2015-11-05 Thread Christoph Hellwig
On Thu, Nov 05, 2015 at 01:07:49PM +0200, Yuval Shaia wrote: > > What do you plan to use it for? We're aiming hard to get rid of FMRs > > from the whole kernel stack ASAP. > We have two drivers that are using FMR. I now RDS still has the totally silly separate code bases for IB vs iWrap and you

Re: [PATCH 4/7] IB/srp: Fix a potential queue overflow in an error path

2015-11-03 Thread Christoph Hellwig
On Tue, Nov 03, 2015 at 12:50:59PM -0800, Bart Van Assche wrote: > Such a check wouldn't be that simple because the only way to perform such a > check is either by doubling the number of ib_map_mr_sg() calls or by > performing additional memory allocations. The other option woud be to disallow

Re: shrink struct ib_send_wr V4

2015-10-29 Thread Christoph Hellwig
On Wed, Oct 28, 2015 at 10:57:59PM -0400, Doug Ledford wrote: > > I had to do a minor hand merge to get this to apply, but it has been > > pulled in for 4.4. > > This breaks all of the drivers in staging BTW. That will need fixed up > before the pull request goes in during the merge window.

Re: merge struct ib_device_attr into struct ib_device V2

2015-10-21 Thread Christoph Hellwig
On Tue, Oct 20, 2015 at 06:07:31PM +0300, Or Gerlitz wrote: > But this makes struct ib_device much much bigger, and this structure > is used in **fast** path, > e.g the ULP traverses through a pointer from struct ib_device to > post_send/recv, poll_cq and friends. But it doesn't get near the end

Re: merge struct ib_device_attr into struct ib_device V2

2015-10-21 Thread Christoph Hellwig
On Tue, Oct 20, 2015 at 06:13:35PM +0300, Sagi Grimberg wrote: >> The networking community will let you work for 10y before they add a >> field to struct net_device exactly b/c >> of the reason I brought. Why here we can come out of the blue and add >> tens if not hundreds of fields to our device

Re: merge struct ib_device_attr into struct ib_device V2

2015-10-21 Thread Christoph Hellwig
On Wed, Oct 21, 2015 at 09:44:41AM +0300, Or Gerlitz wrote: > Fact is that for struct net_device you will not add 333 new fields over > night in the coming 33 years, for sure. That's because they never had this split and added fields to struct netdev as required. One interesting difference in

Re: merge struct ib_device_attr into struct ib_device V2

2015-10-21 Thread Christoph Hellwig
On Wed, Oct 21, 2015 at 10:11:29AM +0300, Or Gerlitz wrote: > > We will have many more device query extensions, None of which use struct ib_device_attr I hope! > but the point I tried to > make here is a bit different -- > we do need to keep the user/kernel device attr struct as part of the

Re: merge struct ib_device_attr into struct ib_device V2

2015-10-21 Thread Christoph Hellwig
On Wed, Oct 21, 2015 at 10:41:13AM +0300, Or Gerlitz wrote: > I know, but a patch that adds caching an attr pointer on the device will > remove these local caches, > we actually had that/similar patch posted here and it was dropped/forgotten. > >> so if you use one of those you're >> already

Re: merge struct ib_device_attr into struct ib_device V2

2015-10-20 Thread Christoph Hellwig
On Tue, Oct 20, 2015 at 04:08:54PM +0300, Sagi Grimberg wrote: > Can you explain what you mean by performance gains? My understanding is > that this patch is not performance critical. It just replaces each and > every ULP device attributes caching. Exactly. The only 'performance' we care about

[PATCH] IB: merge struct ib_device_attr into struct ib_device

2015-10-12 Thread Christoph Hellwig
Avoid the need to query for device attributes and store them in a separate structure by merging struct ib_device_attr into struct ib_device. This matches how the device structures are used in most Linux subsystems. Signed-off-by: Christoph Hellwig <h...@lst.de> --- drivers/infiniband/cor

merge struct ib_device_attr into struct ib_device V2

2015-10-12 Thread Christoph Hellwig
This patch gets rid of struct ib_device_attr and cleans up drivers nicely. It goes on top of my send_wr cleanups and the memory registration udpates from Sagi. Changes since V1: - rebased on top of the Sagi's latest reg_api.6 branch -- To unsubscribe from this list: send the line "unsubscribe

Re: merge struct ib_device_attr into struct ib_device V2

2015-10-12 Thread Christoph Hellwig
On Mon, Oct 12, 2015 at 12:26:06PM +0300, Sagi Grimberg wrote: > First go with this looks OK for mlx4. mlx5 needs the below incremental > patch to be folded in. > > we need dev->ib_dev.max_pkeys set when get_port_caps() is called. Thanks, I've folded your patch and force pushed out the updated

Re: [Ksummit-discuss] [TECH TOPIC] IRQ affinity

2015-10-12 Thread Christoph Hellwig
On Mon, Oct 12, 2015 at 12:09:48PM -0400, Theodore Ts'o wrote: > Hi Christoph, > > Do you think this is still an issue that would be worth discsussing at > the kernel summit as a technical topic? If so, would you be willing > to be responsible for kicking off the discussion for this topic? Hi

Re: shrink struct ib_send_wr V4

2015-10-11 Thread Christoph Hellwig
On Sun, Sep 13, 2015 at 05:13:33PM +0200, Christoph Hellwig wrote: > This series shrinks the WR size by splitting out the different WR > types. > > Patch number one is too large for the mailinglist, so if you didn't > get it grab it here: > > > http://git.infradea

Re: [PATCH] IB: merge struct ib_device_attr into struct ib_device

2015-10-11 Thread Christoph Hellwig
On Sun, Oct 11, 2015 at 07:16:27PM +0300, Sagi Grimberg wrote: > Christoph, would you mind rebasing it on top of 4.3-rc4 or so? I > want to develop over it so I can test it on the fly. I can do a rebase to whatever you want. Initially this was over your reg_api branch. Is a rebase to the latest

Re: [PATCH 1/2] iser: Set block queue_virt_boundary and remove bounce buffering code

2015-10-10 Thread Christoph Hellwig
Looks good, Reviewed-by: Christoph Hellwig <h...@lst.de> -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: [PATCH 2/2] IB/iser: Enable SG clustering

2015-10-10 Thread Christoph Hellwig
Looks good, Reviewed-by: Christoph Hellwig <h...@lst.de> -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 00/26] New fast registration API

2015-10-08 Thread Christoph Hellwig
On Thu, Oct 08, 2015 at 10:50:43AM +0300, Sagi Grimberg wrote: > - Rebased against Doug's for-4.4 tree (4.3.0-rc1) + 4.3-rc fixes Does this means its not on top of the send_wr changes now? I'm fine with that as I think your series is even more important than the send_wr split and I'm happy to

Re: [PATCH v1 00/24] New fast registration API

2015-10-07 Thread Christoph Hellwig
On Tue, Oct 06, 2015 at 11:37:40AM +0300, Sagi Grimberg wrote: > The issue is that the device requires the MR page array to have > an alignment (0x40 for mlx4 and 0x400 for mlx5). When I modified the > page array allocation to be non-coherent I didn't take care of > alignment. Just curious: why

Re: [PATCH v1 00/24] New fast registration API

2015-10-07 Thread Christoph Hellwig
On Wed, Oct 07, 2015 at 12:25:25PM +0300, Sagi Grimberg wrote: > Bart suggested that having to sync once for the entire page list might > perform better than coherent memory. I'll settle either way since using > non-coherent memory might cause higher-order allocations due to > alignment, so it's

Re: [PATCH v1 01/24] IB/core: Introduce new fast registration API

2015-09-28 Thread Christoph Hellwig
On Mon, Sep 28, 2015 at 01:57:52PM -0700, Bart Van Assche wrote: > >Actually I think it doesn't since it is only relevant for the else > >statement where we are passing the page_size boundary. > > Hello Sagi, > > Suppose that the following sg-list is passed to this function as { offset, > length

Re: libmlx4 and libmlx5 git trees? Who is handling those?

2015-09-28 Thread Christoph Hellwig
Hi Robert, getting a package out should not be an issue. master should always be in releasable state, and cutting a release should be a simple shell script doing all the tagging and uploading. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to

[PATCH] IB: merge struct ib_device_attr into struct ib_device

2015-09-23 Thread Christoph Hellwig
Avoid the need to query for device attributes and store them in a separate structure by merging struct ib_device_attr into struct ib_device. This matches how the device structures are used in most Linux subsystems. Signed-off-by: Christoph Hellwig <h...@lst.de> --- drivers/infiniband/cor

Re: [PATCH] IB: merge struct ib_device_attr into struct ib_device

2015-09-23 Thread Christoph Hellwig
On Wed, Sep 23, 2015 at 10:23:15AM -0700, Chuck Lever wrote: > Getting rid of ib_query_device() makes sense. Moving the device > attributes into ib_device is nice. Getting rid of ib_device_attr > is of questionable value. Why do we need to go there? > > IB core API changes generate merge

Re: merge struct ib_device_attr into struct ib_device

2015-09-22 Thread Christoph Hellwig
Hi Yann, looks like the patch was too large and majordomo ate it Here is a link: http://git.infradead.org/users/hch/rdma.git/commitdiff/0e46553467cd01b63ab9c985f87c18c5328880bb -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to

message size, was Re: merge struct ib_device_attr into struct ib_device

2015-09-22 Thread 'Christoph Hellwig'
On Tue, Sep 22, 2015 at 04:06:17PM -0500, Steve Wise wrote: > Can you create a series of smaller patches that will fit on the list? > That would make it easier for everyone to review/comment. I don't see how that is possible, as it's a flag day change. But maybe we really need to bump up the

merge struct ib_device_attr into struct ib_device

2015-09-21 Thread Christoph Hellwig
This patch gets rid of struct ib_device_attr and cleans up drivers nicely. It goes on top of my send_wr cleanups and the memory registration udpates from Sagi. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More

Re: [PATCH v1 00/24] New fast registration API

2015-09-19 Thread Christoph Hellwig
Hi Sagi, I've converted the driver I'm developing to your API and it works great. I think this is an important step towards making the RDMA more usable! -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo

Re: [PATCH rdma-next 02/32] IB/core: Add SEND_LAST_INV and SEND_ONLY_INV opcodes

2015-09-16 Thread Christoph Hellwig
On Wed, Sep 16, 2015 at 04:42:36PM +0300, Kamal Heib wrote: > Intorduce Add SEND_LAST_INV and SEND_ONLY_INV opcodes in ib_pack.h to be > used by RXE for RC. Why does RXE need new public opcodes? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to

[PATCH 2/2] IB: remove xrc_remote_srq_num from struct ib_send_wr

2015-09-13 Thread Christoph Hellwig
The field is only initialized in mlx, but never used. If we want to add proper XRC support it should be done with a new struct ib_xrc_wr. This shrinks the various WR structures by another 4 bytes. Signed-off-by: Christoph Hellwig <h...@lst.de> Reviewed-by: Sagi Grimberg <sa...@mel

Re: RDMA/CM and multiple QPs

2015-09-10 Thread Christoph Hellwig
On Thu, Sep 10, 2015 at 12:50:33PM +0300, Sagi Grimberg wrote: > >What I'm more interested in is a way to tell the CM that I only > >want routes that are using this ib_device that I got from the first > >lookup as all others are useless for me. > > > > I'm not sure I understand what you are

Re: RDMA/CM and multiple QPs

2015-09-08 Thread Christoph Hellwig
On Tue, Sep 08, 2015 at 03:32:11PM +0300, Sagi Grimberg wrote: > The CM is responsible of establishing an RDMA channel. What you are > referring to is a concept of a session. I'm not entirely sure how we can > fit a model where the CM establishes a multi-channel session as the > CM request

Re: RDMA/CM and multiple QPs

2015-09-06 Thread Christoph Hellwig
On Sun, Sep 06, 2015 at 01:24:52PM +0530, Parav Pandit wrote: > Yes. SRP is good example. The point I am trying to make is, SRP > implements failover and request spreading where one QP fails it > delivers to other QP. But SRP doesn't implement that. There are no fail over capabilities in a

RDMA/CM and multiple QPs

2015-09-06 Thread Christoph Hellwig
Hi All, right now RDMA/CM works on a QP basis, but seems very awakward if you want multiple QPs as part of a single logical device, which will be useful for a lot of modern protocols. For example we will need to check in the CM handler that we're not getting a different ib_device if we want to

Re: RDMA/CM and multiple QPs

2015-09-06 Thread Christoph Hellwig
On Sun, Sep 06, 2015 at 01:12:56PM +0530, Parav Pandit wrote: > Hi Christoph, > > Establishing multiple QP is just one part of it. > Bigger challenge is how do we distribute the work request among > multiple QPs For my case I simply rely on the blk-mq layer to have cpu-local queues, so that's a

Re: shrink struct ib_send_wr V3

2015-08-31 Thread Christoph Hellwig
On Sun, Aug 30, 2015 at 06:31:35PM +0300, Sagi Grimberg wrote: >> - patch 2 now explicitly replaces the weird overloading in the mlx5 >> driver with an explicit embedding of struct ib_send_wr, similar >> to what we do for all other MRs. > > That's nice, > > There is one non-trivial spot

shrink struct ib_send_wr V3

2015-08-26 Thread Christoph Hellwig
This series shrinks the WR size by splitting out the different WR types. Patch number two is too large for the mailinglist, so if you didn't get it grab it here: http://git.infradead.org/users/hch/rdma.git/commitdiff/2b63f958de7bd630aba85caf65986831d4372869 or the full git tree at:

[PATCH 3/3] IB: remove xrc_remote_srq_num from struct ib_send_wr

2015-08-26 Thread Christoph Hellwig
The field is only initialized in mlx, but never used. If we want to add proper XRC support it should be done with a new struct ib_xrc_wr. This shrinks the various WR structures by another 4 bytes. Signed-off-by: Christoph Hellwig h...@lst.de Reviewed-by: Sagi Grimberg sa...@mellanox.com

[PATCH 1/3] IB/uverbs: reject invalid or unknown opcodes

2015-08-26 Thread Christoph Hellwig
We have many WR opcodes that are only supported in kernel space and/or require optional information to be copied into the WR structure. Reject all those not explicitly handled so that we can't pass invalid information to drivers. Cc: sta...@vger.kernel.org Signed-off-by: Christoph Hellwig h

ehca driver status?

2015-08-26 Thread Christoph Hellwig
Hi Hoang-Nam, hi Christoph, hi Alexander, do you know what the current state of the eHCA driver and hardware is? The driver hasn't seen any targeted updats since 2010, so we wonder if it's still alive? It's one of the few drivers not supporting FRWRs, and it's also really odd in that it has its

Re: [PATCH 1/3] IB/uverbs: reject invalid or unknown opcodes

2015-08-25 Thread Christoph Hellwig
On Mon, Aug 24, 2015 at 10:59:21AM +0300, Haggai Eran wrote: It looks odd to me as well, but it's not really something I want to change in this series. Note that sparse annoted types like __be32 aren't really common in userspace, but with a bit of effort they can be supported. We have

Re: [PATCH 1/3] IB/uverbs: reject invalid or unknown opcodes

2015-08-24 Thread Christoph Hellwig
On Mon, Aug 24, 2015 at 09:52:14AM +0300, Haggai Eran wrote: Okay. Maybe you can just add a case for IB_WR_SEND in this patch to avoid hurting bisectability. I've done this already, just waiting for more feedback before resending:

Re: [PATCH 1/3] IB/uverbs: reject invalid or unknown opcodes

2015-08-22 Thread Christoph Hellwig
On Sat, Aug 22, 2015 at 06:38:47AM +, Haggai Eran wrote: It looks like the default case in the non-UD branch is currently used to handle plain IB_WR_SEND operations, so the patch would cause these to return an error. Indeed. It's handled fine in patch 2 which splits up the case, but

Re: [PATCH WIP 28/43] IB/core: Introduce new fast registration API

2015-08-21 Thread Christoph Hellwig
On Thu, Aug 20, 2015 at 01:04:13PM -0600, Jason Gunthorpe wrote: Trying to decouple the sub resources, ie by separately pooling the MR/SQE/etc, is just unnecessary complexity, IMHO.. NFS client already had serioues bugs in this area. So, I turn to the idea that every ULP should work as the

Re: [PATCH 1/3] IB/uverbs: reject invalid or unknown opcodes

2015-08-20 Thread Christoph Hellwig
On Wed, Aug 19, 2015 at 07:50:23PM +, Hefty, Sean wrote: AFAIK, this path is rarely (never?) actually used. I think all the drivers we have can post directly from userspace. I didn't think the ipath or qib drivers post from userspace. Makes sense with their software IB stack. Guess

Supported uverbs opcodes?

2015-08-19 Thread Christoph Hellwig
What opcodes are supposed to be submitted by users? Currently we do not define opcodes in the UAPI and kinda rely that userspace uses the same ones as the kernel. For thos defines by libibverbs (RDMA_WRITE, RDMA_WRITE_WITH_IMM, SEND, SEND_WITH_IMM, RDMA_READ, ATOMIC_CMP_AND_SWP and

Re: [PATCH WIP 28/43] IB/core: Introduce new fast registration API

2015-08-19 Thread Christoph Hellwig
On Wed, Aug 19, 2015 at 02:56:24PM +0300, Sagi Grimberg wrote: So I had a go with moving the DMA mapping into ib_map_mr_sg() and it turns out mapping somewhat poorly if the ULP _may_ register memory or just send sg_lists (like storage targets over IB/iWARP). So the ULP will sometimes use the

shrink struct ib_send_wr

2015-08-19 Thread Christoph Hellwig
This series shrinks the WR size by splitting out the different WR types. Patch number two is too large for the mailinglist, so if you didn't get it grab it here: http://git.infradead.org/users/hch/rdma.git/commitdiff/30e522ee6c1d7adb614d7308f09fbfd71c6d3e07 or the full git tree at:

[PATCH 3/3] IB: remove xrc_remote_srq_num from struct ib_send_wr

2015-08-19 Thread Christoph Hellwig
The field is only initialized in mlx, but never used. If we want to add proper XRC support it should be done with a new struct ib_xrc_wr. This shrinks the various WR structures by another 4 bytes. Signed-off-by: Christoph Hellwig h...@lst.de --- drivers/infiniband/hw/mlx5/qp.c | 1 - include

[PATCH 1/3] IB/uverbs: reject invalid or unknown opcodes

2015-08-19 Thread Christoph Hellwig
We have many WR opcodes that are only supported in kernel space and/or require optional information to be copied into the WR structure. Reject all those not explicitly handled so that we can't pass invalid information to drivers. Cc: sta...@vger.kernel.org Signed-off-by: Christoph Hellwig h

Re: [PATCH 1/3] IB/uverbs: reject invalid or unknown opcodes

2015-08-19 Thread Christoph Hellwig
On Wed, Aug 19, 2015 at 11:46:14AM -0600, Jason Gunthorpe wrote: Reviewed-by: Jason Gunthorpe jguntho...@obsidianresearch.com AFAIK, this path is rarely (never?) actually used. I think all the drivers we have can post directly from userspace. Oh, interesting. Is there any chance to

Re: [PATCH WIP 28/43] IB/core: Introduce new fast registration API

2015-08-19 Thread Christoph Hellwig
On Wed, Aug 19, 2015 at 07:09:18PM +0300, Sagi Grimberg wrote: Ok, I was also thinking on moving the access flags to the work request again. Yes, with the current code I don't think we need it in the MR. I'd prefer to get this right with a different helper like Steve suggested: int

Re: [RFC] split struct ib_send_wr

2015-08-17 Thread Christoph Hellwig
On Thu, Aug 13, 2015 at 09:04:39AM -0700, Christoph Hellwig wrote: I'm happy to do that if you're fine with the patch in general. amso1100 should be trivial anyway, while ipath is a mess, just like the new intel driver with the third copy of the soft ib stack. Correct. http

Re: [RFC] split struct ib_send_wr

2015-08-13 Thread Christoph Hellwig
On Thu, Aug 13, 2015 at 11:22:34AM -0600, Jason Gunthorpe wrote: The uverbs change needs to drop/move the original kmalloc: next = kmalloc(ALIGN(sizeof *next, sizeof (struct ib_sge)) + user_wr-num_sge * sizeof (struct ib_sge),

Re: [RFC] split struct ib_send_wr

2015-08-13 Thread Christoph Hellwig
On Thu, Aug 13, 2015 at 09:07:14AM -0400, Doug Ledford wrote: Doug: was your mail a request to fix up the two de-staged drivers? I'm happy to do that if you're fine with the patch in general. amso1100 should be trivial anyway, while ipath is a mess, just like the new intel driver with

Re: [RFC] split struct ib_send_wr

2015-08-13 Thread Christoph Hellwig
On Wed, Aug 12, 2015 at 08:24:49PM +0300, Sagi Grimberg wrote: Just a nit that I've noticed, in mlx4 set_fmr_seg params are not aligned to the parenthesis (maybe in other locations too but I haven't noticed such...) This is just using a normal two tab indent for continued function parameters..

Re: [RFC] split struct ib_send_wr

2015-08-12 Thread Christoph Hellwig
On Wed, Aug 12, 2015 at 07:24:44PM -0700, Chuck Lever wrote: That makes sense, but you already Acked the change that breaks Lustre, and it's going in through the NFS tree. Are you changing that to a NAK? It seems like Doug was mostly concened about to be removed drivers. I defintively refuse to

Re: [RFC] split struct ib_send_wr

2015-08-07 Thread Christoph Hellwig
On Thu, Aug 06, 2015 at 01:58:45PM -0400, Chuck Lever wrote: Wondering if this means we'll have to drop ib_reg_phys_mr() removal until Lustre gets around to removing their call sites from the staging tree. Why? Just because the buildbot catches it? -- To unsubscribe from this list: send the

Re: [RFC] split struct ib_send_wr

2015-08-07 Thread 'Christoph Hellwig'
On Thu, Aug 06, 2015 at 12:44:42PM -0500, Steve Wise wrote: Driver/staging isn't considered in tree for global API change perspective, so I didn't bother with all these staging drivers. The kbuild test bot will probably catch this. It already did catch it for my tree, which is expected.

Re: [RFC] split struct ib_send_wr

2015-08-07 Thread Christoph Hellwig
On Thu, Aug 06, 2015 at 11:08:45PM +0530, Parav Pandit wrote: Do you see value in dividing ib_ud _wr into ib_ud_wr and ib_ud_gsi_wr to save 4 bytes? For now I just wanted to split along the lines of the existing unions. From looking at the various drivers splitting the GSI path might not be a

Re: [RFC] split struct ib_send_wr

2015-08-07 Thread Christoph Hellwig
On Thu, Aug 06, 2015 at 07:46:44PM +0300, Sagi Grimberg wrote: I agree that this is a shame to keep in here for everyone to carry... The only driver I've seen supporting XRC is mlx5 with no consumers. If people are reluctant to remove it, you can put it in ib_xrc_send_wr or something...

Re: [PATCH for-4.3 11/15] iw_cxgb4: Support ib_alloc_mr verb

2015-08-07 Thread Christoph Hellwig
On Fri, Aug 07, 2015 at 10:06:26AM -0500, Steve Wise wrote: If it is too much of a pain to alter this patch, then I'll just submit the NFSRDMA fix and live with the bisect issue... Doug's tree is still to be rebased. So please submit your NFS fix now as ask Doug to merge it before Sagi's

Re: [PATCH for-4.3 11/15] iw_cxgb4: Support ib_alloc_mr verb

2015-08-07 Thread 'Christoph Hellwig'
On Fri, Aug 07, 2015 at 11:29:12AM -0500, Steve Wise wrote: I misspoke. I had the order reversed. The order is such that we can add my new NFS patch after: e20684a xprtrdma, svcrdma: Convert to ib_alloc_mr and before these: af78181 cxgb3: Support ib_alloc_mr verb b7e06cd iw_cxgb4:

Re: [PATCH for-4.3 11/15] iw_cxgb4: Support ib_alloc_mr verb

2015-08-07 Thread 'Christoph Hellwig'
On Fri, Aug 07, 2015 at 11:19:59AM -0500, Steve Wise wrote: I guess I'll post two patches, the NFS fix that preceeds af78181/ b7e06cd, and a reworked patch to replace e20684a. Is that the way to go in your opinion? To me this sounds good. We have a couple patches from Jason's series that

Re: [RFC] split struct ib_send_wr

2015-08-07 Thread Christoph Hellwig
On Fri, Aug 07, 2015 at 10:17:18AM -0400, Chuck Lever wrote: If bot barking doesn't bother anyone, then I'll keep the removal patch. For some such a complaint might be grounds for rejecting the patch. If it's (a) in tree proper and (b) not one of the rare false positives I would consider it a

Re: [RFC] split struct ib_send_wr

2015-08-06 Thread Christoph Hellwig
On Wed, Aug 05, 2015 at 10:40:08PM -0600, Jason Gunthorpe wrote: Any numbers on the struct size reduction? sizeof(struct ib_send_wr) (old): 96 sizeof(struct ib_send_wr): 48 sizeof(struct ib_rdma_wr): 64 sizeof(struct ib_atomic_wr): 96 sizeof(struct ib_ud_wr): 88 sizeof(struct ib_fast_reg_wr):

Re: [RFC] split struct ib_send_wr

2015-08-06 Thread Christoph Hellwig
On Thu, Aug 06, 2015 at 12:04:32PM -0500, Steve Wise wrote: You missed amso1100 (and probably ipath) that have been moved to drivers/staging... Driver/staging isn't considered in tree for global API change perspective, so I didn't bother with all these staging drivers. -- To unsubscribe from

Re: [RFC] split struct ib_send_wr

2015-08-06 Thread Christoph Hellwig
I've pushed out a new version. Updates: - the ib_recv_wr change Bart notices has been fixed. - iser and isert have been converted - the handling of the embedded WR in the qib software queue entry has been fixed. Which means we're basically done now and the patch could use broader testing.

[RFC] split struct ib_send_wr

2015-08-04 Thread Christoph Hellwig
Hi all, please take a look at my RFC patch here: http://git.infradead.org/users/hch/scsi.git/commitdiff/751774250b71da83a26ba8584cff70f5e7bb7b1e the commit contains my explanation, but apparently the patch is too large for the list limit and didn't make it through. -- To unsubscribe

Re: [RFC] split struct ib_send_wr

2015-08-04 Thread Christoph Hellwig
On Tue, Aug 04, 2015 at 04:07:42PM +, Hefty, Sean wrote: This looks like a reasonable start. It may help with feedback if you could just post the changes to ib_verbs.h. Not sure it's all that useful, but here we go: diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index

Re: [PATCH, RFC] rdma: split struct ib_send_wr

2015-08-04 Thread Christoph Hellwig
On Tue, Aug 04, 2015 at 08:06:16PM +0300, Sagi Grimberg wrote: Question though, a ULP may want to keep a couple of WRs around instead of having each allocated in the stack and handled one by one. We need to provide it with a hint of what is the size it needs. Note that with the drastic shrink

Re: [PATCH, RFC] rdma: split struct ib_send_wr

2015-08-04 Thread Christoph Hellwig
On Tue, Aug 04, 2015 at 08:44:26PM +0300, Sagi Grimberg wrote: I do agree that the size on the stack is less of an issue now. What still can matter is handling each wr one by one vs. doing a collective post. But if structured correctly you can still do that with on-stack WRs. I can

Re: [RFC] split struct ib_send_wr

2015-08-04 Thread Christoph Hellwig
On Tue, Aug 04, 2015 at 09:36:49AM -0700, Bart Van Assche wrote: diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h [ ... ] struct ib_recv_wr { +struct ib_send_wr wr; struct ib_recv_wr *next; u64 wr_id; struct ib_sge

Re: [PATCH v2 00/12] IB: Replace safe uses for ib_get_dma_mr with pd-local_dma_lkey

2015-08-03 Thread Christoph Hellwig
On Fri, Jul 31, 2015 at 03:20:40PM -0700, Bart Van Assche wrote: On 07/30/2015 04:22 PM, Jason Gunthorpe wrote: All patches are compile tested. I've done basic testing up to and including the IPoIB patch, the rest required specialized setups I don't have access to, but are fairly

Re: [PATCH v4 00/50] Add OPA gen1 driver

2015-08-03 Thread Christoph Hellwig
On Sat, Aug 01, 2015 at 04:34:31PM -0400, Doug Ledford wrote: Or, I haven't looked at the soft-roce driver (ever). Is it going to need this library as well? ROCE implements the IB protocol, so a software ROCE driver will need a IB protocol implementation sitting ontop of ethernet frames (v1)

<    1   2   3   4   5   >