Re: [PATCH 20/22] IB/iser: Support up to 8MB data transfer in a single command

2015-08-02 Thread Sagi Grimberg
+static void +iser_calc_scsi_params(struct iser_conn *iser_conn, + unsigned int max_sectors) +{ +struct iser_device *device = iser_conn-ib_conn.device; +unsigned short sg_tablesize, sup_sg_tablesize; + +sg_tablesize = DIV_ROUND_UP(max_sectors * 512, SIZE_4K); +

Re: [PATCH 22/22] IB/iser: Chain all iser transaction send work requests

2015-07-30 Thread Sagi Grimberg
On 7/30/2015 1:27 PM, Or Gerlitz wrote: On Thu, Jul 30, 2015 at 11:06 AM, Sagi Grimberg sa...@mellanox.com wrote: Concatination of send work requests benefits performance by reducing the send queue lock contention (acquired in ib_post_send) and saves us HW doorbells which is posted only once

Re: [PATCH 20/22] IB/iser: Support up to 8MB data transfer in a single command

2015-07-30 Thread Sagi Grimberg
On 7/30/2015 1:22 PM, Or Gerlitz wrote: On Thu, Jul 30, 2015 at 11:06 AM, Sagi Grimberg sa...@mellanox.com wrote: iser support up to 512KB data transfer in a single scsi command. In order to support up to 8MB, iser needs to pre-allocate larger memory regions and larger page vectors. We

Re: [PATCH 11/22] IB/iser: Remove dead code in fmr_pool alloc/free

2015-07-30 Thread Sagi Grimberg
On 7/30/2015 1:31 PM, Or Gerlitz wrote: On Thu, Jul 30, 2015 at 11:06 AM, Sagi Grimberg sa...@mellanox.com wrote: In the past the we always tried to allocate an fmr_pool and if it failed on ENOSYS (not supported) then we continued with dma mr. This is not the case anymore and if we tried

Re: [PATCH WIP 28/43] IB/core: Introduce new fast registration API

2015-07-30 Thread Sagi Grimberg
Can you explain what do you mean by downgrades everything to a 2k alignment? If the ULP is responsible for a PAGE_SIZE alignment than how would this get out of alignment with swiotlb? swiotlb copies all DMA maps to a shared buffer below 4G so it can be used with 32 bit devices. The shared

[PATCH 05/22] IB/iser: Get rid of un-maintained counters

2015-07-30 Thread Sagi Grimberg
We don't update those anywhere in the code and they seem pretty useless (no one seem to care about those). qp_tx_queue_full: We never should get this fmr_map_not_avail: We can never get to this eh_abort_cnt: We don't monitor aborts Go ahead and remove them. Signed-off-by: Sagi Grimberg sa

[PATCH 14/22] IB/iser: Introduce iser registration pool struct

2015-07-30 Thread Sagi Grimberg
Instead of having it a part of the connection structure, have it be under a dedicated (embedded) structure in the connection. A logical separation of the registration pool and the connection structure. Signed-off-by: Sagi Grimberg sa...@mellanox.com Signed-off-by: Adir Lev ad...@mellanox.com

[PATCH 10/22] IB/iser: Rename struct fast_reg_descriptor - iser_fr_desc

2015-07-30 Thread Sagi Grimberg
Avoid struct names without iser_ prefix. This patch does not change any functionality. Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/ulp/iser/iscsi_iser.h | 8 drivers/infiniband/ulp/iser/iser_memory.c | 10 +- drivers/infiniband/ulp/iser/iser_verbs.c

[PATCH 22/22] IB/iser: Chain all iser transaction send work requests

2015-07-30 Thread Sagi Grimberg
Concatination of send work requests benefits performance by reducing the send queue lock contention (acquired in ib_post_send) and saves us HW doorbells which is posted only once. Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/ulp/iser/iscsi_iser.c | 1 + drivers

[PATCH 08/22] IB/iser: Remove an unneeded print for unaligned memory

2015-07-30 Thread Sagi Grimberg
We can do it in iser_aligned_data_len instead and it will save us an argument that is passed to fall_to_counce_buf just for the print. Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/ulp/iser/iser_memory.c | 20 +--- 1 file changed, 9 insertions(+), 11

[PATCH 17/22] IB/iser: Make reg_desc_get a per device routine

2015-07-30 Thread Sagi Grimberg
As for fmrs we will hold a single registration descriptor as no need for multiple like in the frwr mode (descriptor for each task). This change helps unifying the duplicate registration code paths. Signed-off-by: Sagi Grimberg sa...@mellanox.com Signed-off-by: Adir Lev ad...@mellanox.com

Re: [PATCH WIP 21/43] mlx5: Allocate a private page list in ib_alloc_mr

2015-07-30 Thread Sagi Grimberg
On 7/28/2015 1:57 PM, Haggai Eran wrote: Hi Sagi, On 22/07/2015 09:55, Sagi Grimberg wrote: Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/hw/mlx5/mlx5_ib.h | 5 drivers/infiniband/hw/mlx5/mr.c | 45 2 files changed

[PATCH for-4.3 07/15] RDS: Convert to ib_alloc_mr

2015-07-30 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- net/rds/iw_rdma.c | 5 +++-- net/rds/iw_send.c | 5 +++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/net/rds/iw_rdma.c b/net/rds/iw_rdma.c index dba8d08..6a8fbd6 100644 --- a/net/rds/iw_rdma.c +++ b/net/rds/iw_rdma.c

[PATCH 11/22] IB/iser: Remove dead code in fmr_pool alloc/free

2015-07-30 Thread Sagi Grimberg
is called is redundant as well as we are guaranteed it exists. Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/ulp/iser/iser_verbs.c | 26 ++ 1 file changed, 10 insertions(+), 16 deletions(-) diff --git a/drivers/infiniband/ulp/iser/iser_verbs.c b

[PATCH 00/22] iser patches for 4.3

2015-07-30 Thread Sagi Grimberg
on the device capabilities. - Optimize Io path by chaining send work requests and posting them only once. Adir Lev (1): IB/iser: Maintain connection fmr_pool under a single registration descriptor Jenny Falkovich (1): IB/iser: Change some module parameters to be RO Sagi Grimberg (20

[PATCH 15/22] IB/iser: Maintain connection fmr_pool under a single registration descriptor

2015-07-30 Thread Sagi Grimberg
descriptor is to avoid taking a redundant spinlock when working with FMRs. We also change the signature of iser_reg_page_vec to make it match iser_fast_reg_mr (and the future indirect registration method). Signed-off-by: Sagi Grimberg sa...@mellanox.com Signed-off-by: Adir Lev ad...@mellanox.com

[PATCH 02/22] IB/iser: Change minor assignments and logging prints

2015-07-30 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/ulp/iser/iscsi_iser.c | 11 --- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/drivers/infiniband/ulp/iser/iscsi_iser.c b/drivers/infiniband/ulp/iser/iscsi_iser.c index c7cea25..7420062 100644

[PATCH 06/22] IB/iser: Fix possible bogus DMA unmapping

2015-07-30 Thread Sagi Grimberg
If iser_initialize_task_headers() routine failed before dma mapping, we should not attempt to unmap in cleanup_task(). Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/ulp/iser/iscsi_iser.c | 12 drivers/infiniband/ulp/iser/iscsi_iser.h | 2 ++ 2 files changed

[PATCH 04/22] IB/iser: Fix missing return status check in iser_send_data_out

2015-07-30 Thread Sagi Grimberg
iser_initialize_task_headers() might fail. Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/ulp/iser/iser_initiator.c | 8 +--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/ulp/iser/iser_initiator.c b/drivers/infiniband/ulp/iser

[PATCH 09/22] IB/iser: Introduce struct iser_reg_resources

2015-07-30 Thread Sagi Grimberg
of the fast_reg_descriptor and a data/protection indicator. Signed-off-by: Sagi Grimberg sa...@mellanox.com Signed-off-by: Adir Lev ad...@mellanox.com --- drivers/infiniband/ulp/iser/iscsi_iser.h | 36 ++- drivers/infiniband/ulp/iser/iser_memory.c | 35 +-- drivers/infiniband/ulp/iser

Re: Potential lost receive WCs (was [PATCH WIP 38/43])

2015-07-30 Thread Sagi Grimberg
The drivers we have that don't dequeue all the CQEs are doing something like NAPI polling and have other mechanisms to guarentee progress. Don't copy something like budget without copying the other mechanisms :) OK, that makes total sense. Thanks for clarifying. IIRC NAPI is soft-IRQ which

[PATCH for-4.3 00/15] Modify MR allocation API

2015-07-30 Thread Sagi Grimberg
for normal memory registration, IB_MR_TYPE_SIGNATURE for a data-integrity capable memory region and future arbitrary SG support capable memory region). Sagi Grimberg (15): IB/core: Get rid of redundant verb ib_destroy_mr IB: Modify ib_create_mr API IB/iser: Convert to ib_alloc_mr iser-target

[PATCH for-4.3 05/15] IB/srp: Convert to ib_alloc_mr

2015-07-30 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/ulp/srp/ib_srp.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c index a800307..4cc2cd4 100644 --- a/drivers/infiniband/ulp/srp

[PATCH for-4.3 04/15] iser-target: Convert to ib_alloc_mr

2015-07-30 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/ulp/isert/ib_isert.c | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c index 23a793a..bf094cc 100644 --- a/drivers

[PATCH for-4.3 03/15] IB/iser: Convert to ib_alloc_mr

2015-07-30 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/ulp/iser/iser_verbs.c | 7 --- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/ulp/iser/iser_verbs.c b/drivers/infiniband/ulp/iser/iser_verbs.c index 9bef5a7..0a7ceb9 100644 --- a/drivers

[PATCH for-4.3 01/15] IB/core: Get rid of redundant verb ib_destroy_mr

2015-07-30 Thread Sagi Grimberg
(iser/isert) accordingly. Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/core/verbs.c | 17 - drivers/infiniband/hw/mlx5/main.c| 1 - drivers/infiniband/hw/mlx5/mlx5_ib.h | 1 - drivers/infiniband/hw/mlx5/mr.c | 43

[PATCH for-4.3 10/15] ocrdma: Support ib_alloc_mr verb

2015-07-30 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/hw/ocrdma/ocrdma_main.c | 2 +- drivers/infiniband/hw/ocrdma/ocrdma_verbs.c | 11 --- drivers/infiniband/hw/ocrdma/ocrdma_verbs.h | 4 +++- 3 files changed, 12 insertions(+), 5 deletions(-) diff --git a/drivers

[PATCH for-4.3 14/15] qib: Support ib_alloc_mr verb

2015-07-30 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/hw/qib/qib_mr.c| 9 +++-- drivers/infiniband/hw/qib/qib_verbs.c | 2 +- drivers/infiniband/hw/qib/qib_verbs.h | 4 +++- 3 files changed, 11 insertions(+), 4 deletions(-) diff --git a/drivers/infiniband/hw/qib/qib_mr.c

[PATCH for-4.3 12/15] cxgb3: Support ib_alloc_mr verb

2015-07-30 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/hw/cxgb3/iwch_provider.c | 14 ++ 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/drivers/infiniband/hw/cxgb3/iwch_provider.c b/drivers/infiniband/hw/cxgb3/iwch_provider.c index b1b7323..7e004c7

[PATCH for-4.3 08/15] mlx5: Drop mlx5_ib_alloc_fast_reg_mr

2015-07-30 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/hw/mlx5/main.c| 1 - drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 -- drivers/infiniband/hw/mlx5/mr.c | 44 3 files changed, 47 deletions(-) diff --git a/drivers/infiniband/hw/mlx5

[PATCH for-4.3 09/15] mlx4: Support ib_alloc_mr verb

2015-07-30 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/hw/mlx4/main.c| 2 +- drivers/infiniband/hw/mlx4/mlx4_ib.h | 5 +++-- drivers/infiniband/hw/mlx4/mr.c | 11 --- 3 files changed, 12 insertions(+), 6 deletions(-) diff --git a/drivers/infiniband/hw/mlx4

[PATCH for-4.3 11/15] iw_cxgb4: Support ib_alloc_mr verb

2015-07-30 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/hw/cxgb4/iw_cxgb4.h | 4 +++- drivers/infiniband/hw/cxgb4/mem.c | 12 +--- drivers/infiniband/hw/cxgb4/provider.c | 2 +- 3 files changed, 13 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/hw

[PATCH for-4.3 15/15] IB/core: Drop ib_alloc_fast_reg_mr

2015-07-30 Thread Sagi Grimberg
Fully replaced by a more generic and suitable ib_alloc_mr. Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/core/verbs.c | 32 +++- include/rdma/ib_verbs.h | 11 --- 2 files changed, 3 insertions(+), 40 deletions(-) diff --git

[PATCH for-4.3 06/15] xprtrdma, svcrdma: Convert to ib_alloc_mr

2015-07-30 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- net/sunrpc/xprtrdma/frwr_ops.c | 6 +++--- net/sunrpc/xprtrdma/svc_rdma_transport.c | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index 63f282e

[PATCH for-4.3 13/15] nes: Support ib_alloc_mr verb

2015-07-30 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/hw/nes/nes_verbs.c | 19 ++- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/hw/nes/nes_verbs.c b/drivers/infiniband/hw/nes/nes_verbs.c index fbc43e5..44cb513 100644

Re: [PATCH RESEND 0/4] Use max_sge_rd device capability

2015-07-28 Thread Sagi Grimberg
://www.spinics.net/lists/linux-rdma/msg27436.html They really aren't part of iSER/iWARP at all, so I've split them out. Bruce: This hits svcrdma, but I suggest they get merged via Doug's tree to avoid any merge problems. --- Looks good, Reviewed-by: Sagi Grimberg sa...@mellanox.com

Re: [PATCH] xprtrdma: take vendor driver refcount at client

2015-07-28 Thread Sagi Grimberg
On 7/28/2015 2:01 AM, Devesh Sharma wrote: Thanks Chuck Lever for the valuable feedback and suggestions. This is a rework of the following patch sent almost a year back: http://www.mail-archive.com/linux-rdma%40vger.kernel.org/msg20730.html In presence of active mount if someone tries to rmmod

Re: Bug: Discover more than 1 iSER gives -- isert: isert_handle_wc: wr... status {9,1} vend_err {8a,d7) -- -- conn error (1011)

2015-07-28 Thread Sagi Grimberg
On 7/28/2015 5:25 AM, james harvey wrote: Two up to date arch systems. Kernel 4.1.2 (Arch -2). 2 Mellanox MT25418 [ConnectX VPI PCIe 2.0 2.5GT/s - IB DDR / 10GigE] (rev a0) running mlx4_core driver v2.2-1 (Feb, 2014.) Both on most recent firmware for PSID MT_04A0110002, FW Version 2.9.1000.

Re: [PATCH V6 1/9] RDMA/iser: Limit sg tablesize and max_sectors to device fastreg max depth

2015-07-26 Thread Sagi Grimberg
On 7/24/2015 10:14 PM, Jason Gunthorpe wrote: On Fri, Jul 24, 2015 at 01:40:17PM -0500, Steve Wise wrote: Huh. How does this relate to the max_page_list_len argument: struct ib_mr *ib_alloc_fast_reg_mr(struct ib_pd *pd, int max_page_list_len) Shouldn't max_fast_reg_page_list_len be checked

Re: [PATCH WIP 01/43] IB: Modify ib_create_mr API

2015-07-26 Thread Sagi Grimberg
On 7/23/2015 10:08 PM, Jason Gunthorpe wrote: On Thu, Jul 23, 2015 at 01:07:56PM +0300, Sagi Grimberg wrote: On 7/22/2015 10:05 PM, Jason Gunthorpe wrote: The reason I named max_entries is because might might not be pages but real SG elements. It stands for maximum registration entries. Do you

Re: [PATCH V6 4/9] svcrdma: Use max_sge_rd for destination read depths

2015-07-26 Thread Sagi Grimberg
@@ -1059,6 +1062,7 @@ static struct svc_xprt *svc_rdma_accept(struct svc_xprt *xprt) ntohs(((struct sockaddr_in *)newxprt-sc_cm_id- route.addr.dst_addr)-sin_port), newxprt-sc_max_sge, + newxprt-sc_max_sge_rd,

Re: [PATCH 00/10] IB: Replace safe uses for ib_get_dma_mr with pd-local_dma_lkey

2015-07-26 Thread Sagi Grimberg
If we want security by default then I propose not only to change the default value of register_always from false into true but also to change the default value of prefer_fr from false into true such that fast registration becomes the default instead of FMR. Yes, I was frowning at that stuff

Re: [PATCH WIP 28/43] IB/core: Introduce new fast registration API

2015-07-26 Thread Sagi Grimberg
On 7/23/2015 9:51 PM, Jason Gunthorpe wrote: On Thu, Jul 23, 2015 at 07:47:14PM +0300, Sagi Grimberg wrote: So we force ULPs to think about what they are doing properly, and we get a chance to actually force lkey to be local use only for IB. The lkey/rkey decision is passed in the fastreg

Re: [PATCH V6 6/9] isert: Rename IO functions to more descriptive names

2015-07-26 Thread Sagi Grimberg
On 7/24/2015 7:18 PM, Steve Wise wrote: This is in preparation for adding new FRMR-only IO handlers for devices that support FRMR and not PI. Steve, I've given this some thought and I think we should avoid splitting logic from PI and iWARP. The reason (other than code duplication) is that

Re: [PATCH] mlx5: Expose correct page_size_cap in device attributes

2015-07-26 Thread Sagi Grimberg
On 7/24/2015 12:48 AM, Jason Gunthorpe wrote: On Thu, Jul 23, 2015 at 05:41:38PM -0400, Doug Ledford wrote: I assume this prevents the driver from working at all on certain arches (like ppc with 64k page size)? Nothing uses page_size_cap correctly, so it has no impact. Sagi, that is a good

Re: [PATCH V6 6/9] isert: Rename IO functions to more descriptive names

2015-07-26 Thread Sagi Grimberg
On 7/26/2015 1:43 PM, Christoph Hellwig wrote: On Sun, Jul 26, 2015 at 01:08:16PM +0300, Sagi Grimberg wrote: I've given this some thought and I think we should avoid splitting logic from PI and iWARP. The reason (other than code duplication) is that currently the iser target support only up

Re: [PATCH WIP 28/43] IB/core: Introduce new fast registration API

2015-07-26 Thread Sagi Grimberg
I would like to see the kdoc for ib_map_mr_sg explain exactly what is required of the caller, maybe just hoist this bit from the ib_sg_to_pages I'll add the kdoc. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More

Re: [PATCH WIP 28/43] IB/core: Introduce new fast registration API

2015-07-26 Thread Sagi Grimberg
On 7/23/2015 8:55 PM, Jason Gunthorpe wrote: On Thu, Jul 23, 2015 at 01:15:16PM +0300, Sagi Grimberg wrote: I was hoping we'd move the DMA flush and translate into here and make it mandatory. Is there any reason not to do that? The reason I didn't added it in was so the ULPs can make sure

Re: [PATCH V6 6/9] isert: Rename IO functions to more descriptive names

2015-07-26 Thread Sagi Grimberg
Ideally, the post contains a chain of all 4 registrations and the rdma_read (and an opportunistic good scsi response). Just to be clear: This example is for IB only, correct? IW would require rkeys with REMOTE_WRITE and 4 read wrs. My assumption is that it would depend on max_sge_rd. IB

Re: [PATCH V6 9/9] isert: Support iWARP transports using FRMRs

2015-07-26 Thread Sagi Grimberg
On 7/24/2015 10:24 PM, Jason Gunthorpe wrote: On Fri, Jul 24, 2015 at 01:48:09PM -0500, Steve Wise wrote: The use of FRWR for RDMA READ should be iWarp specific, IB shouldn't pay that overhead. I am expecting to see a cap_rdma_read_rkey or something in here ? Ok. But cap_rdma_read_rkey()

Re: [PATCH V6 6/9] isert: Rename IO functions to more descriptive names

2015-07-26 Thread Sagi Grimberg
On 7/26/2015 6:53 PM, Christoph Hellwig wrote: On Sun, Jul 26, 2015 at 02:00:51PM +0300, Sagi Grimberg wrote: On the wire iser sends a single rkey, but the target is allowed to transfer the data however it wants to. So you're trying to get above the limit of a single RDMA READ, not above

Re: [PATCH WIP 28/43] IB/core: Introduce new fast registration API

2015-07-23 Thread Sagi Grimberg
On 7/23/2015 7:14 PM, Jason Gunthorpe wrote: On Thu, Jul 23, 2015 at 01:19:16PM +0300, Sagi Grimberg wrote: Again, related to my prior comments, please have two of these: ib_map_mr_sg_rkey() ib_map_mr_sg_lkey() So we force ULPs to think about what they are doing properly, and we get a chance

Re: [PATCH WIP 01/43] IB: Modify ib_create_mr API

2015-07-23 Thread Sagi Grimberg
On 7/22/2015 10:05 PM, Jason Gunthorpe wrote: On Wed, Jul 22, 2015 at 07:58:23PM +0300, Sagi Grimberg wrote: On 7/22/2015 7:44 PM, Christoph Hellwig wrote: On Wed, Jul 22, 2015 at 10:34:05AM -0600, Jason Gunthorpe wrote: +/** + * ib_alloc_mr() - Allocates a memory region + * @pd

Re: [PATCH WIP 01/43] IB: Modify ib_create_mr API

2015-07-23 Thread Sagi Grimberg
On 7/23/2015 12:30 PM, Christoph Hellwig wrote: On Thu, Jul 23, 2015 at 12:57:34AM +, Hefty, Sean wrote: +enum ib_mr_type { + IB_MR_TYPE_FAST_REG, + IB_MR_TYPE_SIGNATURE, If we're going to go through the trouble of changing everything, I vote for dropping the word 'fast'. It's

Re: [PATCH WIP 28/43] IB/core: Introduce new fast registration API

2015-07-23 Thread Sagi Grimberg
On 7/22/2015 8:44 PM, Jason Gunthorpe wrote: On Wed, Jul 22, 2015 at 09:50:12AM -0700, Christoph Hellwig wrote: +/** + * ib_map_mr_sg() - Populates MR with a dma mapped SG list + * @mr:memory region + * @sg:dma mapped scatterlist + * @sg_nents: number of entries in

Re: [PATCH WIP 38/43] iser-target: Port to new memory registration API

2015-07-23 Thread Sagi Grimberg
On 7/22/2015 8:57 PM, Jason Gunthorpe wrote: On Wed, Jul 22, 2015 at 08:33:16PM +0300, Sagi Grimberg wrote: memset(fr_wr, 0, sizeof(fr_wr)); + ib_set_fastreg_wr(mr, mr-lkey, ISER_FASTREG_LI_WRID, + false, fr_wr); Shouldn't ib_set_fastreg_wr take care

Re: [PATCH WIP 40/43] mlx5: Allocate private context for arbitrary scatterlist registration

2015-07-23 Thread Sagi Grimberg
On 7/23/2015 12:25 PM, Christoph Hellwig wrote: On Wed, Jul 22, 2015 at 11:30:48AM -0600, Jason Gunthorpe wrote: On Wed, Jul 22, 2015 at 09:55:40AM +0300, Sagi Grimberg wrote: + size += max_t(int, MLX5_UMR_ALIGN - ARCH_KMALLOC_MINALIGN, 0); + mr-klms = kzalloc(size, GFP_KERNEL

Re: [PATCH WIP 00/43] New fast registration API

2015-07-23 Thread Sagi Grimberg
On 7/23/2015 12:28 PM, Christoph Hellwig wrote: On Wed, Jul 22, 2015 at 08:42:32PM +0300, Sagi Grimberg wrote: We can do that, but I'd prefer not to pollute the API just for this single use case. What we can do, is add a pool API that would take care of that. But even then we might end up

Re: [PATCH WIP 37/43] xprtrdma: Port to new memory registration API

2015-07-23 Thread Sagi Grimberg
On 7/22/2015 7:04 PM, Chuck Lever wrote: On Jul 22, 2015, at 11:41 AM, Sagi Grimberg sa...@dev.mellanox.co.il wrote: + for (i = 0; i nsegs;) { + sg_set_page(frmr-sg[i], seg-mr_page, + seg-mr_len, offset_in_page(seg-mr_offset)); Cautionary

Re: [PATCH 06/10] IB/iser: Use pd-local_dma_lkey

2015-07-23 Thread Sagi Grimberg
-off-by: Jason Gunthorpe jguntho...@obsidianresearch.com Looks good. Reviewed-by: Sagi Grimberg sa...@mellanox.com -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo

Re: [PATCH 08/10] IB/srp: Use pd-local_dma_lkey

2015-07-23 Thread Sagi Grimberg
...@obsidianresearch.com Looks good. Reviewed-by: Sagi Grimberg sa...@mellanox.com -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: [PATCH 07/10] iser-target: Remove ib_get_dma_mr calls

2015-07-23 Thread Sagi Grimberg
On 7/23/2015 2:34 AM, Jason Gunthorpe wrote: The pd now has a local_dma_lkey member which completely replaces ib_get_dma_mr, use it instead. Signed-off-by: Jason Gunthorpe jguntho...@obsidianresearch.com Looks good. Reviewed-by: Sagi Grimberg sa...@mellanox.com -- To unsubscribe from

Re: [PATCH 00/10] IB: Replace safe uses for ib_get_dma_mr with pd-local_dma_lkey

2015-07-23 Thread Sagi Grimberg
Sagi, IB/iser should have special attention paid, as it is less clear to me if it got everything. It looks fine. I'll pull those as well. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at

Re: [PATCH WIP 37/43] xprtrdma: Port to new memory registration API

2015-07-22 Thread Sagi Grimberg
+ for (i = 0; i nsegs;) { + sg_set_page(frmr-sg[i], seg-mr_page, + seg-mr_len, offset_in_page(seg-mr_offset)); Cautionary note: here we’re dealing with both the “contiguous set of pages” case and the “small region of bytes in a single page” case.

Re: [PATCH WIP 38/43] iser-target: Port to new memory registration API

2015-07-22 Thread Sagi Grimberg
On 7/22/2015 8:04 PM, Christoph Hellwig wrote: @@ -2585,11 +2517,9 @@ isert_fast_reg_mr(struct isert_conn *isert_conn, struct isert_device *device = isert_conn-device; struct ib_device *ib_dev = device-ib_device; struct ib_mr *mr; struct ib_send_wr fr_wr, inv_wr;

Re: [PATCH WIP 00/43] New fast registration API

2015-07-22 Thread Sagi Grimberg
On 7/22/2015 8:10 PM, Christoph Hellwig wrote: Thanks Sagi, this looks pretty good in general, various nitpicks nonwithstanding. The one thing I'm curious about is how we can support SRP with it's multiple MR support without too much boilerplate code. One option would be that pass an array of

Re: [PATCH WIP 01/43] IB: Modify ib_create_mr API

2015-07-22 Thread Sagi Grimberg
On 7/22/2015 8:01 PM, Jason Gunthorpe wrote: On Wed, Jul 22, 2015 at 07:59:16PM +0300, Sagi Grimberg wrote: Do we want to pull ib_get_dma_mr() here with type IB_MR_TYPE_DMA? I want to get rid of ib_get_dma_mr... That's why I asked :) So I'll take it as a no... -- To unsubscribe from

Re: [PATCH WIP 01/43] IB: Modify ib_create_mr API

2015-07-22 Thread Sagi Grimberg
On 7/22/2015 7:44 PM, Christoph Hellwig wrote: On Wed, Jul 22, 2015 at 10:34:05AM -0600, Jason Gunthorpe wrote: +/** + * ib_alloc_mr() - Allocates a memory region + * @pd:protection domain associated with the region + * @mr_type: memory region type + * @max_entries: maximum

[PATCH WIP 00/43] New fast registration API

2015-07-22 Thread Sagi Grimberg
to lose their page list duplication. I haven't done that yet. Comments and review are welcomed (and needed!). Sorry for the long series, but it's kinda transverse... The code/patches can be found in: https://github.com/sagigrimberg/linux/tree/fastreg_api_wip Sagi Grimberg (43): IB: Modify

[PATCH WIP 01/43] IB: Modify ib_create_mr API

2015-07-22 Thread Sagi Grimberg
Use ib_alloc_mr with specific parameters. Change the existing callers. Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/core/verbs.c | 20 -- drivers/infiniband/hw/mlx5/main.c| 2 +- drivers/infiniband/hw/mlx5/mlx5_ib.h | 6

[PATCH WIP 07/43] qib: Support ib_alloc_mr verb

2015-07-22 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/hw/qib/qib_mr.c| 23 +++ drivers/infiniband/hw/qib/qib_verbs.c | 1 + drivers/infiniband/hw/qib/qib_verbs.h | 5 + 3 files changed, 29 insertions(+) diff --git a/drivers/infiniband/hw/qib

[PATCH WIP 06/43] nes: Support ib_alloc_mr verb

2015-07-22 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/hw/nes/nes_verbs.c | 73 +++ 1 file changed, 73 insertions(+) diff --git a/drivers/infiniband/hw/nes/nes_verbs.c b/drivers/infiniband/hw/nes/nes_verbs.c index fbc43e5..ac63763 100644

[PATCH WIP 03/43] ocrdma: Support ib_alloc_mr verb

2015-07-22 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/hw/ocrdma/ocrdma_main.c | 1 + drivers/infiniband/hw/ocrdma/ocrdma_verbs.c | 47 + drivers/infiniband/hw/ocrdma/ocrdma_verbs.h | 4 +++ 3 files changed, 52 insertions(+) diff --git a/drivers

[PATCH WIP 05/43] cxgb3: Support ib_alloc_mr verb

2015-07-22 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/hw/cxgb3/iwch_provider.c | 53 + 1 file changed, 53 insertions(+) diff --git a/drivers/infiniband/hw/cxgb3/iwch_provider.c b/drivers/infiniband/hw/cxgb3/iwch_provider.c index b1b7323..d0e9e2d

[PATCH WIP 08/43] IB/iser: Convert to ib_alloc_mr

2015-07-22 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/ulp/iser/iser_verbs.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/infiniband/ulp/iser/iser_verbs.c b/drivers/infiniband/ulp/iser/iser_verbs.c index 6be4d4a..ecc3265 100644 --- a/drivers

[PATCH WIP 15/43] ocrdma: Drop ocrdma_alloc_frmr

2015-07-22 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/hw/ocrdma/ocrdma_main.c | 1 - drivers/infiniband/hw/ocrdma/ocrdma_verbs.c | 41 - drivers/infiniband/hw/ocrdma/ocrdma_verbs.h | 1 - 3 files changed, 43 deletions(-) diff --git a/drivers

[PATCH WIP 04/43] iw_cxgb4: Support ib_alloc_mr verb

2015-07-22 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/hw/cxgb4/iw_cxgb4.h | 4 +++ drivers/infiniband/hw/cxgb4/mem.c | 57 ++ drivers/infiniband/hw/cxgb4/provider.c | 1 + 3 files changed, 62 insertions(+) diff --git a/drivers/infiniband/hw

[PATCH WIP 02/43] IB/mlx4: Support ib_alloc_mr verb

2015-07-22 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/hw/mlx4/main.c| 1 + drivers/infiniband/hw/mlx4/mlx4_ib.h | 4 drivers/infiniband/hw/mlx4/mr.c | 38 3 files changed, 43 insertions(+) diff --git a/drivers/infiniband/hw

[PATCH WIP 23/43] ocrdma: Allocate a private page list in ib_alloc_mr

2015-07-22 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/hw/ocrdma/ocrdma.h | 2 ++ drivers/infiniband/hw/ocrdma/ocrdma_verbs.c | 9 + 2 files changed, 11 insertions(+) diff --git a/drivers/infiniband/hw/ocrdma/ocrdma.h b/drivers/infiniband/hw/ocrdma/ocrdma.h index

[PATCH WIP 27/43] nes: Allocate a private page list in ib_alloc_mr

2015-07-22 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/hw/nes/nes_verbs.c | 27 +++ drivers/infiniband/hw/nes/nes_verbs.h | 5 + 2 files changed, 32 insertions(+) diff --git a/drivers/infiniband/hw/nes/nes_verbs.c b/drivers/infiniband/hw/nes

[PATCH WIP 24/43] cxgb3: Allocate a provate page list in ib_alloc_mr

2015-07-22 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/hw/cxgb3/iwch_provider.c | 9 + drivers/infiniband/hw/cxgb3/iwch_provider.h | 2 ++ 2 files changed, 11 insertions(+) diff --git a/drivers/infiniband/hw/cxgb3/iwch_provider.c b/drivers/infiniband/hw/cxgb3

[PATCH WIP 33/43] cxgb4: Support the new memory registration API

2015-07-22 Thread Sagi Grimberg
Just duplicated the functions to take the needed arguments from the private MR context. The old fast_reg routines will be dropped later. Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/hw/cxgb4/iw_cxgb4.h | 3 ++ drivers/infiniband/hw/cxgb4/mem.c | 11 + drivers

[PATCH WIP 28/43] IB/core: Introduce new fast registration API

2015-07-22 Thread Sagi Grimberg
-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/core/verbs.c | 123 include/rdma/ib_verbs.h | 37 2 files changed, 160 insertions(+) diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c index

[PATCH WIP 42/43] iser: Accept arbitrary sg lists mapping if the device supports it

2015-07-22 Thread Sagi Grimberg
If the device support arbitrary sg list mapping (device cap IB_DEVICE_MAP_ARB_SG set) we allocate the memory regions with IB_MR_MAP_ARB_SG and skip the counce buffer work around. Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/ulp/iser/iser_memory.c | 4 drivers

[PATCH WIP 43/43] iser: Move unaligned counter increment

2015-07-22 Thread Sagi Grimberg
We don't always use bounce buffers, still we update this counter. Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/ulp/iser/iser_memory.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/infiniband/ulp/iser/iser_memory.c b/drivers/infiniband

[PATCH WIP 37/43] xprtrdma: Port to new memory registration API

2015-07-22 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- net/sunrpc/xprtrdma/frwr_ops.c | 80 ++--- net/sunrpc/xprtrdma/xprt_rdma.h | 4 ++- 2 files changed, 47 insertions(+), 37 deletions(-) diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma

[PATCH WIP 40/43] mlx5: Allocate private context for arbitrary scatterlist registration

2015-07-22 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/hw/mlx5/mlx5_ib.h | 6 ++- drivers/infiniband/hw/mlx5/mr.c | 71 ++-- 2 files changed, 64 insertions(+), 13 deletions(-) diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers

[PATCH WIP 41/43] mlx5: Add arbitrary sg list support

2015-07-22 Thread Sagi Grimberg
If ib_alloc_mr is called with IB_MR_MAP_ARB_SG, the driver allocate a private klm list instead of a private page list. And set the UMR wqe correctly when posting the fast registration. Also, expose device cap IB_DEVICE_MAP_ARB_SG Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers

[PATCH WIP 39/43] IB/core: Add arbitrary sg_list support

2015-07-22 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- include/rdma/ib_verbs.h | 8 +++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index d543fee..cc83c39 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h

[PATCH WIP 30/43] mlx4: Support the new memory registration API

2015-07-22 Thread Sagi Grimberg
Just duplicated the functions to take the needed arguments from the private MR context. The old fast_reg routines will be dropped later. Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/hw/mlx4/main.c| 1 + drivers/infiniband/hw/mlx4/mlx4_ib.h | 3 +++ drivers

[PATCH WIP 18/43] cxgb4: Drop c4iw_alloc_fast_reg_mr

2015-07-22 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/hw/cxgb4/iw_cxgb4.h | 1 - drivers/infiniband/hw/cxgb4/mem.c | 51 -- drivers/infiniband/hw/cxgb4/provider.c | 1 - 3 files changed, 53 deletions(-) diff --git a/drivers/infiniband/hw

[PATCH WIP 29/43] mlx5: Support the new memory registration API

2015-07-22 Thread Sagi Grimberg
Just duplicated the functions to take the needed arguments from the private MR context. The old fast_reg routines will be dropped later. Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/hw/mlx5/main.c| 1 + drivers/infiniband/hw/mlx5/mlx5_ib.h | 3 ++ drivers

[PATCH WIP 32/43] cxgb3: Support the new memory registration API

2015-07-22 Thread Sagi Grimberg
Just duplicated the functions to take the needed arguments from the private MR context. The old fast_reg routines will be dropped later. Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/hw/cxgb3/iwch_provider.c | 12 drivers/infiniband/hw/cxgb3/iwch_qp.c | 48

[PATCH WIP 19/43] cxgb3: Drop iwch_alloc_fast_reg_mr

2015-07-22 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/hw/cxgb3/iwch_provider.c | 47 - 1 file changed, 47 deletions(-) diff --git a/drivers/infiniband/hw/cxgb3/iwch_provider.c b/drivers/infiniband/hw/cxgb3/iwch_provider.c index d0e9e2d..af55b79

[PATCH WIP 31/43] ocrdma: Support the new memory registration API

2015-07-22 Thread Sagi Grimberg
Just duplicated the functions to take the needed arguments from the private MR context. The old fast_reg routines will be dropped later. --- drivers/infiniband/hw/ocrdma/ocrdma_main.c | 1 + drivers/infiniband/hw/ocrdma/ocrdma_verbs.c | 67 +

[PATCH WIP 38/43] iser-target: Port to new memory registration API

2015-07-22 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/ulp/isert/ib_isert.c | 116 ++-- drivers/infiniband/ulp/isert/ib_isert.h | 2 - 2 files changed, 19 insertions(+), 99 deletions(-) diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers

[PATCH WIP 36/43] iser: Port to new fast registration api

2015-07-22 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/ulp/iser/iscsi_iser.h | 6 + drivers/infiniband/ulp/iser/iser_memory.c | 40 --- drivers/infiniband/ulp/iser/iser_verbs.c | 16 + 3 files changed, 17 insertions(+), 45 deletions

[PATCH WIP 26/43] qib: Allocate a private page list in ib_alloc_mr

2015-07-22 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/hw/qib/qib_mr.c| 9 + drivers/infiniband/hw/qib/qib_verbs.h | 2 ++ 2 files changed, 11 insertions(+) diff --git a/drivers/infiniband/hw/qib/qib_mr.c b/drivers/infiniband/hw/qib/qib_mr.c index 2a4afea..a58a347

[PATCH WIP 22/43] mlx4: Allocate a private page list in ib_alloc_mr

2015-07-22 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/hw/mlx4/mlx4_ib.h | 5 drivers/infiniband/hw/mlx4/mr.c | 52 +--- 2 files changed, 54 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/hw/mlx4/mlx4_ib.h b/drivers

[PATCH WIP 21/43] mlx5: Allocate a private page list in ib_alloc_mr

2015-07-22 Thread Sagi Grimberg
Signed-off-by: Sagi Grimberg sa...@mellanox.com --- drivers/infiniband/hw/mlx5/mlx5_ib.h | 5 drivers/infiniband/hw/mlx5/mr.c | 45 2 files changed, 50 insertions(+) diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5

<    1   2   3   4   5   6   7   8   9   10   >