Tom,
Vu - I noticed this wasn't a usual kernel patch, so I've recoded it
as a patch against Trond's tree, where I have the connectathon
patches queued too. I can send it to him with your signed-off-by,
if that's ok.
Here is the kernel patch. It's tested with latest connectX FW (2.6.636)
I do want to be sure the patch is tested though. Does it work on
earlier COnnectX firmware that may not support FRMR? I assume
that fw doesn't set the bits the verbs.c file is looking for to detect
FRMR support... correct?
The patch does not work with old FW.
The old fw set the bit; however, it has bug on FRWR
thanks,
-vu
Tom.
At 06:09 PM 3/22/2009, Tom Talpey wrote:
At 02:27 PM 3/22/2009, James Lentini wrote:
On Fri, 20 Mar 2009, Vu Pham wrote:
Hi Jon,
Here is the patch for the file
kernel_patches/fixes/nfsrdma_to_2_6_28_18.patch. This patch will
fix the fast
registration wr for Mellanox ConnectX.
ConnectX FRMR requires local write enable together with remote rdma write
enable
thanks,
-vu
Vu,
Are you going to submit this fix upstream?
Agreed - if setting this bit fixes the FRMR issue, then it should go to
Trond asap for inclusion in 2.6.30 (hopefully). Can someone verify that
it works? I don't have access to ConnectX hardware as I am at the
OFA workshop.
Tom.
-james
---
ofa_kernel-1.4.1.config/kernel_patches/fixes/nfsrdma_to_2.6.28_13.patc
h 2009-03-19 08:42:07.000000000 -0700
+++ ofa_kernel-1.4.1/kernel_patches/fixes/nfsrdma_to_2.6.28_13.patch
2009-03-19 08:49:22.000000000 -0700
@@ -167,7 +167,7 @@
case RPCRDMA_MTHCAFMR:
case RPCRDMA_MEMWINDOWS_ASYNC:
case RPCRDMA_MEMWINDOWS:
-@@ -1391,6 +1457,96 @@ rpcrdma_unmap_one(struct rpcrdma_ia *ia,
struct rpcrdma_mr_seg *seg)
+@@ -1391,6 +1457,98 @@ rpcrdma_unmap_one(struct rpcrdma_ia *ia,
struct rpcrdma_mr_seg *seg)
}
static int
@@ -214,8 +214,10 @@
+ frmr_wr.wr.fast_reg.page_list_len = i;
+ frmr_wr.wr.fast_reg.page_shift = PAGE_SHIFT;
+ frmr_wr.wr.fast_reg.length = i << PAGE_SHIFT;
-+ frmr_wr.wr.fast_reg.access_flags = (writing ?
-+ IB_ACCESS_REMOTE_WRITE : IB_ACCESS_REMOTE_READ);
++ frmr_wr.wr.fast_reg.access_flags =
++ (writing ?
++ (IB_ACCESS_REMOTE_WRITE | IB_ACCESS_LOCAL_WRITE) :
++ IB_ACCESS_REMOTE_READ);
+ frmr_wr.wr.fast_reg.rkey = seg1->mr_chunk.rl_mw->r.frmr.fr_mr->rkey;
+ DECR_CQCOUNT(&r_xprt->rx_ep);
+
ConnectX fast registration WR requires local write enable together with remote
rdma write
Signed-off-by: Vu Pham <[email protected]>
--- a/net/sunrpc/xprtrdma/verbs.c 2009-03-03 17:05:22.000000000 -0800
+++ b/net/sunrpc/xprtrdma/verbs.c 2009-03-04 15:32:21.000000000 -0800
@@ -1495,7 +1500,8 @@
frmr_wr.wr.fast_reg.page_shift = PAGE_SHIFT;
frmr_wr.wr.fast_reg.length = i << PAGE_SHIFT;
frmr_wr.wr.fast_reg.access_flags = (writing ?
- IB_ACCESS_REMOTE_WRITE : IB_ACCESS_REMOTE_READ);
+ (IB_ACCESS_REMOTE_WRITE | IB_ACCESS_LOCAL_WRITE) :
IB_ACCESS_REMOTE_READ);
+
frmr_wr.wr.fast_reg.rkey = seg1->mr_chunk.rl_mw->r.frmr.fr_mr->rkey;
DECR_CQCOUNT(&r_xprt->rx_ep);
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
