lihaidong wrote:
Mr.Wise:
Sorry to Bother you Another problem.
fastreg(Must with local_dma_key to avoid using ib_reg_phys_mr in my
case) with read_inv succeed, yet with server_inv failed.
What does
'krping: cq completion failed with wr_id 0 status 6 opcode -1 vender_err 78'
in client dmesg info mean?
Maybe the mlx4 experts can comment on status 6 vender_err 78?
Why data transferring happened before server waked up from waiting for
CONNECTED?
This can happen. There is a race between getting the CONNECTED event
and the first incoming data completion (like a recv completion).
Client demsg:
krping: proc write
|client,addr=10.10.10.15,port=8888,count=1,verbose,local_dma_lkey,mem_mode=fastreg,server_inv|
client
ipaddr (10.10.10.15)
port 8888
count 1
verbose
using local dma lkey
created cm_id ffff88013cd11000
cma_event type 0 cma_id ffff88013cd11000 (parent)
cma_event type 2 cma_id ffff88013cd11000 (parent)
Fastreg supported - device_cap_flags 0x7c9c76
rdma_resolve_addr - rdma_resolve_route successful
created pd ffff88012a9fff80
created cq ffff8801305d5400
created qp ffff8801305d5800
krping: krping_setup_buffers called on cb ffff88013cd11800
krping: fastreg rkey 0x88001a00 page_list ffff88012b3cee80 page_list_len 1
krping: allocated & registered buffers...
cma_event type 9 cma_id ffff88013cd11000 (parent)
ESTABLISHED
rdma_connect successful
krping: page_list[0] 0x12a40c000
krping: post_inv = 0, fastreg new rkey 0x88001a01 shift 12 len 64 iova_start
12a40c180 page_list_len 1
RDMA addr 12a40c180 rkey 88001a01 len 64
krping: cq completion failed with wr_id 0 status 6 opcode -1 vender_err 78
krping: cq completion in ERROR state
krping: krping_format_send failed
krping_free_buffers called on cb ffff88013cd11800
destroy cm_id ffff88013cd11000
Server dmesg:
krping: proc write
|server,addr=10.10.10.15,port=8888,count=1,verbose,local_dma_lkey,mem_mode=fastreg,server_inv|
server
ipaddr (10.10.10.15)
port 8888
count 1
verbose
using local dma lkey
created cm_id ffff88007e3c5c00
rdma_bind_addr successful
rdma_listen
cma_event type 4 cma_id ffff88003e0dac00 (child)
child cma ffff88003e0dac00
Fastreg supported - device_cap_flags 0x7c9c76
created pd ffff880035d17d20
created cq ffff88001e0f9e00
created qp ffff88001e0f9c00
krping: krping_setup_buffers called on cb ffff88007e2b0000
krping: fastreg rkey 0x68001d00 page_list ffff8800027ab580 page_list_len 1
krping: allocated & registered buffers...
accepting client connection request
cma_event type 9 cma_id ffff88003e0dac00 (child)
ESTABLISHED
cma_event type 10 cma_id ffff88003e0dac00 (child)
krping: DISCONNECT EVENT...
krping: wait for CONNECTED state 10
krping: connect error -1
krping_free_buffers called on cb ffff88007e2b0000
destroy cm_id ffff88007e3c5c00
2010-01-22
------------------------------------------------------------------------
lihaidong
------------------------------------------------------------------------
*发件人：* Steve Wise
*发送时间：* 2010-01-21 06:43:30
*收件人：* lihaidong
*抄送：* linux-rdma
*主题：* Re: Problems using krping
It appears the MLX4 driver does not support kernel mode memory regions.
You'll have to use dma mrs or fast_reg mrs with that device.
Steve.
lihaidong wrote:
> Mr.Wise:
>
> When using mr mode as the memory registration method, krping failed to
> get memory region using ib_reg_phys_mr().Could you help me, please?
>
> dmesg:
> krping_init
> krping: proc write
|client,addr=10.10.10.15,mem_mode=mr,port=9999,count=1,verbose|
> client
> ipaddr (10.10.10.15)
> port 9999
> count 1
> verbose
> created cm_id ffff88013c74c800
> cma_event type 0 cma_id ffff88013c74c800 (parent)
> cma_event type 2 cma_id ffff88013c74c800 (parent)
> rdma_resolve_addr - rdma_resolve_route successful
> created pd ffff880133845280
> created cq ffff88013c1ff400
> created qp ffff88013c1ffe00
> krping: krping_setup_buffers called on cb ffff88013c59f800
> krping: recv buf dma_addr 13c59f968 size 16
> krping: recv_buf reg_mr failed
> krping: krping_setup_buffers failed: -38
> destroy cm_id ffff88013c74c800
> krping: proc write
|client,addr=10.10.10.15,mem_mode=mr,port=9999,count=1,verbose|
> client
> ipaddr (10.10.10.15)
> port 9999
> count 1
> verbose
> created cm_id ffff88013c59f800
> cma_event type 0 cma_id ffff88013c59f800 (parent)
> cma_event type 2 cma_id ffff88013c59f800 (parent)
> rdma_resolve_addr - rdma_resolve_route successful
> created pd ffff88012f5964a0
> created cq ffff88012faee400
> created qp ffff88012faeec00
> krping: krping_setup_buffers called on cb ffff88013c71d400
> krping: recv buf dma_addr 13c71d568 size 16
> krping: recv_buf reg_mr failed
> krping: krping_setup_buffers failed: -38
> destroy cm_id ffff88013c59f800
>
>
> echo "client,addr=10.10.10.15,mem_mode=mr,port=9999,count=1" > /proc/krping
>
> echo "server,addr=10.10.10.15,mem_mode=mr,port=9999" > /proc/krping
>
> Using OFED-1.5 ofa_kernel-1.5
> HardWare:Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE]
>
> put krping source files into drivers/infiniband/hw/mlx4
>
> 2010-01-21
> ------------------------------------------------------------------------
> lihaidong
__________ Information from ESET NOD32 Antivirus, version of virus signature
database 4788 (20100120) __________
The message was checked by ESET NOD32 Antivirus.
http://www.eset.com
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
