Hi Paul, thanks a lot for your quick reply! In my test, client informs the server of its local memory (rkey, addr, size) by sending 4 back to back messages, each message elicits a RDMA read request (RR) from the server.
In other words, client exposes its memory to the server, and server RDMA reads it. As far as RDMA read is concerned, server is a requester, and client is a responder, right? The error I encountered happens at the initial phase, when client sends 4 back to back messages to server(using ibv_post_send ), containing (rkey, addr, size) client's local memory. In these 4 ibv_post_send(), client will see one failure. At server side, server has already posted enough WQs in the RQ. The failures are included in my first email. Looking at the program output, it appears that, server gets messages 1, issues RR 1, gets message 2, issues RR 2. But somehow client reports that "send message 2" fails. On the contrary, server reports "receive message 3" fails. As a result, server gets message 1,2,4, and succeeds with RR 1,2,4. But clients sees that message 2 fails, and succeed with message 1,3,4. This inconsistency is the problem that puzzled me. ------------ By the way, how to interpret the parameters for RDMA, and what are parameters that control RDMA behavior? Below are something I can find, there must be more.... max_qp_rd_atom: 4 max_res_rd_atom: 258048 max_qp_init_rd_atom: 128 qp_attr.max_dest_rd_atomic qp_attr.max_rd_atomic -neutron On Tue, Nov 10, 2009 at 2:04 AM, Paul Grun <[email protected]> wrote: > Is it possible that you exceeded the number of available RDMA Read Resources > available on the server? There is an expectation that the client knows how > many outstanding RDMA Read Requests the responder (server) is capable of > handling; if the requester (client) exceeds that number, the responder will > indeed return a NAK-Invalid Request. Sounds like your server is configured > to accept three outstanding RDMA Read Requests. > This also explains why it works when you pause the program periodically...it > gives the responder time to generate the RDMA Read Responses and therefore > free up some resources to be used in receiving the next incoming RDMA Read > Request. > > -Paul > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of neutron > Sent: Monday, November 09, 2009 9:04 PM > To: [email protected] > Subject: back to back RDMA read fail? > > Hi all, > > I have a simple program that test back to back RDMA read performance. > However I encountered errors for unknown reasons. > > The basic flow of my program is: > > client: > ibv_post_send() to send 4 back to back messages to server (no delay > inbetween). Each message contains the (rkey, addr, size) of a local > buffer. The buffer is registered with remote-read/write/ permissions. > After that, ibv_poll_cq() is called to wait for completion. > > server: > First, enough receive WRs are posted to the RQ. Upon receipt of each > message, immediately post a RDMA read request, using the (rkey, addr, > size) information contained in the originating message. > > -------------- > Both client and server use RC QP. Some errors are observed. > > On client side, ibv_poll_cq() gets 4 CQE, one out of the 4 CQE is an error: > CQ:: wr_id=0x0, wc_opcode=IBV_WC_SEND, wc_status=remote invalid RD > request, wc_flag=0x3b > byte_len=11338758, immdata=1110104528, qp_num=0x0, src_qp=2290530758 > > The other 3 CQE are success. > > On server side, > 3 of the 4 messages are successfully received. One message produces an > error CQE: > CQ:: wr_id=0x8000000000, wc_opcode=Unknow-wc-opcode, > wc_status=unknown, wc_flag=0x0 > byte_len=9569287, immdata=0, qp_num=0x0, src_qp=265551872 > > The 3 RDMA read corresponding to the successful receive all succeed. > > But, if I pause the client program for a short while( usleep(100) for > example ) after calling ibv_post_send(), then no error occurs. > Anyone can point out the pitfall here? Thanks! > > > ----------- > On both client and server, I'm using 'mthca0' type MT25208. The QPs > are initialized with "qp_attr.max_dest_rd_atomic=4, > qp_attr.max_rd_atomic = 4". The QP's "devinfo -v" gives the > information: > > hca_id: mthca0 > fw_ver: 5.1.400 > node_guid: 0002:c902:0023:c04c > sys_image_guid: 0002:c902:0023:c04f > vendor_id: 0x02c9 > vendor_part_id: 25218 > hw_ver: 0xA0 > board_id: MT_0370130002 > phys_port_cnt: 2 > max_mr_size: 0xffffffffffffffff > page_size_cap: 0xfffff000 > max_qp: 64512 > max_qp_wr: 16384 > device_cap_flags: 0x00001c76 > max_sge: 27 > max_sge_rd: 0 > max_cq: 65408 > max_cqe: 131071 > max_mr: 131056 > max_pd: 32764 > max_qp_rd_atom: 4 > max_ee_rd_atom: 0 > max_res_rd_atom: 258048 > max_qp_init_rd_atom: 128 > max_ee_init_rd_atom: 0 > atomic_cap: ATOMIC_HCA (1) > max_ee: 0 > max_rdd: 0 > max_mw: 0 > max_raw_ipv6_qp: 0 > max_raw_ethy_qp: 0 > max_mcast_grp: 8192 > max_mcast_qp_attach: 56 > max_total_mcast_qp_attach: 458752 > max_ah: 0 > max_fmr: 0 > max_srq: 960 > max_srq_wr: 16384 > max_srq_sge: 27 > max_pkeys: 64 > local_ca_ack_delay: 15 > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > the body of a message to [email protected] > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
