Thanks Dotan, probably I didn't word my query correctly. It should have been like is there a 1-1 mapping between the number of polls to send CQ and the number of calls made to ibv_post_send. The only thing I think that can be going wrong is that, I set the IBV_SEND_SIGNALED flag once in every 256 control messages. Which I assume implies I am getting a completion event only for one of every 256 control messages. Hence the outstanding WR queue is never cleared completely and I finally run out of WR that can be posted to the SQ as large messages are also posted to the same SQ. I have 1024 WR associated with the SQ. I poll the send CQ for every large message but this is not true for every control message that I post to the SQ which might be leading to a starvation of available WR's.
Regards, Bharath -----Original Message----- From: Dotan Barak [mailto:[email protected]] Sent: Monday, December 29, 2008 5:59 AM To: Bharath Ramesh Cc: [email protected] Subject: Re: [ofa-general] ibv_post_send returns -1 On Sat, Dec 27, 2008 at 10:17 PM, Bharath Ramesh <[email protected]> wrote: > Thanks for the reply Dotan, I am posting what I am doing currently, probably > I am missing something which you could point out to me. > > 1) I don't have completion notifications or completion channel associated > with my send CQ. > 2) I only have RC QPs. > 3) Every few hundred messages (256 messages to be precise) for my small > messages (64 bytes long) for which I use IBV_WR_SEND with IBV_SEND_INLINE I > set the IBV_SEND_SIGNALED flag. Once I set the flag I poll my send CQ for > completion messages. > 4) All my RDMA buffers for large messages are all registered, and I use > IBV_SEND_SIGNALED flag for the WR and poll the send CQ for completion > immediately after ibv_post_send. > > The only time when I don't poll my send CQ for completion for ibv_post_send > is for my small messages. I presume that polling the send CQ once every 256 > messages for these should be enough to free up the WR from the SQ for the > previously posted WR. Am I wrong in assuming this, should each and every > ibv_post_send should be associated with poll to the send CQ? > No, you are right. You can poll the CQ once in a while and not after every post, but you have to make sure that the total number of outstanding WR in the Send Queue is less that the value that the QP was created with. Dotan _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
