On Wed, Nov 10, 2010 at 09:43:12AM -0800, Roland Dreier wrote: > > Could I get some clarification on where there is no ordering > > guarantee? The WC's do not necessarily come back in the order that the > > sends were posted? > > For a given queue, completions are always returned in the order that > work requests were posted. However there is no ordering between > different queues.
Right, so if you design a scheme where getting a recv grants a send credit then you are doing Post Send #1 ---> Recv WC Recv WC <---- Post Send #2 Send WC #2 Send WC #2 PostSend #3 ---> Recv WC and there is no order guarentee for when you get a Recv WC vs a Send WC, even though by protocol design they are in fact ordered. If you load the HCA the Send Complete WC's do get behind Recv WCs. This is true even if you use the same CQ for Recv WC and Send Complete WCs. In practice this can make the implementation quite troublesome since the posting of a new Send can't necessarily be done in the Recv WC path as no send buffers could be available at that time. I've found properly placing all the activities in a RDMA system to be the hardest challenge of the whole design. Avoiding deadlock and starvation and untangling the various data dependencies can be hard :( Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
