> > What happens if one tries to do RDMA (say write for example) higher > than > > 4 (or 128 in changed case)? does it just wait till previos operation > > is completed? > > I don't remember seeing any error ....it was only > > limited by the send Q-depth which can go much larger value. > > Yes, the limit of outstanding RDMAs is not related to the send queue > depth. Of course you can post many more than 4 RDMAs to a send queue > -- the HCA just won't have more than 4 requests outstanding at a time.
To further clarity, this parameter only affects the number of concurrent outstanding RDMA Reads which the HCA will process. Once it hits this limit, the send Q will stall waiting for issued reads to complete prior to initiating new reads. It does not affect RDMA Writes. It is very analogous to outstanding reads parameters in PCI-X and PCIe (although this parameters is independent from those). The IB spec defines ordering rules for RDMA Reads and Writes. The number of outstanding RDMA Reads is negotiated by the CM during connection establishment and the QP which is sending the RDMA Read must have a value configured for this parameter which is <= the remote ends capability. In previous testing by Mellanox on SDR HCAs they indicated values beyond 2-4 did not improve performance (and in fact required more RDMA resources be allocated for the corresponding QP or HCA). Hence I suspect a very large value like 128 would offer no improvement over values in the 2-8 range. Todd Rimmer _______________________________________________ openib-general mailing list [email protected] http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
