I traced down the main cause of the performance difference between the winverbs and ibal ND providers.
The culprit is CWVDevice::FindPkey(). This converts a pkey to an index and is called before transitioning the QP into the INIT state. The ibal provider hard codes the pkey index to 0, so lacks this problem. After modifying FindPkey() to return an index of 0, the connection rates were 1400 for ibal, versus 1300 for winverbs. (Note that this affects the performance of the librdmacm as well.) Ideally, we want to support multiple pkeys, including handling dynamic changes. The current implementation handles this using a very simple approach - always query for the pkey. It shouldn't be too hard to support multiple pkeys by caching the pkey table in userspace, but handling dynamic changes is a little more difficult. The design of winverbs is such that the query calls always drop into the kernel to retrieve the latest data. Caching was intended to be above the interface, with the caller making use of CWVDevice::Notify() to know when changes have occurred, so that a new query can be issued. I haven't yet decided what to do about this yet, if anything. - Sean _______________________________________________ ofw mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
