The current implementations of the libibverbs and librdmacm libraries are limited in their scalability by using OS event objects to report certain type of IB/RDMA events: asynchronous events on an HCA, CQ completion events, and connection events.
This limitation comes from the limit of WaitForMultipleObjects, which is limited to at most 64 event objects. This limit is easily seen by running rdma_cmatose using the -c option, and setting the number of connections > 64. Because overlapped operations cannot share OS event objects, the best (only?) viable scaling alternative is to use I/O completion ports (IOCPs). However, since an open OS file object can be associated with only one IOCP, that IOCP must be shared between the libibverbs and librdmacm libraries. Additionally, this requires a mechanism on top of the IOCP capable of de-multiplexing IB/RDMA events to the correct channel: ibv_comp_channel, rdma_event_channel, or ibv_context (for async events). Add a new abstraction, the COMP_CHANNEL, capable of de-multiplexing overlapped completion events among multiple queues. The libibverbs and librdmacm are modified to use that abstraction for reporting events. In order to share objects between libibverbs and the librdmacm, add Windows specific APIs to libibverbs to get and release Windows specific data. The COMP_CHANNEL abstraction is exposed directly to applications, which provides them greater control over how events are monitored. (This is similar to how fd's are exposed in the OFED code, which DAPL makes use of.) Finally, in order to scale the number of connections, modify the IB CM to drop, rather than reject, incoming REQs that are not immediately ready for processing because of backlog limitations. This allows the REQ to be retried and the connection established once the backlog has dropped. This matches the behavior that was added to the Linux stack. Signed-off-by: Sean Hefty <sean.he...@intel.com> _______________________________________________ ofw mailing list ofw@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw