Gleb's post earlier today inspired me to [finally] work on the
modularizing the openib btl connection semantics. We need this to
add the RDMA CM support anyway, and it seemed like a natural time to
actually start something on it. I used Gleb's patch as a starting
point.
The idea is to separate out the mechanisms used to make OF
connections from btl_openib_endpoint.[ch] by moving them into
individual component-like .c files (we've discussed this idea off-
list before). Hence, there's one source file for OOB-based
connections and another source file for RDMA CM-based connections (I
anticipate another for appletalk someday, and perhaps pidgeonnet,
too). There's no dlopen or any "real" component stuff, but the
functionality is accessed through function pointers so you can switch
between them at run time (think of it as "components-lite"). The
RDMA CM stuff will be an empty shell at the moment -- to be filled in
later.
I'm still working through the abstractions and will commit everything
on a tmp branch shortly (i.e., Gleb's work plus this new stuff).
There are 2 main ideas:
1. At the beginning of time, the main openib btl component calls the
connection wireup black box and says, "Whenever you finish creating
connection requested on behalf of a remote peer, call function X."
2. Whenever the main openib btl wants to start an outgoing
connection, it calls the wireup black box and says, "Create a
connection to this remote peer; when you're done creating the
connection, call function Y."
Selection between which wireup scheme to use (oob vs. rdma_cm) will
initially be quite stupid: a simple user-specified MCA parameter
defaulting to "oob". Perhaps the selection can be something more
intelligent someday. The wireup schemes can, themselves, export MCA
parameters if they want to (I don't know what RDMA CM's requirements
will be).
Galen's BSRQ stuff has dibs on merging into the trunk because a) he
started first and b) he asked today. So if all goes well, I
anticipate this stuff will come in to the trunk after Galen's,
potentially within a week or two.
--
Jeff Squyres
Cisco Systems
