On Mar 8, 2011, at 12:12 , Damien Guinier wrote: > Hi Jeff
Sorry, your email went on the devel mailing list of Open MPI. > I'm working on large message exchange optimization. My optimization consists > in "choosing > the best protocol for each large message". > In fact, > - for each device, the way to chose the best protocol is different. > - the faster protocol for a given device depends on that device hardware and > on the message > specifications. > > So the device/BTL itself is the best place to dynamically select the fastest > protocol. > > Presently, for large messages, the protocol selection is only based on device > capabilities. > My optimization consists in asking the device/BTL for a "preferred protocol" > and > then make a choice based on : > - the device capabilities and the BTL's recommendation. As a BTL will not randomly change its preferred protocol, one can assume it will depend on the peer. Here is a similar approach to one you describe in your email, but without modification of the BTL interface. https://fs.hlrs.de/projects/eurompi2010/TALKS/WEDNESDAY_AFTERNOON/george_bosilca_locality_and_topology_aware.pdf george. > > Technical view: > The optimization is located in mca_pml_ob1_send_request_start_btl(), after > the device/btl selection. > In the large message section, I call a new function : > mca_pml_ob1_preferred_protocol() => mca_bml_base_preferred_protocol() > This one will try to launch > btl->btl_preferred_protocol() > So, selecting a protocol before a large message in not in the critical path. > It is the BTL's responsibility to define this function to select a preferred > protocol. > > If this function is not defined, nothing changes in the code path > To do this optimization , I had to add an interface to the btl module > structure in "btl.h", this is the drawback. > > ---- > > I have already used this feature to optimize the "shared memory" device/BTL. > I use the "preferred_protocol" feature to enable/disable > KNEM according to intra/inter socket communication. This optimization > increases a "IMB pingping benchmark" bandwidth by ~36%. > > ---- > > The next step is now to use the "preferred protocol" feature with openib ( > with many IB cards) > > > > Attached 2 patches: > 1) BTL_preferred.patch: > introduces the new preferred protocol interface > 2) SM_KNEM_intra_socket.patch: > defines the preferred protocol for the sm btl > Note: Since the "ess" framework can't give us the "socket locality > information", I used hitopo that has been proposed in an RFC > some times ago: > http://www.open-mpi.org/community/lists/devel/2010/11/8677.php > > > > <BTL_preferred.path><SM_KNEM_intra_socket.patch>_______________________________________________ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel "I disapprove of what you say, but I will defend to the death your right to say it" -- Evelyn Beatrice Hall
