Hi George,
This certainly looks like our motivations are close. However, I don't see
in the presentation how you implement it (maybe I misread it), especially
how you manage to not modify the BTL interface.
Do you have any code / SVN commit references for us to better understand
what it's about ?
Thanks,
Sylvain
On Tue, 8 Mar 2011, George Bosilca wrote:
On Mar 8, 2011, at 12:12 , Damien Guinier wrote:
Hi Jeff
Sorry, your email went on the devel mailing list of Open MPI.
I'm working on large message exchange optimization. My optimization consists in
"choosing
the best protocol for each large message".
In fact,
- for each device, the way to chose the best protocol is different.
- the faster protocol for a given device depends on that device hardware and on
the message
specifications.
So the device/BTL itself is the best place to dynamically select the fastest
protocol.
Presently, for large messages, the protocol selection is only based on device
capabilities.
My optimization consists in asking the device/BTL for a "preferred protocol" and
then make a choice based on :
- the device capabilities and the BTL's recommendation.
As a BTL will not randomly change its preferred protocol, one can assume
it will depend on the peer. Here is a similar approach to one you
describe in your email, but without modification of the BTL interface.
https://fs.hlrs.de/projects/eurompi2010/TALKS/WEDNESDAY_AFTERNOON/george_bosilca_locality_and_topology_aware.pdf
george.
Technical view:
The optimization is located in mca_pml_ob1_send_request_start_btl(), after the
device/btl selection.
In the large message section, I call a new function :
mca_pml_ob1_preferred_protocol() => mca_bml_base_preferred_protocol()
This one will try to launch
btl->btl_preferred_protocol()
So, selecting a protocol before a large message in not in the critical path.
It is the BTL's responsibility to define this function to select a preferred
protocol.
If this function is not defined, nothing changes in the code path
To do this optimization , I had to add an interface to the btl module structure in
"btl.h", this is the drawback.
----
I have already used this feature to optimize the "shared memory" device/BTL. I use the
"preferred_protocol" feature to enable/disable
KNEM according to intra/inter socket communication. This optimization increases a
"IMB pingping benchmark" bandwidth by ~36%.
----
The next step is now to use the "preferred protocol" feature with openib ( with
many IB cards)
Attached 2 patches:
1) BTL_preferred.patch:
introduces the new preferred protocol interface
2) SM_KNEM_intra_socket.patch:
defines the preferred protocol for the sm btl
Note: Since the "ess" framework can't give us the "socket locality
information", I used hitopo that has been proposed in an RFC
some times ago:
http://www.open-mpi.org/community/lists/devel/2010/11/8677.php
<BTL_preferred.path><SM_KNEM_intra_socket.patch>_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
"I disapprove of what you say, but I will defend to the death your right to say
it"
-- Evelyn Beatrice Hall
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
