Definitely a side-effect though it could be beneficial in some cases as the RDMA engine in the HCA may be faster than using memcpy (larger than a certain size). I don't know how to best fix this as I need all RDMA capable BTLs to listed for RMA. I though about adding another list to track BTLs that have both RMA and atomics but that would increase the memory footprint of Open MPI by a factor of nranks.
-Nathan On Thu, Feb 26, 2015 at 11:59:41PM +0000, Rolf vandeVaart wrote: > This message is mostly for Nathan, but figured I would go with the wider > distribution. I have noticed some different behaviour that I assume > started with this change. > > > https://github.com/open-mpi/ompi/commit/4bf7a207e90997e75ba1c60d9d191d9d96402d04 > > I am noticing that the openib BTL will also be used for on-node > communication even though the sm (or smcuda) BTL is also available. I > think with the aforementioned change that the openib BTL is listed as an > available BTL that supports RDMA. While looking through the debugger and > looking at the bml_endpoint, it appears that the sm BTL is listed as the > eager and send BTL, but the openib is listed as the RDMA btl. Looking at > the logic in pml_ob1_sendreq.h, it looks like we can end up selecting the > openib btl for some of the communication. I ran with some various > verbosity and saw that this was happening. With v1.8, we only appear to > use the sm (or smcuda) btl. > > I am wondering if this was intentional with this change or maybe a side > effect. > > Rolf > > ---------------------------------------------------------------------- > > This email message is for the sole use of the intended recipient(s) and > may contain confidential information. Any unauthorized review, use, > disclosure or distribution is prohibited. If you are not the intended > recipient, please contact the sender by reply email and destroy all copies > of the original message. > > ---------------------------------------------------------------------- > _______________________________________________ > devel mailing list > [email protected] > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2015/02/17065.php
pgp4ZPssyGRwR.pgp
Description: PGP signature
