This message is mostly for Nathan, but figured I would go with the wider distribution. I have noticed some different behaviour that I assume started with this change.
https://github.com/open-mpi/ompi/commit/4bf7a207e90997e75ba1c60d9d191d9d96402d04 I am noticing that the openib BTL will also be used for on-node communication even though the sm (or smcuda) BTL is also available. I think with the aforementioned change that the openib BTL is listed as an available BTL that supports RDMA. While looking through the debugger and looking at the bml_endpoint, it appears that the sm BTL is listed as the eager and send BTL, but the openib is listed as the RDMA btl. Looking at the logic in pml_ob1_sendreq.h, it looks like we can end up selecting the openib btl for some of the communication. I ran with some various verbosity and saw that this was happening. With v1.8, we only appear to use the sm (or smcuda) btl. I am wondering if this was intentional with this change or maybe a side effect. Rolf ----------------------------------------------------------------------------------- This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. -----------------------------------------------------------------------------------
