Arun, First Open MPI selects a pml for **all** the MPI tasks (for example, pml/ucx or pml/ob1)
Then, if pml/ob1 ends up being selected, a btl component (e.g. btl/uct, btl/vader) is used for each pair of MPI tasks (tasks on the same node will use btl/vader, tasks on different nodes will use btl/uct) Note that if UCX is available, pml/ucx takes the highest priority, so no btl is involved (in your case, if means intra-node communications will be handled by UCX and not btl/vader). You can force ob1 and try different combinations of btl with mpirun --mca pml ob1 --mca btl self,<btl1>,<btl2> ... I expect pml/ucx is faster than pml/ob1 with btl/uct for inter node communications. I have not benchmarked Open MPI for a while and it is possible btl/vader outperforms pml/ucx for intra nodes communications, so if you run on a small number of Infiniband interconnected nodes with a large number of tasks per node, you might be able to get the best performances by forcing pml/ob1. Bottom line, I think it is best for you to benchmark your application and pick the combination that leads to the best performances, and you are more than welcome to share your conclusions. Cheers, Gilles On Mon, Mar 6, 2023 at 3:12 PM Chandran, Arun via users < users@lists.open-mpi.org> wrote: > [Public] > > Hi Folks, > > I can run benchmarks and find the pml+btl (ob1, ucx, uct, vader, etc) > combination that gives the best performance, > but I wanted to hear from the community about what is generally used in > "__high_core_count_intra_node_" cases before jumping into conclusions. > > As I am a newcomer to openMPI I don't want to end up using a combination > only because it fared better in a benchmark (overfitting?) > > Or the choice of pml+btl for the 'intranode' case is not so important as > openmpi is mainly used in 'internode' and the 'networking-equipment' > decides the pml+btl? (UCX for IB) > > --Arun > > -----Original Message----- > From: users <users-boun...@lists.open-mpi.org> On Behalf Of Chandran, > Arun via users > Sent: Thursday, March 2, 2023 4:01 PM > To: users@lists.open-mpi.org > Cc: Chandran, Arun <arun.chand...@amd.com> > Subject: [OMPI users] What is the best choice of pml and btl for intranode > communication > > Hi Folks, > > As the number of cores in a socket is keep on increasing, the right > pml,btl (ucx, ob1, uct, vader, etc) that gives the best performance in > "intra-node" scenario is important. > > For openmpi-4.1.4, which pml, btl combination is the best for intra-node > communication in the case of higher core count scenario? (p-to-p as well as > coll) and why? > Does the answer for the above question holds good for the upcoming ompi5 > release? > > --Arun >
