Hi Stefan, Stefan Riha <[email protected]> skribis:
> I unpacked the pack on my laptop and the mpirun command works (with `-np 8`)! > Then unpacked it on a different workstation (same Archlinux install as > on the previous target), and it did not work, but this time there was > a backtrace (see below, no idea why this error messages did not appear > on the original machine). > > Given that it works in the VM and a laptop, and does not work on two > workstations, I thought maybe it's the selected transport (or whatever they > call it) and tried > > ``` > /gnu/store/8hfv0bjynywx6visp8nq01vqdaqskx3b-openmpi-4.1.6/bin/mpirun -np 8 > --allow-run-as-root -mca pml ucx ./mpas_atmosphere > ``` > i.e. select `pml ucx` explicitly, and it works. Whereas if I do not specify > anything, it chooses: > ``` > selected cm best priority 25 > ``` > Both problematic workstations have Intel E810 nics installed, with the > `ice` and `irdma` drivers installed for RDMA (sorry I should have > mentioned that earlier, but I do not understand why this would impact > a computation on a single node i.e. without any inter-node > communication?). Could that be the reason why it chooses `pml cm`? > According to this site: > https://docs.open-mpi.org/en/v5.0.3/release-notes/networks.html > """ > cm supports a smaller number of networks (and they cannot be used together), > but may provide better overall MPI performance by utilizing MTL (“Matching > Transport Layer”) plugins: > * OpenFabrics Interfaces (“libfabric” tag matching) I don’t have experience with these interconnects. What I do know is that we try hard for Open MPI to pick the right thing, which it already mostly does by itself: https://hpc.guix.info/blog/2019/12/optimized-and-portable-open-mpi-packaging/ There’s currently a patch in Guix to adjust priorities: https://codeberg.org/guix/guix/src/commit/50428b90ef6ca6269a9034f32d3d666ebea7f037/gnu/packages/patches/openmpi-mtl-priorities.patch Could it be that we also need to adjust the default ‘ucx’ priority relative to ‘cm’? Did you try to get guidance from the Open MPI people? Thanks for your feedback, Ludo’.
