I made progress: short story is I think openmpi picks the wrong transport, or the transport that scores highest is somehow broken on my workstations.
I unpacked the pack on my laptop and the mpirun command works (with `-np 8`)! Then unpacked it on a different workstation (same Archlinux install as on the previous target), and it did not work, but this time there was a backtrace (see below, no idea why this error messages did not appear on the original machine). Given that it works in the VM and a laptop, and does not work on two workstations, I thought maybe it's the selected transport (or whatever they call it) and tried ``` /gnu/store/8hfv0bjynywx6visp8nq01vqdaqskx3b-openmpi-4.1.6/bin/mpirun -np 8 --allow-run-as-root -mca pml ucx ./mpas_atmosphere ``` i.e. select `pml ucx` explicitly, and it works. Whereas if I do not specify anything, it chooses: ``` selected cm best priority 25 ``` Both problematic workstations have Intel E810 nics installed, with the `ice` and `irdma` drivers installed for RDMA (sorry I should have mentioned that earlier, but I do not understand why this would impact a computation on a single node i.e. without any inter-node communication?). Could that be the reason why it chooses `pml cm`? According to this site: https://docs.open-mpi.org/en/v5.0.3/release-notes/networks.html """ cm supports a smaller number of networks (and they cannot be used together), but may provide better overall MPI performance by utilizing MTL (“Matching Transport Layer”) plugins: * OpenFabrics Interfaces (“libfabric” tag matching) ... """ I saw that `...-libfabric-1.22.0` is in the pack, and I also saw `ofi` appear in the mpirun logs: ``` [xxx:07447] select: initializing btl component ofi [xxx:07445] select: init of component ofi returned success ``` I don't mind using `pml ucx`, but the end goal is of course to use both worstations with inter-node communication, and ideally I'd like to use the fastest transport. Do you have any experience with Guix openmpi on Intel E810 nic hardware? Thanks in advance for your help. Stefan
