Sounds like a bug in that release - you may have to wait for OMPI v5.0 for a fix.
On Mar 1, 2021, at 7:43 AM, Luis Cebamanos via users <users@lists.open-mpi.org <mailto:users@lists.open-mpi.org> > wrote: I am afraid --map-by ppr:32:socket --bind-to core --cpu-list 0,2,4,6... somehow conflicts internally with other policies. I have also tried with --cpu-set with identical results. Probably rankfile is my only option too. On 28/02/2021 22:44, Ralph Castain via users wrote: The only way I know of to do what you want is --map-by ppr:32:socket --bind-to core --cpu-list 0,2,4,6,... where you list out the exact cpus you want to use. On Feb 28, 2021, at 9:58 AM, Luis Cebamanos via users <users@lists.open-mpi.org <mailto:users@lists.open-mpi.org> > wrote: I could do --map-by ppr:32:socket:PE=1 --bind-to core (output below) but I cannot see the way of mapping every 2 cores 0,2,4,.... [epsilon110:1489563] MCW rank 0 bound to socket 0[core 0[hwt 0-1]]: [BB/../../.. /../../../../../../../../../../../../../../../../../../../../../../../../../../. ./../../../../../../../../../../../../../../../../../../../../../../../../../../ ../../../../../../..][../../../../../../../../../../../../../../../../../../../. ./../../../../../../../../../../../../../../../../../../../../../../../../../../ ../../../../../../../../../../../../../../../../../..] [epsilon110:1489563] MCW rank 1 bound to socket 0[core 1[hwt 0-1]]: [../BB/../.. /../../../../../../../../../../../../../../../../../../../../../../../../../../. ./../../../../../../../../../../../../../../../../../../../../../../../../../../ ../../../../../../..][../../../../../../../../../../../../../../../../../../../. ./../../../../../../../../../../../../../../../../../../../../../../../../../../ ../../../../../../../../../../../../../../../../../..] On 28/02/2021 16:24, Ralph Castain via users wrote: Did you read the documentation on rankfile? The "slot=N" directive saids to "put this proc on core N". In your file, you stipulate that rank 0 is to be placed solely on core 0 rank 1 is to be placed solely on core 2 etc. That is not what you asked for in your mpirun cmd. You asked that each proc be mapped to TWO cores (PE=2) or FOUR threads (PE=4 with bind-to HWT). If you wanted that same thing in a rankfile, it should have said rank 0 slots=0-1 rank 1 slots=2-3 etc. Hence the difference. I was simply correcting your mpirun cmd line as you said you wanted two CORES, and that isn't guaranteed if you are stipulating things in terms of HWTs as not every machine has two HWTs/core. On Feb 28, 2021, at 7:43 AM, Luis Cebamanos via users <users@lists.open-mpi.org <mailto:users@lists.open-mpi.org> > wrote: Hi Ralph, Thanks for this, however --map-by ppr:32:socket:PE=2 --bind-to core reports the same binding than --map-by ppr:32:socket:PE=4 --bind-to hwthread: [epsilon104:2861230] MCW rank 0 bound to socket 0[core 0[hwt 0-1]], socket 0[core 1[hwt 0-1]]: [BB/BB/../../../../ ../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../ ../../../../../../../../../../../../../../../../../../../..][../../../../../../../../../../../../../../../../../.. /../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../.. /../../../../../../../..] [epsilon104:2861230] MCW rank 1 bound to socket 0[core 2[hwt 0-1]], socket 0[core 3[hwt 0-1]]: [../../BB/BB/../../ ../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../ ../../../../../../../../../../../../../../../../../../../..][../../../../../../../../../../../../../../../../../.. /../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../.. /../../../../../../../..] [epsilon104:2861230] MCW rank 2 bound to socket 0[core 4[hwt 0-1]], socket 0[core 5[hwt 0-1]]: [../../../../BB/BB/ ../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../ ../../../../../../../../../../../../../../../../../../../..][../../../../../../../../../../../../../../../../../.. /../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../.. /../../../../../../../..] And this is still different from the output produce using the rankfile. Cheers, Luis On 28/02/2021 14:06, Ralph Castain via users wrote: Your command line is incorrect: --map-by ppr:32:socket:PE=4 --bind-to hwthread should be --map-by ppr:32:socket:PE=2 --bind-to core On Feb 28, 2021, at 5:57 AM, Luis Cebamanos via users <users@lists.open-mpi.org <mailto:users@lists.open-mpi.org> > wrote: I should have said, "I would like to run 128 MPI processes on 2 nodes" and not 64 like I initially said... On Sat, 27 Feb 2021, 15:03 Luis Cebamanos, <luic...@gmail.com <mailto:luic...@gmail.com> > wrote: Hello OMPI users, On 128 core nodes, 2 sockets x 64 cores/socket (2 hwthreads/core) , I am trying to match the behavior of running with a rankfile with manual mapping/ranking/binding. I would like to run 64 MPI processes on 2 nodes, 1 MPI process every 2 cores. This is, I want to run 32 MPI processes per socket on 2 128-core nodes. My mapping should be something like: Node 0 ===== rank 0 - core 0 rank 1 - core 2 rank 3 - core 4 ... rank 63 - core 126 Node 1 ==== rank 64 - core 0 rank 65 - core 2 rank 66 - core 4 ... rank 127- core 126 If I use a rankfile: rank 0=epsilon102 slot=0 rank 1=epsilon102 slot=2 rank 2=epsilon102 slot=4 rank 3=epsilon102 slot=6 rank 4=epsilon102 slot=8 rank 5=epsilon102slot=10 .... rank 123=epsilon103 slot=118 rank 124=epsilon103 slot=120 rank 125=epsilon103 slot=122 rank 126=epsilon103 slot=124 rank 127=epsilon103 slot=126 My --report-binding looks like: [epsilon102:2635370] MCW rank 0 bound to socket 0[core 0[hwt 0-1]]: [BB/../../.. /../../../../../../../../../../../../../../../../../../../../../../../../../../. ./../../../../../../../../../../../../../../../../../../../../../../../../../../ ../../../../../../..][../../../../../../../../../../../../../../../../../../../. ./../../../../../../../../../../../../../../../../../../../../../../../../../../ ../../../../../../../../../../../../../../../../../..] [epsilon102:2635370] MCW rank 1 bound to socket 0[core 2[hwt 0-1]]: [../../BB/.. /../../../../../../../../../../../../../../../../../../../../../../../../../../. ./../../../../../../../../../../../../../../../../../../../../../../../../../../ ../../../../../../..][../../../../../../../../../../../../../../../../../../../. ./../../../../../../../../../../../../../../../../../../../../../../../../../../ ../../../../../../../../../../../../../../../../../..] [epsilon102:2635370] MCW rank 2 bound to socket 0[core 4[hwt 0-1]]: [../../../.. /BB/../../../../../../../../../../../../../../../../../../../../../../../../../. ./../../../../../../../../../../../../../../../../../../../../../../../../../../ ../../../../../../..][../../../../../../../../../../../../../../../../../../../. ./../../../../../../../../../../../../../../../../../../../../../../../../../../ ../../../../../../../../../../../../../../../../../..] However, I cannot match this report-binding output by manually using --map-by and --bind-to. I had the impression that this will be the same: mpirun -np $SLURM_NTASKS --report-bindings --map-by ppr:32:socket:PE=4 --bind-to hwthread But this output is not quite the same: [epsilon102:2631529] MCW rank 0 bound to socket 0[core 0[hwt 0-1]], socket 0[cor e 1[hwt 0-1]]: [BB/BB/../../../../../../../../../../../../../../../../../../../. ./../../../../../../../../../../../../../../../../../../../../../../../../../../ ../../../../../../../../../../../../../../../..][../../../../../../../../../../. ./../../../../../../../../../../../../../../../../../../../../../../../../../../ ../../../../../../../../../../../../../../../../../../../../../../../../../../..] [epsilon102:2631529] MCW rank 1 bound to socket 0[core 2[hwt 0-1]], socket 0[cor e 3[hwt 0-1]]: [../../BB/BB/../../../../../../../../../../../../../../../../../. ./../../../../../../../../../../../../../../../../../../../../../../../../../../ ../../../../../../../../../../../../../../../..][../../../../../../../../../../. ./../../../../../../../../../../../../../../../../../../../../../../../../../../ ../../../../../../../../../../../../../../../../../../../../../../../../../../..] What am I missing to match the rankfile behavior? Regarding performance, what difference does it make between the first and the second outputs? Thanks for your help! Luis
