Hi Mikael and other HPX experts,
Thanks for your suggestions! Unfortunately they did not improve things
for me. To be clear, the only thing I don't understand is the binding
reported by `--hpx:print-bind`. What I do understand is:
- The binding of MPI process ranks to numa nodes, reported by mpirun's
`--display-map`. Process rank 0 is bound to numa node 0, process rank 1
is bound to numa node 1, etc. This is exactly how I want things to be.
- Relation between HPX localities and MPI ranks, printed from my own
code: hpx::get_locality_id() == hpx::util::mpi_environment::rank() ==
std::getenv("OMPI_COMM_WORLD_RANK"). This implies that HPX localities
are ordered the same way as the MPI processes. Locality 0 should be
bound to numa node 0, locality 1 should be bound to numa node 1, etc.
This is exactly how I want things to be.
The weird thing is that, according to `--hpx:print-bind`, localities are
scattered over the numa nodes. Locality 0 always ends up at the first
numa node, but the other ones are bound to numa nodes in a seemingly
random order. When performing scaling tests over numa nodes, the
resulting graphs show artifacts which could be the result of HPX
localities not being ordered according to increasing memory latencies.
At the moment I can only think of `--hpx:print-bind` being wrong, which
is unlikely I guess. But why does it suggest that the localities are
scattered over the numa nodes, when all other information suggests that
they are ordered according to the numa nodes?
Maybe I am just misunderstanding things. To be able to interpret the
results of my scaling tests, I would really like to understand what is
going on.
Thanks in advance for any insights any of you might have for me!
Kor
On 5/21/21 5:02 PM, Simberg Mikael wrote:
> Hi Kor,
>
>
> The nondeterministic nature of your problem is a bit worrying, and I
> don't have any insight into that. However, there's an alternative way to
> set the bindings as well. Would you mind trying the
> --hpx:use-process-mask option to see if you get the expected bindings?
> By default HPX tries to reconstruct the bindings based on various
> environment variables, but if you pass --hpx:use-process-mask it will
> use the process mask that srun/mpi/others typically set, and only spawn
> worker threads on cores in the process mask. Note that the default, even
> with --hpx:use-process-mask, is still to only spawn one worker thread
> per core (not per hyperthread), so if you want exactly the binding you
> ask for with mpirun you should also add --hpx:threads=all.
>
>
> Mikael
>
> ------------------------------------------------------------------------
> *From:* hpx-users-boun...@stellar-group.org
> <hpx-users-boun...@stellar-group.org> on behalf of Kor de Jong
> <k.dejo...@uu.nl>
> *Sent:* Friday, May 21, 2021 4:25:29 PM
> *To:* hpx-users@stellar-group.org
> *Subject:* {Spam?} [hpx-users] Assign HPX localities to NUMA nodes, in
> order
> Dear HPX-experts,
>
> I am trying to spawn 8 hpx processes on a cluster node with 8 numa
> nodes, containing 6 real cpu cores each. All seems well, but the output
> of `--hpx:print-bind` confuses me.
>
> I am using slurm (sbatch command) and openmpi (mpirun command in sbatch
> script). The output of mpirun's `--display-map` makes complete sense.
> All 8 process ranks get assigned to the 6 cores in the 8 numa nodes, in
> order. Process rank 0 is on the first numa node, etc.
>
> The output of `--hpx:print-bind` seems not in sync with this. There is a
> correspondence between mpi ranks and hpx locality ids, but the mapping
> of hpx localities to cpu cores is different now. For example, it seems
> that locality 1 is not on the second numa node (as per mpirun's
> `--display-map`), but on the 7-th (as per hpx's `--print-bind`). Also,
> the output of `--print-bind` differs per invocation.
>
> It is important for me that hpx localities are assigned to numa nodes in
> order. Localities with similar IDs communicate more with each other than
> with other localities.
>
> I have attached the slurm script and outputs mentioned above. Does
> somebody maybe have an idea what is going on and how to fix things? Does
> hpx maybe re-assign the ranks upon initialization? If so, can I
> influence this to make this ordering similar to the ordering of the numa
> nodes?
>
> BTW, I am pretty sure all this worked fine some time ago, when I was
> still using an earlier version of HPX, another version of MPI, and
> started HPX processes using srun instead of mpirun.
>
> Thanks for any info!
>
> Kor
>
>
> _______________________________________________
> hpx-users mailing list
> hpx-users@stellar-group.org
> https://mail.cct.lsu.edu/mailman/listinfo/hpx-users
>
_______________________________________________
hpx-users mailing list
hpx-users@stellar-group.org
https://mail.cct.lsu.edu/mailman/listinfo/hpx-users
