mlock() and mlockall() only guarantee that pages won't be swapped out to
the disk. However, they don't prevent virtual pages from moving to other
physical pages (for instance during migration between NUMA nodes), which
breaks memory registration. At least this was true a couple years ago, I
Thanks Jeff and Alex for your answers and comments.
mlockall(), especially with the MCL_FUTURE argument is indeed interesting.
Thanks Jeff for your clarification of what memory registration really means
(e.g. locking and telling the network stack the virtual to physical mapping).
> On Jun 20, 2016, at 4:15 PM, Audet, Martin
> But now since we have to live with memory registration issues, what changes
> should be done to standard Linux distro so that Open MPI can best use a
> recent Mellanox Infiniband network ?
> I guess
On Jun 20, 2016, at 4:27 PM, Alex A. Granovsky wrote:
> Would be the use of mlockall helpful for this approach?
That's an interesting idea; I didn't know about the existence of
It has a few drawbacks, of course (e.g., processes can't shrink),
Would be the use of mlockall helpful for this approach?
From: Audet, Martin
Sent: Monday, June 20, 2016 11:15 PM
Subject: Re: [OMPI users] Avoiding the memory registration costs by having
memory always registered, is it possible with Linux ?
Thanks Jeff for your
Thanks Jeff for your answer.
It is sad that the approach I mentioned of having all memory registered for
user process on cluster nodes didn't become more popular.
I still believe that such an approach would shorten the executed code path in
MPI libraries, reduce message latency, increase the
Such approaches have been discussed in the past. Indeed, I'm pretty sure that
I've heard of some non-commodity systems / network stacks that do this kind of
Such approaches have not evolved in the commodity Linux space, however. This
kind of support would need
After reading a little the FAQ on the methods used by Open MPI to deal with
memory registration (or pinning) with Infiniband adapter, it seems that we
could avoid all the overhead and complexity of memory
registration/deregistration, registration cache access and update, memory