Re: [OMPI users] Avoiding the memory registration costs by having memory always registered, is it possible with Linux ?

mlock() and mlockall() only guarantee that pages won't be swapped out to the disk. However, they don't prevent virtual pages from moving to other physical pages (for instance during migration between NUMA nodes), which breaks memory registration. At least this was true a couple years ago, I didn't

Re: [OMPI users] Avoiding the memory registration costs by having memory always registered, is it possible with Linux ?

Thanks Jeff and Alex for your answers and comments. mlockall(), especially with the MCL_FUTURE argument is indeed interesting. Thanks Jeff for your clarification of what memory registration really means (e.g. locking and telling the network stack the virtual to physical mapping). Also

Re: [OMPI users] Avoiding the memory registration costs by having memory always registered, is it possible with Linux ?

> On Jun 20, 2016, at 4:15 PM, Audet, Martin > wrote: > > But now since we have to live with memory registration issues, what changes > should be done to standard Linux distro so that Open MPI can best use a > recent Mellanox Infiniband network ? > > I guess

Re: [OMPI users] Avoiding the memory registration costs by having memory always registered, is it possible with Linux ?

On Jun 20, 2016, at 4:27 PM, Alex A. Granovsky wrote: > > Would be the use of mlockall helpful for this approach? That's an interesting idea; I didn't know about the existence of mlockall(MCL_FUTURE). It has a few drawbacks, of course (e.g., processes can't shrink),

Re: [OMPI users] Avoiding the memory registration costs by having memory always registered, is it possible with Linux ?

Would be the use of mlockall helpful for this approach? From: Audet, Martin Sent: Monday, June 20, 2016 11:15 PM To: mailto:us...@open-mpi.org Subject: Re: [OMPI users] Avoiding the memory registration costs by having memory always registered, is it possible with Linux ? Thanks Jeff for your

Re: [OMPI users] Avoiding the memory registration costs by having memory always registered, is it possible with Linux ?

Thanks Jeff for your answer. It is sad that the approach I mentioned of having all memory registered for user process on cluster nodes didn't become more popular. I still believe that such an approach would shorten the executed code path in MPI libraries, reduce message latency, increase the

Re: [OMPI users] Avoiding the memory registration costs by having memory always registered, is it possible with Linux ?

Greetings Martin. Such approaches have been discussed in the past. Indeed, I'm pretty sure that I've heard of some non-commodity systems / network stacks that do this kind of thing. Such approaches have not evolved in the commodity Linux space, however. This kind of support would need