On Thu, May 17, 2007 at 12:30:51PM -0400, Patrick Geoffray wrote:
> Jeff Squyres wrote:
> > Moving to devel; this question seems worthwhile to push out to the
> > general development community.
> >
> > I've been coming across an increasing number of customers and other
> > random OMPI users who use system(). So if there's zero impact on
> > performance and it doesn't make the code [more] incredibly horrible
> > [than it already is], I'm in favor of this change.
>
> I will sound like a broken record, but this is the type of things that a
> MPI implementation should not care about. At least not in the (common)
> protocol layer. That's why the BTL-level abstraction is a bad one,
> device-specific problems bubble up instead of staying hidden in
> device-specific code.
I am glad I provided one more point in favor of your argument :)
>
> > On May 17, 2007, at 7:00 AM, Gleb Natapov wrote:
>
> >> problem is that granularity of registration is HW page (4K), so last
>
> What about huge pages ?
I am saying this again I don't try to solve all problems of interconnects that
were designed by people who ignored 30 or so years of OS design evolution.
Huge page usage is not transparent in linux. If programmer decided to
use it he should understand the consequences.
>
> >> page of the buffer may contain also other application's data and user
> >> may be unaware of this and be very surprised by SIGSEGV. If pipeline
>
> How can a process get a segmentation fault by accessing a page mapped in
> its own address space ?
In child process VMAs, that were registered in parent process, no longer
mapped.
>
> >> so this situation will be avoided. It should have zero impact on
> >> performance. What do you think? How common for MPI applications to
> >> fork()?
>
> The only safe way to support fork() with pinned page is to force the
> duplication of pages at fork time. It makes fork much more expensive,
> but fork should not be in the critical path of HPC applications anyway.
>
It also increases a chance of fork() failure. Otherwise I agree with
you. I even started to write patch once to duplicate only firs and last
pages of pinned region. The chances of such patch to be accepted into
linux less then zero though.
> Playing with registration cache is playing with fire.
The change I propose will not solve any problem if registration cache is
in use.
--
Gleb.
