Hi Jeff
Thank you very much for your detailed answers.
This is the type of information that is hard to find,
even if you read "MPI The Complete Reference" over and over,
highlighting and writing notes on the margins.
Considering what you wrote, and the perhaps only marginal advantage of
copying vs. self-messaging with non-blocking
p2p routines (persistent in my case), I will stick to the latter.
This also makes the code simpler.
Regards,
Gus Correa
Jeff Squyres wrote:
On Oct 14, 2010, at 7:50 PM, Gus Correa wrote:
*1) Can a process send a message to itself?*
The book "MPI: The complete reference, Vol. 1, 2nd Ed.", by Snir et al.,
page 42, section 2.9.9 (Comments on Receive) says:
"Therefore, it is unsafe and non-portable to send self-messages with the
standard mode, blocking send and receive operations described so far,
since this may lead to deadlock."
Correct.
*2) Would it be safe and portable to send messages to self, if one uses
non-blocking send and recv (MPI_Isend, MPI_Irecv), or with persistent
communication (MPI_Init_[Send/Recv], MPI_Startall, MPI_Waitall) ?*
Yes.
On the other hand, the OpenMPI FAQ seems to say something different,
i.e., that self-messaging is OK, as long as the
"self" BTL is turned on:
http://www.open-mpi.org/faq/?category=all#tcp-btl
Also correct.
We made a software engineering choice with the BTLs (i.e., the OB1-based transports) that
they would not have the ability to send to themselves. Instead, we created a
"self" BTL that has all of the logic for sending messages to myself. In this
way, we didn't have to replicate all of that logic in every other BTL.
So it's not really a performance optimization -- it's really a software
engineering issue (i.e., avoid code duplication).
*3) Is this a particular feature or extension provided by OpenMPI,
beyond the MPI standard?*
Nope. Sending to self is something that the MPI standard implicitly guarantees
must work.
*4) If I write a program with self-messaging, will it
be portable and run safely when compiled with other MPI implementations?*
Yes. Provided you use non-blocking communications, per #2.
Of course, I can use an "if" conditional to copy the data to
the output buffer, instead of sending the message to self,
although this makes the code somewhat ugly.
This brings up yet another question:
*5) Which one is better/faster: sending messages to self,
or copying the data to the output buffer?*
If the copy is a straight memcpy, it's probably marginally faster to use memcpy only because you're not making as many function calls to get down into the MPI progression engine. We didn't really take any pains to highly optimize the send-to-self case, but it should perform well.
You might want to benchmark memcpy vs. MPI_Sendrecv() to self (for example) and
see if there's a noticeable difference.
