Re: [OMPI users] self-messaging questions
Hi Jeff Thank you very much for your detailed answers. This is the type of information that is hard to find, even if you read "MPI The Complete Reference" over and over, highlighting and writing notes on the margins. Considering what you wrote, and the perhaps only marginal advantage of copying vs. self-messaging with non-blocking p2p routines (persistent in my case), I will stick to the latter. This also makes the code simpler. Regards, Gus Correa Jeff Squyres wrote: On Oct 14, 2010, at 7:50 PM, Gus Correa wrote: *1) Can a process send a message to itself?* The book "MPI: The complete reference, Vol. 1, 2nd Ed.", by Snir et al., page 42, section 2.9.9 (Comments on Receive) says: "Therefore, it is unsafe and non-portable to send self-messages with the standard mode, blocking send and receive operations described so far, since this may lead to deadlock." Correct. *2) Would it be safe and portable to send messages to self, if one uses non-blocking send and recv (MPI_Isend, MPI_Irecv), or with persistent communication (MPI_Init_[Send/Recv], MPI_Startall, MPI_Waitall) ?* Yes. On the other hand, the OpenMPI FAQ seems to say something different, i.e., that self-messaging is OK, as long as the "self" BTL is turned on: http://www.open-mpi.org/faq/?category=all#tcp-btl Also correct. We made a software engineering choice with the BTLs (i.e., the OB1-based transports) that they would not have the ability to send to themselves. Instead, we created a "self" BTL that has all of the logic for sending messages to myself. In this way, we didn't have to replicate all of that logic in every other BTL. So it's not really a performance optimization -- it's really a software engineering issue (i.e., avoid code duplication). *3) Is this a particular feature or extension provided by OpenMPI, beyond the MPI standard?* Nope. Sending to self is something that the MPI standard implicitly guarantees must work. *4) If I write a program with self-messaging, will it be portable and run safely when compiled with other MPI implementations?* Yes. Provided you use non-blocking communications, per #2. Of course, I can use an "if" conditional to copy the data to the output buffer, instead of sending the message to self, although this makes the code somewhat ugly. This brings up yet another question: *5) Which one is better/faster: sending messages to self, or copying the data to the output buffer?* If the copy is a straight memcpy, it's probably marginally faster to use memcpy only because you're not making as many function calls to get down into the MPI progression engine. We didn't really take any pains to highly optimize the send-to-self case, but it should perform well. You might want to benchmark memcpy vs. MPI_Sendrecv() to self (for example) and see if there's a noticeable difference.
Re: [OMPI users] self-messaging questions
On Oct 14, 2010, at 7:50 PM, Gus Correa wrote: > *1) Can a process send a message to itself?* > > The book "MPI: The complete reference, Vol. 1, 2nd Ed.", by Snir et al., > page 42, section 2.9.9 (Comments on Receive) says: > > "Therefore, it is unsafe and non-portable to send self-messages with the > standard mode, blocking send and receive operations described so far, > since this may lead to deadlock." Correct. > *2) Would it be safe and portable to send messages to self, if one uses > non-blocking send and recv (MPI_Isend, MPI_Irecv), or with persistent > communication (MPI_Init_[Send/Recv], MPI_Startall, MPI_Waitall) ?* Yes. > On the other hand, the OpenMPI FAQ seems to say something different, > i.e., that self-messaging is OK, as long as the > "self" BTL is turned on: > > http://www.open-mpi.org/faq/?category=all#tcp-btl Also correct. We made a software engineering choice with the BTLs (i.e., the OB1-based transports) that they would not have the ability to send to themselves. Instead, we created a "self" BTL that has all of the logic for sending messages to myself. In this way, we didn't have to replicate all of that logic in every other BTL. So it's not really a performance optimization -- it's really a software engineering issue (i.e., avoid code duplication). > *3) Is this a particular feature or extension provided by OpenMPI, > beyond the MPI standard?* Nope. Sending to self is something that the MPI standard implicitly guarantees must work. > *4) If I write a program with self-messaging, will it > be portable and run safely when compiled with other MPI implementations?* Yes. Provided you use non-blocking communications, per #2. > Of course, I can use an "if" conditional to copy the data to > the output buffer, instead of sending the message to self, > although this makes the code somewhat ugly. > This brings up yet another question: > > *5) Which one is better/faster: sending messages to self, > or copying the data to the output buffer?* If the copy is a straight memcpy, it's probably marginally faster to use memcpy only because you're not making as many function calls to get down into the MPI progression engine. We didn't really take any pains to highly optimize the send-to-self case, but it should perform well. You might want to benchmark memcpy vs. MPI_Sendrecv() to self (for example) and see if there's a noticeable difference. -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
[OMPI users] self-messaging questions
Dear OpenMPI Pros These questions may have appeared before. Anyway, I am somewhat confused about self-messaging in MPI and OpenMPI. Questions: *1) Can a process send a message to itself?* The book "MPI: The complete reference, Vol. 1, 2nd Ed.", by Snir et al., page 42, section 2.9.9 (Comments on Receive) says: "Therefore, it is unsafe and non-portable to send self-messages with the standard mode, blocking send and receive operations described so far, since this may lead to deadlock." *2) Would it be safe and portable to send messages to self, if one uses non-blocking send and recv (MPI_Isend, MPI_Irecv), or with persistent communication (MPI_Init_[Send/Recv], MPI_Startall, MPI_Waitall) ?* On the other hand, the OpenMPI FAQ seems to say something different, i.e., that self-messaging is OK, as long as the "self" BTL is turned on: http://www.open-mpi.org/faq/?category=all#tcp-btl *3) Is this a particular feature or extension provided by OpenMPI, beyond the MPI standard?* In other words: *4) If I write a program with self-messaging, will it be portable and run safely when compiled with other MPI implementations?* Of course, I can use an "if" conditional to copy the data to the output buffer, instead of sending the message to self, although this makes the code somewhat ugly. This brings up yet another question: *5) Which one is better/faster: sending messages to self, or copying the data to the output buffer?* (I would guess with the "sm" btl this would make no difference, but who knows ...) Thank you, Gus Correa