On Thu, 19 Jun 2008, Terry Dontje wrote:
But my concern is not the raw performance of MPI_Iprobe in this case but more
of an interaction between MPI and an application. The concern is if it takes
2 MPI_Iprobes to get to the real message (instead of one) then could this
induce a synchronization delay in an application? That is by the
application not receiving the "real" message in the first MPI_Iprobe it may
decide to do other work while the other processes are potentially blocked
waiting for it to do some communications.
I'd have to agree, which is why I proposed calling opal_progres at the
start of every iProbe. By the way, the 40us time hit sounds high, but
really should only happen if you have the TCP BTL actively talking to a
peer. Or you've hit the high water mark and are checking TCP
communication on the OOB connection.
In fact TCP has the potential to exhibit the same behavior. However, TCP
after each successful poll it empty the socket, so it might read more than
one message. As we have to empty the temporary buffer, we interpret most of
the messages inside, and this is why TCP exhibit a different behavior.
I guess this difference in behavior between the SM BTL and TCP BTL is
disturbing to me. Does just processing one fifo entry per sm_progress call
per connection buying us performance? Would draining the acks be detrimental
to performance? Wouldn't providing the messages at the time they arrived
meet the rule of obviousness to application writers?
I know there is a slippery slope here of saying ok you've read one message
should read more until there is none on the fifo. I believe that is really
debatable and could go either way depending on the application. But ack
messages are not visible to the users. Which is why I was only asking about
draining the ack packets.
Galen could say better than I, but I thought the IB BTL did basically what
you propose -- drain until you have a "real" message. That seems to make
the most sense to me and actually seemed to make IB run better for real
jobs, but what do I know?
Brian
