George Bosilca wrote:
On Mar 4, 2009, at 14:44 , Eugene Loh wrote:
Let me try another thought here. Why do we have BTL sendi functions
at all? I'll make an assertion and would appreciate feedback: a
BTL sendi function contributes nothing to optimizing send latency.
To optimize send latency in the "immediate" case, we need *ONLY* PML
work.
Because otherwise you will have to make 2 BTL calls instead of one
What's wrong with that? It's "just a function call", and OMPI is
littered with those.
plus one extra memcpy (or not depending on your network).
Just to check my understanding: sm does not benefit here. Right? So,
it makes no sense really for sm to have a sendi function.
I remain close to giving up, but still am trying to understand the PML
somewhat. I guess there are several send-latency optimizations one
could consider:
1) Not populating the PML send request if the send can be completed
"immediately". This makes sense for BTLs (like sm) that are very fast
(so that shaving off instructions and memory operations makes a
difference) and for messages that are very, very short (one or few
cachelines). It does not rely on having a sendi function, which knows
nothing about PML send requests anyhow.
2) Not copying the user data into a buffer. This makes sense for BTLs
(unlike sm) that can move the data "directly" and for "intermediate"
message sizes (below the eager limit, but still big enough that saving a
memcpy makes a difference). It does rely on having a sendi function.
I guess I've been focussing on #1. I thought that was related to sendi,
but now it seems sendi is related more to #2. So, if I understand
correctly (unclear if this is the case), we're both right. I'm right in
that pruning the PML stack (specifically, populating the send request)
is unrelated to the existence of sendi. You're right in that there is
still a legitimate role for sendi for some BTLs (sm not among them).
First you will have to call btl_alloc to get back a descriptor with
some BTL memory attached to it. The you will put your data (including
the header) in this memory and once ready call btl_send. With sendi
there is only one call from the PML into the BTL, but this time it is
the BTL responsibility to prepare the data that will be sent.
I'm churning a lot and not making much progress, but I'll try
chewing on that idea (unless someone points out it's utterly
ridiculous). I'll look into having PML ignore sendi functions
altogether and just make the "send-immediate" path work fast with
normal send functions. If that works, then we can get rid of sendi
functions and hopefully have a solution that makes sense for everyone.
This is utterly ridiculous (I hope you really expect someone to say it).
It's fine to say that. As you can see, I'll push back when I don't
understand why it's ridiculous.
As I said before, SM is only one of the networks supported by Open
MPI. Independent on how much I would like to have better shared
memory performance, I will not agree with any PML modifications that
are SM oriented. We did that in the past with other BTLs and it
turned out to be a bad idea, so I'm clearly not in favor of doing the
same mistake twice.
Regarding the sendi there are at least 3 networks that can take
advantage of it: Portals, MX and Sicortex. Some of them do this right
now, some others in the near future. Moreover, for these particular
networks there is no way to avoid extra overhead without this feature
(for very obscure reasons such as non contiguous pieces of memory
only known by the BTL that can decrease the number of network
operations).
