I'm a newbie and George is a veteran. So, this feels rather like David
and Goliath. (Hmm, David won and became king. Gee, I kinda like
that.) Anyhow...
George Bosilca wrote:
It doesn't sound reasonable to me. There is a reason for this, and I
think it's a good reason. The sendi function work for some devices as
a fast path for sending data, when the network is not flooded.
However, in the case sendi cannot do the job we expect, the fact that
it return the descriptor save us a call (we don't have to do the
alloc call later).
This does not make any sense to me. In what sense are we "saving a
call"? Not in the sense of run-time performance since the BTL is now
having to allocate a descriptor it did not otherwise need. The amount
of work is the same (one descriptor allocation either way), but you're
just pushing that work into the BTLs.
We are certainly not "saving a call" in the sense of reducing source
code. The PML has to have code to allocate a descriptor anyway since
there may not even be any sendi functions. So, the code to allocate the
descriptor is already in the PML. By asking sendi functions to do the
same, you're replicating that code in every sendi function... possibly
multiple times per BTL since a sendi function might have multiple "out
of resource" return paths.
Therefore, in the PML we already have the descriptor and we can hand
it back to the BTL, which give a chance for asynchronous progress
later on. Without this descriptor, the only option the PML have is to
put the PML request in a queue, and to try to send it later, which is
very expensive.
This also makes no sense to me. We're not talking about doing without
the descriptor. The PML is prepared to allocate it anyhow. The issue
is where the descriptor is allocated in the case that sendi functions
exist but cannot succeed. One alternative is to use a single allocation
point in the PML. The other alternative (what we have today) is to
replicate that code out to multiple sites, adding unnecessary source
code and interface arguments.
The PML code is in
https://svn.open-mpi.org/source/xref/ompi_1.3/ompi/mca/pml/ob1/pml_ob1_sendreq.c#mca_pml_ob1_send_request_start_copy
Existing BTL sendi functions are at
https://svn.open-mpi.org/source/xref/ompi_1.3/ompi/mca/btl/sm/btl_sm.c#mca_btl_sm_sendi
https://svn.open-mpi.org/source/xref/ompi_1.3/ompi/mca/btl/mx/btl_mx.c#mca_btl_mx_sendi
https://svn.open-mpi.org/source/xref/ompi_1.3/ompi/mca/btl/portals/btl_portals_send.c#mca_btl_portals_sendi
