On Feb 23, 2009, at 12:14 , Eugene Loh wrote:
I'm a newbie and George is a veteran. So, this feels rather like
David and Goliath. (Hmm, David won and became king. Gee, I kinda
like that.) Anyhow...
That's an old story, we're living in modern times now ;)
George Bosilca wrote:
It doesn't sound reasonable to me. There is a reason for this, and
I think it's a good reason. The sendi function work for some
devices as a fast path for sending data, when the network is not
flooded. However, in the case sendi cannot do the job we expect,
the fact that it return the descriptor save us a call (we don't
have to do the alloc call later).
This does not make any sense to me. In what sense are we "saving a
call"? Not in the sense of run-time performance since the BTL is
now having to allocate a descriptor it did not otherwise need. The
amount of work is the same (one descriptor allocation either way),
but you're just pushing that work into the BTLs.
The descriptor is a BTL resource. If the sendi doesn't return one, the
PML will have to call the BTL alloc function from the BTL again (in
this case the calls will look like this: btl_sendi followed by
btl_alloc followed by btl_send). I'm not looking only at SM, I want
all of the BTL to have the opportunity to get good performance.
If sendi return a descriptor when it fails to send the data the call
list will be shorter: btl_sendi followed by btl_send. I'm trying to
decrease the number of jumps between the layers (PML/BTL), not the
number of lines of code in the BTL.
We are certainly not "saving a call" in the sense of reducing source
code. The PML has to have code to allocate a descriptor anyway
since there may not even be any sendi functions. So, the code to
allocate the descriptor is already in the PML. By asking sendi
functions to do the same, you're replicating that code in every
sendi function... possibly multiple times per BTL since a sendi
function might have multiple "out of resource" return paths.
Therefore, in the PML we already have the descriptor and we can
hand it back to the BTL, which give a chance for asynchronous
progress later on. Without this descriptor, the only option the
PML have is to put the PML request in a queue, and to try to send
it later, which is very expensive.
This also makes no sense to me. We're not talking about doing
without the descriptor. The PML is prepared to allocate it anyhow.
The issue is where the descriptor is allocated in the case that
sendi functions exist but cannot succeed. One alternative is to use
a single allocation point in the PML. The other alternative (what
we have today) is to replicate that code out to multiple sites,
adding unnecessary source code and interface arguments.
As I said previously, this save one jump from the PML to the BTL by
adding one more return argument to the sendi function and some lines
of code in every BTL. Not a big deal as a correctly written BTL can do
it pretty smartly (as an example special return case where everybody
jumps when an error is detected).
george.
The PML code is in
https://svn.open-mpi.org/source/xref/ompi_1.3/ompi/mca/pml/ob1/pml_ob1_sendreq.c
#mca_pml_ob1_send_request_start_copy
Existing BTL sendi functions are at
https://svn.open-mpi.org/source/xref/ompi_1.3/ompi/mca/btl/sm/
btl_sm.c#mca_btl_sm_sendi
https://svn.open-mpi.org/source/xref/ompi_1.3/ompi/mca/btl/mx/
btl_mx.c#mca_btl_mx_sendi
https://svn.open-mpi.org/source/xref/ompi_1.3/ompi/mca/btl/portals/btl_portals_send.c
#mca_btl_portals_sendi
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
