Aurelien --
I confess to forgetting some of the Paris discussion. :-\
Could the same effect of these pointers also be effected by having a
completion callback function pointer on the request? Or do you need
more than that?
On Nov 29, 2007, at 6:37 PM, Aurelien Bouteiller wrote:
This patch introduces customisable wait/test for requests as
discussed at the face-to-face ompi meeting in Paris.
A new global structure (ompi_request_functions) holding all the
pointers to the wait/test functions have been added.
ompi_request_wait* and ompi_request_test* have been #defined to be
replaced by ompi_request_functions.req_wait. The default
implementations of the wait/test functions names have been changed
from ompi_request_% to ompi_request_default_%. Those functions are
static initializer of the ompi_request_functions structure.
To modify the defaults, a components 1) copy the
ompi_request_functions structure (the type ompi_request_fns_t can be
used to declare a suitable variable), 2) change some of the
functions according to its needs. This is best done at MPI_init time
when there is no threads. Should this component be unloaded it have
to restore the defaults. The ompi_request_default_* functions should
never be called directly anywhere in the code. If a component needs
to access the previously defined implementation of wait, it should
call its local copy of the function. Component implementors should
keep in mind that another component might have already changed the
defaults and needs to be called.
Performance impact on NetPipe -a (async recv mode) does not show
measurable overhead. Here follows the "diff -y" between original and
modified ompi assembly code from ompi/mpi/c/wait.c. The only
significant difference is an extra movl to load the address of the
ompi_request_functions structure in eax. This obviously explains why
there is no measurable cost on latency.
ORIGINAL
MODIFIED
L2:
L2:
movl L_ompi_request_null$non_lazy_ptr-"L00000000001$pb"(%ebx),
%eax movl L_ompi_request_null
$non_lazy_ptr-"L00000000001$pb"(%ebx), %eax
cmpl %eax, (%edi)
cmpl %eax, (%edi)
je L18
je L18
> movl L_ompi_request_functions
$non_lazy_ptr-"L00000000001$pb"(%ebx), %eax
movl %esi, 4(%esp)
movl %esi, 4(%esp)
movl %edi, (%esp)
movl %edi, (%esp)
call L_ompi_request_wait$stub
| call *16(%eax)
Here is the patch for those who want to try themselves.
<custom_request_wait_and_test.patch>
If I receive comments outlining the need, thread safe accessors
could be added to allow components to change the functions at
anytime during execution and not only during MPI_Init/Finalize.
Please make noise if you find this useful.
If comments does not suggest extra work, I expect this code to be
committed in trunk next week.
Aurelien
Le 8 oct. 07 à 06:01, Aurelien Bouteiller a écrit :
For message logging purpose, we need to interface with wait_any,
wait_some, test, test_any, test_some, test_all. It is not possible to
use PMPI for this purpose. During the face-to-face meeting in Paris
(5-12 october 2007) we discussed this issue and came to the
conclusion that the best way to achieve this is to replace direct
calls to ompi_request_wait* and test* by indirect calls (same way as
PML send, recv, etc).
Basic idea is to declare a static structure containing the 8 pointers
to all the functions. This structure is initialized at compilation
time with the current basic wait/test functions. Before end of
MPI_init, any component might replace the basics with specialized
functions.
Expected cost is less than .01us latency according to preliminary
test. The method is consistent with the way we call pml send/recv.
Mechanism could be used later for stripping out grequest from
critical path when they are not used.
--
Aurelien Bouteiller, PhD
Innovative Computing Laboratory - MPI group
+1 865 974 6321
1122 Volunteer Boulevard
Claxton Education Building Suite 350
Knoxville, TN 37996
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
--
Dr. Aurelien Bouteiller, Sr. Research Associate
Innovative Computing Laboratory - MPI group
+1 865 974 6321
1122 Volunteer Boulevard
Claxton Education Building Suite 350
Knoxville, TN 37996
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
--
Jeff Squyres
Cisco Systems
