Forgot the link... https://bitbucket.org/rolfv/ompi-trunk-cuda-rfc2
From: devel-boun...@open-mpi.org [mailto:devel-boun...@open-mpi.org] On Behalf Of Rolf vandeVaart Sent: Tuesday, April 19, 2011 4:45 PM To: Open MPI Developers Subject: [OMPI devel] RFC: Second Try: Add support to send/receive CUDA device memory directly WHAT: Second try to add support to send data directly from CUDA device memory via MPI calls. TIMEOUT: 4/26/2011 DETAILS: Based on all the feedback (thanks to everyone who looked at it), I have whittled down what I hope to accomplish with this RFC. There were suggestions to better modularize the CUDA registration code so I will take a look at that separately. Since the registration code is a performance feature, it will be dropped from this RFC and investigated separately. This significantly reduced the changes being proposed here. With this RFC, all the changes are isolated in datatype and convertor code. As mentioned before, the changes mostly boil down to replacing memcpy with cuMemcpy when moving the data to or from a CUDA device buffer. Per suggestions, the choice to disable the large memory RDMA now happens on a per message basis. This is done by adding a flag to the convertor which tells the BTLs that an intermediate buffer is needed when dealing with device memory. As before, this code would be enabled via a configure option. A mostly completed version is viewable on bitbucket although I know the configure code is sorely lacking. This is the new list of changed files. M opal/config/opal_configure_options.m4 A opal/datatype/opal_datatype_cuda.c A opal/datatype/opal_datatype_cuda.h M opal/datatype/opal_convertor.h M opal/datatype/opal_datatype_copy.c M opal/datatype/opal_datatype_unpack.c M opal/datatype/Makefile.am M opal/datatype/opal_datatype_pack.h M opal/datatype/opal_convertor.c M opal/datatype/opal_datatype_unpack.h M ompi/mca/pml/ob1/pml_ob1_sendreq.h Thanks, Rolf ----------------------------------------------------------------------------------- This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. -----------------------------------------------------------------------------------
