Hi, Ralph. Have you already tried passing the parameter
--mca btl_openib_want_cuda_gdr 1 This may help your case. On Mon, Mar 13, 2017 at 5:52 PM, Ralph Campbell via devel < devel@lists.open-mpi.org> wrote: > I have been looking closely at the source code for openmpi-2.0.2 and I see > what looks like support for GPU direct RDMA but when testing it with a > small GPU direct aware MPI program from Nvidia, I don't see ibv_reg_mr() > ever being called with the GPU UVM address. > Looking more closely, it appears that mca_bml_base_prepare_src() > calls opal_convertor_pack() which calls MEMCPY_CUDA() so I don't see a case > where the GPU memory is accessed peer-to-peer over PCIe by the network card. > > Am I missing something? > > > BTW, I am using cuda-8.0 and opal_config.h defines OPAL_CUDA_GDR_SUPPORT, > OPAL_CUDA_GET_ATTRIBUTES, OPAL_CUDA_SUPPORT, and OPAL_CUDA_SYNC_MEMOPS > as "1". > > _______________________________________________ > devel mailing list > devel@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/devel > -- -Akshay
_______________________________________________ devel mailing list devel@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
