If I understand correctly, the issue that is being asked is: Process 0 has lets say a buffer on GPU 0. That buffer has been imported to PE 1 using the GPU IPC mechanism and is now mapped using a different virtual address into the address space of PE 1. Can PE 1 use that new virtual address in a communication operation with Open MPI? Is my understanding correct?
I think the challenge might be that if PE 1 uses that virtual address in a Send/Recv operation, the internal protocols will try to (potentially) open an IPC handle for that buffer as well, and I am not sure that PE1 can do that, since the owner of that buffer is PE 0. @bosilca ? Thanks Edgar From: 'Pritchard Jr., Howard' via Open MPI users <[email protected]> Sent: Wednesday, May 27, 2026 4:01 PM To: [email protected] Cc: Panyala, Ajay <[email protected]> Subject: Re: [EXTERNAL] [OMPI users] GPU-aware MPI Hello Bruce, I think a little more info is needed. Could you post the output you get from running ompi_info ? double check that the ompi_info you are running is in the same folder as the mpicc you’re using. thanks, Howard From: "'Palmer, Bruce J' via Open MPI users" <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Wednesday, May 27, 2026 at 10:44 AM To: Open MPI Users <[email protected]<mailto:[email protected]>> Cc: "Panyala, Ajay" <[email protected]<mailto:[email protected]>> Subject: [EXTERNAL] [OMPI users] GPU-aware MPI Hi, I’m trying to modify the Global Arrays library so that it supports global arrays hosted on GPU memory. I have a version of the progress ranks runtime that works by copying data to a buffer on the host before sending it to another process located on a different SMP node but I’d like to eliminate the host memory copies by using GPU-aware MPI. I’ve implemented this in the code but it seems to be failing because I can’t use a pointer to GPU memory in an MPI send or receive call that was created via a cudaIpcOpenMemHandle call. Are pointers to GPU memory created from IpcMemHandles supposed to work with GPU-aware MPI? This would be critical for our progress ranks runtime since it would be effectively replacing the POSIX-shared memory strategy that we use for handling messaging related to global arrays hosted on regular host memory. I’ve included a small test code using just Cuda and MPI that reproduces the strategy we want to use inside Global Arrays. Bruce To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<mailto:[email protected]>. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<mailto:[email protected]>. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
