Any additional thoughts on this? Bruce
From: 'Palmer, Bruce J' via Open MPI users <[email protected]> Date: Thursday, May 28, 2026 at 9:00 AM To: [email protected] <[email protected]> Cc: Panyala, Ajay <[email protected]> Subject: Re: [EXTERNAL] [OMPI users] GPU-aware MPI Hi Edgar, I should have posted more information about my reproducer. I’ve been looking at running using 4 processors on 2 SMP nodes with each node having 1 GPU. Ranks 0 and 2 allocate memory on the GPU. Rank 0 sends a message to rank 3 and rand 2 sends a message to rank 1. Rank 3 opens the allocation created by rank 2 using a cudaIpcOpenMemHandle call and uses the pointer returned by cudaIpcOpenMemHandle in an MPI_Irecv call that expects a message from rank 0. Similarly, rank 1 uses a pointer from cudaIpcOpenMemHandle to a GPU allocation created by rank 0 in an MPI_Irecv call that expects a message from rank 2. This reproduces the behavior of a onesided put call in Global Arrays using the progress ranks runtime. The essential feature is that a pointer to GPU memory that was allocated by a different rank is being used in an MPI_Send/Recv call and that pointer is obtained via a cudaIpcOpenMemHandle call. Bruce From: [email protected] <[email protected]> on behalf of Edgar Gabriel <[email protected]> Date: Wednesday, May 27, 2026 at 2:10 PM To: [email protected] <[email protected]> Cc: Panyala, Ajay <[email protected]> Subject: RE: [EXTERNAL] [OMPI users] GPU-aware MPI You don't often get email from [email protected]. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> Check twice before you click! This email originated from outside PNNL. If I understand correctly, the issue that is being asked is: Process 0 has lets say a buffer on GPU 0. That buffer has been imported to PE 1 using the GPU IPC mechanism and is now mapped using a different virtual address into the address space of PE 1. Can PE 1 use that new virtual address in a communication operation with Open MPI? Is my understanding correct? I think the challenge might be that if PE 1 uses that virtual address in a Send/Recv operation, the internal protocols will try to (potentially) open an IPC handle for that buffer as well, and I am not sure that PE1 can do that, since the owner of that buffer is PE 0. @bosilca ? Thanks Edgar From: 'Pritchard Jr., Howard' via Open MPI users <[email protected]> Sent: Wednesday, May 27, 2026 4:01 PM To: [email protected] Cc: Panyala, Ajay <[email protected]> Subject: Re: [EXTERNAL] [OMPI users] GPU-aware MPI Hello Bruce, I think a little more info is needed. Could you post the output you get from running ompi_info ? double check that the ompi_info you are running is in the same folder as the mpicc you’re using. thanks, Howard From: "'Palmer, Bruce J' via Open MPI users" <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Wednesday, May 27, 2026 at 10:44 AM To: Open MPI Users <[email protected]<mailto:[email protected]>> Cc: "Panyala, Ajay" <[email protected]<mailto:[email protected]>> Subject: [EXTERNAL] [OMPI users] GPU-aware MPI Hi, I’m trying to modify the Global Arrays library so that it supports global arrays hosted on GPU memory. I have a version of the progress ranks runtime that works by copying data to a buffer on the host before sending it to another process located on a different SMP node but I’d like to eliminate the host memory copies by using GPU-aware MPI. I’ve implemented this in the code but it seems to be failing because I can’t use a pointer to GPU memory in an MPI send or receive call that was created via a cudaIpcOpenMemHandle call. Are pointers to GPU memory created from IpcMemHandles supposed to work with GPU-aware MPI? This would be critical for our progress ranks runtime since it would be effectively replacing the POSIX-shared memory strategy that we use for handling messaging related to global arrays hosted on regular host memory. I’ve included a small test code using just Cuda and MPI that reproduces the strategy we want to use inside Global Arrays. Bruce To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<mailto:[email protected]>. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<mailto:[email protected]>. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<mailto:[email protected]>. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<mailto:[email protected]>. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
