Fair enough Ralph! I was implicitly assuming a "build once / run everywhere" use case, my bad for not making my assumption clear.
If the container is built to run on a specific host, there are indeed other options to achieve near native performances. Cheers, Gilles On Thu, Jan 27, 2022 at 4:02 PM Ralph Castain via users < users@lists.open-mpi.org> wrote: > I'll disagree a bit there. You do want to use an MPI library in your > container that is configued to perform on the host cluster. However, that > doesn't mean you are constrained as Gilles describes. It takes a little > more setup knowledge, true, but there are lots of instructions and > knowledgeable people out there to help. Experiments have shown that using > non-system MPIs provide at least equivalent performance to the native MPIs > when configured. Matching the internal/external MPI implementations may > simplify the mechanics of setting it up, but it is definitely not required. > > > On Jan 26, 2022, at 8:55 PM, Gilles Gouaillardet via users < > users@lists.open-mpi.org> wrote: > > Brian, > > FWIW > > Keep in mind that when running a container on a supercomputer, it is > generally recommended to use the supercomputer MPI implementation > (fine tuned and with support for the high speed interconnect) instead of > the one of the container (generally a vanilla MPI with basic > support for TCP and shared memory). > That scenario implies several additional constraints, and one of them is > the MPI library of the host and the container are (oversimplified) ABI > compatible. > > In your case, you would have to rebuild your container with MPICH (instead > of Open MPI) so it can be "substituted" at run time with Intel MPI (MPICH > based and ABI compatible). > > Cheers, > > Gilles > > On Thu, Jan 27, 2022 at 1:07 PM Brian Dobbins via users < > users@lists.open-mpi.org> wrote: > >> >> Hi Ralph, >> >> Thanks for the explanation - in hindsight, that makes perfect sense, >> since each process is operating inside the container and will of course >> load up identical libraries, so data types/sizes can't be inconsistent. I >> don't know why I didn't realize that before. I imagine the past issues I'd >> experienced were just due to the PMI differences in the different MPI >> implementations at the time. I owe you a beer or something at the next >> in-person SC conference! >> >> Cheers, >> - Brian >> >> >> On Wed, Jan 26, 2022 at 4:54 PM Ralph Castain via users < >> users@lists.open-mpi.org> wrote: >> >>> There is indeed an ABI difference. However, the _launcher_ doesn't have >>> anything to do with the MPI library. All that is needed is a launcher that >>> can provide the key exchange required to wireup the MPI processes. At this >>> point, both MPICH and OMPI have PMIx support, so you can use the same >>> launcher for both. IMPI does not, and so the IMPI launcher will only >>> support PMI-1 or PMI-2 (I forget which one). >>> >>> You can, however, work around that problem. For example, if the host >>> system is using Slurm, then you could "srun" the containers and let Slurm >>> perform the wireup. Again, you'd have to ensure that OMPI was built to >>> support whatever wireup protocol the Slurm installation supported (which >>> might well be PMIx today). Also works on Cray/ALPS. Completely bypasses the >>> IMPI issue. >>> >>> Another option I've seen used is to have the host system start the >>> containers (using ssh or whatever), providing the containers with access to >>> a "hostfile" identifying the TCP address of each container. It is then easy >>> for OMPI's mpirun to launch the job across the containers. I use this every >>> day on my machine (using Docker Desktop with Docker containers, but the >>> container tech is irrelevant here) to test OMPI. Pretty easy to set that >>> up, and I should think the sys admins could do so for their users. >>> >>> Finally, you could always install the PMIx Reference RTE (PRRTE) on the >>> cluster as that executes at user level, and then use PRRTE to launch your >>> OMPI containers. OMPI runs very well under PRRTE - in fact, PRRTE is the >>> RTE embedded in OMPI starting with the v5.0 release. >>> >>> Regardless of your choice of method, the presence of IMPI doesn't >>> preclude using OMPI containers so long as the OMPI library is fully >>> contained in that container. Choice of launch method just depends on how >>> your system is setup. >>> >>> Ralph >>> >>> >>> On Jan 26, 2022, at 3:17 PM, Brian Dobbins <bdobb...@gmail.com> wrote: >>> >>> >>> Hi Ralph, >>> >>> Afraid I don't understand. If your image has the OMPI libraries >>>> installed in it, what difference does it make what is on your host? You'll >>>> never see the IMPI installation. >>>> >>> >>>> We have been supporting people running that way since Singularity was >>>> originally released, without any problems. The only time you can hit an >>>> issue is if you try to mount the MPI libraries from the host (i.e., violate >>>> the container boundary) - so don't do that and you should be fine. >>>> >>> >>> Can you clarify what you mean here? I thought there was an ABI >>> difference between the various MPICH-based MPIs and OpenMPI, meaning you >>> can't use a host's Intel MPI to launch a container's OpenMPI-compiled >>> program. You *can* use the internal-to-the-container OpenMPI to launch >>> everything, which is easy for single-node runs but more challenging for >>> multi-node ones. Maybe my understanding is wrong or out of date though? >>> >>> Thanks, >>> - Brian >>> >>> >>> >>>> >>>> >>>> On Jan 26, 2022, at 12:19 PM, Luis Alfredo Pires Barbosa < >>>> luis_pire...@hotmail.com> wrote: >>>> >>>> Hi Ralph, >>>> >>>> My singularity image has OpenMPI, but my host doesnt (Intel MPI). And I >>>> am not sure if I the system would work with Intel + OpenMPI. >>>> >>>> Luis >>>> >>>> Enviado do Email <https://go.microsoft.com/fwlink/?LinkId=550986> >>>> para Windows >>>> >>>> *De: *Ralph Castain via users <users@lists.open-mpi.org> >>>> *Enviado:*quarta-feira, 26 de janeiro de 2022 16:01 >>>> *Para: *Open MPI Users <users@lists.open-mpi.org> >>>> *Cc:*Ralph Castain <r...@open-mpi.org> >>>> *Assunto: *Re: [OMPI users] OpenMPI - Intel MPI >>>> >>>> Err...the whole point of a container is to put all the library >>>> dependencies _inside_ it. So why don't you just install OMPI in your >>>> singularity image? >>>> >>>> >>>> >>>> On Jan 26, 2022, at 6:42 AM, Luis Alfredo Pires Barbosa via users < >>>> users@lists.open-mpi.org> wrote: >>>> >>>> Hello all, >>>> >>>> I have Intel MPI in my cluster but I am running singularity image of a >>>> software which uses OpenMPI. >>>> >>>> Since they may not be compatible and I dont think it is possible to get >>>> these two different MPI running in the system. >>>> I wounder if there is some work arround for this issue. >>>> >>>> Any insight would be welcome. >>>> Luis >>>> >>>> >>>> >>> >
