Re: [OMPI users] RES: OpenMPI - Intel MPI

[Ralph Castain via users]

Sure - but then we aren't talking about containers any more, just vendor vs 
OMPI. I'm not getting in the middle of that one!


On Jan 27, 2022, at 6:28 PM, Gilles Gouaillardet via users 
mailto:users@lists.open-mpi.org> > wrote:

Thanks Ralph,

Now I get what you had in mind.

Strictly speaking, you are making the assumption that Open MPI performance 
matches the system MPI performances.

This is generally true for common interconnects and/or those that feature 
providers for libfabric or UCX, but not so for "exotic" interconnects (that 
might not be supported natively by Open MPI or abstraction layers) and/or with 
an uncommon topology (for which collective communications are not fully 
optimized by Open MPI). In the latter case, using the system/vendor MPI is the 
best option performance wise.

Cheers,

Gilles

On Fri, Jan 28, 2022 at 2:23 AM Ralph Castain via users 
mailto:users@lists.open-mpi.org> > wrote:
Just to complete this - there is always a lingering question regarding shared 
memory support. There are two ways to resolve that one:

* run one container per physical node, launching multiple procs in each 
container. The procs can then utilize shared memory _inside_ the container. 
This is the cleanest solution (i.e., minimizes container boundary violations), 
but some users need/want per-process isolation.

* run one container per MPI process, having each container then mount an 
_external_ common directory to an internal mount point. This allows each 
process to access the common shared memory location. As with the device 
drivers, you typically specify that external mount location when launching the 
container.

Using those combined methods, you can certainly have a "generic" container that 
suffers no performance impact from bare metal. The problem has been that it 
takes a certain degree of "container savvy" to set this up and make it work - 
which is beyond what most users really want to learn. I'm sure the container 
community is working on ways to reduce that burden (I'm not really plugged into 
those efforts, but others on this list might be).

Ralph


> On Jan 27, 2022, at 7:39 AM, Ralph H Castain   > wrote:
> 
>> Fair enough Ralph! I was implicitly assuming a "build once / run everywhere" 
>> use case, my bad for not making my assumption clear.
>> If the container is built to run on a specific host, there are indeed other 
>> options to achieve near native performances.
>> 
> 
> Err...that isn't actually what I meant, nor what we did. You can, in fact, 
> build a container that can "run everywhere" while still employing high-speed 
> fabric support. What you do is:
> 
> * configure OMPI with all the fabrics enabled (or at least all the ones you 
> care about)
> 
> * don't include the fabric drivers in your container. These can/will vary 
> across deployments, especially those (like NVIDIA's) that involve kernel 
> modules
> 
> * setup your container to mount specified external device driver locations 
> onto the locations where you configured OMPI to find them. Sadly, this does 
> violate the container boundary - but nobody has come up with another 
> solution, and at least the violation is confined to just the device drivers. 
> Typically, you specify the external locations that are to be mounted using an 
> envar or some other mechanism appropriate to your container, and then include 
> the relevant information when launching the containers.

> 
> When OMPI initializes, it will do its normal procedure of attempting to load 
> each fabric's drivers, selecting the transports whose drivers it can load. 
> NOTE: beginning with OMPI v5, you'll need to explicitly tell OMPI to build 
> without statically linking in the fabric plugins or else this probably will 
> fail.
> 
> At least one vendor now distributes OMPI containers preconfigured with their 
> fabric support based on this method. So using a "generic" container doesn't 
> mean you lose performance - in fact, our tests showed zero impact on 
> performance using this method.
> 
> HTH
> Ralph
> 

Re: [OMPI users] RES: OpenMPI - Intel MPI

[Gilles Gouaillardet via users]

Thanks Ralph,

Now I get what you had in mind.

Strictly speaking, you are making the assumption that Open MPI performance
matches the system MPI performances.

This is generally true for common interconnects and/or those that feature
providers for libfabric or UCX, but not so for "exotic" interconnects (that
might not be supported natively by Open MPI or abstraction layers) and/or
with an uncommon topology (for which collective communications are not
fully optimized by Open MPI). In the latter case, using the system/vendor
MPI is the best option performance wise.

Cheers,

Gilles

On Fri, Jan 28, 2022 at 2:23 AM Ralph Castain via users <
users@lists.open-mpi.org> wrote:

> Just to complete this - there is always a lingering question regarding
> shared memory support. There are two ways to resolve that one:
>
> * run one container per physical node, launching multiple procs in each
> container. The procs can then utilize shared memory _inside_ the container.
> This is the cleanest solution (i.e., minimizes container boundary
> violations), but some users need/want per-process isolation.
>
> * run one container per MPI process, having each container then mount an
> _external_ common directory to an internal mount point. This allows each
> process to access the common shared memory location. As with the device
> drivers, you typically specify that external mount location when launching
> the container.
>
> Using those combined methods, you can certainly have a "generic" container
> that suffers no performance impact from bare metal. The problem has been
> that it takes a certain degree of "container savvy" to set this up and make
> it work - which is beyond what most users really want to learn. I'm sure
> the container community is working on ways to reduce that burden (I'm not
> really plugged into those efforts, but others on this list might be).
>
> Ralph
>
>
> > On Jan 27, 2022, at 7:39 AM, Ralph H Castain  wrote:
> >
> >> Fair enough Ralph! I was implicitly assuming a "build once / run
> everywhere" use case, my bad for not making my assumption clear.
> >> If the container is built to run on a specific host, there are indeed
> other options to achieve near native performances.
> >>
> >
> > Err...that isn't actually what I meant, nor what we did. You can, in
> fact, build a container that can "run everywhere" while still employing
> high-speed fabric support. What you do is:
> >
> > * configure OMPI with all the fabrics enabled (or at least all the ones
> you care about)
> >
> > * don't include the fabric drivers in your container. These can/will
> vary across deployments, especially those (like NVIDIA's) that involve
> kernel modules
> >
> > * setup your container to mount specified external device driver
> locations onto the locations where you configured OMPI to find them. Sadly,
> this does violate the container boundary - but nobody has come up with
> another solution, and at least the violation is confined to just the device
> drivers. Typically, you specify the external locations that are to be
> mounted using an envar or some other mechanism appropriate to your
> container, and then include the relevant information when launching the
> containers.
> >
> > When OMPI initializes, it will do its normal procedure of attempting to
> load each fabric's drivers, selecting the transports whose drivers it can
> load. NOTE: beginning with OMPI v5, you'll need to explicitly tell OMPI to
> build without statically linking in the fabric plugins or else this
> probably will fail.
> >
> > At least one vendor now distributes OMPI containers preconfigured with
> their fabric support based on this method. So using a "generic" container
> doesn't mean you lose performance - in fact, our tests showed zero impact
> on performance using this method.
> >
> > HTH
> > Ralph
> >
>
>
>

Re: [OMPI users] RES: OpenMPI - Intel MPI

[Ralph Castain via users]

See inline
Ralph

On Jan 27, 2022, at 10:05 AM, Brian Dobbins mailto:bdobb...@gmail.com> > wrote:


Hi Ralph,

  Thanks again for this wealth of information - we've successfully run the same 
container instance across multiple systems without issues, even surpassing 
'native' performance in edge cases, presumably because the native host MPI is 
either older or simply tuned differently (eg, 'eager limit' differences or 
medium/large message optimizations) than the one in the container.  That's 
actually really useful already, and coupled with what you've already pointed 
out about the launcher vs the MPI when it comes to ABI issues, makes me pretty 
happy.  

  But I would love to dive a little deeper on two of the more complex things 
you've brought up:

  1) With regards to not including the fabric drivers in the container and 
mounting in the device drivers, how many different sets of drivers are there?  
I know you mentioned you're not really plugged into the container community, so 
maybe this is more a question for them, but I'd think if there's a relatively 
small set that accounts for most systems, you might be able to include them 
all, and have the dlopen facility find the correct one at launch?  (Eg, the 
'host' launcher could transmit some information as to which drivers  -like 
'mlnx5'- are desired, and it looks for those inside the container?)  

  I definitely agree that mounting in specific drivers is beyond what most 
users are comfortable with, so understanding the plans to address this would be 
great.  Mind you, for most of our work, even just including the 'inbox' OFED 
drivers works well enough right now.  

So there are two problems you can encounter when using internal drivers. The 
first is a compatibility issue between the drivers in the container and the 
firmware in the local hardware. Vendors assume that the drivers and firmware 
get updated as a package since that is how they deliver it. In the case of a 
container, however, you have a driver that is "frozen" in time while the 
firmware is moving around. You could wind up on a system whose firmware is more 
recent, but also on a system with "stone age" firmware someone hasn't 
wanted/needed to update for some time. Thus, your drivers could run into 
problems.

One way you can work around that is to create your own compatibility table - 
i.e., embed some logic in your container that queries the firmware level of the 
local hardware and checks for compatibility with your embedded driver. If it 
isn't compatible, then you can either report that problem and cleanly abort, or 
you can mount the external drivers. Little more work, and I'm not sure what it 
really buys you since you'd have to be prepared to mount the external drivers 
anyway - but still something one could do.

Second problem is that some drivers actually include kernel modules (e.g., if 
you are using CUDA). In those cases, you have no choice but to mount the local 
drivers as your container isn't going to have the system's addresses in it.

Frankly, the latter issue is the one that usually gets you - unless you have an 
overriding reason to bring your own drivers, probably better to mount them.


  2) With regards to launching multiple processes within one container for 
shared memory access, how is this done?  Or is it automatic now with modern 
launchers?  Eg, if the launch commands knows it's running 96 copies of the same 
container (via either 'host:96' or '-ppn 96' or something), is it 'smart' 
enough to do this?   This also hasn't been a problem for us, since we're 
typically rate-limited by inter-node comms, not intra-node ones, but it'd be 
good to ensure we're doing it 'right'.

It depends on how you are launching the containers. If the system is launching 
the containers and you use mpirun (from inside one of the containers) to spawn 
the processes, then mpirun "sees" each container as being a "node". Thus, you 
just launch like normal.

If you are using something outside to not only start the containers, but also 
to start the application processes within the container, then it's a little 
less "normal". You'd need a container-aware launcher to do it - i.e., something 
that understands it is doing a two-stage launch, how to separate mapping of the 
processes from placement of the containers, and how to "inject" processes into 
the container. This is feasible and I actually had this working at one time 
(with Singularity, at least), but it has bit-rotted. If there is interest, I 
can put it on my "to-do" list to revive in PRRTE, though I can't promise a 
release date for that feature. You can see some info on the general idea here:

[PDF] https://openpmix.github.io/uploads/2019/04/PMIxSUG2019.pdf 
 
[PPX] https://www.slideshare.net/rcastain/pmix-bridging-the-container-boundary 
 
[video] 

Re: [OMPI users] RES: OpenMPI - Intel MPI

[Brian Dobbins via users]

Hi Ralph,

  Thanks again for this wealth of information - we've successfully run the
same container instance across multiple systems without issues, even
surpassing 'native' performance in edge cases, presumably because the
native host MPI is either older or simply tuned differently (eg, 'eager
limit' differences or medium/large message optimizations) than the one in
the container.  That's actually really useful already, and coupled with
what you've already pointed out about the launcher vs the MPI when it comes
to ABI issues, makes me pretty happy.

  But I would love to dive a little deeper on two of the more complex
things you've brought up:

  1) With regards to not including the fabric drivers in the container and
mounting in the device drivers, how many different sets of drivers
*are* there?
I know you mentioned you're not really plugged into the container
community, so maybe this is more a question for them, but I'd think if
there's a relatively small set that accounts for most systems, you might be
able to include them all, and have the dlopen facility find the correct one
at launch?  (Eg, the 'host' launcher could transmit some information as to
which drivers  -like 'mlnx5'- are desired, and it looks for those inside
the container?)

  I definitely agree that mounting in specific drivers is beyond what most
users are comfortable with, so understanding the plans to address this
would be great.  Mind you, for most of our work, even just including the
'inbox' OFED drivers works well enough right now.

  2) With regards to launching multiple processes within one container for
shared memory access, how is this done?  Or is it automatic now with modern
launchers?  Eg, if the launch commands knows it's running 96 copies of the
same container (via either 'host:96' or '-ppn 96' or something), is it
'smart' enough to do this?   This also hasn't been a problem for us, since
we're typically rate-limited by inter-node comms, not intra-node ones, but
it'd be good to ensure we're doing it 'right'.

  Thanks again,
  - Brian


On Thu, Jan 27, 2022 at 10:22 AM Ralph Castain via users <
users@lists.open-mpi.org> wrote:

> Just to complete this - there is always a lingering question regarding
> shared memory support. There are two ways to resolve that one:
>
> * run one container per physical node, launching multiple procs in each
> container. The procs can then utilize shared memory _inside_ the container.
> This is the cleanest solution (i.e., minimizes container boundary
> violations), but some users need/want per-process isolation.
>
> * run one container per MPI process, having each container then mount an
> _external_ common directory to an internal mount point. This allows each
> process to access the common shared memory location. As with the device
> drivers, you typically specify that external mount location when launching
> the container.
>
> Using those combined methods, you can certainly have a "generic" container
> that suffers no performance impact from bare metal. The problem has been
> that it takes a certain degree of "container savvy" to set this up and make
> it work - which is beyond what most users really want to learn. I'm sure
> the container community is working on ways to reduce that burden (I'm not
> really plugged into those efforts, but others on this list might be).
>
> Ralph
>
>
> > On Jan 27, 2022, at 7:39 AM, Ralph H Castain  wrote:
> >
> >> Fair enough Ralph! I was implicitly assuming a "build once / run
> everywhere" use case, my bad for not making my assumption clear.
> >> If the container is built to run on a specific host, there are indeed
> other options to achieve near native performances.
> >>
> >
> > Err...that isn't actually what I meant, nor what we did. You can, in
> fact, build a container that can "run everywhere" while still employing
> high-speed fabric support. What you do is:
> >
> > * configure OMPI with all the fabrics enabled (or at least all the ones
> you care about)
> >
> > * don't include the fabric drivers in your container. These can/will
> vary across deployments, especially those (like NVIDIA's) that involve
> kernel modules
> >
> > * setup your container to mount specified external device driver
> locations onto the locations where you configured OMPI to find them. Sadly,
> this does violate the container boundary - but nobody has come up with
> another solution, and at least the violation is confined to just the device
> drivers. Typically, you specify the external locations that are to be
> mounted using an envar or some other mechanism appropriate to your
> container, and then include the relevant information when launching the
> containers.
> >
> > When OMPI initializes, it will do its normal procedure of attempting to
> load each fabric's drivers, selecting the transports whose drivers it can
> load. NOTE: beginning with OMPI v5, you'll need to explicitly tell OMPI to
> build without statically linking in the fabric plugins or else this
> probably 

Re: [OMPI users] RES: OpenMPI - Intel MPI

[Ralph Castain via users]

Just to complete this - there is always a lingering question regarding shared 
memory support. There are two ways to resolve that one:

* run one container per physical node, launching multiple procs in each 
container. The procs can then utilize shared memory _inside_ the container. 
This is the cleanest solution (i.e., minimizes container boundary violations), 
but some users need/want per-process isolation.

* run one container per MPI process, having each container then mount an 
_external_ common directory to an internal mount point. This allows each 
process to access the common shared memory location. As with the device 
drivers, you typically specify that external mount location when launching the 
container.

Using those combined methods, you can certainly have a "generic" container that 
suffers no performance impact from bare metal. The problem has been that it 
takes a certain degree of "container savvy" to set this up and make it work - 
which is beyond what most users really want to learn. I'm sure the container 
community is working on ways to reduce that burden (I'm not really plugged into 
those efforts, but others on this list might be).

Ralph


> On Jan 27, 2022, at 7:39 AM, Ralph H Castain  wrote:
> 
>> Fair enough Ralph! I was implicitly assuming a "build once / run everywhere" 
>> use case, my bad for not making my assumption clear.
>> If the container is built to run on a specific host, there are indeed other 
>> options to achieve near native performances.
>> 
> 
> Err...that isn't actually what I meant, nor what we did. You can, in fact, 
> build a container that can "run everywhere" while still employing high-speed 
> fabric support. What you do is:
> 
> * configure OMPI with all the fabrics enabled (or at least all the ones you 
> care about)
> 
> * don't include the fabric drivers in your container. These can/will vary 
> across deployments, especially those (like NVIDIA's) that involve kernel 
> modules
> 
> * setup your container to mount specified external device driver locations 
> onto the locations where you configured OMPI to find them. Sadly, this does 
> violate the container boundary - but nobody has come up with another 
> solution, and at least the violation is confined to just the device drivers. 
> Typically, you specify the external locations that are to be mounted using an 
> envar or some other mechanism appropriate to your container, and then include 
> the relevant information when launching the containers.
> 
> When OMPI initializes, it will do its normal procedure of attempting to load 
> each fabric's drivers, selecting the transports whose drivers it can load. 
> NOTE: beginning with OMPI v5, you'll need to explicitly tell OMPI to build 
> without statically linking in the fabric plugins or else this probably will 
> fail.
> 
> At least one vendor now distributes OMPI containers preconfigured with their 
> fabric support based on this method. So using a "generic" container doesn't 
> mean you lose performance - in fact, our tests showed zero impact on 
> performance using this method.
> 
> HTH
> Ralph
> 

Re: [OMPI users] RES: OpenMPI - Intel MPI

[Ralph Castain via users]

> Fair enough Ralph! I was implicitly assuming a "build once / run everywhere" 
> use case, my bad for not making my assumption clear.
> If the container is built to run on a specific host, there are indeed other 
> options to achieve near native performances.
> 

Err...that isn't actually what I meant, nor what we did. You can, in fact, 
build a container that can "run everywhere" while still employing high-speed 
fabric support. What you do is:

* configure OMPI with all the fabrics enabled (or at least all the ones you 
care about)

* don't include the fabric drivers in your container. These can/will vary 
across deployments, especially those (like NVIDIA's) that involve kernel modules

* setup your container to mount specified external device driver locations onto 
the locations where you configured OMPI to find them. Sadly, this does violate 
the container boundary - but nobody has come up with another solution, and at 
least the violation is confined to just the device drivers. Typically, you 
specify the external locations that are to be mounted using an envar or some 
other mechanism appropriate to your container, and then include the relevant 
information when launching the containers.

When OMPI initializes, it will do its normal procedure of attempting to load 
each fabric's drivers, selecting the transports whose drivers it can load. 
NOTE: beginning with OMPI v5, you'll need to explicitly tell OMPI to build 
without statically linking in the fabric plugins or else this probably will 
fail.

At least one vendor now distributes OMPI containers preconfigured with their 
fabric support based on this method. So using a "generic" container doesn't 
mean you lose performance - in fact, our tests showed zero impact on 
performance using this method.

HTH
Ralph

Re: [OMPI users] RES: OpenMPI - Intel MPI

[Jeff Squyres (jsquyres) via users]

This is part of the challenge of HPC: there are general solutions, but no 
specific silver bullet that works in all scenarios.  In short: everyone's setup 
is different.  So we can offer advice, but not necessarily a 100%-guaranteed 
solution that will work in your environment.

In general, we advise users to:

* Configure/build Open MPI with their favorite compilers (either a proprietary 
compiler or modern/recent GCC or clang).  More recent compilers tend to give 
better performance than older compilers.
* Configure/build Open MPI against the communication library for your HPC 
interconnect.  These days, it's mostly Libfabric or UCX.  If you have 
InfiniBand, it's UCX.

That's probably table stakes right there; you can tweak more beyond that, but 
with those 2 things, you'll go pretty far.

FWIW, we did a series of talks about Open MPI in conjunction with the EasyBuild 
community recently.  The videos and slides are available here:

* https://www.open-mpi.org/video/?category=general#abcs-of-open-mpi-part-1
* https://www.open-mpi.org/video/?category=general#abcs-of-open-mpi-part-2
* https://www.open-mpi.org/video/?category=general#abcs-of-open-mpi-part-3

For a beginner, parts 1 and 2 are probably the most relevant, and you can 
probably skip the parts about PMIx (circle back to that later for more advanced 
knowledge).

--
Jeff Squyres
jsquy...@cisco.com


From: users  on behalf of Diego Zuccato via 
users 
Sent: Thursday, January 27, 2022 2:59 AM
To: users@lists.open-mpi.org
Cc: Diego Zuccato
Subject: Re: [OMPI users] RES: OpenMPI - Intel MPI

Sorry for the noob question, but: what should I configure for OpenMPI
"to perform on the host cluster"? Any link to a guide would be welcome!

Slightly extended rationale for the question: I'm currently using
"unconfigured" Debian packages and getting some strange behaviour...
Maybe it's just something that a little tuning can fix easily.

Il 27/01/2022 07:58, Ralph Castain via users ha scritto:
> I'll disagree a bit there. You do want to use an MPI library in your
> container that is configued to perform on the host cluster. However,
> that doesn't mean you are constrained as Gilles describes. It takes a
> little more setup knowledge, true, but there are lots of instructions
> and knowledgeable people out there to help. Experiments have shown that
> using non-system MPIs provide at least equivalent performance to the
> native MPIs when configured. Matching the internal/external MPI
> implementations may simplify the mechanics of setting it up, but it is
> definitely not required.
>
>
>> On Jan 26, 2022, at 8:55 PM, Gilles Gouaillardet via users
>> mailto:users@lists.open-mpi.org>> wrote:
>>
>> Brian,
>>
>> FWIW
>>
>> Keep in mind that when running a container on a supercomputer, it is
>> generally recommended to use the supercomputer MPI implementation
>> (fine tuned and with support for the high speed interconnect) instead
>> of the one of the container (generally a vanilla MPI with basic
>> support for TCP and shared memory).
>> That scenario implies several additional constraints, and one of them
>> is the MPI library of the host and the container are (oversimplified)
>> ABI compatible.
>>
>> In your case, you would have to rebuild your container with MPICH
>> (instead of Open MPI) so it can be "substituted" at run time with
>> Intel MPI (MPICH based and ABI compatible).
>>
>> Cheers,
>>
>> Gilles
>>
>> On Thu, Jan 27, 2022 at 1:07 PM Brian Dobbins via users
>> mailto:users@lists.open-mpi.org>> wrote:
>>
>>
>> Hi Ralph,
>>
>>   Thanks for the explanation - in hindsight, that makes perfect
>> sense, since each process is operating inside the container and
>> will of course load up identical libraries, so data types/sizes
>> can't be inconsistent.  I don't know why I didn't realize that
>> before.  I imagine the past issues I'd experienced were just due
>> to the PMI differences in the different MPI implementations at the
>> time.  I owe you a beer or something at the next in-person SC
>> conference!
>>
>>   Cheers,
>>   - Brian
>>
>>
>> On Wed, Jan 26, 2022 at 4:54 PM Ralph Castain via users
>> mailto:users@lists.open-mpi.org>> wrote:
>>
>> There is indeed an ABI difference. However, the _launcher_
>> doesn't have anything to do with the MPI library. All that is
>> needed is a launcher that can provide the key exchange
>> required to wireup the MPI processes. At this point, both
>> MPICH and OMPI have PMIx support, so you can use the same
>&g

Re: [OMPI users] RES: OpenMPI - Intel MPI

[Diego Zuccato via users]

Sorry for the noob question, but: what should I configure for OpenMPI 
"to perform on the host cluster"? Any link to a guide would be welcome!


Slightly extended rationale for the question: I'm currently using 
"unconfigured" Debian packages and getting some strange behaviour... 
Maybe it's just something that a little tuning can fix easily.


Il 27/01/2022 07:58, Ralph Castain via users ha scritto:
I'll disagree a bit there. You do want to use an MPI library in your 
container that is configued to perform on the host cluster. However, 
that doesn't mean you are constrained as Gilles describes. It takes a 
little more setup knowledge, true, but there are lots of instructions 
and knowledgeable people out there to help. Experiments have shown that 
using non-system MPIs provide at least equivalent performance to the 
native MPIs when configured. Matching the internal/external MPI 
implementations may simplify the mechanics of setting it up, but it is 
definitely not required.



On Jan 26, 2022, at 8:55 PM, Gilles Gouaillardet via users 
mailto:users@lists.open-mpi.org>> wrote:


Brian,

FWIW

Keep in mind that when running a container on a supercomputer, it is 
generally recommended to use the supercomputer MPI implementation
(fine tuned and with support for the high speed interconnect) instead 
of the one of the container (generally a vanilla MPI with basic

support for TCP and shared memory).
That scenario implies several additional constraints, and one of them 
is the MPI library of the host and the container are (oversimplified) 
ABI compatible.


In your case, you would have to rebuild your container with MPICH 
(instead of Open MPI) so it can be "substituted" at run time with 
Intel MPI (MPICH based and ABI compatible).


Cheers,

Gilles

On Thu, Jan 27, 2022 at 1:07 PM Brian Dobbins via users 
mailto:users@lists.open-mpi.org>> wrote:



Hi Ralph,

  Thanks for the explanation - in hindsight, that makes perfect
sense, since each process is operating inside the container and
will of course load up identical libraries, so data types/sizes
can't be inconsistent.  I don't know why I didn't realize that
before.  I imagine the past issues I'd experienced were just due
to the PMI differences in the different MPI implementations at the
time.  I owe you a beer or something at the next in-person SC
conference!

  Cheers,
  - Brian


On Wed, Jan 26, 2022 at 4:54 PM Ralph Castain via users
mailto:users@lists.open-mpi.org>> wrote:

There is indeed an ABI difference. However, the _launcher_
doesn't have anything to do with the MPI library. All that is
needed is a launcher that can provide the key exchange
required to wireup the MPI processes. At this point, both
MPICH and OMPI have PMIx support, so you can use the same
launcher for both. IMPI does not, and so the IMPI launcher
will only support PMI-1 or PMI-2 (I forget which one).

You can, however, work around that problem. For example, if
the host system is using Slurm, then you could "srun" the
containers and let Slurm perform the wireup. Again, you'd have
to ensure that OMPI was built to support whatever wireup
protocol the Slurm installation supported (which might well be
PMIx today). Also works on Cray/ALPS. Completely bypasses the
IMPI issue.

Another option I've seen used is to have the host system start
the containers (using ssh or whatever), providing the
containers with access to a "hostfile" identifying the TCP
address of each container. It is then easy for OMPI's mpirun
to launch the job across the containers. I use this every day
on my machine (using Docker Desktop with Docker containers,
but the container tech is irrelevant here) to test OMPI.
Pretty easy to set that up, and I should think the sys admins
could do so for their users.

Finally, you could always install the PMIx Reference RTE
(PRRTE) on the cluster as that executes at user level, and
then use PRRTE to launch your OMPI containers. OMPI runs very
well under PRRTE - in fact, PRRTE is the RTE embedded in OMPI
starting with the v5.0 release.

Regardless of your choice of method, the presence of IMPI
doesn't preclude using OMPI containers so long as the OMPI
library is fully contained in that container. Choice of launch
method just depends on how your system is setup.

Ralph



On Jan 26, 2022, at 3:17 PM, Brian Dobbins
mailto:bdobb...@gmail.com>> wrote:


Hi Ralph,

Afraid I don't understand. If your image has the OMPI
libraries installed in it, what difference does it make
what is on your host? You'll never see the IMPI installation.


We have been supporting people running that way since

Re: [OMPI users] RES: OpenMPI - Intel MPI

[Gilles Gouaillardet via users]

Fair enough Ralph!

I was implicitly assuming a "build once / run everywhere" use case, my bad
for not making my assumption clear.

If the container is built to run on a specific host, there are indeed other
options to achieve near native performances.

Cheers,

Gilles

On Thu, Jan 27, 2022 at 4:02 PM Ralph Castain via users <
users@lists.open-mpi.org> wrote:

> I'll disagree a bit there. You do want to use an MPI library in your
> container that is configued to perform on the host cluster. However, that
> doesn't mean you are constrained as Gilles describes. It takes a little
> more setup knowledge, true, but there are lots of instructions and
> knowledgeable people out there to help. Experiments have shown that using
> non-system MPIs provide at least equivalent performance to the native MPIs
> when configured. Matching the internal/external MPI implementations may
> simplify the mechanics of setting it up, but it is definitely not required.
>
>
> On Jan 26, 2022, at 8:55 PM, Gilles Gouaillardet via users <
> users@lists.open-mpi.org> wrote:
>
> Brian,
>
> FWIW
>
> Keep in mind that when running a container on a supercomputer, it is
> generally recommended to use the supercomputer MPI implementation
> (fine tuned and with support for the high speed interconnect) instead of
> the one of the container (generally a vanilla MPI with basic
> support for TCP and shared memory).
> That scenario implies several additional constraints, and one of them is
> the MPI library of the host and the container are (oversimplified) ABI
> compatible.
>
> In your case, you would have to rebuild your container with MPICH (instead
> of Open MPI) so it can be "substituted" at run time with Intel MPI (MPICH
> based and ABI compatible).
>
> Cheers,
>
> Gilles
>
> On Thu, Jan 27, 2022 at 1:07 PM Brian Dobbins via users <
> users@lists.open-mpi.org> wrote:
>
>>
>> Hi Ralph,
>>
>>   Thanks for the explanation - in hindsight, that makes perfect sense,
>> since each process is operating inside the container and will of course
>> load up identical libraries, so data types/sizes can't be inconsistent.  I
>> don't know why I didn't realize that before.  I imagine the past issues I'd
>> experienced were just due to the PMI differences in the different MPI
>> implementations at the time.  I owe you a beer or something at the next
>> in-person SC conference!
>>
>>   Cheers,
>>   - Brian
>>
>>
>> On Wed, Jan 26, 2022 at 4:54 PM Ralph Castain via users <
>> users@lists.open-mpi.org> wrote:
>>
>>> There is indeed an ABI difference. However, the _launcher_ doesn't have
>>> anything to do with the MPI library. All that is needed is a launcher that
>>> can provide the key exchange required to wireup the MPI processes. At this
>>> point, both MPICH and OMPI have PMIx support, so you can use the same
>>> launcher for both. IMPI does not, and so the IMPI launcher will only
>>> support PMI-1 or PMI-2 (I forget which one).
>>>
>>> You can, however, work around that problem. For example, if the host
>>> system is using Slurm, then you could "srun" the containers and let Slurm
>>> perform the wireup. Again, you'd have to ensure that OMPI was built to
>>> support whatever wireup protocol the Slurm installation supported (which
>>> might well be PMIx today). Also works on Cray/ALPS. Completely bypasses the
>>> IMPI issue.
>>>
>>> Another option I've seen used is to have the host system start the
>>> containers (using ssh or whatever), providing the containers with access to
>>> a "hostfile" identifying the TCP address of each container. It is then easy
>>> for OMPI's mpirun to launch the job across the containers. I use this every
>>> day on my machine (using Docker Desktop with Docker containers, but the
>>> container tech is irrelevant here) to test OMPI. Pretty easy to set that
>>> up, and I should think the sys admins could do so for their users.
>>>
>>> Finally, you could always install the PMIx Reference RTE (PRRTE) on the
>>> cluster as that executes at user level, and then use PRRTE to launch your
>>> OMPI containers. OMPI runs very well under PRRTE - in fact, PRRTE is the
>>> RTE embedded in OMPI starting with the v5.0 release.
>>>
>>> Regardless of your choice of method, the presence of IMPI doesn't
>>> preclude using OMPI containers so long as the OMPI library is fully
>>> contained in that container. Choice of launch method just depends on how
>>> your system is setup.
>>>
>>> Ralph
>>>
>>>
>>> On Jan 26, 2022, at 3:17 PM, Brian Dobbins  wrote:
>>>
>>>
>>> Hi Ralph,
>>>
>>> Afraid I don't understand. If your image has the OMPI libraries
 installed in it, what difference does it make what is on your host? You'll
 never see the IMPI installation.

>>>
 We have been supporting people running that way since Singularity was
 originally released, without any problems. The only time you can hit an
 issue is if you try to mount the MPI libraries from the host (i.e., violate
 the container boundary) - so don't do that and 

Re: [OMPI users] RES: OpenMPI - Intel MPI

[Ralph Castain via users]

I'll disagree a bit there. You do want to use an MPI library in your container 
that is configued to perform on the host cluster. However, that doesn't mean 
you are constrained as Gilles describes. It takes a little more setup 
knowledge, true, but there are lots of instructions and knowledgeable people 
out there to help. Experiments have shown that using non-system MPIs provide at 
least equivalent performance to the native MPIs when configured. Matching the 
internal/external MPI implementations may simplify the mechanics of setting it 
up, but it is definitely not required.


On Jan 26, 2022, at 8:55 PM, Gilles Gouaillardet via users 
mailto:users@lists.open-mpi.org> > wrote:

Brian,

FWIW

Keep in mind that when running a container on a supercomputer, it is generally 
recommended to use the supercomputer MPI implementation
(fine tuned and with support for the high speed interconnect) instead of the 
one of the container (generally a vanilla MPI with basic
support for TCP and shared memory).
That scenario implies several additional constraints, and one of them is the 
MPI library of the host and the container are (oversimplified) ABI compatible.

In your case, you would have to rebuild your container with MPICH (instead of 
Open MPI) so it can be "substituted" at run time with Intel MPI (MPICH based 
and ABI compatible).

Cheers,

Gilles

On Thu, Jan 27, 2022 at 1:07 PM Brian Dobbins via users 
mailto:users@lists.open-mpi.org> > wrote:

Hi Ralph,

  Thanks for the explanation - in hindsight, that makes perfect sense, since 
each process is operating inside the container and will of course load up 
identical libraries, so data types/sizes can't be inconsistent.  I don't know 
why I didn't realize that before.  I imagine the past issues I'd experienced 
were just due to the PMI differences in the different MPI implementations at 
the time.  I owe you a beer or something at the next in-person SC conference!

  Cheers,
  - Brian


On Wed, Jan 26, 2022 at 4:54 PM Ralph Castain via users 
mailto:users@lists.open-mpi.org> > wrote:
There is indeed an ABI difference. However, the _launcher_ doesn't have 
anything to do with the MPI library. All that is needed is a launcher that can 
provide the key exchange required to wireup the MPI processes. At this point, 
both MPICH and OMPI have PMIx support, so you can use the same launcher for 
both. IMPI does not, and so the IMPI launcher will only support PMI-1 or PMI-2 
(I forget which one).

You can, however, work around that problem. For example, if the host system is 
using Slurm, then you could "srun" the containers and let Slurm perform the 
wireup. Again, you'd have to ensure that OMPI was built to support whatever 
wireup protocol the Slurm installation supported (which might well be PMIx 
today). Also works on Cray/ALPS. Completely bypasses the IMPI issue.

Another option I've seen used is to have the host system start the containers 
(using ssh or whatever), providing the containers with access to a "hostfile" 
identifying the TCP address of each container. It is then easy for OMPI's 
mpirun to launch the job across the containers. I use this every day on my 
machine (using Docker Desktop with Docker containers, but the container tech is 
irrelevant here) to test OMPI. Pretty easy to set that up, and I should think 
the sys admins could do so for their users.

Finally, you could always install the PMIx Reference RTE (PRRTE) on the cluster 
as that executes at user level, and then use PRRTE to launch your OMPI 
containers. OMPI runs very well under PRRTE - in fact, PRRTE is the RTE 
embedded in OMPI starting with the v5.0 release.

Regardless of your choice of method, the presence of IMPI doesn't preclude 
using OMPI containers so long as the OMPI library is fully contained in that 
container. Choice of launch method just depends on how your system is setup.

Ralph


On Jan 26, 2022, at 3:17 PM, Brian Dobbins mailto:bdobb...@gmail.com> > wrote:


Hi Ralph,

Afraid I don't understand. If your image has the OMPI libraries installed in 
it, what difference does it make what is on your host? You'll never see the 
IMPI installation.

We have been supporting people running that way since Singularity was 
originally released, without any problems. The only time you can hit an issue 
is if you try to mount the MPI libraries from the host (i.e., violate the 
container boundary) - so don't do that and you should be fine.

  Can you clarify what you mean here?  I thought there was an ABI difference 
between the various MPICH-based MPIs and OpenMPI, meaning you can't use a 
host's Intel MPI to launch a container's OpenMPI-compiled program.  You can use 
the internal-to-the-container OpenMPI to launch everything, which is easy for 
single-node runs but more challenging for multi-node ones.  Maybe my 
understanding is wrong or out of date though?

  Thanks,
  - Brian

 

On Jan 26, 2022, at 12:19 PM, Luis Alfredo Pires Barbosa 
mailto:luis_pire...@hotmail.com> > wrote:

Re: [OMPI users] RES: OpenMPI - Intel MPI

[Gilles Gouaillardet via users]

Brian,

FWIW

Keep in mind that when running a container on a supercomputer, it is
generally recommended to use the supercomputer MPI implementation
(fine tuned and with support for the high speed interconnect) instead of
the one of the container (generally a vanilla MPI with basic
support for TCP and shared memory).
That scenario implies several additional constraints, and one of them is
the MPI library of the host and the container are (oversimplified) ABI
compatible.

In your case, you would have to rebuild your container with MPICH (instead
of Open MPI) so it can be "substituted" at run time with Intel MPI (MPICH
based and ABI compatible).

Cheers,

Gilles

On Thu, Jan 27, 2022 at 1:07 PM Brian Dobbins via users <
users@lists.open-mpi.org> wrote:

>
> Hi Ralph,
>
>   Thanks for the explanation - in hindsight, that makes perfect sense,
> since each process is operating inside the container and will of course
> load up identical libraries, so data types/sizes can't be inconsistent.  I
> don't know why I didn't realize that before.  I imagine the past issues I'd
> experienced were just due to the PMI differences in the different MPI
> implementations at the time.  I owe you a beer or something at the next
> in-person SC conference!
>
>   Cheers,
>   - Brian
>
>
> On Wed, Jan 26, 2022 at 4:54 PM Ralph Castain via users <
> users@lists.open-mpi.org> wrote:
>
>> There is indeed an ABI difference. However, the _launcher_ doesn't have
>> anything to do with the MPI library. All that is needed is a launcher that
>> can provide the key exchange required to wireup the MPI processes. At this
>> point, both MPICH and OMPI have PMIx support, so you can use the same
>> launcher for both. IMPI does not, and so the IMPI launcher will only
>> support PMI-1 or PMI-2 (I forget which one).
>>
>> You can, however, work around that problem. For example, if the host
>> system is using Slurm, then you could "srun" the containers and let Slurm
>> perform the wireup. Again, you'd have to ensure that OMPI was built to
>> support whatever wireup protocol the Slurm installation supported (which
>> might well be PMIx today). Also works on Cray/ALPS. Completely bypasses the
>> IMPI issue.
>>
>> Another option I've seen used is to have the host system start the
>> containers (using ssh or whatever), providing the containers with access to
>> a "hostfile" identifying the TCP address of each container. It is then easy
>> for OMPI's mpirun to launch the job across the containers. I use this every
>> day on my machine (using Docker Desktop with Docker containers, but the
>> container tech is irrelevant here) to test OMPI. Pretty easy to set that
>> up, and I should think the sys admins could do so for their users.
>>
>> Finally, you could always install the PMIx Reference RTE (PRRTE) on the
>> cluster as that executes at user level, and then use PRRTE to launch your
>> OMPI containers. OMPI runs very well under PRRTE - in fact, PRRTE is the
>> RTE embedded in OMPI starting with the v5.0 release.
>>
>> Regardless of your choice of method, the presence of IMPI doesn't
>> preclude using OMPI containers so long as the OMPI library is fully
>> contained in that container. Choice of launch method just depends on how
>> your system is setup.
>>
>> Ralph
>>
>>
>> On Jan 26, 2022, at 3:17 PM, Brian Dobbins  wrote:
>>
>>
>> Hi Ralph,
>>
>> Afraid I don't understand. If your image has the OMPI libraries installed
>>> in it, what difference does it make what is on your host? You'll never see
>>> the IMPI installation.
>>>
>>
>>> We have been supporting people running that way since Singularity was
>>> originally released, without any problems. The only time you can hit an
>>> issue is if you try to mount the MPI libraries from the host (i.e., violate
>>> the container boundary) - so don't do that and you should be fine.
>>>
>>
>>   Can you clarify what you mean here?  I thought there was an ABI
>> difference between the various MPICH-based MPIs and OpenMPI, meaning you
>> can't use a host's Intel MPI to launch a container's OpenMPI-compiled
>> program.  You *can* use the internal-to-the-container OpenMPI to launch
>> everything, which is easy for single-node runs but more challenging for
>> multi-node ones.  Maybe my understanding is wrong or out of date though?
>>
>>   Thanks,
>>   - Brian
>>
>>
>>
>>>
>>>
>>> On Jan 26, 2022, at 12:19 PM, Luis Alfredo Pires Barbosa <
>>> luis_pire...@hotmail.com> wrote:
>>>
>>> Hi Ralph,
>>>
>>> My singularity image has OpenMPI, but my host doesnt (Intel MPI). And I
>>> am not sure if I the system would work with Intel + OpenMPI.
>>>
>>> Luis
>>>
>>> Enviado do Email 
>>> para Windows
>>>
>>> *De: *Ralph Castain via users 
>>> *Enviado:*quarta-feira, 26 de janeiro de 2022 16:01
>>> *Para: *Open MPI Users 
>>> *Cc:*Ralph Castain 
>>> *Assunto: *Re: [OMPI users] OpenMPI - Intel MPI
>>>
>>> Err...the whole point of a container is to put all the library
>>> dependencies 

Re: [OMPI users] RES: OpenMPI - Intel MPI

[Brian Dobbins via users]

Hi Ralph,

  Thanks for the explanation - in hindsight, that makes perfect sense,
since each process is operating inside the container and will of course
load up identical libraries, so data types/sizes can't be inconsistent.  I
don't know why I didn't realize that before.  I imagine the past issues I'd
experienced were just due to the PMI differences in the different MPI
implementations at the time.  I owe you a beer or something at the next
in-person SC conference!

  Cheers,
  - Brian


On Wed, Jan 26, 2022 at 4:54 PM Ralph Castain via users <
users@lists.open-mpi.org> wrote:

> There is indeed an ABI difference. However, the _launcher_ doesn't have
> anything to do with the MPI library. All that is needed is a launcher that
> can provide the key exchange required to wireup the MPI processes. At this
> point, both MPICH and OMPI have PMIx support, so you can use the same
> launcher for both. IMPI does not, and so the IMPI launcher will only
> support PMI-1 or PMI-2 (I forget which one).
>
> You can, however, work around that problem. For example, if the host
> system is using Slurm, then you could "srun" the containers and let Slurm
> perform the wireup. Again, you'd have to ensure that OMPI was built to
> support whatever wireup protocol the Slurm installation supported (which
> might well be PMIx today). Also works on Cray/ALPS. Completely bypasses the
> IMPI issue.
>
> Another option I've seen used is to have the host system start the
> containers (using ssh or whatever), providing the containers with access to
> a "hostfile" identifying the TCP address of each container. It is then easy
> for OMPI's mpirun to launch the job across the containers. I use this every
> day on my machine (using Docker Desktop with Docker containers, but the
> container tech is irrelevant here) to test OMPI. Pretty easy to set that
> up, and I should think the sys admins could do so for their users.
>
> Finally, you could always install the PMIx Reference RTE (PRRTE) on the
> cluster as that executes at user level, and then use PRRTE to launch your
> OMPI containers. OMPI runs very well under PRRTE - in fact, PRRTE is the
> RTE embedded in OMPI starting with the v5.0 release.
>
> Regardless of your choice of method, the presence of IMPI doesn't preclude
> using OMPI containers so long as the OMPI library is fully contained in
> that container. Choice of launch method just depends on how your system is
> setup.
>
> Ralph
>
>
> On Jan 26, 2022, at 3:17 PM, Brian Dobbins  wrote:
>
>
> Hi Ralph,
>
> Afraid I don't understand. If your image has the OMPI libraries installed
>> in it, what difference does it make what is on your host? You'll never see
>> the IMPI installation.
>>
>
>> We have been supporting people running that way since Singularity was
>> originally released, without any problems. The only time you can hit an
>> issue is if you try to mount the MPI libraries from the host (i.e., violate
>> the container boundary) - so don't do that and you should be fine.
>>
>
>   Can you clarify what you mean here?  I thought there was an ABI
> difference between the various MPICH-based MPIs and OpenMPI, meaning you
> can't use a host's Intel MPI to launch a container's OpenMPI-compiled
> program.  You *can* use the internal-to-the-container OpenMPI to launch
> everything, which is easy for single-node runs but more challenging for
> multi-node ones.  Maybe my understanding is wrong or out of date though?
>
>   Thanks,
>   - Brian
>
>
>
>>
>>
>> On Jan 26, 2022, at 12:19 PM, Luis Alfredo Pires Barbosa <
>> luis_pire...@hotmail.com> wrote:
>>
>> Hi Ralph,
>>
>> My singularity image has OpenMPI, but my host doesnt (Intel MPI). And I
>> am not sure if I the system would work with Intel + OpenMPI.
>>
>> Luis
>>
>> Enviado do Email 
>> para Windows
>>
>> *De: *Ralph Castain via users 
>> *Enviado:*quarta-feira, 26 de janeiro de 2022 16:01
>> *Para: *Open MPI Users 
>> *Cc:*Ralph Castain 
>> *Assunto: *Re: [OMPI users] OpenMPI - Intel MPI
>>
>> Err...the whole point of a container is to put all the library
>> dependencies _inside_ it. So why don't you just install OMPI in your
>> singularity image?
>>
>>
>>
>> On Jan 26, 2022, at 6:42 AM, Luis Alfredo Pires Barbosa via users <
>> users@lists.open-mpi.org> wrote:
>>
>> Hello all,
>>
>> I have Intel MPI in my cluster but I am running singularity image of a
>> software which uses OpenMPI.
>>
>> Since they may not be compatible and I dont think it is possible to get
>> these two different MPI running in the system.
>> I wounder if there is some work arround for this issue.
>>
>> Any insight would be welcome.
>> Luis
>>
>>
>>
>

Re: [OMPI users] RES: OpenMPI - Intel MPI

[Ralph Castain via users]

There is indeed an ABI difference. However, the _launcher_ doesn't have 
anything to do with the MPI library. All that is needed is a launcher that can 
provide the key exchange required to wireup the MPI processes. At this point, 
both MPICH and OMPI have PMIx support, so you can use the same launcher for 
both. IMPI does not, and so the IMPI launcher will only support PMI-1 or PMI-2 
(I forget which one).

You can, however, work around that problem. For example, if the host system is 
using Slurm, then you could "srun" the containers and let Slurm perform the 
wireup. Again, you'd have to ensure that OMPI was built to support whatever 
wireup protocol the Slurm installation supported (which might well be PMIx 
today). Also works on Cray/ALPS. Completely bypasses the IMPI issue.

Another option I've seen used is to have the host system start the containers 
(using ssh or whatever), providing the containers with access to a "hostfile" 
identifying the TCP address of each container. It is then easy for OMPI's 
mpirun to launch the job across the containers. I use this every day on my 
machine (using Docker Desktop with Docker containers, but the container tech is 
irrelevant here) to test OMPI. Pretty easy to set that up, and I should think 
the sys admins could do so for their users.

Finally, you could always install the PMIx Reference RTE (PRRTE) on the cluster 
as that executes at user level, and then use PRRTE to launch your OMPI 
containers. OMPI runs very well under PRRTE - in fact, PRRTE is the RTE 
embedded in OMPI starting with the v5.0 release.

Regardless of your choice of method, the presence of IMPI doesn't preclude 
using OMPI containers so long as the OMPI library is fully contained in that 
container. Choice of launch method just depends on how your system is setup.

Ralph


On Jan 26, 2022, at 3:17 PM, Brian Dobbins mailto:bdobb...@gmail.com> > wrote:


Hi Ralph,

Afraid I don't understand. If your image has the OMPI libraries installed in 
it, what difference does it make what is on your host? You'll never see the 
IMPI installation.

We have been supporting people running that way since Singularity was 
originally released, without any problems. The only time you can hit an issue 
is if you try to mount the MPI libraries from the host (i.e., violate the 
container boundary) - so don't do that and you should be fine.

  Can you clarify what you mean here?  I thought there was an ABI difference 
between the various MPICH-based MPIs and OpenMPI, meaning you can't use a 
host's Intel MPI to launch a container's OpenMPI-compiled program.  You can use 
the internal-to-the-container OpenMPI to launch everything, which is easy for 
single-node runs but more challenging for multi-node ones.  Maybe my 
understanding is wrong or out of date though?

  Thanks,
  - Brian

 

On Jan 26, 2022, at 12:19 PM, Luis Alfredo Pires Barbosa 
mailto:luis_pire...@hotmail.com> > wrote:

Hi Ralph,
 My singularity image has OpenMPI, but my host doesnt (Intel MPI). And I am not 
sure if I the system would work with Intel + OpenMPI.
 Luis
 Enviado do Email   para Windows
 De: Ralph Castain via users  
Enviado:quarta-feira, 26 de janeiro de 2022 16:01
Para: Open MPI Users  
Cc:Ralph Castain  
Assunto: Re: [OMPI users] OpenMPI - Intel MPI
 Err...the whole point of a container is to put all the library dependencies 
_inside_ it. So why don't you just install OMPI in your singularity image?
 

On Jan 26, 2022, at 6:42 AM, Luis Alfredo Pires Barbosa via users 
mailto:users@lists.open-mpi.org> > wrote:
 Hello all,
 I have Intel MPI in my cluster but I am running singularity image of a 
software which uses OpenMPI.
 Since they may not be compatible and I dont think it is possible to get these 
two different MPI running in the system.
I wounder if there is some work arround for this issue.
 Any insight would be welcome.
Luis

Re: [OMPI users] RES: OpenMPI - Intel MPI

[Brian Dobbins via users]

Hi Ralph,

Afraid I don't understand. If your image has the OMPI libraries installed
> in it, what difference does it make what is on your host? You'll never see
> the IMPI installation.
>

> We have been supporting people running that way since Singularity was
> originally released, without any problems. The only time you can hit an
> issue is if you try to mount the MPI libraries from the host (i.e., violate
> the container boundary) - so don't do that and you should be fine.
>

  Can you clarify what you mean here?  I thought there was an ABI
difference between the various MPICH-based MPIs and OpenMPI, meaning you
can't use a host's Intel MPI to launch a container's OpenMPI-compiled
program.  You *can* use the internal-to-the-container OpenMPI to launch
everything, which is easy for single-node runs but more challenging for
multi-node ones.  Maybe my understanding is wrong or out of date though?

  Thanks,
  - Brian



>
>
> On Jan 26, 2022, at 12:19 PM, Luis Alfredo Pires Barbosa <
> luis_pire...@hotmail.com> wrote:
>
> Hi Ralph,
>
> My singularity image has OpenMPI, but my host doesnt (Intel MPI). And I am
> not sure if I the system would work with Intel + OpenMPI.
>
> Luis
>
> Enviado do Email 
> para Windows
>
> *De: *Ralph Castain via users 
> *Enviado:*quarta-feira, 26 de janeiro de 2022 16:01
> *Para: *Open MPI Users 
> *Cc:*Ralph Castain 
> *Assunto: *Re: [OMPI users] OpenMPI - Intel MPI
>
> Err...the whole point of a container is to put all the library
> dependencies _inside_ it. So why don't you just install OMPI in your
> singularity image?
>
>
>
> On Jan 26, 2022, at 6:42 AM, Luis Alfredo Pires Barbosa via users <
> users@lists.open-mpi.org> wrote:
>
> Hello all,
>
> I have Intel MPI in my cluster but I am running singularity image of a
> software which uses OpenMPI.
>
> Since they may not be compatible and I dont think it is possible to get
> these two different MPI running in the system.
> I wounder if there is some work arround for this issue.
>
> Any insight would be welcome.
> Luis
>
>
>

Re: [OMPI users] RES: OpenMPI - Intel MPI

[Bennet Fauber via users]

Luis,

Can you install OpenMPI into your home directory (or other shared
filesystem) and use that?  You may also want contact your cluster
admins to see if they can help do that or offer another solution.

On Wed, Jan 26, 2022 at 3:21 PM Luis Alfredo Pires Barbosa via users
 wrote:
>
> Hi Ralph,
>
>
>
> My singularity image has OpenMPI, but my host doesnt (Intel MPI). And I am 
> not sure if I the system would work with Intel + OpenMPI.
>
>
>
> Luis
>
>
>
> Enviado do Email para Windows
>
>
>
> De: Ralph Castain via users
> Enviado:quarta-feira, 26 de janeiro de 2022 16:01
> Para: Open MPI Users
> Cc:Ralph Castain
> Assunto: Re: [OMPI users] OpenMPI - Intel MPI
>
>
>
> Err...the whole point of a container is to put all the library dependencies 
> _inside_ it. So why don't you just install OMPI in your singularity image?
>
>
>
>
>
> On Jan 26, 2022, at 6:42 AM, Luis Alfredo Pires Barbosa via users 
>  wrote:
>
>
>
> Hello all,
>
>
>
> I have Intel MPI in my cluster but I am running singularity image of a 
> software which uses OpenMPI.
>
>
>
> Since they may not be compatible and I dont think it is possible to get these 
> two different MPI running in the system.
>
> I wounder if there is some work arround for this issue.
>
>
>
> Any insight would be welcome.
>
> Luis
>
>
>
>

Re: [OMPI users] RES: OpenMPI - Intel MPI

[Ralph Castain via users]

Afraid I don't understand. If your image has the OMPI libraries installed in 
it, what difference does it make what is on your host? You'll never see the 
IMPI installation.

We have been supporting people running that way since Singularity was 
originally released, without any problems. The only time you can hit an issue 
is if you try to mount the MPI libraries from the host (i.e., violate the 
container boundary) - so don't do that and you should be fine.


On Jan 26, 2022, at 12:19 PM, Luis Alfredo Pires Barbosa 
mailto:luis_pire...@hotmail.com> > wrote:

Hi Ralph,
 My singularity image has OpenMPI, but my host doesnt (Intel MPI). And I am not 
sure if I the system would work with Intel + OpenMPI.
 Luis
 Enviado do Email   para Windows
 De: Ralph Castain via users  
Enviado:quarta-feira, 26 de janeiro de 2022 16:01
Para: Open MPI Users  
Cc:Ralph Castain  
Assunto: Re: [OMPI users] OpenMPI - Intel MPI
 Err...the whole point of a container is to put all the library dependencies 
_inside_ it. So why don't you just install OMPI in your singularity image?
 

On Jan 26, 2022, at 6:42 AM, Luis Alfredo Pires Barbosa via users 
mailto:users@lists.open-mpi.org> > wrote:
 Hello all,
 I have Intel MPI in my cluster but I am running singularity image of a 
software which uses OpenMPI.
 Since they may not be compatible and I dont think it is possible to get these 
two different MPI running in the system.
I wounder if there is some work arround for this issue.
 Any insight would be welcome.
Luis

[OMPI users] RES: OpenMPI - Intel MPI

[Luis Alfredo Pires Barbosa via users]

Hi Ralph,

My singularity image has OpenMPI, but my host doesnt (Intel MPI). And I am not 
sure if I the system would work with Intel + OpenMPI.

Luis

Enviado do Email para Windows

De: Ralph Castain via users
Enviado:quarta-feira, 26 de janeiro de 2022 16:01
Para: Open MPI Users
Cc:Ralph Castain
Assunto: Re: [OMPI users] OpenMPI - Intel MPI

Err...the whole point of a container is to put all the library dependencies 
_inside_ it. So why don't you just install OMPI in your singularity image?



On Jan 26, 2022, at 6:42 AM, Luis Alfredo Pires Barbosa via users 
mailto:users@lists.open-mpi.org>> wrote:

Hello all,

I have Intel MPI in my cluster but I am running singularity image of a software 
which uses OpenMPI.

Since they may not be compatible and I dont think it is possible to get these 
two different MPI running in the system.
I wounder if there is some work arround for this issue.

Any insight would be welcome.
Luis