Re: [OMPI users] Pros and cons of --enable-heterogeneous

[Jeff Squyres]

On Oct 8, 2010, at 2:21 PM, Jeff Squyres wrote:

> No.  Open MPI does not have MPD -- I think you're thinking of a different MPI 
> implementation.  :-)

Someone pointed out to me off-list that I wasn't quite clear here...

*If* all of your architectures are the same endian and otherwise equivalent, 
you can just compile 32 bit and be ok.  If your architectures are a mix of 
endians and/or other representations, you need the heterogeneous support.

Also note that there is a --hetero flag to mpirun that you will need to use.  
Here's the relevant bit from mpirun(1):

   --hetero
  Indicates that multiple app_contexts are being provided that are
  a mix of 32/64-bit binaries.

It seems a little klunky to have to specify --enable-heterogeneous in configure 
*and* specify --hetero on the command line, but as I mentioned, this is 
unfortunately not a common scenario and we therefore haven't really optimized 
it.  :-\

I *swear* I just read in some of our documentation somewhere the part about 
mixing 32 and 64 bit MPI_DOUBLE (for example), but for the life of me, I can't 
find it now.  :-(

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] Pros and cons of --enable-heterogeneous

[Jeff Squyres]

On Oct 7, 2010, at 3:53 PM, Durga Choudhury wrote:

> If I compile with --enable-heterogenous flag for different
> *architectures* (I have a mix of old 32 bit x86, newer x86_64 and some
> Cell BE based boxes (PS3)), would I be able to form a MPD ring between
> all these different machines?

No.  Open MPI does not have MPD -- I think you're thinking of a different MPI 
implementation.  :-)

If you can run Linux on all these machines, then hypothetically the 
heterogeneous flag would allow a single Open MPI job to run across all of them. 
 I say "hypothetically" because I think we have done basic testing to ensure 
this stuff works, but it is used so infrequently (in reality, most people don't 
mix and match too often) that we don't stress test it, nor do we test it in 
exotic environments.

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] Pros and cons of --enable-heterogeneous

[Jeff Squyres]

On Oct 7, 2010, at 3:44 PM, David Ronis wrote:

> I have various boxes that run openmpi and I can't seem to use all of
> them at once because they have different CPU's (e.g., pentiums, athlons
> (both 32 bit) vs Intel I7 (64 bit)).  

You could also just compile everything 32 bit, and then not use the 
heterogeneous flag.

> I'm about the build 1.4.3 and was
> wondering if I should add --enable-heterogenous to the configure flags.

Keep in mind that you may still lose precision if you, for example, send a 64 
MPI_DOUBLE to a machine where MPI_DOUBLE is 32 bit.  There's not too much that 
OMPI can do there.

The heterogeneous flag is more traditionally used for endian flipping kinds of 
things.

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] Pros and cons of --enable-heterogeneous

[Eugene Loh]

David Ronis wrote:

Ralph, thanks for the reply.   


If I build with enable-heterogeneous and then decide to run on a
homogeneous set of nodes, does the additional "overhead" go away or
become completely negligible; i.e., if no conversion is necessary.
 

I'm no expert, but I think the overhead does not go away.  Even if you 
run on a homogeneous set of nodes, a local node does not know that.  It 
prepares a message without knowing if the destination is "same" or 
"different".  (There may be an exception with the sm BTL, which is only 
for processes on the same node and where it it assumed that a node 
comprises homogeneous processors.)


Whether the overhead is significant or negligible is another matter.  A 
subjective matter.  I suppose you could try some tests and judge for 
yourself for your case.

Re: [OMPI users] Pros and cons of --enable-heterogeneous

[David Ronis]

Ralph, thanks for the reply.   

If I build with enable-heterogeneous and then decide to run on a
homogeneous set of nodes, does the additional "overhead" go away or
become completely negligible; i.e., if no conversion is necessary.

David


On Thu, 2010-10-07 at 15:17 -0600, Ralph Castain wrote:
> Hetero operations tend to lose a little performance due to the need to
> convert data, but otherwise there is no real negative. We don't do it
> by default solely because the majority of installations don't need to,
> and there is no reason to lose even a little performance if it isn't
> necessary.
> 
> 
> If you want an application to be able to span that mix, then you'll
> need to set that configure flag.
> 
> On Thu, Oct 7, 2010 at 1:44 PM, David Ronis 
> wrote:
> I have various boxes that run openmpi and I can't seem to use
> all of
> them at once because they have different CPU's (e.g.,
> pentiums, athlons
> (both 32 bit) vs Intel I7 (64 bit)).   I'm about the build
> 1.4.3 and was
> wondering if I should add --enable-heterogenous to the
> configure flags.
> Any advice as to why or why not would be appreciated.
> 
> David
> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
> 

[OMPI users] Pros and cons of --enable-heterogeneous

[David Ronis]

I have various boxes that run openmpi and I can't seem to use all of
them at once because they have different CPU's (e.g., pentiums, athlons
(both 32 bit) vs Intel I7 (64 bit)).   I'm about the build 1.4.3 and was
wondering if I should add --enable-heterogenous to the configure flags.
Any advice as to why or why not would be appreciated.

David

Re: [OMPI users] Pros and cons of --enable-heterogeneous

[Ralph Castain]

The short answer is "yes". It should work.


On Thu, Oct 7, 2010 at 1:53 PM, Durga Choudhury  wrote:

> I'd like to add to this question the following:
>
> If I compile with --enable-heterogenous flag for different
> *architectures* (I have a mix of old 32 bit x86, newer x86_64 and some
> Cell BE based boxes (PS3)), would I be able to form a MPD ring between
> all these different machines?
>
> Best regards
> Durga
>
> On Thu, Oct 7, 2010 at 3:44 PM, David Ronis  wrote:
> > I have various boxes that run openmpi and I can't seem to use all of
> > them at once because they have different CPU's (e.g., pentiums, athlons
> > (both 32 bit) vs Intel I7 (64 bit)).   I'm about the build 1.4.3 and was
> > wondering if I should add --enable-heterogenous to the configure flags.
> > Any advice as to why or why not would be appreciated.
> >
> > David
> >
> >
> > ___
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
> >
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

Re: [OMPI users] Pros and cons of --enable-heterogeneous

[Ralph Castain]

Hetero operations tend to lose a little performance due to the need to
convert data, but otherwise there is no real negative. We don't do it by
default solely because the majority of installations don't need to, and
there is no reason to lose even a little performance if it isn't necessary.

If you want an application to be able to span that mix, then you'll need to
set that configure flag.

On Thu, Oct 7, 2010 at 1:44 PM, David Ronis  wrote:

> I have various boxes that run openmpi and I can't seem to use all of
> them at once because they have different CPU's (e.g., pentiums, athlons
> (both 32 bit) vs Intel I7 (64 bit)).   I'm about the build 1.4.3 and was
> wondering if I should add --enable-heterogenous to the configure flags.
> Any advice as to why or why not would be appreciated.
>
> David
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

Re: [OMPI users] Pros and cons of --enable-heterogeneous

[Durga Choudhury]

I'd like to add to this question the following:

If I compile with --enable-heterogenous flag for different
*architectures* (I have a mix of old 32 bit x86, newer x86_64 and some
Cell BE based boxes (PS3)), would I be able to form a MPD ring between
all these different machines?

Best regards
Durga

On Thu, Oct 7, 2010 at 3:44 PM, David Ronis  wrote:
> I have various boxes that run openmpi and I can't seem to use all of
> them at once because they have different CPU's (e.g., pentiums, athlons
> (both 32 bit) vs Intel I7 (64 bit)).   I'm about the build 1.4.3 and was
> wondering if I should add --enable-heterogenous to the configure flags.
> Any advice as to why or why not would be appreciated.
>
> David
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

[OMPI users] Pros and cons of --enable-heterogeneous

[David Ronis]

I have various boxes that run openmpi and I can't seem to use all of
them at once because they have different CPU's (e.g., pentiums, athlons
(both 32 bit) vs Intel I7 (64 bit)).   I'm about the build 1.4.3 and was
wondering if I should add --enable-heterogenous to the configure flags.
Any advice as to why or why not would be appreciated.

David