Re: [OMPI users] How closely tied is a specific release of OpenMPI to the host operating system and other system software?

[Joe Landman]

On 2/1/2011 5:02 PM, Jeffrey A Cummings wrote:

I use OpenMPI on a variety of platforms: stand-alone servers running
Solaris on sparc boxes and Linux (mostly CentOS) on AMD/Intel boxes,
also Linux (again CentOS) on large clusters of AMD/Intel boxes. These
platforms all have some version of the 1.3 OpenMPI stream. I recently
requested an upgrade on all systems to 1.4.3 (for production work) and
1.5.1 (for experimentation). I'm getting a lot of push back from the
SysAdmin folks claiming that OpenMPI is closely intertwined with the
specific version of the operating system and/or other system software
(i.e., Rocks on the clusters). I need to know if they are telling me the
truth or if they're just making excuses to avoid the work. To state my
question another way: Apparently each release of Linux and/or Rocks
comes with some version of OpenMPI bundled in. Is it dangerous in some
way to upgrade to a newer version of OpenMPI? Thanks in advance for any
insight anyone can provide.


Jeff:

  The issue is that the way Rocks integrates MPI stacks, it is quite 
hard to upgrade the baseline stacks.


  This said, there is no issue in placing new stacks in 
/shared/openmpi/version/compiler/  or similar.  We've done this for many 
of our Rocks customers.


  For Rocks, the admins usually need RPMs, then they have to work on 
some extend-compute.xml magic to make it work.  You can do this, or as I 
did when we ran on a shared Rocks cluster that the admin was ... er ... 
reluctant ... to make needed changes.  You can easily install it to your 
home directory.  No admin needed for this.  Worked nicely.  Solved our 
problems.


Regards,

Joe

--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.,
email: land...@scalableinformatics.com
web  : http://www.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 866 888 3112
cell : +1 734 612 4615

Re: [OMPI users] How closely tied is a specific release of OpenMPI to the host operating system and other system software?

[Jeffrey A Cummings]

Thanks for all the good replies on this thread.  I don't know if I'll be 
able to make a dent in the corporate IT bureaucracy but I'm going to try.




From:   Prentice Bisbal 
To: Open MPI Users 
List-Post: users@lists.open-mpi.org
Date:   02/02/2011 11:35 AM
Subject:Re: [OMPI users] How closely tied is a specific release of 
OpenMPI to the host operating system and other system software?
Sent by:users-boun...@open-mpi.org



Jeffrey A Cummings wrote:
> I use OpenMPI on a variety of platforms:  stand-alone servers running
> Solaris on sparc boxes and Linux (mostly CentOS) on AMD/Intel boxes,
> also Linux (again CentOS) on large clusters of AMD/Intel boxes.  These
> platforms all have some version of the 1.3 OpenMPI stream.  I recently
> requested an upgrade on all systems to 1.4.3 (for production work) and
> 1.5.1 (for experimentation).  I'm getting a lot of push back from the
> SysAdmin folks claiming that OpenMPI is closely intertwined with the
> specific version of the operating system and/or other system software
> (i.e., Rocks on the clusters).  I need to know if they are telling me
> the truth or if they're just making excuses to avoid the work.  To state
> my question another way:  Apparently each release of Linux and/or Rocks
> comes with some version of OpenMPI bundled in.  Is it dangerous in some
> way to upgrade to a newer version of OpenMPI?  Thanks in advance for any
> insight anyone can provide.
> 
> - Jeff
> 

Jeff,

OpenMPI is more or less a user-space program, and isn't that tightly
coupled to the OS at all. As long as the OS has the correct network
drivers (ethernet, IB, or other), that's all OpenMPI needs to do it's
job. In fact, you can install it yourself in your own home directory (if
 your home directory is shared amongst the cluster nodes you want to
use), and run it from there - no special privileges needed.

I have many different versions of OpenMPI installed on my systems,
without a problem.

As a system administrator responsible for maintaining OpenMPI on several
clusters, it sounds like one of two things:

1. Your system administrators really don't know what they're talking
about, or,

2. They're lying to you to avoid doing work.

--
Prentice
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] OpenMPI version syntax?

[Jeffrey A Cummings]

The context was wrt the OpenMPI version that is bundled with a specific 
version of CentOS Linux which my IT folks are about to install on one of 
our servers.  Since the most recent 1.4 stream version is 1.4.3, I'm 
afraid that 1.4-4 is really some variant of 1.4 (i.e., 1.4.0) and hence 
not that new.




From:   Jeff Squyres 
To: Open MPI Users 
List-Post: users@lists.open-mpi.org
Date:   02/02/2011 07:38 PM
Subject:Re: [OMPI users] OpenMPI version syntax?
Sent by:users-boun...@open-mpi.org



On Feb 2, 2011, at 1:44 PM, Jeffrey A Cummings wrote:

> I've encountered a supposed OpenMPI version of 1.4-4.  Is the hyphen a 
typo or is this syntax correct and if so what does it mean? 

Is this an RPM version number?  It's fairly common for RPMs to add "-X" at 
the end of the version number.  The "X" indicates the RPM version number 
(i.e., the version number of the packaging -- not the package itself).

Open MPI's version number scheme is explained here:

http://www.open-mpi.org/software/ompi/versions/

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Calculate time spent on non blocking communication?

[Eugene Loh]

Again, you can try the Peruse instrumentation.  Configure OMPI with 
--enable-peruse.  The instrumentation points might help you decide how 
you want to define the time you want to measure.  Again, you really have 
to spend a bunch of your own time deciding what is meaningful to measure.


Gustavo Correa wrote:

However, OpenMPI may give this info, with non-MPI (hence non-portable) 
functions, I'd guess.



From: Eugene Loh 

Anyhow, the Peruse instrumentation in OMPI might help.

Re: [OMPI users] OpenMPI version syntax?

[Jeff Squyres]

On Feb 2, 2011, at 1:44 PM, Jeffrey A Cummings wrote:

> I've encountered a supposed OpenMPI version of 1.4-4.  Is the hyphen a typo 
> or is this syntax correct and if so what does it mean? 

Is this an RPM version number?  It's fairly common for RPMs to add "-X" at the 
end of the version number.  The "X" indicates the RPM version number (i.e., the 
version number of the packaging -- not the package itself).

Open MPI's version number scheme is explained here:

http://www.open-mpi.org/software/ompi/versions/

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] Calculate time spent on non blocking communication?

[Gustavo Correa]

On Feb 2, 2011, at 7:35 AM, Bibrak Qamar wrote:

> Gus Correa, But it will include the time of computation which took place 
> before waitAll( ).
> 
Correct.
Now I realize you want the communication time only,
not the overall time.

Even to define precisely what this means may be a bit tricky.
I guess Eugene Loh was hinting this difficulty.

Is it the time MPI_Isend takes to return?
It returns as soon as the send is posted, I think,
but this doesn't mean that the message was received at the other end,
or that you are free to use the send buffer either.
You could do it with 
start=MPI_Wtime()
call MPI_Isend(...)
end=MPI_Wtime()
but this may not be very useful.

Is it the time until MPI_[I]recv returns?
Well, that one could be measured the way I suggested,
but it will probably contain also time used to compute.

Is it something else, perhaps?

My guess is that it may be hard to nail down communication time alone, 
particularly in non-blocking case,
at least if you try to get it only with MPI calls.
Non-blocking is really designed to overlap communication and computation,
not to separate them.
Also, I guess MPI itself doesn't give user access to the details of *how* it 
provides communication, timing information included.
However, OpenMPI may give this info, with non-MPI (hence non-portable) 
functions, I'd guess.

Have you tried inserting MPI_Test calls as Jeff suggested? 
(Perhaps along with MPI_Wtime)

Maybe there is more timing information available outside MPI,
at another  OpenMPI internal level.
Perhaps this is accessible from  within the program, via non-MPI calls,
but I don't really know.
Maybe the OpenMPI developers have a hint.
Have you read Eugene Loh's suggestions?

Now I am asking more questions than answering anything. :)

Gus Correa
> 
> Date: Tue, 1 Feb 2011 10:09:03 +0400
> From: Bibrak Qamar 
> Subject: [OMPI users] Calculate time spent on non blocking
>communication?
> To: us...@open-mpi.org
> Message-ID:
>
> Content-Type: text/plain; charset="iso-8859-1"
> 
> Hello All,
> 
> I am using non-blocking send and receive, and i want to calculate the time
> it took for the communication. Is there any method or a way to do this using
> openmpi.
> 
> Thanks
> Bibrak Qamar
> Undergraduate Student BIT-9
> Member Center for High Performance Scientific Computing
> NUST-School of Electrical Engineering and Computer Science.
> -- next part --
> HTML attachment scrubbed and removed
> 
> --
> 
> Message: 4
> Date: Mon, 31 Jan 2011 22:14:53 -0800
> From: Eugene Loh 
> Subject: Re: [OMPI users] Calculate time spent on non blocking
>communication?
> To: Open MPI Users 
> Message-ID: <4d47a4dd.5010...@oracle.com>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
> 
> Bibrak Qamar wrote:
> 
> > Hello All,
> >
> > I am using non-blocking send and receive, and i want to calculate the
> > time it took for the communication. Is there any method or a way to do
> > this using openmpi.
> 
> You probably have to start by defining what you mean by "the time it
> took for the communication".  Anyhow, the Peruse instrumentation in OMPI
> might help.
> 
> 
> --
> 
> Message: 5
> Date: Tue, 1 Feb 2011 01:20:36 -0500
> From: Gustavo Correa 
> Subject: Re: [OMPI users] Calculate time spent on non blocking
>communication?
> To: Open MPI Users 
> Message-ID: <8f16054c-6fca-4e65-9c83-5efbfcb18...@ldeo.columbia.edu
> >
> Content-Type: text/plain; charset=us-ascii
> 
> 
> On Feb 1, 2011, at 1:09 AM, Bibrak Qamar wrote:
> 
> 
> 
> 
> > Hello All,
> >
> > I am using non-blocking send and receive, and i want to calculate the time 
> > it took for the communication. Is there any method or a way to do this 
> > using openmpi.
> >
> > Thanks
> > Bibrak Qamar
> > Undergraduate Student BIT-9
> > Member Center for High Performance Scientific Computing
> > NUST-School of Electrical Engineering and Computer Science.
> > ___
> 
> About the same as with blocking communication, I guess.
> 
> Would this do work for you?
> 
> start=MPI_Wtime()
> MPI_Isend(...)
> ...
> MPI_Irecv(...)
> ...
> MPI_Wait[all](...)
> end=MPI_Wtime()
> print *, 'walltime = ', end-start
> 
> My two cents,
> Gus Correa
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Check whether non-blocking communication has finished?

[David Zhang]

I think ultimately it would comes down to whether code execution is
more important than receiving the message in a timely fashion.

On 2/2/11, amjad ali  wrote:
> Perhaps often it is more useful to use MPI_WAIT rather than MPI_TEST type
> fucntions, because at MPI_WAIT point it will be taken care of communication
> completion, automatically, which may be necessary before going ahead. with
> MPI_TEST it would become the responsibility of the programmer to handle the
> situation if the test is FALSE/FAILED.
>
> On Wed, Feb 2, 2011 at 7:50 AM, Jeff Squyres  wrote:
>
>> Yes; you can use any of the various flavors of the MPI_TEST* functions.
>>
>> On Feb 2, 2011, at 7:03 AM, Bibrak Qamar wrote:
>>
>> > Hello All,
>> >
>> > Is there any way to find whether a non blocking communication has
>> finished without calling the wait( ) function.
>> >
>> >
>> > Thanks
>> > Bibrak Qamar
>> > Undergraduate Student BIT-9
>> > Member Center for High Performance Scientific Computing
>> > NUST-School of Electrical Engineering and Computer Science.
>> > ___
>> > users mailing list
>> > us...@open-mpi.org
>> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>>
>> --
>> Jeff Squyres
>> jsquy...@cisco.com
>> For corporate legal information go to:
>> http://www.cisco.com/web/about/doing_business/legal/cri/
>>
>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>

-- 
Sent from my mobile device

David Zhang
University of California, San Diego

Re: [OMPI users] Check whether non-blocking communication has finished?

[amjad ali]

Perhaps often it is more useful to use MPI_WAIT rather than MPI_TEST type
fucntions, because at MPI_WAIT point it will be taken care of communication
completion, automatically, which may be necessary before going ahead. with
MPI_TEST it would become the responsibility of the programmer to handle the
situation if the test is FALSE/FAILED.

On Wed, Feb 2, 2011 at 7:50 AM, Jeff Squyres  wrote:

> Yes; you can use any of the various flavors of the MPI_TEST* functions.
>
> On Feb 2, 2011, at 7:03 AM, Bibrak Qamar wrote:
>
> > Hello All,
> >
> > Is there any way to find whether a non blocking communication has
> finished without calling the wait( ) function.
> >
> >
> > Thanks
> > Bibrak Qamar
> > Undergraduate Student BIT-9
> > Member Center for High Performance Scientific Computing
> > NUST-School of Electrical Engineering and Computer Science.
> > ___
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

[OMPI users] OpenMPI version syntax?

[Jeffrey A Cummings]

I've encountered a supposed OpenMPI version of 1.4-4.  Is the hyphen a 
typo or is this syntax correct and if so what does it mean?

- Jeff

Re: [OMPI users] heterogenous cluster

[jody]

Thaks all

I did the simple copying of the 32Bit applications and now it works.

Thanks
  Jody

On Wed, Feb 2, 2011 at 5:47 PM, David Mathog  wrote:
> jody  wrote:
>
>> How can i force OpenMPI to be built as a 32Bit application on a 64Bit
> machine?
>
> THe easiest way is not to - just copy over a build from a 32 bit
> machine, it will run on your 64 bit machine if the proper 32 bit
> libraries have been installed there.  Otherwise,  you need to put -m32
> on the gcc commmand lines.  Generally one does that by something like:
>
>  export CFLAGS=-m32
>
> before running configure to generate Makefiles.
>
> Regards,
>
>
> David Mathog
> mat...@caltech.edu
> Manager, Sequence Analysis Facility, Biology Division, Caltech
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

Re: [OMPI users] Segmentation fault with SLURM and non-local nodes

[Samuel K. Gutierrez]

Hi,

We'll try to reproduce the problem.

Thanks,

--
Samuel K. Gutierrez
Los Alamos National Laboratory


On Feb 2, 2011, at 2:55 AM, Michael Curtis wrote:



On 28/01/2011, at 8:16 PM, Michael Curtis wrote:



On 27/01/2011, at 4:51 PM, Michael Curtis wrote:

Some more debugging information:
Is anyone able to help with this problem?  As far as I can tell it's  
a stock-standard recently installed SLURM installation.


I can try 1.5.1 but hesitant to deploy this as it would require a  
recompile of some rather large pieces of software.  Should I re-post  
to the -devel lists?


Regards,


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] heterogenous cluster

[David Mathog]

jody  wrote:

> How can i force OpenMPI to be built as a 32Bit application on a 64Bit
machine?

THe easiest way is not to - just copy over a build from a 32 bit
machine, it will run on your 64 bit machine if the proper 32 bit
libraries have been installed there.  Otherwise,  you need to put -m32
on the gcc commmand lines.  Generally one does that by something like:

 export CFLAGS=-m32

before running configure to generate Makefiles.

Regards,


David Mathog
mat...@caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech

Re: [OMPI users] How closely tied is a specific release of OpenMPI to the host operating system and other system software?

[Prentice Bisbal]

Jeffrey A Cummings wrote:
> I use OpenMPI on a variety of platforms:  stand-alone servers running
> Solaris on sparc boxes and Linux (mostly CentOS) on AMD/Intel boxes,
> also Linux (again CentOS) on large clusters of AMD/Intel boxes.  These
> platforms all have some version of the 1.3 OpenMPI stream.  I recently
> requested an upgrade on all systems to 1.4.3 (for production work) and
> 1.5.1 (for experimentation).  I'm getting a lot of push back from the
> SysAdmin folks claiming that OpenMPI is closely intertwined with the
> specific version of the operating system and/or other system software
> (i.e., Rocks on the clusters).  I need to know if they are telling me
> the truth or if they're just making excuses to avoid the work.  To state
> my question another way:  Apparently each release of Linux and/or Rocks
> comes with some version of OpenMPI bundled in.  Is it dangerous in some
> way to upgrade to a newer version of OpenMPI?  Thanks in advance for any
> insight anyone can provide.
> 
> - Jeff
> 

Jeff,

OpenMPI is more or less a user-space program, and isn't that tightly
coupled to the OS at all. As long as the OS has the correct network
drivers (ethernet, IB, or other), that's all OpenMPI needs to do it's
job. In fact, you can install it yourself in your own home directory (if
 your home directory is shared amongst the cluster nodes you want to
use), and run it from there - no special privileges needed.

I have many different versions of OpenMPI installed on my systems,
without a problem.

As a system administrator responsible for maintaining OpenMPI on several
clusters, it sounds like one of two things:

1. Your system administrators really don't know what they're talking
about, or,

2. They're lying to you to avoid doing work.

--
Prentice

Re: [OMPI users] Calculate time spent on non blocking communication?

[Eugene Loh]

Bibrak Qamar wrote:

  Gus Correa, But it will include the time of
computation which took place before waitAll( ).
  

What's wrong with that?

  From: Bibrak Qamar 
I am using non-blocking send and receive, and i want to calculate the
time
it took for the communication.
  
From: Eugene Loh 
You probably have to start by defining what you mean by "the time it
took for the communication".  Anyhow, the Peruse instrumentation in OMPI
might help.
  

Again, you should probably start by thinking more precisely about what
time you want to measure.  E.g., ask yourself why the answer even
matters.

Re: [OMPI users] How closely tied is a specific release of OpenMPIto the host operating system and other system software?

[Mehdi Bozzo-Rey]

Hey,

I work for Platform Computing, we do have our clustering tool as well
and I play a lot with others (PCM, ROCKS, xCAT, Unicluster ...). I never
ran into any problem switching OpenMPI versions. In my case I use
environment modules and it just works like a charm :-) You can also have
modules dependencies (like app A was compiled with a very specific
version of OpenMPI so when you load the environment for app A you are
basically "forced" to load the corresponding environment module for
OpenMPI).
Now it may be a little bit more difficult if you are in an heterogeneous
environment (from an OS point of view), like nodes running different
versions of RHEL and SLES. 

Cheers,

Mehdi

Mehdi Bozzo-Rey
HPC Solutions Architect
Platform computing
Phone: +1 905 948 4649



-Original Message-
From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On
Behalf Of Jeff Squyres
Sent: February-01-11 7:35 PM
To: Open MPI Users
Subject: Re: [OMPI users] How closely tied is a specific release of
OpenMPIto the host operating system and other system software?

On Feb 1, 2011, at 5:02 PM, Jeffrey A Cummings wrote:

> I'm getting a lot of push back from the SysAdmin folks claiming that
OpenMPI is closely intertwined with the specific version of the
operating system and/or other system software (i.e., Rocks on the
clusters).  

I wouldn't say that this is true.  We test across a wide variety of OS's
and compilers.  I'm sure that there are particular
platforms/environments that can trip up some kind of problem (it's
happened before), but in general, Open MPI is pretty portable.

> To state my question another way:  Apparently each release of Linux
and/or Rocks comes with some version of OpenMPI bundled in.  Is it
dangerous in some way to upgrade to a newer version of OpenMPI?  

Not at all.  Others have said it, but I'm one of the developers and I'll
reinforce their answers: I regularly have about a dozen different
installations of Open MPI on my cluster at any given time (all in
different stages of development -- all installed to different prefixes).
I switch between them quite easily by changing my PATH and
LD_LIBRARY_PATH (both locally and on remote nodes).

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] How closely tied is a specific release of OpenMPI to the host operating system and other system software?

[Jeff Squyres]

On Feb 2, 2011, at 7:02 AM, Terry Dontje wrote:

> 2.  The system libraries on different linux versions are not always the same. 
>  At Oracle we build a binary distribution of OMPI that we test out on several 
> different versions of Linux.  The key here is building on a machine that is 
> essentially the lowest common denominator of all the system software that 
> exists on the machines one will be running on.  This is essentially why 
> Oracle states a bounded set of OS versions a distribution runs on.  An 
> example of this is there is a component in OMPI that was relying on a version 
> of libbfd that changed significantly between Linux version.  Once we got rid 
> of the usage of that library we were ok.  There are not "a lot" of these 
> instances but the number is not zero.  

+1

If your systems are not running exactly the same versions of software (e.g., 
different OS's), we typically categorize this as a "heterogeneous" scenario -- 
with all the caveats that Terry mentions.  You may well likely require a 
separate OMPI install for each OS, and possibly (likely) even a different 
compilation of your application for each OS.

In general, I try to limit my OMPI jobs to homogeneous platforms.  
Heterogeneous can be done; it's just somewhat of a pain to get right.

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] Check whether non-blocking communication has finished?

[Jeff Squyres]

Yes; you can use any of the various flavors of the MPI_TEST* functions.

On Feb 2, 2011, at 7:03 AM, Bibrak Qamar wrote:

> Hello All,
> 
> Is there any way to find whether a non blocking communication has finished 
> without calling the wait( ) function.
> 
> 
> Thanks
> Bibrak Qamar
> Undergraduate Student BIT-9
> Member Center for High Performance Scientific Computing
> NUST-School of Electrical Engineering and Computer Science.
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users


-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] Calculate time spent on non blocking communication?

[Bibrak Qamar]

Gus Correa, But it will include the time of computation which took place
before waitAll( ).


List-Post: users@lists.open-mpi.org
Date: Tue, 1 Feb 2011 10:09:03 +0400
From: Bibrak Qamar 
Subject: [OMPI users] Calculate time spent on non blocking
   communication?
To: us...@open-mpi.org
Message-ID:
   
Content-Type: text/plain; charset="iso-8859-1"

Hello All,

I am using non-blocking send and receive, and i want to calculate the time
it took for the communication. Is there any method or a way to do this using
openmpi.

Thanks
Bibrak Qamar
Undergraduate Student BIT-9
Member Center for High Performance Scientific Computing
NUST-School of Electrical Engineering and Computer Science.
-- next part --
HTML attachment scrubbed and removed

--

Message: 4
List-Post: users@lists.open-mpi.org
Date: Mon, 31 Jan 2011 22:14:53 -0800
From: Eugene Loh 
Subject: Re: [OMPI users] Calculate time spent on non blocking
   communication?
To: Open MPI Users 
Message-ID: <4d47a4dd.5010...@oracle.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

Bibrak Qamar wrote:

> Hello All,
>
> I am using non-blocking send and receive, and i want to calculate the
> time it took for the communication. Is there any method or a way to do
> this using openmpi.

You probably have to start by defining what you mean by "the time it
took for the communication".  Anyhow, the Peruse instrumentation in OMPI
might help.


--

Message: 5
List-Post: users@lists.open-mpi.org
Date: Tue, 1 Feb 2011 01:20:36 -0500
From: Gustavo Correa 
Subject: Re: [OMPI users] Calculate time spent on non blocking
   communication?
To: Open MPI Users 
Message-ID: <8f16054c-6fca-4e65-9c83-5efbfcb18...@ldeo.columbia.edu
>
Content-Type: text/plain; charset=us-ascii


On Feb 1, 2011, at 1:09 AM, Bibrak Qamar wrote:




> Hello All,
>
> I am using non-blocking send and receive, and i want to calculate the time
it took for the communication. Is there any method or a way to do this using
openmpi.
>
> Thanks
> Bibrak Qamar
> Undergraduate Student BIT-9
> Member Center for High Performance Scientific Computing
> NUST-School of Electrical Engineering and Computer Science.
> ___

About the same as with blocking communication, I guess.

Would this do work for you?

start=MPI_Wtime()
MPI_Isend(...)
...
MPI_Irecv(...)
...
MPI_Wait[all](...)
end=MPI_Wtime()
print *, 'walltime = ', end-start

My two cents,
Gus Correa

[OMPI users] Check whether non-blocking communication has finished?

[Bibrak Qamar]

Hello All,

Is there any way to find whether a non blocking communication has finished
without calling the wait( ) function.


Thanks
Bibrak Qamar
Undergraduate Student BIT-9
Member Center for High Performance Scientific Computing
NUST-School of Electrical Engineering and Computer Science.

Re: [OMPI users] How closely tied is a specific release of OpenMPI to the host operating system and other system software?

[Terry Dontje]

On 02/01/2011 07:34 PM, Jeff Squyres wrote:

On Feb 1, 2011, at 5:02 PM, Jeffrey A Cummings wrote:


I'm getting a lot of push back from the SysAdmin folks claiming that OpenMPI is 
closely intertwined with the specific version of the operating system and/or 
other system software (i.e., Rocks on the clusters).

I wouldn't say that this is true.  We test across a wide variety of OS's and 
compilers.  I'm sure that there are particular platforms/environments that can 
trip up some kind of problem (it's happened before), but in general, Open MPI 
is pretty portable.


To state my question another way:  Apparently each release of Linux and/or 
Rocks comes with some version of OpenMPI bundled in.  Is it dangerous in some 
way to upgrade to a newer version of OpenMPI?

Not at all.  Others have said it, but I'm one of the developers and I'll 
reinforce their answers: I regularly have about a dozen different installations 
of Open MPI on my cluster at any given time (all in different stages of 
development -- all installed to different prefixes).  I switch between them 
quite easily by changing my PATH and LD_LIBRARY_PATH (both locally and on 
remote nodes).
Not to be a lone descenting opinion here is my experience in doing the 
above.


First if you are always recompiling your application with a specific 
version of OMPI then I would agree with everything Jeff said above.  
That is you can build many versions of OMPI on many linux versions and 
have them run.


But there are definite pitfalls once you start trying to keep one set of 
executables and OMPI binaries across different Linux versions.


1.  You may see executables not be able to use OMPI libraries that 
differ in the first dot number release (eg 1.3 vs 1.4 or 1.5 branches).  
We the community try to avoid these incompatibilities as much as 
possible but it happens on occasion (I think 1.3 to 1.4 is one such 
occasion).


2.  The system libraries on different linux versions are not always the 
same.  At Oracle we build a binary distribution of OMPI that we test out 
on several different versions of Linux.  The key here is building on a 
machine that is essentially the lowest common denominator of all the 
system software that exists on the machines one will be running on.  
This is essentially why Oracle states a bounded set of OS versions a 
distribution runs on.  An example of this is there is a component in 
OMPI that was relying on a version of libbfd that changed significantly 
between Linux version.  Once we got rid of the usage of that library we 
were ok.  There are not "a lot" of these instances but the number is not 
zero.


--
Oracle
Terry D. Dontje | Principal Software Engineer
Developer Tools Engineering | +1.781.442.2631
Oracle *- Performance Technologies*
95 Network Drive, Burlington, MA 01803
Email terry.don...@oracle.com 

Re: [OMPI users] Segmentation fault with SLURM and non-local nodes

[Michael Curtis]

On 28/01/2011, at 8:16 PM, Michael Curtis wrote:

> 
> On 27/01/2011, at 4:51 PM, Michael Curtis wrote:
> 
> Some more debugging information:
Is anyone able to help with this problem?  As far as I can tell it's a 
stock-standard recently installed SLURM installation.

I can try 1.5.1 but hesitant to deploy this as it would require a recompile of 
some rather large pieces of software.  Should I re-post to the -devel lists?

Regards,

Re: [OMPI users] heterogenous cluster

[Doug Reeder]

Jody,

With the gnu compilers the -m32  flag works. With other compilire's the same or 
other flag should work. 

Doug Reeder
On Feb 1, 2011, at 11:46 PM, jody wrote:

> Thanks for your reply.
> 
> If i try your suggestion, every process fails with the following message:
> 
> *** The MPI_Init() function was called before MPI_INIT was invoked.
> *** This is disallowed by the MPI standard.
> *** Your MPI job will now abort.
> [aim-triops:15460] Abort before MPI_INIT completed successfully; not
> able to guarantee that all other processes were killed!
> 
> I think this is caused by the fact that on the 64Bit machine Open MPI
> is also built as a 64 bit application.
> How can i force OpenMPI to be built as a 32Bit application on a 64Bit machine?
> 
> Thank You
> Jody
> 
> On Tue, Feb 1, 2011 at 9:00 PM, David Mathog  wrote:
>> 
>>> I have sofar used a homogenous 32-bit cluster.
>>> Now i have added a new machine which is 64 bit
>>> 
>>> This means i have to reconfigure open MPI with
>> `--enable-heterogeneous`, right?
>> 
>> Not necessarily.  If you don't need the 64bit capabilities you could run
>> 32 bit binaries along with a 32 bit version of OpenMPI.  At least that
>> approach has worked so far for me.
>> 
>> Regards,
>> 
>> David Mathog
>> mat...@caltech.edu
>> Manager, Sequence Analysis Facility, Biology Division, Caltech
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] heterogenous cluster

[jody]

Thanks for your reply.

If i try your suggestion, every process fails with the following message:

*** The MPI_Init() function was called before MPI_INIT was invoked.
*** This is disallowed by the MPI standard.
*** Your MPI job will now abort.
[aim-triops:15460] Abort before MPI_INIT completed successfully; not
able to guarantee that all other processes were killed!

I think this is caused by the fact that on the 64Bit machine Open MPI
is also built as a 64 bit application.
How can i force OpenMPI to be built as a 32Bit application on a 64Bit machine?

Thank You
Jody

On Tue, Feb 1, 2011 at 9:00 PM, David Mathog  wrote:
>
>> I have sofar used a homogenous 32-bit cluster.
>> Now i have added a new machine which is 64 bit
>>
>> This means i have to reconfigure open MPI with
> `--enable-heterogeneous`, right?
>
> Not necessarily.  If you don't need the 64bit capabilities you could run
> 32 bit binaries along with a 32 bit version of OpenMPI.  At least that
> approach has worked so far for me.
>
> Regards,
>
> David Mathog
> mat...@caltech.edu
> Manager, Sequence Analysis Facility, Biology Division, Caltech
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>