Re: [OMPI users] MVAPI Options on Job Submission

[Brock Palen]

You will know if it doesn't,  You will have a bunch of messages about  
not finding a ib card and that openMPI is falling back to another  
transport.


Do all your nodes have infiniband?

Brock Palen
Center for Advanced Computing
bro...@umich.edu
(734)936-1985


On Aug 23, 2007, at 9:27 PM, Jeff Pummill wrote:


I have successfully compiled Open MPI 1.2.3 against Intel 8.1 compiler
suite and old (3 years) mvapi stack using the following configure:

configure --prefix=/nfsutil/openmpi-1.2.3
--with-mvapi=/usr/local/topspin/ CC=icc CXX=icpc F77=ifort FC=ifort

Do I need to assign any particular flags to the command line  
submission

to ensure that it is using the IB network instead of the TCP? Or
possibly disable the Gig-E with ^tcp to see if it still runs  
successfully?


I just want to be sure that Open MPI is actually USING the IB network
and mvapi.

Thanks!

Jeff Pummill

___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

[OMPI users] MVAPI Options on Job Submission

[Jeff Pummill]

I have successfully compiled Open MPI 1.2.3 against Intel 8.1 compiler 
suite and old (3 years) mvapi stack using the following configure:


configure --prefix=/nfsutil/openmpi-1.2.3 
--with-mvapi=/usr/local/topspin/ CC=icc CXX=icpc F77=ifort FC=ifort


Do I need to assign any particular flags to the command line submission 
to ensure that it is using the IB network instead of the TCP? Or 
possibly disable the Gig-E with ^tcp to see if it still runs successfully?


I just want to be sure that Open MPI is actually USING the IB network 
and mvapi.


Thanks!

Jeff Pummill

Re: [OMPI users] openmpi realloc() holding onto memory when glibc doesn't

[Tim Prins]

Hi Josh,

I am not an expert in this area of the code, but I'll give it a shot. 

(I assume you are using linux due to your email address) When using the memory 
manager (which is the default on linux), we wrap malloc/realloc/etc with 
ptmalloc2 (which is the same allocator used in glibc 2.3.x).

What I believe is happening is that ptmalloc2 is requesting more memory than 
necessary from the OS, and then lazily releasing it back. Try looking in the 
ompi source at opal/mca/memory/ptmalloc2/README ( 
https://svn.open-mpi.org/trac/ompi/browser/tags/v1.2-series/v1.2.3/opal/mca/memory/ptmalloc2/README#L121
 ).

This mentions some environment variables that can be set to alter ptmalloc2's 
behavior, although I have no idea if they work.

Alternatively, if you are not using a high performance network, there is 
little reason to use the memory manager, so you could just disable it.

Tim

On Thursday 23 August 2007 10:18:45 am Josh Aune wrote:
> I have found that the infiniserv MPI that comes with our IB software
> distribution tracks the same behaviour as gcc (releaseing memory on
> realloc).  I have also found that building openmpi with
> --without-memory-manager makes openmpi track the same behaviour as
> glibc.   I'm guessing that there is a bug in the pinned pages caching
> code?
>
> On 8/21/07, Josh Aune  wrote:
> > The realloc included with openmpi 1.2.3 is not releasing memory to the
> > OS and is causing apps to go into swap.  Attached is a little test
> > program that shows calls to realloc not releasing the memory when
> > compiled using mpicc, but when compiled directly with gcc (or icc)
> > calling realloc() frees any memory no longer needed.
> >
> > Is this a bug?
> >
> > If not, how can I force openmpi to free the memory that the allocator
> > is sitting on?
> >
> > Thanks,
> > Josh
> >
> > Sample output.  Note the delta between 'total' and 'malloc held' when
> > compiled with mpicc and how the gcc compiled versions track perfectly.
> >
> > $ mpicc -o realloc_test realloc_test.c
> > $ ./realloc_test
> > ...
> > malloc/realloc/free test
> > malloc()50 MB, total   50 MB, malloc held   50 MB
> > realloc()1 MB, total1 MB, malloc held   50 MB
> > malloc()50 MB, total   51 MB, malloc held  100 MB
> > realloc()1 MB, total2 MB, malloc held  100 MB
> > malloc()50 MB, total   52 MB, malloc held  150 MB
> > realloc()1 MB, total3 MB, malloc held  150 MB
> > malloc()50 MB, total   53 MB, malloc held  200 MB
> > realloc()1 MB, total4 MB, malloc held  200 MB
> > malloc()50 MB, total   54 MB, malloc held  250 MB
> > realloc()1 MB, total5 MB, malloc held  250 MB
> > free()   1 MB, total4 MB, malloc held  200 MB
> > free()   1 MB, total3 MB, malloc held  150 MB
> > free()   1 MB, total2 MB, malloc held  100 MB
> > free()   1 MB, total1 MB, malloc held   50 MB
> > free()   1 MB, total0 MB, malloc held0 MB
> > ...
> >
> > $ gcc -o realloc_test realloc_test.c
> > $ ./realloc_test
> > ...
> > malloc/realloc/free test
> > malloc()50 MB, total   50 MB, malloc held   50 MB
> > realloc()1 MB, total1 MB, malloc held1 MB
> > malloc()50 MB, total   51 MB, malloc held   51 MB
> > realloc()1 MB, total2 MB, malloc held2 MB
> > malloc()50 MB, total   52 MB, malloc held   52 MB
> > realloc()1 MB, total3 MB, malloc held3 MB
> > malloc()50 MB, total   53 MB, malloc held   53 MB
> > realloc()1 MB, total4 MB, malloc held4 MB
> > malloc()50 MB, total   54 MB, malloc held   54 MB
> > realloc()1 MB, total5 MB, malloc held5 MB
> > free()   1 MB, total4 MB, malloc held4 MB
> > free()   1 MB, total3 MB, malloc held3 MB
> > free()   1 MB, total2 MB, malloc held2 MB
> > free()   1 MB, total1 MB, malloc held1 MB
> > free()   1 MB, total0 MB, malloc held0 MB
> > ...
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] MPI_FILE_NULL

[Brian Barrett]

On Aug 23, 2007, at 4:33 AM, Bernd Schubert wrote:

I need to compile a benchmarking program and absolutely so far do  
not have

any experience with any MPI.
However, this looks like a general open-mpi problem, doesn't it?

bschubert@lanczos MPI_IO> make
cp ../globals.f90 ./; mpif90 -O2 -c ../globals.f90
mpif90 -O2 -c main.f90
mpif90 -O2 -c reader.f90
fortcom: Error: reader.f90, line 24: This name does not have a  
type, and

must have an explicit type.   [MPI_FILE_NULL]
call MPI_File_set_errhandler (MPI_FILE_NULL, MPI_ERRORS_ARE_FATAL,  
ierror)


Yeah, that looks like a mistake on our part.  It will be fixed in  
Open MPI 1.2.4.  Your quick fix should work until then.



Thanks,

Brian

--
  Brian W. Barrett
  Networking Team, CCS-1
  Los Alamos National Laboratory

Re: [OMPI users] openmpi realloc() holding onto memory when glibc doesn't

[Josh Aune]

I have found that the infiniserv MPI that comes with our IB software
distribution tracks the same behaviour as gcc (releaseing memory on
realloc).  I have also found that building openmpi with
--without-memory-manager makes openmpi track the same behaviour as
glibc.   I'm guessing that there is a bug in the pinned pages caching
code?

On 8/21/07, Josh Aune  wrote:
> The realloc included with openmpi 1.2.3 is not releasing memory to the
> OS and is causing apps to go into swap.  Attached is a little test
> program that shows calls to realloc not releasing the memory when
> compiled using mpicc, but when compiled directly with gcc (or icc)
> calling realloc() frees any memory no longer needed.
>
> Is this a bug?
>
> If not, how can I force openmpi to free the memory that the allocator
> is sitting on?
>
> Thanks,
> Josh
>
> Sample output.  Note the delta between 'total' and 'malloc held' when
> compiled with mpicc and how the gcc compiled versions track perfectly.
>
> $ mpicc -o realloc_test realloc_test.c
> $ ./realloc_test
> ...
> malloc/realloc/free test
> malloc()50 MB, total   50 MB, malloc held   50 MB
> realloc()1 MB, total1 MB, malloc held   50 MB
> malloc()50 MB, total   51 MB, malloc held  100 MB
> realloc()1 MB, total2 MB, malloc held  100 MB
> malloc()50 MB, total   52 MB, malloc held  150 MB
> realloc()1 MB, total3 MB, malloc held  150 MB
> malloc()50 MB, total   53 MB, malloc held  200 MB
> realloc()1 MB, total4 MB, malloc held  200 MB
> malloc()50 MB, total   54 MB, malloc held  250 MB
> realloc()1 MB, total5 MB, malloc held  250 MB
> free()   1 MB, total4 MB, malloc held  200 MB
> free()   1 MB, total3 MB, malloc held  150 MB
> free()   1 MB, total2 MB, malloc held  100 MB
> free()   1 MB, total1 MB, malloc held   50 MB
> free()   1 MB, total0 MB, malloc held0 MB
> ...
>
> $ gcc -o realloc_test realloc_test.c
> $ ./realloc_test
> ...
> malloc/realloc/free test
> malloc()50 MB, total   50 MB, malloc held   50 MB
> realloc()1 MB, total1 MB, malloc held1 MB
> malloc()50 MB, total   51 MB, malloc held   51 MB
> realloc()1 MB, total2 MB, malloc held2 MB
> malloc()50 MB, total   52 MB, malloc held   52 MB
> realloc()1 MB, total3 MB, malloc held3 MB
> malloc()50 MB, total   53 MB, malloc held   53 MB
> realloc()1 MB, total4 MB, malloc held4 MB
> malloc()50 MB, total   54 MB, malloc held   54 MB
> realloc()1 MB, total5 MB, malloc held5 MB
> free()   1 MB, total4 MB, malloc held4 MB
> free()   1 MB, total3 MB, malloc held3 MB
> free()   1 MB, total2 MB, malloc held2 MB
> free()   1 MB, total1 MB, malloc held1 MB
> free()   1 MB, total0 MB, malloc held0 MB
> ...
>
>

[OMPI users] MPI_FILE_NULL

[Bernd Schubert]

Hi,

I need to compile a benchmarking program and absolutely so far do not have
any experience with any MPI.
However, this looks like a general open-mpi problem, doesn't it?

bschubert@lanczos MPI_IO> make
cp ../globals.f90 ./; mpif90 -O2 -c ../globals.f90
mpif90 -O2 -c main.f90
mpif90 -O2 -c reader.f90
fortcom: Error: reader.f90, line 24: This name does not have a type, and
must have an explicit type.   [MPI_FILE_NULL]
call MPI_File_set_errhandler (MPI_FILE_NULL, MPI_ERRORS_ARE_FATAL, ierror)
--^


When I define

integer::MPI_FILE_NULL=0

compiling works fine.


Thanks for any help,
Bernd