Re: [OMPI users] FW: mpirun hangs when used on more than 2 CPUs ( mpirun compiled without thread support )
The thought occurs to me... (disclaimer: I know just about zero about OpenFoam and how to install/use it) If your customer has been dealing with binaries, I wonder if there is some kind of ABI incompatibility going on here. Open MPI did not provide any ABI guarantees until Open MPI v1.3.2 -- see http://www.open-mpi.org/software/ompi/versions/ for details. Also, Open MPI v1.3.2 is a bit old. There have been many bug fixes since then -- 1.4.4 is the latest stable. There will be a 1.4.5 shortly, but that will be the last on the 1.4 series. On Jan 19, 2012, at 5:51 AM, Theiner, Andre wrote: > Hi all, > I have to stop my investigations and repairs on the request of my customer. > I will unsubscribe from this list soon. > > I found out that OpenFoam does not use threaded MPI-calls. > My next step would have been to compile openmpi-1.4.4 and have the user try > this. > In case it would have also not worked I would have compiled the whole > OpenFoam from the sources. > Up to now the user uses a rpm binary version of OF 2.0.1. > > Thanks for all your support. > > > Andre > > > -Original Message- > From: Theiner, Andre > Sent: Mittwoch, 18. Januar 2012 10:15 > To: 'Open MPI Users' > Subject: RE: [OMPI users] mpirun hangs when used on more than 2 CPUs ( mpirun > compiled without thread support ) > Importance: High > > Thanks, Jeff and Ralph for your good help. > I do not know yet, whether OpenFoam uses threads with OpenMPI but I will find > out. > > I ran "ompi_info" and it output the lines in the next chapter. > The important line is " Thread support: posix (mpi: no, progress: no)". > At first sight the above line made me think that I found the cause of the > problem > but I compared the output to the output of the same command run on another > machine > where OpenFoam runs fine. The OpenMPI version of that machine is 1.3.2-1.1 > and it > also does not have thread support. > The difference though is that that machine's OpenFoam version is 1.7.1 and > not 2.0.1 and the > OS is SUSE Linux Enterprise Desktop 11 SP1 and not openSUSE 11.3. > So I am at the beginning of the search for the cause of the problem. > > Package: Open MPI abuild@build30 Distribution >Open MPI: 1.3.2 > Open MPI SVN revision: r21054 > Open MPI release date: Apr 21, 2009 >Open RTE: 1.3.2 > Open RTE SVN revision: r21054 > Open RTE release date: Apr 21, 2009 >OPAL: 1.3.2 > OPAL SVN revision: r21054 > OPAL release date: Apr 21, 2009 >Ident string: 1.3.2 > Prefix: /usr/lib64/mpi/gcc/openmpi > Configured architecture: x86_64-unknown-linux-gnu > Configure host: build30 > Configured by: abuild > Configured on: Fri Sep 23 05:58:54 UTC 2011 > Configure host: build30 >Built by: abuild >Built on: Fri Sep 23 06:11:31 UTC 2011 > Built host: build30 > C bindings: yes >C++ bindings: yes > Fortran77 bindings: yes (all) > Fortran90 bindings: yes > Fortran90 bindings size: small > C compiler: gcc > C compiler absolute: /usr/bin/gcc >C++ compiler: g++ > C++ compiler absolute: /usr/bin/g++ > Fortran77 compiler: gfortran > Fortran77 compiler abs: /usr/bin/gfortran > Fortran90 compiler: gfortran > Fortran90 compiler abs: /usr/bin/gfortran > C profiling: yes > C++ profiling: yes > Fortran77 profiling: yes > Fortran90 profiling: yes > C++ exceptions: no > Thread support: posix (mpi: no, progress: no) > Sparse Groups: no > Internal debug support: no > MPI parameter check: runtime > Memory profiling support: no > Memory debugging support: no > libltdl support: yes > Heterogeneous support: no > mpirun default --prefix: no > MPI I/O support: yes > MPI_WTIME support: gettimeofday > Symbol visibility support: yes > FT Checkpoint support: no (checkpoint thread: no) > MCA backtrace: execinfo (MCA v2.0, API v2.0, Component v1.3.2) > MCA memory: ptmalloc2 (MCA v2.0, API v2.0, Component v1.3.2) > MCA paffinity: linux (MCA v2.0, API v2.0, Component v1.3.2) > MCA carto: auto_detect (MCA v2.0, API v2.0, Component v1.3.2) > MCA carto: file (MCA v2.0, API v2.0, Component v1.3.2) > MCA maffinity: first_use (MCA v2.0, API v2.0, Component v1.3.2) > MCA timer: linux (MCA v2.0, API v2.0, Component v1.3.2) > MCA installdirs: env (MCA v2.0, API v2.0, Component v1.3.2) > MCA installdirs: config (MCA v2.0, API v2.0, Component v1.3.2) > MCA dpm: orte (MCA v2.0, API v2.0, Component v1.3.2) > MCA pubsub: orte (MCA v2.0, API v2.0, Component v1.3.2) > MCA allocator: basic (MCA v2.0, API v2.0, Component v1.3.2) > MCA allocator: bucket (MCA v2.0, API
[OMPI users] FW: mpirun hangs when used on more than 2 CPUs ( mpirun compiled without thread support )
Hi all, I have to stop my investigations and repairs on the request of my customer. I will unsubscribe from this list soon. I found out that OpenFoam does not use threaded MPI-calls. My next step would have been to compile openmpi-1.4.4 and have the user try this. In case it would have also not worked I would have compiled the whole OpenFoam from the sources. Up to now the user uses a rpm binary version of OF 2.0.1. Thanks for all your support. Andre -Original Message- From: Theiner, Andre Sent: Mittwoch, 18. Januar 2012 10:15 To: 'Open MPI Users' Subject: RE: [OMPI users] mpirun hangs when used on more than 2 CPUs ( mpirun compiled without thread support ) Importance: High Thanks, Jeff and Ralph for your good help. I do not know yet, whether OpenFoam uses threads with OpenMPI but I will find out. I ran "ompi_info" and it output the lines in the next chapter. The important line is " Thread support: posix (mpi: no, progress: no)". At first sight the above line made me think that I found the cause of the problem but I compared the output to the output of the same command run on another machine where OpenFoam runs fine. The OpenMPI version of that machine is 1.3.2-1.1 and it also does not have thread support. The difference though is that that machine's OpenFoam version is 1.7.1 and not 2.0.1 and the OS is SUSE Linux Enterprise Desktop 11 SP1 and not openSUSE 11.3. So I am at the beginning of the search for the cause of the problem. Package: Open MPI abuild@build30 Distribution Open MPI: 1.3.2 Open MPI SVN revision: r21054 Open MPI release date: Apr 21, 2009 Open RTE: 1.3.2 Open RTE SVN revision: r21054 Open RTE release date: Apr 21, 2009 OPAL: 1.3.2 OPAL SVN revision: r21054 OPAL release date: Apr 21, 2009 Ident string: 1.3.2 Prefix: /usr/lib64/mpi/gcc/openmpi Configured architecture: x86_64-unknown-linux-gnu Configure host: build30 Configured by: abuild Configured on: Fri Sep 23 05:58:54 UTC 2011 Configure host: build30 Built by: abuild Built on: Fri Sep 23 06:11:31 UTC 2011 Built host: build30 C bindings: yes C++ bindings: yes Fortran77 bindings: yes (all) Fortran90 bindings: yes Fortran90 bindings size: small C compiler: gcc C compiler absolute: /usr/bin/gcc C++ compiler: g++ C++ compiler absolute: /usr/bin/g++ Fortran77 compiler: gfortran Fortran77 compiler abs: /usr/bin/gfortran Fortran90 compiler: gfortran Fortran90 compiler abs: /usr/bin/gfortran C profiling: yes C++ profiling: yes Fortran77 profiling: yes Fortran90 profiling: yes C++ exceptions: no Thread support: posix (mpi: no, progress: no) Sparse Groups: no Internal debug support: no MPI parameter check: runtime Memory profiling support: no Memory debugging support: no libltdl support: yes Heterogeneous support: no mpirun default --prefix: no MPI I/O support: yes MPI_WTIME support: gettimeofday Symbol visibility support: yes FT Checkpoint support: no (checkpoint thread: no) MCA backtrace: execinfo (MCA v2.0, API v2.0, Component v1.3.2) MCA memory: ptmalloc2 (MCA v2.0, API v2.0, Component v1.3.2) MCA paffinity: linux (MCA v2.0, API v2.0, Component v1.3.2) MCA carto: auto_detect (MCA v2.0, API v2.0, Component v1.3.2) MCA carto: file (MCA v2.0, API v2.0, Component v1.3.2) MCA maffinity: first_use (MCA v2.0, API v2.0, Component v1.3.2) MCA timer: linux (MCA v2.0, API v2.0, Component v1.3.2) MCA installdirs: env (MCA v2.0, API v2.0, Component v1.3.2) MCA installdirs: config (MCA v2.0, API v2.0, Component v1.3.2) MCA dpm: orte (MCA v2.0, API v2.0, Component v1.3.2) MCA pubsub: orte (MCA v2.0, API v2.0, Component v1.3.2) MCA allocator: basic (MCA v2.0, API v2.0, Component v1.3.2) MCA allocator: bucket (MCA v2.0, API v2.0, Component v1.3.2) MCA coll: basic (MCA v2.0, API v2.0, Component v1.3.2) MCA coll: hierarch (MCA v2.0, API v2.0, Component v1.3.2) MCA coll: inter (MCA v2.0, API v2.0, Component v1.3.2) MCA coll: self (MCA v2.0, API v2.0, Component v1.3.2) MCA coll: sm (MCA v2.0, API v2.0, Component v1.3.2) MCA coll: sync (MCA v2.0, API v2.0, Component v1.3.2) MCA coll: tuned (MCA v2.0, API v2.0, Component v1.3.2) MCA io: romio (MCA v2.0, API v2.0, Component v1.3.2) MCA mpool: fake (MCA v2.0, API v2.0, Component v1.3.2) MCA mpool: rdma (MCA v2.0, API v2.0, Component v1.3.2) MCA mpool: sm (MCA
