Well, the whole situation is really bizarre.
I just uninstalled openmpi 1.3.2 on my system. Then I installed OFED
1.4.1 to see if that resolves this situation.
Here's what I get:
[root@master ~]# ompi_info --config
Configured by: root
Configured on: Wed Jun 24 11:10:00 EDT 2009
Configure host: master.org
Built by: root
Built on: Wed Jun 24 11:13:22 EDT 2009
Built host: master.org
C bindings: yes
C++ bindings: yes
Fortran77 bindings: yes (all)
Fortran90 bindings: yes
Fortran90 bindings size: small
C compiler: gcc
C compiler absolute: /usr/bin/gcc
C char size: 1
C bool size: 1
C short size: 2
C int size: 4
C long size: 8
C float size: 4
C double size: 8
C pointer size: 8
C char align: 1
C bool align: 1
C int align: 4
C float align: 4
C double align: 8
C++ compiler: g++
C++ compiler absolute: /usr/bin/g++
Fortran77 compiler: gfortran
Fortran77 compiler abs: /usr/bin/gfortran
Fortran90 compiler: gfortran
Fortran90 compiler abs: /usr/bin/gfortran
Fort integer size: 4
Fort logical size: 4
Fort logical value true: 1
Fort have integer1: yes
Fort have integer2: yes
Fort have integer4: yes
Fort have integer8: yes
Fort have integer16: no
Fort have real4: yes
Fort have real8: yes
Fort have real16: no
Fort have complex8: yes
Fort have complex16: yes
Fort have complex32: no
Fort integer1 size: 1
Fort integer2 size: 2
Fort integer4 size: 4
Fort integer8 size: 8
Fort integer16 size: -1
Fort real size: 4
Fort real4 size: 4
Fort real8 size: 8
Fort real16 size: -1
Fort dbl prec size: 4
Fort cplx size: 4
Fort dbl cplx size: 4
Fort cplx8 size: 8
Fort cplx16 size: 16
Fort cplx32 size: -1
Fort integer align: 4
Fort integer1 align: 1
Fort integer2 align: 2
Fort integer4 align: 4
Fort integer8 align: 8
Fort integer16 align: -1
Fort real align: 4
Fort real4 align: 4
Fort real8 align: 8
Fort real16 align: -1
Fort dbl prec align: 4
Fort cplx align: 4
Fort dbl cplx align: 4
Fort cplx8 align: 4
Fort cplx16 align: 8
Fort cplx32 align: -1
C profiling: yes
C++ profiling: yes
Fortran77 profiling: yes
Fortran90 profiling: yes
C++ exceptions: no
Thread support: posix (mpi: no, progress: no)
Sparse Groups: no
Build CFLAGS: -DNDEBUG -O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2
-fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m64 -mtune=generic
-finline-functions -fno-strict-aliasing
-pthread
-fvisibility=hidden
Build CXXFLAGS: -DNDEBUG -O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2
-fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m64 -mtune=generic
-finline-functions -pthread
Build FFLAGS: -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
-fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m64 -mtune=generic
Build FCFLAGS: -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
-fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m64 -mtune=generic
Build LDFLAGS: -export-dynamic
Build LIBS: -lnsl -lutil -lm
Wrapper extra CFLAGS: -pthread
Wrapper extra CXXFLAGS: -pthread
Wrapper extra FFLAGS: -pthread
Wrapper extra FCFLAGS: -pthread
Wrapper extra LDFLAGS:
Wrapper extra LIBS: -ldl -Wl,--export-dynamic -lnsl -lutil -lm
-ldl
Internal debug support: no
MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
libltdl support: yes
Heterogeneous support: no
mpirun default --prefix: yes
MPI I/O support: yes
MPI_WTIME support: gettimeofday
Symbol visibility support: yes
FT Checkpoint support: no (checkpoint thread: no)
[root@master ~]#
so you see, even the OFED 1.4.1 installation fails to put -libverbs etc
into openmpi.
Also, I think its
--enable-static
that is putting the -libverbs into your openmpi. I'll try it and see
what happens.
What are libnuma and tm? Do I need to worry about them?
Finally, I have forgotton what I do with all the RPMs OFED generates.
Do I install them all on my compute nodes or just a subset?
Thanks for the help.
Jim
On Wed, 2009-06-24 at 17:22 -0400, Gus Correa wrote:
> Hi Jim
>
>
> Jim Kress wrote:
> > Noam, Gus and List,
> >
> > Did you statically link your openmpi when you built it? If you did (the
> > default is NOT to do this) then that could explain the discrepancy.
> >
> > Jim
>
> No, I didn't link statically.
>
> Did you link statically?
>
> Actually, I tried to do it, and it didn't work.
> I wouldn't get OpenMPI with IB if I tried to
> link statically (i.e. by passing -static or equivalent to CFLAGS,
> FFLAGS, etc).
> When I removed the "-static" I got OpenMPI with IB.
> I always dump the configure output (and the make output, etc) to
> log files to check these things out after it is done.
> I really suggest you do this, it pays off, saves time, costs nothing.
> I don't remember exactly what symptoms I found on the log,
> whether the log definitely said that there was no IB support,
> or if it didn't have the right flags (-libverbs, etc) like yours.
> However, when I suppressed the "-static" from the compiler flags
> then I've got all the IB goodies! :)
>
> Here is how I run configure (CFLAGS etc only have optimization flags,
> no "-static"):
>
> ./configure \
> --prefix=/my/directory \
> --with-libnuma=/usr \
> --with-tm=/usr \
> --with-openib=/usr \
> --enable-static \
> 2>&1 configure.log
>
> Note, "--enable-static" means OpenMPI will build static libraries
> (besides the shared ones).
> OpenMPI is not being linked statically to system libraries,
> or to IB libraries, etc.
>
> Some switches may not be needed,
> in particularly the explicit use of /usr directory.
> However, at some point the OpenMPI configure
> would not work without being
> told this (at least for libnuma).
>
> BTW, I didn't claim your OpenMPI doesn't have IB support.
> Not a categorical syllogism like
> "you don't have the -libverbs flag, hence you don't have IB".
> It is hard to make definitive statements like this
> in a complex environment like this (OpenMPI build, parallel programs),
> and with limited information via email.
> After all, the list is peer reviewed! :)
> Hence, I only guessed, as I usually do in these exchanges.
> However, considering all the trouble you've been through, who knows,
> maybe it was a guess in the right direction.
>
> I wonder if there may still be a glitch in the OpenMPI configure
> script, on how it searches for and uses libraries like IB, NUMA, etc,
> which may be causing the problem.
> Jeff: Is this possible?
>
> In any case, we have different "Wrapper extra LIBS".
> I have -lrdmacm -libverbs, you and Noam don't have them.
> (Noam: I am not saying you don't have IB support! :))
> My configure explicitly asks for ib support, Noam's (and maybe yours)
> doesn't.
> Somehow, slight differences in how one invokes
> the configure script seems to produce different results.
>
> I hope this helps,
> Gus Correa
> ---------------------------------------------------------------------
> Gustavo Correa
> Lamont-Doherty Earth Observatory - Columbia University
> Palisades, NY, 10964-8000 - USA
> ---------------------------------------------------------------------
> >
> >> -----Original Message-----
> >> From: users-boun...@open-mpi.org
> >> [mailto:users-boun...@open-mpi.org] On Behalf Of Noam Bernstein
> >> Sent: Wednesday, June 24, 2009 9:38 AM
> >> To: Open MPI Users
> >> Subject: Re: [OMPI users] 50% performance reduction due to
> >> OpenMPI v 1.3.2forcing all MPI traffic over Ethernet instead
> >> of using Infiniband
> >>
> >>
> >> On Jun 23, 2009, at 6:19 PM, Gus Correa wrote:
> >>
> >>> Hi Jim, list
> >>>
> >>> On my OpenMPI 1.3.2 ompi_info -config gives:
> >>>
> >>> Wrapper extra LIBS: -lrdmacm -libverbs -ltorque -lnuma -ldl -Wl,--
> >>> export-dynamic -lnsl -lutil -lm -ldl
> >>>
> >>> Yours doesn't seem to have the IB libraries: -lrdmacm -libverbs
> >>>
> >>> So, I would guess your OpenMPI 1.3.2 build doesn't have IB support.
> >> The second of these statements doesn't follow from the first.
> >>
> >> My "ompi_info -config" returns
> >>
> >> ompi_info -config | grep LIBS
> >> Build LIBS: -lnsl -lutil -lm
> >> Wrapper extra LIBS: -ldl -Wl,--export-dynamic
> >> -lnsl -lutil -
> >> lm -ldl
> >>
> >> But it does have openib
> >>
> >> ompi_info | grep openib
> >> MCA btl: openib (MCA v2.0, API v2.0,
> >> Component v1.3.2)
> >>
> >> and osu_bibw returns
> >>
> >> # OSU MPI Bi-Directional Bandwidth Test v3.0
> >> # Size Bi-Bandwidth (MB/s)
> >> 4194304 1717.43
> >>
> >> which it's sure not getting over ethernet. I think Jeff
> >> Squyres' test (ompi_info | grep openib) must be more definitive.
> >>
> >>
> >> Noam
> >> _______________________________________________
> >> users mailing list
> >> us...@open-mpi.org
> >> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >
> > _______________________________________________
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
