Sorry I haven't jumped in this thread earlier -- I've been a bit behind.
The multi-lib support worked at one time, and I can't think of why it
would have changed. The one condition is that libdir, includedir,
etc. *MUST* be specified relative to $prefix for it to work. It looks
like you were defining them as absolute paths, so you'd have to set
libdir directly, which will never work in multi-lib because mpirun and
the app likely have different word sizes and therefore different
libdirs. More information is on the multilib page in the wiki:
https://svn.open-mpi.org/trac/ompi/wiki/MultiLib
There is actually one condition we do not handle properly, the prefix
flag to mpirun. The LD_LIBRARY_PATH will only be set for the word
size of mpirun, and not the executable. Really, both would have to be
added (so that both orted, which is likely always 32 bit in a multilib
situation and the app both find their libraries).
Brian
On Jan 5, 2009, at 6:02 PM, Jeff Squyres wrote:
I honestly haven't thought through the ramifications of doing a
multi-lib build with OPAL_PREFIX et al. :-\
If you setenv OPAL_LIBDIR, it'll use whatever you set it to, so it
doesn't matter what you configured --libdir with. Additionally mca/
installdirs/config/install_dirs.h has this by default:
#define OPAL_LIBDIR "${exec_prefix}/lib"
Hence, if you use a default --libdir and setenv OPAL_PREFIX, then
the libdir should pick up the right thing (because it's based on the
prefix). But if you use --libdir that is *not* based on $
{exec_prefix}, then you might run into problems.
Perhaps you can '--libdir="${exec_prefix}/lib64"' so that you can
have your custom libdir, but still have it dependent upon the prefix
that gets expanded at run time...?
(again, I'm not thinking all of this through -- just offering a few
suggestions off the top of my head that you'll need to test / trace
the code to be sure...)
On Jan 5, 2009, at 1:35 PM, Ethan Mallove wrote:
On Thu, Dec/25/2008 08:12:49AM, Jeff Squyres wrote:
It's quite possible that we don't handle this situation properly.
Won't
you need to libdir's (one for the 32 bit OMPI executables, and one
for the
64 bit MPI apps)?
I don't need an OPAL environment variable for the executables, just a
single OPAL_LIBDIR var for the libraries. (One set of 32-bit
executables runs with both 32-bit and 64-bit libraries.) I'm guessing
OPAL_LIBDIR will not work for you if you configure with a non-
standard
--libdir option.
-Ethan
On Dec 23, 2008, at 3:58 PM, Ethan Mallove wrote:
I think the problem is that I am doing a multi-lib build. I have
32-bit libraries in lib/, and 64-bit libraries in lib/64. I
assume I
do not see the issue for 32-bit tests, because all the dependencies
are where Open MPI expects them to be. For the 64-bit case, I tried
setting OPAL_LIBDIR to /opt/openmpi-relocated/lib/lib64, but no
luck.
Given the below configure arguments, what do my OPAL_* env vars
need
to be? (Also, could using --enable-orterun-prefix-by-default
interfere
with OPAL_PREFIX?)
$ ./configure CC=cc CXX=CC F77=f77 FC=f90 --with-openib
--without-udapl --disable-openib-ibcm --enable-heterogeneous
--enable-cxx-exceptions --enable-shared --enable-orterun-prefix-
by-default
--with-sge --enable-mpi-f90 --with-mpi-f90-size=small
--disable-mpi-threads --disable-progress-threads --disable-debug
CFLAGS="-m32 -xO5" CXXFLAGS="-m32 -xO5" FFLAGS="-m32 -xO5"
FCFLAGS="-m32
-xO5"
--prefix=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
tarball-testing/installs/DGQx/install
--mandir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
tarball-testing/installs/DGQx/install/man
--libdir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
tarball-testing/installs/DGQx/install/lib
--includedir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/
ompi-tarball-testing/installs/DGQx/install/include
--without-mx --with-tm=/ws/ompi-tools/orte/torque/current/shared-
install32
--with-contrib-vt-flags="--prefix=/workspace/em162155/hpc/mtt-
scratch/burl-ct-v!
20z-12/ompi-tarball-testing/installs/DGQx/install
--mandir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
tarball-testing/installs/DGQx/install/man
--libdir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
tarball-testing/installs/DGQx/install/lib
--includedir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/
ompi-tarball-testing/installs/DGQx/install/include
LDFLAGS=-R/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/
ompi-tarball-testing/installs/DGQx/install/lib"
$ ./confgiure CC=cc CXX=CC F77=f77 FC=f90 --with-openib
--without-udapl --disable-openib-ibcm --enable-heterogeneous
--enable-cxx-exceptions --enable-shared --enable-orterun-prefix-
by-default
--with-sge --enable-mpi-f90 --with-mpi-f90-size=small
--disable-mpi-threads --disable-progress-threads --disable-debug
CFLAGS="-m64 -xO5" CXXFLAGS="-m64 -xO5" FFLAGS="-m64 -xO5"
FCFLAGS="-m64
-xO5"
--prefix=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
tarball-testing/installs/DGQx/install
--mandir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
tarball-testing/installs/DGQx/install/man
--libdir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
tarball-testing/installs/DGQx/install/lib/lib64
--includedir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/
ompi-tarball-testing/installs/DGQx/install/include/64
--without-mx --with-tm=/ws/ompi-tools/orte/torque/current/shared-
install64
--with-contrib-vt-flags="--prefix=/workspace/em162155/hpc/mtt-
scratch/!
burl-ct-v20z-12/ompi-tarball-testing/installs/DGQx/install
--mandir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
tarball-testing/installs/DGQx/install/man
--libdir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
tarball-testing/installs/DGQx/install/lib/lib64
--includedir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/
ompi-tarball-testing/installs/DGQx/install/include/64
LDFLAGS=-R/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/
ompi-tarball-testing/installs/DGQx/install/lib"
--disable-binaries
-Ethan
On Dec 22, 2008, at 12:42 PM, Ethan Mallove wrote:
Can anyone get OPAL_PREFIX to work on Linux? A simple test is
to see
if the following works for any mpicc/mpirun:
$ mv <openmpi-installation> /tmp/foo
$ set OPAL_PREFIX /tmp/foo
$ mpicc ...
$ mpirun ...
If you are able to get the above to run successfully, I'm
interested
in your config.log file.
Thanks,
Ethan
On Thu, Dec/18/2008 11:03:25AM, Ethan Mallove wrote:
Hello,
The below FAQ lists instructions on how to use a relocated
Open MPI
installation:
http://www.open-mpi.org/faq/?category=building#installdirs
On Solaris, OPAL_PREFIX and friends (documented in the FAQ)
work for
me with both MPI (hello_c) and non-MPI (hostname) programs. On
Linux,
I can only get the non-MPI case to work. Here are the
environment
variables I am setting:
$ cat setenv_opal_prefix.csh
set opal_prefix = "/opt/openmpi-relocated"
setenv OPAL_PREFIX $opal_prefix
setenv OPAL_BINDIR $opal_prefix/bin
setenv OPAL_SBINDIR $opal_prefix/sbin
setenv OPAL_DATAROOTDIR $opal_prefix/share
setenv OPAL_SYSCONFDIR $opal_prefix/etc
setenv OPAL_SHAREDSTATEDIR $opal_prefix/com
setenv OPAL_LOCALSTATEDIR $opal_prefix/var
setenv OPAL_LIBDIR $opal_prefix/lib
setenv OPAL_INCLUDEDIR $opal_prefix/include
setenv OPAL_INFODIR $opal_prefix/info
setenv OPAL_MANDIR $opal_prefix/man
setenv PATH $opal_prefix/bin:$PATH
setenv LD_LIBRARY_PATH $opal_prefix/lib:$opal_prefix/lib/64
Here is the error I get:
$ mpirun -np 2 hello_c
--------------------------------------------------------------------------
It looks like opal_init failed for some reason; your parallel
process
is
likely to abort. There are many reasons that a parallel
process can
fail during opal_init; some of which are due to configuration or
environment problems. This failure appears to be an internal
failure;
here's some additional information (which may only be relevant
to an
Open MPI developer):
opal_carto_base_select failed
--> Returned value -13 instead of OPAL_SUCCESS
--------------------------------------------------------------------------
[burl-ct-v20z-0:27737] [[INVALID],INVALID] ORTE_ERROR_LOG: Not
found in
file runtime/orte_init.c at line 77
Any ideas on what's going on?
Thanks,
Ethan
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
--
Jeff Squyres
Cisco Systems
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
--
Jeff Squyres
Cisco Systems
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
--
Jeff Squyres
Cisco Systems
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
