https://www.alcf.anl.gov/user-guides/compiling-and-linking-faq#how-do-i-compile-and-link-a-shared-library?
Looks like I need to specify '-Wl,-dy' for the linker to find the shared
library.
[balay@vestalac1 tutorials]$ make ex2 CFLAGS="-Wl,-dy
-Wl,-rpath,/home/balay/petsc/arch-xl-opt-nopkg/lib"
mpixlc_r -o ex2.o -c -qpic -O3 -qhot=level=0 -qsimd=auto -qmaxmem=-1 -qstrict
-qstrict_induction -Wl,-dy -Wl,-rpath,/home/balay/petsc/arch-xl-opt-nop
kg/lib -I/home/balay/petsc/include
-I/home/balay/petsc/arch-xl-opt-nopkg/include
-I/bgsys/drivers/V1R2M1/ppc64/comm/include -I/bgsys/drivers/V1R2M1/pp
c64/comm/lib/gnu -I/bgsys/drivers/V1R2M1/ppc64
-I/bgsys/drivers/V1R2M1/ppc64/comm/sys/include
-I/bgsys/drivers/V1R2M1/ppc64/spi/include
-I/bgsys/drivers/V1R2M1/ppc64/spi/include/kernel/cnk `pwd`/ex2.c
mpixlc_r -qpic -O3 -qhot=level=0 -qsimd=auto -qmaxmem=-1 -qstrict
-qstrict_induction -Wl,-dy -Wl,-rpath,/home/balay/petsc/arch-xl-opt-nopkg/lib
-o ex2 ex2.o -L/home/balay/petsc/arch-xl-opt-nopkg/lib -lpetsc
-L/soft/libraries/alcf/current/xl/LAPACK/lib -llapack
-L/soft/libraries/alcf/current/xl/BLAS/lib -lblas -lpthread
-L/bgsys/drivers/V1R2M1/ppc64/comm/lib -L/bgsys/drivers/V1R2M1/ppc64/comm/lib64
-L/bgsys/drivers/V1R2M1/ppc64/spi/lib
-L/bgsys/drivers/V1R2M1/ppc64/comm/sys/lib
-L/soft/compilers/ibmcmp-may2014/xlsmp/bg/3.1/bglib64
-L/soft/compilers/ibmcmp-may2014/xlmass/bg/7.3/bglib64
-L/soft/compilers/ibmcmp-may2014/vac/bg/12.1/bglib64
-L/soft/compilers/ibmcmp-may2014/vacpp/bg/12.1/bglib64
-L/bgsys/drivers/toolchain/V1R2M1_base-efix033/gnu-linux/lib/gcc/powerpc64-bgq-linux/4.4.6
-L/bgsys/drivers/toolchain/V1R2M1_base-efix033/gnu-linux/lib/gcc
-L/bgsys/drivers/toolchain/V1R2M1_base-efix033/gnu-linux/powerpc64-bgq-linux/lib
-L/soft/compilers/ibmcmp-may2014/xlf/bg/14.1/bglib64
-Wl,-rpath,/soft/compilers/ibmcmp-may2014/lib64/bg/bglib64 -lmpichf90-gcc
-lxlf90_r -lxlomp_ser -lxlfmath -lm -lmpichcxx-gcc -libmc++ -ldl -lmpich-gcc
-lopa-gcc -lmpl-gcc -lpami-gcc -lSPI -lSPI_cnk -lrt -lpthread -lstdc++ -lxlopt
-lxl -lnss_files -lnss_dns -lresolv -lgcc_eh -ldl
/bin/rm -f ex2.o
[balay@vestalac1 tutorials]$ ldd ex2
linux-vdso64.so.1 => (0x00000fff8d020000)
libpetsc.so => /home/balay/petsc/arch-xl-opt-nopkg/lib/libpetsc.so
(0x00000fff8bfb0000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00000fff8bf50000)
libmpichf90-gcc.so.8 =>
/bgsys/drivers/ppcfloor/comm/lib/libmpichf90-gcc.so.8 (0x00000fff8be40000)
libxlf90_r.so.1 =>
/soft/compilers/ibmcmp-may2014/lib64/bg/bglib64/libxlf90_r.so.1
(0x00000fff8b7a0000)
libxlomp_ser.so.1 =>
/soft/compilers/ibmcmp-may2014/lib64/bg/bglib64/libxlomp_ser.so.1
(0x00000fff8b690000)
libxlfmath.so.1 =>
/soft/compilers/ibmcmp-may2014/lib64/bg/bglib64/libxlfmath.so.1
(0x00000fff8b580000)
libm.so.6 => /lib64/libm.so.6 (0x00000fff8b480000)
libmpichcxx-gcc.so.8 =>
/bgsys/drivers/ppcfloor/comm/lib/libmpichcxx-gcc.so.8 (0x00000fff8b370000)
libibmc++.so.1 =>
/soft/compilers/ibmcmp-may2014/lib64/bg/bglib64/libibmc++.so.1
(0x00000fff8b260000)
libdl.so.2 => /lib64/libdl.so.2 (0x00000fff8b230000)
libmpich-gcc.so.8 => /bgsys/drivers/ppcfloor/comm/lib/libmpich-gcc.so.8
(0x00000fff8ae90000)
libopa-gcc.so.0 => /bgsys/drivers/ppcfloor/comm/lib/libopa-gcc.so.0
(0x00000fff8ad80000)
libmpl-gcc.so.1 => /bgsys/drivers/ppcfloor/comm/lib/libmpl-gcc.so.1
(0x00000fff8ac70000)
libpami-gcc.so => /bgsys/drivers/ppcfloor/comm/lib/libpami-gcc.so
(0x00000fff8a700000)
librt.so.1 => /lib64/librt.so.1 (0x00000fff8a6d0000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00000080ea910000)
libnss_files.so.2 => /lib64/libnss_files.so.2 (0x00000fff8a6a0000)
libnss_dns.so.2 => /lib64/libnss_dns.so.2 (0x00000fff8a670000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00000fff8a630000)
libc.so.6 => /lib64/libc.so.6 (0x00000fff8a440000)
/lib64/bgq/ld64.so.1 => /lib64/ld64.so.1 (0x00000000264e0000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000080ea720000)
libgfortran.so.3 =>
/bgsys/drivers/toolchain/V1R2M2_base/gnu-linux/powerpc64-bgq-linux/lib/libgfortran.so.3
(0x00000fff8a220000)
And I could run this binary successfully..
Satish
On Wed, 30 Jul 2014, Satish Balay wrote:
> On Wed, 30 Jul 2014, Alp Dener wrote:
>
> > Hi,
> >
> > I’m trying to build PETSc 3.4.5 on an IBM BlueGene/Q at the moment using
> > the XL toolchain. I need to build the PETSc shared library and to
> > accomplish that, I use the following configure options:
> >
> > ./configure \
> > --with-shared-libraries=1 \
> > --with-debugging=1 \
> > --with-fortran-interfaces=1 \
> > --with-mpi=/bgsys/drivers/ppcfloor/comm \
> > --with-blas-lib="-L/gpfs/u/software/bgq/lapack/xl/3.5.0/lib -lblas" \
> > --with-lapack-lib="-L/gpfs/u/software/bgq/lapack/xl/3.5.0/lib -llapack"
> > \
> > --with-metis-include=/gpfs/u/software/bgq/parmetis/xl/4.0.2/include \
> > --with-metis-lib="-L/gpfs/u/software/bgq/parmetis/xl/4.0.2/lib -lmetis"
> > \
> > --with-parmetis-include=/gpfs/u/software/bgq/parmetis/xl/4.0.2/include \
> > --with-parmetis-lib="-L/gpfs/u/software/bgq/parmetis/xl/4.0.2/lib
> > -lparmetis" \
> >
> > --download-superlu=/gpfs/u/barn/ODLC/shared/src/PETSc/superlu_4.3.tar.gz \
> >
> > --download-superlu_dist=/gpfs/u/barn/ODLC/shared/src/PETSc/superlu_dist_3.3.tar.gz
> > \
> > --prefix=/gpfs/u/barn/ODLC/shared/xl/petsc/3.4.5 \
> > --PETSC_ARCH=xl-dbg \
> > FFLAGS="-qextname”
> >
> > Despite specifying the “—with-shared-libraries” option, the configuration
> > report shows “shared libraries : disabled”. The configure log doesn’t
> > appear to contain any warnings or errors about this (or maybe I couldn’t
> > find it).
> >
> > Normally, compiling shared objects with XL compilers require a -qmkshrobj
> > flag. I’ve attempted passing this to PETSc using LDFLAGS but this causes
> > the configuration to fail with an error, stating that it cannot run
> > executables created with the C compiler.
> >
> > This problem seems to be persistent across a number of PETSc versions. I
> > need to use 3.4.5 for compatibility reasons, but I’ve attempted the same
> > build with 3.5, 3.3 and 3.0 with the exact same results.
> >
> > Would really appreciate it if anyone can offer guidance on how to force
> > PETSc to build its shared library. The configure log from the above options
> > is provided in the attachment.
>
> Shared libraries with IBM compilers is something I've not tired. If its just
> a matter
> of using '-qmkshrobj' - you could try to build it manually - and see if it
> works.
>
> [when I try it I get errors.]
>
> Satish
>
> ----------
>
> [balay@vestalac1 ~]$ cd petsc/arch-xl-opt-nopkg/lib/
> [balay@vestalac1 lib]$ ar x libpetsc.a
> [balay@vestalac1 lib]$ mpixlc -qmkshrobj -o libpetsc.so *.o
> [balay@vestalac1 lib]$ file libpetsc.so
> libpetsc.so: ELF 64-bit MSB shared object, 64-bit PowerPC or cisco 7500,
> version 1 (SYSV), dynamically linked, not stripped
> [balay@vestalac1 lib]$ mv libpetsc.a libpetsc-bkp.a
> [balay@vestalac1 lib]$ cd $PETSC_DIR
> [balay@vestalac1 petsc]$ cd src/ksp/ksp/examples/tutorials/
> [balay@vestalac1 tutorials]$ make ex2
> mpixlc_r -o ex2.o -c -qpic -O3 -qhot=level=0 -qsimd=auto -qmaxmem=-1 -qstrict
> -qstrict_induction -I/home/balay/petsc/include
> -I/home/balay/petsc/arch-xl-opt-nopkg/include
> -I/bgsys/drivers/V1R2M1/ppc64/comm/include
> -I/bgsys/drivers/V1R2M1/ppc64/comm/lib/gnu -I/bgsys/drivers/V1R2M1/ppc64
> -I/bgsys/drivers/V1R2M1/ppc64/comm/sys/include
> -I/bgsys/drivers/V1R2M1/ppc64/spi/include
> -I/bgsys/drivers/V1R2M1/ppc64/spi/include/kernel/cnk `pwd`/ex2.c
> mpixlc_r -qpic -O3 -qhot=level=0 -qsimd=auto -qmaxmem=-1 -qstrict
> -qstrict_induction -o ex2 ex2.o -L/home/balay/petsc/arch-xl-opt-nopkg/lib
> -lpetsc -L/soft/libraries/alcf/current/xl/LAPACK/lib -llapack
> -L/soft/libraries/alcf/current/xl/BLAS/lib -lblas -lpthread
> -L/bgsys/drivers/V1R2M1/ppc64/comm/lib
> -L/bgsys/drivers/V1R2M1/ppc64/comm/lib64
> -L/bgsys/drivers/V1R2M1/ppc64/spi/lib
> -L/bgsys/drivers/V1R2M1/ppc64/comm/sys/lib
> -L/soft/compilers/ibmcmp-may2014/xlsmp/bg/3.1/bglib64
> -L/soft/compilers/ibmcmp-may2014/xlmass/bg/7.3/bglib64
> -L/soft/compilers/ibmcmp-may2014/vac/bg/12.1/bglib64
> -L/soft/compilers/ibmcmp-may2014/vacpp/bg/12.1/bglib64
> -L/bgsys/drivers/toolchain/V1R2M1_base-efix033/gnu-linux/lib/gcc/powerpc64-bgq-linux/4.4.6
> -L/bgsys/drivers/toolchain/V1R2M1_base-efix033/gnu-linux/lib/gcc
> -L/bgsys/drivers/toolchain/V1R2M1_base-efix033/gnu-linux/powerpc64-bgq-linux/lib
> -L/soft/compilers/ibmcmp-may2014/xlf/bg/14.1/bglib64
> -Wl,-rpath,/soft/compilers/ibmcmp-may2014/lib64/bg/bglib64 -lmpichf90-gcc
> -lxlf90_r -lxlomp_ser -lxlfmath -lm -lmpichcxx-gcc -libmc++ -ldl -lmpich-gcc
> -lopa-gcc -lmpl-gcc -lpami-gcc -lSPI -lSPI_cnk -lrt -lpthread -lstdc++
> -lxlopt -lxl -lnss_files -lnss_dns -lresolv -lgcc_eh -ldl
> /bgsys/drivers/ppcfloor/gnu-linux/powerpc64-bgq-linux/bin/ld: cannot find
> -lpetsc
> make: [ex2] Error 1 (ignored)
> /bin/rm -f ex2.o
> [balay@vestalac1 tutorials]$
