Oh sorry, the error was at link time: 01:18 bint01 master ~/petsc_private/src/ksp/ksp/examples/tutorials$ make PETSC_DIR=/global/homes/m/madams/petsc_private PETSC_ARCH=arch-knc-dbg ex1 mpiicc -o ex1.o -c -mmic -mkl -fp-model precise -g -I/global/homes/m/madams/petsc_private/include -I/global/homes/m/madams/petsc_private/arch-knc-dbg/include -I/chos/global/u2/m/madams/petsc_private/arch-knc-dbg/include -I/global/babbage/nsg/opt/intel/impi/4.1.3.048/intel64/include `pwd`/ex1.c mpiicc -mmic -mkl -fp-model precise -g -o ex1 ex1.o -L/global/homes/m/madams/petsc_private/arch-knc-dbg/lib -lpetsc -Wl,-rpath,/chos/global/u2/m/madams/petsc_private/arch-knc-dbg/lib -L/chos/global/u2/m/madams/petsc_private/arch-knc-dbg/lib -lsuperlu_4.3 -lsuperlu_dist_3.3 -lparmetis -lmetis -lpthread -L/global/babbage/nsg/opt/intel/impi/4.1.3.048/mic/lib -L/global/babbage/nsg/opt/intel/composerxe/composer_xe_2015.0.090/compiler/lib/mic -L/global/babbage/nsg/opt/intel/composerxe/composer_xe_2015.0.090/mkl/lib/mic -L/global/babbage/nsg/opt/intel/composerxe/composer_xe_2015.0.090/tbb/lib/mic -L/chos/global/babbage/nsg/opt/intel/composerxe/composer_xe_2015.0.090/compiler/lib/mic -L/usr/linux-k1om-4.7/lib/gcc/x86_64-k1om-linux/4.7.0 -L/usr/linux-k1om-4.7/lib/gcc -L/usr/linux-k1om-4.7/linux-k1om/lib64 -L/usr/linux-k1om-4.7/linux-k1om/usr/lib64 -Wl,-rpath,/global/babbage/nsg/opt/intel/impi/4.1.3.048/mic/lib -Wl,-rpath,/opt/intel/mpi-rt/4.1 -lifport -lifcore -lm -Wl,-rpath,/chos/global/u2/m/madams/petsc_private/or -lmpigc4 -ldl -L/global/babbage/nsg/opt/intel/impi/4.1.3.048/mic/lib -lmpigf -lmpi_dbg -lmpigi -lrt -lpthread -L/global/babbage/nsg/opt/intel/impi/ 4.1.3.048/mic/lib -L/global/babbage/nsg/opt/intel/composerxe/composer_xe_2015.0.090/compiler/lib/mic -L/global/babbage/nsg/opt/intel/composerxe/composer_xe_2015.0.090/mkl/lib/mic -L/global/babbage/nsg/opt/intel/composerxe/composer_xe_2015.0.090/tbb/lib/mic -L/chos/global/babbage/nsg/opt/intel/composerxe/composer_xe_2015.0.090/compiler/lib/mic -L/usr/linux-k1om-4.7/lib/gcc/x86_64-k1om-linux/4.7.0 -L/usr/linux-k1om-4.7/lib/gcc -L/usr/linux-k1om-4.7/linux-k1om/lib64 -L/usr/linux-k1om-4.7/linux-k1om/usr/lib64 -L/usr/linux-k1om-4.7/linux-k1om/lib64 -L/usr/linux-k1om-4.7/linux-k1om/usr/lib64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -Wl,-rpath,/global/babbage/nsg/opt/intel/impi/4.1.3.048/mic/lib -Wl,-rpath,/opt/intel/mpi-rt/4.1 -L/global/babbage/nsg/opt/intel/composerxe/composer_xe_2015.0.090/mkl/lib/mic -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lirc_s -L/global/babbage/nsg/opt/intel/impi/4.1.3.048/mic/lib -L/global/babbage/nsg/opt/intel/composerxe/composer_xe_2015.0.090/compiler/lib/mic -L/global/babbage/nsg/opt/intel/composerxe/composer_xe_2015.0.090/mkl/lib/mic -L/global/babbage/nsg/opt/intel/composerxe/composer_xe_2015.0.090/tbb/lib/mic -L/chos/global/babbage/nsg/opt/intel/composerxe/composer_xe_2015.0.090/compiler/lib/mic -L/usr/linux-k1om-4.7/lib/gcc/x86_64-k1om-linux/4.7.0 -L/usr/linux-k1om-4.7/lib/gcc -L/usr/linux-k1om-4.7/linux-k1om/lib64 -L/usr/linux-k1om-4.7/linux-k1om/usr/lib64 -L/usr/linux-k1om-4.7/linux-k1om/lib64 -L/usr/linux-k1om-4.7/linux-k1om/usr/lib64 -L/global/babbage/nsg/opt/intel/composerxe/composer_xe_2015.0.090/mkl/lib/mic -ldl x86_64-k1om-linux-ld: warning: libintlc.so.5, needed by /global/babbage/nsg/opt/intel/composerxe/composer_xe_2015.0.090/compiler/lib/mic/libifport.so.5, not found (try using -rpath or -rpath-link) x86_64-k1om-linux-ld: i386:x86-64 architecture of input file `/global/homes/m/madams/petsc_private/arch-knc-dbg/lib/libsuperlu_4.3.a(superlu_timer.o)' is incompatible with k1om output x86_64-k1om-linux-ld: /global/homes/m/madams/petsc_private/arch-knc-dbg/lib/libsuperlu_4.3.a(superlu_timer.o)(.text+0xc): reloc against `sysconf@@GLIBC_2.14': error 4 x86_64-k1om-linux-ld: final link failed: Nonrepresentable section on output make: [ex1] Error 1 (ignored) /bin/rm -f ex1.o
On Wed, Nov 5, 2014 at 12:29 PM, Satish Balay <ba...@mcs.anl.gov> wrote: > configure.log looks fine. Perhaps the errors are in test.log? > > There is stuff like: > > >>>> > Possible ERROR while running linker: > stderr: > x86_64-k1om-linux-ld: warning: libintlc.so.5, needed by > /global/babbage/nsg/opt/intel/composerxe/composer_xe_2015.0.090/compiler/lib/mic/libifport.so.5, > not found (try using -rpath or -rpath-link) > <<<< > > But perhaps thats a compiler issue [or an issue with autodetect stuff.] > > Satish > > On Wed, 5 Nov 2014, Mark Adams wrote: > > > I am thinking I need to get a manual build of SuperLU but let me know if > > you have any suggestions. > > Thanks, > > Mark > > > > On Wed, Nov 5, 2014 at 11:59 AM, Satish Balay <ba...@mcs.anl.gov> wrote: > > > > > No log attached - so I can't comment.. > > > > > > satish > > > > > > On Wed, 5 Nov 2014, Mark Adams wrote: > > > > > > > Thanks, it is working now. Links fail with a SuperLU error. I'm > > > guessing > > > > I need to use a manually built SuperLU. > > > > Mark > > > > > > > > On Tue, Nov 4, 2014 at 9:13 PM, Satish Balay <ba...@mcs.anl.gov> > wrote: > > > > > > > > > >>>>>> > > > > > checking whether the C compiler works...configure: error: in > > > > > > > > > `/chos/global/u2/m/madams/petsc_private/arch-knc-dbg64/externalpackages/hypre-2.9.1a/src': > > > > > configure: error: cannot run C compiled programs. > > > > > If you meant to cross compile, use `--host'. > > > > > <<<<< > > > > > > > > > > we really don't have support for batch build of > > > > > externalpacakges. [they usually work - which is great - but it > won't > > > > > work always]. > > > > > > > > > > So it has to be installed manually. > > > > > > > > > > BTW: --with-cc="mpiicc " --with-cxx="mpiicpc " has extra > > > > > 'spaces'. Perhaps its not causing problems yet. [cmake usually > barfs > > > > > with this type of usage] > > > > > > > > > > Satish > > > > > > > > > > On Tue, 4 Nov 2014, Mark Adams wrote: > > > > > > > > > > > Thanks, getting better ... I now get this error with hypre. > hypre > > > is > > > > > not > > > > > > critical here but if someone has ideas ... > > > > > > > > > > > > On Tue, Nov 4, 2014 at 7:43 PM, Satish Balay <ba...@mcs.anl.gov> > > > wrote: > > > > > > > > > > > > > --CFLAGS=""-mmic -mkl -fp-model precise"" > > > > > > > > > > > > > > remove the extra set of quotes from your ../arch-knc-dbg64.py. > > > > > > > For eg: petsc/config/examples/arch-cray-xt6-pkgs-opt.py has: > > > > > > > > > > > > > > '--COPTFLAGS=-fast -mp', > > > > > > > > > > > > > > The equivalent shell would be: > > > > > > > > > > > > > > --COPTFLAGS="-fast -mp" > > > > > > > > > > > > > > Satish > > > > > > > > > > > > > > On Tue, 4 Nov 2014, Mark Adams wrote: > > > > > > > > > > > > > > > Thanks this helped. I am now getting METIS errors. > > > > > > > > > > > > > > > > On Tue, Nov 4, 2014 at 6:30 PM, Satish Balay < > ba...@mcs.anl.gov> > > > > > wrote: > > > > > > > > > > > > > > > > > Ah - didn't check the configure comand closely enough.. > > > > > > > > > > > > > > > > > > > --with-mpicc=/global/babbage/nsg/opt/intel/impi/ > > > > > > > > > 5.0.1.035/intel64/bin/mpicc > > > > > > > > > > --with-mpicxx=/global/babbage/nsg/opt/intel/impi/ > > > > > > > > > 5.0.1.035/intel64/bin/mpicxx > > > > > > > > > > --with-mpif90=/global/babbage/nsg/opt/intel/impi/ > > > > > > > > > 5.0.1.035/intel64/bin/mpif90 > > > > > > > > > > > > > > > > > > petsc configure doesn't provide any such options.. > > > > > > > > > > > > > > > > > > > --with-mpi-lib=/global/babbage/nsg/opt/intel/impi/ > > > > > > > 5.0.1.035/intel64/lib > > > > > > > > > > > > > > > > > > This would be the wrong way to provide MPI lib. > > > > > > > > > > > > > > > > > > > > > > > > > > > The correct way to use Intel MPI is: > > > > > > > > > > > > > > > > > > --with-cc=mpiipc --with-cxx=mpiicpc --with-fc=mpiifort > [i.e no > > > > > > > > > --with-mpi-include or --with-mpi-lib options] > > > > > > > > > > > > > > > > > > But as Richard mentioned - the wrappers in intel impi-5 are > > > > > > > > > broken. Alternative is to use impi-4 as suggested. > > > > > > > > > > > > > > > > > > Or something like the following (for compilers/mpi): > > > > > > > > > > > > > > > > > > ./configure --with-cc=icc --with-fc=ifort --with-cxx=0 > > > > > > > > > --with-mpi-include=/global/babbage/nsg/opt/intel/impi/ > > > > > > > > > 5.0.1.035/intel64/include > > > > > > > > > --with-mpi-lib="-L/global/babbage/nsg/opt/intel/impi/ > > > > > > > > > 5.0.1.035/intel64/lib/release > > > > > -L/global/babbage/nsg/opt/intel/impi/ > > > > > > > > > 5.0.1.035/intel64/lib -lmpifort -lmpi -lmpigi -ldl -lrt > > > -lpthread" > > > > > > > > > > > > > > > > > > Satish > > > > > > > > > > > > > > > > > > On Tue, 4 Nov 2014, Richard Mills wrote: > > > > > > > > > > > > > > > > > > > Hi Mark, > > > > > > > > > > > > > > > > > > > > I noticed that you are using the wrong compiler wrappers. > > > > > Instead of > > > > > > > > > > 'mpicc', 'mpif90', and 'mpicxx' (which will invoke GNU > > > > > compilers), > > > > > > > you > > > > > > > > > > probably want 'mpiicc', 'mpiifort', and 'mpiicpc' > (invoke the > > > > > Intel > > > > > > > > > > compilers). > > > > > > > > > > > > > > > > > > > > You will probably also want to unload the IMPI 5.x and > load > > > the > > > > > IMPI > > > > > > > > > 4.1.3 > > > > > > > > > > module right now because of a stupid bug in the compiler > > > wrappers > > > > > > > (for > > > > > > > > > > which I've filed a bug report--will be fixed in next IMPI > > > > > release, > > > > > > > > > sometime > > > > > > > > > > before SC14). > > > > > > > > > > > > > > > > > > > > --Richard > > > > > > > > > > > > > > > > > > > > On Tue, Nov 4, 2014 at 12:59 PM, Mark Adams < > mfad...@lbl.gov > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Thanks Victor, > > > > > > > > > > > > > > > > > > > > > > I am getting cc error. Any ideas? > > > > > > > > > > > Mark > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 4, 2014 at 12:38 PM, Victor Eijkhout < > > > > > > > > > eijkh...@tacc.utexas.edu > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > >> > On Nov 4, 2014, at 11:32 AM, Mark Adams < > > > mfad...@lbl.gov> > > > > > > > wrote: > > > > > > > > > > >> > > > > > > > > > > > >> > Has anyone built PETSc on KNC (Babbage at NERSC to > be > > > > > specific)? > > > > > > > > > > >> > > > > > > > > > > >> At TACC under intel 14: > > > > > > > > > > >> > > > > > > > > > > >> CEE_OPTIONS="-mmic -mkl -fp-model precise" > > > > > > > > > > >> ./configure --PETSC_ARCH=mic --with-fc=0 > --with-debug=0 \ > > > > > > > > > > >> --with-batch=1 --CPPFLAGS=-mmic \ > > > > > > > > > > >> --CFLAGS="${CEE_OPTIONS}" > --CXXFLAGS="${CEE_OPTIONS}" > > > > > > > > > > >> --FFLAGS="${CEE_OPTIONS}" \ > > > > > > > > > > >> --with-mpi=1 --known-mpi-shared-libraries=1 \ > > > > > > > > > > >> --with-mpi-include=${MPICH_HOME}/mic/include \ > > > > > > > > > > >> --with-mpi-lib=${MPICH_HOME}/mic/lib \ > > > > > > > > > > >> --with-mpicc=/opt/apps/intel13/impi/ > > > > > > > > > 4.1.0.030/intel64/bin/mpicc \ > > > > > > > > > > >> --with-mpicxx=/opt/apps/intel13/impi/ > > > > > > > > > 4.1.0.030/intel64/bin/mpicxx > > > > > > > > > > >> \ > > > > > > > > > > >> --with-mpif90=/opt/apps/intel13/impi/ > > > > > > > > > 4.1.0.030/intel64/bin/mpif90 > > > > > > > > > > >> > > > > > > > > > > >> That first line is still giving me problems. Just nix > the > > > > > > > fp-model. > > > > > > > > > > >> > > > > > > > > > > >> Victor. > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >