-------- Forwarded Message --------
Subject: Re: [petsc-users] PETSc doesn't allow use of multithreaded MKL
with MUMPS + fblaslapack?
Date: Sun, 12 Aug 2018 14:20:42 -0500
From: Satish Balay <[email protected]>
Reply-To: petsc-users <[email protected]>
To: Appel, Thibaut <[email protected]>
CC: [email protected] <[email protected]>
Hm - its just a default - so you can always change the default value
to a more suitable one for your usage. [i.e use --with-blaslapack-lib
option instead of --with-blaslapack-dir option]
For regular petsc build - we think that sequential MKL is the best match.
For a build with C/Pardiso - I believe its best to use threaded MKL.
Wrt with-openmp=1 - if threaded MKL is preferable - I guess we could
change the default. But a default does not prevent one from using a
preferred blas [whatever that is]
Wrt fblaslapack - yes its not multi-threaded. But I believe openblas is
multi-threaded [so you could use --download-openblas as alternative]
The usual issue is - one cannot use threaded MKL [or any threaded
library] as a black box. They would have to always be aware of how
many mpi procs, and openmp threads are being used - and tweak these
parameters constantly. The default for OpenMPI is to use the whole
machine - i.e it expects 1 mpi task per node. If one users more mpi
tasks per node - and does not reduce threads per node - they get bad
performance. Hence we avoid using threaded MKL as a default..
Satish
On Sun, 12 Aug 2018, Appel, Thibaut wrote:
Good afternoon,
I have an application code written in pure MPI but wanted to exploit
multithreading in MUMPS (contained in calls to BLAS routines)
On a high-end parallel cluster I’m using, I’m linking with the Intel MKL
library but it seems that PETSc won’t configure the way I want:
./configure […] —with-openmp=1 --with-pic=1 --with-cc=mpiicc --with-cxx=mpiicpc
--with-fc=mpiifort --with-blaslapack-dir=${MKLROOT}
--with-scalapack-lib="-L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64
-lmkl_blacs_intelmpi_lp64" --with-scalapack-include=${MKLROOT}/include
--download-metis --download-parmetis --download-mumps
yields BLAS/LAPACK: -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread
while if I configure with cpardiso on top of the same flags
./configure […] —with-openmp=1 —with-pic=1 --with-cc=mpiicc --with-cxx=mpiicpc
--with-fc=mpiifort --with-blaslapack-dir=${MKLROOT}
--with-scalapack-lib="-L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64
-lmkl_blacs_intelmpi_lp64" --with-scalapack-include=${MKLROOT}/include
--with-mkl_cpardiso-dir=${MKLROOT} --download-metis --download-parmetis --download-mumps
the configure script says
===============================================
BLASLAPACK: Looking for Multithreaded MKL for C/Pardiso
===============================================
and yields BLAS/LAPACK: -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread
-lmkl_blacs_intelmpi_lp64 -liomp5 -ldl -lpthread
In other words, there is no current possibility of activating multithreaded
BLAS with MUMPS in spite of the option —with-openmp=1, as libmkl_sequential is
linked. Is it not possible to fix that and use libmkl_intel_thread by default?
On another smaller cluster, I do not have MKL and configure PETSc with BLAS
downloaded with —download-fblaslapack, which is not multithreaded.
Could you confirm I would need to link with a multithreaded BLAS library I
downloaded myself and use —with-openmp=1? Would it be `recognized` by the MUMPS
installed by PETSc?
Thanks for your support,
Thibaut
