Hi all,

I have been working in the last months with PETSC in a FEM program written
on FORTRAN, so far sequential. Now, I want to parallelize it with OpenMP
and I have found some problems. Finally, I have built a mockup program
trying to localize the error.

1. I have compiled PETSC with these options:
./configure --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort
--with-blas-lapack-dir=/opt/intel/mkl/lib/intel64/ --with-debugging=1
--with-scalar-type=complex --with-threadcomm --with-pthreadclasses
--with-openmp
--with-openmp-include=/opt/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64_lin
--with-openmp-lib=/opt/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64_lin/libiomp5.a
PETSC_ARCH=linux-intel-dbg PETSC-AVOID-MPIF-H=1

(I have tried also removing   --with-threadcomm --with-pthreadclasses and
with libiomp5.so).

2. The program to be executed is composed of two files, one is
hellocount.F90:
MODULE hello_count
  use omp_lib
  IMPLICIT none

  CONTAINS
  subroutine hello_print ()
     integer :: nthreads,mythread

   !pragma hello-who-omp-f
   !$omp parallel
     nthreads = omp_get_num_threads()
     mythread = omp_get_thread_num()
     write(*,'("Hello from",i3," out of",i3)') mythread,nthreads
   !$omp end parallel
   !pragma end
   end subroutine hello_print
END MODULE hello_count

and the other one is hellocount_main.F90:
Program Hello

   USE hello_count

   call hello_print

   STOP

end Program Hello

3. To compile these two functions I use:
rm -rf _obj
mkdir _obj

ifort -E -I/home/aamor/petsc/include
-I/home/aamor/petsc/linux-intel-dbg/include -c hellocount.F90
>_obj/hellocount.f90
ifort -E -I/home/aamor/petsc/include
-I/home/aamor/petsc/linux-intel-dbg/include -c hellocount_main.F90
>_obj/hellocount_main.f90

mpiifort -CB -g -warn all -O0 -shared-intel -check:none -qopenmp -module
_obj -I./_obj -I/home/aamor/MUMPS_5.1.2/include
 -I/opt/intel/compilers_and_libraries_2016.1.150/linux/mkl/include
-I/opt/intel/compilers_and_libraries_2016.1.150/linux/mkl/include/intel64/lp64/
-I/home/aamor/petsc/include -I/home/aamor/petsc/linux-intel-dbg/include -o
_obj/hellocount.o -c _obj/hellocount.f90
mpiifort -CB -g -warn all -O0 -shared-intel -check:none -qopenmp -module
_obj -I./_obj -I/home/aamor/MUMPS_5.1.2/include
 -I/opt/intel/compilers_and_libraries_2016.1.150/linux/mkl/include
-I/opt/intel/compilers_and_libraries_2016.1.150/linux/mkl/include/intel64/lp64/
-I/home/aamor/petsc/include -I/home/aamor/petsc/linux-intel-dbg/include -o
_obj/hellocount_main.o -c _obj/hellocount_main.f90

mpiifort -CB -g -warn all -O0 -shared-intel -check:none -qopenmp -module
_obj -I./_obj -o exec/HELLO _obj/hellocount.o _obj/hellocount_main.o
/home/aamor/lib_tmp/libarpack_LinuxIntel15.a
/home/aamor/MUMPS_5.1.2/lib/libzmumps.a
/home/aamor/MUMPS_5.1.2/lib/libmumps_common.a
/home/aamor/MUMPS_5.1.2/lib/libpord.a
/home/aamor/parmetis-4.0.3/lib/libparmetis.a
/home/aamor/parmetis-4.0.3/lib/libmetis.a
-L/opt/intel/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64
-lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -lpetsc -lmkl_intel_lp64
-lmkl_intel_thread -lmkl_core -lmkl_lapack95_lp64 -liomp5 -lpthread -lm
-L/home/aamor/lib_tmp -lgidpost -lz /home/aamor/lua-5.3.3/src/liblua.a
/home/aamor/ESEAS-master/libeseas.a
-Wl,-rpath,/home/aamor/petsc/linux-intel-dbg/lib
-L/home/aamor/petsc/linux-intel-dbg/lib
-Wl,-rpath,/opt/intel/mkl/lib/intel64 -L/opt/intel/mkl/lib/intel64
-Wl,-rpath,/opt/intel/impi/5.1.2.150/intel64/lib/debug_mt -L/opt/intel/impi/
5.1.2.150/intel64/lib/debug_mt -Wl,-rpath,/opt/intel/impi/
5.1.2.150/intel64/lib -L/opt/intel/impi/5.1.2.150/intel64/lib
-Wl,-rpath,/opt/intel/compilers_and_libraries_2016/linux/mkl/lib/intel64
-L/opt/intel/compilers_and_libraries_2016/linux/mkl/lib/intel64
-Wl,-rpath,/opt/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64_lin
-L/opt/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64_lin
-Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.7
-L/usr/lib/gcc/x86_64-redhat-linux/4.4.7
-Wl,-rpath,/opt/intel/mpi-rt/5.1/intel64/lib/debug_mt
-Wl,-rpath,/opt/intel/mpi-rt/5.1/intel64/lib -lmkl_intel_lp64
-lmkl_sequential -lmkl_core -lpthread -lX11 -lssl -lcrypto -lifport
-lifcore_pic -lmpicxx -ldl -Wl,-rpath,/opt/intel/impi/
5.1.2.150/intel64/lib/debug_mt -L/opt/intel/impi/
5.1.2.150/intel64/lib/debug_mt -Wl,-rpath,/opt/intel/impi/
5.1.2.150/intel64/lib -L/opt/intel/impi/5.1.2.150/intel64/lib -lmpifort
-lmpi -lmpigi -lrt -lpthread -Wl,-rpath,/opt/intel/impi/
5.1.2.150/intel64/lib/debug_mt -L/opt/intel/impi/
5.1.2.150/intel64/lib/debug_mt -Wl,-rpath,/opt/intel/impi/
5.1.2.150/intel64/lib -L/opt/intel/impi/5.1.2.150/intel64/lib
-Wl,-rpath,/opt/intel/compilers_and_libraries_2016/linux/mkl/lib/intel64
-L/opt/intel/compilers_and_libraries_2016/linux/mkl/lib/intel64
-Wl,-rpath,/opt/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64_lin
-L/opt/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64_lin
-Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.7
-L/usr/lib/gcc/x86_64-redhat-linux/4.4.7
-Wl,-rpath,/opt/intel/compilers_and_libraries_2016/linux/mkl/lib/intel64
-L/opt/intel/compilers_and_libraries_2016/linux/mkl/lib/intel64
-Wl,-rpath,/opt/intel/impi/5.1.2.150/intel64/lib/debug_mt
-Wl,-rpath,/opt/intel/impi/5.1.2.150/intel64/lib
-Wl,-rpath,/opt/intel/mpi-rt/5.1/intel64/lib/debug_mt
-Wl,-rpath,/opt/intel/mpi-rt/5.1/intel64/lib -limf -lsvml -lirng -lm -lipgo
-ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lirc_s
-Wl,-rpath,/opt/intel/impi/5.1.2.150/intel64/lib/debug_mt -L/opt/intel/impi/
5.1.2.150/intel64/lib/debug_mt -Wl,-rpath,/opt/intel/impi/
5.1.2.150/intel64/lib -L/opt/intel/impi/5.1.2.150/intel64/lib
-Wl,-rpath,/opt/intel/compilers_and_libraries_2016/linux/mkl/lib/intel64
-L/opt/intel/compilers_and_libraries_2016/linux/mkl/lib/intel64
-Wl,-rpath,/opt/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64_lin
-L/opt/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64_lin
-Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.7
-L/usr/lib/gcc/x86_64-redhat-linux/4.4.7
-Wl,-rpath,/opt/intel/compilers_and_libraries_2016/linux/mkl/lib/intel64
-L/opt/intel/compilers_and_libraries_2016/linux/mkl/lib/intel64 -ldl

exec/HELLO

4. Then I have seen that:
4.1. If I set OMP_NUM_THREADS=2 and I remove -lpetsc and -lifcore_pic from
the last step, I got:
Hello from  0 out of  2
Hello from  1 out of  2
4.2 But if add -lpetsc and -lifcore_pic (because I want to use PETSC) I get
this error:
Hello from  0 out of  2
forrtl: severe (40): recursive I/O operation, unit -1, file unknown
Image              PC                Routine            Line        Source
HELLO              000000000041665C  Unknown               Unknown  Unknown
HELLO              00000000004083C8  Unknown               Unknown  Unknown
libiomp5.so        00007F9C603566A3  Unknown               Unknown  Unknown
libiomp5.so        00007F9C60325007  Unknown               Unknown  Unknown
libiomp5.so        00007F9C603246F5  Unknown               Unknown  Unknown
libiomp5.so        00007F9C603569C3  Unknown               Unknown  Unknown
libpthread.so.0    0000003CE76079D1  Unknown               Unknown  Unknown
libc.so.6          0000003CE6AE88FD  Unknown               Unknown  Unknown
If you set OMP_NUM_THREADS to 8, I get:
forrtl: severe (40): recursive I/O operation, unit -1, file unknown
forrtl: severe (40): recursive I/O operation, unit -1, file unknown
forrtl: severe (40): recursive I/O operation, unit -1, file unknown

I am sorry if this is a trivial problem because I guess that lots of people
use PETSC with OpenMP in FORTRAN, but I have really done my best to figure
out where the error is. Can you help me?

Thanks a lot!

Adrian.

Reply via email to