Thank you very much Barry, it works indeed to build petsc with the qopenmp flags!
Best Timothee 2018-01-30 18:46 GMT+01:00 Satish Balay <[email protected]>: > Try building PETSc with -qopenmp flag and see if this makes a difference. > > i.e use CFLAGS=-qopenmp FFLAGS=-qopenmp CXXFLAGS=-qopenmp > > Potential explanation: > > Compilers do different things internally [for ex - link to different > libraries] when different compile flags are used. > > When -qopenmp is used - its possible some openmp library gets linked > in. And when -qopenmp is not used - some equivalnet stub library might > get linked in. Only one lib is supporsed to be used at anytime. With > the way petsc gets built - both might get used - causing some > conflict. > > You can verify with: > > <compare logs from> > mpicc -v -qopenmp dummy.c > mpicc -v dummy.c > > And 'ldd libpetsc.so' > > Satish > > On Tue, 30 Jan 2018, Timothée Nicolas wrote: > > > Thank you for your answer, > > > > I am not responsible for the choices in this code to begin with, but in > any > > case, as you can see in the example, I am not using any petsc call. Also > in > > our code, even though openmp is used, this is never in interaction with > > petsc, the latter being used only at a specific (but crucial) place in > the > > code, and without openMP. In that sense, the model we are using is NOT > some > > 'Petsc with threads model', as described in the page you sent me. > > Nevertheless, it appears that just linking petsc is enough to perturb the > > behavior of openMP. I believe it may require further investigation. On > top > > of the bug I report in my first message, I have noticed very weird openMP > > behavior lately, the kind of bugs that 'don't make sense' at first sight > > (would be too long/boring to give details at this stage). I am starting > to > > realize that it may have something to do with petsc. > > > > Again, the webpage you have sent me seems to adopt the point of view that > > hybrid MPI/openMP is useless from the performance point of view, and/or > > that Petsc routines should not be threaded. It does not, however, explain > > why the behaviour of openMP may be changed by merely linking a program > with > > petsc, without a single petsc call in the program. > > > > Best > > > > 2018-01-30 18:02 GMT+01:00 Smith, Barry F. <[email protected]>: > > > > > > > > I don't know what you are trying to do with OpenMP and PETSc, nor do > I > > > understand why anyone would use OpenMP, but you cannot call virtually > any > > > PETSc function or operation while you are using threads. > > > > > > Best to use PETSc as intended, with one MPI process per core or > hardware > > > thread and not use OpenMP at all. http://www.mcs.anl.gov/petsc/ > > > miscellaneous/petscthreads.html > > > > > > > > > Barry > > > > > > > > > > On Jan 30, 2018, at 10:57 AM, Timothée Nicolas < > > > [email protected]> wrote: > > > > > > > > Dear petsc team, > > > > > > > > For a while, I have been wondering why I have never managed to print > > > what threads are doing in an openMP region in my FORTRAN programs. Some > > > people told me it was normal because threads will get confused all > trying > > > to write at the same time. However I realised today that the problem > seems > > > to be related to petsc. I have a super simple "hello world" example > that > > > reproduces the problem : > > > > > > > > program hello > > > > !$ use omp_lib > > > > implicit none > > > > integer nthreads, tid > > > > > > > > !$omp parallel private(nthreads, tid) > > > > tid = omp_get_thread_num() > > > > print *, 'hello world from thread = ', tid > > > > > > > > if (tid .eq. 0) then > > > > nthreads = omp_get_num_threads() > > > > print *, 'number of threads = ', nthreads > > > > end if > > > > !$omp end parallel > > > > > > > > end program hello > > > > > > > > If I compile it with > > > > > > > > mpif90 -qopenmp -o omp main.f90 > > > > > > > > Then I get no problem. But if I link petsc library (as is the case > in my > > > code) : > > > > > > > > mpif90 -qopenmp -o omp main.f90 -L/home/timotheenicolas/petsc- > 3.7.3/arch-linux2-c-debug/lib > > > -lpetsc > > > > > > > > > > > > Then I get the following error after execution of export > > > OMP_NUM_THREADS=2;./omp > > > > > > > > hello world from thread = 0 > > > > number of threads = 2 > > > > forrtl: severe (40): recursive I/O operation, unit -1, file unknown > > > > Image PC Routine Line > > > Source > > > > omp 0000000000403BC8 Unknown Unknown > > > Unknown > > > > omp 0000000000403572 Unknown Unknown > > > Unknown > > > > libiomp5.so 00002AAAAD3146A3 Unknown Unknown > > > Unknown > > > > libiomp5.so 00002AAAAD2E3007 Unknown Unknown > > > Unknown > > > > libiomp5.so 00002AAAAD2E26F5 Unknown Unknown > > > Unknown > > > > libiomp5.so 00002AAAAD3149C3 Unknown Unknown > > > Unknown > > > > libpthread.so.0 00002AAAAD5BADC5 Unknown Unknown > > > Unknown > > > > libc.so.6 00002AAAAD8C5CED Unknown Unknown > > > Unknown > > > > > > > > > > > > This means that in my programs using petsc, I can't use prints to see > > > exactly what the threads are doing, which is a pain when debug is > required > > > (that is to say, all the time). Is this issue expected ? > > > > > > > > Thx in advance > > > > > > > > Timothee NICOLAS > > > > > > > > > > > > >
