[petsc-users] Strange efficiency in PETSc-dev using OpenMP

[Danyang Su]

Hi Shri,







Iterations     6 *time_assembly 0.4137E-01* time_ksp  0.9296E-01


Iterations     6 *time_assembly 0.3595E+03* time_ksp  0.2907E+00


                do i = istart, iend - 1
                   ii = ia_in(i+1)
                   jj = ia_in(i+2)

                end do

The log files for both PETSc-3.4.2 and PETSc-dev are attached.



Thanks and regards,

Danyang

On 21/09/2013 2:09 PM, Shri wrote:

Thanks for your info. It can work with the option -threadcomm_type openmp. But another problem arises, as described as follows. The sparse matrix is 53760*53760 with 1067392 non-zero entries. If the codes is compiled using PETSc-3.4.2, it works fine, the equations can be solved quickly and I can see the speedup. But if the code is compiled using PETSc-dev with OpenMP option, it takes a long time in solving the equations and I cannot see any speedup when more processors are used. For PETSc-3.4.2, run by "mpiexec -n 4 ksp_inhm_d -log_summary log_mpi4_petsc3.4.2.log", the iteration and runtime are: For PETSc-dev, run by "mpiexec -n 1 ksp_inhm_d -threadcomm_type openmp -threadcomm_nthreads 4 -log_summary log_openmp_petsc_dev.log", the iteration and runtime are: Most of the time *'time_assembly 0.3595E+03*' is spent on the following codes call MatSetValues(a, ione, i, jj-ii, ja_in(ii:jj-1)-1, a_in(ii:jj-1), Insert_Values, ierr) Is there anything wrong with my codes or with running option? The above codes works fine when using MPICH.

There are three thread communicator types in PETSc. The default is "no thread" which is basically a non-threaded version. The other two types are "openmp" and "pthread". If you want to use OpenMP then use the option -threadcomm_type openmp.

Shri

On Sep 21, 2013, at 3:46 PM, Danyang Su <danyang...@gmail.com <mailto:danyang...@gmail.com>> wrote:

Hi Barry,

Thanks for the quick reply.

After changing
/#if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP) /
to
/#if defined(PETSC_HAVE_PTHREADCLASSES)/
and comment out
/#elif defined(PETSC_HAVE_OPENMP)//
//PETSC_EXTERN PetscStack *petscstack;/

It can be compiled and validated with "make test".





Thanks and regards,

Danyang



On 21/09/2013 7:26 AM, Barry Smith wrote:

But I still have questions on running the examples. After rebuild the codes (e.g., ksp_ex2f.f), I can run it with "mpiexec -n 1 ksp_ex2f", or "mpiexec -n 4 ksp_ex2f", or "mpiexec -n 1 ksp_ex2f -threadcomm_nthreads 1", but if I run it with "mpiexec -n 1 ksp_ex2f -threadcomm_nthreads 4", there will be a lot of error information (attached). The codes is not modified and there is no OpenMP routines in it. For the current development in my project, I want to keep the OpenMP codes in calculating matrix values, but want to solve it with PETSc (OpenMP). Is it possible?

   Danyang,

      I don't think the  || defined (PETSC_HAVE_OPENMP)   belongs in the code 
below.

/*  Linux functions CPU_SET and others don't work if sched.h is not included 
before
     including pthread.h. Also, these functions are active only if either 
_GNU_SOURCE
     or __USE_GNU is not set (see /usr/include/sched.h and 
/usr/include/features.h), hence
     set these first.
*/
#if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP)

Edit include/petscerror.h and locate these lines and remove that part and then 
rerun make all.  Let us know if it works or not.

    Barry

i.e. replace

#if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP)

with

#if defined(PETSC_HAVE_PTHREADCLASSES)

On Sep 21, 2013, at 6:53 AM, Matthew Knepley<petsc-ma...@mcs.anl.gov>  wrote:

On Sat, Sep 21, 2013 at 12:18 AM, Danyang Su<danyang...@gmail.com>  wrote:
Hi All,

I got error information in compiling petsc-dev with openmp in cygwin. Before, I 
have successfully compiled petsc-3.4.2 and it works fine.
The log files have been attached.

The OpenMP configure test is wrong. It clearly fails to find pthread.h, but the 
test passes. Then in petscerror.h
we guard pthread.h using PETSC_HAVE_OPENMP. Can someone who knows OpenMP fix 
this?

     Matt


Danyang



--
What most experimenters take for granted before they begin their experiments is 
infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

Thanks,

<error.txt>

log_mpi4_petsc3.4.2.log
Description: Binary data

log_openmp_petsc_dev.log
Description: Binary data