I changed the code as you suggested, and I updated the nightly tarball. 

Unfortunately, I still got the error below with number of processors, say 22, 
55, 61, 62 ... on the Linux cluster.  And I think the number might be different 
on your machine. Could you please try other numbers, say 50~63?

It seems the errors could be from my side. Could you please give me some idea 
on how to debug such a problem? 

Thank you.

[60]PETSC ERROR: 
------------------------------------------------------------------------
[60]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch 
system) has told this process to end
[60]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[60]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[60]PETSC ERROR: or 
try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory 
corruption errors
[60]PETSC ERROR: likely location of problem given in stack below
[60]PETSC ERROR: ---------------------  Stack Frames 
------------------------------------
[60]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[60]PETSC ERROR:       INSTEAD the line number of the start of the function
[60]PETSC ERROR:       is given.
[60]PETSC ERROR: [60] MatAssemblyBegin_MPISBAIJ line 483 
/home/xiao/Local/petsc-dev-debug/src/mat/impls/sbaij/mpi/mpisbaij.c
[60]PETSC ERROR: [60] MatAssemblyBegin line 4854 
/home/xiao/Local/petsc-dev-debug/src/mat/interface/matrix.c
[60]PETSC ERROR: [60] DMCreateMatrix_DA_3d_MPISBAIJ line 1679 
/home/xiao/Local/petsc-dev-debug/src/dm/impls/da/fdda.c
[60]PETSC ERROR: [60] DMCreateMatrix_DA line 625 
/home/xiao/Local/petsc-dev-debug/src/dm/impls/da/fdda.c
[60]PETSC ERROR: [60] DMCreateMatrix line 961 
/home/xiao/Local/petsc-dev-debug/src/dm/interface/dm.c
[60]PETSC ERROR: --------------------- Error Message 
------------------------------------
[60]PETSC ERROR: Signal received!
[60]PETSC ERROR: 
------------------------------------------------------------------------
[60]PETSC ERROR: Petsc Development GIT revision: v3.4.3-2332-g54f71ec  GIT 
Date: 2014-01-20 14:12:11 -0700
[60]PETSC ERROR: See docs/changes/index.html for recent updates.
[60]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[60]PETSC ERROR: See docs/index.html for manual pages.
[60]PETSC ERROR: 
------------------------------------------------------------------------
[60]PETSC ERROR: ./ex44f on a linux-gnu named cluster08 by xiao Wed Jan 22 
23:51:36 2014
[60]PETSC ERROR: Libraries linked from 
/home/xiao/Local/petsc-dev-debug/linux-gnu/lib
[60]PETSC ERROR: Configure run at Wed Jan 22 23:45:50 2014
[60]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 
--with-cxx=mpicxx --with-debugging=1
[60]PETSC ERROR: 
------------------------------------------------------------------------

JJ
_________
_______________________________
From: Barry Smith [[email protected]]
Sent: Wednesday, January 22, 2014 9:29 PM
To: Xiao, Jianjun (IKET)
Cc: [email protected]; [email protected]
Subject: Re: [petsc-users] Error in creation of MPISBAIJ matrix with 
DMCreateMatrix

  There was an error in your code; because you did not use implicit none in the 
main program it did not tell you
      CALL DMDACreate3d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,               &
                                                           1
Error: Symbol 'dmda_boundary_none' at (1) has no IMPLICIT type
ex1f.F90:14.26:

     &    DMDA_STENCIL_BOX,-1002,-3,-3,PETSC_DECIDE,PETSC_DECIDE,&
                          1
Error: Symbol 'dmda_stencil_box' at (1) has no IMPLICIT type


   I fixed the code by adding use petscdmda and the implicit none and it ran 
without error on those number of processes on both a MacOS and a Linux system

     program main   !   Solves the linear system  J x = f
      use petscksp; use petscdm; use petscdmda
      implicit none
#include <finclude/petscdef.h>

   Please let me know if you still have a problem.

   Barry


On Jan 22, 2014, at 5:15 AM, Xiao, Jianjun (IKET) <[email protected]> wrote:

> Dear Barry,
>
> I modified ex44f.F90, and ran the case on my 64-processors cluster. Please 
> find the file in the attachment.
>
> I tried various number of processors. It seems that for number of processors: 
> 1, 2, 3, 4, 5, 6, 7, 8, 16, 32 and 64 it works fine. For numbers such as 20, 
> 30, 31, 60, 61 and 62, I got the error below. This time, even for MATMPIBAIJ 
> I got error. I did not try all the numbers between 1-64. If you need more 
> information, please let me know.
>
> Thank you for your help.
>
> Best regards
> JJ
>
>
> For MATMPISBAIJ, I got the error like this:
>
> [55]PETSC ERROR: [55] MatAssemblyBegin_MPISBAIJ line 483 
> src/mat/impls/sbaij/mpi/mpisbaij.c
> [55]PETSC ERROR: [55] MatAssemblyBegin line 4865 src/mat/interface/matrix.c
> [55]PETSC ERROR: [55] DMCreateMatrix_DA_3d_MPISBAIJ line 1694 
> src/dm/impls/da/fdda.c
> [55]PETSC ERROR: [55] DMCreateMatrix_DA line 626 src/dm/impls/da/fdda.c
> [55]PETSC ERROR: [55] DMCreateMatrix line 1002 src/dm/interface/dm.c
> [55]PETSC ERROR: --------------------- Error Message 
> ------------------------------------
> [55]PETSC ERROR: Signal received!
> [55]PETSC ERROR: 
> ------------------------------------------------------------------------
> [55]PETSC ERROR: Petsc Development GIT revision: 
> f7404d5510646a3c64be49fff6ce547efef07b3d  GIT Date: 2013-11-27 00:12:54 +0100
> [55]PETSC ERROR: See docs/changes/index.html for recent updates.
> [55]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [55]PETSC ERROR: See docs/index.html for manual pages.
> [55]PETSC ERROR: 
> ------------------------------------------------------------------------
> [55]PETSC ERROR: ./ex44f on a linux-gnu named cluster07 by xiao Wed Jan 22 
> 11:54:32 2014
> [55]PETSC ERROR: Libraries linked from 
> /home/xiao/Local/petsc-dev-debug/linux-gnu/lib
> [55]PETSC ERROR: Configure run at Tue Jan 14 15:27:57 2014
> [55]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 
> --with-cxx=mpicxx --with-debugging=1
> [55]PETSC ERROR: 
> ------------------------------------------------------------------------
> [55]PETSC ERROR: User provided function() line 0 in  unknown file
>
>
> For MATMPIBAIJ, I got the error like this:
>
> [44]PETSC ERROR: --------------------- Error Message 
> ------------------------------------
> [44]PETSC ERROR: Argument out of range!
> [44]PETSC ERROR: Trying to set preallocation for row 6678 less than first 
> local row 6714!
> [44]PETSC ERROR: 
> ------------------------------------------------------------------------
> [44]PETSC ERROR: Petsc Development GIT revision: 
> f7404d5510646a3c64be49fff6ce547efef07b3d  GIT Date: 2013-11-27 00:12:54 +0100
> [44]PETSC ERROR: See docs/changes/index.html for recent updates.
> [44]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [44]PETSC ERROR: See docs/index.html for manual pages.
> [44]PETSC ERROR: 
> ------------------------------------------------------------------------
> [44]PETSC ERROR: ./ex44f on a linux-gnu named cluster06 by xiao Wed Jan 22 
> 11:52:08 2014
> [44]PETSC ERROR: Libraries linked from 
> /home/xiao/Local/petsc-dev-debug/linux-gnu/lib
> [44]PETSC ERROR: Configure run at Tue Jan 14 15:27:57 2014
> [44]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 
> --with-cxx=mpicxx --with-debugging=1
> [44]PETSC ERROR: 
> ------------------------------------------------------------------------
> [44]PETSC ERROR: DMCreateMatrix_DA_3d_MPIBAIJ() line 1508 in 
> src/dm/impls/da/fdda.c
> [44]PETSC ERROR: DMCreateMatrix_DA() line 771 in src/dm/impls/da/fdda.c
> [44]PETSC ERROR: DMCreateMatrix() line 1007 in src/dm/interface/dm.c
> [44]PETSC ERROR: --------------------- Error Message 
> ------------------------------------
> [44]PETSC ERROR: Corrupt argument:
> see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind!
> [44]PETSC ERROR: Invalid Pointer to Object: Parameter # 1!
> [44]PETSC ERROR: 
> ------------------------------------------------------------------------
> [44]PETSC ERROR: Petsc Development GIT revision: 
> f7404d5510646a3c64be49fff6ce547efef07b3d  GIT Date: 2013-11-27 00:12:54 +0100
> [44]PETSC ERROR: See docs/changes/index.html for recent updates.
> [44]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [44]PETSC ERROR: See docs/index.html for manual pages.
> [44]PETSC ERROR: 
> ------------------------------------------------------------------------
> [44]PETSC ERROR: ./ex44f on a linux-gnu named cluster06 by xiao Wed Jan 22 
> 11:52:08 2014
> [44]PETSC ERROR: Libraries linked from 
> /home/xiao/Local/petsc-dev-debug/linux-gnu/lib
> [44]PETSC ERROR: Configure run at Tue Jan 14 15:27:57 2014
> [44]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 
> --with-cxx=mpicxx --with-debugging=1
> [44]PETSC ERROR: 
> ------------------------------------------------------------------------
> [44]PETSC ERROR: MatDestroy() line 1029 in src/mat/interface/matrix.c
>
> ________________________________________
> From: Barry Smith [[email protected]]
> Sent: Monday, January 20, 2014 8:37 PM
> To: Xiao, Jianjun (IKET)
> Cc: [email protected]; [email protected]
> Subject: Re: [petsc-users] Error in creation of MPISBAIJ matrix with 
> DMCreateMatrix
>
>   Thanks for reporting the problem. This is our error. Could you please send 
> us the code that generates the error so we can reproduce the problem, 
> determine the cause and fix it.
>
>   Barry
>
> We need to know the exact values of imax,jmax, kmax etc to reproduce the 
> problem.
>
>
>
> On Jan 20, 2014, at 9:33 AM, Xiao, Jianjun (IKET) <[email protected]> 
> wrote:
>
>> Dear developers,
>>
>> I am using petsc-dev. I tried to create a MPISBAIJ matrix as shown below, 
>> and it seems that the matrix creation is sensitive to the number of 
>> processors.
>>
>>     CALL DMDACreate3d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,            &
>>    &    DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,                        &
>>    &    DMDA_STENCIL_BOX,-imax,-jmax,-kmax,PETSC_DECIDE,PETSC_DECIDE,&
>>    &    PETSC_DECIDE,1,1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,       &
>>    &    PETSC_NULL_INTEGER,da,ierr)
>>
>>      CALL DMSetMatType(da,MATMPISBAIJ,ierr)
>>      CALL DMCreateMatrix(da,mat,ierr)
>>
>> A cluster with 64 processors was used for the testing.
>>
>> When the number of procssors are 1,2,3,4,5,6,7,8,16,32 and 64, the code 
>> always works quite well.
>>
>> For some other numbers, the code works not so stable. Sometimes, the matrix 
>> was created successfully. Sometimes, it failed.
>>
>> When the number of procssors are 20, 33, 63 or some relatively bigger 
>> numbers , the code always got the error below.
>>
>>
>> mpirun: [25]PETSC ERROR: --------------------- Error Message 
>> ------------------------------------
>> mpirun: [25]PETSC ERROR: Argument out of range!
>> mpirun: [25]PETSC ERROR: New nonzero at (198,7038) caused a malloc!
>> mpirun: [25]PETSC ERROR: 
>> ------------------------------------------------------------------------
>> mpirun: [25]PETSC ERROR: Petsc Development GIT revision: 
>> f7404d5510646a3c64be49fff6ce547efef07b3d GIT Date: 2013-11-27 00:12:54 +0100
>> mpirun: [25]PETSC ERROR: See docs/changes/index.html for recent updates.
>> mpirun: [25]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> mpirun: [25]PETSC ERROR: See docs/index.html for manual pages.
>> mpirun: [25]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 
>> --with-cxx=mpicxx --with-debugging=1
>> mpirun: [25]PETSC ERROR: 
>> ------------------------------------------------------------------------
>> mpirun: [25]PETSC ERROR: MatSetValuesBlocked_SeqBAIJ() line 1836 in 
>> src/mat/impls/baij/seq/baij.c
>> mpirun: [25]PETSC ERROR: MatSetValuesBlocked_MPISBAIJ() line 339 in 
>> src/mat/impls/sbaij/mpi/mpisbaij.c
>> mpirun: [25]PETSC ERROR: MatSetValuesBlocked() line 1658 in 
>> src/mat/interface/matrix.c
>> mpirun: [25]PETSC ERROR: DMCreateMatrix_DA_3d_MPISBAIJ() line 1780 in 
>> src/dm/impls/da/fdda.c
>> mpirun: [25]PETSC ERROR: DMCreateMatrix_DA() line 777 in 
>> src/dm/impls/da/fdda.c
>> mpirun: [25]PETSC ERROR: DMCreateMatrix() line 1007 in src/dm/interface/dm.c
>>
>> Then I changed the matrix format to MATMPIBAIJ. DMCreateMatrix worked fine 
>> for any number of processors.
>>
>>      CALL DMSetMatType(da,MATMPIBAIJ,ierr)
>>      CALL DMCreateMatrix(da,gfmat,ierr)
>>
>> Could you please let me know how can I fix this problem? If you need more 
>> information, please let me know. Thank you.
>>
>> JJ
>
> <ex44f.F90>

      program main   !   Solves the linear system  J x = f
      use petscksp; use petscdm; use petscdmda
      implicit none
#include <finclude/petscdef.h>
      Vec x,f
      Mat J
      DM da
      KSP ksp
      PetscErrorCode ierr
      call PetscInitialize(PETSC_NULL_CHARACTER,ierr)

      CALL DMDACreate3d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,               &
     &    DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,                        &
     &    DMDA_STENCIL_BOX,-1002,-3,-3,PETSC_DECIDE,PETSC_DECIDE,&
     &    PETSC_DECIDE,1,1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,       &
     &    PETSC_NULL_INTEGER,da,ierr)

!      call DMCreateGlobalVector(da,x,ierr)
!      call VecDuplicate(x,f,ierr)

      call DMSetMatType(da,MATMPISBAIJ,ierr)
      call DMCreateMatrix(da,J,ierr)
      call MatView(J,PETSC_VIEWER_STDOUT_WORLD,ierr)

!      call ComputeRHS(da,f,ierr)
!      call ComputeMatrix(da,J,ierr)

!      call KSPCreate(MPI_COMM_WORLD,ksp,ierr)
!      call KSPSetOperators(ksp,J,J,SAME_NONZERO_PATTERN,ierr)
!      call KSPSetFromOptions(ksp,ierr)
!      call KSPSolve(ksp,f,x,ierr)

      call MatDestroy(J,ierr)
!      call VecDestroy(x,ierr)
!      call VecDestroy(f,ierr)
!      call KSPDestroy(ksp,ierr)
      call DMDestroy(da,ierr)
      call PetscFinalize(ierr)
      end

!      subroutine  ComputeRHS(da,x,ierr)
!#include <finclude/petscdef.h>
!      use petscdm
!      DM da
!      Vec x
!      PetscErrorCode ierr
!      PetscInt xs,xm,i,mx
!      PetscScalar hx
!      PetscScalar, pointer :: xx(:)
!      call DMDAGetInfo(da,PETSC_NULL_INTEGER,mx,PETSC_NULL_INTEGER,     &
!     &  PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,       &
!     &  PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,       &
!     &  PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,       &
!     &  PETSC_NULL_INTEGER,ierr)
!      call DMDAGetCorners(da,xs,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,  &
!     &  xm,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr)
!      hx     = 1.d0/(mx-1)
!      call VecGetArrayF90(x,xx,ierr)
!      do i=xs,xs+xm-1
!          xx(i) = i*hx
!      enddo
!      call VecRestoreArrayF90(x,xx,ierr)
!      return
!      end
!      subroutine ComputeMatrix(da,J,ierr)
!#include <finclude/petscdef.h>
!      use petscdm
!      Mat J
!      DM da
!      PetscErrorCode ierr
!      PetscInt xs,xm,i,mx
!      PetscScalar hx
!      call DMDAGetInfo(da,PETSC_NULL_INTEGER,mx,PETSC_NULL_INTEGER,     &
!     &  PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,       &
!     &  PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,       &
!     &  PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,       &
!     &  PETSC_NULL_INTEGER,ierr)
!      call DMDAGetCorners(da,xs,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,  &
!     &  xm,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr)
!      hx     = 1.d0/(mx-1)
!      do i=xs,xs+xm-1
!        if ((i .eq. 0) .or. (i .eq. mx-1)) then
!          call MatSetValue(J,i,i,1d0,INSERT_VALUES,ierr)
!        else
!          call MatSetValue(J,i,i-1,-hx,INSERT_VALUES,ierr)
!          call MatSetValue(J,i,i+1,-hx,INSERT_VALUES,ierr)
!          call MatSetValue(J,i,i,2*hx,INSERT_VALUES,ierr)
!        endif
!      enddo
!      call MatAssemblyBegin(J,MAT_FINAL_ASSEMBLY,ierr)
!      call MatAssemblyEnd(J,MAT_FINAL_ASSEMBLY,ierr)
!      return
!      end

Reply via email to