Hi everyone,
Thanks for your responses, i understand communicating on this way to this
level of technicality can be hard, i still think we can work ways to solve
this problem though,
I can say the following at this point:
- Program works without issues in any other non-gpu preconditioner, just
by calling -pc_type ### .
- Program works correctly with ViennaCL preconditioners with mpirun -n
1, that is on one processor.
- Program breaks with every ViennaCL preconditioner when attempting any
more than 1 processors.
- I haven't tried other GPU preconditioners because ViennaCL gave the
best performance in 1 processor before.
- I have made sure the matrix type is mpiaijviennacl as it is printed
right before the error log.
- I am attaching what i think are the backtrace logs of the run with two
cores, as i have limited experience with this kind of debugging, and i
cannot make sense of what they say, please let me know if you need me to do
something else.
Current options i can think of are giving you permission to clone my model
repo, or for you to guide me trough the process remotely,
Thanks,
.-.-.-.-
The error i get is still:
Matrix type: mpiaijviennacl
Of sizes: 125 x 125
Matrix type: mpiaijviennacl
Of sizes: 125 x 125
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: No support for this operation for this object type
[0]PETSC ERROR: Currently only handles ViennaCL matrices
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.9.2-549-g779ab53 GIT
Date: 2018-05-31 17:31:13 +0300
[0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera Wed Aug
29 11:43:25 2018
[0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
--with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2
--FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1
--CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 --download-viennacl
[0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in
/home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu
[0]PETSC ERROR: #2 PCSetUp() line 932 in
/home/valera/petsc/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #3 KSPSetUp() line 381 in
/home/valera/petsc/src/ksp/ksp/interface/itfunc.c
[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
probably memory access out of range
[1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[1]PETSC ERROR: or see
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X
to find memory corruption errors
[1]PETSC ERROR: likely location of problem given in stack below
[1]PETSC ERROR: --------------------- Stack Frames
------------------------------------
[1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[1]PETSC ERROR: INSTEAD the line number of the start of the function
[1]PETSC ERROR: is given.
[1]PETSC ERROR: [1] PetscTraceBackErrorHandler line 182
/home/valera/petsc/src/sys/error/errtrace.c
[1]PETSC ERROR: [1] PetscError line 352
/home/valera/petsc/src/sys/error/err.c
[1]PETSC ERROR: [1] PCSetUp_SAVIENNACL line 45
/home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu
[1]PETSC ERROR: [1] PCSetUp line 894
/home/valera/petsc/src/ksp/pc/interface/precon.c
[1]PETSC ERROR: [1] KSPSetUp line 294
/home/valera/petsc/src/ksp/ksp/interface/itfunc.c
[1]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[1]PETSC ERROR: Signal received
[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[1]PETSC ERROR: Petsc Development GIT revision: v3.9.2-549-g779ab53 GIT
Date: 2018-05-31 17:31:13 +0300
[1]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera Wed Aug
29 11:43:25 2018
[1]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
--with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2
--FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1
--CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 --download-viennacl
[1]PETSC ERROR: #1 User provided function() line 0 in unknown file
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
with errorcode 59.
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
batch system) has told this process to end
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X
to find memory corruption errors
[0]PETSC ERROR: likely location of problem given in stack below
[0]PETSC ERROR: --------------------- Stack Frames
------------------------------------
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR: INSTEAD the line number of the start of the function
[0]PETSC ERROR: is given.
[0]PETSC ERROR: [0] MatSetErrorIfFailure line 116
/home/valera/petsc/src/mat/utils/gcreate.c
[0]PETSC ERROR: [0] PCSetUp line 894
/home/valera/petsc/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: [0] PCSetUp_SAVIENNACL line 45
/home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu
[0]PETSC ERROR: [0] PCSetUp line 894
/home/valera/petsc/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: [0] KSPSetUp line 294
/home/valera/petsc/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Signal received
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.9.2-549-g779ab53 GIT
Date: 2018-05-31 17:31:13 +0300
[0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera Wed Aug
29 11:43:25 2018
[0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
--with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2
--FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1
--CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 --download-viennacl
[0]PETSC ERROR: #4 User provided function() line 0 in unknown file
[node50:32783] 1 more process has sent help message help-mpi-api.txt /
mpi-abort
[node50:32783] Set MCA parameter "orte_base_help_aggregate" to 0 to see all
help / error messages
On Tue, Aug 28, 2018 at 9:34 PM, Karl Rupp <[email protected]> wrote:
> Hi Manuel,
>
> as Barry said, it is hard for us to provide any help without having a more
> complete picture of what is going on.
>
> The error you report seems to come from the AMG preconditioner in
> ViennaCL. This can have many origins. Do other preconditioners run without
> error? Can you run in a debugger and provide a complete backtrace?
>
> Thanks and best regards,
> Karli
>
>
> On 08/29/2018 01:33 AM, Manuel Valera wrote:
>
>> Talked too fast,
>>
>> After fixing that problem, i tried more than one mpi processor and got
>> the following:
>>
>> Matrix type: mpiaijviennacl
>> Of sizes: 125 x 125
>> Matrix type: mpiaijviennacl
>> Of sizes: 125 x 125
>> [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: No support for this operation for this object type
>> [0]PETSC ERROR: Currently only handles ViennaCL matrices
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9.2-549-g779ab53 GIT
>> Date: 2018-05-31 17:31:13 +0300
>> [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera Tue
>> Aug 28 16:30:02 2018
>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2
>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1
>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64
>> --download-viennacl
>> [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in
>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu <
>> http://saviennacl.cu>
>> [0]PETSC ERROR: #2 PCSetUp() line 932 in /home/valera/petsc/src/ksp/pc/
>> interface/precon.c
>> [1]PETSC ERROR: ------------------------------
>> ------------------------------------------
>> [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
>> probably memory access out of range
>> [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>> [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d
>> ocumentation/faq.html#valgrind
>> [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS
>> X to find memory corruption errors
>> [1]PETSC ERROR: likely location of problem given in stack below
>> [1]PETSC ERROR: --------------------- Stack Frames
>> ------------------------------------
>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>> available,
>> [1]PETSC ERROR: INSTEAD the line number of the start of the function
>> [1]PETSC ERROR: is given.
>> [1]PETSC ERROR: [1] PetscTraceBackErrorHandler line 182
>> /home/valera/petsc/src/sys/error/errtrace.c
>> [1]PETSC ERROR: [1] PetscError line 352 /home/valera/petsc/src/sys/err
>> or/err.c
>> [1]PETSC ERROR: [1] PCSetUp_SAVIENNACL line 45
>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu <
>> http://saviennacl.cu>
>>
>> [1]PETSC ERROR: [1] PCSetUp line 894 /home/valera/petsc/src/ksp/pc/
>> interface/precon.c
>> [1]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [1]PETSC ERROR: Signal received
>> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [1]PETSC ERROR: Petsc Development GIT revision: v3.9.2-549-g779ab53 GIT
>> Date: 2018-05-31 17:31:13 +0300
>> [1]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera Tue
>> Aug 28 16:30:02 2018
>> [1]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2
>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1
>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64
>> --download-viennacl
>> [1]PETSC ERROR: #1 User provided function() line 0 in unknown file
>> ------------------------------------------------------------
>> --------------
>> MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
>> with errorcode 59.
>>
>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
>> You may or may not see output from other processes, depending on
>> exactly when Open MPI kills them.
>> ------------------------------------------------------------
>> --------------
>> [0]PETSC ERROR: ------------------------------
>> ------------------------------------------
>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
>> batch system) has told this process to end
>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d
>> ocumentation/faq.html#valgrind
>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS
>> X to find memory corruption errors
>> [0]PETSC ERROR: likely location of problem given in stack below
>> [0]PETSC ERROR: --------------------- Stack Frames
>> ------------------------------------
>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>> available,
>> [0]PETSC ERROR: INSTEAD the line number of the start of the function
>> [0]PETSC ERROR: is given.
>> [0]PETSC ERROR: [0] PetscCommDuplicate line 130
>> /home/valera/petsc/src/sys/objects/tagm.c
>> [0]PETSC ERROR: [0] PetscHeaderCreate_Private line 34
>> /home/valera/petsc/src/sys/objects/inherit.c
>> [0]PETSC ERROR: [0] ISCreate line 35 /home/valera/petsc/src/vec/is/
>> is/interface/isreg.c
>> [0]PETSC ERROR: [0] ISCreateGeneral line 668
>> /home/valera/petsc/src/vec/is/is/impls/general/general.c
>> [0]PETSC ERROR: [0] PCSetUp_SAVIENNACL line 45
>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu <
>> http://saviennacl.cu>
>> [0]PETSC ERROR: [0] PCSetUp line 894 /home/valera/petsc/src/ksp/pc/
>> interface/precon.c
>> [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: Signal received
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9.2-549-g779ab53 GIT
>> Date: 2018-05-31 17:31:13 +0300
>> [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera Tue
>> Aug 28 16:30:02 2018
>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2
>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1
>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64
>> --download-viennacl
>> [0]PETSC ERROR: #3 User provided function() line 0 in unknown file
>> [node50:30582] 1 more process has sent help message help-mpi-api.txt /
>> mpi-abort
>> [node50:30582] Set MCA parameter "orte_base_help_aggregate" to 0 to see
>> all help / error messages
>>
>>
>>
>> It is currently running in 1 mpi processor + GPU but i would like to call
>> at least 16 mpi processors + GPU to do the rest of the data management who
>> is not part of the main laplacian on the mpi and the laplacian solution on
>> the GPU, is this currently possible?
>>
>> Thanks for your help,
>>
>>
>>
>> On Tue, Aug 28, 2018 at 4:21 PM, Manuel Valera <[email protected]
>> <mailto:[email protected]>> wrote:
>>
>> Ok, i found the culprit and we can close this thread,
>>
>> The problem was a missing variable for setting the maximum columns,
>> which i deleted at some point without realizing. The error message
>> was too ambiguous to catch this so i had to compare with a previous
>> working version of the arguments of MatSetValues, it was evident then.
>>
>> Good news is that i can now set the values with the viennacl types
>> too,
>>
>> Thanks for your kind help,
>>
>> Manuel
>>
>> On Tue, Aug 28, 2018 at 11:25 AM, Smith, Barry F.
>> <[email protected] <mailto:[email protected]>> wrote:
>>
>>
>> 1) PetscMalloc() is never valid or needed in Fortran
>>
>> 2) there is no reason to use DMSetMatrixPreallocateOnly()
>> just use DMCreateMatrix() assuming that using a DM (DMDA,
>> DMPLEX, etc) is suitable for your problem.
>>
>> At this end we are totally guessing at what you are doing
>> and so have little help we can provide. A simple, nonworking
>> code that tries to do what you would need would help us a great
>> deal in understanding that you are trying to do.
>>
>> Barry
>>
>>
>>
>>
>>
>> > On Aug 28, 2018, at 1:18 PM, Manuel Valera
>> <[email protected] <mailto:[email protected]>> wrote:
>> >
>> > Matthew, PetscMalloc gives the same error,
>> >
>> > Barry, it would be very hard for me to get the code to a
>> minimum working example, i guess all i need to understand is how
>> to setup a DM matrix with DMSetMatrixPreallocateOnly() instead
>> of MatMPIAIJSetPreallocation() as we were doing before, is there
>> a simple example who does this in Fortran?
>> >
>> > Is the PetscMalloc call needed? is 'call
>> PetscMalloc(1,row,ierr)' a valid, compilable call to
>> PetscMalloc? what other reason may there be for this error to
>> happen ?
>> >
>> > Just remembering, that trying to setup the matrix with the
>> MatAIJSetPreallocation() brings up an error to acknowledge the
>> viennacl datatypes and that's why i'm trying to make this change
>> on your recommendation,
>> >
>> > Thanks for your help,
>> >
>> >
>> >
>> >
>> >
>> > On Mon, Aug 27, 2018 at 7:35 PM, Smith, Barry F.
>> <[email protected] <mailto:[email protected]>> wrote:
>> >
>> > Send your code in a way we can compile and run it; it must
>> be some simple issue that is hard to communicate in email.
>> >
>> > Barry
>> >
>> >
>> > > On Aug 27, 2018, at 5:51 PM, Manuel Valera
>> <[email protected] <mailto:[email protected]>> wrote:
>> > >
>> > > Hello everyone,
>> > >
>> > > I just had time to work on this again, and checked the code
>> for errors on the matrix entries, this is the exact code i was
>> using for creating the matrix without
>> DMSetMatrixPreallocateOnly, using MatMPIAIJSetPreallocation and
>> it worked that way, but trying this way i get the same 'Column
>> too large' error using any number at the column position of
>> MatSetValues,
>> > >
>> > > I have set up my code to print the column argument (n) of
>> MatSetValues and in this case is 7 (lower than 124), it still
>> gives error, even entering a specific number in the MatSetValues
>> column argument position gives the same error.
>> > >
>> > > So next i went back to ex.50 here:
>> http://www.mcs.anl.gov/petsc/petsc-current/src/ts/examples/t
>> utorials/ex50.c.html
>> <http://www.mcs.anl.gov/petsc/petsc-current/src/ts/examples/
>> tutorials/ex50.c.html>
>> and it has a very similar structure except the PetscMalloc1()
>> call, so i tried adding that and got:
>> > >
>> > > /home/valera/ParGCCOM/Src/DMDALaplacian.f90:114: undefined
>> reference to `petscmalloc1_'
>> > >
>> > > Any ideas on this behaviour?
>> > >
>> > > Thanks so much,
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > On Thu, Aug 16, 2018 at 11:20 AM, Smith, Barry F.
>> <[email protected] <mailto:[email protected]>> wrote:
>> > >
>> > > Column too large: col 10980 max 124
>> > >
>> > > You need to check the code that is generating the matrix
>> entries. The matrix has 124 columns but you are attempting to
>> put a value at column 10980
>> > >
>> > > Barry
>> > >
>> > >
>> > > > On Aug 15, 2018, at 9:44 PM, Manuel Valera
>> <[email protected] <mailto:[email protected]>> wrote:
>> > > >
>> > > > Thanks Matthew and Barry,
>> > > >
>> > > > Now my code looks like:
>> > > >
>> > > > call DMSetMatrixPreallocateOnly(daDummy,PETSC_TRUE,ierr)
>> > > > call DMSetMatType(daDummy,MATMPIAIJVIENNACL,ierr)
>> > > > call DMSetVecType(daDummy,VECMPIVIENNACL,ierr)
>> > > > call DMCreateMatrix(daDummy,A,ierr)
>> > > > call MatSetFromOptions(A,ierr)
>> > > > call MatSetUp(A,ierr)
>> > > > [...]
>> > > > call
>> MatSetValues(A,1,row,sumpos,pos(0:iter-1),vals(0:iter-1),INS
>> ERT_VALUES,ierr)
>> > > > [...]
>> > > > call MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY, ierr)
>> > > > call MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY, ierr)
>> > > >
>> > > > And i get a different error, now is:
>> > > >
>> > > > [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> > > > [0]PETSC ERROR: Argument out of range
>> > > > [0]PETSC ERROR: Column too large: col 10980 max 124
>> > > > [0]PETSC ERROR: See
>> http://www.mcs.anl.gov/petsc/documentation/faq.html
>> <http://www.mcs.anl.gov/petsc/documentation/faq.html> for
>> trouble shooting.
>> > > > [0]PETSC ERROR: Petsc Development GIT revision:
>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300
>> > > > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50
>> by valera Wed Aug 15 19:40:00 2018
>> > > > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2
>> --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 --with-shared-libraries=1
>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60
>> --with-blaslapack-dir=/usr/lib64 --download-viennacl
>> > > > [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line 442 in
>> /home/valera/petsc/src/mat/impls/aij/seq/aij.c
>> > > > [0]PETSC ERROR: #2 MatSetValues() line 1339 in
>> /home/valera/petsc/src/mat/interface/matrix.c
>> > > >
>> > > >
>> > > > Thanks again,
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Wed, Aug 15, 2018 at 7:02 PM, Smith, Barry F.
>> <[email protected] <mailto:[email protected]>> wrote:
>> > > >
>> > > > Should be
>> > > >
>> > > > call DMSetMatType(daDummy,MATMPIAIJVIENNACL,ierr)
>> > > > call DMSetVecType(daDummy,VECMPIVIENNACL,ierr)
>> > > > call DMCreateMatrix(daDummy,A,ierr)
>> > > >
>> > > > and remove the rest. You need to set the type of Mat
>> you want the DM to return BEFORE you create the matrix.
>> > > >
>> > > > Barry
>> > > >
>> > > >
>> > > >
>> > > > > On Aug 15, 2018, at 4:45 PM, Manuel Valera
>> <[email protected] <mailto:[email protected]>> wrote:
>> > > > >
>> > > > > Ok thanks for clarifying that, i wasn't sure if there
>> were different types,
>> > > > >
>> > > > > Here is a stripped down version of my code, it seems
>> like the preallocation is working now since the matrix
>> population part is working without problem, but here it is for
>> illustration purposes:
>> > > > >
>> > > > > call DMSetMatrixPreallocateOnly(daDummy,PETSC_TRUE,ierr)
>> > > > > call DMCreateMatrix(daDummy,A,ierr)
>> > > > > call MatSetFromOptions(A,ierr)
>> > > > > call DMSetMatType(daDummy,MATMPIAIJVIENNACL,ierr)
>> > > > > call DMSetVecType(daDummy,VECMPIVIENNACL,ierr)
>> > > > > call
>> MatMPIAIJSetPreallocation(A,19,PETSC_NULL_INTEGER,19,PETSC_
>> NULL_INTEGER,ierr)
>> > > > > call MatSetUp(A,ierr)
>> > > > > [...]
>> > > > > call
>> MatSetValues(A,1,row,sumpos,pos(0:iter-1),vals(0:iter-1),INS
>> ERT_VALUES,ierr)
>> > > > > [...]
>> > > > > call MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY, ierr)
>> > > > > call MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY, ierr)
>> > > > >
>> > > > > Adding the first line there did the trick,
>> > > > >
>> > > > > Now the problem seems to be the program is not
>> recognizing the matrix as ViennaCL type when i try with more
>> than one processor, i get now:
>> > > > >
>> > > > > [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> > > > > [0]PETSC ERROR: No support for this operation for this
>> object type
>> > > > > [0]PETSC ERROR: Currently only handles ViennaCL matrices
>> > > > > [0]PETSC ERROR: See
>> http://www.mcs.anl.gov/petsc/documentation/faq.html
>> <http://www.mcs.anl.gov/petsc/documentation/faq.html> for
>> trouble shooting.
>> > > > > [0]PETSC ERROR: Petsc Development GIT revision:
>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300
>> > > > > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named
>> node50 by valera Wed Aug 15 14:44:22 2018
>> > > > > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2
>> --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 --with-shared-libraries=1
>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60
>> --with-blaslapack-dir=/usr/lib64 --download-viennacl
>> > > > > [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in
>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu
>> <http://saviennacl.cu>
>> > > > > [0]PETSC ERROR: #2 PCSetUp() line 932 in
>> /home/valera/petsc/src/ksp/pc/interface/precon.c
>> > > > > [0]PETSC ERROR: #3 KSPSetUp() line 381 in
>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c
>> > > > >
>> > > > > When running with:
>> > > > >
>> > > > > mpirun -n 1 ./gcmLEP.GPU
>> tc=TestCases/LockRelease/LE_6x6x6/ jid=tiny_cuda_test_n2
>> -ksp_type cg -dm_vec_type viennacl -dm_mat_type aijviennacl
>> -pc_type saviennacl -log_view
>> > > > >
>> > > > >
>> > > > > Thanks,
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Wed, Aug 15, 2018 at 2:32 PM, Matthew Knepley
>> <[email protected] <mailto:[email protected]>> wrote:
>> > > > > On Wed, Aug 15, 2018 at 5:20 PM Manuel Valera
>> <[email protected] <mailto:[email protected]>> wrote:
>> > > > > It seems to be resumed on: I do not know how to
>> preallocate a DM Matrix correctly.
>> > > > >
>> > > > > There is only one matrix type, Mat. There are no
>> separate DM matrices. A DM can create a matrix for you
>> > > > > using DMCreateMatrix(), but that is a Mat and it is
>> preallocated correctly. I am not sure what you are doing.
>> > > > >
>> > > > > Thanks,
>> > > > >
>> > > > > Matt
>> > > > >
>> > > > > The interesting part is that it only breaks when i need
>> to populate a GPU matrix from MPI, so kudos on that, but it
>> seems i need to do better on my code to get this setup working,
>> > > > >
>> > > > > Any help would be appreciated,
>> > > > >
>> > > > > Thanks,
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Wed, Aug 15, 2018 at 2:15 PM, Matthew Knepley
>> <[email protected] <mailto:[email protected]>> wrote:
>> > > > > On Wed, Aug 15, 2018 at 4:53 PM Manuel Valera
>> <[email protected] <mailto:[email protected]>> wrote:
>> > > > > Thanks Matthew,
>> > > > >
>> > > > > I try to do that when calling:
>> > > > >
>> > > > > call
>> MatMPIAIJSetPreallocation(A,19,PETSC_NULL_INTEGER,19,PETSC_
>> NULL_INTEGER,ierr)
>> > > > >
>> > > > > But i am not aware on how to do this for the DM if it
>> needs something more specific/different,
>> > > > >
>> > > > > The error says that your preallocation is wrong for the
>> values you are putting in. The DM does not control either,
>> > > > > so I do not understand your email.
>> > > > >
>> > > > > Thanks,
>> > > > >
>> > > > > Matt
>> > > > >
>> > > > > Thanks,
>> > > > >
>> > > > > On Wed, Aug 15, 2018 at 1:51 PM, Matthew Knepley
>> <[email protected] <mailto:[email protected]>> wrote:
>> > > > > On Wed, Aug 15, 2018 at 4:39 PM Manuel Valera
>> <[email protected] <mailto:[email protected]>> wrote:
>> > > > > Hello PETSc devs,
>> > > > >
>> > > > > I am running into an error when trying to use the
>> MATMPIAIJVIENNACL Matrix type in MPI calls, the same code runs
>> for MATSEQAIJVIENNACL type in one processor. The error happens
>> when calling MatSetValues for this specific configuration. It
>> does not occur when using MPI DMMatrix types only.
>> > > > >
>> > > > > The DM properly preallocates the matrix. I am assuming
>> you do not here.
>> > > > >
>> > > > > Matt
>> > > > >
>> > > > > Any help will be appreciated,
>> > > > >
>> > > > > Thanks,
>> > > > >
>> > > > >
>> > > > >
>> > > > > My program call:
>> > > > >
>> > > > > mpirun -n 2 ./gcmLEP.GPU
>> tc=TestCases/LockRelease/LE_6x6x6/ jid=tiny_cuda_test_n2
>> -ksp_type cg -dm_vec_type viennacl -dm_mat_type aijviennacl
>> -pc_type saviennacl -log_view
>> > > > >
>> > > > >
>> > > > > The error (repeats after each MatSetValues call):
>> > > > >
>> > > > > [1]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> > > > > [1]PETSC ERROR: Argument out of range
>> > > > > [1]PETSC ERROR: Inserting a new nonzero at global
>> row/column (75, 50) into matrix
>> > > > > [1]PETSC ERROR: See
>> http://www.mcs.anl.gov/petsc/documentation/faq.html
>> <http://www.mcs.anl.gov/petsc/documentation/faq.html> for
>> trouble shooting.
>> > > > > [1]PETSC ERROR: Petsc Development GIT revision:
>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300
>> > > > > [1]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named
>> node50 by valera Wed Aug 15 13:10:44 2018
>> > > > > [1]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug
>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2
>> --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 --with-shared-libraries=1
>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60
>> --with-blaslapack-dir=/usr/lib64 --download-viennacl
>> > > > > [1]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 608 in
>> /home/valera/petsc/src/mat/impls/aij/mpi/mpiaij.c
>> > > > > [1]PETSC ERROR: #2 MatSetValues() line 1339 in
>> /home/valera/petsc/src/mat/interface/matrix.c
>> > > > >
>> > > > >
>> > > > > My Code structure:
>> > > > >
>> > > > > call DMCreateMatrix(daDummy,A,ierr)
>> > > > > call MatSetFromOptions(A,ierr)
>> > > > > call MPI_Comm_size(PETSC_COMM_WORLD, numprocs, ierr)
>> > > > > if (numprocs > 1) then ! set matrix type parallel
>> > > > > ! Get local size
>> > > > > call DMDACreateNaturalVector(daDummy,Tmpnat,ierr)
>> > > > > call VecGetLocalSize(Tmpnat,locsize,ierr)
>> > > > > call VecDestroy(Tmpnat,ierr)
>> > > > > ! Set matrix
>> > > > > #ifdef GPU
>> > > > > call MatSetType(A,MATAIJVIENNACL,ierr)
>> > > > > call DMSetMatType(daDummy,MATMPIAIJVIENNACL,ierr)
>> > > > > call DMSetVecType(daDummy,VECMPIVIENNACL,ierr)
>> > > > > print*,'SETTING GPU TYPES'
>> > > > > #else
>> > > > > call DMSetMatType(daDummy,MATMPIAIJ,ierr)
>> > > > > call DMSetMatType(daDummy,VECMPI,ierr)
>> > > > > call MatSetType(A,MATMPIAIJ,ierr)!
>> > > > > #endif
>> > > > > call
>> MatMPIAIJSetPreallocation(A,19,PETSC_NULL_INTEGER,19,PETSC_
>> NULL_INTEGER,ierr)
>> > > > > else ! set matrix type sequential
>> > > > > #ifdef GPU
>> > > > > call DMSetMatType(daDummy,MATSEQAIJVIENNACL,ierr)
>> > > > > call DMSetVecType(daDummy,VECSEQVIENNACL,ierr)
>> > > > > call MatSetType(A,MATSEQAIJVIENNACL,ierr)
>> > > > > print*,'SETTING GPU TYPES'
>> > > > > #else
>> > > > > call DMSetMatType(daDummy,MATSEQAIJ,ierr)
>> > > > > call DMSetMatType(daDummy,VECSEQ,ierr)
>> > > > > call MatSetType(A,MATSEQAIJ,ierr)
>> > > > > #endif
>> > > > > call MatSetUp(A,ierr)
>> > > > > call
>> getCenterInfo(daGrid,xstart,ystart,zstart,xend,yend,zend)
>> > > > >
>> > > > > do k=zstart,zend-1
>> > > > > do j=ystart,yend-1
>> > > > > do i=xstart,xend-1
>> > > > > [..]
>> > > > > call
>> MatSetValues(A,1,row,sumpos,pos(0:iter-1),vals(0:iter-1),INS
>> ERT_VALUES,ierr)
>> > > > > [..]
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > What most experimenters take for granted before they
>> begin their experiments is infinitely more interesting than any
>> results to which their experiments lead.
>> > > > > -- Norbert Wiener
>> > > > >
>> > > > > https://www.cse.buffalo.edu/~knepley/
>> <https://www.cse.buffalo.edu/%7Eknepley/>
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > What most experimenters take for granted before they
>> begin their experiments is infinitely more interesting than any
>> results to which their experiments lead.
>> > > > > -- Norbert Wiener
>> > > > >
>> > > > > https://www.cse.buffalo.edu/~knepley/
>> <https://www.cse.buffalo.edu/%7Eknepley/>
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > What most experimenters take for granted before they
>> begin their experiments is infinitely more interesting than any
>> results to which their experiments lead.
>> > > > > -- Norbert Wiener
>> > > > >
>> > > > > https://www.cse.buffalo.edu/~knepley/
>> <https://www.cse.buffalo.edu/%7Eknepley/>
>> > > > >
>> > > >
>> > > >
>> > >
>> > >
>> >
>> >
>>
>>
>>
>>
Starting program: /usr/scratch/valera/ParGCCOM/./gcmLEP.GPU tc=TestCases/LockRelease/LE_6x6x6/
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib64/libthread_db.so.1".
[New Thread 0x2aaac822d700 (LWP 32604)]
[New Thread 0x2aaac9882700 (LWP 32607)]
[New Thread 0x2aaac9a83700 (LWP 32611)]
[New Thread 0x2aaacd3ed700 (LWP 32621)]
[New Thread 0x2aaadd932700 (LWP 32627)]
Program received signal SIGABRT, Aborted.
0x00002aaab10f91f7 in raise () from /usr/lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install blas-3.4.2-8.el7.x86_64 cyrus-sasl-lib-2.1.26-20.el7_2.x86_64 glibc-2.17-196.el7_4.2.x86_64 hdf5-1.8.12-7.el7.x86_64 hwloc-libs-1.7-5.el7.x86_64 infinipath-psm-3.3-0.g6f42cdb1bb8.2.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.13.2-12.el7_2.x86_64 lapack-3.4.2-8.el7.x86_64 libX11-1.6.5-1.el7.x86_64 libXau-1.0.8-2.1.el7.x86_64 libcom_err-1.42.9-7.el7.x86_64 libcurl-7.29.0-25.el7.centos.x86_64 libcxgb3-1.3.1-8.el7.x86_64 libcxgb4-1.3.5-3.el7.x86_64 libfabric-1.1.0-2.el7.x86_64 libgcc-4.8.5-16.el7_4.1.x86_64 libgfortran-4.8.5-16.el7_4.1.x86_64 libibumad-1.3.10.2-1.el7.x86_64 libibverbs-1.1.8-8.el7.x86_64 libidn-1.28-4.el7.x86_64 libipathverbs-1.3-2.el7.x86_64 libjpeg-turbo-1.2.90-5.el7.x86_64 libmlx4-1.0.6-5.el7.x86_64 libmlx5-1.0.2-1.el7.x86_64 libmthca-1.0.6-13.el7.x86_64 libnes-1.1.4-2.el7.x86_64 libnl3-3.2.28-4.el7.x86_64 libpciaccess-0.13.4-2.el7.x86_64 libquadmath-4.8.5-16.el7_4.1.x86_64 librdmacm-1.0.21-1.el7.x86_64 libselinux-2.2.2-6.el7.x86_64 libssh2-1.4.3-10.el7_2.1.x86_64 libstdc++-4.8.5-16.el7_4.1.x86_64 libxcb-1.12-1.el7.x86_64 libxml2-2.9.1-6.el7_2.2.x86_64 netcdf-4.3.3.1-5.el7.x86_64 netcdf-fortran-4.2-16.el7.x86_64 nspr-4.11.0-1.el7_2.x86_64 nss-3.21.0-9.el7_2.x86_64 nss-softokn-freebl-3.16.2.3-14.2.el7_2.x86_64 nss-util-3.21.0-2.2.el7_2.x86_64 numactl-libs-2.0.9-6.el7_2.x86_64 openldap-2.4.40-9.el7_2.x86_64 openmpi-1.10.0-10.el7.x86_64 opensm-libs-3.3.19-1.el7.x86_64 openssl-libs-1.0.1e-51.el7_2.4.x86_64 pcre-8.32-15.el7.x86_64 xz-libs-5.1.2-12alpha.el7.x86_64 zlib-1.2.7-15.el7.x86_64
#0 0x00002aaab10f91f7 in raise () from /usr/lib64/libc.so.6
#1 0x00002aaab10fa8e8 in abort () from /usr/lib64/libc.so.6
#2 0x00002aaaaaeb92ab in PetscTraceBackErrorHandler ()
from /home/valera/petsc/cuda-debug/lib/libpetsc.so.3.09
#3 0x00002aaaaaeb4cc1 in PetscError ()
from /home/valera/petsc/cuda-debug/lib/libpetsc.so.3.09
#4 0x00002aaaab852849 in PCSetUp_SAVIENNACL (pc=0x16d0780)
at /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu:47
#5 0x00002aaaaba10030 in PCSetUp ()
from /home/valera/petsc/cuda-debug/lib/libpetsc.so.3.09
#6 0x00002aaaaba1b08c in pcsetup_ ()
from /home/valera/petsc/cuda-debug/lib/libpetsc.so.3.09
#7 0x000000000051c3d5 in matrixobjs::setupmatrixobjs () at matrixobjs.f90:69
#8 0x0000000000454957 in dmdalaplacian () at DMDALaplacian.f90:349
#9 0x00000000005142ae in MAIN__ () at Main.f90:145
#0 0x00002aaab10f91f7 in raise () from /usr/lib64/libc.so.6
#1 0x00002aaab10fa8e8 in abort () from /usr/lib64/libc.so.6
#2 0x00002aaaaaeb92ab in PetscTraceBackErrorHandler ()
from /home/valera/petsc/cuda-debug/lib/libpetsc.so.3.09
#3 0x00002aaaaaeb4cc1 in PetscError ()
from /home/valera/petsc/cuda-debug/lib/libpetsc.so.3.09
#4 0x00002aaaab852849 in PCSetUp_SAVIENNACL (pc=0x16d0780)
at /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu:47
#5 0x00002aaaaba10030 in PCSetUp ()
from /home/valera/petsc/cuda-debug/lib/libpetsc.so.3.09
#6 0x00002aaaaba1b08c in pcsetup_ ()
from /home/valera/petsc/cuda-debug/lib/libpetsc.so.3.09
#7 0x000000000051c3d5 in matrixobjs::setupmatrixobjs () at matrixobjs.f90:69
#8 0x0000000000454957 in dmdalaplacian () at DMDALaplacian.f90:349
#9 0x00000000005142ae in MAIN__ () at Main.f90:145
Continuing.
Program received signal SIGSEGV, Segmentation fault.
0x00002aaaca630060 in ?? ()
No stack.
Starting program: /usr/scratch/valera/ParGCCOM/./gcmLEP.GPU tc=TestCases/LockRelease/LE_6x6x6/
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib64/libthread_db.so.1".
[New Thread 0x2aaac822d700 (LWP 32593)]
[New Thread 0x2aaac9882700 (LWP 32596)]
[New Thread 0x2aaac9a83700 (LWP 32612)]
[New Thread 0x2aaacd3ed700 (LWP 32622)]
Detaching after fork from child process 32623.
Detaching after fork from child process 32624.
Detaching after fork from child process 32625.
Detaching after fork from child process 32626.
[New Thread 0x2aaadd932700 (LWP 32628)]
Program received signal SIGINT, Interrupt.
0x00002aaac0aeaecd in opal_progress ()
from /usr/lib64/openmpi/lib/libopen-pal.so.13
Missing separate debuginfos, use: debuginfo-install blas-3.4.2-8.el7.x86_64 cyrus-sasl-lib-2.1.26-20.el7_2.x86_64 glibc-2.17-196.el7_4.2.x86_64 hdf5-1.8.12-7.el7.x86_64 hwloc-libs-1.7-5.el7.x86_64 infinipath-psm-3.3-0.g6f42cdb1bb8.2.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.13.2-12.el7_2.x86_64 lapack-3.4.2-8.el7.x86_64 libX11-1.6.5-1.el7.x86_64 libXau-1.0.8-2.1.el7.x86_64 libcom_err-1.42.9-7.el7.x86_64 libcurl-7.29.0-25.el7.centos.x86_64 libcxgb3-1.3.1-8.el7.x86_64 libcxgb4-1.3.5-3.el7.x86_64 libfabric-1.1.0-2.el7.x86_64 libgcc-4.8.5-16.el7_4.1.x86_64 libgfortran-4.8.5-16.el7_4.1.x86_64 libibumad-1.3.10.2-1.el7.x86_64 libibverbs-1.1.8-8.el7.x86_64 libidn-1.28-4.el7.x86_64 libipathverbs-1.3-2.el7.x86_64 libjpeg-turbo-1.2.90-5.el7.x86_64 libmlx4-1.0.6-5.el7.x86_64 libmlx5-1.0.2-1.el7.x86_64 libmthca-1.0.6-13.el7.x86_64 libnes-1.1.4-2.el7.x86_64 libnl3-3.2.28-4.el7.x86_64 libpciaccess-0.13.4-2.el7.x86_64 libquadmath-4.8.5-16.el7_4.1.x86_64 librdmacm-1.0.21-1.el7.x86_64 libselinux-2.2.2-6.el7.x86_64 libssh2-1.4.3-10.el7_2.1.x86_64 libstdc++-4.8.5-16.el7_4.1.x86_64 libxcb-1.12-1.el7.x86_64 libxml2-2.9.1-6.el7_2.2.x86_64 netcdf-4.3.3.1-5.el7.x86_64 netcdf-fortran-4.2-16.el7.x86_64 nspr-4.11.0-1.el7_2.x86_64 nss-3.21.0-9.el7_2.x86_64 nss-softokn-freebl-3.16.2.3-14.2.el7_2.x86_64 nss-util-3.21.0-2.2.el7_2.x86_64 numactl-libs-2.0.9-6.el7_2.x86_64 openldap-2.4.40-9.el7_2.x86_64 openmpi-1.10.0-10.el7.x86_64 opensm-libs-3.3.19-1.el7.x86_64 openssl-libs-1.0.1e-51.el7_2.4.x86_64 pcre-8.32-15.el7.x86_64 xz-libs-5.1.2-12alpha.el7.x86_64 zlib-1.2.7-15.el7.x86_64
#0 0x00002aaac0aeaecd in opal_progress ()
from /usr/lib64/openmpi/lib/libopen-pal.so.13
#1 0x00002aaaac6df565 in ompi_request_default_wait_all ()
from /usr/lib64/openmpi/lib/libmpi.so.12
#2 0x00002aaacc7a54c1 in ompi_coll_tuned_barrier_intra_two_procs ()
from /usr/lib64/openmpi/lib/openmpi/mca_coll_tuned.so
#3 0x00002aaaac6f3606 in PMPI_Barrier ()
from /usr/lib64/openmpi/lib/libmpi.so.12
#4 0x00002aaaaadb4c9d in PetscCommDuplicate ()
from /home/valera/petsc/cuda-debug/lib/libpetsc.so.3.09
#5 0x00002aaaaadb99c6 in PetscHeaderCreate_Private ()
from /home/valera/petsc/cuda-debug/lib/libpetsc.so.3.09
#6 0x00002aaaaafb88fc in ISCreate ()
from /home/valera/petsc/cuda-debug/lib/libpetsc.so.3.09
#7 0x00002aaaaafaed24 in ISCreateGeneral ()
from /home/valera/petsc/cuda-debug/lib/libpetsc.so.3.09
#8 0x00002aaaaafafed6 in iscreategeneral_ ()
from /home/valera/petsc/cuda-debug/lib/libpetsc.so.3.09
#9 0x000000000051c8d7 in matrixobjs::setupmatrixobjs () at matrixobjs.f90:88
#10 0x0000000000454957 in dmdalaplacian () at DMDALaplacian.f90:349
#11 0x00000000005142ae in MAIN__ () at Main.f90:145
Continuing.
Program received signal SIGINT, Interrupt.
0x00002aaaca83ea7b in mca_btl_vader_component_progress ()
from /usr/lib64/openmpi/lib/openmpi/mca_btl_vader.so
