I understand that Matthew, thanks, I am trying to see if i get any speedup in this configuration, otherwise i will be waiting for the MPIAIJ support on PETSc,
Is there any other gpu preconditioner that works on parallel matrices directly? Thanks again, On Wed, Aug 29, 2018 at 3:05 PM, Matthew Knepley <[email protected]> wrote: > On Wed, Aug 29, 2018 at 5:49 PM Manuel Valera <[email protected]> wrote: > >> Update: >> >> I made it work like you suggested Barry, i had to comment the code line >> to set up the pc_type saviennacl, that way i am getting as ksp_view: >> >> KSP Object: 2 MPI processes >> type: cg >> maximum iterations=10000, initial guess is zero >> tolerances: relative=1e-07, absolute=1e-50, divergence=10000. >> left preconditioning >> using PRECONDITIONED norm type for convergence test >> PC Object: 2 MPI processes >> type: bjacobi >> number of blocks = 2 >> Local solve is same for all blocks, in the following KSP and PC >> objects: >> KSP Object: (sub_) 1 MPI processes >> type: preonly >> maximum iterations=10000, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000. >> left preconditioning >> using NONE norm type for convergence test >> PC Object: (sub_) 1 MPI processes >> type: ilu >> out-of-place factorization >> 0 levels of fill >> tolerance for zero pivot 2.22045e-14 >> matrix ordering: natural >> factor fill ratio given 1., needed 1. >> Factored matrix follows: >> Mat Object: 1 MPI processes >> type: seqaij >> rows=75, cols=75 >> package used to perform factorization: petsc >> total: nonzeros=927, allocated nonzeros=927 >> total number of mallocs used during MatSetValues calls =0 >> not using I-node routines >> linear system matrix = precond matrix: >> Mat Object: 1 MPI processes >> type: seqaijviennacl >> rows=75, cols=75 >> total: nonzeros=927, allocated nonzeros=1335 >> total number of mallocs used during MatSetValues calls =84 >> not using I-node routines >> linear system matrix = precond matrix: >> Mat Object: 2 MPI processes >> type: mpiaijviennacl >> rows=125, cols=125 >> total: nonzeros=1685, allocated nonzeros=2885 >> total number of mallocs used during MatSetValues calls =184 >> has attached near null space >> Press_Max: 1.9995807630564922E-004 >> >> >> This is what we want, right? >> >> Still if i try using pc_type saviennacl directly it breaks, >> > > You are missing the point of the error message. > > We do not support SAVIENNACL on parallel matrices (MPIAIJ). We do support > it > on serial matrices (SEQAIJ). When you are setting in directly, you are > applying it > to a parallel matrix. If you use Block-Jacobi, you are applying it to each > submatrix, > and those submatrices are all serial. > > Thanks, > > Matt > > >> Thanks for your help i'll try my speedup like this, >> >> >> >> >> >> On Wed, Aug 29, 2018 at 2:21 PM, Manuel Valera <[email protected]> >> wrote: >> >>> Ok, executing with: >>> >>> mpirun -n 2 ./gcmLEP.GPU tc=TestCases/LockRelease/LE_6x6x6/ >>> jid=tiny_cuda_test_n1 -pc_type bjacobi -pc_sub_type saviennacl -ksp_view >>> >>> >>> I get: >>> >>> >>> SETTING GPU TYPES >>> Matrix type: mpiaijviennacl >>> >>> Of sizes: 125 x 125 >>> Matrix type: mpiaijviennacl >>> >>> Of sizes: 125 x 125 >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: No support for this operation for this object type >>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9.3-1264-g6fb0cd4 >>> GIT Date: 2018-08-29 12:42:36 -0600 >>> [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera Wed >>> Aug 29 14:20:42 2018 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --with-cc=mpicc >>> --with-cxx=mpic++ --with-fc=mpifort --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2 >>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 >>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 >>> --download-viennacl --download-cusp >>> [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>> [0]PETSC ERROR: #2 PCSetUp() line 932 in /home/valera/petsc/src/ksp/pc/ >>> interface/precon.c >>> [1]PETSC ERROR: ------------------------------ >>> ------------------------------------------ >>> [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >>> probably memory access out of range >>> [1]PETSC ERROR: Try option -start_in_debugger or >>> -on_error_attach_debugger >>> [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ >>> documentation/faq.html#valgrind >>> [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac >>> OS X to find memory corruption errors >>> [1]PETSC ERROR: likely location of problem given in stack below >>> [1]PETSC ERROR: --------------------- Stack Frames >>> ------------------------------------ >>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>> available, >>> [1]PETSC ERROR: INSTEAD the line number of the start of the >>> function >>> [1]PETSC ERROR: is given. >>> [1]PETSC ERROR: [1] PetscTraceBackErrorHandler line 182 >>> /home/valera/petsc/src/sys/error/errtrace.c >>> [1]PETSC ERROR: [1] PetscError line 352 /home/valera/petsc/src/sys/ >>> error/err.c >>> [1]PETSC ERROR: [1] PCSetUp_SAVIENNACL line 45 >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>> [1]PETSC ERROR: [1] PCSetUp line 894 /home/valera/petsc/src/ksp/pc/ >>> interface/precon.c >>> [1]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [1]PETSC ERROR: Signal received >>> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [1]PETSC ERROR: Petsc Development GIT revision: v3.9.3-1264-g6fb0cd4 >>> GIT Date: 2018-08-29 12:42:36 -0600 >>> [1]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera Wed >>> Aug 29 14:20:42 2018 >>> [1]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --with-cc=mpicc >>> --with-cxx=mpic++ --with-fc=mpifort --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2 >>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 >>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 >>> --download-viennacl --download-cusp >>> [1]PETSC ERROR: #1 User provided function() line 0 in unknown file >>> ------------------------------------------------------------ >>> -------------- >>> MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD >>> with errorcode 59. >>> >>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>> You may or may not see output from other processes, depending on >>> exactly when Open MPI kills them. >>> ------------------------------------------------------------ >>> -------------- >>> [0]PETSC ERROR: ------------------------------ >>> ------------------------------------------ >>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the >>> batch system) has told this process to end >>> [0]PETSC ERROR: Try option -start_in_debugger or >>> -on_error_attach_debugger >>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ >>> documentation/faq.html#valgrind >>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac >>> OS X to find memory corruption errors >>> [0]PETSC ERROR: likely location of problem given in stack below >>> [0]PETSC ERROR: --------------------- Stack Frames >>> ------------------------------------ >>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>> available, >>> [0]PETSC ERROR: INSTEAD the line number of the start of the >>> function >>> [0]PETSC ERROR: is given. >>> [0]PETSC ERROR: [0] PetscCommDuplicate line 130 >>> /home/valera/petsc/src/sys/objects/tagm.c >>> [0]PETSC ERROR: [0] PetscHeaderCreate_Private line 41 >>> /home/valera/petsc/src/sys/objects/inherit.c >>> [0]PETSC ERROR: [0] ISCreate line 35 /home/valera/petsc/src/vec/is/ >>> is/interface/isreg.c >>> [0]PETSC ERROR: [0] ISCreateGeneral line 658 >>> /home/valera/petsc/src/vec/is/is/impls/general/general.c >>> [0]PETSC ERROR: [0] PCSetUp_SAVIENNACL line 45 >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>> [0]PETSC ERROR: [0] PCSetUp line 894 /home/valera/petsc/src/ksp/pc/ >>> interface/precon.c >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Signal received >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9.3-1264-g6fb0cd4 >>> GIT Date: 2018-08-29 12:42:36 -0600 >>> [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera Wed >>> Aug 29 14:20:42 2018 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --with-cc=mpicc >>> --with-cxx=mpic++ --with-fc=mpifort --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2 >>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 >>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 >>> --download-viennacl --download-cusp >>> [0]PETSC ERROR: #3 User provided function() line 0 in unknown file >>> [node50:77836] 1 more process has sent help message help-mpi-api.txt / >>> mpi-abort >>> [node50:77836] Set MCA parameter "orte_base_help_aggregate" to 0 to see >>> all help / error messages >>> >>> >>> >>> On Wed, Aug 29, 2018 at 2:02 PM, Smith, Barry F. <[email protected]> >>> wrote: >>> >>>> Please send complete error message >>>> >>>> >>>> > On Aug 29, 2018, at 3:40 PM, Manuel Valera <[email protected]> >>>> wrote: >>>> > >>>> > Yeah, no sorry, i get the same error with -pc_type bjacobi >>>> -sub_pc_type SAVIENNACL : "Currently only handles ViennaCL matrices" >>>> > >>>> > >>>> > Thanks and let me know of any progress on this issue, >>>> > >>>> > >>>> > >>>> > On Wed, Aug 29, 2018 at 1:37 PM, Manuel Valera <[email protected]> >>>> wrote: >>>> > Awesome, thanks! >>>> > >>>> > On Wed, Aug 29, 2018 at 1:29 PM, Smith, Barry F. <[email protected]> >>>> wrote: >>>> > >>>> > >>>> > > On Aug 29, 2018, at 3:26 PM, Manuel Valera <[email protected]> >>>> wrote: >>>> > > >>>> > > >>>> > > >>>> > > You may need to use just plain PCBJACOBI or PCASM for parallelism >>>> and then SAVIENNACL sequentially on each block. >>>> > > >>>> > > >>>> > > Can you elaborate a little in how to do this? do you mean i can >>>> only use viennacl in serial at the moment or this is a suggestion of a >>>> workaround on how to make it work? >>>> > >>>> > -pc_type bjacobi -sub_pc_type SAVIENNACL >>>> > >>>> > This will run in parallel and utilize the GPU for each block of >>>> the preconditioner. >>>> > >>>> > Barry >>>> > >>>> > > >>>> > > >>>> > > Thanks, >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > > On Aug 29, 2018, at 1:50 PM, Manuel Valera <[email protected]> >>>> wrote: >>>> > > > >>>> > > > Hi everyone, >>>> > > > >>>> > > > Thanks for your responses, i understand communicating on this way >>>> to this level of technicality can be hard, i still think we can work ways >>>> to solve this problem though, >>>> > > > >>>> > > > I can say the following at this point: >>>> > > > >>>> > > > • Program works without issues in any other non-gpu >>>> preconditioner, just by calling -pc_type ### . >>>> > > > • Program works correctly with ViennaCL preconditioners >>>> with mpirun -n 1, that is on one processor. >>>> > > > • Program breaks with every ViennaCL preconditioner when >>>> attempting any more than 1 processors. >>>> > > > • I haven't tried other GPU preconditioners because >>>> ViennaCL gave the best performance in 1 processor before. >>>> > > > • I have made sure the matrix type is mpiaijviennacl as it >>>> is printed right before the error log. >>>> > > > • I am attaching what i think are the backtrace logs of the >>>> run with two cores, as i have limited experience with this kind of >>>> debugging, and i cannot make sense of what they say, please let me know if >>>> you need me to do something else. >>>> > > > >>>> > > > Current options i can think of are giving you permission to clone >>>> my model repo, or for you to guide me trough the process remotely, >>>> > > > >>>> > > > Thanks, >>>> > > > >>>> > > > .-.-.-.- >>>> > > > >>>> > > > >>>> > > > The error i get is still: >>>> > > > >>>> > > > Matrix type: mpiaijviennacl >>>> >>>> > > > Of sizes: 125 x 125 >>>> > > > Matrix type: mpiaijviennacl >>>> >>>> > > > Of sizes: 125 x 125 >>>> > > > [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> > > > [0]PETSC ERROR: No support for this operation for this object type >>>> > > > [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>> > > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/ >>>> documentation/faq.html for trouble shooting. >>>> > > > [0]PETSC ERROR: Petsc Development GIT revision: >>>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300 >>>> > > > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by >>>> valera Wed Aug 29 11:43:25 2018 >>>> > > > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2 >>>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 >>>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 >>>> --download-viennacl >>>> > > > [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in >>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>> > > > [0]PETSC ERROR: #2 PCSetUp() line 932 in >>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>> > > > [0]PETSC ERROR: #3 KSPSetUp() line 381 in >>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>> > > > [1]PETSC ERROR: ------------------------------ >>>> ------------------------------------------ >>>> > > > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation >>>> Violation, probably memory access out of range >>>> > > > [1]PETSC ERROR: Try option -start_in_debugger or >>>> -on_error_attach_debugger >>>> > > > [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ >>>> documentation/faq.html#valgrind >>>> > > > [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and >>>> Apple Mac OS X to find memory corruption errors >>>> > > > [1]PETSC ERROR: likely location of problem given in stack below >>>> > > > [1]PETSC ERROR: --------------------- Stack Frames >>>> ------------------------------------ >>>> > > > [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>>> available, >>>> > > > [1]PETSC ERROR: INSTEAD the line number of the start of the >>>> function >>>> > > > [1]PETSC ERROR: is given. >>>> > > > [1]PETSC ERROR: [1] PetscTraceBackErrorHandler line 182 >>>> /home/valera/petsc/src/sys/error/errtrace.c >>>> > > > [1]PETSC ERROR: [1] PetscError line 352 >>>> /home/valera/petsc/src/sys/error/err.c >>>> > > > [1]PETSC ERROR: [1] PCSetUp_SAVIENNACL line 45 >>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>> > > > [1]PETSC ERROR: [1] PCSetUp line 894 >>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>> > > > [1]PETSC ERROR: [1] KSPSetUp line 294 /home/valera/petsc/src/ksp/ >>>> ksp/interface/itfunc.c >>>> > > > [1]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> > > > [1]PETSC ERROR: Signal received >>>> > > > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/ >>>> documentation/faq.html for trouble shooting. >>>> > > > [1]PETSC ERROR: Petsc Development GIT revision: >>>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300 >>>> > > > [1]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by >>>> valera Wed Aug 29 11:43:25 2018 >>>> > > > [1]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2 >>>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 >>>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 >>>> --download-viennacl >>>> > > > [1]PETSC ERROR: #1 User provided function() line 0 in unknown >>>> file >>>> > > > ------------------------------------------------------------ >>>> -------------- >>>> > > > MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD >>>> > > > with errorcode 59. >>>> > > > >>>> > > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI >>>> processes. >>>> > > > You may or may not see output from other processes, depending on >>>> > > > exactly when Open MPI kills them. >>>> > > > ------------------------------------------------------------ >>>> -------------- >>>> > > > [0]PETSC ERROR: ------------------------------ >>>> ------------------------------------------ >>>> > > > [0]PETSC ERROR: Caught signal number 15 Terminate: Some process >>>> (or the batch system) has told this process to end >>>> > > > [0]PETSC ERROR: Try option -start_in_debugger or >>>> -on_error_attach_debugger >>>> > > > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ >>>> documentation/faq.html#valgrind >>>> > > > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and >>>> Apple Mac OS X to find memory corruption errors >>>> > > > [0]PETSC ERROR: likely location of problem given in stack below >>>> > > > [0]PETSC ERROR: --------------------- Stack Frames >>>> ------------------------------------ >>>> > > > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>>> available, >>>> > > > [0]PETSC ERROR: INSTEAD the line number of the start of the >>>> function >>>> > > > [0]PETSC ERROR: is given. >>>> > > > [0]PETSC ERROR: [0] MatSetErrorIfFailure line 116 >>>> /home/valera/petsc/src/mat/utils/gcreate.c >>>> > > > [0]PETSC ERROR: [0] PCSetUp line 894 >>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>> > > > [0]PETSC ERROR: [0] PCSetUp_SAVIENNACL line 45 >>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>>> > > > [0]PETSC ERROR: [0] PCSetUp line 894 >>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>> > > > [0]PETSC ERROR: [0] KSPSetUp line 294 /home/valera/petsc/src/ksp/ >>>> ksp/interface/itfunc.c >>>> > > > [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> > > > [0]PETSC ERROR: Signal received >>>> > > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/ >>>> documentation/faq.html for trouble shooting. >>>> > > > [0]PETSC ERROR: Petsc Development GIT revision: >>>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300 >>>> > > > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by >>>> valera Wed Aug 29 11:43:25 2018 >>>> > > > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2 >>>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 >>>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 >>>> --download-viennacl >>>> > > > [0]PETSC ERROR: #4 User provided function() line 0 in unknown >>>> file >>>> > > > [node50:32783] 1 more process has sent help message >>>> help-mpi-api.txt / mpi-abort >>>> > > > [node50:32783] Set MCA parameter "orte_base_help_aggregate" to 0 >>>> to see all help / error messages >>>> > > > >>>> > > > >>>> > > > On Tue, Aug 28, 2018 at 9:34 PM, Karl Rupp <[email protected]> >>>> wrote: >>>> > > > Hi Manuel, >>>> > > > >>>> > > > as Barry said, it is hard for us to provide any help without >>>> having a more complete picture of what is going on. >>>> > > > >>>> > > > The error you report seems to come from the AMG preconditioner in >>>> ViennaCL. This can have many origins. Do other preconditioners run without >>>> error? Can you run in a debugger and provide a complete backtrace? >>>> > > > >>>> > > > Thanks and best regards, >>>> > > > Karli >>>> > > > >>>> > > > >>>> > > > On 08/29/2018 01:33 AM, Manuel Valera wrote: >>>> > > > Talked too fast, >>>> > > > >>>> > > > After fixing that problem, i tried more than one mpi processor >>>> and got the following: >>>> > > > >>>> > > > Matrix type: mpiaijviennacl >>>> > > > Of sizes: 125 x 125 >>>> > > > Matrix type: mpiaijviennacl >>>> > > > Of sizes: 125 x 125 >>>> > > > [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> > > > [0]PETSC ERROR: No support for this operation for this object type >>>> > > > [0]PETSC ERROR: Currently only handles ViennaCL matrices >>>> > > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/ >>>> documentation/faq.html for trouble shooting. >>>> > > > [0]PETSC ERROR: Petsc Development GIT revision: >>>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300 >>>> > > > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by >>>> valera Tue Aug 28 16:30:02 2018 >>>> > > > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2 >>>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 >>>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 >>>> --download-viennacl >>>> > > > [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in >>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu < >>>> http://saviennacl.cu> >>>> > > > [0]PETSC ERROR: #2 PCSetUp() line 932 in >>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>> > > > [1]PETSC ERROR: ------------------------------ >>>> ------------------------------------------ >>>> > > > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation >>>> Violation, probably memory access out of range >>>> > > > [1]PETSC ERROR: Try option -start_in_debugger or >>>> -on_error_attach_debugger >>>> > > > [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ >>>> documentation/faq.html#valgrind >>>> > > > [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and >>>> Apple Mac OS X to find memory corruption errors >>>> > > > [1]PETSC ERROR: likely location of problem given in stack below >>>> > > > [1]PETSC ERROR: --------------------- Stack Frames >>>> ------------------------------------ >>>> > > > [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>>> available, >>>> > > > [1]PETSC ERROR: INSTEAD the line number of the start of the >>>> function >>>> > > > [1]PETSC ERROR: is given. >>>> > > > [1]PETSC ERROR: [1] PetscTraceBackErrorHandler line 182 >>>> /home/valera/petsc/src/sys/error/errtrace.c >>>> > > > [1]PETSC ERROR: [1] PetscError line 352 >>>> /home/valera/petsc/src/sys/error/err.c >>>> > > > [1]PETSC ERROR: [1] PCSetUp_SAVIENNACL line 45 >>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu< >>>> http://saviennacl.cu> >>>> > > > >>>> > > > [1]PETSC ERROR: [1] PCSetUp line 894 >>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>> > > > [1]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> > > > [1]PETSC ERROR: Signal received >>>> > > > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/ >>>> documentation/faq.html for trouble shooting. >>>> > > > [1]PETSC ERROR: Petsc Development GIT revision: >>>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300 >>>> > > > [1]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by >>>> valera Tue Aug 28 16:30:02 2018 >>>> > > > [1]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2 >>>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 >>>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 >>>> --download-viennacl >>>> > > > [1]PETSC ERROR: #1 User provided function() line 0 in unknown >>>> file >>>> > > > ------------------------------------------------------------ >>>> -------------- >>>> > > > MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD >>>> > > > with errorcode 59. >>>> > > > >>>> > > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI >>>> processes. >>>> > > > You may or may not see output from other processes, depending on >>>> > > > exactly when Open MPI kills them. >>>> > > > ------------------------------------------------------------ >>>> -------------- >>>> > > > [0]PETSC ERROR: ------------------------------ >>>> ------------------------------------------ >>>> > > > [0]PETSC ERROR: Caught signal number 15 Terminate: Some process >>>> (or the batch system) has told this process to end >>>> > > > [0]PETSC ERROR: Try option -start_in_debugger or >>>> -on_error_attach_debugger >>>> > > > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ >>>> documentation/faq.html#valgrind >>>> > > > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and >>>> Apple Mac OS X to find memory corruption errors >>>> > > > [0]PETSC ERROR: likely location of problem given in stack below >>>> > > > [0]PETSC ERROR: --------------------- Stack Frames >>>> ------------------------------------ >>>> > > > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>>> available, >>>> > > > [0]PETSC ERROR: INSTEAD the line number of the start of the >>>> function >>>> > > > [0]PETSC ERROR: is given. >>>> > > > [0]PETSC ERROR: [0] PetscCommDuplicate line 130 >>>> /home/valera/petsc/src/sys/objects/tagm.c >>>> > > > [0]PETSC ERROR: [0] PetscHeaderCreate_Private line 34 >>>> /home/valera/petsc/src/sys/objects/inherit.c >>>> > > > [0]PETSC ERROR: [0] ISCreate line 35 >>>> /home/valera/petsc/src/vec/is/is/interface/isreg.c >>>> > > > [0]PETSC ERROR: [0] ISCreateGeneral line 668 >>>> /home/valera/petsc/src/vec/is/is/impls/general/general.c >>>> > > > [0]PETSC ERROR: [0] PCSetUp_SAVIENNACL line 45 >>>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu< >>>> http://saviennacl.cu> >>>> > > > [0]PETSC ERROR: [0] PCSetUp line 894 >>>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>>> > > > [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> > > > [0]PETSC ERROR: Signal received >>>> > > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/ >>>> documentation/faq.html for trouble shooting. >>>> > > > [0]PETSC ERROR: Petsc Development GIT revision: >>>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300 >>>> > > > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by >>>> valera Tue Aug 28 16:30:02 2018 >>>> > > > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2 >>>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 >>>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 >>>> --download-viennacl >>>> > > > [0]PETSC ERROR: #3 User provided function() line 0 in unknown >>>> file >>>> > > > [node50:30582] 1 more process has sent help message >>>> help-mpi-api.txt / mpi-abort >>>> > > > [node50:30582] Set MCA parameter "orte_base_help_aggregate" to 0 >>>> to see all help / error messages >>>> > > > >>>> > > > >>>> > > > >>>> > > > It is currently running in 1 mpi processor + GPU but i would like >>>> to call at least 16 mpi processors + GPU to do the rest of the data >>>> management who is not part of the main laplacian on the mpi and the >>>> laplacian solution on the GPU, is this currently possible? >>>> > > > >>>> > > > Thanks for your help, >>>> > > > >>>> > > > >>>> > > > >>>> > > > On Tue, Aug 28, 2018 at 4:21 PM, Manuel Valera < >>>> [email protected] <mailto:[email protected]>> wrote: >>>> > > > >>>> > > > Ok, i found the culprit and we can close this thread, >>>> > > > >>>> > > > The problem was a missing variable for setting the maximum >>>> columns, >>>> > > > which i deleted at some point without realizing. The error >>>> message >>>> > > > was too ambiguous to catch this so i had to compare with a >>>> previous >>>> > > > working version of the arguments of MatSetValues, it was >>>> evident then. >>>> > > > >>>> > > > Good news is that i can now set the values with the viennacl >>>> types too, >>>> > > > >>>> > > > Thanks for your kind help, >>>> > > > >>>> > > > Manuel >>>> > > > >>>> > > > On Tue, Aug 28, 2018 at 11:25 AM, Smith, Barry F. >>>> > > > <[email protected] <mailto:[email protected]>> wrote: >>>> > > > >>>> > > > >>>> > > > 1) PetscMalloc() is never valid or needed in Fortran >>>> > > > >>>> > > > 2) there is no reason to use >>>> DMSetMatrixPreallocateOnly() >>>> > > > just use DMCreateMatrix() assuming that using a DM (DMDA, >>>> > > > DMPLEX, etc) is suitable for your problem. >>>> > > > >>>> > > > At this end we are totally guessing at what you are >>>> doing >>>> > > > and so have little help we can provide. A simple, >>>> nonworking >>>> > > > code that tries to do what you would need would help us a >>>> great >>>> > > > deal in understanding that you are trying to do. >>>> > > > >>>> > > > Barry >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > > On Aug 28, 2018, at 1:18 PM, Manuel Valera >>>> > > > <[email protected] <mailto:[email protected]>> wrote: >>>> > > > > >>>> > > > > Matthew, PetscMalloc gives the same error, >>>> > > > > >>>> > > > > Barry, it would be very hard for me to get the code to >>>> a >>>> > > > minimum working example, i guess all i need to understand >>>> is how >>>> > > > to setup a DM matrix with DMSetMatrixPreallocateOnly() >>>> instead >>>> > > > of MatMPIAIJSetPreallocation() as we were doing before, >>>> is there >>>> > > > a simple example who does this in Fortran? >>>> > > > > >>>> > > > > Is the PetscMalloc call needed? is 'call >>>> > > > PetscMalloc(1,row,ierr)' a valid, compilable call to >>>> > > > PetscMalloc? what other reason may there be for this >>>> error to >>>> > > > happen ? >>>> > > > > >>>> > > > > Just remembering, that trying to setup the matrix with >>>> the >>>> > > > MatAIJSetPreallocation() brings up an error to >>>> acknowledge the >>>> > > > viennacl datatypes and that's why i'm trying to make this >>>> change >>>> > > > on your recommendation, >>>> > > > > >>>> > > > > Thanks for your help, >>>> > > > > >>>> > > > > >>>> > > > > >>>> > > > > >>>> > > > > >>>> > > > > On Mon, Aug 27, 2018 at 7:35 PM, Smith, Barry F. >>>> > > > <[email protected] <mailto:[email protected]>> wrote: >>>> > > > > >>>> > > > > Send your code in a way we can compile and run it; >>>> it must >>>> > > > be some simple issue that is hard to communicate in email. >>>> > > > > >>>> > > > > Barry >>>> > > > > >>>> > > > > >>>> > > > > > On Aug 27, 2018, at 5:51 PM, Manuel Valera >>>> > > > <[email protected] <mailto:[email protected]>> wrote: >>>> > > > > > >>>> > > > > > Hello everyone, >>>> > > > > > >>>> > > > > > I just had time to work on this again, and checked >>>> the code >>>> > > > for errors on the matrix entries, this is the exact code >>>> i was >>>> > > > using for creating the matrix without >>>> > > > DMSetMatrixPreallocateOnly, using >>>> MatMPIAIJSetPreallocation and >>>> > > > it worked that way, but trying this way i get the same >>>> 'Column >>>> > > > too large' error using any number at the column position >>>> of >>>> > > > MatSetValues, >>>> > > > > > >>>> > > > > > I have set up my code to print the column argument >>>> (n) of >>>> > > > MatSetValues and in this case is 7 (lower than 124), it >>>> still >>>> > > > gives error, even entering a specific number in the >>>> MatSetValues >>>> > > > column argument position gives the same error. >>>> > > > > > >>>> > > > > > So next i went back to ex.50 here: >>>> > > > http://www.mcs.anl.gov/petsc/ >>>> petsc-current/src/ts/examples/tutorials/ex50.c.html >>>> > > > <http://www.mcs.anl.gov/petsc/petsc-current/src/ts/ >>>> examples/tutorials/ex50.c.html> >>>> > > > and it has a very similar structure except the >>>> PetscMalloc1() >>>> > > > call, so i tried adding that and got: >>>> > > > > > >>>> > > > > > /home/valera/ParGCCOM/Src/DMDALaplacian.f90:114: >>>> undefined >>>> > > > reference to `petscmalloc1_' >>>> > > > > > >>>> > > > > > Any ideas on this behaviour? >>>> > > > > > >>>> > > > > > Thanks so much, >>>> > > > > > >>>> > > > > > >>>> > > > > > >>>> > > > > > >>>> > > > > > >>>> > > > > > >>>> > > > > > On Thu, Aug 16, 2018 at 11:20 AM, Smith, Barry F. >>>> > > > <[email protected] <mailto:[email protected]>> wrote: >>>> > > > > > >>>> > > > > > Column too large: col 10980 max 124 >>>> > > > > > >>>> > > > > > You need to check the code that is generating the >>>> matrix >>>> > > > entries. The matrix has 124 columns but you are >>>> attempting to >>>> > > > put a value at column 10980 >>>> > > > > > >>>> > > > > > Barry >>>> > > > > > >>>> > > > > > >>>> > > > > > > On Aug 15, 2018, at 9:44 PM, Manuel Valera >>>> > > > <[email protected] <mailto:[email protected]>> wrote: >>>> > > > > > > >>>> > > > > > > Thanks Matthew and Barry, >>>> > > > > > > >>>> > > > > > > Now my code looks like: >>>> > > > > > > >>>> > > > > > > call DMSetMatrixPreallocateOnly( >>>> daDummy,PETSC_TRUE,ierr) >>>> > > > > > > call DMSetMatType(daDummy,MATMPIAIJVIENNACL,ierr) >>>> > > > > > > call DMSetVecType(daDummy,VECMPIVIENNACL,ierr) >>>> > > > > > > call DMCreateMatrix(daDummy,A,ierr) >>>> > > > > > > call MatSetFromOptions(A,ierr) >>>> > > > > > > call MatSetUp(A,ierr) >>>> > > > > > > [...] >>>> > > > > > > call >>>> > > > MatSetValues(A,1,row,sumpos,pos(0:iter-1),vals(0:iter-1), >>>> INSERT_VALUES,ierr) >>>> > > > > > > [...] >>>> > > > > > > call MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY, ierr) >>>> > > > > > > call MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY, ierr) >>>> > > > > > > >>>> > > > > > > And i get a different error, now is: >>>> > > > > > > >>>> > > > > > > [0]PETSC ERROR: --------------------- Error Message >>>> > > > ----------------------------- >>>> --------------------------------- >>>> > > > > > > [0]PETSC ERROR: Argument out of range >>>> > > > > > > [0]PETSC ERROR: Column too large: col 10980 max 124 >>>> > > > > > > [0]PETSC ERROR: See >>>> > > > http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> > > > <http://www.mcs.anl.gov/petsc/documentation/faq.html> for >>>> > > > trouble shooting. >>>> > > > > > > [0]PETSC ERROR: Petsc Development GIT revision: >>>> > > > v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300 >>>> > > > > > > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named >>>> node50 >>>> > > > by valera Wed Aug 15 19:40:00 2018 >>>> > > > > > > [0]PETSC ERROR: Configure options >>>> PETSC_ARCH=cuda-debug >>>> > > > --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 >>>> > > > --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 >>>> --with-shared-libraries=1 >>>> > > > --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 >>>> > > > --with-blaslapack-dir=/usr/lib64 --download-viennacl >>>> > > > > > > [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line 442 >>>> in >>>> > > > /home/valera/petsc/src/mat/impls/aij/seq/aij.c >>>> > > > > > > [0]PETSC ERROR: #2 MatSetValues() line 1339 in >>>> > > > /home/valera/petsc/src/mat/interface/matrix.c >>>> > > > > > > >>>> > > > > > > >>>> > > > > > > Thanks again, >>>> > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > > > On Wed, Aug 15, 2018 at 7:02 PM, Smith, Barry F. >>>> > > > <[email protected] <mailto:[email protected]>> wrote: >>>> > > > > > > >>>> > > > > > > Should be >>>> > > > > > > >>>> > > > > > > call DMSetMatType(daDummy,MATMPIAIJVIENNACL,ierr) >>>> > > > > > > call DMSetVecType(daDummy,VECMPIVIENNACL,ierr) >>>> > > > > > > call DMCreateMatrix(daDummy,A,ierr) >>>> > > > > > > >>>> > > > > > > and remove the rest. You need to set the type of >>>> Mat >>>> > > > you want the DM to return BEFORE you create the matrix. >>>> > > > > > > >>>> > > > > > > Barry >>>> > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > > > > On Aug 15, 2018, at 4:45 PM, Manuel Valera >>>> > > > <[email protected] <mailto:[email protected]>> wrote: >>>> > > > > > > > >>>> > > > > > > > Ok thanks for clarifying that, i wasn't sure if >>>> there >>>> > > > were different types, >>>> > > > > > > > >>>> > > > > > > > Here is a stripped down version of my code, it >>>> seems >>>> > > > like the preallocation is working now since the matrix >>>> > > > population part is working without problem, but here it >>>> is for >>>> > > > illustration purposes: >>>> > > > > > > > >>>> > > > > > > > call DMSetMatrixPreallocateOnly( >>>> daDummy,PETSC_TRUE,ierr) >>>> > > > > > > > call DMCreateMatrix(daDummy,A,ierr) >>>> > > > > > > > call MatSetFromOptions(A,ierr) >>>> > > > > > > > call DMSetMatType(daDummy, >>>> MATMPIAIJVIENNACL,ierr) >>>> > > > > > > > call DMSetVecType(daDummy,VECMPIVIENNACL,ierr) >>>> > > > > > > > call >>>> > > > MatMPIAIJSetPreallocation(A,19,PETSC_NULL_INTEGER,19, >>>> PETSC_NULL_INTEGER,ierr) >>>> > > > > > > > call MatSetUp(A,ierr) >>>> > > > > > > > [...] >>>> > > > > > > > call >>>> > > > MatSetValues(A,1,row,sumpos,pos(0:iter-1),vals(0:iter-1), >>>> INSERT_VALUES,ierr) >>>> > > > > > > > [...] >>>> > > > > > > > call MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY, >>>> ierr) >>>> > > > > > > > call MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY, ierr) >>>> > > > > > > > >>>> > > > > > > > Adding the first line there did the trick, >>>> > > > > > > > >>>> > > > > > > > Now the problem seems to be the program is not >>>> > > > recognizing the matrix as ViennaCL type when i try with >>>> more >>>> > > > than one processor, i get now: >>>> > > > > > > > >>>> > > > > > > > [0]PETSC ERROR: --------------------- Error >>>> Message >>>> > > > ----------------------------- >>>> --------------------------------- >>>> > > > > > > > [0]PETSC ERROR: No support for this operation >>>> for this >>>> > > > object type >>>> > > > > > > > [0]PETSC ERROR: Currently only handles ViennaCL >>>> matrices >>>> > > > > > > > [0]PETSC ERROR: See >>>> > > > http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> > > > <http://www.mcs.anl.gov/petsc/documentation/faq.html> for >>>> > > > trouble shooting. >>>> > > > > > > > [0]PETSC ERROR: Petsc Development GIT revision: >>>> > > > v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300 >>>> > > > > > > > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug >>>> named >>>> > > > node50 by valera Wed Aug 15 14:44:22 2018 >>>> > > > > > > > [0]PETSC ERROR: Configure options >>>> PETSC_ARCH=cuda-debug >>>> > > > --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 >>>> > > > --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 >>>> --with-shared-libraries=1 >>>> > > > --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 >>>> > > > --with-blaslapack-dir=/usr/lib64 --download-viennacl >>>> > > > > > > > [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 >>>> in >>>> > > > /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/savien >>>> nacl.cu >>>> > > > <http://saviennacl.cu> >>>> > > > > > > > [0]PETSC ERROR: #2 PCSetUp() line 932 in >>>> > > > /home/valera/petsc/src/ksp/pc/interface/precon.c >>>> > > > > > > > [0]PETSC ERROR: #3 KSPSetUp() line 381 in >>>> > > > /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>> > > > > > > > >>>> > > > > > > > When running with: >>>> > > > > > > > >>>> > > > > > > > mpirun -n 1 ./gcmLEP.GPU >>>> > > > tc=TestCases/LockRelease/LE_6x6x6/ jid=tiny_cuda_test_n2 >>>> > > > -ksp_type cg -dm_vec_type viennacl -dm_mat_type >>>> aijviennacl >>>> > > > -pc_type saviennacl -log_view >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > Thanks, >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > On Wed, Aug 15, 2018 at 2:32 PM, Matthew Knepley >>>> > > > <[email protected] <mailto:[email protected]>> wrote: >>>> > > > > > > > On Wed, Aug 15, 2018 at 5:20 PM Manuel Valera >>>> > > > <[email protected] <mailto:[email protected]>> wrote: >>>> > > > > > > > It seems to be resumed on: I do not know how to >>>> > > > preallocate a DM Matrix correctly. >>>> > > > > > > > >>>> > > > > > > > There is only one matrix type, Mat. There are no >>>> > > > separate DM matrices. A DM can create a matrix for you >>>> > > > > > > > using DMCreateMatrix(), but that is a Mat and it >>>> is >>>> > > > preallocated correctly. I am not sure what you are doing. >>>> > > > > > > > >>>> > > > > > > > Thanks, >>>> > > > > > > > >>>> > > > > > > > Matt >>>> > > > > > > > >>>> > > > > > > > The interesting part is that it only breaks when >>>> i need >>>> > > > to populate a GPU matrix from MPI, so kudos on that, but >>>> it >>>> > > > seems i need to do better on my code to get this setup >>>> working, >>>> > > > > > > > >>>> > > > > > > > Any help would be appreciated, >>>> > > > > > > > >>>> > > > > > > > Thanks, >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > On Wed, Aug 15, 2018 at 2:15 PM, Matthew Knepley >>>> > > > <[email protected] <mailto:[email protected]>> wrote: >>>> > > > > > > > On Wed, Aug 15, 2018 at 4:53 PM Manuel Valera >>>> > > > <[email protected] <mailto:[email protected]>> wrote: >>>> > > > > > > > Thanks Matthew, >>>> > > > > > > > >>>> > > > > > > > I try to do that when calling: >>>> > > > > > > > >>>> > > > > > > > call >>>> > > > MatMPIAIJSetPreallocation(A,19,PETSC_NULL_INTEGER,19, >>>> PETSC_NULL_INTEGER,ierr) >>>> > > > > > > > >>>> > > > > > > > But i am not aware on how to do this for the DM >>>> if it >>>> > > > needs something more specific/different, >>>> > > > > > > > >>>> > > > > > > > The error says that your preallocation is wrong >>>> for the >>>> > > > values you are putting in. The DM does not control either, >>>> > > > > > > > so I do not understand your email. >>>> > > > > > > > >>>> > > > > > > > Thanks, >>>> > > > > > > > >>>> > > > > > > > Matt >>>> > > > > > > > >>>> > > > > > > > Thanks, >>>> > > > > > > > >>>> > > > > > > > On Wed, Aug 15, 2018 at 1:51 PM, Matthew Knepley >>>> > > > <[email protected] <mailto:[email protected]>> wrote: >>>> > > > > > > > On Wed, Aug 15, 2018 at 4:39 PM Manuel Valera >>>> > > > <[email protected] <mailto:[email protected]>> wrote: >>>> > > > > > > > Hello PETSc devs, >>>> > > > > > > > >>>> > > > > > > > I am running into an error when trying to use the >>>> > > > MATMPIAIJVIENNACL Matrix type in MPI calls, the same code >>>> runs >>>> > > > for MATSEQAIJVIENNACL type in one processor. The error >>>> happens >>>> > > > when calling MatSetValues for this specific >>>> configuration. It >>>> > > > does not occur when using MPI DMMatrix types only. >>>> > > > > > > > >>>> > > > > > > > The DM properly preallocates the matrix. I am >>>> assuming >>>> > > > you do not here. >>>> > > > > > > > >>>> > > > > > > > Matt >>>> > > > > > > > >>>> > > > > > > > Any help will be appreciated, >>>> > > > > > > > >>>> > > > > > > > Thanks, >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > My program call: >>>> > > > > > > > >>>> > > > > > > > mpirun -n 2 ./gcmLEP.GPU >>>> > > > tc=TestCases/LockRelease/LE_6x6x6/ jid=tiny_cuda_test_n2 >>>> > > > -ksp_type cg -dm_vec_type viennacl -dm_mat_type >>>> aijviennacl >>>> > > > -pc_type saviennacl -log_view >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > The error (repeats after each MatSetValues call): >>>> > > > > > > > >>>> > > > > > > > [1]PETSC ERROR: --------------------- Error >>>> Message >>>> > > > ----------------------------- >>>> --------------------------------- >>>> > > > > > > > [1]PETSC ERROR: Argument out of range >>>> > > > > > > > [1]PETSC ERROR: Inserting a new nonzero at global >>>> > > > row/column (75, 50) into matrix >>>> > > > > > > > [1]PETSC ERROR: See >>>> > > > http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> > > > <http://www.mcs.anl.gov/petsc/documentation/faq.html> for >>>> > > > trouble shooting. >>>> > > > > > > > [1]PETSC ERROR: Petsc Development GIT revision: >>>> > > > v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300 >>>> > > > > > > > [1]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug >>>> named >>>> > > > node50 by valera Wed Aug 15 13:10:44 2018 >>>> > > > > > > > [1]PETSC ERROR: Configure options >>>> PETSC_ARCH=cuda-debug >>>> > > > --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 >>>> > > > --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 >>>> --with-shared-libraries=1 >>>> > > > --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 >>>> > > > --with-blaslapack-dir=/usr/lib64 --download-viennacl >>>> > > > > > > > [1]PETSC ERROR: #1 MatSetValues_MPIAIJ() line >>>> 608 in >>>> > > > /home/valera/petsc/src/mat/impls/aij/mpi/mpiaij.c >>>> > > > > > > > [1]PETSC ERROR: #2 MatSetValues() line 1339 in >>>> > > > /home/valera/petsc/src/mat/interface/matrix.c >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > My Code structure: >>>> > > > > > > > >>>> > > > > > > > call DMCreateMatrix(daDummy,A,ierr) >>>> > > > > > > > call MatSetFromOptions(A,ierr) >>>> > > > > > > > call MPI_Comm_size(PETSC_COMM_WORLD, numprocs, >>>> ierr) >>>> > > > > > > > if (numprocs > 1) then ! set matrix type >>>> parallel >>>> > > > > > > > ! Get local size >>>> > > > > > > > call DMDACreateNaturalVector( >>>> daDummy,Tmpnat,ierr) >>>> > > > > > > > call VecGetLocalSize(Tmpnat,locsize,ierr) >>>> > > > > > > > call VecDestroy(Tmpnat,ierr) >>>> > > > > > > > ! Set matrix >>>> > > > > > > > #ifdef GPU >>>> > > > > > > > call MatSetType(A,MATAIJVIENNACL,ierr) >>>> > > > > > > > call DMSetMatType(daDummy, >>>> MATMPIAIJVIENNACL,ierr) >>>> > > > > > > > call DMSetVecType(daDummy, >>>> VECMPIVIENNACL,ierr) >>>> > > > > > > > print*,'SETTING GPU TYPES' >>>> > > > > > > > #else >>>> > > > > > > > call DMSetMatType(daDummy,MATMPIAIJ,ierr) >>>> > > > > > > > call DMSetMatType(daDummy,VECMPI,ierr) >>>> > > > > > > > call MatSetType(A,MATMPIAIJ,ierr)! >>>> > > > > > > > #endif >>>> > > > > > > > call >>>> > > > MatMPIAIJSetPreallocation(A,19,PETSC_NULL_INTEGER,19, >>>> PETSC_NULL_INTEGER,ierr) >>>> > > > > > > > else ! set matrix type >>>> sequential >>>> > > > > > > > #ifdef GPU >>>> > > > > > > > call DMSetMatType(daDummy, >>>> MATSEQAIJVIENNACL,ierr) >>>> > > > > > > > call DMSetVecType(daDummy, >>>> VECSEQVIENNACL,ierr) >>>> > > > > > > > call MatSetType(A,MATSEQAIJVIENNACL,ierr) >>>> > > > > > > > print*,'SETTING GPU TYPES' >>>> > > > > > > > #else >>>> > > > > > > > call DMSetMatType(daDummy,MATSEQAIJ,ierr) >>>> > > > > > > > call DMSetMatType(daDummy,VECSEQ,ierr) >>>> > > > > > > > call MatSetType(A,MATSEQAIJ,ierr) >>>> > > > > > > > #endif >>>> > > > > > > > call MatSetUp(A,ierr) >>>> > > > > > > > call >>>> > > > getCenterInfo(daGrid,xstart,ystart,zstart,xend,yend,zend) >>>> > > > > > > > >>>> > > > > > > > do k=zstart,zend-1 >>>> > > > > > > > do j=ystart,yend-1 >>>> > > > > > > > do i=xstart,xend-1 >>>> > > > > > > > [..] >>>> > > > > > > > call >>>> > > > MatSetValues(A,1,row,sumpos,pos(0:iter-1),vals(0:iter-1), >>>> INSERT_VALUES,ierr) >>>> > > > > > > > [..] >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > -- >>>> > > > > > > > What most experimenters take for granted before >>>> they >>>> > > > begin their experiments is infinitely more interesting >>>> than any >>>> > > > results to which their experiments lead. >>>> > > > > > > > -- Norbert Wiener >>>> > > > > > > > >>>> > > > > > > > https://www.cse.buffalo.edu/~knepley/ >>>> > > > <https://www.cse.buffalo.edu/%7Eknepley/> >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > -- >>>> > > > > > > > What most experimenters take for granted before >>>> they >>>> > > > begin their experiments is infinitely more interesting >>>> than any >>>> > > > results to which their experiments lead. >>>> > > > > > > > -- Norbert Wiener >>>> > > > > > > > >>>> > > > > > > > https://www.cse.buffalo.edu/~knepley/ >>>> > > > <https://www.cse.buffalo.edu/%7Eknepley/> >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > -- >>>> > > > > > > > What most experimenters take for granted before >>>> they >>>> > > > begin their experiments is infinitely more interesting >>>> than any >>>> > > > results to which their experiments lead. >>>> > > > > > > > -- Norbert Wiener >>>> > > > > > > > >>>> > > > > > > > https://www.cse.buffalo.edu/~knepley/ >>>> > > > <https://www.cse.buffalo.edu/%7Eknepley/> >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > > >>>> > > > > >>>> > > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > <backt-2.log><backt.log> >>>> > >>>> > >>>> > >>>> >>>> >>> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> >
