On Wed, Aug 15, 2018 at 5:45 PM Manuel Valera <[email protected]> wrote:
> Ok thanks for clarifying that, i wasn't sure if there were different types, > > Here is a stripped down version of my code, it seems like the > preallocation is working now since the matrix population part is working > without problem, but here it is for illustration purposes: > Good. > call DMSetMatrixPreallocateOnly(daDummy,PETSC_TRUE,ierr) >> call DMCreateMatrix(daDummy,A,ierr) >> call MatSetFromOptions(A,ierr) >> call DMSetMatType(daDummy,MATMPIAIJVIENNACL,ierr) >> call DMSetVecType(daDummy,VECMPIVIENNACL,ierr) >> > The two statements above should be unnecessary with the command line options you have. > call >> MatMPIAIJSetPreallocation(A,19,PETSC_NULL_INTEGER,19,PETSC_NULL_INTEGER,ierr) >> > This should not be necessary since the matrix you get from DMCreateMatrix() should be preallocated. > call MatSetUp(A,ierr) >> [...] >> call >> MatSetValues(A,1,row,sumpos,pos(0:iter-1),vals(0:iter-1),INSERT_VALUES,ierr) >> [...] >> call MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY, ierr) >> call MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY, ierr) > > > Adding the first line there did the trick, > > Now the problem seems to be the program is not recognizing the matrix as > ViennaCL type when i try with more than one processor, i get now: > This is a bad error message since it does not tell us the type of the matrix. You could look in the debugger to see what type the matrix is. I would get rid of all the extraneous statements above so we can figure out what is happening. Matt > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: No support for this operation for this object type > [0]PETSC ERROR: Currently only handles ViennaCL matrices > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.9.2-549-g779ab53 GIT > Date: 2018-05-31 17:31:13 +0300 > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera Wed > Aug 15 14:44:22 2018 > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug > --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2 > --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 > --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 --download-viennacl > [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in > /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu > [0]PETSC ERROR: #2 PCSetUp() line 932 in > /home/valera/petsc/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #3 KSPSetUp() line 381 in > /home/valera/petsc/src/ksp/ksp/interface/itfunc.c > > When running with: > > mpirun -n 1 ./gcmLEP.GPU tc=TestCases/LockRelease/LE_6x6x6/ > jid=tiny_cuda_test_n2 -ksp_type cg -dm_vec_type viennacl -dm_mat_type > aijviennacl -pc_type saviennacl -log_view > > > Thanks, > > > > > > > > > > > On Wed, Aug 15, 2018 at 2:32 PM, Matthew Knepley <[email protected]> > wrote: > >> On Wed, Aug 15, 2018 at 5:20 PM Manuel Valera <[email protected]> wrote: >> >>> It seems to be resumed on: I do not know how to preallocate a DM Matrix >>> correctly. >>> >> >> There is only one matrix type, Mat. There are no separate DM matrices. A >> DM can create a matrix for you >> using DMCreateMatrix(), but that is a Mat and it is preallocated >> correctly. I am not sure what you are doing. >> >> Thanks, >> >> Matt >> >> >>> The interesting part is that it only breaks when i need to populate a >>> GPU matrix from MPI, so kudos on that, but it seems i need to do better on >>> my code to get this setup working, >>> >>> Any help would be appreciated, >>> >>> Thanks, >>> >>> >>> >>> On Wed, Aug 15, 2018 at 2:15 PM, Matthew Knepley <[email protected]> >>> wrote: >>> >>>> On Wed, Aug 15, 2018 at 4:53 PM Manuel Valera <[email protected]> >>>> wrote: >>>> >>>>> Thanks Matthew, >>>>> >>>>> I try to do that when calling: >>>>> >>>>> call MatMPIAIJSetPreallocation(A,19,PETSC_NULL_INTEGER,19, >>>>> PETSC_NULL_INTEGER,ierr) >>>>> >>>>> But i am not aware on how to do this for the DM if it needs something >>>>> more specific/different, >>>>> >>>> >>>> The error says that your preallocation is wrong for the values you are >>>> putting in. The DM does not control either, >>>> so I do not understand your email. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Thanks, >>>>> >>>>> On Wed, Aug 15, 2018 at 1:51 PM, Matthew Knepley <[email protected]> >>>>> wrote: >>>>> >>>>>> On Wed, Aug 15, 2018 at 4:39 PM Manuel Valera <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hello PETSc devs, >>>>>>> >>>>>>> I am running into an error when trying to use the MATMPIAIJVIENNACL >>>>>>> Matrix type in MPI calls, the same code runs for MATSEQAIJVIENNACL type >>>>>>> in >>>>>>> one processor. The error happens when calling MatSetValues for this >>>>>>> specific configuration. It does not occur when using MPI DMMatrix types >>>>>>> only. >>>>>>> >>>>>> >>>>>> The DM properly preallocates the matrix. I am assuming you do not >>>>>> here. >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> Any help will be appreciated, >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> >>>>>>> >>>>>>> My program call: >>>>>>> >>>>>>> mpirun -n 2 ./gcmLEP.GPU tc=TestCases/LockRelease/LE_6x6x6/ >>>>>>> jid=tiny_cuda_test_n2 -ksp_type cg -dm_vec_type viennacl -dm_mat_type >>>>>>> aijviennacl -pc_type saviennacl -log_view >>>>>>> >>>>>>> >>>>>>> The error (repeats after each MatSetValues call): >>>>>>> >>>>>>> [1]PETSC ERROR: --------------------- Error Message >>>>>>> -------------------------------------------------------------- >>>>>>> [1]PETSC ERROR: Argument out of range >>>>>>> [1]PETSC ERROR: Inserting a new nonzero at global row/column (75, >>>>>>> 50) into matrix >>>>>>> [1]PETSC ERROR: See >>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >>>>>>> shooting. >>>>>>> [1]PETSC ERROR: Petsc Development GIT revision: v3.9.2-549-g779ab53 >>>>>>> GIT Date: 2018-05-31 17:31:13 +0300 >>>>>>> [1]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera >>>>>>> Wed Aug 15 13:10:44 2018 >>>>>>> [1]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>>>>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2 >>>>>>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 >>>>>>> --with-cuda=1 >>>>>>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 >>>>>>> --download-viennacl >>>>>>> [1]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 608 in >>>>>>> /home/valera/petsc/src/mat/impls/aij/mpi/mpiaij.c >>>>>>> [1]PETSC ERROR: #2 MatSetValues() line 1339 in >>>>>>> /home/valera/petsc/src/mat/interface/matrix.c >>>>>>> >>>>>>> >>>>>>> My Code structure: >>>>>>> >>>>>>> call DMCreateMatrix(daDummy,A,ierr) >>>>>>>> call MatSetFromOptions(A,ierr) >>>>>>>> call MPI_Comm_size(PETSC_COMM_WORLD, numprocs, ierr) >>>>>>>> if (numprocs > 1) then ! set matrix type parallel >>>>>>>> ! Get local size >>>>>>>> call DMDACreateNaturalVector(daDummy,Tmpnat,ierr) >>>>>>>> call VecGetLocalSize(Tmpnat,locsize,ierr) >>>>>>>> call VecDestroy(Tmpnat,ierr) >>>>>>>> ! Set matrix >>>>>>>> #ifdef GPU >>>>>>>> call MatSetType(A,MATAIJVIENNACL,ierr) >>>>>>>> call DMSetMatType(daDummy,MATMPIAIJVIENNACL,ierr) >>>>>>>> call DMSetVecType(daDummy,VECMPIVIENNACL,ierr) >>>>>>>> print*,'SETTING GPU TYPES' >>>>>>>> #else >>>>>>>> call DMSetMatType(daDummy,MATMPIAIJ,ierr) >>>>>>>> call DMSetMatType(daDummy,VECMPI,ierr) >>>>>>>> call MatSetType(A,MATMPIAIJ,ierr)! >>>>>>>> #endif >>>>>>>> call >>>>>>>> MatMPIAIJSetPreallocation(A,19,PETSC_NULL_INTEGER,19,PETSC_NULL_INTEGER,ierr) >>>>>>>> else ! set matrix type sequential >>>>>>>> #ifdef GPU >>>>>>>> call DMSetMatType(daDummy,MATSEQAIJVIENNACL,ierr) >>>>>>>> call DMSetVecType(daDummy,VECSEQVIENNACL,ierr) >>>>>>>> call MatSetType(A,MATSEQAIJVIENNACL,ierr) >>>>>>>> print*,'SETTING GPU TYPES' >>>>>>>> #else >>>>>>>> call DMSetMatType(daDummy,MATSEQAIJ,ierr) >>>>>>>> call DMSetMatType(daDummy,VECSEQ,ierr) >>>>>>>> call MatSetType(A,MATSEQAIJ,ierr) >>>>>>>> #endif >>>>>>>> call MatSetUp(A,ierr) >>>>>>>> call getCenterInfo(daGrid,xstart,ystart,zstart,xend,yend,zend) >>>>>>>> >>>>>>> >>>>>>> >>>>>>>> do k=zstart,zend-1 >>>>>>>> do j=ystart,yend-1 >>>>>>>> do i=xstart,xend-1 >>>>>>>> [..] >>>>>>>> call >>>>>>>> MatSetValues(A,1,row,sumpos,pos(0:iter-1),vals(0:iter-1),INSERT_VALUES,ierr) >>>>>>>> [..] >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which >>>>>> their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>> <http://www.caam.rice.edu/~mk51/> >>>>>> >>>>> >>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/~mk51/> >>>> >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/~mk51/> >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/~mk51/>
