On Wed, Aug 15, 2018 at 4:39 PM Manuel Valera <[email protected]> wrote:
> Hello PETSc devs, > > I am running into an error when trying to use the MATMPIAIJVIENNACL Matrix > type in MPI calls, the same code runs for MATSEQAIJVIENNACL type in one > processor. The error happens when calling MatSetValues for this specific > configuration. It does not occur when using MPI DMMatrix types only. > The DM properly preallocates the matrix. I am assuming you do not here. Matt > Any help will be appreciated, > > Thanks, > > > > My program call: > > mpirun -n 2 ./gcmLEP.GPU tc=TestCases/LockRelease/LE_6x6x6/ > jid=tiny_cuda_test_n2 -ksp_type cg -dm_vec_type viennacl -dm_mat_type > aijviennacl -pc_type saviennacl -log_view > > > The error (repeats after each MatSetValues call): > > [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [1]PETSC ERROR: Argument out of range > [1]PETSC ERROR: Inserting a new nonzero at global row/column (75, 50) into > matrix > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [1]PETSC ERROR: Petsc Development GIT revision: v3.9.2-549-g779ab53 GIT > Date: 2018-05-31 17:31:13 +0300 > [1]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera Wed > Aug 15 13:10:44 2018 > [1]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug > --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2 > --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 > --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 --download-viennacl > [1]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 608 in > /home/valera/petsc/src/mat/impls/aij/mpi/mpiaij.c > [1]PETSC ERROR: #2 MatSetValues() line 1339 in > /home/valera/petsc/src/mat/interface/matrix.c > > > My Code structure: > > call DMCreateMatrix(daDummy,A,ierr) >> call MatSetFromOptions(A,ierr) >> call MPI_Comm_size(PETSC_COMM_WORLD, numprocs, ierr) >> if (numprocs > 1) then ! set matrix type parallel >> ! Get local size >> call DMDACreateNaturalVector(daDummy,Tmpnat,ierr) >> call VecGetLocalSize(Tmpnat,locsize,ierr) >> call VecDestroy(Tmpnat,ierr) >> ! Set matrix >> #ifdef GPU >> call MatSetType(A,MATAIJVIENNACL,ierr) >> call DMSetMatType(daDummy,MATMPIAIJVIENNACL,ierr) >> call DMSetVecType(daDummy,VECMPIVIENNACL,ierr) >> print*,'SETTING GPU TYPES' >> #else >> call DMSetMatType(daDummy,MATMPIAIJ,ierr) >> call DMSetMatType(daDummy,VECMPI,ierr) >> call MatSetType(A,MATMPIAIJ,ierr)! >> #endif >> call >> MatMPIAIJSetPreallocation(A,19,PETSC_NULL_INTEGER,19,PETSC_NULL_INTEGER,ierr) >> else ! set matrix type sequential >> #ifdef GPU >> call DMSetMatType(daDummy,MATSEQAIJVIENNACL,ierr) >> call DMSetVecType(daDummy,VECSEQVIENNACL,ierr) >> call MatSetType(A,MATSEQAIJVIENNACL,ierr) >> print*,'SETTING GPU TYPES' >> #else >> call DMSetMatType(daDummy,MATSEQAIJ,ierr) >> call DMSetMatType(daDummy,VECSEQ,ierr) >> call MatSetType(A,MATSEQAIJ,ierr) >> #endif >> call MatSetUp(A,ierr) >> call getCenterInfo(daGrid,xstart,ystart,zstart,xend,yend,zend) >> > > >> do k=zstart,zend-1 >> do j=ystart,yend-1 >> do i=xstart,xend-1 >> [..] >> call >> MatSetValues(A,1,row,sumpos,pos(0:iter-1),vals(0:iter-1),INSERT_VALUES,ierr) >> [..] > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/~mk51/>
