The code is very slow for one of two reasons 1) the matrix preallocation is not correct and it is spending a great deal of time in the MatSetValues() calls. To check this run the code with -info and grep the output for malloc. My guess is that you are not preallocating for the diagonal and hence the preallocation is not enough. See http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#efficient-assembly
2) the convergence of the solver is very slow. Run with -ksp_monitor and see how many iterations it is taking to converge. Is it thousands? If so, you will need to select a better preconditioner or use a direct solver -pc_type lu. Barry On May 9, 2011, at 11:02 AM, Danesh Daroui wrote: > > Hi again, > > The issue is solved and it was due to a bug in my code. But now the > problem is that PETSc is extremely slow when I try to solve even a small > equation. I fill the matrix and right-hand-side and call the solver as > below: > > ierr=KSPCreate(PETSC_COMM_WORLD, &ksp); > ierr=KSPSetOperators(ksp, Mp, Mp, DIFFERENT_NONZERO_PATTERN); > ierr=KSPSetTolerances(ksp, 1.e-2/Msize, 1.e-50, PETSC_DEFAULT, > PETSC_DEFAULT); > ierr=KSPSetFromOptions(ksp); > ierr=KSPSolve(ksp, bp, xp); > > The Msize is the size of right-hand-side vector. I use C++ and am > running PETSc in sequential mode so no MPI is used, but anyway, it > shouldn't be that slow! Am I right or I am missing something? > > Thanks, > > Danesh > > > > > On Sun, 2011-05-08 at 14:41 +0200, dan at ltu.se wrote: >> Hi, >> >> I checked my code and there is no zero on the diagonal. I also modified the >> code >> so if there is any zero on the diagonal it will be stored anyway. After this >> change the problem still exists! I am not sure if I have called PETSc >> functions >> correctly or in correct sequence. But anyway, it returns the error: >> >> [0]PETSC ERROR: Object is in wrong state! >> >> >> that and the rest of the error message is exactly as I posted before. Any >> idea >> to solve this problem? >> >> Thanks, >> >> D. >> >> >> >> Quoting Barry Smith <bsmith at mcs.anl.gov>: >> >>> >>> Danesh, >>> >>> Some of the PETSc solvers (like the default ILU) require that you put >>> entries on all the diagonal locations of the matrix, even if they are zero. >>> So you should explicitly put a 0 on the diagonal locations that have 0 and >>> the issue will go away. >>> >>> Barry >>> >>> On May 6, 2011, at 1:15 PM, Danesh Daroui wrote: >>> >>>> >>>> Dear PETSc users, >>>> >>>> I have recently started to use PETSc in our code. As the first try, I >>>> assemble the matrix row by row and then put each non-zero element in >>>> each row into PETSc matrix. Then I start to solve the equation. I use >>>> PETSc on a multi-core machine, so I will not use MPI in my program. I >>>> compiled PETSc without MPI. The PETSc parts of my code are: >>>> >>>> >>>> Mat Mp; >>>> Vec xp, bp; >>>> KSP ksp; >>>> PetscErrorCode ierr; >>>> >>>> ierr=MatCreate(PETSC_COMM_WORLD, &Mp); >>>> ierr=MatSetSizes(Mp, Msize, Msize, Msize, Msize); >>>> ierr=MatSetType(Mp, MATSEQAIJ); >>>> ierr=MatSetUp(Mp); >>>> >>>> ierr=VecCreate(PETSC_COMM_WORLD, &xp); >>>> ierr=VecSetSizes(xp, Msize, Msize); >>>> ierr=VecSetFromOptions(xp); >>>> >>>> ierr=VecCreate(PETSC_COMM_WORLD, &bp); >>>> ierr=VecSetSizes(bp, Msize, Msize); >>>> ierr=VecSetFromOptions(bp); >>>> >>>> . >>>> . >>>> . >>>> >>>> // each non zero element of row "i" and column "k" is put in >>>> // PETSc matrix >>>> if (M_row[k]!=0.0) >>>> ierr=MatSetValues(Mp, 1, &i, 1, &k, &M_row[k], >>>> ADD_VALUES); >>>> >>>> >>>> ierr=MatAssemblyBegin(Mp, MAT_FINAL_ASSEMBLY); >>>> ierr=MatAssemblyEnd(Mp, MAT_FINAL_ASSEMBLY); >>>> >>>> . >>>> . >>>> . >>>> >>>> // copy right-hand-side to PETSc vector >>>> for (int i=0; i<Msize; ++i) >>>> VecSetValue(bp, i, b[i], INSERT_VALUES); >>>> >>>> VecAssemblyBegin(bp); >>>> VecAssemblyEnd(bp); >>>> >>>> ierr=KSPCreate(PETSC_COMM_WORLD, &ksp); >>>> ierr=KSPSetOperators(ksp, Mp, Mp, DIFFERENT_NONZERO_PATTERN); >>>> ierr=KSPSetTolerances(ksp, 1.e-2/Msize, 1.e-50, PETSC_DEFAULT, >>>> PETSC_DEFAULT); >>>> ierr=KSPSetFromOptions(ksp); >>>> ierr=KSPSolve(ksp, bp, xp); >>>> >>>> >>>> >>>> >>>> >>>> The problem is that whenever I run the code I get following output: >>>> >>>> >>>> >>>> >>>> [0]PETSC ERROR: --------------------- Error Message >>>> ------------------------------------ >>>> [0]PETSC ERROR: Object is in wrong state! >>>> [0]PETSC ERROR: Matrix is missing diagonal entry 309! >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 >>>> 13:37:48 CDT 2011 >>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: Unknown Name on a linux-gnu named linux-sclt.site by dan >>>> Fri May 6 19:34:19 2011 >>>> [0]PETSC ERROR: Libraries linked >>>> from /home/danesh/petsc-3.1-p8/linux-gnu-intel/lib >>>> [0]PETSC ERROR: Configure run at Fri May 6 15:50:26 2011 >>>> [0]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-intel >>>> --with-scalar-type=complex --with-clanguage=c++ --with-mpi=0 >>>> --with-cc=icc --with-fc=gfortran --with-cxx=icpc >>>> --with-blas-lapack-dir=/home/danesh/intel/mkl/lib/intel64/ >>>> [0]PETSC ERROR: >>>> ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: MatILUFactorSymbolic_SeqAIJ_ilu0() line 1627 in >>>> src/mat/impls/aij/seq/aijfact.c >>>> [0]PETSC ERROR: MatILUFactorSymbolic_SeqAIJ() line 1731 in >>>> src/mat/impls/aij/seq/aijfact.c >>>> [0]PETSC ERROR: MatILUFactorSymbolic() line 5464 in >>>> src/mat/interface/matrix.c >>>> [0]PETSC ERROR: PCSetUp_ILU() line 204 in >>>> src/ksp/pc/impls/factor/ilu/ilu.c >>>> [0]PETSC ERROR: PCSetUp() line 795 in src/ksp/pc/interface/precon.c >>>> [0]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c >>>> [0]PETSC ERROR: KSPSolve() line 353 in src/ksp/ksp/interface/itfunc.c >>>> >>>> >>>> >>>> I am sure that the matrix is correct and the system is not singular >>>> because the system can be solved using LAPACK's direct solver. I >>>> probably format the PETSc matrix incorrectly or some PETSc parameters >>>> are wrong. I would appreciate if you give me any clue about possible >>>> problem(s). >>>> >>>> Regards, >>>> >>>> Danesh >>>> >>>> >>> >>> >>> >
