I don't think it is the zero entries. Please run both the first two version below with the option -ksp_view_mat binary -ksp_view_rhs binary and in each case email to [email protected] the resulting file called binaryoutput
Barry > On Dec 7, 2016, at 2:49 AM, Leidy Catherine Ramirez Villalba > <[email protected]> wrote: > > Hi Barry, > > Thanks for your reply! > > I must say that I still do not master still the different solvers and > options, so my problem might be due to a wrong formulation for the solver, or > the final state of the matrix is not good, even if I verify that is assembled > and the exit looks quite ok. Following the whole thing: > > > After assembling, manually and using petsc, I compare the matrices using a > ASCII file (PETSC_VIEWER_ASCII_MATLAB option), and there i see no difference > but the zero in some rows > > 1664 560 -1.6995706110114395e+06 > | 1664 560 -1.6995706110114395e+06 > 1664 1660 -1.7462298274040222e-10 > | 1664 1660 -1.7462298274040222e-10 > 1664 1661 6.6919512730949128e+05 > | 1664 1661 6.6919512730949128e+05 > 1664 1663 0.0000000000000000e+00 > | 1664 1664 5.4708211855417928e+06 > > 1664 1664 5.4707211855417928e+06 > | > ------------------------------------------------------------------------------------------------------- > 1664 4600 -1.7462298274040222e-10 > | 1664 4600 -1.7462298274040222e-10 > > > then, I solve both systems using MUMPS solve, with same rhs vector and > finally I compare the output vector Ur. > > Below how I solve the system: > > ! ---------------------------------------SOLVING 1: manual matrix: mpiA > ---------------------------------------------------------------- > call KSPCreate(PETSC_COMM_WORLD,kspSolver,petscIerr) > call KSPSetOperators(kspSolver,mpiA,mpiA,petscIerr) > > call KSPSetType(kspSolver, KSPPREONLY, petscIerr) > call KSPSetFromOptions(kspSolver,petscIerr) > > call KSPGetPC(kspSolver,precond,petscIerr) > call PCSetType(precond,PCLU,petscIerr) > call PCFactorSetMatSolverPackage(precond,MATSOLVERMUMPS,petscIerr) > call PCFactorSetUpMatSolverPackage(precond,petscIerr) > > call PCSetFromOptions(precond,petscIerr) > > relativeTol=1.e-15 > absoluteTol=1.e-15 > divergenceTol=1.e4 > maxIter=200 > call > KSPSetTolerances(kspSolver,relativeTol,absoluteTol,divergenceTol,maxIter,petscIerr) > > call KSPSolve(kspSolver,mpiVecRhs,mpiVecLocSolution,petscIerr) > > call KSPGetSolution(kspSolver,mpiVecLocSolution,petscIerr) > > call > VecScatterCreateToAll(mpiVecLocSolution,vecScatterEnv,mpiVecGlobSolution,petscIerr) > call > VecScatterBegin(vecScatterEnv,mpiVecLocSolution,mpiVecGlobSolution,INSERT_VALUES,SCATTER_FORWARD,petscIerr) > call > VecScatterEnd(vecScatterEnv,mpiVecLocSolution,mpiVecGlobSolution,INSERT_VALUES,SCATTER_FORWARD,petscIerr) > > call VecGetArrayF90(mpiVecGlobSolution,pVecSolution,petscIerr) > Ur(1:ddl) = pVecSolution(:) > call VecRestoreArrayF90(mpiVecLocSolution,pVecSolution,petscIerr) > ! > ------------------------------------------------------------------------------------------------------- > > > For the assembled matrix I have tried the same sequence, but: > > 1. changing the system and preconditioner specified matrix for the solver: > > call KSPSetOperators(kspSolver,mpiMassMatrix,mpiMassMatrix,petscIerr) > > There, I'm getting infinite for the solution for the petsc assembled matrix > and the proper solution for the manual assembled one. > > %Vec Object:mpiVecLocSolution 1 MPI processes | > %Vec Object:mpiVecLocSolution 1 MPI processes > % type: seq | > % type: seq > mpiVecLocSolution = [ | > mpiVecLocSolution = [ > inf | > -2.2389524508816294e-20 > > inf | > -1.5265035169220699e-20 > > inf | > 0.0000000000000000e+00 > > > > 2. I thought it was due to the extra zeros, then I tried no passing the same > matrix for the preconditioner argument. > > call > KSPSetOperators(kspSolver,mpiMassMatrix,PETSC_NULL_OBJECT,petscIerr) > > There I have the errors listed below and a solution vector full of zeros for > the petsc assembled matrix. > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: You can only call this routine after the matrix object has > been provided to the solver, for example with KSPSetOperators() or > SNESSetJacobian() > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for > trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.7.4, unknown > [0]PETSC ERROR: ./aravanti on a arch-linux2-c-debug named > localhost.localdomain by catherine Wed Dec 7 09:36:39 2016 > [0]PETSC ERROR: Configure options --download-mumps > --with-mpi-dir=/home/catherine/local/intel2016/ --download-scalapack > --download-parmetis --download-metis --download-fblaslapack > [0]PETSC ERROR: #1 PCFactorSetUpMatSolverPackage_Factor() line 15 in > /home/catherine/petsc/src/ksp/pc/impls/factor/factimpl.c > [0]PETSC ERROR: #2 PCFactorSetUpMatSolverPackage() line 26 in > /home/catherine/petsc/src/ksp/pc/impls/factor/factor.c > solving the system > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: Not for unassembled matrix > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for > trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.7.4, unknown > [0]PETSC ERROR: ./aravanti on a arch-linux2-c-debug named > localhost.localdomain by catherine Wed Dec 7 09:36:39 2016 > [0]PETSC ERROR: Configure options --download-mumps > --with-mpi-dir=/home/catherine/local/intel2016/ --download-scalapack > --download-parmetis --download-metis --download-fblaslapack > [0]PETSC ERROR: #3 MatGetOrdering() line 189 in > /home/catherine/petsc/src/mat/order/sorder.c > [0]PETSC ERROR: #4 PCSetUp_LU() line 125 in > /home/catherine/petsc/src/ksp/pc/impls/factor/lu/lu.c > [0]PETSC ERROR: #5 PCSetUp() line 968 in > /home/catherine/petsc/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #6 KSPSetUp() line 390 in > /home/catherine/petsc/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #7 KSPSolve() line 599 in > /home/catherine/petsc/src/ksp/ksp/interface/itfunc.c > > > > 3. Or simply the default option (direct solver LU if i'm not mistaken) > call KSPCreate(PETSC_COMM_WORLD,kspSolver,petscIerr) > call KSPSetOperators(kspSolver,mpiMassMatrix,mpiMassMatrix,petscIerr) > call KSPSolve(kspSolver,mpiVecRhs,mpiVecLocSolution,petscIerr) > > and then again I have an error and an exit of zeros, and a closer answer for > the matrix assembled manually. > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: Matrix is missing diagonal entry 5955 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for > trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.7.4, unknown > [0]PETSC ERROR: ./aravanti on a arch-linux2-c-debug named > localhost.localdomain by catherine Wed Dec 7 09:26:29 2016 > [0]PETSC ERROR: Configure options --download-mumps > --with-mpi-dir=/home/catherine/local/intel2016/ --download-scalapack > --download-parmetis --download-metis --download-fblaslapack > [0]PETSC ERROR: #1 MatILUFactorSymbolic_SeqAIJ() line 1733 in > /home/catherine/petsc/src/mat/impls/aij/seq/aijfact.c > [0]PETSC ERROR: #2 MatILUFactorSymbolic() line 6579 in > /home/catherine/petsc/src/mat/interface/matrix.c > [0]PETSC ERROR: #3 PCSetUp_ILU() line 213 in > /home/catherine/petsc/src/ksp/pc/impls/factor/ilu/ilu.c > [0]PETSC ERROR: #4 PCSetUp() line 968 in > /home/catherine/petsc/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #5 KSPSetUp() line 390 in > /home/catherine/petsc/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #6 KSPSolve() line 599 in > /home/catherine/petsc/src/ksp/ksp/interface/itfunc.c > > > > I hope information is clear enough. > > Thanks again, > Catherine > > > > ----- Mail original ----- > De: "Barry Smith" <[email protected]> > À: "Leidy Catherine Ramirez Villalba" > <[email protected]> > Cc: [email protected] > Envoyé: Mardi 6 Décembre 2016 18:45:52 > Objet: Re: [petsc-users] Way to remove zero entries from an assembled matrix > >> On Dec 6, 2016, at 11:25 AM, Leidy Catherine Ramirez Villalba >> <[email protected]> wrote: >> >> Hello PETSc team: >> >> I'm doing the parallelization of the assembling of a system, previously >> assembled in a serial way (manual), but solved using PETSc in parallel. >> Therefore I have the old assembled matrix to compare with the one assembled >> with PETSc. >> >> While doing the assembling of the matrix, I avoid the zero entries using the >> option 'MAT_IGNORE_ZERO_ENTRIES' in MatSetOption, however the final matrix >> has zero values (or almost) due to the addition of non zero elements. >> Below the example of added values: >> >> 1664 i 1663 j -165509.423650377 >> 1664 i 1663 j 165509.423650377 >> 1664 i 1663 j -165509.423650377 >> 1664 i 1663 j 165509.423650377 >> >> Due to this difference between the two matrices I'm not able to get the >> solution for the matrix assembled with petsc. > > Catherine, > > The "extra" zeros should not prevent getting the solution to the system. > The computed solution may be a little different but if you use an iterative > method with smaller and smaller tolerances the solutions should converge to > the same value. With direct solvers the solutions should be very similar. > > Could you please provide more details about not getting the solution for > the parallel system? Send output of -ksp_monitor_true_residual for example > > Barry > > >> >> Therefore I wonder if there is a way to remove zero entries from an >> assembled matrix, or which solver to use so that zeros will not be a problem. >> >> Thanks in advance, >> Catherine >> >> >> >>
