On Thu, Feb 11, 2016 at 12:30 AM, Dave May <[email protected]> wrote:
> > > On 11 February 2016 at 07:05, Michele Rosso <[email protected]> wrote: > >> I tried setting -mat_superlu_dist_replacetinypivot true: it does help to >> advance the run past the previous "critical" point but eventually it stops >> later with the same error. >> I forgot to mention my system is singular: I remove the constant null >> space but I am not sure if the coarse solver needs to be explicity informed >> of this. >> > > Right - are you using pure Newmann boundary conditions? > > To make the solution unique, are you > (a) imposing a single Dichletet boundary condition on your field > by explicitly modifying the matrix > (b) imposing a a condition like > \int \phi dV = 0 > via something like -ksp_constant_null_space > > If you removed removed the null space by modifying the matrix explicitly > (a), the sparse direct solver > should go through. If you use (b), then this method cannot be used to help > the direct solver. > > If this is the intended target problem size (16x16), gather the matrix and > using petsc Cholesky or Umfpack. > Cholesky is more stable than LU and can usually deal with a single zero > eigenvaue without resorting to tricks. Umfpack will solve the problem > easily as it uses clever re-ordering. If you have access to MKL-Pardiso, > that will also work great. > An easy fix is just to use -pc_type svd on the coarse grid. Matt > Thanks, > Dave > > > >> >> >> Michele >> >> >> On Wed, 2016-02-10 at 22:15 -0600, Barry Smith wrote: >> >> You can try the option >> >> -mat_superlu_dist_replacetinypivot true >> >> if you are luck it get you past the zero pivot but still give an adequate >> preconditioner. >> >> Barry >> > On Feb 10, 2016, at 9:49 PM, Michele Rosso <[email protected]> wrote:> > >> > Hong,> > here if the output of grep -info:> > using diagonal shift >> > on blocks to prevent zero pivot [INBLOCKS]> Replace tiny >> > pivots FALSE> tolerance for zero pivot 2.22045e-14> > It seems it >> > is not replacing small pivots: could this be the problem?> I will also try >> > Barry's suggestion to diagnose the problem.> > Thanks,> Michele> > > On >> > Wed, 2016-02-10 at 21:22 -0600, Barry Smith wrote:>> > On Feb 10, 2016, at >> > 9:00 PM, Hong <[email protected]> wrote:>> > >> > Michele :>> > >> > Superlu_dist LU is used for coarse grid PC, which likely produces a >> > zero-pivot.>> > Run your code with '-info |grep pivot' to verify.>> >> >> >> > Michele>> >> You can also run with -ksp_error_if_not_converged in or >> > not in the debugger and it will stop immediately when the problem is >> > detected and hopefully provide additional useful information about what >> > has happened.>> >> Barry>> >> >> > >> > Hong>> > >> > Hi Matt,>> > >> > >> > the ksp_view output was an attachment to my previous email.>> > Here it >> > is:>> > >> > KSP Object: 1 MPI processes>> > type: cg>> > maximum >> > iterations=10000>> > tolerances: relative=1e-08, absolute=1e-50, >> > divergence=10000.>> > left preconditioning>> > using nonzero initial >> > guess>> > using UNPRECONDITIONED norm type for convergence test>> > PC >> > Object: 1 MPI processes>> > type: mg>> > MG: type is MULTIPLICATIVE, >> > levels=4 cycles=v>> > Cycles per PCApply=1>> > Using Galerkin >> > computed coarse grid matrices>> > Coarse grid solver -- level >> > ------------------------------->> > KSP Object: (mg_coarse_) 1 >> > MPI processes>> > type: preonly>> > maximum iterations=1, >> > initial guess is zero>> > tolerances: relative=1e-05, >> > absolute=1e-50, divergence=10000.>> > left preconditioning>> > >> > using NONE norm type for convergence test>> > PC Object: >> > (mg_coarse_) 1 MPI processes>> > type: lu>> > LU: >> > out-of-place factorization>> > tolerance for zero pivot >> > 2.22045e-14>> > using diagonal shift on blocks to prevent zero >> > pivot [INBLOCKS]>> > matrix ordering: nd>> > factor fill >> > ratio given 0., needed 0.>> > Factored matrix follows:>> > >> > Mat Object: 1 MPI processes>> > type: >> > seqaij>> > rows=16, cols=16>> > package used >> > to perform factorization: superlu_dist>> > total: >> > nonzeros=0, allocated nonzeros=0>> > total number of mallocs >> > used during MatSetValues calls =0>> > SuperLU_DIST run >> > parameters:>> > Process grid nprow 1 x npcol 1 >> > >> > Equilibrate matrix TRUE >> > Matrix input >> > mode 0 >> > Replace tiny pivots FALSE >> > >> > Use iterative refinement FALSE >> > Processors in >> > row 1 col partition 1 >> > Row permutation LargeDiag >> >> > > Column permutation METIS_AT_PLUS_A>> > >> > Parallel symbolic factorization FALSE >> > Repeated >> > factorization SamePattern>> > linear system matrix = precond >> > matrix:>> > Mat Object: 1 MPI processes>> > type: >> > seqaij>> > rows=16, cols=16>> > total: nonzeros=72, >> > allocated nonzeros=72>> > total number of mallocs used during >> > MatSetValues calls =0>> > not using I-node routines>> > Down >> > solver (pre-smoother) on level 1 ------------------------------->> > >> > KSP Object: (mg_levels_1_) 1 MPI processes>> > type: >> > richardson>> > Richardson: damping factor=1.>> > maximum >> > iterations=2>> > tolerances: relative=1e-05, absolute=1e-50, >> > divergence=10000.>> > left preconditioning>> > using nonzero >> > initial guess>> > using NONE norm type for convergence test>> > >> > PC Object: (mg_levels_1_) 1 MPI processes>> > type: sor>> > >> > SOR: type = local_symmetric, iterations = 1, local iterations = 1, >> > omega = 1.>> > linear system matrix = precond matrix:>> > Mat >> > Object: 1 MPI processes>> > type: seqaij>> > >> > rows=64, cols=64>> > total: nonzeros=304, allocated nonzeros=304>> >> > > total number of mallocs used during MatSetValues calls =0>> > >> > not using I-node routines>> > Up solver (post-smoother) same as >> > down solver (pre-smoother)>> > Down solver (pre-smoother) on level 2 >> > ------------------------------->> > KSP Object: (mg_levels_2_) >> > 1 MPI processes>> > type: richardson>> > Richardson: damping >> > factor=1.>> > maximum iterations=2>> > tolerances: >> > relative=1e-05, absolute=1e-50, divergence=10000.>> > left >> > preconditioning>> > using nonzero initial guess>> > using NONE >> > norm type for convergence test>> > PC Object: (mg_levels_2_) 1 >> > MPI processes>> > type: sor>> > SOR: type = local_symmetric, >> > iterations = 1, local iterations = 1, omega = 1.>> > linear system >> > matrix = precond matrix:>> > Mat Object: 1 MPI processes>> > >> > type: seqaij>> > rows=256, cols=256>> > total: >> > nonzeros=1248, allocated nonzeros=1248>> > total number of mallocs >> > used during MatSetValues calls =0>> > not using I-node >> > routines>> > Up solver (post-smoother) same as down solver >> > (pre-smoother)>> > Down solver (pre-smoother) on level 3 >> > ------------------------------->> > KSP Object: (mg_levels_3_) >> > 1 MPI processes>> > type: richardson>> > Richardson: damping >> > factor=1.>> > maximum iterations=2>> > tolerances: >> > relative=1e-05, absolute=1e-50, divergence=10000.>> > left >> > preconditioning>> > using nonzero initial guess>> > using NONE >> > norm type for convergence test>> > PC Object: (mg_levels_3_) 1 >> > MPI processes>> > type: sor>> > SOR: type = local_symmetric, >> > iterations = 1, local iterations = 1, omega = 1.>> > linear system >> > matrix = precond matrix:>> > Mat Object: 1 MPI processes>> > >> > type: seqaij>> > rows=1024, cols=1024>> > total: >> > nonzeros=5056, allocated nonzeros=5056>> > total number of mallocs >> > used during MatSetValues calls =0>> > has attached null space>> >> > > not using I-node routines>> > Up solver (post-smoother) same >> > as down solver (pre-smoother)>> > linear system matrix = precond >> > matrix:>> > Mat Object: 1 MPI processes>> > type: seqaij>> > >> > rows=1024, cols=1024>> > total: nonzeros=5056, allocated >> > nonzeros=5056>> > total number of mallocs used during MatSetValues >> > calls =0>> > has attached null space>> > not using I-node >> > routines>> > >> > >> > Michele>> > >> > >> > >> > >> > On Wed, 2016-02-10 >> > at 19:37 -0600, Matthew Knepley wrote:>> >> On Wed, Feb 10, 2016 at 7:33 >> > PM, Michele Rosso <[email protected]> wrote:>> >> Hi,>> >> >> >> I >> > encountered the following error while solving a symmetric positive defined >> > system:>> >> >> >> Linear solve did not converge due to >> > DIVERGED_PCSETUP_FAILED iterations 0>> >> PCSETUP_FAILED >> > due to SUBPC_ERROR >> >> >> >> This error appears only if I use the >> > optimized version of both petsc and my code ( compiler: gfortran, flags: >> > -O3 ).>> >> It is weird since I am solving a time-dependent problem and >> > everything, i.e. results and convergence rate, are as expected until the >> > above error shows up. If I run both petsc and my code in debug mode, >> > everything goes smooth till the end of the simulation.>> >> However, if I >> > reduce the ksp_rtol, even the debug run fails, after running as expected >> > for a while, because of a KSP_DIVERGED_INDEFINITE_PC . >> >> The options I >> > am using are:>> >> >> >> -ksp_type cg>> >> -ksp_norm_type >> > unpreconditioned>> >> -ksp_rtol 1e-8>> >> -ksp_lag_norm>> >> >> > -ksp_initial_guess_nonzero yes>> >> -pc_type mg>> >> -pc_mg_galerkin>> >> >> > -pc_mg_levels 4>> >> -mg_levels_ksp_type richardson>> >> >> > -mg_coarse_ksp_constant_null_space>> >> -mg_coarse_pc_type lu>> >> >> > -mg_coarse_pc_factor_mat_solver_package superlu_dist>> >> -options_left>> >> > >> >> >> I attached a copy of ksp_view. I am currently using petsc-master >> > (last updated yesterday).>> >> I would appreciate any suggestion on this >> > matter.>> >> >> >> >> >> >> >> I suspect you have a nonlinear PC. Can you >> > send the output of -ksp_view?>> >> >> >> >> >> Matt>> >> >> >> >> > Thanks,>> >> Michele>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> -->> >> What most experimenters take for granted >> > before they begin their experiments is infinitely more interesting than >> > any results to which their experiments lead.>> >> -- Norbert Wiener>> > >> >> > > >> >> >> > >> >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
