I am using periodic along x and Neumann along y. I remove the nullspace via -ksp_constant_null_space.
Matt's suggestion worked; I will also give a try to umfpack and cholesky.

Thanks,
Michele

On 02/11/2016 06:50 AM, Matthew Knepley wrote:
On Thu, Feb 11, 2016 at 12:30 AM, Dave May <[email protected] <mailto:[email protected]>> wrote:



    On 11 February 2016 at 07:05, Michele Rosso <[email protected]
    <mailto:[email protected]>> wrote:

        I tried setting -mat_superlu_dist_replacetinypivot true: it
does help to advance the run past the previous "critical" point but eventually it stops later with the same error.
        I forgot to mention my system is singular: I remove the
        constant null space but I am not sure if the coarse solver
        needs to be explicity informed of this.


    Right - are you using pure Newmann boundary conditions?

    To make the solution unique, are you
    (a) imposing a single Dichletet boundary condition on your field
    by explicitly modifying the matrix
    (b) imposing a a condition like
      \int \phi dV = 0
    via something like -ksp_constant_null_space

    If you removed removed the null space by modifying the matrix
    explicitly (a), the sparse direct solver
    should go through. If you use (b), then this method cannot be used
    to help the direct solver.

    If this is the intended target problem size (16x16), gather the
    matrix and using petsc Cholesky or Umfpack.
    Cholesky is more stable than LU and can usually deal with a single
    zero eigenvaue without resorting to tricks. Umfpack will solve the
    problem easily as it uses clever re-ordering. If you have access
    to MKL-Pardiso, that will also work great.


An easy fix is just to use -pc_type svd on the coarse grid.

  Matt

    Thanks,
      Dave



        Michele


        On Wed, 2016-02-10 at 22:15 -0600, Barry Smith wrote:
            You can try the option

        -mat_superlu_dist_replacetinypivot true

        if you are luck it get you past the zero pivot but still give an 
adequate preconditioner.

           Barry

        > On Feb 10, 2016, at 9:49 PM, Michele Rosso <[email protected]
        <mailto:[email protected]>> wrote:
        >
        > Hong,
        >
        > here if the output of grep -info:
        >
        > using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
        > Replace tiny pivots FALSE
        > tolerance for zero pivot 2.22045e-14
        >
        > It seems it is not replacing small pivots: could this be
        the problem?
        > I will also try Barry's suggestion to diagnose the problem.
        >
        > Thanks,
        > Michele
        >
        >
        > On Wed, 2016-02-10 at 21:22 -0600, Barry Smith wrote:
        >> > On Feb 10, 2016, at 9:00 PM, Hong <[email protected]
        <mailto:[email protected]>> wrote:
        >> >
        >> > Michele :
        >> > Superlu_dist LU is used for coarse grid PC, which likely
        produces a zero-pivot.
        >> > Run your code with '-info |grep pivot' to verify.
        >>
        >>
        >> Michele
        >>
        >> You can also run with -ksp_error_if_not_converged in or
        not in the debugger and it will stop immediately when the
        problem is detected and hopefully provide additional useful
        information about what has happened.
        >>
        >> Barry
        >>
        >>
        >> >
        >> > Hong
        >> >
        >> > Hi Matt,
        >> >
        >> > the ksp_view output was an attachment to my previous email.
        >> > Here it is:
        >> >
        >> > KSP Object: 1 MPI processes
        >> > type: cg
        >> > maximum iterations=10000
        >> > tolerances: relative=1e-08, absolute=1e-50,
        divergence=10000.
        >> > left preconditioning
        >> > using nonzero initial guess
        >> > using UNPRECONDITIONED norm type for convergence test
        >> > PC Object: 1 MPI processes
        >> > type: mg
        >> > MG: type is MULTIPLICATIVE, levels=4 cycles=v
        >> > Cycles per PCApply=1
        >> > Using Galerkin computed coarse grid matrices
        >> > Coarse grid solver -- level -------------------------------
        >> > KSP Object: (mg_coarse_) 1 MPI processes
        >> > type: preonly
        >> > maximum iterations=1, initial guess is zero
        >> > tolerances: relative=1e-05, absolute=1e-50,
        divergence=10000.
        >> > left preconditioning
        >> > using NONE norm type for convergence test
        >> > PC Object: (mg_coarse_) 1 MPI processes
        >> > type: lu
        >> > LU: out-of-place factorization
        >> > tolerance for zero pivot 2.22045e-14
        >> > using diagonal shift on blocks to prevent zero pivot
        [INBLOCKS]
        >> > matrix ordering: nd
        >> > factor fill ratio given 0., needed 0.
        >> > Factored matrix follows:
        >> > Mat Object: 1 MPI processes
        >> > type: seqaij
        >> > rows=16, cols=16
        >> > package used to perform factorization: superlu_dist
        >> > total: nonzeros=0, allocated nonzeros=0
        >> > total number of mallocs used during MatSetValues calls =0
        >> > SuperLU_DIST run parameters:
        >> > Process grid nprow 1 x npcol 1
        >> > Equilibrate matrix TRUE
        >> > Matrix input mode 0
        >> > Replace tiny pivots FALSE
        >> > Use iterative refinement FALSE
        >> > Processors in row 1 col partition 1
        >> > Row permutation LargeDiag
        >> > Column permutation METIS_AT_PLUS_A
        >> > Parallel symbolic factorization FALSE
        >> > Repeated factorization SamePattern
        >> > linear system matrix = precond matrix:
        >> > Mat Object: 1 MPI processes
        >> > type: seqaij
        >> > rows=16, cols=16
        >> > total: nonzeros=72, allocated nonzeros=72
        >> > total number of mallocs used during MatSetValues calls =0
        >> > not using I-node routines
        >> > Down solver (pre-smoother) on level 1
        -------------------------------
        >> > KSP Object: (mg_levels_1_) 1 MPI processes
        >> > type: richardson
        >> > Richardson: damping factor=1.
        >> > maximum iterations=2
        >> > tolerances: relative=1e-05, absolute=1e-50,
        divergence=10000.
        >> > left preconditioning
        >> > using nonzero initial guess
        >> > using NONE norm type for convergence test
        >> > PC Object: (mg_levels_1_) 1 MPI processes
        >> > type: sor
        >> > SOR: type = local_symmetric, iterations = 1, local
        iterations = 1, omega = 1.
        >> > linear system matrix = precond matrix:
        >> > Mat Object: 1 MPI processes
        >> > type: seqaij
        >> > rows=64, cols=64
        >> > total: nonzeros=304, allocated nonzeros=304
        >> > total number of mallocs used during MatSetValues calls =0
        >> > not using I-node routines
        >> > Up solver (post-smoother) same as down solver (pre-smoother)
        >> > Down solver (pre-smoother) on level 2
        -------------------------------
        >> > KSP Object: (mg_levels_2_) 1 MPI processes
        >> > type: richardson
        >> > Richardson: damping factor=1.
        >> > maximum iterations=2
        >> > tolerances: relative=1e-05, absolute=1e-50,
        divergence=10000.
        >> > left preconditioning
        >> > using nonzero initial guess
        >> > using NONE norm type for convergence test
        >> > PC Object: (mg_levels_2_) 1 MPI processes
        >> > type: sor
        >> > SOR: type = local_symmetric, iterations = 1, local
        iterations = 1, omega = 1.
        >> > linear system matrix = precond matrix:
        >> > Mat Object: 1 MPI processes
        >> > type: seqaij
        >> > rows=256, cols=256
        >> > total: nonzeros=1248, allocated nonzeros=1248
        >> > total number of mallocs used during MatSetValues calls =0
        >> > not using I-node routines
        >> > Up solver (post-smoother) same as down solver (pre-smoother)
        >> > Down solver (pre-smoother) on level 3
        -------------------------------
        >> > KSP Object: (mg_levels_3_) 1 MPI processes
        >> > type: richardson
        >> > Richardson: damping factor=1.
        >> > maximum iterations=2
        >> > tolerances: relative=1e-05, absolute=1e-50,
        divergence=10000.
        >> > left preconditioning
        >> > using nonzero initial guess
        >> > using NONE norm type for convergence test
        >> > PC Object: (mg_levels_3_) 1 MPI processes
        >> > type: sor
        >> > SOR: type = local_symmetric, iterations = 1, local
        iterations = 1, omega = 1.
        >> > linear system matrix = precond matrix:
        >> > Mat Object: 1 MPI processes
        >> > type: seqaij
        >> > rows=1024, cols=1024
        >> > total: nonzeros=5056, allocated nonzeros=5056
        >> > total number of mallocs used during MatSetValues calls =0
        >> > has attached null space
        >> > not using I-node routines
        >> > Up solver (post-smoother) same as down solver (pre-smoother)
        >> > linear system matrix = precond matrix:
        >> > Mat Object: 1 MPI processes
        >> > type: seqaij
        >> > rows=1024, cols=1024
        >> > total: nonzeros=5056, allocated nonzeros=5056
        >> > total number of mallocs used during MatSetValues calls =0
        >> > has attached null space
        >> > not using I-node routines
        >> >
        >> >
        >> > Michele
        >> >
        >> >
        >> >
        >> >
        >> > On Wed, 2016-02-10 at 19:37 -0600, Matthew Knepley wrote:
        >> >> On Wed, Feb 10, 2016 at 7:33 PM, Michele Rosso
        <[email protected] <mailto:[email protected]>> wrote:
        >> >> Hi,
        >> >>
        >> >> I encountered the following error while solving a
        symmetric positive defined system:
        >> >>
        >> >> Linear solve did not converge due to
        DIVERGED_PCSETUP_FAILED iterations 0
        >> >> PCSETUP_FAILED due to SUBPC_ERROR
        >> >>
        >> >> This error appears only if I use the optimized version
        of both petsc and my code ( compiler: gfortran, flags: -O3 ).
        >> >> It is weird since I am solving a time-dependent problem
        and everything, i.e. results and convergence rate, are as
        expected until the above error shows up. If I run both petsc
        and my code in debug mode, everything goes smooth till the
        end of the simulation.
        >> >> However, if I reduce the ksp_rtol, even the debug run
        fails, after running as expected for a while, because of a
        KSP_DIVERGED_INDEFINITE_PC .
        >> >> The options I am using are:
        >> >>
        >> >> -ksp_type cg
        >> >> -ksp_norm_type unpreconditioned
        >> >> -ksp_rtol 1e-8
        >> >> -ksp_lag_norm
        >> >> -ksp_initial_guess_nonzero yes
        >> >> -pc_type mg
        >> >> -pc_mg_galerkin
        >> >> -pc_mg_levels 4
        >> >> -mg_levels_ksp_type richardson
        >> >> -mg_coarse_ksp_constant_null_space
        >> >> -mg_coarse_pc_type lu
        >> >> -mg_coarse_pc_factor_mat_solver_package superlu_dist
        >> >> -options_left
        >> >>
        >> >> I attached a copy of ksp_view. I am currently using
        petsc-master (last updated yesterday).
        >> >> I would appreciate any suggestion on this matter.
        >> >>
        >> >>
        >> >>
        >> >> I suspect you have a nonlinear PC. Can you send the
        output of -ksp_view?
        >> >>
        >> >>
        >> >> Matt
        >> >>
        >> >> Thanks,
        >> >> Michele
        >> >>
        >> >>
        >> >>
        >> >>
        >> >>
        >> >>
        >> >>
        >> >>
        >> >>
        >> >>
        >> >>
        >> >>
        >> >> --
        >> >> What most experimenters take for granted before they
        begin their experiments is infinitely more interesting than
        any results to which their experiments lead.
        >> >> -- Norbert Wiener
        >> >
        >> >
        >>
        >>
        >>
        >






--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

Reply via email to