On Thu, Feb 11, 2016 at 12:30 AM, Dave May <[email protected]> wrote:

>
>
> On 11 February 2016 at 07:05, Michele Rosso <[email protected]> wrote:
>
>> I tried setting -mat_superlu_dist_replacetinypivot true: it does help to
>> advance the run past the previous "critical"  point but eventually it stops
>> later with the same error.
>> I forgot to mention my system is singular: I remove the constant null
>> space but I am not sure if the coarse solver needs to be explicity informed
>> of this.
>>
>
> Right - are you using pure Newmann boundary conditions?
>
> To make the solution unique, are you
> (a) imposing a single Dichletet boundary condition on your field
> by explicitly modifying the matrix
> (b) imposing a a condition like
>   \int \phi dV = 0
> via something like -ksp_constant_null_space
>
> If you removed removed the null space by modifying the matrix explicitly
> (a), the sparse direct solver
> should go through. If you use (b), then this method cannot be used to help
> the direct solver.
>
> If this is the intended target problem size (16x16), gather the matrix and
> using petsc Cholesky or Umfpack.
> Cholesky is more stable than LU and can usually deal with a single zero
> eigenvaue without resorting to tricks. Umfpack will solve the problem
> easily as it uses clever re-ordering. If you have access to MKL-Pardiso,
> that will also work great.
>

An easy fix is just to use -pc_type svd on the coarse grid.

  Matt


> Thanks,
>   Dave
>
>
>
>>
>>
>> Michele
>>
>>
>> On Wed, 2016-02-10 at 22:15 -0600, Barry Smith wrote:
>>
>>    You can try the option
>>
>> -mat_superlu_dist_replacetinypivot true
>>
>> if you are luck it get you past the zero pivot but still give an adequate 
>> preconditioner.
>>
>>   Barry
>> > On Feb 10, 2016, at 9:49 PM, Michele Rosso <[email protected]> wrote:> > 
>> > Hong,> > here if the output of grep -info:> >         using diagonal shift 
>> > on blocks to prevent zero pivot [INBLOCKS]>                   Replace tiny 
>> > pivots FALSE>         tolerance for zero pivot 2.22045e-14> > It seems it 
>> > is not replacing small pivots: could this be the problem?> I will also try 
>> > Barry's suggestion to diagnose the problem.> > Thanks,> Michele> > > On 
>> > Wed, 2016-02-10 at 21:22 -0600, Barry Smith wrote:>> > On Feb 10, 2016, at 
>> > 9:00 PM, Hong <[email protected]> wrote:>> > >> > Michele :>> > 
>> > Superlu_dist LU is used for coarse grid PC, which likely produces a 
>> > zero-pivot.>> > Run your code with '-info |grep pivot' to  verify.>> >> >> 
>> >   Michele>> >>    You can also run with -ksp_error_if_not_converged in or 
>> > not in the debugger and it will stop immediately when the problem is 
>> > detected and hopefully provide additional useful  information about what 
>> > has happened.>> >>   Barry>> >> >> > >> > Hong>> > >> > Hi Matt,>> > >> > 
>> > the ksp_view output was an attachment to my previous email.>> > Here it 
>> > is:>> > >> > KSP Object: 1 MPI processes>> >   type: cg>> >   maximum 
>> > iterations=10000>> >   tolerances:  relative=1e-08, absolute=1e-50, 
>> > divergence=10000.>> >   left preconditioning>> >   using nonzero initial 
>> > guess>> >   using UNPRECONDITIONED norm type for convergence test>> > PC 
>> > Object: 1 MPI processes>> >   type: mg>> >     MG: type is MULTIPLICATIVE, 
>> > levels=4 cycles=v>> >       Cycles per PCApply=1>> >       Using Galerkin 
>> > computed coarse grid matrices>> >   Coarse grid solver -- level 
>> > ------------------------------->> >     KSP Object:    (mg_coarse_)     1 
>> > MPI processes>> >       type: preonly>> >       maximum iterations=1, 
>> > initial guess is zero>> >       tolerances:  relative=1e-05, 
>> > absolute=1e-50, divergence=10000.>> >       left preconditioning>> >       
>> > using NONE norm type for convergence test>> >     PC Object:    
>> > (mg_coarse_)     1 MPI processes>> >       type: lu>> >         LU: 
>> > out-of-place factorization>> >         tolerance for zero pivot 
>> > 2.22045e-14>> >         using diagonal shift on blocks to prevent zero 
>> > pivot [INBLOCKS]>> >         matrix ordering: nd>> >         factor fill 
>> > ratio given 0., needed 0.>> >           Factored matrix follows:>> >       
>> >       Mat Object:             1 MPI processes>> >               type: 
>> > seqaij>> >               rows=16, cols=16>> >               package used 
>> > to perform factorization: superlu_dist>> >               total: 
>> > nonzeros=0, allocated nonzeros=0>> >               total number of mallocs 
>> > used during MatSetValues calls =0>> >                 SuperLU_DIST run 
>> > parameters:>> >                   Process grid nprow 1 x npcol 1 >> >      
>> >              Equilibrate matrix TRUE >> >                   Matrix input 
>> > mode 0 >> >                   Replace tiny pivots FALSE >> >               
>> >     Use iterative refinement FALSE >> >                   Processors in 
>> > row 1 col partition 1 >> >                   Row permutation LargeDiag >> 
>> > >                   Column permutation METIS_AT_PLUS_A>> >                 
>> >   Parallel symbolic factorization FALSE >> >                   Repeated 
>> > factorization SamePattern>> >       linear system matrix = precond 
>> > matrix:>> >       Mat Object:       1 MPI processes>> >         type: 
>> > seqaij>> >         rows=16, cols=16>> >         total: nonzeros=72, 
>> > allocated nonzeros=72>> >         total number of mallocs used during 
>> > MatSetValues calls =0>> >           not using I-node routines>> >   Down 
>> > solver (pre-smoother) on level 1 ------------------------------->> >     
>> > KSP Object:    (mg_levels_1_)     1 MPI processes>> >       type: 
>> > richardson>> >         Richardson: damping factor=1.>> >       maximum 
>> > iterations=2>> >       tolerances:  relative=1e-05, absolute=1e-50, 
>> > divergence=10000.>> >       left preconditioning>> >       using nonzero 
>> > initial guess>> >       using NONE norm type for convergence test>> >     
>> > PC Object:    (mg_levels_1_)     1 MPI processes>> >       type: sor>> >   
>> >       SOR: type = local_symmetric, iterations = 1, local iterations = 1, 
>> > omega = 1.>> >       linear system matrix = precond matrix:>> >       Mat 
>> > Object:       1 MPI processes>> >         type: seqaij>> >         
>> > rows=64, cols=64>> >         total: nonzeros=304, allocated nonzeros=304>> 
>> > >         total number of mallocs used during MatSetValues calls =0>> >    
>> >        not using I-node routines>> >   Up solver (post-smoother) same as 
>> > down solver (pre-smoother)>> >   Down solver (pre-smoother) on level 2 
>> > ------------------------------->> >     KSP Object:    (mg_levels_2_)     
>> > 1 MPI processes>> >       type: richardson>> >         Richardson: damping 
>> > factor=1.>> >       maximum iterations=2>> >       tolerances:  
>> > relative=1e-05, absolute=1e-50, divergence=10000.>> >       left 
>> > preconditioning>> >       using nonzero initial guess>> >       using NONE 
>> > norm type for convergence test>> >     PC Object:    (mg_levels_2_)     1 
>> > MPI processes>> >       type: sor>> >         SOR: type = local_symmetric, 
>> > iterations = 1, local iterations = 1, omega = 1.>> >       linear system 
>> > matrix = precond matrix:>> >       Mat Object:       1 MPI processes>> >   
>> >       type: seqaij>> >         rows=256, cols=256>> >         total: 
>> > nonzeros=1248, allocated nonzeros=1248>> >         total number of mallocs 
>> > used during MatSetValues calls =0>> >           not using I-node 
>> > routines>> >   Up solver (post-smoother) same as down solver 
>> > (pre-smoother)>> >   Down solver (pre-smoother) on level 3 
>> > ------------------------------->> >     KSP Object:    (mg_levels_3_)     
>> > 1 MPI processes>> >       type: richardson>> >         Richardson: damping 
>> > factor=1.>> >       maximum iterations=2>> >       tolerances:  
>> > relative=1e-05, absolute=1e-50, divergence=10000.>> >       left 
>> > preconditioning>> >       using nonzero initial guess>> >       using NONE 
>> > norm type for convergence test>> >     PC Object:    (mg_levels_3_)     1 
>> > MPI processes>> >       type: sor>> >         SOR: type = local_symmetric, 
>> > iterations = 1, local iterations = 1, omega = 1.>> >       linear system 
>> > matrix = precond matrix:>> >       Mat Object:       1 MPI processes>> >   
>> >       type: seqaij>> >         rows=1024, cols=1024>> >         total: 
>> > nonzeros=5056, allocated nonzeros=5056>> >         total number of mallocs 
>> > used during MatSetValues calls =0>> >           has attached null space>> 
>> > >           not using I-node routines>> >   Up solver (post-smoother) same 
>> > as down solver (pre-smoother)>> >   linear system matrix = precond 
>> > matrix:>> >   Mat Object:   1 MPI processes>> >     type: seqaij>> >     
>> > rows=1024, cols=1024>> >     total: nonzeros=5056, allocated 
>> > nonzeros=5056>> >     total number of mallocs used during MatSetValues 
>> > calls =0>> >       has attached null space>> >       not using I-node 
>> > routines>> > >> > >> > Michele>> > >> > >> > >> > >> > On Wed, 2016-02-10 
>> > at 19:37 -0600, Matthew Knepley wrote:>> >> On Wed, Feb 10, 2016 at 7:33 
>> > PM, Michele Rosso <[email protected]> wrote:>> >> Hi,>> >> >> >> I 
>> > encountered the following error while solving a symmetric positive defined 
>> > system:>> >> >> >> Linear solve did not converge due to 
>> > DIVERGED_PCSETUP_FAILED iterations 0>> >>                PCSETUP_FAILED 
>> > due to SUBPC_ERROR >> >> >> >> This error appears only if I use the  
>> > optimized version of both petsc and my code ( compiler: gfortran, flags: 
>> > -O3 ).>> >> It is weird since I am solving  a time-dependent problem and 
>> > everything, i.e. results and convergence rate, are as expected until the 
>> > above error shows up. If I run both petsc and my code in debug mode, 
>> > everything goes smooth till the end of the simulation.>> >> However, if I 
>> > reduce the ksp_rtol, even the debug run fails, after running as expected 
>> > for a while, because of a KSP_DIVERGED_INDEFINITE_PC . >> >> The options I 
>> > am using are:>> >> >> >> -ksp_type        cg>> >> -ksp_norm_type   
>> > unpreconditioned>> >> -ksp_rtol        1e-8>> >> -ksp_lag_norm>> >> 
>> > -ksp_initial_guess_nonzero  yes>> >> -pc_type mg>> >> -pc_mg_galerkin>> >> 
>> > -pc_mg_levels 4>> >> -mg_levels_ksp_type richardson>> >> 
>> > -mg_coarse_ksp_constant_null_space>> >> -mg_coarse_pc_type lu>> >> 
>> > -mg_coarse_pc_factor_mat_solver_package superlu_dist>> >> -options_left>> 
>> > >> >> >> I attached a copy of ksp_view. I am currently using petsc-master 
>> > (last updated yesterday).>> >> I would appreciate any suggestion on this 
>> > matter.>> >> >> >> >> >> >> >> I suspect you have a nonlinear PC. Can you 
>> > send the output of -ksp_view?>> >> >> >> >> >>    Matt>> >>  >> >> 
>> > Thanks,>> >> Michele>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> 
>> > >> >> >> >> >> >> >> >> -->> >> What most experimenters take for granted 
>> > before they begin their experiments is infinitely more interesting than 
>> > any results to which their experiments lead.>> >> -- Norbert Wiener>> > >> 
>> > > >> >> >> >
>>
>>
>>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

Reply via email to