Re: [petsc-users] nest matrix and the corresponding vector format

[Matthew Knepley]

On Mon, Aug 6, 2018 at 5:06 PM Mike Wick 
wrote:

> Hi PETSc team:
>
> I have a question regarding the compatibility between the nest matrix and
> the vec object. My understanding is that one should create and use nest
> vectors when the matrix is in the nest form.
>
> In the tutorial example
>
> http://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex70.c.html
>
> I noticed that the vectors are created as regular MPI vectors, with the
> local partition left for PETSC_DECIDE, in lines 171-173. I wander if the
> underlying structure of the vectors are compatible with the nest matrix. If
> not, will that create some unnecessary communication work?
>
> Look forward to your suggestions.
>

Nest matrices work with any vector.

  Thanks,

   Matt


> Best,
>
> Mike
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

[petsc-users] nest matrix and the corresponding vector format

[Mike Wick]

Hi PETSc team:

I have a question regarding the compatibility between the nest matrix and
the vec object. My understanding is that one should create and use nest
vectors when the matrix is in the nest form.

In the tutorial example
http://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex70.c.html

I noticed that the vectors are created as regular MPI vectors, with the
local partition left for PETSC_DECIDE, in lines 171-173. I wander if the
underlying structure of the vectors are compatible with the nest matrix. If
not, will that create some unnecessary communication work?

Look forward to your suggestions.

Best,

Mike

Re: [petsc-users] large number of iteration.

[Smith, Barry F.]

Ok, I think you want to use PCFIELDSPLIT with two fields u and c. Start by 
using PC_COMPOSITE_MULTIPLICATIVE with direct solvers (LU) for each field. Then 
try replacing the direct solvers LU with hypre BoomerAMG first one at a time. 

Let us know how it goes and we'll make additional suggestions,

Barry


> On Aug 6, 2018, at 12:58 PM, Josh L  wrote:
> 
> Hi Barry,
> 
> (1)
> I am basically solving the equilibrium coupled with phase field equation:
> 
> 1-D example
> u is displacement and c is phase-field parameter. 
> 
> equilibrium: (c^2)*E* (\Delta u) = 0
> 
> phase-field equation:  2*G_c*L*(\Delta c)-E*(\epsilon)^2*c+G_c/2L(1-c)=0
> 
> I solve the above 2 equations coupled. 
> it is the formulation for quasi-static fracture problem, where G_c is the 
> critical energy release rate and L is the length scale in phase-field method 
> (both constant).
> 
> The free energy I am using now is 
> 1-D:
> 
> \psi=c^2*0.5*E*\epsilon^2+G_c/4L*(1-c)^2+G_c*L*grad c * grad c
> 
> 
> it is not convex w.r.t u and c simultaneously. 
> 
> (2)
> 
> with an appropriate step size. Newton's method usually converges with 5 
> iteration, and I can see the quadratic convergence of Newton's method.
> 
> 
> 
> 
> Thanks,
> Josh
> 
> 
> 
> 2018-08-05 22:23 GMT-05:00 Smith, Barry F. :
> 
> 
> > On Aug 5, 2018, at 9:38 PM, Josh L  wrote:
> > 
> > Hi,
> > 
> > I am using GRMES to solve the linear systems within Newton Raphson. 
> > I am using finite element method to solve fracture problem with phase field 
> > method.(the formulation gives symmetric but indefinite hessian so can't use 
> > CG) 
> 
> What exact equations are you using for the phase field method?
> 
>  With the direct LU solver roughly how many iterations does your Newton's 
> method take?
> 
> 
>Barry
> 
> > 
> > The total iteration number to solve one linear system is often very large( 
> > > 2000) even for a small sized problem(120K degree of freedom). I've tried 
> > different preconditioner, but it doesn't help a lot.  I have checked my 
> > hessian and right hand side, and sure they are correct. 
> > 
> > I think it might stem from the great difference in element 
> > size(max/min=100) and damage stiffness around crack. 
> > 
> > Anyone has idea to reduce the number of iteration?
> > 
> > Now, I get away with it by using LU, which is not ideal for large size 
> > problem (I will have to run it in the future)
> > 
> > 
> > Thanks,
> > Josh
> > 
> 
> 

Re: [petsc-users] large number of iteration.

[Josh L]

Hi Barry,

(1)
I am basically solving the equilibrium coupled with phase field equation:

1-D example
u is displacement and c is phase-field parameter.

equilibrium: (c^2)*E* (\Delta u) = 0

phase-field equation:  2*G_c*L*(\Delta c)-E*(\epsilon)^2*c+G_c/2L(1-c)=0

I solve the above 2 equations coupled.
it is the formulation for quasi-static fracture problem, where G_c is the
critical energy release rate and L is the length scale in phase-field
method (both constant).

The free energy I am using now is
1-D:

\psi=c^2*0.5*E*\epsilon^2+G_c/4L*(1-c)^2+G_c*L*grad c * grad c


it is not convex w.r.t u and c simultaneously.

(2)

with an appropriate step size. Newton's method usually converges with 5
iteration, and I can see the quadratic convergence of Newton's method.




Thanks,
Josh



2018-08-05 22:23 GMT-05:00 Smith, Barry F. :

>
>
> > On Aug 5, 2018, at 9:38 PM, Josh L  wrote:
> >
> > Hi,
> >
> > I am using GRMES to solve the linear systems within Newton Raphson.
> > I am using finite element method to solve fracture problem with phase
> field method.(the formulation gives symmetric but indefinite hessian so
> can't use CG)
>
> What exact equations are you using for the phase field method?
>
>  With the direct LU solver roughly how many iterations does your
> Newton's method take?
>
>
>Barry
>
> >
> > The total iteration number to solve one linear system is often very
> large( > 2000) even for a small sized problem(120K degree of freedom). I've
> tried different preconditioner, but it doesn't help a lot.  I have checked
> my hessian and right hand side, and sure they are correct.
> >
> > I think it might stem from the great difference in element
> size(max/min=100) and damage stiffness around crack.
> >
> > Anyone has idea to reduce the number of iteration?
> >
> > Now, I get away with it by using LU, which is not ideal for large size
> problem (I will have to run it in the future)
> >
> >
> > Thanks,
> > Josh
> >
>
>

Re: [petsc-users] memory corruption when using harmonic extraction with SLEPc

[Dave May]

Please always use "reply-all" so that your messages go to the list.
This is standard mailing list etiquette.  It is important to preserve
threading for people who find this discussion later and so that we do
not waste our time re-answering the same questions that have already
been answered in private side-conversations.  You'll likely get an
answer faster that way too.




On 6 August 2018 at 17:17, Moritz Cygorek  wrote:

> Hi,
>
>
> I have found that the memory corruption takes place at the end of the
> first iteration or the beginning of the second iteration, i.e., at a time
> where I expect the harmonic extraction part to do some calculations.
>

Where is the report from valgrind indicating the exact file/function/line
where the problem occurred?



> I've then played around with the command line options and I found that
> harmonic extraction works when -eps_ncv is set to smaller values.
>
> I have the feeling that the memory needs to sustain this number of vectors
> and the eigenvectors for all processes individually. If I run multiple
> processes on a single computer, much more memory is required for storage
> than when I only use a single process, eventually allocating all of the
> available memory
>
>
> Do you know if this behavior of the harmonic extraction routine
> intended/necessary?
>
>
> Regards,
> Moritz
>
>
>
>
>
>
>
> --
> *From:* Dave May 
> *Sent:* Friday, August 3, 2018 12:59:54 AM
> *To:* Moritz Cygorek
> *Cc:* petsc-users@mcs.anl.gov
> *Subject:* Re: [petsc-users] memory corruption when using harmonic
> extraction with SLEPc
>
> On Thu, 2 Aug 2018 at 21:32, Moritz Cygorek  wrote:
>
>> Hi,
>>
>>
>> I want to diagonalize a huge sparse matrix and I'm using the Kryov-Schur
>> method with harmonic extraction (command line option -eps_harmonic )
>> implemented in SLEPc.
>>
>>
>> I manually distribute a sparse matrix across several CPUs and everything
>> works fine when:
>>
>> - I do _not_ use harmonic extraction
>>
>> - I use harmonic extraction on only a single CPU
>>
>>
>> If I try do use harmonic extraction on multiple CPUs, I get a memory
>> corruption.
>>
>> I'm not quite sure where to look at, but somewhere in the output, I find:
>>
>>
>>
>> [1]PETSC ERROR: PetscMallocValidate: error detected at
>> PetscSignalHandlerDefault() line 145 in /home/applications/sources/
>> libraries/petsc-3.9.3/src/sys/error/signal.c
>> [1]PETSC ERROR: Memory [id=0(9072)] at address 0x145bcd0 is corrupted
>> (probably write past end of array)
>> [1]PETSC ERROR: Memory originally allocated in DSAllocateWork_Private()
>> line 74 in /home/applications/sources/libraries/slepc-3.9.2/src/sys/
>> classes/ds/interface/dspriv.c
>>
>>
>> Now, I have the feeling that this might be a bug in SLEPc because, if I
>> had messed up the matrix initialization and distribution, I should also get
>> a memory corruption when I don't use harmonic extraction, right?
>>
>
> Not necessarily
>
>
>> Any suggestions what might be going on?
>>
>
> Run your code through valgrind and make sure that your application code is
> clean. See here
>
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>
> If errors are detected in your application code, fix them and see if the
> slepc errors go away. If your code is valgrind clean, send through the
> relevant chunk of the valgrind report indicating exactly where the error is
> occurring
>
> Thanks,
>   Dave
>
>
>
>> Regards,
>>
>> Moritz
>>
>>
>>
>>
>>
>>
>>
>>

Re: [petsc-users] SLEPc - Davidson-type solvers options

[Jose E. Roman]

> El 6 ago 2018, a las 14:44, Michael Werner  escribió:
> 
> 
> Michael Werner writes:
> 
>> Hello, I want to use a Davidson-type solver (probably jd) to find the 
>> eigenvalues with the smallest real part, but so far I'm strugglung to get 
>> them to converge. So I was hoping to get some advice on the various  options 
>> available for those solvers.
>> 
>> For my test case, I know the smallest eigenvalue is -0.04+0.71i (calculated 
>> with shift-and-invert and lu preconditioner). However, when I try to use jd 
>> or gd, the eigensolver never converges. After a few iterations, the "first 
>> unconverged value" is more or less as expected, but it fluctuates around the 
>> correct value, and the residual never drops below 1e-2. The best results so 
>> far were achieved with the following set of commandline options:
>> 
>> -st_type precond -st_pc_type asm -st_sub_pc_type ilu  
>> -st_sub_pc_factor_levels 2 -st_ksp_max_it 15 -st_ksp_rtol 1e-2  -eps_type jd 
>> -eps_tol 1e-4 -eps_monitor -eps_nev 5 -eps_target  -0.08 -eps_target_real 
>> -eps_harmonic_largest
>> 
>> I also tried using more ksp iterations or different st_sub_pc_factor_levels. 
>> I also tried using eps_smallest_real instead of a target value, without any 
>> success. So far, I noticed that the extraction type had the largest 
>> influence. Only for eps_harmonic_largest could I observe any eigenvalues 
>> with negative real part, for all the other extraction types I only got 
>> eigenvalues with positive real parts.
>> 
>> I also had a look at the available options for the Davidson solvers, but I 
>> couldn't find a good explanation for 
> 
> Sorry, I didn't mean to send it already. What I wanted to say:
> 
> I also had a look at the available options for the Davidson solvers, but I 
> couldn't find a good explanation for many of the additional options. Is there 
> any recommendation to chose -eps_mpd or -eps_jd_minv? And can -eps_jd_plusk 
> be used for GNHEP-type problems? I already had a look at the paper of E. 
> Romero and J. E. Roman concerning the implementation of the Davidson-type 
> solvers in SLEPc, but it didn't answer all my questions.
> 
> Kind regards,
> Michael

You may need to enforce more accuracy on the KSP. Try replacing
-st_ksp_max_it 15 -st_ksp_rtol 1e-2 
by
-st_ksp_rtol 1e-5

Normally, you don't need to set the -eps_mpd parameter, except if many 
eigenvalues (nev) are requested. The parameter jd_minv controls how the restart 
is done, keeping more or less information from previous restarts - it may 
affect convergece, but not too much. And yes, jd_plusk can be used for GNHEP 
problems.

Jose

Re: [petsc-users] SLEPc - Davidson-type solvers options

[Michael Werner]

Michael Werner writes:

Hello, I want to use a Davidson-type solver (probably jd) to 
find 
the eigenvalues with the smallest real part, but so far I'm 
strugglung to get them to converge. So I was hoping to get some 
advice on the various  options available for those solvers.


For my test case, I know the smallest eigenvalue is -0.04+0.71i 
(calculated with shift-and-invert and lu 
preconditioner). However, 
when I try to use jd or gd, the eigensolver never 
converges. After 
a few iterations, the "first unconverged value" is more or less 
as 
expected, but it fluctuates around the correct value, and the 
residual never drops below 1e-2. The best results so far were 
achieved with the following set of commandline options:


-st_type precond -st_pc_type asm -st_sub_pc_type ilu 
 -st_sub_pc_factor_levels 2 -st_ksp_max_it 15 -st_ksp_rtol 1e-2 
 -eps_type jd -eps_tol 1e-4 -eps_monitor -eps_nev 5 -eps_target 
 -0.08 -eps_target_real -eps_harmonic_largest


I also tried using more ksp iterations or different 
st_sub_pc_factor_levels. I also tried using eps_smallest_real 
instead of a target value, without any success. So far, I 
noticed 
that the extraction type had the largest influence. Only for 
eps_harmonic_largest could I observe any eigenvalues with 
negative 
real part, for all the other extraction types I only got 
eigenvalues with positive real parts.


I also had a look at the available options for the Davidson 
solvers, but I couldn't find a good explanation for 


Sorry, I didn't mean to send it already. What I wanted to say:

I also had a look at the available options for the Davidson 
solvers, but I couldn't find a good explanation for many of the 
additional options. Is there any recommendation to chose -eps_mpd 
or -eps_jd_minv? And can -eps_jd_plusk be used for GNHEP-type 
problems? I already had a look at the paper of E. Romero and 
J. E. Roman concerning the implementation of the Davidson-type 
solvers in SLEPc, but it didn't answer all my questions.


Kind regards,
Michael

[petsc-users] SLEPc - Davidson-type solvers options

[Michael Werner]

Hello, I want to use a Davidson-type solver (probably jd) to find 
the eigenvalues with the smallest real part, but so far I'm 
strugglung to get them to converge. So I was hoping to get some 
advice on the various  options available for those solvers.


For my test case, I know the smallest eigenvalue is -0.04+0.71i 
(calculated with shift-and-invert and lu preconditioner). However, 
when I try to use jd or gd, the eigensolver never converges. After 
a few iterations, the "first unconverged value" is more or less as 
expected, but it fluctuates around the correct value, and the 
residual never drops below 1e-2. The best results so far were 
achieved with the following set of commandline options:


-st_type precond -st_pc_type asm -st_sub_pc_type ilu 
-st_sub_pc_factor_levels 2 -st_ksp_max_it 15 -st_ksp_rtol 1e-2 
-eps_type jd -eps_tol 1e-4 -eps_monitor -eps_nev 5 -eps_target 
-0.08 -eps_target_real -eps_harmonic_largest


I also tried using more ksp iterations or different 
st_sub_pc_factor_levels. I also tried using eps_smallest_real 
instead of a target value, without any success. So far, I noticed 
that the extraction type had the largest influence. Only for 
eps_harmonic_largest could I observe any eigenvalues with negative 
real part, for all the other extraction types I only got 
eigenvalues with positive real parts.


I also had a look at the available options for the Davidson 
solvers, but I couldn't find a good explanation for 



--


Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR) 
Institut für Aerodynamik und Strömungstechnik | Bunsenstr. 10 | 
37073 Göttingen


Michael Werner
Telefon 0551 709-2627 | Telefax 0551 709-2811 | 
michael.wer...@dlr.de

DLR.de