Re: [petsc-users] Patching in generalized eigen value problems

[Matthew Knepley]

On Sun, Apr 9, 2017 at 2:47 PM, Daralagodu Dattatreya Jois, Sathwik Bharadw
 wrote:

> Dear Barry,
>
>
> These operations comes into picture when I have to map a
>
> plane surface to a closed surface (lets say cylinder). As you
>
> can imagine nodes (hence nodal values) at 2 opposite sides
>
> of the plane have to add up to give a closed geometry.
>
> Instead of assembling both, why not just make one row into constraints,
namely
1 for the primal node and -1 for the host node. Then you do not need
communication,
and its sparser. IMHO, these "extra" nodes should just be eliminated from
the system.

   Matt

> Matrices can be as large as 30,000*30,000 or more depending
>
> on the density of the mesh. Since, in effect different elements sits in
>
> different processes doing this in assembly level will be tricky.
>
>
> Sathwik Bharadwaj
> --
> *From:* Barry Smith 
> *Sent:* Sunday, April 9, 2017 3:34:13 PM
> *To:* Daralagodu Dattatreya Jois, Sathwik Bharadw
> *Cc:* petsc-users@mcs.anl.gov
> *Subject:* Re: [petsc-users] Patching in generalized eigen value problems
>
>
> > On Apr 9, 2017, at 2:21 PM, Daralagodu Dattatreya Jois, Sathwik Bharadw <
> sdaralagoduda...@wpi.edu> wrote:
> >
> > Dear petsc users,
> >
> > I am solving for generalized eigen value problems using petsc and slepc.
> > Our equation will be of the form,
> >
> > A X=λ B X.
> >
> > I am constructing the A and B matrix of type MATMPIAIJ. Let us consider
> that
> > both of my matrices are of dimension 10*10. When we are solving for a
> closed
> > geometry, we require to add all the entries of the last (9th) row and
> column to
> > the first (0th) row and column respectively for both matrices. In a high
> density mesh,
> > I will have a large number of such row to row and column to column
> additions.
> > For example, I may have to add last 200 rows and columns to first 200
> rows and columns
> > respectively. We will then zero the copied row and column expect the
> diagonal
> > element (9th row/column in the former case).
>
>Where is this "strange" operation coming from?
>
>Boundary conditions?
>
>   Is there any way to assemble matrices initially with these sums instead
> of doing it after the fact?
>
>  Why is it always the "last rows" and the "first rows"?
>
>   What happens when you run in parallel where first and last rows are on
> different processes?
>
>   How large will the matrices get?
>
>Are the matrices symmetric?
>
>
>
>
> >
> > I understand that MatGetRow, MatGetColumnVector, MatGetValues or any
> other
> > MatGet- or VecGet- functions are not collective. Can you suggest any
> > efficient algorithm or function to achieve this way of patching?
> >
> > One way I can think of is to obtain the column vector using
> MatGetColumnVector and
> > row vector by MatZeroRows and then scatter these vectors to all
> processes. Once we have
> > entire row/column vector entries in each process, we can add the values
> to the matrix by
> > their global index. Of course, care should be taken to add the values of
> diagonal element
> > only once. But this will be a quite slow process.
> > Any ideas are appreciated.
> >
> > Thanks,
> > Sathwik Bharadwaj
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

Re: [petsc-users] Patching in generalized eigen value problems

[Daralagodu Dattatreya Jois, Sathwik Bharadw]

Dear Barry,


These operations comes into picture when I have to map a

plane surface to a closed surface (lets say cylinder). As you

can imagine nodes (hence nodal values) at 2 opposite sides

of the plane have to add up to give a closed geometry.

Matrices can be as large as 30,000*30,000 or more depending

on the density of the mesh. Since, in effect different elements sits in

different processes doing this in assembly level will be tricky.


Sathwik Bharadwaj

From: Barry Smith 
Sent: Sunday, April 9, 2017 3:34:13 PM
To: Daralagodu Dattatreya Jois, Sathwik Bharadw
Cc: petsc-users@mcs.anl.gov
Subject: Re: [petsc-users] Patching in generalized eigen value problems


> On Apr 9, 2017, at 2:21 PM, Daralagodu Dattatreya Jois, Sathwik Bharadw 
>  wrote:
>
> Dear petsc users,
>
> I am solving for generalized eigen value problems using petsc and slepc.
> Our equation will be of the form,
>
> A X=λ B X.
>
> I am constructing the A and B matrix of type MATMPIAIJ. Let us consider that
> both of my matrices are of dimension 10*10. When we are solving for a closed
> geometry, we require to add all the entries of the last (9th) row and column 
> to
> the first (0th) row and column respectively for both matrices. In a high 
> density mesh,
> I will have a large number of such row to row and column to column additions.
> For example, I may have to add last 200 rows and columns to first 200 rows 
> and columns
> respectively. We will then zero the copied row and column expect the diagonal
> element (9th row/column in the former case).

   Where is this "strange" operation coming from?

   Boundary conditions?

  Is there any way to assemble matrices initially with these sums instead of 
doing it after the fact?

 Why is it always the "last rows" and the "first rows"?

  What happens when you run in parallel where first and last rows are on 
different processes?

  How large will the matrices get?

   Are the matrices symmetric?




>
> I understand that MatGetRow, MatGetColumnVector, MatGetValues or any other
> MatGet- or VecGet- functions are not collective. Can you suggest any
> efficient algorithm or function to achieve this way of patching?
>
> One way I can think of is to obtain the column vector using 
> MatGetColumnVector and
> row vector by MatZeroRows and then scatter these vectors to all processes. 
> Once we have
> entire row/column vector entries in each process, we can add the values to 
> the matrix by
> their global index. Of course, care should be taken to add the values of 
> diagonal element
> only once. But this will be a quite slow process.
> Any ideas are appreciated.
>
> Thanks,
> Sathwik Bharadwaj

Re: [petsc-users] Patching in generalized eigen value problems

[Barry Smith]

> On Apr 9, 2017, at 2:21 PM, Daralagodu Dattatreya Jois, Sathwik Bharadw 
>  wrote:
> 
> Dear petsc users,
> 
> I am solving for generalized eigen value problems using petsc and slepc. 
> Our equation will be of the form, 
> 
> A X=λ B X. 
> 
> I am constructing the A and B matrix of type MATMPIAIJ. Let us consider that 
> both of my matrices are of dimension 10*10. When we are solving for a closed 
> geometry, we require to add all the entries of the last (9th) row and column 
> to 
> the first (0th) row and column respectively for both matrices. In a high 
> density mesh, 
> I will have a large number of such row to row and column to column additions. 
> For example, I may have to add last 200 rows and columns to first 200 rows 
> and columns
> respectively. We will then zero the copied row and column expect the diagonal
> element (9th row/column in the former case). 

   Where is this "strange" operation coming from?

   Boundary conditions? 

  Is there any way to assemble matrices initially with these sums instead of 
doing it after the fact? 

 Why is it always the "last rows" and the "first rows"? 

  What happens when you run in parallel where first and last rows are on 
different processes? 

  How large will the matrices get?

   Are the matrices symmetric?




> 
> I understand that MatGetRow, MatGetColumnVector, MatGetValues or any other 
> MatGet- or VecGet- functions are not collective. Can you suggest any
> efficient algorithm or function to achieve this way of patching? 
> 
> One way I can think of is to obtain the column vector using 
> MatGetColumnVector and
> row vector by MatZeroRows and then scatter these vectors to all processes. 
> Once we have 
> entire row/column vector entries in each process, we can add the values to 
> the matrix by 
> their global index. Of course, care should be taken to add the values of 
> diagonal element 
> only once. But this will be a quite slow process. 
> Any ideas are appreciated. 
>  
> Thanks,
> Sathwik Bharadwaj

[petsc-users] Patching in generalized eigen value problems

[Daralagodu Dattatreya Jois, Sathwik Bharadw]

Dear petsc users,


I am solving for generalized eigen value problems using petsc and slepc.

Our equation will be of the form,


A X=λ B X.


I am constructing the A and B matrix of type MATMPIAIJ. Let us consider that

both of my matrices are of dimension 10*10. When we are solving for a closed

geometry, we require to add all the entries of the last (9th) row and column to

the first (0th) row and column respectively for both matrices. In a high 
density mesh,

I will have a large number of such row to row and column to column additions.

For example, I may have to add last 200 rows and columns to first 200 rows and 
columns

respectively. We will then zero the copied row and column expect the diagonal

element (9th row/column in the former case).


I understand that MatGetRow, MatGetColumnVector, MatGetValues or any other

MatGet- or VecGet- functions are not collective. Can you suggest any

efficient algorithm or function to achieve this way of patching?


One way I can think of is to obtain the column vector using MatGetColumnVector 
and

row vector by MatZeroRows and then scatter these vectors to all processes. Once 
we have

entire row/column vector entries in each process, we can add the values to the 
matrix by

their global index. Of course, care should be taken to add the values of 
diagonal element

only once. But this will be a quite slow process.

Any ideas are appreciated.



Thanks,

Sathwik Bharadwaj

Re: [petsc-users] GAMG for the unsymmetrical matrix

[Mark Adams]

You seem to have two levels here and 3M eqs on the fine grid and 37 on
the coarse grid. I don't understand that.

You are also calling the AMG setup a lot, but not spending much time
in it. Try running with -info and grep on "GAMG".


On Fri, Apr 7, 2017 at 5:29 PM, Kong, Fande  wrote:
> Thanks, Barry.
>
> It works.
>
> GAMG is three times better than ASM in terms of the number of linear
> iterations, but it is five times slower than ASM. Any suggestions to improve
> the performance of GAMG? Log files are attached.
>
> Fande,
>
> On Thu, Apr 6, 2017 at 3:39 PM, Barry Smith  wrote:
>>
>>
>> > On Apr 6, 2017, at 9:39 AM, Kong, Fande  wrote:
>> >
>> > Thanks, Mark and Barry,
>> >
>> > It works pretty wells in terms of the number of linear iterations (using
>> > "-pc_gamg_sym_graph true"), but it is horrible in the compute time. I am
>> > using the two-level method via "-pc_mg_levels 2". The reason why the 
>> > compute
>> > time is larger than other preconditioning options is that a matrix free
>> > method is used in the fine level and in my particular problem the function
>> > evaluation is expensive.
>> >
>> > I am using "-snes_mf_operator 1" to turn on the Jacobian-free Newton,
>> > but I do not think I want to make the preconditioning part matrix-free.  Do
>> > you guys know how to turn off the matrix-free method for GAMG?
>>
>>-pc_use_amat false
>>
>> >
>> > Here is the detailed solver:
>> >
>> > SNES Object: 384 MPI processes
>> >   type: newtonls
>> >   maximum iterations=200, maximum function evaluations=1
>> >   tolerances: relative=1e-08, absolute=1e-08, solution=1e-50
>> >   total number of linear solver iterations=20
>> >   total number of function evaluations=166
>> >   norm schedule ALWAYS
>> >   SNESLineSearch Object:   384 MPI processes
>> > type: bt
>> >   interpolation: cubic
>> >   alpha=1.00e-04
>> > maxstep=1.00e+08, minlambda=1.00e-12
>> > tolerances: relative=1.00e-08, absolute=1.00e-15,
>> > lambda=1.00e-08
>> > maximum iterations=40
>> >   KSP Object:   384 MPI processes
>> > type: gmres
>> >   GMRES: restart=100, using Classical (unmodified) Gram-Schmidt
>> > Orthogonalization with no iterative refinement
>> >   GMRES: happy breakdown tolerance 1e-30
>> > maximum iterations=100, initial guess is zero
>> > tolerances:  relative=0.001, absolute=1e-50, divergence=1.
>> > right preconditioning
>> > using UNPRECONDITIONED norm type for convergence test
>> >   PC Object:   384 MPI processes
>> > type: gamg
>> >   MG: type is MULTIPLICATIVE, levels=2 cycles=v
>> > Cycles per PCApply=1
>> > Using Galerkin computed coarse grid matrices
>> > GAMG specific options
>> >   Threshold for dropping small values from graph 0.
>> >   AGG specific options
>> > Symmetric graph true
>> > Coarse grid solver -- level ---
>> >   KSP Object:  (mg_coarse_)   384 MPI processes
>> > type: preonly
>> > maximum iterations=1, initial guess is zero
>> > tolerances:  relative=1e-05, absolute=1e-50, divergence=1.
>> > left preconditioning
>> > using NONE norm type for convergence test
>> >   PC Object:  (mg_coarse_)   384 MPI processes
>> > type: bjacobi
>> >   block Jacobi: number of blocks = 384
>> >   Local solve is same for all blocks, in the following KSP and
>> > PC objects:
>> > KSP Object:(mg_coarse_sub_) 1 MPI processes
>> >   type: preonly
>> >   maximum iterations=1, initial guess is zero
>> >   tolerances:  relative=1e-05, absolute=1e-50, divergence=1.
>> >   left preconditioning
>> >   using NONE norm type for convergence test
>> > PC Object:(mg_coarse_sub_) 1 MPI processes
>> >   type: lu
>> > LU: out-of-place factorization
>> > tolerance for zero pivot 2.22045e-14
>> > using diagonal shift on blocks to prevent zero pivot
>> > [INBLOCKS]
>> > matrix ordering: nd
>> > factor fill ratio given 5., needed 1.31367
>> >   Factored matrix follows:
>> > Mat Object: 1 MPI processes
>> >   type: seqaij
>> >   rows=37, cols=37
>> >   package used to perform factorization: petsc
>> >   total: nonzeros=913, allocated nonzeros=913
>> >   total number of mallocs used during MatSetValues calls
>> > =0
>> > not using I-node routines
>> >   linear system matrix = precond matrix:
>> >   Mat Object:   1 MPI processes
>> > type: seqaij
>> > rows=37, cols=37
>> > total: nonzeros=695, allocated nonzeros=695
>> > total number