Re: [petsc-users] Question about DMDAGetElements

[Mark Adams via petsc-users]

The local indices of the local mesh and local vectors, which includes ghost
vertices. There are global-to-local methods to fill in ghost values and
local-to-global methods to create global vectors that you can use for
computation.

On Mon, Nov 19, 2018 at 5:16 PM Sajid Ali 
wrote:

> Bingo!
>
> So, DMDAGetElements gives the indices of the mesh, right ?
>
>
> Thank you !
>

Re: [petsc-users] Question about DMDAGetElements

[Sajid Ali via petsc-users]

Bingo!

So, DMDAGetElements gives the indices of the mesh, right ?


Thank you !

Re: [petsc-users] Question about DMDAGetElements

[Mark Adams via petsc-users]

You seem to be confusing the degree of the mesh and the "degree" of the
matrix and vector.

A matrix is always N x M (2D if you like), a vector is always N (or 1 x N,
or 1D if you like).

The mesh in a DM or DA can be 1, 2 or 3D.

On Mon, Nov 19, 2018 at 4:44 PM Sajid Ali via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> So, DMDA is used for sparse matrices arising from FD/FE and
> MatCreateMPIAIJ can be used for dense matrices (though it is strongly
> discouraged).
>
> My confusion stemmed from DMDAGetElements giving the element indices for
> the 1D mesh/vector of size N(when the DMDACreate1d is used). But these
> indices were then used as local indices to set the values for a 2D matrix
> (via MatSetValuesLocal ) created using the same DM (now we have NxN
> elements).
>
>

Re: [petsc-users] Question about DMDAGetElements

[Sajid Ali via petsc-users]

So, DMDA is used for sparse matrices arising from FD/FE and MatCreateMPIAIJ
can be used for dense matrices (though it is strongly discouraged).

My confusion stemmed from DMDAGetElements giving the element indices for
the 1D mesh/vector of size N(when the DMDACreate1d is used). But these
indices were then used as local indices to set the values for a 2D matrix
(via MatSetValuesLocal ) created using the same DM (now we have NxN
elements).

Re: [petsc-users] Question about DMDAGetElements

[Smith, Barry F. via petsc-users]

> On Nov 19, 2018, at 3:30 PM, Sajid Ali  
> wrote:
> 
> I think what confused me was the fact that using DMCreateMatrix(da,) 
> created a 12x12 matrix with 144 elements but summing up nel*nen from each 
> rank gives only 2*2+3*2+3*2+3*2=20 elements. So this means that 
> DMDAGetElements returns the elements for the vector created on the mesh and 
> this happens to be used for indexing the matrix (created using the same DM 
> object) via the local indices. Luckily since this code just creates a 
> tridiagonal matrix, this works here (since if I wanted to create a dense 
> matrix I'd want to have access to all the indices which would happen if the 
> addressable indices at each rank (for the submatrix stored locally) add up to 
> the size of the total matrix).

   I don't understand. This code would not work for a dense matrix where each 
process can set all the values via a local index. It only works when each 
process sets values for its local and ghost points (where the ghost points are 
defined by the stencil).   DMDA is not intended for problems with completely 
coupling of unknowns (that is dense matrices). It is meant for matrices arising 
from finite differences or finite elements.

  Barry


> Is my understanding correct? 
> 
> Thanks for the help! 
> 
> -- 
> Sajid Ali
> Applied Physics
> Northwestern University

Re: [petsc-users] Question about DMDAGetElements

[Sajid Ali via petsc-users]

I think what confused me was the fact that using DMCreateMatrix(da,)
created a 12x12 matrix with 144 elements but summing up nel*nen from each
rank gives only 2*2+3*2+3*2+3*2=20 elements. So this means that
DMDAGetElements returns the elements for the vector created on the mesh and
this happens to be used for indexing the matrix (created using the same DM
object) via the local indices. Luckily since this code just creates a
tridiagonal matrix, this works here (since if I wanted to create a dense
matrix I'd want to have access to all the indices which would happen if the
addressable indices at each rank (for the submatrix stored locally) add up
to the size of the total matrix). Is my understanding correct?

Thanks for the help!

-- 
Sajid Ali
Applied Physics
Northwestern University

Re: [petsc-users] Question about DMDAGetElements

[Smith, Barry F. via petsc-users]

> On Nov 13, 2018, at 3:03 PM, Jed Brown via petsc-users 
>  wrote:
> 
> Sajid Ali  writes:
> 
>> I'm still confused and have the following questions :
>> 
>> 1) Suppose as in the case above a DM object (created using DMCreate1D) is
>> used to created a matrix A using DMCreateMatrix, how does one convert the
>> indices obtained from DMDAGetElements to the row and column indices for the
>> matrix A ? Is there a function telling me exactly which sub-matrix each
>> rank is storing ?
> 
> Submatrix?
> 
>> 2) From the ex6.c example above, it looks like e corresponds to row index
>> and nen is the number of columns and i is the column index (which runs from
>> 0 to nel) when addressing the matrix elements via MatSetValuesLocal. Is
>> this correct?
> 
> Yes, that's how local indices are used.  It's also referenced in the man
> page.
> 
>> 3) How would the indices obtained from DMDAGetElements correspond to the
>> indices for a vector created using DMCreateGlobalVector ?
> 
> They don't; you want a local vector.

   Similar to the MatSetValuesLocal() there is a VecSetValuesLocal() that can 
be used with a vector obtained with DMCreateGlobalVector() to assemble the load.

   Barry

Re: [petsc-users] Question about DMDAGetElements

[Jed Brown via petsc-users]

Sajid Ali  writes:

> I'm still confused and have the following questions :
>
> 1) Suppose as in the case above a DM object (created using DMCreate1D) is
> used to created a matrix A using DMCreateMatrix, how does one convert the
> indices obtained from DMDAGetElements to the row and column indices for the
> matrix A ? Is there a function telling me exactly which sub-matrix each
> rank is storing ?

Submatrix?

> 2) From the ex6.c example above, it looks like e corresponds to row index
> and nen is the number of columns and i is the column index (which runs from
> 0 to nel) when addressing the matrix elements via MatSetValuesLocal. Is
> this correct?

Yes, that's how local indices are used.  It's also referenced in the man
page.

> 3) How would the indices obtained from DMDAGetElements correspond to the
> indices for a vector created using DMCreateGlobalVector ?

They don't; you want a local vector.

Re: [petsc-users] Question about DMDAGetElements

[Sajid Ali via petsc-users]

I'm still confused and have the following questions :

1) Suppose as in the case above a DM object (created using DMCreate1D) is
used to created a matrix A using DMCreateMatrix, how does one convert the
indices obtained from DMDAGetElements to the row and column indices for the
matrix A ? Is there a function telling me exactly which sub-matrix each
rank is storing ?

2) From the ex6.c example above, it looks like e corresponds to row index
and nen is the number of columns and i is the column index (which runs from
0 to nel) when addressing the matrix elements via MatSetValuesLocal. Is
this correct?

3) How would the indices obtained from DMDAGetElements correspond to the
indices for a vector created using DMCreateGlobalVector ?

Thank You,
Sajid Ali
Applied Physics
Northwestern University

Re: [petsc-users] Question about DMDAGetElements

[Jed Brown via petsc-users]

Sajid Ali via petsc-users  writes:

> Hi,
>
> I'm trying to understand this example
> 
> from a tutorial. DM is used as an alternative to setting up the matrix and
> vector separately (as was done in the previous example
> ).
>
>
> Since DMDACreate1d was used to create the mesh, can someone explain this
> logic more clearly :
>
>   /*
> Gets an array containing the indices (in local coordinates)
> of all the local elements.
> nel   - number of local elements
> nen   - number of one element's nodes
> e - the local indices of the elements' vertices
>   */
>   ierr = DMDAGetElements(da, , , );CHKERRQ(ierr);
>
>   /*
>  Assemble matrix and RHS
>   */
>   value[0] = 1.0; value[1] = -1.0; value[2] = -1.0; value[3] = 1.0;
>   bvalue[0] = 1.0; bvalue[1] = 1.0;
>   for (i=0; i ierr = MatSetValuesLocal(A, 2, e+nen*i, 2, e+nen*i, value,
> ADD_VALUES);CHKERRQ(ierr);
> //TODO use VecSetValuesLocal() to set the values
>   }
>   ierr = DMDARestoreElements(da, , , );CHKERRQ(ierr);
>
>
> The dm was created using
>
>   ierr = MPI_Allreduce(, , 1, MPI_INT, MPI_SUM,
> PETSC_COMM_WORLD);CHKERRQ(ierr);
>   ierr = 
> DMDACreate1d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,N,1,1,NULL,);CHKERRQ(ierr);

So each process owns 3 vertices.  The first process owns two elements,
then each subsequent process gets three elements.  Only the first
nel*nen entries in that array are valid.  nen=2 in all cases because
these are 1D linear elements.

Use DMPlex if you want more general finite element mesh handling.

>
> Using n = 3 and I get the following results when I print the values of
> nel,nen and e using this statement
> PetscPrintf(PETSC_COMM_SELF,"nel : %d,nen : %d, e elements :
> %d,%d,%d,%d,%d,%d,%d,%d ,rank
> %d\n",nel,nen,e[0],e[1],e[2],e[3],e[4],e[5],e[6],e[7],rank);
>
> The output comes out as  :
>
> [sajid@xrm exercises]$ mpirun -np 4 ./ex6
> nel : 2,nen : 2, e elements : 0,1,1,2,1936617321,0,3,0 ,rank 0
> nel : 3,nen : 2, e elements : 0,1,1,2,2,3,4,0 ,rank 1
> nel : 3,nen : 2, e elements : 0,1,1,2,2,3,4,0 ,rank 2
> nel : 3,nen : 2, e elements : 0,1,1,2,2,3,3,0 ,rank 3
>
> Does this mean that I'm supposed to read the first 4 elements for rank 0 as
> row0, col0, row1, col1?
>
> And how does differ If I'm creating a vector using DMCreate1d ?
>
>
> Thank You,
> Sajid Ali
> Applied Physics
> Northwestern University

[petsc-users] Question about DMDAGetElements

[Sajid Ali via petsc-users]

Hi,

I'm trying to understand this example

from a tutorial. DM is used as an alternative to setting up the matrix and
vector separately (as was done in the previous example
).


Since DMDACreate1d was used to create the mesh, can someone explain this
logic more clearly :

  /*
Gets an array containing the indices (in local coordinates)
of all the local elements.
nel - number of local elements
nen - number of one element's nodes
e   - the local indices of the elements' vertices
  */
  ierr = DMDAGetElements(da, , , );CHKERRQ(ierr);

  /*
 Assemble matrix and RHS
  */
  value[0] = 1.0; value[1] = -1.0; value[2] = -1.0; value[3] = 1.0;
  bvalue[0] = 1.0; bvalue[1] = 1.0;
  for (i=0; i