Alex,
Ahh, some over-zealous error checking used for the sparse case that is not
there otherwise.
You can edit src/dm/impls/da/fdda.c and remove the lines
if (bx == DM_BOUNDARY_PERIODIC && (m % col))
SETERRQ(PetscObjectComm((PetscObject)da),PETSC_ERR_SUP,"For coloring efficiency
ensure number of grid points in X is divisible\n\
by 2*stencil_width + 1\n");
if (by == DM_BOUNDARY_PERIODIC && (n % col))
SETERRQ(PetscObjectComm((PetscObject)da),PETSC_ERR_SUP,"For coloring efficiency
ensure number of grid points in Y is divisible\n\
by 2*stencil_width + 1\n");
if (bz == DM_BOUNDARY_PERIODIC && (p % col))
SETERRQ(PetscObjectComm((PetscObject)da),PETSC_ERR_SUP,"For coloring efficiency
ensure number of grid points in Z is divisible\n\
by 2*stencil_width + 1\n");
then do make libs in a PETSc directory.
Regarding the slowdown. What becomes slower? Creating the DM or somewhere
later in the code?
My guess is that what becomes slow is your matrix assembly process and this
happens because you provide the "wrong" sparsity pattern. That is your matrix
actually has some entries that are not represented in the sparsity pattern. It
will take forever to fill the matrix if this is the case. You can call
MatSetOption(mat,MAT_NEW_NONZERO_LOCATION_ERR,PETSC_TRUE) before putting values
into the matrix to see if that is the case.
Barry
> On Aug 13, 2020, at 9:16 PM, Alex Koshkarov <[email protected]> wrote:
>
> Hello All,
>
> I have a problem with DMDA matrix, I hope you can help. I am solving big PDE
> with TS object and my main data structure is 3D DMDA which has large number
> of degrees of freedom. I solve this system implicitly with JFNK. Essentially,
> I do something like this (J is jacobian, P is preconditioner):
>
> MatCreateSNESMF(snes,&J);
> DMCreateMatrix(da,&P);
> SNESSetJacobian(snes,J,P,form_P,env);
>
> In form_P I form P, but not J, and all works. I also sometimes use 3D DMDA
> for 2D and 1D problems, just setting Ny=Nz=1 and periodic boundary
> conditions. I have many degrees of freedom in my DMDA, so I need blocks to be
> sparse. So I tried to use DMDASetBlockFillsSparse, before DMCreateMatrix.
> However, it gives me an error connected that size of each dimension should be
> divisible by (2*stencil_size +1) to make efficient coloring, which makes
> using 3D DMDA for 2D problems not optimal. Moreover, it becomes super slow… I
> am not sure why it needs coloring. Essentially, everything works fine without
> DMDASetBlockFillsSparse, but once I use it, it complains and becomes slow. Do
> I do something wrong here? Do I need this coloring? All I need is more sparse
> DMDA matrix… I am somewhat lost.
>
> Thank you very much and best regards,
> Alex Koshkarov.
