Re: [petsc-users] A number of questions about DMDA with SNES and Quasi-Newton methods

[Smith, Barry F.]

> On Oct 31, 2017, at 11:59 AM, zakaryah . <zakar...@gmail.com> wrote:
> 
> First I set up the composite DM, which contains one redundant field coupled 
> to everything else, and a DMDA, which of course only has local couplings 
> given by the stencil.  I then create the matrix with DMCreateMatrix, with the 
> composite, packer, as the first argument.  I immediately call MatSetOption 
> with MAT_NEW_NONZERO_ALLOCATION_ERR set to FALSE.  The matrix values are set 
> by first calling MatGetLocalSubMatrix, and then MatSetValuesLocal.  
> Apparently this is working, but when I move to larger problems, setting the 
> off-diagonal submatrices is very slow, even when I set the entire submatrix 
> in a single call to MatSetValuesLocal.  I call the submatrices Jbb, Jbh, Jhb, 
> and Jhh, where b is the redundant field and h are the displacements in the 
> DMDA.  Then Jbh is a 1xN matrix where N is the size of the DMDA, and all of 
> its values are non-zero.  When N is around 3e6, setting the values in Jbh 
> takes several hours, even when I only call MatSetValuesLocal on it once.
> 
> I think the preallocation for the DMDA is working well, because if I forego 
> setting the "off-diagonal" submatrices Jbh and Jhb, then setting Jhh is very 
> fast (seconds).  I assume it's the allocation on the fly that is killing 
> performance, because Jhh has 57 times as many non-zero elements as Jbh and 
> Jhb, but setting it is several orders of magnitudes FASTER.
>