On Jan 15, 2014, at 4:44 PM, Matthew Knepley <[email protected]> wrote:
> On Wed, Jan 15, 2014 at 4:36 PM, Barry Smith <[email protected]> wrote: > > This problem comes up for MPIAIJ matrices when the “off-diagona” matrix is > compressed to only the columns that have non zeros in them. When a > MatDuplicate() is done on this it detects the inconsistency that the number > of local columns may no longer be divisible by the block size. We could > handle this by > > 1) when the matrix is compressed (removing empty columns) we just mark the > block size to 1 if the new number of local columns is not divisible by the > block size > > but then if the more values are put into the matrix when the matrix is > uncompressed it no longer has the block size > 1 (possibly hurting > efficiency) so we could > > 2) stash the old block size in the matrix and put it back when we uncompress > it. > > What do you think? Should I do 1)? 2)? > > 2) seems better, but I am worried about the complexity. I do not quite > understand the data structures either. I thought > B was just a MatSeqAIJ + column map. However, there is some > compression/decompression code I guess. What is > the compressed matrix, It is the same matrix, just the column indices are changed to shrink down on compress and go back up on non-compression. > and where is the compressed flag? It doesn’t actually have a flag, the off-diagonal is always compressed on assembly. I just looked again at the code and realized that 2) is actually already handled correctly. So I can fix 1) easily and will get to it soon. BTW: longer term issue is TS is keeping too many matrices around (memory hog). Barry > > Matt > > Barry > > Request-assigned: Barry fix MatDuplicate() problem on MPIAIJ matrices where > block size of off diagonal matrix is no longer valid when compressed > > Begin forwarded message: > > > From: "Jay J. Billings" <[email protected]> > > Subject: [petsc-users] Local size 7 not compatible with block size 13 > > Date: January 15, 2014 at 1:14:02 PM CST > > To: <[email protected]> > > > > Everyone, > > > > I'm trying to run PETSc in parallel and I'm getting an error I haven't seen > > before that the local and block size arguments don't match. Can anyone > > point me in the right direction on this? It happens right after I call > > TSSolve and, according to the backtrace, when TSSolve is trying to form the > > Jacobian. It doesn't look like it is actually calling my Jacobian routine, > > although I could be wrong about that. > > > > Jay > > > > [1]PETSC ERROR: --------------------- Error Message > > ------------------------------------ > > [1]PETSC ERROR: [0]PETSC ERROR: --------------------- Error Message > > ------------------------------------ > > [0]PETSC ERROR: Arguments are incompatible! > > [1]PETSC ERROR: Arguments are incompatible! > > [0]PETSC ERROR: Local size 7 not compatible with block size 13! > > [1]PETSC ERROR: Local size 7 not compatible with block size 13! > > [0]PETSC ERROR: > > ------------------------------------------------------------------------ > > [1]PETSC ERROR: Petsc Development GIT revision: v3.4.3-2305-g52cd893 GIT > > Date: 2014-01-13 09:37:41 -0500 > > [1]PETSC ERROR: See docs/changes/index.html for recent updates. > > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > [1]PETSC ERROR: See docs/index.html for manual pages. > > [1]PETSC ERROR: > > ------------------------------------------------------------------------ > > [1]PETSC ERROR: ./xolotl on a arch-linux2-c-opt named box.ornl.gov by jay > > Wed Jan 15 14:12:23 2014 > > [1]PETSC ERROR: Libraries linked from /opt/petsc-latest_mpich-3.0.1/lib > > [1]PETSC ERROR: Configure run at Mon Jan 13 11:11:18 2014 > > [1]PETSC ERROR: Configure options --prefix=/opt/petsc-latest_mpich-3.0.1 > > --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif77 --with-debugging=no > > --download-f-blas-lapack=1 --FOPTFLAGS= --with-shared-libraries=1 > > --download-hypre=yes --with-debugging=0 > > [1]PETSC ERROR: > > ------------------------------------------------------------------------ > > [1]PETSC ERROR: PetscLayoutSetBlockSize() line 464 in > > /home/jay/Programs/petsc/src/vec/is/utils/pmap.c > > [1]PETSC ERROR: MatSetBlockSizes() line 6772 in > > /home/jay/Programs/petsc/src/mat/interface/matrix.c > > [1]PETSC ERROR: MatDuplicate_SeqAIJ() line 4158 in > > /home/jay/Programs/petsc/src/mat/impls/aij/seq/aij.c > > [1]PETSC ERROR: MatDuplicate() line 4116 in > > /home/jay/Programs/petsc/src/mat/interface/matrix.c > > [1]PETSC ERROR: MatDuplicate_MPIAIJ() line 3395 in > > /home/jay/Programs/petsc/src/mat/impls/aij/mpi/mpiaij.c > > [1]PETSC ERROR: MatDuplicate() line 4116 in > > /home/jay/Programs/petsc/src/mat/interface/matrix.c > > [1]PETSC ERROR: TSGetRHSMats_Private() line 605 in > > /home/jay/Programs/petsc/src/ts/interface/ts.c > > [1]PETSC ERROR: TSComputeIJacobian() line 782 in > > /home/jay/Programs/petsc/src/ts/interface/ts.c > > [1]PETSC ERROR: SNESTSFormJacobian_ARKIMEX() line 1056 in > > /home/jay/Programs/petsc/src/ts/impls/arkimex/arkimex.c > > [1]PETSC ERROR: SNESTSFormJacobian() line 3541 in > > /home/jay/Programs/petsc/src/ts/interface/ts.c > > [1]PETSC ERROR: SNESComputeJacobian() line 2253 in > > /home/jay/Programs/petsc/src/snes/interface/snes.c > > [1]PETSC ERROR: SNESSolve_NEWTONLS() line 231 in > > /home/jay/Programs/petsc/src/snes/impls/ls/ls.c > > [1]PETSC ERROR: SNESSolve() line 3812 in > > /home/jay/Programs/petsc/src/snes/interface/snes.c > > [1]PETSC ERROR: TSStep_ARKIMEX() line 776 in > > /home/jay/Programs/petsc/src/ts/impls/arkimex/arkimex.c > > [1]PETSC ERROR: TSStep() line 2625 in > > /home/jay/Programs/petsc/src/ts/interface/ts.c > > [1]PETSC ERROR: TSSolve() line 2741 in > > /home/jay/Programs/petsc/src/ts/interface/ts.c > > [1]PETSC ERROR: checkPetscError() line 65 in > > /home/jay/research/xolotl/xolotl_workspace/xolotl-Source@xolotl/xolotlSolver/PetscSolver.cpp > > ------------------------------------------------------------------------ > > [0]PETSC ERROR: Petsc Development GIT revision: v3.4.3-2305-g52cd893 GIT > > Date: 2014-01-13 09:37:41 -0500 > > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > > [0]PETSC ERROR: See docs/index.html for manual pages. > > [0]PETSC ERROR: > > ------------------------------------------------------------------------ > > [0]PETSC ERROR: ./xolotl on a arch-linux2-c-opt named box.ornl.gov by jay > > Wed Jan 15 14:12:23 2014 > > [0]PETSC ERROR: Libraries linked from /opt/petsc-latest_mpich-3.0.1/lib > > [0]PETSC ERROR: Configure run at Mon Jan 13 11:11:18 2014 > > [0]PETSC ERROR: Configure options --prefix=/opt/petsc-latest_mpich-3.0.1 > > --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif77 --with-debugging=no > > --download-f-blas-lapack=1 --FOPTFLAGS= --with-shared-libraries=1 > > --download-hypre=yes --with-debugging=0 > > [0]PETSC ERROR: > > ------------------------------------------------------------------------ > > [0]PETSC ERROR: PetscLayoutSetBlockSize() line 464 in > > /home/jay/Programs/petsc/src/vec/is/utils/pmap.c > > [0]PETSC ERROR: MatSetBlockSizes() line 6772 in > > /home/jay/Programs/petsc/src/mat/interface/matrix.c > > [0]PETSC ERROR: MatDuplicate_SeqAIJ() line 4158 in > > /home/jay/Programs/petsc/src/mat/impls/aij/seq/aij.c > > [0]PETSC ERROR: MatDuplicate() line 4116 in > > /home/jay/Programs/petsc/src/mat/interface/matrix.c > > [0]PETSC ERROR: MatDuplicate_MPIAIJ() line 3395 in > > /home/jay/Programs/petsc/src/mat/impls/aij/mpi/mpiaij.c > > [0]PETSC ERROR: MatDuplicate() line 4116 in > > /home/jay/Programs/petsc/src/mat/interface/matrix.c > > [0]PETSC ERROR: TSGetRHSMats_Private() line 605 in > > /home/jay/Programs/petsc/src/ts/interface/ts.c > > [0]PETSC ERROR: TSComputeIJacobian() line 782 in > > /home/jay/Programs/petsc/src/ts/interface/ts.c > > [0]PETSC ERROR: SNESTSFormJacobian_ARKIMEX() line 1056 in > > /home/jay/Programs/petsc/src/ts/impls/arkimex/arkimex.c > > [0]PETSC ERROR: SNESTSFormJacobian() line 3541 in > > /home/jay/Programs/petsc/src/ts/interface/ts.c > > [0]PETSC ERROR: SNESComputeJacobian() line 2253 in > > /home/jay/Programs/petsc/src/snes/interface/snes.c > > [0]PETSC ERROR: SNESSolve_NEWTONLS() line 231 in > > /home/jay/Programs/petsc/src/snes/impls/ls/ls.c > > [0]PETSC ERROR: SNESSolve() line 3812 in > > /home/jay/Programs/petsc/src/snes/interface/snes.c > > [0]PETSC ERROR: TSStep_ARKIMEX() line 776 in > > /home/jay/Programs/petsc/src/ts/impls/arkimex/arkimex.c > > [0]PETSC ERROR: TSStep() line 2625 in > > /home/jay/Programs/petsc/src/ts/interface/ts.c > > [0]PETSC ERROR: TSSolve() line 2741 in > > /home/jay/Programs/petsc/src/ts/interface/ts.c > > [0]PETSC ERROR: checkPetscError() line 65 in > > /home/jay/research/xolotl/xolotl_workspace/xolotl-Source@xolotl/xolotlSolver/PetscSolver.cpp > > > > -- > > Jay Jay Billings > > Oak Ridge National Laboratory > > Twitter Handle: @jayjaybillings > > > > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener
