> On Jan 20, 2017, at 3:01 PM, Zhang, Hong <[email protected]> wrote: > > We set C->submat_singleis = false as default. > User turns it on when he knows a single IS will be used for > MatGetSubMatrices(), e.g., when asm is used. > User should turn the flag off after it is being used.
Right, this is why I am confused why this is messing up Mat since presumably he is not setting it? > > This flag is for optimization -- avoid MPI_Allreduce, which is expensive for > np>10k. > MatGetSubMatrices() works without this flag turns on. > I just merged the improved MatGetSubMatrices_MPIAIJ() which only calls > MPI_Allreduce once > for repeated use. Previously, it calls MPI_Allreduce twice for each call of > MatGetSubMatrices_MPIAIJ(). > > Hong > > ________________________________________ > From: Barry Smith [[email protected]] > Sent: Friday, January 20, 2017 1:11 PM > To: Matthew Knepley > Cc: Zhang, Hong; PETSc > Subject: Re: [petsc-dev] Bug introduced in MatGetSubmatrices() > >> On Jan 20, 2017, at 12:45 PM, Matthew Knepley <[email protected]> wrote: >> >> On Fri, Jan 20, 2017 at 11:55 AM, Barry Smith <[email protected]> wrote: >> >>> On Jan 20, 2017, at 11:49 AM, Matthew Knepley <[email protected]> wrote: >>> >>> On Fri, Jan 20, 2017 at 10:49 AM, Hong <[email protected]> wrote: >>> Matt, >>> By default, the flag C->submat_singleis = false. >>> In PCSetUp_ASM(), we set it as 'true' to use >>> MatGetSubMatrices_MPIAIJ_SingleIS(). >>> >>> Can you check the value of this flag in your case? >>> >>> The problem is the following: >>> >>> 1) We use MatGetSubMatrices() to extract small matrices in order to form a >>> preconditioner >>> >>> 2) We do this at each Newton iteration >>> >>> 3) We use ASM as a preconditioner for the eventual Newton solve >>> >>> 4) The second time we call MatGetSubMatrices(), it has this flag set, even >>> though we are using multiple ISes >>> >>> Solution: ALSO check that the user is in fact passing a single IS. >>> >> This requires a communication. I am confused, is the number of IS >> changing each time? If not why is the flag set? >> >> ASM sets this flag because it knows that IT is going to call >> MatGetSubMatrices() later, but it unsafe if any user calls >> MatGetSubMatrices() as well. I think overall its a fragile design and should >> be scrapped. > > You mean the user calls the SAME MatGetSubMatrices with the same matrices, > right? Not a completely different unrelated MatGetSubMatrices() which should > not be affect by the previous unrelated call. > > > Barry > >> >> Matt >> >> >> Barry >> >>> Matt >>> >>> Hong >>> >>> It comes from here: >>> >>> >>> https://bitbucket.org/petsc/petsc/commits/c10200c1442b553b7ad65c70101560db4fa22e78 >>> >>> If we ask for more than 1 matrix, it dispatches to >>> >>> MatGetSubMatrices_MPIAIJ_SingleIS() >>> >>> but then fails here >>> >>> >>> https://bitbucket.org/petsc/petsc/annotate/2e559809f9aee9c95ee79eb0939630cfe5502c8d/src/mat/impls/aij/mpi/mpiov.c?at=master&fileviewer=file-view-default#mpiov.c-1306 >>> >>> because ismax > 1. I think the ismax check needs to move up to here >>> >>> >>> https://bitbucket.org/petsc/petsc/annotate/2e559809f9aee9c95ee79eb0939630cfe5502c8d/src/mat/impls/aij/mpi/mpiov.c?at=master&fileviewer=file-view-default#mpiov.c-2012 >>> >>> but I don't know for sure. Please fix this since it is breaking PyLith. >>> >>> Matt >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments >> is infinitely more interesting than any results to which their experiments >> lead. >> -- Norbert Wiener > >
