I just tried building against petsc/master, but there needs to be more work on libMesh before it can work with petsc/master: the new VecLockPush()/Pop() stuff isn't respected by vector manipulation in libMesh. I put a hack equivalent to MatReset() into your branch (patch attached, in case you want to take a look at it), but it generates the same error in MatCreateColmap that you reported earlier. It's odd that it occurs on the second nonlinear iteration. I'll have to dig a bit deeper to see what's going on.
Dmitry. On Mon Feb 23 2015 at 10:03:33 AM David Knezevic <[email protected]> wrote: > Hi Dmitry, > > OK, good to hear we're seeing the same behavior for the example. > > Regarding this comment: > > > libMesh needs to change the way configure extracts PETSc information -- >> configuration data were moved: >> conf --> lib/petsc-conf >> ${PETSC_ARCH}/conf --> ${PETSC_ARCH}/lib/petsc-conf >> >> At one point I started looking at m4/petsc.m4, but that got put on the >> back burner. For now making the relevant symlinks by hand lets you >> configure and build libMesh with petsc/master. >> > > > So do you suggest that the next step here is to build libmesh against > petsc/master so that we can try your PETSc pull request that implements > MatReset() to see if that gets this example working? > > David > > > > > >> On Mon Feb 23 2015 at 9:15:44 AM David Knezevic < >> [email protected]> wrote: >> >>> Hi Dmitry, >>> >>> Thanks very much for testing out the example. >>> >>> examples/systems_of_equations/ex8 works fine for me in serial, but it >>> fails for me if I run with more than 1 MPI process. Can you try it with, >>> say, 2 or 4 MPI processes? >>> >>> If we need access to MatReset in libMesh to get this to work, I'll be >>> happy to work on a libMesh pull request for that. >>> >>> David >>> >>> >>> -- >>> >>> David J. Knezevic | CTO >>> Akselos | 17 Bay State Road | Boston, MA | 02215 >>> Phone (office): +1-857-265-2238 >>> Phone (mobile): +1-617-599-4755 >>> Web: http://www.akselos.com >>> >>> >>> On Mon, Feb 23, 2015 at 10:08 AM, Dmitry Karpeyev <[email protected]> >>> wrote: >>> >>>> David, >>>> >>>> What code are you running when you encounter this error? I'm trying to >>>> reproduce it and >>>> I tried examples/systems_of_equations/ex8, but it ran for me, >>>> ostensibly to completion. >>>> >>>> I have a small PETSc pull request that implements MatReset(), which >>>> passes a small PETSc test, >>>> but libMesh needs some work to be able to build against petsc/master >>>> because of some recent >>>> changes to PETSc. >>>> >>>> Dmitry. >>>> >>>> On Mon Feb 23 2015 at 7:17:06 AM David Knezevic < >>>> [email protected]> wrote: >>>> >>>>> Hi Barry, hi Dmitry, >>>>> >>>>> I set the matrix to BAIJ and back to AIJ, and the code got a bit >>>>> further. But I now run into the error pasted below (Note that I'm now >>>>> using >>>>> "--with-debugging=1"): >>>>> >>>>> PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> PETSC ERROR: Petsc has generated inconsistent data >>>>> PETSC ERROR: MPIAIJ Matrix was assembled but is missing garray >>>>> PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>> for trouble shooting. >>>>> PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014 >>>>> PETSC ERROR: ./example-dbg on a arch-linux2-c-debug named david-Lenovo >>>>> by dknez Mon Feb 23 08:05:44 2015 >>>>> PETSC ERROR: Configure options --with-shared-libraries=1 >>>>> --with-debugging=1 --download-suitesparse=1 --download-parmetis=1 >>>>> --download-blacs=1 --download-scalapack=1 --download-mumps=1 >>>>> --download-metis --download-superlu_dist >>>>> --prefix=/home/dknez/software/libmesh_install/dbg_real/petsc >>>>> --download-hypre >>>>> PETSC ERROR: #1 MatCreateColmap_MPIAIJ_Private() line 361 in >>>>> /home/dknez/software/petsc-3.5.2/src/mat/impls/aij/mpi/mpiaij.c >>>>> PETSC ERROR: #2 MatSetValues_MPIAIJ() line 538 in >>>>> /home/dknez/software/petsc-3.5.2/src/mat/impls/aij/mpi/mpiaij.c >>>>> PETSC ERROR: #3 MatSetValues() line 1136 in >>>>> /home/dknez/software/petsc-3.5.2/src/mat/interface/matrix.c >>>>> PETSC ERROR: #4 add_matrix() line 765 in /home/dknez/software/libmesh- >>>>> src/src/numerics/petsc_matrix.C >>>>> ------------------------------------------------------------ >>>>> -------------- >>>>> >>>>> This occurs when I try to set some entries of the matrix. Do you have >>>>> any suggestions on how I can resolve this? >>>>> >>>>> Thanks! >>>>> David >>>>> >>>>> >>>>> >>>>> >>>>> On Sun, Feb 22, 2015 at 10:22 PM, Dmitry Karpeyev <[email protected]> >>>>> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Sun Feb 22 2015 at 9:15:22 PM Barry Smith <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> > On Feb 22, 2015, at 9:09 PM, David Knezevic < >>>>>>> [email protected]> wrote: >>>>>>> > >>>>>>> > Hi Dmitry, >>>>>>> > >>>>>>> > Thanks for the suggestion. I tried MatSetType(mat,MATMPIAIJ) >>>>>>> followed by MatXAIJSetPreallocation(...), but unfortunately this still >>>>>>> gives me the same error as before: "nnz cannot be greater than row >>>>>>> length: >>>>>>> local row 168 value 24 rowlength 0". >>>>>>> > >>>>>>> > I gather that the idea here is that MatSetType builds a new matrix >>>>>>> object, and then I should be able to pre-allocate for that new matrix >>>>>>> however I like, right? Was I supposed to clear the matrix object somehow >>>>>>> before calling MatSetType? (I didn't do any sort of clear operation.) >>>>>>> >>>>>>> If the type doesn't change then MatSetType() won't do anything. >>>>>>> You can try setting the type to BAIJ and then setting the type back to >>>>>>> AIJ. >>>>>>> This may/should clear out the matrix. >>>>>>> >>>>>> Ah, yes. If the type is the same as before it does quit early, but >>>>>> changing the type and then back will clear out and rebuild the matrix. >>>>>> We >>>>>> need >>>>>> something like MatReset() to do the equivalent thing. >>>>>> >>>>>>> >>>>>>> > >>>>>>> > As I said earlier, I'll make a dbg PETSc build, so hopefully that >>>>>>> will help shed some light on what's going wrong for me. >>>>>>> >>>>>> I think it's always a good idea to have a dbg build of PETSc when you >>>>>> doing things like these. >>>>>> >>>>>> Dmitry. >>>>>> >>>>>>> >>>>>>> Don't bother, what I suggested won't work. >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>> >>>>>>> > >>>>>>> > Thanks, >>>>>>> > David >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > On Sun, Feb 22, 2015 at 6:02 PM, Dmitry Karpeyev < >>>>>>> [email protected]> wrote: >>>>>>> > David, >>>>>>> > It might be easier to just rebuild the whole matrix from scratch: >>>>>>> you would in effect be doing all that with disassembling and resetting >>>>>>> the >>>>>>> preallocation. >>>>>>> > MatSetType(mat,MATMPIAIJ) >>>>>>> > or >>>>>>> > PetscObjectGetType((PetscObject)mat,&type); >>>>>>> > MatSetType(mat,type); >>>>>>> > followed by >>>>>>> > MatXAIJSetPreallocation(...); >>>>>>> > should do. >>>>>>> > Dmitry. >>>>>>> > >>>>>>> > >>>>>>> > On Sun Feb 22 2015 at 4:45:46 PM Barry Smith <[email protected]> >>>>>>> wrote: >>>>>>> > >>>>>>> > Do not call for SeqAIJ matrix. Do not call before the first time >>>>>>> you have preallocated and put entries in the matrix and done the >>>>>>> MatAssemblyBegin/End() >>>>>>> > >>>>>>> > If it still crashes you'll need to try the debugger >>>>>>> > >>>>>>> > Barry >>>>>>> > >>>>>>> > > On Feb 22, 2015, at 4:09 PM, David Knezevic < >>>>>>> [email protected]> wrote: >>>>>>> > > >>>>>>> > > Hi Barry, >>>>>>> > > >>>>>>> > > Thanks for your help, much appreciated. >>>>>>> > > >>>>>>> > > I added a prototype for MatDisAssemble_MPIAIJ: >>>>>>> > > PETSC_INTERN PetscErrorCode MatDisAssemble_MPIAIJ(Mat); >>>>>>> > > >>>>>>> > > and I added a call to MatDisAssemble_MPIAIJ before >>>>>>> MatMPIAIJSetPreallocation. However, I get a segfault on the call to >>>>>>> MatDisAssemble_MPIAIJ. The segfault occurs in both serial and parallel. >>>>>>> > > >>>>>>> > > FYI, I'm using Petsc 3.5.2, and I'm not using a non-debug build >>>>>>> (though I could rebuild PETSc in debug mode if you think that would help >>>>>>> figure out what's happening here). >>>>>>> > > >>>>>>> > > Thanks, >>>>>>> > > David >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > On Sun, Feb 22, 2015 at 1:13 PM, Barry Smith <[email protected]> >>>>>>> wrote: >>>>>>> > > >>>>>>> > > David, >>>>>>> > > >>>>>>> > > This is an obscure little feature of MatMPIAIJ, each time >>>>>>> you change the sparsity pattern before you call the >>>>>>> MatMPIAIJSetPreallocation you need to call MatDisAssemble_MPIAIJ(Mat >>>>>>> mat). This is a private PETSc function so you need to provide your >>>>>>> own >>>>>>> prototype for it above the function you use it in. >>>>>>> > > >>>>>>> > > Let us know if this resolves the problem. >>>>>>> > > >>>>>>> > > Barry >>>>>>> > > >>>>>>> > > We never really intended that people would call >>>>>>> MatMPIAIJSetPreallocation() AFTER they had already used the matrix. >>>>>>> > > >>>>>>> > > >>>>>>> > > > On Feb 22, 2015, at 6:50 AM, David Knezevic < >>>>>>> [email protected]> wrote: >>>>>>> > > > >>>>>>> > > > Hi all, >>>>>>> > > > >>>>>>> > > > I've implemented a solver for a contact problem using SNES. >>>>>>> The sparsity pattern of the jacobian matrix needs to change at each >>>>>>> nonlinear iteration (because the elements which are in contact can >>>>>>> change), >>>>>>> so I tried to deal with this by calling MatSeqAIJSetPreallocation and >>>>>>> MatMPIAIJSetPreallocation during each iteration in order to update the >>>>>>> preallocation. >>>>>>> > > > >>>>>>> > > > This seems to work fine in serial, but with two or more MPI >>>>>>> processes I run into the error "nnz cannot be greater than row length", >>>>>>> e.g.: >>>>>>> > > > nnz cannot be greater than row length: local row 528 value 12 >>>>>>> rowlength 0 >>>>>>> > > > >>>>>>> > > > This error is from the call to >>>>>>> > > > MatSeqAIJSetPreallocation(b->B,o_nz,o_nnz); in >>>>>>> MatMPIAIJSetPreallocation_MPIAIJ. >>>>>>> > > > >>>>>>> > > > Any guidance on what the problem might be would be most >>>>>>> appreciated. For example, I was wondering if there is a problem with >>>>>>> calling SetPreallocation on a matrix that has already been preallocated? >>>>>>> > > > >>>>>>> > > > Some notes: >>>>>>> > > > - I'm using PETSc via libMesh >>>>>>> > > > - The code that triggers this issue is available as a PR on >>>>>>> the libMesh github repo, in case anyone is interested: >>>>>>> https://github.com/libMesh/libmesh/pull/460/ >>>>>>> > > > - I can try to make a minimal pure-PETSc example that >>>>>>> reproduces this error, if that would be helpful. >>>>>>> > > > >>>>>>> > > > Many thanks, >>>>>>> > > > David >>>>>>> > > > >>>>>>> > > >>>>>>> > > >>>>>>> > >>>>>>> > >>>>>>> >>>>>>> >>>>> >>>
mat-reset-hack.patch
Description: Binary data
