Adrian, I could reproduce with 3 MPI ranks.
Another error I had to fix. I also added a test example SInce I rebased the branch you will need to do something like git fetch git checkout main git branch -D barry/2024-05-27/fix-bug-baij-setvaluesblocked/release git checkout barry/2024-05-27/fix-bug-baij-setvaluesblocked/release Thanks for your patience Barry > On May 27, 2024, at 10:42 PM, Adrian Croucher <[email protected]> > wrote: > > Hmm, that's a bit weird. I haven't modified the test code - I checked the > file date to make sure. I also tried deleting my PETSc build dir and > rebuilding it, then rebuilding the test code. I still get the error if I run > on 2 ranks with -dm_mat_type (or -mat_type) baij or mpibaij, but it's fine on > aij or mpiaij. > > My actual (non-test) code is however working ok now, in serial or parallel. > So I don't think this should hold up merging your bugfix. > > - Adrian > > On 28/05/24 2:13 pm, Barry Smith wrote: >> >> When I run the exact code you sent with two ranks and—mat_type mpibaij, it >> runs as expected. If you modified the code in any way to demonstrate the >> bug, please send the modified code. >> >> >> >>> On May 27, 2024, at 9:37 PM, Adrian Croucher <[email protected]> >>> <mailto:[email protected]> wrote: >>> >>> hi Barry, >>> >>> On 28/05/24 7:46 am, Barry Smith wrote: >>>> >>>> Thanks for reporting this. It is a bug. I have a fixed branch >>>> barry/2024-05-27/fix-bug-baij-setvaluesblocked/release and associated >>>> merge request >>>> https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7578__;!!G_uCfscf7eWS!doaVfskPKURsqYgA0bpocyW5cbPLsGwTjv41wqfhw_nlPc6a7scHMULs9UqHfAz74BFdHtbPJ-LBlRkYElrUKjE$ >>>> >>>> >>> Thanks very much, that does appear to fix the bug when I run my test >>> program in serial. >>> >>> If I run it on 2 processes, it is OK with AIJ matrix type, but with BAIJ I >>> get an error (see below). Is there another problem, or I am doing something >>> else wrong? >>> >>> - Adrian >>> >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Petsc has generated inconsistent data >>> [0]PETSC ERROR: Incorrect colmap >>> [0]PETSC ERROR: See >>> https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!doaVfskPKURsqYgA0bpocyW5cbPLsGwTjv41wqfhw_nlPc6a7scHMULs9UqHfAz74BFdHtbPJ-LBlRkY0Lao4Fs$ >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.21.1, unknown >>> [0]PETSC ERROR: ./matmodify on a main-debug named EN438880 by acro018 Tue >>> May 28 13:33:46 2024 >>> [0]PETSC ERROR: Configure options --with-x --download-hdf5 --download-zlib >>> --download-netcdf --download-pnetcdf --download-exodusii >>> --download-triangle --download-ptscotch --download-chaco --download-hypre >>> [0]PETSC ERROR: #1 MatSetValuesBlocked_MPIBAIJ() at >>> /home/acro018/software/PETSc/code/src/mat/impls/baij/mpi/mpibaij.c:448 >>> [0]PETSC ERROR: #2 MatSetValuesBlocked() at >>> /home/acro018/software/PETSc/code/src/mat/interface/matrix.c:2030 >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: It appears a new error in the code was triggered after a >>> previous error, possibly because: >>> [0]PETSC ERROR: - The first error was not properly handled via (for >>> example) the use of >>> [0]PETSC ERROR: PetscCall(TheFunctionThatErrors()); or >>> [0]PETSC ERROR: - The second error was triggered while handling the >>> first error. >>> [0]PETSC ERROR: Above is the traceback for the previous unhandled error, >>> below the traceback for the next error >>> [0]PETSC ERROR: ALL ERRORS in the PETSc libraries are fatal, you should >>> add the appropriate error checking to the code >>> [0]PETSC ERROR: Petsc has generated inconsistent data >>> [0]PETSC ERROR: Incorrect colmap >>> [0]PETSC ERROR: See >>> https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!doaVfskPKURsqYgA0bpocyW5cbPLsGwTjv41wqfhw_nlPc6a7scHMULs9UqHfAz74BFdHtbPJ-LBlRkY0Lao4Fs$ >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.21.1, unknown >>> [0]PETSC ERROR: ./matmodify on a main-debug named EN438880 by acro018 Tue >>> May 28 13:33:46 2024 >>> [0]PETSC ERROR: Configure options --with-x --download-hdf5 --download-zlib >>> --download-netcdf --download-pnetcdf --download-exodusii >>> --download-triangle --download-ptscotch --download-chaco --download-hypre >>> [0]PETSC ERROR: #1 MatSetValuesBlocked_MPIBAIJ() at >>> /home/acro018/software/PETSc/code/src/mat/impls/baij/mpi/mpibaij.c:448 >>> [0]PETSC ERROR: #2 MatAssemblyEnd_MPIBAIJ() at >>> /home/acro018/software/PETSc/code/src/mat/impls/baij/mpi/mpibaij.c:943 >>> [0]PETSC ERROR: #3 MatAssemblyEnd() at >>> /home/acro018/software/PETSc/code/src/mat/interface/matrix.c:5820 >>> [0]PETSC ERROR: #4 matmodify.F90:40 >>> >>> >>> >>>> Barry >>>> >>>> >>>>> On May 26, 2024, at 10:45 PM, Adrian Croucher <[email protected]> >>>>> <mailto:[email protected]> wrote: >>>>> >>>>> hi, >>>>> >>>>> I've been trying creating a matrix with DMCreateMatrix() and then adding >>>>> extra blocks of nonzeros into it using MatSetValuesBlocked(), but getting >>>>> some unexpected results if I set the matrix type to BAIJ. It seems to >>>>> behave as expected if I use matrix type AIJ. >>>>> >>>>> I've attached a minimal example program. It reads in the DMPlex from >>>>> file, sets up a section on it, creates a matrix (blocksize 2) and then >>>>> inserts a single 2x2 block at global block indices (0,7). It views the >>>>> matrix before and after the insertion. >>>>> >>>>> If I run with "-dm_mat_type aij" it gives the expected results, but with >>>>> "-dm_mat_type baij" it doesn't - e.g. if run in serial, it adds the new >>>>> nonzeros in the right place but also adds a whole lot of other duplicated >>>>> entries in block row 0. >>>>> >>>>> Is there something I'm not understanding about BAIJ, or about >>>>> MatSetValuesBlocked()? or possibly some other mistake? >>>>> >>>>> - Adrian >>>>> >>>>> On 20/05/24 12:24 pm, Barry Smith wrote: >>>>>> >>>>>> You can call MatSetOption(mat,MAT_NEW_NONZERO_LOCATION_ERR) then >>>>>> insert the new values. If it is just a handful of new insertions the >>>>>> extra time should be small. >>>>>> >>>>>> Making a copy of the matrix won't give you a new matrix that is any >>>>>> faster to insert into so best to just use the same matrix. >>>>>> >>>>>> Barry >>>>>> >>>>>> >>>>>>> On May 19, 2024, at 7:44 PM, Adrian Croucher >>>>>>> <[email protected]> <mailto:[email protected]> wrote: >>>>>>> >>>>>>> This Message Is From an External Sender >>>>>>> This message came from outside your organization. >>>>>>> hi, >>>>>>> >>>>>>> I have a Jacobian matrix created using DMCreateMatrix(). What would be >>>>>>> the best way to add extra nonzero entries into it? >>>>>>> >>>>>>> I'm guessing that DMCreateMatrix() allocates the storage so the nonzero >>>>>>> structure can't really be easily modified. Would it be a case of >>>>>>> creating a new matrix, copying the nonzero entries from the original >>>>>>> one >>>>>>> and then adding the extra ones, before calling MatSetUp() or similar? >>>>>>> If >>>>>>> so, how exactly would you copy the nonzero structure from the original >>>>>>> matrix? >>>>>>> >>>>>>> Background: the flow problem I'm solving (on a DMPlex with finite >>>>>>> volume >>>>>>> method) has complex source terms that depend on the solution (e.g. >>>>>>> pressure), and can also depend on other source terms. A simple example >>>>>>> is when fluid is extracted from one location, with a pressure-dependent >>>>>>> flow rate, and some of it is then reinjected in another location. This >>>>>>> can result in poor nonlinear solver convergence. I think the reason is >>>>>>> that there are effectively missing Jacobian entries in the row for the >>>>>>> reinjection cell, which should have an additional dependence on the >>>>>>> solution in the cell where fluid is extracted. >>>>>>> >>>>>>> - Adrian >>>>>>> >>>>>>> -- >>>>>>> Dr Adrian Croucher >>>>>>> Senior Research Fellow >>>>>>> Department of Engineering Science >>>>>>> Waipapa Taumata Rau / University of Auckland, New Zealand >>>>>>> email: [email protected] <mailto:[email protected]> >>>>>>> tel: +64 (0)9 923 4611 >>>>>>> >>>>>>> >>>>>> >>>>> -- >>>>> Dr Adrian Croucher >>>>> Senior Research Fellow >>>>> Department of Engineering Science >>>>> Waipapa Taumata Rau / University of Auckland, New Zealand >>>>> email: [email protected] <mailto:[email protected]> >>>>> tel: +64 (0)9 923 4611 >>>>> <3x3grid.exo><matmodify.F90> >>>> >>> -- >>> Dr Adrian Croucher >>> Senior Research Fellow >>> Department of Engineering Science >>> Waipapa Taumata Rau / University of Auckland, New Zealand >>> email: [email protected] <mailto:[email protected]> >>> tel: +64 (0)9 923 4611 >> > -- > Dr Adrian Croucher > Senior Research Fellow > Department of Engineering Science > Waipapa Taumata Rau / University of Auckland, New Zealand > email: [email protected] <mailto:[email protected]> > tel: +64 (0)9 923 4611
