Great.
> On May 28, 2024, at 9:33 PM, Adrian Croucher <[email protected]>
> wrote:
>
> Thanks again Barry, it's working fine for me now too.
>
> - Adrian
>
> On 29/05/24 1:27 pm, Barry Smith wrote:
>>
>> There was a bug in my fix for parallel which I have fixed. You will need
>> to
>>
>> git fetch
>> git checkout main
>> git branch -D barry/2024-05-27/fix-bug-baij-setvaluesblocked/release
>> git checkout barry/2024-05-27/fix-bug-baij-setvaluesblocked/release
>>
>> I get the same results with your example with 1,2, and 3 ranks
>>
>>
>>> On May 28, 2024, at 7:45 PM, Adrian Croucher <[email protected]>
>>> <mailto:[email protected]> wrote:
>>>
>>> hi Barry,
>>>
>>> Thanks, that change has fixed the error on 2 ranks for me.
>>>
>>> When I run on 3 ranks, there is no error, but it doesn't actually add the
>>> extra values in to the matrix. Do you see that behaviour too?
>>>
>>> - Adrian
>>>
>>> On 29/05/24 4:33 am, Barry Smith wrote:
>>>>
>>>> Adrian,
>>>>
>>>> I could reproduce with 3 MPI ranks.
>>>>
>>>> Another error I had to fix. I also added a test example
>>>>
>>>> SInce I rebased the branch you will need to do something like
>>>>
>>>> git fetch
>>>> git checkout main
>>>> git branch -D barry/2024-05-27/fix-bug-baij-setvaluesblocked/release
>>>> git checkout barry/2024-05-27/fix-bug-baij-setvaluesblocked/release
>>>>
>>>> Thanks for your patience
>>>>
>>>> Barry
>>>>
>>>>
>>>>> On May 27, 2024, at 10:42 PM, Adrian Croucher <[email protected]>
>>>>> <mailto:[email protected]> wrote:
>>>>>
>>>>> Hmm, that's a bit weird. I haven't modified the test code - I checked the
>>>>> file date to make sure. I also tried deleting my PETSc build dir and
>>>>> rebuilding it, then rebuilding the test code. I still get the error if I
>>>>> run on 2 ranks with -dm_mat_type (or -mat_type) baij or mpibaij, but it's
>>>>> fine on aij or mpiaij.
>>>>>
>>>>> My actual (non-test) code is however working ok now, in serial or
>>>>> parallel. So I don't think this should hold up merging your bugfix.
>>>>>
>>>>> - Adrian
>>>>>
>>>>> On 28/05/24 2:13 pm, Barry Smith wrote:
>>>>>>
>>>>>> When I run the exact code you sent with two ranks and—mat_type
>>>>>> mpibaij, it runs as expected. If you modified the code in any way to
>>>>>> demonstrate the bug, please send the modified code.
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On May 27, 2024, at 9:37 PM, Adrian Croucher
>>>>>>> <[email protected]> <mailto:[email protected]> wrote:
>>>>>>>
>>>>>>> hi Barry,
>>>>>>>
>>>>>>> On 28/05/24 7:46 am, Barry Smith wrote:
>>>>>>>>
>>>>>>>> Thanks for reporting this. It is a bug. I have a fixed branch
>>>>>>>> barry/2024-05-27/fix-bug-baij-setvaluesblocked/release and associated
>>>>>>>> merge request
>>>>>>>> https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7578__;!!G_uCfscf7eWS!fIiz5r9h5T8TN2nNlSaM1BCjjyPOZBnzUxuj-00QOJrLaVhRGf5XYwqp5VpSOIMbnc4p5SeyUR-czmNEWSP3vBA$
>>>>>>>>
>>>>>>>>
>>>>>>> Thanks very much, that does appear to fix the bug when I run my test
>>>>>>> program in serial.
>>>>>>>
>>>>>>> If I run it on 2 processes, it is OK with AIJ matrix type, but with
>>>>>>> BAIJ I get an error (see below). Is there another problem, or I am
>>>>>>> doing something else wrong?
>>>>>>>
>>>>>>> - Adrian
>>>>>>>
>>>>>>> [0]PETSC ERROR: --------------------- Error Message
>>>>>>> --------------------------------------------------------------
>>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data
>>>>>>> [0]PETSC ERROR: Incorrect colmap
>>>>>>> [0]PETSC ERROR: See
>>>>>>> https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!fIiz5r9h5T8TN2nNlSaM1BCjjyPOZBnzUxuj-00QOJrLaVhRGf5XYwqp5VpSOIMbnc4p5SeyUR-czmNEwgg8Qv0$
>>>>>>> for trouble shooting.
>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.21.1, unknown
>>>>>>> [0]PETSC ERROR: ./matmodify on a main-debug named EN438880 by acro018
>>>>>>> Tue May 28 13:33:46 2024
>>>>>>> [0]PETSC ERROR: Configure options --with-x --download-hdf5
>>>>>>> --download-zlib --download-netcdf --download-pnetcdf
>>>>>>> --download-exodusii --download-triangle --download-ptscotch
>>>>>>> --download-chaco --download-hypre
>>>>>>> [0]PETSC ERROR: #1 MatSetValuesBlocked_MPIBAIJ() at
>>>>>>> /home/acro018/software/PETSc/code/src/mat/impls/baij/mpi/mpibaij.c:448
>>>>>>> [0]PETSC ERROR: #2 MatSetValuesBlocked() at
>>>>>>> /home/acro018/software/PETSc/code/src/mat/interface/matrix.c:2030
>>>>>>> [0]PETSC ERROR: --------------------- Error Message
>>>>>>> --------------------------------------------------------------
>>>>>>> [0]PETSC ERROR: It appears a new error in the code was triggered
>>>>>>> after a previous error, possibly because:
>>>>>>> [0]PETSC ERROR: - The first error was not properly handled via (for
>>>>>>> example) the use of
>>>>>>> [0]PETSC ERROR: PetscCall(TheFunctionThatErrors()); or
>>>>>>> [0]PETSC ERROR: - The second error was triggered while handling the
>>>>>>> first error.
>>>>>>> [0]PETSC ERROR: Above is the traceback for the previous unhandled
>>>>>>> error, below the traceback for the next error
>>>>>>> [0]PETSC ERROR: ALL ERRORS in the PETSc libraries are fatal, you
>>>>>>> should add the appropriate error checking to the code
>>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data
>>>>>>> [0]PETSC ERROR: Incorrect colmap
>>>>>>> [0]PETSC ERROR: See
>>>>>>> https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!fIiz5r9h5T8TN2nNlSaM1BCjjyPOZBnzUxuj-00QOJrLaVhRGf5XYwqp5VpSOIMbnc4p5SeyUR-czmNEwgg8Qv0$
>>>>>>> for trouble shooting.
>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.21.1, unknown
>>>>>>> [0]PETSC ERROR: ./matmodify on a main-debug named EN438880 by acro018
>>>>>>> Tue May 28 13:33:46 2024
>>>>>>> [0]PETSC ERROR: Configure options --with-x --download-hdf5
>>>>>>> --download-zlib --download-netcdf --download-pnetcdf
>>>>>>> --download-exodusii --download-triangle --download-ptscotch
>>>>>>> --download-chaco --download-hypre
>>>>>>> [0]PETSC ERROR: #1 MatSetValuesBlocked_MPIBAIJ() at
>>>>>>> /home/acro018/software/PETSc/code/src/mat/impls/baij/mpi/mpibaij.c:448
>>>>>>> [0]PETSC ERROR: #2 MatAssemblyEnd_MPIBAIJ() at
>>>>>>> /home/acro018/software/PETSc/code/src/mat/impls/baij/mpi/mpibaij.c:943
>>>>>>> [0]PETSC ERROR: #3 MatAssemblyEnd() at
>>>>>>> /home/acro018/software/PETSc/code/src/mat/interface/matrix.c:5820
>>>>>>> [0]PETSC ERROR: #4 matmodify.F90:40
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Barry
>>>>>>>>
>>>>>>>>
>>>>>>>>> On May 26, 2024, at 10:45 PM, Adrian Croucher
>>>>>>>>> <[email protected]> <mailto:[email protected]> wrote:
>>>>>>>>>
>>>>>>>>> hi,
>>>>>>>>>
>>>>>>>>> I've been trying creating a matrix with DMCreateMatrix() and then
>>>>>>>>> adding extra blocks of nonzeros into it using MatSetValuesBlocked(),
>>>>>>>>> but getting some unexpected results if I set the matrix type to BAIJ.
>>>>>>>>> It seems to behave as expected if I use matrix type AIJ.
>>>>>>>>>
>>>>>>>>> I've attached a minimal example program. It reads in the DMPlex from
>>>>>>>>> file, sets up a section on it, creates a matrix (blocksize 2) and
>>>>>>>>> then inserts a single 2x2 block at global block indices (0,7). It
>>>>>>>>> views the matrix before and after the insertion.
>>>>>>>>>
>>>>>>>>> If I run with "-dm_mat_type aij" it gives the expected results, but
>>>>>>>>> with "-dm_mat_type baij" it doesn't - e.g. if run in serial, it adds
>>>>>>>>> the new nonzeros in the right place but also adds a whole lot of
>>>>>>>>> other duplicated entries in block row 0.
>>>>>>>>>
>>>>>>>>> Is there something I'm not understanding about BAIJ, or about
>>>>>>>>> MatSetValuesBlocked()? or possibly some other mistake?
>>>>>>>>>
>>>>>>>>> - Adrian
>>>>>>>>>
>>>>>>>>> On 20/05/24 12:24 pm, Barry Smith wrote:
>>>>>>>>>>
>>>>>>>>>> You can call MatSetOption(mat,MAT_NEW_NONZERO_LOCATION_ERR) then
>>>>>>>>>> insert the new values. If it is just a handful of new insertions the
>>>>>>>>>> extra time should be small.
>>>>>>>>>>
>>>>>>>>>> Making a copy of the matrix won't give you a new matrix that is
>>>>>>>>>> any faster to insert into so best to just use the same matrix.
>>>>>>>>>>
>>>>>>>>>> Barry
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On May 19, 2024, at 7:44 PM, Adrian Croucher
>>>>>>>>>>> <[email protected]> <mailto:[email protected]>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> This Message Is From an External Sender
>>>>>>>>>>> This message came from outside your organization.
>>>>>>>>>>> hi,
>>>>>>>>>>>
>>>>>>>>>>> I have a Jacobian matrix created using DMCreateMatrix(). What would
>>>>>>>>>>> be
>>>>>>>>>>> the best way to add extra nonzero entries into it?
>>>>>>>>>>>
>>>>>>>>>>> I'm guessing that DMCreateMatrix() allocates the storage so the
>>>>>>>>>>> nonzero
>>>>>>>>>>> structure can't really be easily modified. Would it be a case of
>>>>>>>>>>> creating a new matrix, copying the nonzero entries from the
>>>>>>>>>>> original one
>>>>>>>>>>> and then adding the extra ones, before calling MatSetUp() or
>>>>>>>>>>> similar? If
>>>>>>>>>>> so, how exactly would you copy the nonzero structure from the
>>>>>>>>>>> original
>>>>>>>>>>> matrix?
>>>>>>>>>>>
>>>>>>>>>>> Background: the flow problem I'm solving (on a DMPlex with finite
>>>>>>>>>>> volume
>>>>>>>>>>> method) has complex source terms that depend on the solution (e.g.
>>>>>>>>>>> pressure), and can also depend on other source terms. A simple
>>>>>>>>>>> example
>>>>>>>>>>> is when fluid is extracted from one location, with a
>>>>>>>>>>> pressure-dependent
>>>>>>>>>>> flow rate, and some of it is then reinjected in another location.
>>>>>>>>>>> This
>>>>>>>>>>> can result in poor nonlinear solver convergence. I think the reason
>>>>>>>>>>> is
>>>>>>>>>>> that there are effectively missing Jacobian entries in the row for
>>>>>>>>>>> the
>>>>>>>>>>> reinjection cell, which should have an additional dependence on the
>>>>>>>>>>> solution in the cell where fluid is extracted.
>>>>>>>>>>>
>>>>>>>>>>> - Adrian
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Dr Adrian Croucher
>>>>>>>>>>> Senior Research Fellow
>>>>>>>>>>> Department of Engineering Science
>>>>>>>>>>> Waipapa Taumata Rau / University of Auckland, New Zealand
>>>>>>>>>>> email: [email protected] <mailto:[email protected]>
>>>>>>>>>>> tel: +64 (0)9 923 4611
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Dr Adrian Croucher
>>>>>>>>> Senior Research Fellow
>>>>>>>>> Department of Engineering Science
>>>>>>>>> Waipapa Taumata Rau / University of Auckland, New Zealand
>>>>>>>>> email: [email protected] <mailto:[email protected]>
>>>>>>>>> tel: +64 (0)9 923 4611
>>>>>>>>> <3x3grid.exo><matmodify.F90>
>>>>>>>>
>>>>>>> --
>>>>>>> Dr Adrian Croucher
>>>>>>> Senior Research Fellow
>>>>>>> Department of Engineering Science
>>>>>>> Waipapa Taumata Rau / University of Auckland, New Zealand
>>>>>>> email: [email protected] <mailto:[email protected]>
>>>>>>> tel: +64 (0)9 923 4611
>>>>>>
>>>>> --
>>>>> Dr Adrian Croucher
>>>>> Senior Research Fellow
>>>>> Department of Engineering Science
>>>>> Waipapa Taumata Rau / University of Auckland, New Zealand
>>>>> email: [email protected] <mailto:[email protected]>
>>>>> tel: +64 (0)9 923 4611
>>>>
>>> --
>>> Dr Adrian Croucher
>>> Senior Research Fellow
>>> Department of Engineering Science
>>> Waipapa Taumata Rau / University of Auckland, New Zealand
>>> email: [email protected] <mailto:[email protected]>
>>> tel: +64 (0)9 923 4611
>>
> --
> Dr Adrian Croucher
> Senior Research Fellow
> Department of Engineering Science
> Waipapa Taumata Rau / University of Auckland, New Zealand
> email: [email protected] <mailto:[email protected]>
> tel: +64 (0)9 923 4611
