Re: [petsc-users] Diagnosing Convergence Issue in Fieldsplit Problem

[Matthew Knepley]

I put in stuff to propagate the nullspace if you use DM.

   Matt

On Thu, May 23, 2024 at 11:04 PM Barry Smith  wrote:

> On May 23, 2024, at 3: 48 PM, Stefano Zampini  com> wrote: the null space of the Schur complement is the restriction of
> the original null space. I guess if fieldsplit is Schur type then we could
> in principle extract
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
>
>
> On May 23, 2024, at 3:48 PM, Stefano Zampini 
> wrote:
>
> the null space of the Schur complement is the restriction of the original
> null space. I guess if fieldsplit is Schur type then we could in principle
> extract the sub vectors and renormalize them
>
>
>Is this true if A is singular?   Or are you assuming the Schur
> complement form is only used if A is nonsingular? Would the user need to
> somehow indicate A is nonsingular?
>
>
>
>
> On Thu, May 23, 2024, 22:13 Jed Brown  wrote:
>
>> Barry Smith  writes: > Unfortunately it cannot
>> automatically because -pc_fieldsplit_detect_saddle_point just grabs part of
>> the matrix (having no concept of "what part" so doesn't know to grab the
>> null space information.
>> ZjQcmQRYFpfptBannerStart
>> This Message Is From an External Sender
>> This message came from outside your organization.
>>
>> ZjQcmQRYFpfptBannerEnd
>>
>> Barry Smith  writes:
>>
>> >Unfortunately it cannot automatically because 
>> > -pc_fieldsplit_detect_saddle_point just grabs part of the matrix (having 
>> > no concept of "what part" so doesn't know to grab the null space 
>> > information.
>> >
>> >It would be possible for PCFIELDSPLIT to access the null space of the 
>> > larger matrix directly as vectors and check if they are all zero in the 00 
>> > block, then it would know that the null space only applied to the second 
>> > block and could use it for the Schur complement.
>> >
>> >Matt, Jed, Stefano, Pierre does this make sense?
>>
>> I think that would work (also need to check that the has_cnst flag is 
>> false), though if you've gone to the effort of filling in that Vec, you 
>> might as well provide the IS.
>>
>> I also wonder if the RHS is consistent.
>>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cl36t5n0xoPzg3CNPvl8rF34swIpHS7UzkeJ7NGQpRF4ZxJeeQqaIuD7nTY5hCChJbFlaXCbk0pIovfR2pFG$
  

Re: [petsc-users] Modify matrix nonzero structure

[Matthew Knepley]

On Sun, May 19, 2024 at 11:25 PM Barry Smith  wrote:

> Certainly missing Jacobian entries can dramatically change the Newton
> direction and hence the convergence. Even if the optimal (in time) setup
> skips some Jacobian entries it is always good to have runs with all the
> entries to see the "best
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
>
>   Certainly missing Jacobian entries can dramatically change the Newton
> direction and hence the convergence. Even if the optimal (in time) setup
> skips some Jacobian entries it is always good to have runs with all the
> entries to see the "best possible" convergence.
>

Let me expand on this. If you are missing Jacobian entries, one option is
to use the -snes_mf_operator mode for the solve. In this mode, you provide
an approximate Jacobian that is used to generate the preconditioner, but
the action of the Jacobian is calculated by finite differences using the
residual function. Thus, the Jaccbian "matrix" should be consistent with
the residual, but the preconditioner is approximate.

  Thanks,

  Matt


>   Barry
>
>
> On May 19, 2024, at 10:44 PM, Adrian Croucher 
> wrote:
>
> Great, it sounds like this might be easier than I expected. Thanks very
> much.
>
> Did you have any thoughts on my diagnosis of the problem (the poor
> nonlinear solver convergence being caused by missing Jacobian elements
> representing interaction between the sources)?
>
> - Adrian
> On 20/05/24 12:41 pm, Matthew Knepley wrote:
>
> On Sun, May 19, 2024 at 8:25 PM Barry Smith  wrote:
>
>> You can call MatSetOption(mat,MAT_NEW_NONZERO_LOCATION_ERR) then insert
>> the new values. If it is just a handful of new insertions the extra time
>> should be small. Making a copy of the matrix won't give you a new matrix
>> that is any faster to
>> ZjQcmQRYFpfptBannerStart
>> This Message Is From an External Sender
>> This message came from outside your organization.
>>
>> ZjQcmQRYFpfptBannerEnd
>>
>>You can call MatSetOption(mat,MAT_NEW_NONZERO_LOCATION_ERR) then
>> insert the new values. If it is just a handful of new insertions the extra
>> time should be small.
>>
>> Making a copy of the matrix won't give you a new matrix that is any
>> faster to insert into so best to just use the same matrix.
>>
>
> Let me add to Barry's answer. The preallocation infrastructure is now not
> strictly necessary. It is possible to just add all your nonzeros in and
> assembly,  and the performance will be pretty good (uses hashing etc). So
> if just adding a few nonzeros does not work, we can go this route.
>
>   Thanks,
>
>  Matt
>
>
>>   Barry
>>
>>
>> On May 19, 2024, at 7:44 PM, Adrian Croucher 
>> wrote:
>>
>> This Message Is From an External Sender
>> This message came from outside your organization.
>>
>> hi,
>>
>> I have a Jacobian matrix created using DMCreateMatrix(). What would be
>> the best way to add extra nonzero entries into it?
>>
>> I'm guessing that DMCreateMatrix() allocates the storage so the nonzero
>> structure can't really be easily modified. Would it be a case of
>> creating a new matrix, copying the nonzero entries from the original one
>> and then adding the extra ones, before calling MatSetUp() or similar? If
>> so, how exactly would you copy the nonzero structure from the original
>> matrix?
>>
>> Background: the flow problem I'm solving (on a DMPlex with finite volume
>> method) has complex source terms that depend on the solution (e.g.
>> pressure), and can also depend on other source terms. A simple example
>> is when fluid is extracted from one location, with a pressure-dependent
>> flow rate, and some of it is then reinjected in another location. This
>> can result in poor nonlinear solver convergence. I think the reason is
>> that there are effectively missing Jacobian entries in the row for the
>> reinjection cell, which should have an additional dependence on the
>> solution in the cell where fluid is extracted.
>>
>> - Adrian
>>
>>
> --
> Dr Adrian Croucher
> Senior Research Fellow
> Department of Engineering Science
> Waipapa Taumata Rau / University of Auckland, New Zealand
> email: a.crouc...@auckland.ac.nz
> tel: +64 (0)9 923 4611
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eoJE8Eq9V_WNo4aDVL3rN9-rq3xKlASAxI-BbyITAUnKZ2Se208TWSxPRfxptNTBq0ZDwJ7rPdCyNyy2-i4i$
  
<https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eoJE8Eq9V_WNo4aDVL3rN9-rq3xKlASAxI-BbyITAUnKZ2Se208TWSxPRfxptNTBq0ZDwJ7rPdCyN0s5AfvC$
 >

Re: [petsc-users] Modify matrix nonzero structure

[Matthew Knepley]

On Sun, May 19, 2024 at 8:25 PM Barry Smith  wrote:

> You can call MatSetOption(mat,MAT_NEW_NONZERO_LOCATION_ERR) then insert
> the new values. If it is just a handful of new insertions the extra time
> should be small. Making a copy of the matrix won't give you a new matrix
> that is any faster to
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
>
>You can call MatSetOption(mat,MAT_NEW_NONZERO_LOCATION_ERR) then insert
> the new values. If it is just a handful of new insertions the extra time
> should be small.
>
> Making a copy of the matrix won't give you a new matrix that is any
> faster to insert into so best to just use the same matrix.
>

Let me add to Barry's answer. The preallocation infrastructure is now not
strictly necessary. It is possible to just add all your nonzeros in and
assembly,  and the performance will be pretty good (uses hashing etc). So
if just adding a few nonzeros does not work, we can go this route.

  Thanks,

 Matt


>   Barry
>
>
> On May 19, 2024, at 7:44 PM, Adrian Croucher 
> wrote:
>
> This Message Is From an External Sender
> This message came from outside your organization.
>
> hi,
>
> I have a Jacobian matrix created using DMCreateMatrix(). What would be
> the best way to add extra nonzero entries into it?
>
> I'm guessing that DMCreateMatrix() allocates the storage so the nonzero
> structure can't really be easily modified. Would it be a case of
> creating a new matrix, copying the nonzero entries from the original one
> and then adding the extra ones, before calling MatSetUp() or similar? If
> so, how exactly would you copy the nonzero structure from the original
> matrix?
>
> Background: the flow problem I'm solving (on a DMPlex with finite volume
> method) has complex source terms that depend on the solution (e.g.
> pressure), and can also depend on other source terms. A simple example
> is when fluid is extracted from one location, with a pressure-dependent
> flow rate, and some of it is then reinjected in another location. This
> can result in poor nonlinear solver convergence. I think the reason is
> that there are effectively missing Jacobian entries in the row for the
> reinjection cell, which should have an additional dependence on the
> solution in the cell where fluid is extracted.
>
> - Adrian
>
> --
> Dr Adrian Croucher
> Senior Research Fellow
> Department of Engineering Science
> Waipapa Taumata Rau / University of Auckland, New Zealand
> email: a.crouc...@auckland.ac.nz
> tel: +64 (0)9 923 4611
>
>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ZAicuPSGpbQ0tcP9ZNf7TPN7b-DE4XuRcqUtAUEsr_yk9p2EAWn5wSAIdkxE7SYHsDeBftI_qWDRw8aUfXLK$
  

Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

[Matthew Knepley]

On Sat, May 18, 2024 at 7:29 PM Shatanawi, Sawsan Muhammad <
sawsan.shatan...@wsu.edu> wrote:

> Hello everyone,
>
> Thank you all for your feedback, it helped me a lot.
>
> I read the PETSc document and examples related to the Jacobian and
> modified my code.
> Now I am getting errors related to the memory access.  I tried to debug
> but couldn't find out how to fix it.
>

I think the right way to find the bug is to start with either

a) Valgrind 
(https://urldefense.us/v3/__http://www.valgrind.org__;!!G_uCfscf7eWS!an0a5TBb_2F0Yl9de3r2Rr5Lde7DTjTZJaHL4BLVhBks6EHjOrKhtOsm9NFyMydeBwa6UGgfAXF46n-sW_2y$
 )

or

b) Address Sanitizer, which is a feature of the clang compiler
(-fsanitize=address)

You can also run with -malloc_debug in PETSc, which can detect some errors.

  Thanks,

  Matt


> sshatanawi/SawSim: Sawsan's Simulation (SawSim) is a groundwater dynamics
> scheme to be integrated into Noah MP land surface model (github.com)
> <https://urldefense.us/v3/__https://github.com/sshatanawi/SawSim__;!!G_uCfscf7eWS!an0a5TBb_2F0Yl9de3r2Rr5Lde7DTjTZJaHL4BLVhBks6EHjOrKhtOsm9NFyMydeBwa6UGgfAXF46t8nZWfw$
>  >   is the link to the code in the
> GitHub, I would appreciate it if you could have a look at it and guide me
> how to fix it.
> I believe the problem is in the memory of LHS and res_vector because they
> are new vectors I created.
>
> Thank you in advance for your help, I really appreciate it
>
> Bests,
> Sawsan
> --
> *From:* Matthew Knepley 
> *Sent:* Saturday, May 11, 2024 1:56 AM
> *To:* Shatanawi, Sawsan Muhammad 
> *Cc:* Barry Smith ; petsc-users@mcs.anl.gov <
> petsc-users@mcs.anl.gov>
> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran
> Groundwater Flow Simulation Code
>
>
> *[EXTERNAL EMAIL]*
> On Fri, May 10, 2024 at 6:30 PM Shatanawi, Sawsan Muhammad via petsc-users
>  wrote:
>
> Good afternoon, I have tried SNESComputeJacobianDefaultColor(), but the
> arguments needed are confusing me. Would you please have a look at my code
> and the error messages I am getting? I didn't understand what the nonzero
> values of the sparse
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
> Good afternoon,
>
> I have tried SNESComputeJacobianDefaultColor(), but the arguments needed
> are confusing me.
>
> Would you please have a look at my code and the error messages I am
> getting?
> I didn't understand what the nonzero values of the sparse Jacobian would
> be.
>
>
> 1) You are not checking any of the return values from PETSc calls. Look at
> the PETSc Fortran examples.
>  You  should wrap all PETSc calls in PetscCall() or PetscCallA().
>
> 2) You are not intended to directly
> call SNESComputeJacobianDefaultColor(). PETSc will do this automatically if
> you do not set of Jacobian function.
>
> 3) As Barry points out, coloring will not work unless we understand the
> nonzero structure of your Jacobian. This can happen by either:
>
>   a) Using a DM: This is the easiest. Find the type that matches your
> grid, or
>
>   b) Preallocating your Jacobian: Here you give the nonzero structure of
> your Jacobian, but not the values.
>Currently you do not do this. Instead you just give the size, not
> the nonzero structure (columns for
>each row).
>
>   Thanks,
>
>  Matt
>
>
> Thank you for your patience and help
> Bests,
> Sawsan
>
>
> --
> *From:* Barry Smith 
> *Sent:* Thursday, May 9, 2024 12:05 PM
> *To:* Shatanawi, Sawsan Muhammad 
> *Cc:* petsc-users@mcs.anl.gov 
> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran
> Groundwater Flow Simulation Code
>
>
> *[EXTERNAL EMAIL]*
>
>
> On May 9, 2024, at 2:52 PM, Shatanawi, Sawsan Muhammad via petsc-users <
> petsc-users@mcs.anl.gov> wrote:
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Hello everyone,
>
> Thank you for your responses and feedback,
>
> I checked PFLOTRAN and found that it is a model to simulate groundwater
> flow, contaminant transport, and other subsurface processes.
> my goal is not to simulate the groundwater flow, my goal is to develop a
> code from scratch to simulate the groundwater flow with specific
> conditions, and then integrate this code with land surface models.
> Later, the simulation of this code will be on a large scale.
>
> I want PETSc to calculate the Jacobian because the system is large and has
> complex nonlinear behavior, and I don’t risk calculating the derivative by
> myself.
> My A

Re: [petsc-users] GMSH entities

[Matthew Knepley]

On Tue, May 14, 2024 at 9:07 AM Matthew Knepley  wrote:

> On Mon, May 13, 2024 at 10:04 PM Adrian Croucher <
> a.crouc...@auckland.ac.nz> wrote:
>
>> On 14/05/24 1:44 pm, Matthew Knepley wrote:
>>
>> I wish GMsh was clearer about what is optional:
>> https://urldefense.us/v3/__https://gmsh.info/doc/texinfo/gmsh.html*MSH-file-format__;Iw!!G_uCfscf7eWS!fewn3DodeIVBmUE15mfIIEWSAHMhFjmoORD8nz_G4QnsaxhItWNFYbe4Lm0noV3Fuzj8ep504ww-M4Z0oroq$
>>  
>> They do talk about it, but not exhaustively. GMsh always writes and
>> $Entities block from what I can tell.
>> I can make it optional, it just might take until after the PETSc Meeting.
>>
>> Looks like $Entities are optional:
>>
>>
>> https://urldefense.us/v3/__https://gitlab.onelab.info/gmsh/gmsh/-/commit/b5feba2af57181ffa946d3f0c494b014603c6efa__;!!G_uCfscf7eWS!fewn3DodeIVBmUE15mfIIEWSAHMhFjmoORD8nz_G4QnsaxhItWNFYbe4Lm0noV3Fuzj8ep504ww-M57ksfou$
>>  
>>
>> I can also load a GMSH 4.1 file without $Entities into GMSH itself and it
>> doesn't complain, suggesting that they are indeed optional.
>>
>> Yes, but they are not careful to specify when a file can be inconsistent.
> For instance, omitting the $Entities, but then specifying entity numbers in
> the $Nodes block. I think they also thought this was inconsistent, but then
> got user complaints. The minimal example they show does exactly this.
>
>> If the $Entities aren't strictly needed for anything in DMPlex (which I'm
>> guessing they aren't, as the GMSH file format 2.2 doesn't even have them)
>> then it would be useful not to require them.
>>
> I put in some code for this:
> https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7546__;!!G_uCfscf7eWS!fewn3DodeIVBmUE15mfIIEWSAHMhFjmoORD8nz_G4QnsaxhItWNFYbe4Lm0noV3Fuzj8ep504ww-M-tMQE24$
>  
>
> It just ignores entity numbers when there is no section.
>

This merged, so now it should be fixed for you.

  Thanks,

 Matt


>   Thanks,
>
>  Matt
>
>> - Adrian
>>
>> --
>> Dr Adrian Croucher
>> Senior Research Fellow
>> Department of Engineering Science
>> Waipapa Taumata Rau / University of Auckland, New Zealand
>> email: a.crouc...@auckland.ac.nz
>> tel: +64 (0)9 923 4611
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fewn3DodeIVBmUE15mfIIEWSAHMhFjmoORD8nz_G4QnsaxhItWNFYbe4Lm0noV3Fuzj8ep504ww-M7SPCKFm$
>  
> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fewn3DodeIVBmUE15mfIIEWSAHMhFjmoORD8nz_G4QnsaxhItWNFYbe4Lm0noV3Fuzj8ep504ww-MzikdTxP$
>  >
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fewn3DodeIVBmUE15mfIIEWSAHMhFjmoORD8nz_G4QnsaxhItWNFYbe4Lm0noV3Fuzj8ep504ww-M7SPCKFm$
  
<https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fewn3DodeIVBmUE15mfIIEWSAHMhFjmoORD8nz_G4QnsaxhItWNFYbe4Lm0noV3Fuzj8ep504ww-MzikdTxP$
 >

Re: [petsc-users] GMSH entities

[Matthew Knepley]

On Mon, May 13, 2024 at 10:04 PM Adrian Croucher 
wrote:

> On 14/05/24 1:44 pm, Matthew Knepley wrote:
>
> I wish GMsh was clearer about what is optional:
> https://urldefense.us/v3/__https://gmsh.info/doc/texinfo/gmsh.html*MSH-file-format__;Iw!!G_uCfscf7eWS!YJN5how37EbmfjwDvfPAsVSCQdWejJn8symxZ83hj94omk6Mh9imh2qOrFqZbsZRM_3W3G5YIn5lZK2KzQlj$
>  
> They do talk about it, but not exhaustively. GMsh always writes and
> $Entities block from what I can tell.
> I can make it optional, it just might take until after the PETSc Meeting.
>
> Looks like $Entities are optional:
>
>
> https://urldefense.us/v3/__https://gitlab.onelab.info/gmsh/gmsh/-/commit/b5feba2af57181ffa946d3f0c494b014603c6efa__;!!G_uCfscf7eWS!YJN5how37EbmfjwDvfPAsVSCQdWejJn8symxZ83hj94omk6Mh9imh2qOrFqZbsZRM_3W3G5YIn5lZMbTrLlQ$
>  
>
> I can also load a GMSH 4.1 file without $Entities into GMSH itself and it
> doesn't complain, suggesting that they are indeed optional.
>
> Yes, but they are not careful to specify when a file can be inconsistent.
For instance, omitting the $Entities, but then specifying entity numbers in
the $Nodes block. I think they also thought this was inconsistent, but then
got user complaints. The minimal example they show does exactly this.

> If the $Entities aren't strictly needed for anything in DMPlex (which I'm
> guessing they aren't, as the GMSH file format 2.2 doesn't even have them)
> then it would be useful not to require them.
>
I put in some code for this:
https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7546__;!!G_uCfscf7eWS!YJN5how37EbmfjwDvfPAsVSCQdWejJn8symxZ83hj94omk6Mh9imh2qOrFqZbsZRM_3W3G5YIn5lZHVFZ3e3$
 

It just ignores entity numbers when there is no section.

  Thanks,

 Matt

> - Adrian
>
> --
> Dr Adrian Croucher
> Senior Research Fellow
> Department of Engineering Science
> Waipapa Taumata Rau / University of Auckland, New Zealand
> email: a.crouc...@auckland.ac.nz
> tel: +64 (0)9 923 4611
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YJN5how37EbmfjwDvfPAsVSCQdWejJn8symxZ83hj94omk6Mh9imh2qOrFqZbsZRM_3W3G5YIn5lZFEALcjs$
  
<https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YJN5how37EbmfjwDvfPAsVSCQdWejJn8symxZ83hj94omk6Mh9imh2qOrFqZbsZRM_3W3G5YIn5lZFz1kNMU$
 >

Re: [petsc-users] DMPlex periodic face coordinates

[Matthew Knepley]

On Tue, May 14, 2024 at 12:14 AM Matteo Semplice <
matteo.sempl...@uninsubria.it> wrote:

> Dear petsc-users, I am playing with DMPlexGetCellCoordinates and observing
> that it returns correct periodic coordinates for cells, but not for faces.
> More precisely, adding PetscCall(DMPlexGetHeightStratum(dm, 1, ,
> )); for
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
>
> Dear petsc-users,
>
> I am playing with DMPlexGetCellCoordinates and observing that it
> returns correct periodic coordinates for cells, but not for faces.
>
> More precisely, adding
>
> PetscCall(DMPlexGetHeightStratum(dm, 1, , ));
> for (f = fStart; f < fEnd; ++f) {
>   const PetscScalar *array;
>   PetscScalar   *x = NULL;
>   PetscInt   ndof;
>   PetscBool  isDG;
>
>   PetscCall(DMPlexGetCellCoordinates(dm, f, , , , ));
>   PetscCheck(ndof % cdim == 0, PETSC_COMM_SELF, PETSC_ERR_ARG_INCOMP,
> "ndof not divisible by cdim");
>   PetscCall(PetscPrintf(PETSC_COMM_SELF, "Face #%" PetscInt_FMT "
> coordinates\n", f - fStart));
>   for (PetscInt i = 0; i < ndof; i += cdim)
> PetscCall(PetscScalarView(cdim, [i], PETSC_VIEWER_STDOUT_SELF));
>   PetscCall(DMPlexRestoreCellCoordinates(dm, f, , , ,
> ));
> }
>
> to src/dm/impls/plex/tutorials/ex8.c, I get
>
> $ ./ex8 -dm_plex_dim 2 -petscspace_degree 1 -dm_plex_simplex 0
> -dm_plex_box_faces 3,2 -dm_plex_box_bd periodic,none -dm_view -view_coord
> DM Object: box 1 MPI process
>   type: plex
> box in 2 dimensions:
>   Number of 0-cells per rank: 9
>   Number of 1-cells per rank: 15
>   Number of 2-cells per rank: 6
> Periodic mesh (PERIODIC, NONE) coordinates localized
>
> [...]
>
> Element #2 coordinates
>  0:   6.6667e-01   0.e+00
>  0:   1.e+00   0.e+00  <<<- correct
>  0:   1.e+00   5.e-01
>  0:   6.6667e-01   5.e-01
> [...]
>
> Face #0 coordinates
>  0:   0.e+00   0.e+00
>  0:   3.e-01   0.e+00
> Face #1 coordinates
>  0:   3.e-01   0.e+00
>  0:   6.6667e-01   0.e+00
> Face #2 coordinates
>  0:   6.6667e-01   0.e+00
>  0:   0.e+00   0.e+00  <<< should be (0.66,0.00) and
> (1.00,0.00)
>
> Is there a way to recover correct periodic coordinates in the case of
> periodic DMPLex?
>
> The way that periodic coordinates work is that it stores a DG coordinate
field by cell. Faces default back to the vertices. You could think about
also putting DG coordinates on faces, but no one had asked, and it is
potentially expensive.

If you really need them to keep going, face coordinates can be extracted
from cell coordinates. Otherwise, I can do it after the PETSc Meeting. Or,
we are happy to take contributions adding this.

  Thanks,

 Matt

> Thanks in advance
>
> Matteo
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!Z9-i-9O8yBcNimBXW2alc9XHsn9MWWFwaobajlfkDcoghfrtZlVX4CRMW0BdrXaX2hOabPupZN8g9FrtoNf2$
  

Re: [petsc-users] GMSH entities

[Matthew Knepley]

On Mon, May 13, 2024 at 9:33 PM Adrian Croucher 
wrote:

> hi, We often create meshes in GMSH format using the meshio library. This
> works OK if we stick to GMSH file format 2. 2. If we use GMSH file format
> 4. 1, DMPlex can't read them because it expects the "Entities" section to
> be present: [0]PETSC
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
>
> hi,
>
> We often create meshes in GMSH format using the meshio library. This
> works OK if we stick to GMSH file format 2.2.
>
> If we use GMSH file format 4.1, DMPlex can't read them because it
> expects the "Entities" section to be present:
>
> [0]PETSC ERROR: Unexpected data in file
> [0]PETSC ERROR: File is not a valid Gmsh file, expecting $Entities, not
> $Nodes
> [0]PETSC ERROR: See 
> https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!ZfBS1KM5EBZ7ZJIu6lBKFcclVMmXteXsW8m9HBEZ5tIf0u_3duEFt9eXKF7FcorQSAQqD5SbJrYh4C8rX676S_IMI_sp6naX$
>  for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision:
> v3.21.1-124-g2d06e2faec8  GIT Date: 2024-05-08 19:31:33 +
> [0]PETSC ERROR: waiwera on a main-debug named EN438880 by acro018 Tue
> May 14 11:25:54 2024
> [0]PETSC ERROR: Configure options --with-x --download-hdf5
> --download-zlib --download-netcdf --download-pnetcdf --download-exodusii
> --download-triangle --download-ptscotch --download-chaco --download-hypre
> [0]PETSC ERROR: #1 GmshExpect() at
> /home/acro018/software/PETSc/code/src/dm/impls/plex/plexgmsh.c:270
> [0]PETSC ERROR: #2 DMPlexCreateGmsh() at
> /home/acro018/software/PETSc/code/src/dm/impls/plex/plexgmsh.c:1608
> [0]PETSC ERROR: #3 DMPlexCreateGmshFromFile() at
> /home/acro018/software/PETSc/code/src/dm/impls/plex/plexgmsh.c:1469
> [0]PETSC ERROR: #4 DMPlexCreateFromFile() at
> /home/acro018/software/PETSc/code/src/dm/impls/plex/plexcreate.c:5804
>
> By default meshio doesn't seem to write the Entities section. From what
> I can gather, it is optional.
>
> Am I right that this section is not optional in DMPlex?
>
>
I wish GMsh was clearer about what is optional:
https://urldefense.us/v3/__https://gmsh.info/doc/texinfo/gmsh.html*MSH-file-format__;Iw!!G_uCfscf7eWS!fS2tI8aC3rUnGOv-LbLPj1PRyijAPB-EO54CwUBP6pNFD-VKeGE98pOHepNMSTP_krHBKTZNk92Bx01NrC-T$
 
They do talk about it, but not exhaustively. GMsh always writes and
$Entities block from what I can tell.
I can make it optional, it just might take until after the PETSc Meeting.

  Thanks,

Matt

- Adrian
>
> --
> Dr Adrian Croucher
> Senior Research Fellow
> Department of Engineering Science
> Waipapa Taumata Rau / University of Auckland, New Zealand
> email: a.crouc...@auckland.ac.nz
> tel: +64 (0)9 923 4611
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fS2tI8aC3rUnGOv-LbLPj1PRyijAPB-EO54CwUBP6pNFD-VKeGE98pOHepNMSTP_krHBKTZNk92Bx1gj7y2J$
  

Re: [petsc-users] [SLEPc] Best method to compute all eigenvalues of a MatShell

[Matthew Knepley]

On Mon, May 13, 2024 at 1:40 PM Sreeram R Venkat 
wrote:

> I have a MatShell object that computes matrix-vector products of a dense
> symmetric matrix of size NxN. The MatShell does not actually form the dense
> matrix, so it is never in memory/storage. For my application, N ranges from
> 1e4 to 1e5. I want
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
> I have a MatShell object that computes matrix-vector products of a dense
> symmetric matrix of size NxN. The MatShell does not actually form the dense
> matrix, so it is never in memory/storage. For my application, N ranges from
> 1e4 to 1e5.
>
> I want to compute the full spectrum of this matrix. For an example with N
> ~1e4, I was able to use SLEPc's Krylov-Schur solver to get the spectrum in
> about 3 hours running on 6 A100 GPUs. There, I had set the MPD to 2000.
> Before moving on to larger matrices, I wanted to check whether this is the
> best way to go about it. I saw on other posts that for computing full
> spectra of dense matrices, it is generally better to go with
> LAPACK/SCALAPACK. Is the same true for MatShells of dense matrices? The
> matrix-vector products with the shell themselves are really cheap, so I can
> form the dense matrix with MatComputeOperator() and store it to later
> compute with another solver if needed. If SLEPc is not a bad option, what
> is a good way to select MPD/NCV?
>

You can select LAPACK through SLEPc. I would definitely try it since it is
easy. For the complete spectrum, it is likely to be faster (modulo how fast
KS converges).

  Thanks,

Matt


> I do need the full spectrum here since I am trying to analyze how the
> spectral decay changes for different problem configurations.
>
> Thanks for your help,
> Sreeram
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cLlaAsyk3G1fWIZGlTet7-bB1VMxzmbrmnjxP0IPtxxAA-mYn4vGvI9tzI2x7GXRMPqOjNBQZZrrDoH5JihG$
  

Re: [petsc-users] HDF5 time step count

[Matthew Knepley]

On Sun, May 12, 2024 at 10:42 PM Adrian Croucher 
wrote:

> hi Matt,
> On 11/05/24 4:12 am, Matthew Knepley wrote:
>
> Thanks. I tried it out and the error below was raised, looks like it's
>> when it tries to destroy the viewer. It still runs ok when
>> DMGetOutputSequenceLength() isn't called.
>>
> Sorry, it looks like I did not close the group. I pushed an update. If
> that is alright, I will merge it.
>
> Thanks, it does appear to work now. That's going to be very useful!
>
> Excellent! I will get it merged in.

  Thanks,

Matt

> - Adrian
>
> --
> Dr Adrian Croucher
> Senior Research Fellow
> Department of Engineering Science
> Waipapa Taumata Rau / University of Auckland, New Zealand
> email: a.crouc...@auckland.ac.nz
> tel: +64 (0)9 923 4611
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fa_sX8leJlX2EqUMrX8Mgt-V_CgjZo1q5KtjgUiqXJKLzHcUH1Ko3Xgwt49Ji5tcIxwutLtuTM1N_Y15u93u$
  
<https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fa_sX8leJlX2EqUMrX8Mgt-V_CgjZo1q5KtjgUiqXJKLzHcUH1Ko3Xgwt49Ji5tcIxwutLtuTM1N_RFBcCyv$
 >

Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

[Matthew Knepley]

On Fri, May 10, 2024 at 6:30 PM Shatanawi, Sawsan Muhammad via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Good afternoon, I have tried SNESComputeJacobianDefaultColor(), but the
> arguments needed are confusing me. Would you please have a look at my code
> and the error messages I am getting? I didn't understand what the nonzero
> values of the sparse
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
> Good afternoon,
>
> I have tried SNESComputeJacobianDefaultColor(), but the arguments needed
> are confusing me.
>
> Would you please have a look at my code and the error messages I am
> getting?
> I didn't understand what the nonzero values of the sparse Jacobian would
> be.
>

1) You are not checking any of the return values from PETSc calls. Look at
the PETSc Fortran examples.
 You  should wrap all PETSc calls in PetscCall() or PetscCallA().

2) You are not intended to directly call SNESComputeJacobianDefaultColor().
PETSc will do this automatically if you do not set of Jacobian function.

3) As Barry points out, coloring will not work unless we understand the
nonzero structure of your Jacobian. This can happen by either:

  a) Using a DM: This is the easiest. Find the type that matches your grid,
or

  b) Preallocating your Jacobian: Here you give the nonzero structure of
your Jacobian, but not the values.
   Currently you do not do this. Instead you just give the size, not
the nonzero structure (columns for
   each row).

  Thanks,

 Matt


> Thank you for your patience and help
> Bests,
> Sawsan
>
>
> --
> *From:* Barry Smith 
> *Sent:* Thursday, May 9, 2024 12:05 PM
> *To:* Shatanawi, Sawsan Muhammad 
> *Cc:* petsc-users@mcs.anl.gov 
> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran
> Groundwater Flow Simulation Code
>
>
> *[EXTERNAL EMAIL]*
>
>
> On May 9, 2024, at 2:52 PM, Shatanawi, Sawsan Muhammad via petsc-users <
> petsc-users@mcs.anl.gov> wrote:
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Hello everyone,
>
> Thank you for your responses and feedback,
>
> I checked PFLOTRAN and found that it is a model to simulate groundwater
> flow, contaminant transport, and other subsurface processes.
> my goal is not to simulate the groundwater flow, my goal is to develop a
> code from scratch to simulate the groundwater flow with specific
> conditions, and then integrate this code with land surface models.
> Later, the simulation of this code will be on a large scale.
>
> I want PETSc to calculate the Jacobian because the system is large and has
> complex nonlinear behavior, and I don’t risk calculating the derivative by
> myself.
> My A-Matrix has parts of source terms that depend on the flow fields, and
> independent parts will be in the RHS vector.
>
>
> With coloring SNESComputeJacobianDefaultColor() PETSc can compute
> Jacobian's pretty efficiently. You do not to provide the residual function
> and you need to provide the nonzero pattern of the sparse Jacobian; that is
> what residual components f_i are coupled to what input variables in the
> array x_i. This information comes from your PDE and discretization and
> appears implicitly in your residual function.
>
>   Barry
>
>
> I hope I have answered your questions, and I apologize that I wasn’t clear
> from the beginning, I was trying to keep my descriptions brief.
>
> Bests,
> Sawsan
> --
> *From:* Matthew Knepley 
> *Sent:* Tuesday, May 7, 2024 5:17 PM
> *To:* Shatanawi, Sawsan Muhammad 
> *Cc:* petsc-users@mcs.anl.gov 
> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran
> Groundwater Flow Simulation Code
>
> *[EXTERNAL EMAIL]*
> On Tue, May 7, 2024 at 2:23 PM Shatanawi, Sawsan Muhammad via petsc-users <
> petsc-users@mcs.anl.gov> wrote:
>
> This Message Is From an External Sender
> This message came from outside your organization.
>
> Hello everyone,
>
>
> I hope this email finds you well.
>
>
>  My Name is Sawsan Shatanawi, and I was developing a Fortran code for
> simulating groundwater flow in a 3D system with nonlinear behavior.  I
> solved the nonlinear system using the PCG solver and Picard iteration, but
> I did not get good results although I checked my matrix and RHS and
> everything, I decided to change my solver to Newton Rapson method.
> I checked PETSc documents but I have a few questions:
> 1) My groundwater system is time-dependent, so should I use TS only
> instead of SNES?
>
>
> You could use TS, but it is not necessary. You could

Re: [petsc-users] HDF5 time step count

[Matthew Knepley]

On Fri, May 10, 2024 at 1:01 AM Adrian Croucher 
wrote:

> hi Matt,
> On 10/05/24 12:15 am, Matthew Knepley wrote:
>
> I just tried to test it, but there doesn't seem to be a Fortran interface
>> for DMGetOutputSequenceLength().
>>
> Pushed.
>
> Thanks. I tried it out and the error below was raised, looks like it's
> when it tries to destroy the viewer. It still runs ok when
> DMGetOutputSequenceLength() isn't called.
>
> Sorry, it looks like I did not close the group. I pushed an update. If
that is alright, I will merge it.

  Thanks

 Matt

> - Adrian
>
> [0]PETSC ERROR: Error in external library
> [0]PETSC ERROR: Error in HDF5 call H5Fclose() Status -1
> [0]PETSC ERROR: See 
> https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!cr7LJSXkch_z_JwHbQJLw6lOHNHIwlYHSs2EKuOgdO4WPY3yBsl64nAdyRiQX9blQC8KVSJZObA0YA9zG9uD$
>   for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.21.1-124-g2d06e2faec8
> GIT Date: 2024-05-08 19:31:33 +
> [0]PETSC ERROR: ./initial_test on a main-debug named EN438880 by acro018
> Fri May 10 16:55:34 2024
> [0]PETSC ERROR: Configure options --with-x --download-hdf5 --download-zlib
> --download-netcdf --download-pnetcdf --download-exodusii
> --download-triangle --download-ptscotch --download-chaco --download-hypre
> [0]PETSC ERROR: #1 PetscViewerFileClose_HDF5() at
> /home/acro018/software/PETSc/code/src/sys/classes/viewer/impls/hdf5/hdf5v.c:107
> [0]PETSC ERROR: #2 PetscViewerDestroy_HDF5() at
> /home/acro018/software/PETSc/code/src/sys/classes/viewer/impls/hdf5/hdf5v.c:126
> [0]PETSC ERROR: #3 PetscViewerDestroy() at
> /home/acro018/software/PETSc/code/src/sys/classes/viewer/interface/view.c:101
> [0]PETSC ERROR: #4 ../src/initial.F90:497
>
> --
> Dr Adrian Croucher
> Senior Research Fellow
> Department of Engineering Science
> Waipapa Taumata Rau / University of Auckland, New Zealand
> email: a.crouc...@auckland.ac.nz
> tel: +64 (0)9 923 4611
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cr7LJSXkch_z_JwHbQJLw6lOHNHIwlYHSs2EKuOgdO4WPY3yBsl64nAdyRiQX9blQC8KVSJZObA0YLHcR0C8$
  
<https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cr7LJSXkch_z_JwHbQJLw6lOHNHIwlYHSs2EKuOgdO4WPY3yBsl64nAdyRiQX9blQC8KVSJZObA0YDfia_zg$
 >

Re: [petsc-users] HDF5 time step count

[Matthew Knepley]

On Wed, May 8, 2024 at 10:46 PM Adrian Croucher 
wrote:

> hi Matt,
> On 9/05/24 3:00 am, Matthew Knepley wrote:
>
> Sorry about the delay. I had lost track of this. Can you look at branch
>
>   knepley/feature-hdf5-seq-len
>
> I have not made a test yet, but if this works for you, I will make a test
> and merge it in.
>
> Thanks for looking at that.
>
> I just tried to test it, but there doesn't seem to be a Fortran interface
> for DMGetOutputSequenceLength().
>
Pushed.

> Another odd thing I found was that DMPlexConstructGhostCells() seemed to
> complain if I passed in PETSC_NULL_INTEGER for the parameter numGhostCells
> (which works ok in v3.21.1) - the compiler complained about a rank-1/scalar
> mismatch. Can you no longer pass in a null integer for that? It works if I
> declare a dummy integer and pass it in.
>
Oh, Barry has been redoing the Fortran stubs to try and get more automated.
He might have eliminated the hand copy by mistake (since we cannot yet
automate checking for NULL). I will ask him. For now, can we put a dummy in?

  Thanks,

 Matt

> - Adrian
>
> --
> Dr Adrian Croucher
> Senior Research Fellow
> Department of Engineering Science
> Waipapa Taumata Rau / University of Auckland, New Zealand
> email: a.crouc...@auckland.ac.nz
> tel: +64 (0)9 923 4611
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YEiiqBTCh7zSAbsfhvbrdIgnFEYpbB4rUltLdnm3IhphEaC4qf6ny1HMQl-j374n_GwyG96OGNk1F-AzQUVb$
  
<https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YEiiqBTCh7zSAbsfhvbrdIgnFEYpbB4rUltLdnm3IhphEaC4qf6ny1HMQl-j374n_GwyG96OGNk1F8dmZwZL$
 >

Re: [petsc-users] HDF5 time step count

[Matthew Knepley]

On Wed, May 1, 2024 at 11:03 PM Adrian Croucher 
wrote:

> hi Matt & all,
>
> I just had a query from one of my users which prompted me to see if any
> progress had been made on the issue below - using PETSc to get the number
> of time steps in an HDF5 file.
>
> I can't see anything new in PETSc on this - I did try using
> PetscViewerHDF5ReadSizes() to see if that would do it, but it seems it
> doesn't. If I use that on the "time" dataset it just returns 1.
>
> Sorry about the delay. I had lost track of this. Can you look at branch

  knepley/feature-hdf5-seq-len

I have not made a test yet, but if this works for you, I will make a test
and merge it in.

  Thanks!

 Matt

> Regards, Adrian
> On 11/10/21 2:08 pm, Adrian Croucher wrote:
>
> On 10/11/21 11:59 AM, Matthew Knepley wrote:
>
> On Sun, Oct 10, 2021 at 6:51 PM Adrian Croucher 
> wrote:
>
>> hi
>>
>> Is there any way to query the PETSc HDF5 viewer to find the number of
>> time steps in the file?
>>
>> A common use case I have is that an HDF5 file from a previous simulation
>> is used to get initial conditions for a subsequent run. The most common
>> thing you want to do is restart from the last set of results in the
>> previous output. To do that you need to know how many time steps there
>> are, so you can set the output index to be the last one.
>>
>> I thought maybe I could just query the size of the "time" dataset, but I
>> can't even see any obvious way to do that using the viewer functions.
>>
>
> There is nothing in there that does it right now. Do you know how to do it
> in HDF5?
> If so, I can put it in. Otherwise, I will have to learn more HDF5 :)
>
>
> I haven't actually tried this myself but it looks like what you do is:
>
> 1) get the dataspace for the dataset (in our case the "time" dataset):
>
> hid_t dspace = H5Dget_space(dset);
>
> 2) Get the dimensions of the dataspace:
>
> const int ndims = 1;
>
> hsize_t dims[ndims];
> H5Sget_simple_extent_dims(dspace, dims, NULL);
>
> The first element of dims should be the number of time steps. Here I've
> assumed the number of dimensions of the time dataset is 1. In general you
> can instead query the rank of the dataspace using
> H5Sget_simple_extent_ndims() to get the rank ndims.
>
> Regards, Adrian
>
> --
> Dr Adrian Croucher
> Senior Research Fellow
> Department of Engineering Science
> Waipapa Taumata Rau / University of Auckland, New Zealand
> email: a.crouc...@auckland.ac.nz
> tel: +64 (0)9 923 4611
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eWw8ZyC03JCY81KS4nuvF6OwPoyRvldIwdZsw_6BlM1OLpJ4cucgOSJ6wetJu-YCCDUSYq0ZI_4rgu343kFn$
  
<https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eWw8ZyC03JCY81KS4nuvF6OwPoyRvldIwdZsw_6BlM1OLpJ4cucgOSJ6wetJu-YCCDUSYq0ZI_4rgriZqJYl$
 >

Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

[Matthew Knepley]

On Tue, May 7, 2024 at 2:23 PM Shatanawi, Sawsan Muhammad via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Hello everyone, I hope this email finds you well. My Name is Sawsan
> Shatanawi, and I was developing a Fortran code for simulating groundwater
> flow in a 3D system with nonlinear behavior. I solved the nonlinear system
> using the PCG solver and
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
>
> Hello everyone,
>
>
>
> I hope this email finds you well.
>
>
>
>  My Name is Sawsan Shatanawi, and I was developing a Fortran code for
> simulating groundwater flow in a 3D system with nonlinear behavior.  I
> solved the nonlinear system using the PCG solver and Picard iteration, but
> I did not get good results although I checked my matrix and RHS and
> everything, I decided to change my solver to Newton Rapson method.
> I checked PETSc documents but I have a few questions:
> 1) My groundwater system is time-dependent, so should I use TS only
> instead of SNES?
>

You could use TS, but it is not necessary. You could use SNES and your own
timestepping. THe advantage of TS is that you can try many different
timesteppers without recoding (just like you can
try many different linear and nonlinear solvers).


> 2) My system has its deltaT, would using deltaT as dt affect my solver, or
> is it better to use TS-PETSc dt? Also, would using PETSc dt affect the
> simulation of the groundwater system
>

It sounds like your dt comes from your timestepper. If you use TS, you
would use the dt from that.


> 3) I want my Jacobian matrix to be calculated by PETSc automatically
>

PETSc can calculate a full Jacobian for smaller problems, or a
finite-difference Jacobian for any problem (but this impacts the solver).
It should be straightfoward to code up the analytic Jacobian. Is there a
reason it would be a problem?


> 4) Do I need to define and calculate the residual vector?
>

Yes.


> My A-Matrix contains coefficients and external sources and my RHS vector
> includes the boundary conditions
>

It is strange that your matrix would contain source terms. Do they depend
on the flow fields?

Barry is right, you should consider PFlotran, and at least know why it
would not work for your problem if you don't use it.

  Thanks,

 Matt


>
> Please find the attached file contains a draft of my code
>
> Thank you in advance for your time and help.
>
>
> Best regards,
>
>  Sawsan
>
> ----------
> *From:* Shatanawi, Sawsan Muhammad 
> *Sent:* Tuesday, January 16, 2024 10:43 AM
> *To:* Junchao Zhang 
> *Cc:* Barry Smith ; Matthew Knepley ;
> Mark Adams ; petsc-users@mcs.anl.gov <
> petsc-users@mcs.anl.gov>
> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran
> Groundwater Flow Simulation Code
>
> Hello all,
>
> Thank you for your valuable help. I will do your recommendations and hope
> it will run without any issues.
>
> Bests,
> Sawsan
>
> --
> *From:* Junchao Zhang 
> *Sent:* Friday, January 12, 2024 8:46 AM
> *To:* Shatanawi, Sawsan Muhammad 
> *Cc:* Barry Smith ; Matthew Knepley ;
> Mark Adams ; petsc-users@mcs.anl.gov <
> petsc-users@mcs.anl.gov>
> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran
> Groundwater Flow Simulation Code
>
>
> *[EXTERNAL EMAIL]*
> Hi, Sawsan,
>First in test_main.F90, you need to call VecGetArrayF90(temp_solution,
> H_vector, ierr) and  VecRestoreArrayF90 (temp_solution, H_vector, ierr)  as
> Barry mentioned.
>Secondly, in the loop of test_main.F90, it calls GW_solver(). Within
> it, it calls PetscInitialize()/PetscFinalize(). But without MPI being
> initialized, PetscInitialize()/PetscFinalize()* can only be called once. *
> do timestep =2 , NTSP
>call GW_boundary_conditions(timestep-1)
> !print *,HNEW(1,1,1)
>call GW_elevation()
>! print *, GWTOP(2,2,2)
>call GW_conductance()
>! print *, CC(2,2,2)
>call GW_recharge()
>! print *, B_Rech(5,4)
>call GW_pumping(timestep-1)
>! print *, B_pump(2,2,2)
>call GW_SW(timestep-1)
> print *,B_RIVER (2,2,2)
>call GW_solver(timestep-1,N)
>call GW_deallocate_loop()
> end do
>
> A solution is to delete PetscInitialize()/PetscFinalize() in
> GW_solver_try.F90 and move it to test_main.F90,  outside the do loop.
>
> diff --git a/test_main.F90 b/test_main.F90
> index b5997c55..107bd3ee 100644
> --- a/test_main.F9

Re: [petsc-users] Reasons for breakdown in preconditioned LSQR

[Matthew Knepley]

On Tue, May 7, 2024 at 8:28 AM Mark Adams  wrote:

> "A^T A being similar to a finite difference Poisson matrix if the rows
> were permuted randomly."
> Normal eqs are a good option in general for rectangular systems and we
> have Poisson solvers.
>

I missed that. Why not first permute back to the Poisson matrix? Then it
would be trivial.

  Thanks,

Matt


> I'm not sure what you mean by "permuted randomly." A random permutation
> of the matrix can kill performance but not math.
>
> Mark
>
>
> On Tue, May 7, 2024 at 8:03 AM Matthew Knepley  wrote:
>
>> On Tue, May 7, 2024 at 5: 12 AM Pierre Jolivet 
>> wrote: On 7 May 2024, at 9: 10 AM, Marco Seiz 
>> wrote: Thanks for the quick response! On 07. 05. 24 14: 24, Pierre Jolivet
>> wrote: On 7 May 2024,
>> ZjQcmQRYFpfptBannerStart
>> This Message Is From an External Sender
>> This message came from outside your organization.
>>
>> ZjQcmQRYFpfptBannerEnd
>> On Tue, May 7, 2024 at 5:12 AM Pierre Jolivet  wrote:
>>
>>> On 7 May 2024, at 9: 10 AM, Marco Seiz  wrote:
>>> Thanks for the quick response! On 07. 05. 24 14: 24, Pierre Jolivet wrote:
>>> On 7 May 2024, at 7: 04 AM, Marco Seiz  wrote: This
>>> Message Is From an External
>>> ZjQcmQRYFpfptBannerStart
>>> This Message Is From an External Sender
>>> This message came from outside your organization.
>>>
>>> ZjQcmQRYFpfptBannerEnd
>>>
>>>
>>> On 7 May 2024, at 9:10 AM, Marco Seiz  wrote:
>>>
>>> Thanks for the quick response!
>>>
>>> On 07.05.24 14:24, Pierre Jolivet wrote:
>>>
>>>
>>>
>>> On 7 May 2024, at 7:04 AM, Marco Seiz  wrote:
>>>
>>> This Message Is From an External Sender
>>> This message came from outside your organization.
>>> Hello,
>>>
>>> something a bit different from my last question, since that didn't
>>> progress so well:
>>> I have a related model which generally produces a rectangular matrix A,
>>> so I am using LSQR to solve the system.
>>> The matrix A has two nonzeros (1, -1) per row, with A^T A being similar
>>> to a finite difference Poisson matrix if the rows were permuted randomly.
>>> The problem is singular in that the solution is only specified up to a
>>> constant from the matrix, with my target solution being a weighted zero
>>> average one, which I can handle by adding a nullspace to my matrix.
>>> However, I'd also like to pin (potentially many) DOFs in the future so I
>>> also tried pinning a single value, and afterwards subtracting the
>>> average from the KSP solution.
>>> This leads to the KSP *sometimes* diverging when I use a preconditioner;
>>> the target size of the matrix will be something like ([1,20] N) x N,
>>> with N ~ [2, 1e6] so for the higher end I will require a preconditioner
>>> for reasonable execution time.
>>>
>>> For a smaller example system, I set up my application to dump the input
>>> to the KSP when it breaks down and I've attached a simple python script
>>> + data using petsc4py to demonstrate the divergence for those specific
>>> systems.
>>> With `python3 lsdiv.py -pc_type lu -ksp_converged_reason` that
>>> particular system shows breakdown, but if I remove the pinned DOF and
>>> add the nullspace (pass -usens) it converges. I did try different PCs
>>> but they tend to break down at different steps, e.g. `python3 lsdiv.py
>>> -usenormal -qrdiv -pc_type qr -ksp_converged_reason` shows the breakdown
>>> for PCQR when I use MatCreateNormal for creating the PC mat, but
>>> interestingly it doesn't break down when I explicitly form A^T A (don't
>>> pass -usenormal).
>>>
>>>
>>> What version are you using? All those commands are returning
>>>  Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1
>>> So I cannot reproduce any breakdown, but there have been recent changes
>>> to KSPLSQR.
>>>
>>> For those tests I've been using PETSc 3.20.5 (last githash was
>>> 4b82c11ab5d ).
>>> I pulled the latest version from gitlab ( 6b3135e3cbe ) and compiled it,
>>> but I had to drop --download-suitesparse=1 from my earlier config due to
>>> errors.
>>> Should I write a separate mail about this?
>>>
>>> The LU example still behaves the same for me (`python3 lsdiv.py -pc_type
>>> lu -ksp_converged_reason` gives DIVERGED_BREAKDOWN, `python3 lsdiv.py
>>> -usens -pc_type lu -ksp_converged_reason` gi

Re: [petsc-users] Reasons for breakdown in preconditioned LSQR

[Matthew Knepley]

On Tue, May 7, 2024 at 5:12 AM Pierre Jolivet  wrote:

> On 7 May 2024, at 9: 10 AM, Marco Seiz  wrote: Thanks
> for the quick response! On 07. 05. 24 14: 24, Pierre Jolivet wrote: On 7
> May 2024, at 7: 04 AM, Marco Seiz  wrote: This
> Message Is From an External
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
>
>
> On 7 May 2024, at 9:10 AM, Marco Seiz  wrote:
>
> Thanks for the quick response!
>
> On 07.05.24 14:24, Pierre Jolivet wrote:
>
>
>
> On 7 May 2024, at 7:04 AM, Marco Seiz  wrote:
>
> This Message Is From an External Sender
> This message came from outside your organization.
> Hello,
>
> something a bit different from my last question, since that didn't
> progress so well:
> I have a related model which generally produces a rectangular matrix A,
> so I am using LSQR to solve the system.
> The matrix A has two nonzeros (1, -1) per row, with A^T A being similar
> to a finite difference Poisson matrix if the rows were permuted randomly.
> The problem is singular in that the solution is only specified up to a
> constant from the matrix, with my target solution being a weighted zero
> average one, which I can handle by adding a nullspace to my matrix.
> However, I'd also like to pin (potentially many) DOFs in the future so I
> also tried pinning a single value, and afterwards subtracting the
> average from the KSP solution.
> This leads to the KSP *sometimes* diverging when I use a preconditioner;
> the target size of the matrix will be something like ([1,20] N) x N,
> with N ~ [2, 1e6] so for the higher end I will require a preconditioner
> for reasonable execution time.
>
> For a smaller example system, I set up my application to dump the input
> to the KSP when it breaks down and I've attached a simple python script
> + data using petsc4py to demonstrate the divergence for those specific
> systems.
> With `python3 lsdiv.py -pc_type lu -ksp_converged_reason` that
> particular system shows breakdown, but if I remove the pinned DOF and
> add the nullspace (pass -usens) it converges. I did try different PCs
> but they tend to break down at different steps, e.g. `python3 lsdiv.py
> -usenormal -qrdiv -pc_type qr -ksp_converged_reason` shows the breakdown
> for PCQR when I use MatCreateNormal for creating the PC mat, but
> interestingly it doesn't break down when I explicitly form A^T A (don't
> pass -usenormal).
>
>
> What version are you using? All those commands are returning
>  Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1
> So I cannot reproduce any breakdown, but there have been recent changes to
> KSPLSQR.
>
> For those tests I've been using PETSc 3.20.5 (last githash was
> 4b82c11ab5d ).
> I pulled the latest version from gitlab ( 6b3135e3cbe ) and compiled it,
> but I had to drop --download-suitesparse=1 from my earlier config due to
> errors.
> Should I write a separate mail about this?
>
> The LU example still behaves the same for me (`python3 lsdiv.py -pc_type
> lu -ksp_converged_reason` gives DIVERGED_BREAKDOWN, `python3 lsdiv.py
> -usens -pc_type lu -ksp_converged_reason` gives CONVERGED_RTOL_NORMAL)
> but the QR example fails since I had to remove suitesparse.
> petsc4py.__version__ reports 3.21.1 and if I rebuild my application,
> then `ldd app` gives me `libpetsc.so
> .3.21
> =>
> /opt/petsc/linux-c-opt/lib/libpetsc.so
> .3.21`
> so it should be using the
> newly built one.
> The application then still eventually yields a DIVERGED_BREAKDOWN.
> I don't have a ~/.petscrc and PETSC_OPTIONS is unset, so if we are on
> the same version and there's still a discrepancy it is quite weird.
>
>
> Quite weird indeed…
> $ python3 lsdiv.py -pc_type lu -ksp_converged_reason
>   Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1
> $ python3 lsdiv.py -usens -pc_type lu -ksp_converged_reason
>   Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1
> $ python3 lsdiv.py -pc_type qr -ksp_converged_reason
>   Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1
> $ python3 lsdiv.py -usens -pc_type qr -ksp_converged_reason
>   Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1
>
> For the moment I can work by adding the nullspace but eventually the
> need for pinning DOFs will resurface, so I'd like to ask where the
> breakdown is coming from. What causes the breakdowns? Is that a generic
> problem occurring when adding (dof_i = val) rows to least-squares
> systems which prevents these preconditioners from being robust? If so,
> what preconditioners could be robust?
> I did a minimal sweep of the available PCs by going over the possible
> 

Re: [petsc-users] PETSc options

[Matthew Knepley]

On Mon, May 6, 2024 at 1:14 PM Mark Adams  wrote:

> But that will hardwire disabling -options_left. right?
>

True. Your suggestion would only filter out those two in particular.

  Thanks,

 Matt


> On Mon, May 6, 2024 at 11:30 AM Matthew Knepley  wrote:
>
>> On Mon, May 6, 2024 at 11: 15 AM Pierre Jolivet 
>> wrote: On 6 May 2024, at 3: 14 PM, Matthew Knepley 
>> wrote: This Message Is From an External Sender This message came from
>> outside your organization. On
>> ZjQcmQRYFpfptBannerStart
>> This Message Is From an External Sender
>> This message came from outside your organization.
>>
>> ZjQcmQRYFpfptBannerEnd
>> On Mon, May 6, 2024 at 11:15 AM Pierre Jolivet  wrote:
>>
>>> On 6 May 2024, at 3:14 PM, Matthew Knepley  wrote:
>>>
>>> This Message Is From an External Sender
>>> This message came from outside your organization.
>>> On Mon, May 6, 2024 at 1:04 AM Adrian Croucher <
>>> a.crouc...@auckland.ac.nz> wrote:
>>>
>>>> This Message Is From an External Sender
>>>> This message came from outside your organization.
>>>>
>>>>
>>>> hi,
>>>>
>>>> My code has some optional command line arguments -v and -h for output of
>>>> version number and usage help. These are processed using Fortran's
>>>> get_command_argument().
>>>>
>>>> Since updating PETSc to version 3.21, I get some extra warnings after
>>>> the output:
>>>>
>>>> acro018@EN438880:~$ waiwera -v
>>>> 1.5.0b1
>>>> WARNING! There are options you set that were not used!
>>>> WARNING! could be spelling mistake, etc!
>>>> There is one unused database option. It is:
>>>> Option left: name:-v (no value) source: command line
>>>>
>>>> That didn't used to happen. What should I do to make them go away?
>>>>
>>>>
>>> Hi Adrian,
>>>
>>> Barry and Mark's suggestions will make this go away. However, it should
>>> not happen
>>> in the first place.
>>>
>>>
>>> It should happen if Adrian was previously not using 3.19.X or below.
>>> See 
>>> https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/6601__;!!G_uCfscf7eWS!aNM60JUO9Vbf5saWU1g8lA5UztQH35oMhYJcuYQfRU3hKT9s_58lqzRoWbtXuf0G1azqCD_9R9cWfJxmWrGE$
>>>  
>>> <https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/6601__;!!G_uCfscf7eWS!bVUciSLxU1Cw3mv04eO2KtlHR8v6ZR7CRE_CpMzmxmCn4x58fUdEJ4L-dknPq_yMbgnc9OkN_O8aOSu_aPK3$>
>>>
>>
>> I forgot. Not my favorite change.
>>
>> Okay. You can shut this off using
>>
>>   PetscCall(PetscOptionsSetValue(NULL, "-options_left", "0"))
>>
>>   Thanks,
>>
>>  Matt
>>
>>
>>> Thanks,
>>> Pierre
>>>
>>> We should try to figure this out.
>>>
>>> This warning is usually activated by the -options_left argument. Could
>>> that be in the
>>> PETSC_OPTIONS env variable, or in ~/.petscrc?
>>>
>>>   Thanks,
>>>
>>>  Matt
>>>
>>>
>>>> Regards, Adrian
>>>>
>>>> --
>>>> Dr Adrian Croucher
>>>> Senior Research Fellow
>>>> Department of Engineering Science
>>>> Waipapa Taumata Rau / University of Auckland, New Zealand
>>>> email: a.crouc...@auckland.ac.nz
>>>> tel: +64 (0)9 923 4611
>>>>
>>>>
>>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aNM60JUO9Vbf5saWU1g8lA5UztQH35oMhYJcuYQfRU3hKT9s_58lqzRoWbtXuf0G1azqCD_9R9cWfFIGaM_Q$
>>>  
>>> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fo7-rCLfudbdd6yHxfeF1pyKANLJjDLd2VvUtM7xc31s9AoJjSSu-vjiyaVVAhNAqV6c4s-9aJPkL3lFP1DF$>
>>>
>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aNM60JUO9Vbf5saWU1g8lA5UztQH35oMhYJcuYQfRU3hKT9s_58lqzRoWbtXuf0G1azqCD_9R9cWfFIGaM_Q$
>>  
>> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bVUciSLxU1Cw3mv04eO2KtlHR8v6ZR7CRE_CpMzmxmCn4x58fUdEJ4L-dknPq_yMbgnc9OkN_O8aOXbl1p93$>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aNM60JUO9Vbf5saWU1g8lA5UztQH35oMhYJcuYQfRU3hKT9s_58lqzRoWbtXuf0G1azqCD_9R9cWfFIGaM_Q$
  
<https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aNM60JUO9Vbf5saWU1g8lA5UztQH35oMhYJcuYQfRU3hKT9s_58lqzRoWbtXuf0G1azqCD_9R9cWfCOgoX4N$
 >

Re: [petsc-users] PETSc options

[Matthew Knepley]

On Mon, May 6, 2024 at 11:15 AM Pierre Jolivet  wrote:

> On 6 May 2024, at 3:14 PM, Matthew Knepley  wrote:
>
> This Message Is From an External Sender
> This message came from outside your organization.
> On Mon, May 6, 2024 at 1:04 AM Adrian Croucher 
> wrote:
>
>> This Message Is From an External Sender
>> This message came from outside your organization.
>>
>>
>> hi,
>>
>> My code has some optional command line arguments -v and -h for output of
>> version number and usage help. These are processed using Fortran's
>> get_command_argument().
>>
>> Since updating PETSc to version 3.21, I get some extra warnings after
>> the output:
>>
>> acro018@EN438880:~$ waiwera -v
>> 1.5.0b1
>> WARNING! There are options you set that were not used!
>> WARNING! could be spelling mistake, etc!
>> There is one unused database option. It is:
>> Option left: name:-v (no value) source: command line
>>
>> That didn't used to happen. What should I do to make them go away?
>>
>>
> Hi Adrian,
>
> Barry and Mark's suggestions will make this go away. However, it should
> not happen
> in the first place.
>
>
> It should happen if Adrian was previously not using 3.19.X or below.
> See 
> https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/6601__;!!G_uCfscf7eWS!bVUciSLxU1Cw3mv04eO2KtlHR8v6ZR7CRE_CpMzmxmCn4x58fUdEJ4L-dknPq_yMbgnc9OkN_O8aOSu_aPK3$
>  
>

I forgot. Not my favorite change.

Okay. You can shut this off using

  PetscCall(PetscOptionsSetValue(NULL, "-options_left", "0"))

  Thanks,

 Matt


> Thanks,
> Pierre
>
> We should try to figure this out.
>
> This warning is usually activated by the -options_left argument. Could
> that be in the
> PETSC_OPTIONS env variable, or in ~/.petscrc?
>
>   Thanks,
>
>  Matt
>
>
>> Regards, Adrian
>>
>> --
>> Dr Adrian Croucher
>> Senior Research Fellow
>> Department of Engineering Science
>> Waipapa Taumata Rau / University of Auckland, New Zealand
>> email: a.crouc...@auckland.ac.nz
>> tel: +64 (0)9 923 4611
>>
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bVUciSLxU1Cw3mv04eO2KtlHR8v6ZR7CRE_CpMzmxmCn4x58fUdEJ4L-dknPq_yMbgnc9OkN_O8aOQZGyQDX$
>  
> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fo7-rCLfudbdd6yHxfeF1pyKANLJjDLd2VvUtM7xc31s9AoJjSSu-vjiyaVVAhNAqV6c4s-9aJPkL3lFP1DF$>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bVUciSLxU1Cw3mv04eO2KtlHR8v6ZR7CRE_CpMzmxmCn4x58fUdEJ4L-dknPq_yMbgnc9OkN_O8aOQZGyQDX$
  
<https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bVUciSLxU1Cw3mv04eO2KtlHR8v6ZR7CRE_CpMzmxmCn4x58fUdEJ4L-dknPq_yMbgnc9OkN_O8aOXbl1p93$
 >

Re: [petsc-users] PETSc options

[Matthew Knepley]

On Mon, May 6, 2024 at 1:04 AM Adrian Croucher 
wrote:

> hi, My code has some optional command line arguments -v and -h for output
> of version number and usage help. These are processed using Fortran's
> get_command_argument(). Since updating PETSc to version 3. 21, I get some
> extra warnings after the
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
>
> hi,
>
> My code has some optional command line arguments -v and -h for output of
> version number and usage help. These are processed using Fortran's
> get_command_argument().
>
> Since updating PETSc to version 3.21, I get some extra warnings after
> the output:
>
> acro018@EN438880:~$ waiwera -v
> 1.5.0b1
> WARNING! There are options you set that were not used!
> WARNING! could be spelling mistake, etc!
> There is one unused database option. It is:
> Option left: name:-v (no value) source: command line
>
> That didn't used to happen. What should I do to make them go away?
>
>
Hi Adrian,

Barry and Mark's suggestions will make this go away. However, it should not
happen
in the first place. We should try to figure this out.

This warning is usually activated by the -options_left argument. Could that
be in the
PETSC_OPTIONS env variable, or in ~/.petscrc?

  Thanks,

 Matt


> Regards, Adrian
>
> --
> Dr Adrian Croucher
> Senior Research Fellow
> Department of Engineering Science
> Waipapa Taumata Rau / University of Auckland, New Zealand
> email: a.crouc...@auckland.ac.nz
> tel: +64 (0)9 923 4611
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fo7-rCLfudbdd6yHxfeF1pyKANLJjDLd2VvUtM7xc31s9AoJjSSu-vjiyaVVAhNAqV6c4s-9aJPkL_AI6ctk$
  

Re: [petsc-users] [EXTERNAL] Re: Is there anything like a "DMPlexSetCones()" ?

[Matthew Knepley]

On Wed, May 1, 2024 at 6:30 PM Ferrand, Jesus A. 
wrote:

> "You know the number of cells if you can read connectivity. Do you mean
> that the format does not tell you
> the number of vertices?"
>
> Sort of...
> So, the format does provide the number of vertices, however, due to the
> way I read the data in parallel I don't know immediately how many local
> vertices there will be.
> Which prevents me from knowing the chart a priori.
> I figured out the number of vertices using PetscHashSetI to determine the
> number of unique entries in the connectivity list.
>

Right, but that means that the connectivity you read in is not in local
numbering. You will have to renumber it, so making a copy is probably
necessary anyway. This is the same sort of processing I do for the parallel
read.

  Thanks,

 Matt


> ------
> *From:* Matthew Knepley 
> *Sent:* Wednesday, May 1, 2024 8:52 PM
> *To:* Ferrand, Jesus A. 
> *Cc:* petsc-users@mcs.anl.gov 
> *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a
> "DMPlexSetCones()" ?
>
> On Wed, May 1, 2024 at 4:23 PM Ferrand, Jesus A. 
> wrote:
>
> Matt:
>
> My bad again, I need to clarify something that I just realized doesn't
> make sense.
> I said: "The nature of the I/O makes it so that I need to read the
> connectivity before I get a semblance of buffer sizes."
> Scratch that, obviously, if I can read the connectivity, I know the buffer
> sizes.
>
> What I should have said is that I cannot know the "chart" that one sets in
> DMPlexSetChart() a priori like in most cases.
> I need to determine the chart from the connectivity lists.
>
>
> You know the number of cells if you can read connectivity. Do you mean
> that the format does not tell you
> the number of vertices?
>
>   Thanks,
>
> Matt
>
>
> --
> *From:* Ferrand, Jesus A. 
> *Sent:* Wednesday, May 1, 2024 8:17 PM
> *To:* Matthew Knepley 
> *Cc:* petsc-users@mcs.anl.gov 
> *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a
> "DMPlexSetCones()" ?
>
> Matt:
>
> "I do not understand the "flag check". What is that?"
>
> My bad, I should have referred to the "dm->setupcalled".
> I believe this PetscBool is checked by the other DM (not just DMPlex) APIs.
> The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to
> say.
> Here's a copy of DMSetUp().
>
> *PetscErrorCode 
> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscErrorCode/__;!!G_uCfscf7eWS!e603Zn0jXSCVvzNnzzRiius-0-C5jw6GGzCOFJ7EW5nKTGdX8aI6qchOlBW23rM9TlwMTiidfBsowpJ3s88P$
>  > DMSetUp 
> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/DM/DMSetUp/__;!!G_uCfscf7eWS!e603Zn0jXSCVvzNnzzRiius-0-C5jw6GGzCOFJ7EW5nKTGdX8aI6qchOlBW23rM9TlwMTiidfBsowpy3zGSd$
>  >(DM 
> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/DM/DM/__;!!G_uCfscf7eWS!e603Zn0jXSCVvzNnzzRiius-0-C5jw6GGzCOFJ7EW5nKTGdX8aI6qchOlBW23rM9TlwMTiidfBsowt_sd2eh$
>  > dm)*
> 817: {
> 818:   PetscFunctionBegin 
> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscFunctionBegin/__;!!G_uCfscf7eWS!e603Zn0jXSCVvzNnzzRiius-0-C5jw6GGzCOFJ7EW5nKTGdX8aI6qchOlBW23rM9TlwMTiidfBsowuMbcxmD$
>  >;
> 820:   if (dm->setupcalled) PetscFunctionReturn 
> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscFunctionReturn/__;!!G_uCfscf7eWS!e603Zn0jXSCVvzNnzzRiius-0-C5jw6GGzCOFJ7EW5nKTGdX8aI6qchOlBW23rM9TlwMTiidfBsowhC986W-$
>  >(PETSC_SUCCESS 
> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscErrorCode/__;!!G_uCfscf7eWS!e603Zn0jXSCVvzNnzzRiius-0-C5jw6GGzCOFJ7EW5nKTGdX8aI6qchOlBW23rM9TlwMTiidfBsowpJ3s88P$
>  >);
> 821:   PetscTryTypeMethod 
> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscTryTypeMethod/__;!!G_uCfscf7eWS!e603Zn0jXSCVvzNnzzRiius-0-C5jw6GGzCOFJ7EW5nKTGdX8aI6qchOlBW23rM9TlwMTiidfBsowrbaxERF$
>  >(dm, setup);
> 822:   dm->setupcalled = PETSC_TRUE 
> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PETSC_TRUE/__;!!G_uCfscf7eWS!e603Zn0jXSCVvzNnzzRiius-0-C5jw6GGzCOFJ7EW5nKTGdX8aI6qchOlBW23rM9TlwMTiidfBsowoILDY_P$
>  >;
> 823:   PetscFunctionReturn 
> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscFunctionReturn/__;!!G_uCfscf7eWS!e603Zn0jXSCVvzNnzzRiius-0-C5jw6GGzCOFJ7EW5nKTGdX8aI6qchOlBW23rM9TlwMTiidfBsowhC986W-$
>  >(PETSC_SUCCESS 
> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscErrorCode/__;!!G_uCfscf7eWS!e603Zn0jXSCVvzNnzzRiius-0-C5jw6GGzCOFJ7EW5nKTGdX8aI6qchOlBW23rM

Re: [petsc-users] Question about petsc4py createWithArray function

[Matthew Knepley]

On Thu, May 2, 2024 at 12:53 PM Samar Khatiwala <
samar.khatiw...@earth.ox.ac.uk> wrote:

> Hello, I have a couple of questions about createWithArray in petsc4py: 1)
> What is the correct usage for creating a standard MPI Vec with it?
> Something like this seems to work but is it right?: On each rank do: a =
> np. zeros(localSize) v = PETSc. Vec(). createWithArray(a,
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
>
> Hello,
>
> I have a couple of questions about createWithArray in petsc4py:
>
> 1) What is the correct usage for creating a standard MPI Vec with it? 
> Something like this seems to work but is it right?:
>
> On each rank do:
> a = np.zeros(localSize)
> v = PETSc.Vec().createWithArray(a, comm=PETSc.COMM_WORLD)
>
> Is that all it takes?
>
>
That looks right to me.

> 2) Who ‘owns’ the underlying memory for a Vec created with the 
> createWithArray method, i.e., who is responsible for managing it and doing 
> garbage collection? In my problem, the numpy array is created in a Cython 
> module where memory is allocated, and a pointer to it is associated with a 
> numpy ndarray via PyArray_SimpleNewFromData and PyArray_SetBaseObject. I have 
> a deallocator method of my own that is called when the numpy array is 
> deleted/goes out of scope/whenever python does garbage collection. All of 
> that works fine. But if I use this array to create a Vec with createWithArray 
> what happens when the Vec is, e.g., destroyed? Will my deallocator be called?
>
> No. The PETSc struct will be deallocated, but the storage will not be
touched.

  Thanks,

 Matt

> Or does petsc4py know that it doesn’t own the memory and won’t attempt to 
> free it? I can’t quite figure out from the petsc4py code what is going on. 
> And help would be appreciated.
>
> Thanks very much.
>
> Samar
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!anRKQn_8v127_CSkzk22FfTFRodKT0G2BLwgi_kPAUQt_eqCiySYWSg3ctwewCceXLeJY4FU01ONUG3xfSeD$
  

Re: [petsc-users] [EXTERNAL] Re: Is there anything like a "DMPlexSetCones()" ?

[Matthew Knepley]

On Wed, May 1, 2024 at 4:23 PM Ferrand, Jesus A. 
wrote:

> Matt:
>
> My bad again, I need to clarify something that I just realized doesn't
> make sense.
> I said: "The nature of the I/O makes it so that I need to read the
> connectivity before I get a semblance of buffer sizes."
> Scratch that, obviously, if I can read the connectivity, I know the buffer
> sizes.
>
> What I should have said is that I cannot know the "chart" that one sets in
> DMPlexSetChart() a priori like in most cases.
> I need to determine the chart from the connectivity lists.
>

You know the number of cells if you can read connectivity. Do you mean that
the format does not tell you
the number of vertices?

  Thanks,

Matt


> --
> *From:* Ferrand, Jesus A. 
> *Sent:* Wednesday, May 1, 2024 8:17 PM
> *To:* Matthew Knepley 
> *Cc:* petsc-users@mcs.anl.gov 
> *Subject:* Re: [EXTERNAL] Re: [petsc-users] Is there anything like a
> "DMPlexSetCones()" ?
>
> Matt:
>
> "I do not understand the "flag check". What is that?"
>
> My bad, I should have referred to the "dm->setupcalled".
> I believe this PetscBool is checked by the other DM (not just DMPlex) APIs.
> The subsequent checks for dm->setupcalled == PETSC_TRUE is what I meant to
> say.
> Here's a copy of DMSetUp().
>
> *PetscErrorCode 
> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscErrorCode/__;!!G_uCfscf7eWS!dYZsj6bNyjrW3n2ZUHSK0N1eCUS5ylIPHxjjS0SQT-DSlMCOeipzRVmuuasQCL9GHi7tXPQbRftLbEGErkJ-$
>  > DMSetUp 
> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/DM/DMSetUp/__;!!G_uCfscf7eWS!dYZsj6bNyjrW3n2ZUHSK0N1eCUS5ylIPHxjjS0SQT-DSlMCOeipzRVmuuasQCL9GHi7tXPQbRftLbHlKVRsv$
>  >(DM 
> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/DM/DM/__;!!G_uCfscf7eWS!dYZsj6bNyjrW3n2ZUHSK0N1eCUS5ylIPHxjjS0SQT-DSlMCOeipzRVmuuasQCL9GHi7tXPQbRftLbLTlKr33$
>  > dm)*
> 817: {
> 818:   PetscFunctionBegin 
> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscFunctionBegin/__;!!G_uCfscf7eWS!dYZsj6bNyjrW3n2ZUHSK0N1eCUS5ylIPHxjjS0SQT-DSlMCOeipzRVmuuasQCL9GHi7tXPQbRftLbO3Lr_yl$
>  >;
> 820:   if (dm->setupcalled) PetscFunctionReturn 
> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscFunctionReturn/__;!!G_uCfscf7eWS!dYZsj6bNyjrW3n2ZUHSK0N1eCUS5ylIPHxjjS0SQT-DSlMCOeipzRVmuuasQCL9GHi7tXPQbRftLbIQ_4i0Z$
>  >(PETSC_SUCCESS 
> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscErrorCode/__;!!G_uCfscf7eWS!dYZsj6bNyjrW3n2ZUHSK0N1eCUS5ylIPHxjjS0SQT-DSlMCOeipzRVmuuasQCL9GHi7tXPQbRftLbEGErkJ-$
>  >);
> 821:   PetscTryTypeMethod 
> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscTryTypeMethod/__;!!G_uCfscf7eWS!dYZsj6bNyjrW3n2ZUHSK0N1eCUS5ylIPHxjjS0SQT-DSlMCOeipzRVmuuasQCL9GHi7tXPQbRftLbAvZ6cYM$
>  >(dm, setup);
> 822:   dm->setupcalled = PETSC_TRUE 
> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PETSC_TRUE/__;!!G_uCfscf7eWS!dYZsj6bNyjrW3n2ZUHSK0N1eCUS5ylIPHxjjS0SQT-DSlMCOeipzRVmuuasQCL9GHi7tXPQbRftLbDXZdzLl$
>  >;
> 823:   PetscFunctionReturn 
> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscFunctionReturn/__;!!G_uCfscf7eWS!dYZsj6bNyjrW3n2ZUHSK0N1eCUS5ylIPHxjjS0SQT-DSlMCOeipzRVmuuasQCL9GHi7tXPQbRftLbIQ_4i0Z$
>  >(PETSC_SUCCESS 
> <https://urldefense.us/v3/__https://petsc.org/release/manualpages/Sys/PetscErrorCode/__;!!G_uCfscf7eWS!dYZsj6bNyjrW3n2ZUHSK0N1eCUS5ylIPHxjjS0SQT-DSlMCOeipzRVmuuasQCL9GHi7tXPQbRftLbEGErkJ-$
>  >);
> 824: }
>
> "We could make a DMPlexSetCones(), but as you point out, the workflow for
> DMSetUp() would have to change. Where does the memory come from for your
> connectivity?"
> It's a memory that I myself allocate based on a file's contents.
> The nature of the I/O makes it so that I need to read the connectivity
> before I get a semblance of buffer sizes.
> Otherwise, I would stick to the tried and tested way.
>
> Also, when replying to the PETSc developers, users must reply to
> petsc-users@mcs.anl.gov and not just to the individual email accounts of
> the developers, right?
>
>
>
>
> --
> *From:* Matthew Knepley 
> *Sent:* Wednesday, May 1, 2024 8:07 PM
> *To:* Ferrand, Jesus A. 
> *Cc:* petsc-users@mcs.anl.gov 
> *Subject:* [EXTERNAL] Re: [petsc-users] Is there anything like a
> "DMPlexSetCones()" ?
>
> *CAUTION:* This email originated outside of Embry-Riddle Aeronautical
> University. Do not click links or open attachments unless you recognize the
> sender and know the content is safe.
>
> On Wed, May 1, 2024 at 3:3

Re: [petsc-users] Is there anything like a "DMPlexSetCones()" ?

[Matthew Knepley]

On Wed, May 1, 2024 at 3:34 PM Ferrand, Jesus A. 
wrote:

> Dear PETSc team: For a project that I'm working on, I need to manually
> build a DMPlex. From studying the source code of the various APIs in which
> the plex is built from some supported file format, I get that the workflow
> is this: DMPlexSetChart()
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
> Dear PETSc team:
>
> For a project that I'm working on, I need to manually build a DMPlex.
> From studying the source code of the various APIs in which the plex is
> built from some supported file format, I get that the workflow is this:
>
>
>1. DMPlexSetChart() <-- Input nCells + nVerts
>2. DMPlexSetConeSize() <-- Input ConeSize for each point in [0,nCells)
>3. DMSetUp() – Allocates memory internally.
>4. DMPlexGetCones() --> Gives you the memory onto which to write the
>cell connectivity.
>5. *Write connectivity*
>6. DMPlexReorderCell() <-- For each point in [0,nCells)
>
>
> I'm in a situation where the memory given by step (4) is available
> a-priori.
> I was hoping to skip steps 2, 3 , and 4 with something like a
> "DMPlexSetCones()", but such an API does not exist.
> My current workaround is to implement steps 2 through 4 as always and have
> double the memory allocated in the interim (my instance + DM's internal
> instance).
> I was thinking of looking for the name of the struct member and = it to my
> memory, but I can't overcome the flag check in DMSetUp() during later calls
> to DMPlexGetCones() or DMPlexGetTransitiveClosure().
>

I do not understand the "flag check". What is that?

We could make a DMPlexSetCones(), but as you point out, the workflow for
DMSetUp() would have to change. Where does the memory come from for your
connectivity?

  Thanks,

Matt


>
> Sincerely:
>
> *J.A. Ferrand*
>
> Embry-Riddle Aeronautical University - Daytona Beach - FL
> Ph.D. Candidate, Aerospace Engineering
>
> M.Sc. Aerospace Engineering
>
> B.Sc. Aerospace Engineering
>
> B.Sc. Computational Mathematics
>
>
> *Phone:* (386)-843-1829
>
> *Email(s):* ferra...@my.erau.edu
>
> jesus.ferr...@gmail.com
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ZR1_fNBzyO9hgjbXCkFEbX3CMsLXVmoeah8FCeMOb9AMhbszC-gLvEU6mbq67TLMx9kDRY2b--nt1pUHr_Np$
  

Re: [petsc-users] PETSc-GPU

[Matthew Knepley]

On Fri, Apr 26, 2024 at 7:23 AM Karthikeyan Chockalingam - STFC UKRI via
petsc-users  wrote:

> Hello, When PETSc is installed with GPU support, will it run only on GPUs
> or can it be run on CPUs (without GPUs)? Currently, PETSc crashes when run
> on CPUs. Thank you. Best regards, Karthik. -- Karthik Chockalingam, Ph. D.
> Senior Research Software
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
>
> Hello,
>
>
>
> When PETSc is installed with GPU support, will it run only on GPUs or can
> it be run on CPUs (without GPUs)? Currently, PETSc crashes when run on CPUs.
>

It should run on both. Can you send the crash? I think we did fix a problem
with it eagerly initializing GPUs when they were absent.

  Thanks,

 Matt


> Thank you.
>
>
>
> Best regards,
>
> Karthik.
>
>
>
> --
>
> *Karthik Chockalingam, Ph.D.*
>
> Senior Research Software Engineer
>
> High Performance Systems Engineering Group
>
> Hartree Centre | Science and Technology Facilities Council
>
> karthikeyan.chockalin...@stfc.ac.uk
>
>
>
>  [image: signature_3970890138]
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!a0VV8buEHYPmWgGr_wyPE35z-vFWz2YwIgw3_bJI-_ydHr3nvKWF8LjXSvvoCPZ0U1zxOqXdxcrOKx8-ahM8$
  

Re: [petsc-users] [petsc-maint] Inquiry about Multithreading Capabilities in PETSc's KSPSolver

[Matthew Knepley]

On Tue, Apr 23, 2024 at 4:00 PM Yongzhong Li 
wrote:

> Thanks Barry! Does this mean that the sparse matrix-vector products, which
> actually constitute the majority of the computations in my GMRES routine in
> PETSc, don’t utilize multithreading? Only basic vector operations such as
> VecAXPY and VecDot
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
>
> Thanks Barry! Does this mean that the sparse matrix-vector products, which
> actually constitute the majority of the computations in my GMRES routine in
> PETSc, don’t utilize multithreading? Only basic vector operations such as
> VecAXPY and VecDot or dense matrix operations in PETSc will benefit from
> multithreading, is it correct?
>

I am not sure what your point is above.

SpMV performance is mainly controlled by memory bandwidth (latency plays
very little role). If you think
the MPI processes are not using the full bandwidth, use more processes. If
they are, and you think using
threads will speed anything up, you are incorrect. In fact, the only
difference between a thread and a
process is the default memory sharing flag, MPI will perform at least as
well (and usually better), than
adding threads to SpMV.

There are dozens of publications showing this.

  Thanks,

 Matt


> Best,
>
> Yongzhong
>
>
>
> *From: *Barry Smith 
> *Date: *Tuesday, April 23, 2024 at 3:35 PM
> *To: *Yongzhong Li 
> *Cc: *petsc-users@mcs.anl.gov ,
> petsc-ma...@mcs.anl.gov , Piero Triverio <
> piero.trive...@utoronto.ca>
> *Subject: *Re: [petsc-maint] Inquiry about Multithreading Capabilities in
> PETSc's KSPSolver
>
> 你通常不会收到来自 bsm...@petsc.dev 的电子邮件。了解这一点为什么很重要
> 
>
>
>
>Intel MKL or OpenBLAS are the best bet, but for vector operations they
> will not be significant since the vector operations do not dominate the
> computations.
>
>
>
> On Apr 23, 2024, at 3:23 PM, Yongzhong Li 
> wrote:
>
>
>
> Hi Barry,
>
> Thank you for the information provided!
>
> Do you think different BLAS implementation will affect the multithreading
> performance of some vectors operations in GMERS in PETSc?
>
>
>
> I am now using OpenBLAS but didn’t see much improvement when theb
> multithreading are enabled, do you think other implementation such as
> netlib and intel-mkl will help?
>
> Best,
>
> Yongzhong
>
>
>
> *From: *Barry Smith 
> *Date: *Monday, April 22, 2024 at 4:20 PM
> *To: *Yongzhong Li 
> *Cc: *petsc-users@mcs.anl.gov ,
> petsc-ma...@mcs.anl.gov , Piero Triverio <
> piero.trive...@utoronto.ca>
> *Subject: *Re: [petsc-maint] Inquiry about Multithreading Capabilities in
> PETSc's KSPSolver
>
> 你通常不会收到来自 bsm...@petsc.dev 的电子邮件。了解这一点为什么很重要
> 
>
>
>
>PETSc provided solvers do not directly use threads.
>
>
>
>The BLAS used by LAPACK and PETSc may use threads depending on what
> BLAS is being used and how it was configured.
>
>
>
>Some of the vector operations in GMRES in PETSc use BLAS that can use
> threads, including axpy, dot, etc. For sufficiently large problems, the use
> of threaded BLAS can help with these routines, but not significantly for
> the solver.
>
>
>
>Dense matrix-vector products MatMult() and dense matrix direct solvers
> PCLU use BLAS and thus can benefit from threading. The benefit can be
> significant for large enough problems with good hardware, especially with
> PCLU.
>
>
>
>If you run with -blas_view  PETSc tries to indicate information about
> the threading of BLAS. You can also use -blas_num_threads  to set the
> number of threads, equivalent to setting the environmental variable.  For
> dense solvers you can vary the number of threads and run with -log_view to
> see what it helps to improve and what it does not effect.
>
>
>
>
>
>
>
> On Apr 22, 2024, at 4:06 PM, Yongzhong Li 
> wrote:
>
>
>
> This Message Is From an External Sender
>
> This message came from outside your organization.
>
> Hello all,
>
>
>
> I am writing to ask if PETSc’s KSPSolver makes use of
> OpenMP/multithreading, specifically when performing iterative solutions
> with the GMRES algorithm.
>
>
>
> The questions appeared when I was running a large numerical program based
> on boundary element method. I used the PETSc's GMRES algorithm in KSPSolve
> to solve a shell matrix system iteratively. I observed that threads were
> being utilized, controlled by the *OPENBLAS_NUM_THREADS* environment
> variable. However, I noticed no significant performance difference between
> running the solver with multiple threads versus a single thread.
>
> Could you please *confirm if GMRES in KSPSolve 

Re: [petsc-users] Periodic boundary conditions using swarm

[Matthew Knepley]

On Sun, Apr 21, 2024 at 9:38 AM MIGUEL MOLINOS PEREZ  wrote:

> Dear Matt,
>
> Thank you for your answer. In addition to your suggestion I solved a bug
> in the test (I was not updating the local size integer during the time
> loop). Indeed if I turn off periodicity it works. Furthermore, if I use
> instead DM_BOUNDARY_TWIST instead, it works too.
>
> However, if I turn on DM_BOUNDARY_PERIODIC, I got an error in the search
> algorithm I implemented for the domain decomposition inspired by (
> https://urldefense.us/v3/__https://petsc.org/main/src/dm/tutorials/swarm_ex3.c.html__;!!G_uCfscf7eWS!flWhzJ-Vd_S7zFFxywsQXO7UelXYvJprWVsqngT5k0SlO7wMjGMHQ_D_3lQ-2_FNdN9pw-ztQ4gVIcl8_Lle$
>  ). The algorithm
> is not capable of finding some of the particles at the initial stage of the
> simulation (without transport).
>
> Looks like the error is on my end, however it is puzzling why it works for
> DM_BOUNDARY_TWIST but not for DM_BOUNDARY_PERIODIC.
>

I usually solve these things by making a simple example. We could make
another test in Swarm test ex1 that uses periodicity. If you send a draft
over that fails, I can help you debug it. It would make a fantastic
contribution to PETSc.

   Thanks,

 Matt


> Thanks,
> Miguel
>
> On 21 Apr 2024, at 14:53, Matthew Knepley  wrote:
>
> On Thu, Apr 18, 2024 at 8:23 AM MIGUEL MOLINOS PEREZ 
> wrote:
>
>>  Dear all, I am working on the implementation of periodic bcc using a
>> discretisation (PIC-style). I am working with a test case which consists on
>> solving the advection of a set of particles inside of a box (DMDA mesh)
>> with periodic bcc on
>> ZjQcmQRYFpfptBannerStart
>> This Message Is From an External Sender
>> This message came from outside your organization.
>>
>> ZjQcmQRYFpfptBannerEnd
>> 
>> Dear all,
>>
>> I am working on the implementation of periodic bcc using a discretisation
>> (PIC-style). I am working with a test case which consists on solving the
>> advection of a set of particles inside of a box (DMDA mesh) with periodic
>> bcc on the x axis.
>>
>> My implementation updates the position of each particle with a velocity
>> field, afterwards I check if the particle is inside, or not, the supercell
>> (periodic box). If not, I correct the position using bcc conditions. Once
>> this step is done, I call Dmswarmmigrate.
>>
>> It works in serial, but crashes in parallel with MPI (see attached nohup
>> file). I have checked some of the Dmswarmmigrate examples, and they
>> looks similar to my implementation. However they do not use periodic bcc.
>>
>> I am missing any step in addition to Dmswarmmigrate?
>>
>
> It does not sound like it. We do have parallel examples of periodic
> migration, such as Swarm ex9.
>
> What happens if you turn off periodicity and just let particles fall out
> of the box? Does it still crash?
>
>   Thanks,
>
> Matt
>
>
>> Best regards
>> Miguel
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!flWhzJ-Vd_S7zFFxywsQXO7UelXYvJprWVsqngT5k0SlO7wMjGMHQ_D_3lQ-2_FNdN9pw-ztQ4gVIfmWX52U$
>  
> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!flWhzJ-Vd_S7zFFxywsQXO7UelXYvJprWVsqngT5k0SlO7wMjGMHQ_D_3lQ-2_FNdN9pw-ztQ4gVIbVCp3cZ$
>  >
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!flWhzJ-Vd_S7zFFxywsQXO7UelXYvJprWVsqngT5k0SlO7wMjGMHQ_D_3lQ-2_FNdN9pw-ztQ4gVIfmWX52U$
  
<https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!flWhzJ-Vd_S7zFFxywsQXO7UelXYvJprWVsqngT5k0SlO7wMjGMHQ_D_3lQ-2_FNdN9pw-ztQ4gVIbVCp3cZ$
 >

Re: [petsc-users] Periodic boundary conditions using swarm

[Matthew Knepley]

On Thu, Apr 18, 2024 at 8:23 AM MIGUEL MOLINOS PEREZ  wrote:

>  Dear all, I am working on the implementation of periodic bcc using a
> discretisation (PIC-style). I am working with a test case which consists on
> solving the advection of a set of particles inside of a box (DMDA mesh)
> with periodic bcc on
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
> 
> Dear all,
>
> I am working on the implementation of periodic bcc using a discretisation
> (PIC-style). I am working with a test case which consists on solving the
> advection of a set of particles inside of a box (DMDA mesh) with periodic
> bcc on the x axis.
>
> My implementation updates the position of each particle with a velocity
> field, afterwards I check if the particle is inside, or not, the supercell
> (periodic box). If not, I correct the position using bcc conditions. Once
> this step is done, I call Dmswarmmigrate.
>
> It works in serial, but crashes in parallel with MPI (see attached nohup
> file). I have checked some of the Dmswarmmigrate examples, and they looks
> similar to my implementation. However they do not use periodic bcc.
>
> I am missing any step in addition to Dmswarmmigrate?
>

It does not sound like it. We do have parallel examples of periodic
migration, such as Swarm ex9.

What happens if you turn off periodicity and just let particles fall out of
the box? Does it still crash?

  Thanks,

Matt


> Best regards
> Miguel
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YAPx7Q5VFXWaibGKwqJDGzxvP0snoJmOJZWb4-jtxTRnZNS4wl35_Rp74soLPXHoOsFuG3UoInaqqVUz6Az-$
  

Re: [petsc-users] About recent changes in GAMG

[Matthew Knepley]

On Fri, Apr 19, 2024 at 3:52 PM Ashish Patel  wrote:

> Hi Jed, VmRss is on a higher side and seems to match what
> PetscMallocGetMaximumUsage is reporting. HugetlbPages was 0 for me. Mark,
> running without the near nullspace also gives similar results. I have
> attached the malloc_view and gamg info
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
> Hi Jed,
> VmRss is on a higher side and seems to match what
> PetscMallocGetMaximumUsage is reporting. HugetlbPages was 0 for me.
>
> Mark, running without the near nullspace also gives similar results. I
> have attached the malloc_view and gamg info for serial and 2 core runs.
> Some of the standout functions on rank 0 for parallel run seems to be
> 5.3 GB MatSeqAIJSetPreallocation_SeqAIJ
> 7.7 GB MatStashSortCompress_Private
> 10.1 GB PetscMatStashSpaceGet
>

This is strange. We would expect the MatStash to be much smaller than the
allocation, but it is larger.
That suggests that you are sending a large number of off-process values. Is
this by design?

  Thanks,

 Matt


> 7.7 GB  PetscSegBufferAlloc_Private
>
> malloc_view also says the following
> [0] Maximum memory PetscMalloc()ed 32387548912 maximum size of entire
> process 8270635008
> which fits the PetscMallocGetMaximumUsage > PetscMemoryGetMaximumUsage
>  output.
>
> Let me know if you need some other info.
>
> Thanks,
> Ashish
>
> --
> *From:* Jed Brown 
> *Sent:* Thursday, April 18, 2024 2:16 PM
> *To:* Mark Adams ; Ashish Patel ;
> PETSc users list 
> *Cc:* Scott McClennan 
> *Subject:* Re: [petsc-users] About recent changes in GAMG
>
> [External Sender]
>
> Mark Adams  writes:
>
> >>> Yea, my interpretation of these methods is also that "
> > PetscMemoryGetMaximumUsage" should be >= "PetscMallocGetMaximumUsage".
> >>> But you are seeing the opposite.
> >
> >
> > We are using PETSc main and have found a case where memory consumption
> > explodes in parallel.
> > Also, we see a non-negligible difference between
> PetscMemoryGetMaximumUsage()
> > and PetscMallocGetMaximumUsage().
> > Running in serial through /usr/bin/time, the max. resident set size
> matches
> > the PetscMallocGetMaximumUsage() result.
> > I would have expected it to match PetscMemoryGetMaximumUsage() instead.
>
> PetscMemoryGetMaximumUsage uses procfs (if PETSC_USE_PROCFS_FOR_SIZE,
> which should be typical on Linux anyway) in PetscHeaderDestroy to update a
> static variable. If you haven't destroyed an object yet, its value will be
> nonsense.
>
> If your program is using huge pages, it might also be inaccurate (I don't
> know). You can look at /proc//statm to see what PETSc is reading
> (second field, which is number of pages in RSS). You can also look at the
> VmRss field in /proc//status, which reads in kB. See also the
> HugetlbPages field in /proc//status.
>
> https://urldefense.us/v3/__https://www.kernel.org/doc/Documentation/filesystems/proc.txt__;!!G_uCfscf7eWS!ed35Hheva03XvvROCiYSMw0awizDuqiHG-IvZWEe-6j6XOY7z0eYVj_VFWQsUtaWNm-JLkMBQQT5wHbCEp3F$
>  
> 
>
> If your app is swapping, these will be inaccurate because swapped memory
> is not resident. We don't use the first field (VmSize) because there are
> reasons why programs sometimes map much more memory than they'll actually
> use, making such numbers irrelevant for most purposes.
>
> >
> >
> > PetscMemoryGetMaximumUsage
> > PetscMallocGetMaximumUsage
> >  Time
> > Serial + Option 1
> >  4.8 GB
> >  7.4 GB
> > 112 sec
> > 2 core + Option1
> > 15.2 GB
> > 45.5 GB
> > 150 sec
> > Serial + Option 2
> > 3.1 GB
> > 3.8 GB
> >  167 sec
> > 2 core + Option2
> > 13.1 GB
> > 17.4 GB
> > 89 sec
> > Serial + Option 3
> > 4.7GB
> > 5.2GB
> > 693 sec
> > 2 core + Option 3
> > 23.2 GB
> > 26.4 GB
> > 411 sec
> >
> >
> > On Thu, Apr 18, 2024 at 4:13 PM Mark Adams  wrote:
> >
> >> The next thing you might try is not using the null space argument.
> >> Hypre does not use it, but GAMG does.
> >> You could also run with -malloc_view to see some info on mallocs. It is
> >> probably in the Mat objects.
> >> You can also run with "-info" and grep on GAMG in the output and send
> that.
> >>
> >> Mark
> >>
> >> On Thu, Apr 18, 2024 at 12:03 PM Ashish Patel 
> >> wrote:
> >>
> >>> Hi Mark,
> >>>
> >>> Thanks for your response and suggestion. With hypre both memory and
> time
> >>> looks good, here is the data for that
> >>>
> >>> PetscMemoryGetMaximumUsage
> >>> PetscMallocGetMaximumUsage
> >>>  Time
> >>> Serial + Option 4
> >>> 5.55 GB
> >>>  5.17 GB
> >>> 15.7 sec
> >>> 2 core + Option 4
> >>> 5.85 GB
> >>> 4.69 GB
> >>> 21.9 sec
> >>>
> >>> Option 4
> >>> mpirun -n _ ./ex1 -A_name matrix.dat -b_name vector.dat -n_name
> >>> _null_space.dat -num_near_nullspace 6 

Re: [petsc-users] Using PETSc GPU backend

[Matthew Knepley]

> >
> >
> >> On Aug 9, 2023, at 5:59 PM, Ng, Cho-Kuen  wrote:
> >>
> >> Barry and Matt,
> >>
> >> Thanks for your help. Now I can use petsc GPU backend on Perlmutter: 1
> node, 4 MPI tasks and 4 GPUs. However, I ran into problems with multiple
> nodes: 2 nodes, 8 MPI tasks and 8 GPUs. The run hung on KSPSolve. How can I
> fix this?
> >>
> >> Best,
> >> Cho
> >>
> >>  From: Barry Smith 
> >> Sent: Monday, July 17, 2023 6:58 AM
> >> To: Ng, Cho-Kuen 
> >> Cc: petsc-users@mcs.anl.gov 
> >> Subject: Re: [petsc-users] Using PETSc GPU backend
> >>
> >>  The examples that use DM, in particular DMDA all trivially support
> using the GPU with -dm_mat_type aijcusparse -dm_vec_type cuda
> >>
> >>
> >>
> >>> On Jul 17, 2023, at 1:45 AM, Ng, Cho-Kuen 
> wrote:
> >>>
> >>> Barry,
> >>>
> >>> Thank you so much for the clarification.
> >>>
> >>> I see that ex104.c and ex300.c use  MatXAIJSetPreallocation(). Are
> there other tutorials available?
> >>>
> >>> Cho
> >>>  From: Barry Smith 
> >>> Sent: Saturday, July 15, 2023 8:36 AM
> >>> To: Ng, Cho-Kuen 
> >>> Cc: petsc-users@mcs.anl.gov 
> >>> Subject: Re: [petsc-users] Using PETSc GPU backend
> >>>
> >>>  Cho,
> >>>
> >>> We currently have a crappy API for turning on GPU support, and our
> documentation is misleading in places.
> >>>
> >>> People constantly say "to use GPU's with PETSc you only need to
> use -mat_type aijcusparse (for example)" This is incorrect.
> >>>
> >>>  This does not work with code that uses the convenience Mat
> constructors such as MatCreateAIJ(), MatCreateAIJWithArrays etc. It only
> works if you use the constructor approach of MatCreate(), MatSetSizes(),
> MatSetFromOptions(), MatXXXSetPreallocation(). ...  Similarly you need to
> use VecCreate(), VecSetSizes(), VecSetFromOptions() and -vec_type cuda
> >>>
> >>>If you use DM to create the matrices and vectors then you can use
> -dm_mat_type aijcusparse -dm_vec_type cuda
> >>>
> >>>Sorry for the confusion.
> >>>
> >>>Barry
> >>>
> >>>
> >>>
> >>>
> >>>> On Jul 15, 2023, at 8:03 AM, Matthew Knepley 
> wrote:
> >>>>
> >>>> On Sat, Jul 15, 2023 at 1:44 AM Ng, Cho-Kuen 
> wrote:
> >>>> Matt,
> >>>>
> >>>> After inserting 2 lines in the code:
> >>>>
> >>>>   ierr =
> MatCreate(PETSC_COMM_WORLD,);CHKERRQ(ierr);
> >>>>   ierr = MatSetFromOptions(A);CHKERRQ(ierr);
> >>>>   ierr = MatCreateAIJ(PETSC_COMM_WORLD,mlocal,mlocal,m,n,
> >>>>
> d_nz,PETSC_NULL,o_nz,PETSC_NULL,);;CHKERRQ(ierr);
> >>>>
> >>>> "There are no unused options." However, there is no improvement on
> the GPU performance.
> >>>>
> >>>> 1. MatCreateAIJ() sets the type, and in fact it overwrites the Mat
> you created in steps 1 and 2. This is detailed in the manual.
> >>>>
> >>>> 2. You should replace MatCreateAIJ(), with MatSetSizes() before
> MatSetFromOptions().
> >>>>
> >>>>   THanks,
> >>>>
> >>>> Matt
> >>>>  Thanks,
> >>>> Cho
> >>>>
> >>>>  From: Matthew Knepley 
> >>>> Sent: Friday, July 14, 2023 5:57 PM
> >>>> To: Ng, Cho-Kuen 
> >>>> Cc: Barry Smith ; Mark Adams ;
> petsc-users@mcs.anl.gov 
> >>>> Subject: Re: [petsc-users] Using PETSc GPU backend
> >>>>  On Fri, Jul 14, 2023 at 7:57 PM Ng, Cho-Kuen 
> wrote:
> >>>> I managed to pass the following options to PETSc using a GPU node on
> Perlmutter.
> >>>>
> >>>> -mat_type aijcusparse -vec_type cuda -log_view -options_left
> >>>>
> >>>> Below is a summary of the test using 4 MPI tasks and 1 GPU per task.
> >>>>
> >>>> o #PETSc Option Table entries:
> >>>> -log_view
> >>>> -mat_type aijcusparse
> >>>>-options_left
> >>>>-vec_type cuda
> >>>>#End of PETSc Option Table entries
> >>>>WARNING! There are options you set that were not used!
> >>>>

Re: [petsc-users] Exposing further detail in -log_view for Hypre withe PETSc

[Matthew Knepley]

On Wed, Apr 10, 2024 at 7:49 AM Khurana, Parv 
wrote:

> Hello PETSc users community, Thank you in advance for your help as always.
> I am using BoomerAMG from Hypre via PETSc as a part of preconditioner in my
> software (Nektar++). I am trying to understand the time profiling
> information that is printed
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
>
> Hello PETSc users community,
>
>
>
> Thank you in advance for your help as always.
>
>
>
> I am using BoomerAMG from Hypre via PETSc as a part of preconditioner in
> my software (Nektar++). I am trying to understand the time profiling
> information that is printed using the -log_view option.
>

I believe that our Hypre interface plugs into the PETSc MG callbacks. Thus
you should be able to use

  -pc_mg_log

to split the various levels into different logging stages. This still will
not directly log Hypre time, but it should be better.

  Thanks,

 MAtt


> I want to understand how much time is spent in the smoothening step vs the
> time to solve on the coarsest grid I reach. The output that I get from
> -log_view (pasted below) gives me information of the KSPSolve and MatMult,
> but I think I need more granular time information to see a further
> breakdown of time spent within my routines. I would like to hear if anyone
> has any recommendations on obtaining this information?
>
>
>
> Best
>
> Parv Khurana
>
>
>
>
>
> PETSc database options used for solve:
>
>
>
> -ksp_monitor # (source: file)
>
> -ksp_type preonly # (source: file)
>
> -log_view # (source: file)
>
> -pc_hypre_boomeramg_coarsen_type hmis # (source: file)
>
> -pc_hypre_boomeramg_grid_sweeps_all 2 # (source: file)
>
> -pc_hypre_boomeramg_interp_type ext+i # (source: file)
>
> -pc_hypre_boomeramg_max_iter 1 # (source: file)
>
> -pc_hypre_boomeramg_P_max 2 # (source: file)
>
> -pc_hypre_boomeramg_print_debug 1 # (source: file)
>
> -pc_hypre_boomeramg_print_statistics 1 # (source: file)
>
> -pc_hypre_boomeramg_relax_type_all sor/jacobi # (source: file)
>
> -pc_hypre_boomeramg_strong_threshold 0.7 # (source: file)
>
> -pc_hypre_boomeramg_truncfactor 0.3 # (source: file)
>
> -pc_hypre_type boomeramg # (source: file)
>
> -pc_type hypre # (source: file)
>
>
>
>
>
> PETSc log_view output:
>
>
>
>
> 
>
> EventCount  Time (sec) Flop
>--- Global ---  --- Stage   Total
>
>Max Ratio  Max Ratio   Max  Ratio  Mess   AvgLen
> Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
>
>
> 
>
>
>
> --- Event Stage 0: Main Stage
>
>
>
> BuildTwoSided  1 1.0 9.6900e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0 0
>
> MatMult   14 1.0 1.6315e-01 1.0 1.65e+08 1.0 0.0e+00 0.0e+00
> 0.0e+00  0 86  0  0  0   0 86  0  0  0  1011
>
> MatConvert 1 1.0 4.3092e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0 0
>
> MatAssemblyBegin   3 1.0 3.1680e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0 0
>
> MatAssemblyEnd 3 1.0 9.4178e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0 0
>
> MatGetRowIJ2 1.0 1.1630e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0 0
>
> MatSetPreallCOO1 1.0 3.2132e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0 0
>
> MatSetValuesCOO1 1.0 2.9956e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0 0
>
> VecNorm   28 1.0 1.3981e-02 1.0 2.10e+07 1.0 0.0e+00 0.0e+00
> 0.0e+00  0 11  0  0  0   0 11  0  0  0  1499
>
> VecSet13 1.0 6.5185e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0 0
>
> VecAYPX   14 1.0 7.1511e-03 1.0 5.24e+06 1.0 0.0e+00 0.0e+00
> 0.0e+00  0  3  0  0  0   0  3  0  0  0   733
>
> VecAssemblyBegin  14 1.0 1.3998e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0 0
>
> VecAssemblyEnd14 1.0 4.2560e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0 0
>
> VecScatterBegin   14 1.0 8.2761e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0 0
>
> VecScatterEnd 14 1.0 4.4665e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0 0
>
> SFSetGraph 1 1.0 6.5993e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0 0
>
> SFSetUp1 1.0 7.9212e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0 0
>
> SFPack 

Re: [petsc-users] Correct way to set/track global numberings in DMPlex?

[Matthew Knepley]

On Wed, Apr 3, 2024 at 2:28 PM Ferrand, Jesus A. 
wrote:

> Dear PETSc team: (Hoping to get a hold of Jed and/or Matt for this one)
> (Also, sorry for the mouthful below) I'm developing routines that will
> read/write CGNS files to DMPlex and vice versa. One of the recurring
> challenges is the bookkeeping
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
> Dear PETSc team:
>
> (Hoping to get a hold of Jed and/or Matt for this one)
> (Also, sorry for the mouthful below)
>
> I'm developing routines that will read/write CGNS files to DMPlex and vice
> versa.
> One of the recurring challenges is the bookkeeping of global numbering for
> vertices and cells.
> Currently, I am restricting my support to single Zone CGNS files, in which
> the file provides global numbers for vertices and cells.
>

I thought Jed had put in parallel CGNS loading. If so, maybe you can
transition to that. If not, we should get your stuff integrated.


> I used PetscHSetI as exemplified in DMPlexBuildFromCellListParallel() to
> obtain local DAG numbers from the global numbers provided by the file.
> I also used PetscSFCreateByMatchingIndices() to establish a basic DAG
> point distribution over the MPI processes.
> I use this PointSF to manually assemble a global PetscSection.
> For owned DAG points  (per the PointSF) , I call
> PetscSectionSetOffset(section, point, file_offset);
> For ghost DAG points  (per the PointSF) I call
> PetscSectionSetOffset(section, point, -(file_offset + 1));
>

This sounds alright to me, although I admit to not understanding exactly
what is being done.

All of what I have just described happens in my CGNS version of
> DMPlexTopologyLoad().
> My intention is to retain those numbers into the DMPlex, and reuse them in
> my CGNS analogues of DMPlexCoordinatesLoad(), DMPlexLabelsLoad(), and
> DMPlexGlobalVectorLoad().
> Anyhow, is this a good wait to track global numbers?
>

The way I think about it, for example in DMPlexBuildFromCellListParallel(),
you load all parallel things in blocks (cell adjacency, vertex coordinates,
etc). Then if you have to redistribute afterwards, you make a PetscSF to do
it. I first make one mapping points to points. With a PetscSection, you can
easily convert this into  dofs to dofs. For example, we load sets of
vertices, but we want vertices distributed as they are attached to cells.
So we create a PetscSF mapping uniform blocks to the division attached to
cells. Then we use the PetscSection for coordinates to make a new PetscSF
and redistribute coordinates.

  Thanks,

 Matt


> Also, I need (for other applications) to eventually call
> DMPlexInterpolate() and DMPlexDistribute(), will the global PetscSection
> offsets be preserved after calling those two?
>
>
> Sincerely:
>
> *J.A. Ferrand*
>
> Embry-Riddle Aeronautical University - Daytona Beach - FL
> Ph.D. Candidate, Aerospace Engineering
>
> M.Sc. Aerospace Engineering
>
> B.Sc. Aerospace Engineering
>
> B.Sc. Computational Mathematics
>
>
> *Phone:* (386)-843-1829
>
> *Email(s):* ferra...@my.erau.edu
>
> jesus.ferr...@gmail.com
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!Z30NkeNY4hgcjCs5RtVH3AgiBI0E4BBkGDPYdLNB10LWOF050wW1AXJDMcOtZ0G3u9nPiKrc0MX9YIYzdUyK$
  

Re: [petsc-users] FE Tabulation values

[Matthew Knepley]

On Tue, Apr 2, 2024 at 4:42 AM Noam T.  wrote:

> Thank you for the clarification.
>
> Are there references specifically for this tabulation method and its
> construction? I have seen some references about  the "FIAT" algorithm, but
> from a quick look I could not find all details.
>

The basis for tabulation is this algorithm:
https://urldefense.us/v3/__https://dl.acm.org/doi/10.1145/1039813.1039820__;!!G_uCfscf7eWS!f6jEiH1CuazpOliQttQfGv2Idl6TtDFefOQi3LMsOc1x75ElNLSY-l0BBvHSJhRf_B0HkkN0nvj0yEAhY_eJ$
 


> ---
>
> On a related note, I stated the values of Nq, Nc and Nb, as they can be
> checked. But to be sure; for the given 2D example:
>
>- Nc = 2 refers to the two compoents as in x/y in 2D
>
> It means that you have a vector quantity here.

>
>-
>
>
>- Nb = 3 * 2 i.e. 3 shape functions (or nodes) times 2 components
>
> Yes. I am explicitly representing the tensor product structure, since
sometimes you do not have a
tensor product and I wanted to be general.


> Testing with a 3D mesh (e.g. a 4-node linear tetrahedron), Nc = 3 and Nb =
> 12, so the same math seems to work, but perhaps there is a different idea
> behind it.
>

Yes, that is right.

  Thanks,

    Matt


> Thanks.
> Noam
> On Tuesday, March 26th, 2024 at 11:17 PM, Matthew Knepley <
> knep...@gmail.com> wrote:
>
> On Tue, Mar 26, 2024 at 2:23 PM Noam T. via petsc-users <
> petsc-users@mcs.anl.gov> wrote:
>
>> Hello, I am trying to understand the FE Tabulation data obtained from e.
>> g . PetscFEComputeTabulation. Using a 2D mesh with a single triangle, first
>> order, with vertices (0,0), (0,1), (1,0) (see msh file attached), and a
>> single quadrature point
>> ZjQcmQRYFpfptBannerStart
>> This Message Is From an External Sender
>> This message came from outside your organization.
>> ZjQcmQRYFpfptBannerEnd
>> Hello,
>>
>> I am trying to understand the FE Tabulation data obtained from e.g .
>> PetscFEComputeTabulation. Using a 2D mesh with a single triangle, first
>> order, with vertices (0,0), (0,1), (1,0) (see msh file attached), and a
>> single quadrature point at (1/3, 1/3), one gets Nb = 6, Nc = 2, Nq = 1, and
>> the arrays for the basis and first derivatives are of sizes [Nq x Nb x Nc]
>> = 12 and[Nq x Nb x Nc x dim] = 24, respectively
>>
>
> The tabulations from PetscFE are recorded on the reference cell. For
> triangles, the reference cell is
> (-1, -1) -- (1, -1) -- (-1, 1). The linear basis functions at these nodes
> are
>
> phi_0: -(x + y) / 2
> phi_1: (x + 1) / 2
> phi_2: (y + 1) / 2
>
> and then you use the tensor product for Nc = 2.
>
> / phi_0 \ / 0 \ etc.
> \ 0 / \ phi_0 /
>
> The values of these two arrays are:
>> basis (T->T[0])
>> [-1/3, 0, 0, -1/3, 2/3, 0,
>> 0, 2/3, 2/3, 0, 0, 2/3]
>>
>
> So these values are indeed the evaluations of those basis functions at
> (1/3, 1/3). The derivatives are similar.
>
> These are the evaluations you want if you are integrating in reference
> space, as we do for the finite element integrals, and also the only way we
> could use a single tabulation for the mesh.
>
> Thanks,
>
> Matt
>
>> deriv (T->T[1])
>> [-1/2, -1/2, 0, 0, 0, 0,
>> -1/2, -1/2, 1/2, 0, 0, 0,
>> 0, 0, 1/2, 0, 0, 1/2,
>> 0, 0, 0, 0, 0, 1/2]
>>
>> How does one get these values? I can't quite find a way to relate them to
>> evaluating the basis functions of a P1 triangle in the given quadrature
>> point.
>>
>> Thanks,
>> Noam
>>
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!f6jEiH1CuazpOliQttQfGv2Idl6TtDFefOQi3LMsOc1x75ElNLSY-l0BBvHSJhRf_B0HkkN0nvj0yC6i1AIh$
>  
> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!f6jEiH1CuazpOliQttQfGv2Idl6TtDFefOQi3LMsOc1x75ElNLSY-l0BBvHSJhRf_B0HkkN0nvj0yADZFpMQ$
>  >
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!f6jEiH1CuazpOliQttQfGv2Idl6TtDFefOQi3LMsOc1x75ElNLSY-l0BBvHSJhRf_B0HkkN0nvj0yC6i1AIh$
  
<https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!f6jEiH1CuazpOliQttQfGv2Idl6TtDFefOQi3LMsOc1x75ElNLSY-l0BBvHSJhRf_B0HkkN0nvj0yADZFpMQ$
 >

Re: [petsc-users] PETSC Matrix debugging

[Matthew Knepley]

On Mon, Apr 1, 2024 at 1:57 PM Shatanawi, Sawsan Muhammad via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Hello everyone, I hope this email finds you well. Is there a way we can
> check how the matrix looks like after setting it. I have tried debugging it
> with gdb- break points- and print statements, but it only gave me one value
> instead of a matrix.
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
> Hello everyone,
>
> I hope this email finds you well.
>
> Is there a way we can check how the matrix looks like after setting it.
> I have tried debugging it with gdb- break points- and print statements,
> but it only gave me one value instead of a matrix.
>
> Thank you in advance for your time and assistance.
>
> I usually use MatView(), which can print to the screen. Is that what you
want?

  Thanks,

 Matt

> Best regards,
>
>  Sawsan
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!cFl9o86vmX7ZWINcjhpIy1gzc938wF-1bgwtrPGwm8zQQ71WUgVstEpSos7o1OWqfmTup5bgaekGgfUuG7aF$
  

Re: [petsc-users] Error loading data coming from a .hdf5 file into a DMSwarm

[Matthew Knepley]

On Mon, Apr 1, 2024 at 11:13 AM MIGUEL MOLINOS PEREZ  wrote:

> Dear Matthew,
>
> Thank you for your suggestion. I tried to update the vector with the
> information coming from the hdf5 file inside the main function. Then I
> print the vector two times (see the lines below), the first time it has the
> correct data. However, the second time, it has the same values like I never
> updated it with VecLoad.
>
> It is an alternative way of initialise a vector coming from DMSWarm with
> previously stored information (keeping the parallel structure)?
>

Oh, you cannot use the Swarm vectors for VecLoad(). They are just a view
into the particle data, and that view is destroyed on restore. Swarm data
is stored in a particle-like data structure, not in Vecs. If you want to
load this Vec, you have to duplicate exactly as you did. This interface is
likely to change in the next year to make this problem go away.

  Thanks,

Matt


> Miguel
>
> //! Load Hdf5 viewer
> PetscViewer viewer_hdf5;
> REQUIRE_NOTHROW(PetscViewerHDF5Open(MPI_COMM_WORLD, Output_hdf5_file,
> FILE_MODE_READ, _hdf5));
> REQUIRE_NOTHROW(PetscViewerHDF5PushTimestepping(viewer_hdf5));
>
> //! Load the vector and fill it with the information from the .hdf5 file
> Vec stdv_q;
> REQUIRE_NOTHROW(DMSwarmCreateGlobalVectorFromField(
> Simulation.atomistic_data, "stdv-q", _q));
>
> REQUIRE_NOTHROW(PetscViewerHDF5PushGroup(viewer_hdf5, "/particle_fields"
> ));
> REQUIRE_NOTHROW(VecLoad(stdv_q, viewer_hdf5));
> REQUIRE_NOTHROW(VecView(stdv_q, PETSC_VIEWER_STDOUT_WORLD));
> REQUIRE_NOTHROW(PetscViewerHDF5PopGroup(viewer_hdf5));
>
> REQUIRE_NOTHROW(DMSwarmDestroyGlobalVectorFromField(
> Simulation.atomistic_data, "stdv-q", _q));
>
> //! Destoy HDF5 context
> REQUIRE_NOTHROW(PetscViewerDestroy(_hdf5));
>
> //! Load the vector again and print
> REQUIRE_NOTHROW(DMSwarmCreateGlobalVectorFromField(
> Simulation.atomistic_data, "stdv-q", _q));
>
> REQUIRE_NOTHROW(VecView(stdv_q, PETSC_VIEWER_STDOUT_WORLD));
>
> REQUIRE_NOTHROW(DMSwarmDestroyGlobalVectorFromField(
> Simulation.atomistic_data, "stdv-q", _q));
>
> Best,
> Miguel
>
> Miguel Molinos
> Investigador postdoctoral
> Juan de la Cierva
> Dpto. Mecánica de Medios Continuos y Teoría de Estructuras - ETSI
> Universidad de Sevilla
> Camino de los descubrimientos, s/n
> 41092 Sevilla
>
>
> [image: us_logo.jpg]
>
>
>
>
> https://urldefense.us/v3/__http://www.us.es__;!!G_uCfscf7eWS!YcVqv_PA_UdFsx4fZHfzMMaZZ7auMp-poE49ThdEDk0fT7DAXF6-0fOpeWMEOJcrVL5grSHp-QCF8PdvRl51$
>  
> Este correo electrónico y, en su caso, cualquier fichero anexo al
> mismo, contiene información de carácter confidencial exclusivamente
> dirigida a su destinatario o destinatarios. Si no es UD. el
> destinatario del mensaje, le ruego lo destruya sin hacer copia digital o
> física, comunicando al emisor por esta misma vía la recepción del presente
> mensaje. Gracias
>
>
>
>
> On 1 Apr 2024, at 16:28, Matthew Knepley  wrote:
>
> From the description, my guess is that this is pointer confusion. The
> vector inside the function is different from the vector outside the
> function.
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YcVqv_PA_UdFsx4fZHfzMMaZZ7auMp-poE49ThdEDk0fT7DAXF6-0fOpeWMEOJcrVL5grSHp-QCF8EnZVeTC$
  
<https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YcVqv_PA_UdFsx4fZHfzMMaZZ7auMp-poE49ThdEDk0fT7DAXF6-0fOpeWMEOJcrVL5grSHp-QCF8AYgu-dB$
 >

Re: [petsc-users] Error loading data coming from a .hdf5 file into a DMSwarm

[Matthew Knepley]

On Sun, Mar 31, 2024 at 4:08 PM MIGUEL MOLINOS PEREZ  wrote:

> Dear all, I am writing a function which store datasets (Vectors) coming
> from a DMSwarm structure into a hdf5 file. This step is done nicely
> write_function(){ PetscViewerHDF5Open(…) PetscViewerHDF5PushTimestepping(…)
> DMSwarmCreateGlobalVectorFromField(…)
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
> Dear all,
>
> I am writing a function which store datasets (Vectors) coming from a DMSwarm
> structure into a hdf5 file. This step is done nicely
>
> write_function(){
> PetscViewerHDF5Open(…)
> PetscViewerHDF5PushTimestepping(…)
> DMSwarmCreateGlobalVectorFromField(…)
> VecLoad(…)
> DMSwarmDestroyGlobalVectorFromField(…)
> }
>
> The resulting hdf5 file looks good after an inspection using python’s
> library h5py.
>
> However, I am finding difficulties when I try to use this .hdf5 file as a
> fresh start for my application. The target field is not properly updated
> when I try to load the stored data (it keeps the default one).
>
> read_function(){
> …
> PetscViewerHDF5Open(…)
> PetscViewerHDF5PushTimestepping(…)
> DMSwarmCreateGlobalVectorFromField(…)
> VecLoad(… )
> DMSwarmDestroyGlobalVectorFromField(…)
> ...
> }
>
> The puzzling part is: if I print the “updated” vector inside of
> read_function() using VecView after VecLoad, the vector seem to hold the
> updated values. However, If I print the field in the main function after
> the call to read_function(), the field remains the same it was before
> calling to read_function() and I do not get any erro message.
>
> It is there something wrong with the logic of my programing? Maybe I am
> missing something.
>

>From the description, my guess is that this is pointer confusion. The
vector inside the function is different from the vector outside the
function.

  Thanks,

Matt


> Thank you in advance.
>
> Best regards,
> Miguel
>
> Miguel Molinos
> Investigador postdoctoral
> Juan de la Cierva
> Dpto. Mecánica de Medios Continuos y Teoría de Estructuras - ETSI
> Universidad de Sevilla
> Camino de los descubrimientos, s/n
> 41092 Sevilla
>
>
> [image: us_logo.jpg]
>
>
>
>
>
> https://urldefense.us/v3/__http://www.us.es__;!!G_uCfscf7eWS!YnltLCiNPOvMtX9m3CBEtxQ-FLDRM3Ef6ZAOt3huF3xEquvYmdHNozTGvZpnCoX8m3gZ_6W9SXKOxFP20r74$
>  
> 
> Este correo electrónico y, en su caso, cualquier fichero anexo al
> mismo, contiene información de carácter confidencial exclusivamente
> dirigida a su destinatario o destinatarios. Si no es UD. el
> destinatario del mensaje, le ruego lo destruya sin hacer copia digital o
> física, comunicando al emisor por esta misma vía la recepción del presente
> mensaje. Gracias
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YnltLCiNPOvMtX9m3CBEtxQ-FLDRM3Ef6ZAOt3huF3xEquvYmdHNozTGvZpnCoX8m3gZ_6W9SXKOxFq8o9DM$
  

Re: [petsc-users] FE Tabulation values

[Matthew Knepley]

On Tue, Mar 26, 2024 at 2:23 PM Noam T. via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Hello, I am trying to understand the FE Tabulation data obtained from e. g
> . PetscFEComputeTabulation. Using a 2D mesh with a single triangle, first
> order, with vertices (0,0), (0,1), (1,0) (see msh file attached), and a
> single quadrature point
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
> Hello,
>
> I am trying to understand the FE Tabulation data obtained from e.g .
> PetscFEComputeTabulation. Using a  2D mesh with a single triangle, first
> order, with vertices (0,0), (0,1), (1,0) (see msh file attached), and a
> single quadrature point at (1/3, 1/3), one gets Nb = 6, Nc = 2, Nq = 1, and
> the arrays for the basis and first derivatives are of sizes [Nq x Nb x Nc]
> = 12 and[Nq x Nb x Nc x dim] = 24, respectively
>

The tabulations from PetscFE are recorded on the reference cell. For
triangles, the reference cell is
(-1, -1) -- (1, -1) -- (-1, 1). The linear basis functions at these nodes
are

phi_0: -(x + y) / 2
phi_1: (x + 1) / 2
phi_2: (y + 1) / 2

and then you use the tensor product for Nc = 2.

/ phi_0 \  /0 \   etc.
\0 /  \ phi_0 /

The values of these two arrays are:
> basis (T->T[0])
> [-1/3, 0, 0, -1/3, 2/3, 0,
>  0, 2/3, 2/3, 0, 0, 2/3]
>

So these values are indeed the evaluations of those basis functions at
(1/3, 1/3). The derivatives are similar.

These are the evaluations you want if you are integrating in reference
space, as we do for the finite element integrals, and also the only way we
could use a single tabulation for the mesh.

  Thanks,

Matt


> deriv (T->T[1])
> [-1/2, -1/2, 0, 0, 0, 0,
>  -1/2, -1/2, 1/2, 0, 0, 0,
>  0, 0, 1/2, 0, 0, 1/2,
>  0, 0, 0, 0, 0, 1/2]
>
> How does one get these values? I can't quite find a way to relate them to
> evaluating the basis functions of a P1 triangle in the given quadrature
> point.
>
> Thanks,
> Noam
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!d8GhOqpkJjEyg-Jdk6hvwyR7vg2vFqL9LClNxZuIOEKhjfIIOknalB0ZPB8qlb-PFQ9QOFJpkI3yqYMPcrLX$
  

Re: [petsc-users] Question about how to use DS to do the discretization

[Matthew Knepley]

On Sat, Mar 23, 2024 at 8:03 AM Gong Yujie 
wrote:

> Dear PETSc group, I'm reading the DS part for the discretization start
> from SNES ex17. c which is a demo for solving linear elasticity problem. I
> have two questions for the details. The first question is for the residual
> function. Is the residual
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
> Dear PETSc group,
>
> I'm reading the DS part for the discretization start from SNES ex17.c
> which is a demo for solving linear elasticity problem. I have two questions
> for the details.
>
> The first question is for the residual function. Is the residual
> calculated as this? The dot product is a little weird because of the
> dimension of the result.
> Here \sigma is the stress tensor, \phi_i is the test function for the i-th
> function (Linear elasticity in 3D contains three equations).
>

The stress term in the momentum equation is

  (-div sigma) . psi = d_i sigma_{ij} psi_j

which is then integrated by parts

  sigma_{ij} d_i psi_j

This is linear isotropic elasticity, so

  sigma = \mu (d_i u_j + d_j u_i) + \lambda \delta_{ij} sigma_{kk}

In PETSc, we phrase the term in the weak form as

  f^1_{ij} d_i psi_j

so f1[i * dim + j] below is sigma_{ij}, from line 298 of ex17.c

  for (PetscInt c = 0; c < Nc; ++c) {
for (PetscInt d = 0; d < dim; ++d) {
  f1[c * dim + d] += mu * (u_x[c * dim + d] + u_x[d * dim + c]);
  f1[c * dim + c] += lambda * u_x[d * dim + d];
}
  }


> The second question is how to derive the Jacobian of the system (line 330
> in ex17.c). As shown in the PetscDSSetJacobian, we need to provide function
> g3() which I think is a 4th-order tensor with size 3*3*3*3 in this linear
> elasticity case. I'm not sure how to get it. Are there any references on
> how to get this Jacobian?
>

The Jacobian indices are shown here:
https://urldefense.us/v3/__https://petsc.org/main/manualpages/FE/PetscFEIntegrateJacobian/__;!!G_uCfscf7eWS!eNdXZ9QpzXPTMCtLneeS7td5OYVeKmhdOrw_fswDze2u7v2JaO7kBmFTVakyHDmWJOCHjf-0GrqGc53QKQec$
 
where the g3 term is

  \nabla\psi^{fc}_f(q) \cdot g3_{fc,gc,df,dg}(u, \nabla u)
\nabla\phi^{gc}_g(q)

To get the Jacobian, we use u = \sum_i u_i psi_i, where psi_i is a vector,
and then differentiate the expression with respect to the coefficient u_i.
Since the operator is already linear,this is just matching indices

  for (PetscInt c = 0; c < Nc; ++c) {
for (PetscInt d = 0; d < dim; ++d) {
  g3[((c * Nc + c) * dim + d) * dim + d] += mu;
  g3[((c * Nc + d) * dim + d) * dim + c] += mu;
  g3[((c * Nc + d) * dim + c) * dim + d] += lambda;
}
  }

Take the first mu term, mu (d_c u_d ) (d_c psi_d). We know that fc == gc
and df == dg, so we get

  g3[((c * Nc + c) * dim + d) * dim + d] += mu;

For the second term, we transpose grad u, so fc == dg and gc == df,

  g3[((c * Nc + d) * dim + d) * dim + c] += mu;

Finally, for the lambda term, fc == df and gc == dg because we are matching
terms in the sum

  g3[((c * Nc + d) * dim + c) * dim + d] += lambda;

  Thanks,

 Matt


> I've checked about the comment before this Jacobian function (line 330 in
> ex17.c) but don't know how to get this.
>
> Thanks in advance!
>
> Best Regards,
> Yujie
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eNdXZ9QpzXPTMCtLneeS7td5OYVeKmhdOrw_fswDze2u7v2JaO7kBmFTVakyHDmWJOCHjf-0GrqGc4DkrV1c$
  

Re: [petsc-users] MATSETVALUES: Fortran problem

[Matthew Knepley]

On Fri, Mar 15, 2024 at 9:55 AM Frank Bramkamp  wrote:

> Dear PETSc Team, I am using the latest petsc version 3. 20. 5. I would
> like to create a matrix using MatCreateSeqAIJ To insert values, I use
> MatSetValues. It seems that the Fortran interface/stubs are missing for
> MatsetValues, as the linker does
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
>
> Dear PETSc Team,
>
> I am using the latest petsc version 3.20.5.
>
>
> I would like to create a matrix using
> MatCreateSeqAIJ
>
> To insert values, I use MatSetValues.
> It seems that the Fortran interface/stubs are missing for MatsetValues, as 
> the linker does not find any subroutine with that name.
> MatSetValueLocal seems to be fine.
>
>
Here is a Fortran example calling MatSetValues():


https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/ksp/tutorials/ex1f.F90?ref_type=heads__;!!G_uCfscf7eWS!ag7gfsg4qE_beGvp2_VPt5PyN1bZcFr8xkVlbukEpUnqYHx_awYDepWWZfT-7rtVF5-lHC2GXj1ETtSIaOLc$
 


> Typically I am using a blocked matrix format (BAIJ), which works fine in 
> fortran.
> Soon we want to try PETSC on GPUs, using the format MATAIJCUSPARSE, since 
> there seems not to be a blocked format available in PETSC for GPUs so far.
>
> You can use the blocked input API, like MatSetValuesBlocked(), with all
the storage formats. There is no explicit blocking in the MATAIJCUSPARSE format
because Nvidia handles the optimization differently.

  Thanks,

 Matt

Therefore I first want to try the pointwise format MatCreateSeqAIJ
format on a CPU, before using the GPU format.
>
> I think that CUDA also supports a block format now ?! Maybe that would be 
> also useful to have one day.
>
>
> Greetings, Frank Bramkamp
>
>
>
>
>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ag7gfsg4qE_beGvp2_VPt5PyN1bZcFr8xkVlbukEpUnqYHx_awYDepWWZfT-7rtVF5-lHC2GXj1ETryTqsT3$
  

Re: [petsc-users] PAMI error on Summit

[Matthew Knepley]

On Thu, Feb 29, 2024 at 4:22 PM Blondel, Sophie via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> I still get the same error when deactivating GPU-aware MPI. I also tried
> unloading spectrum MPI and using openMPI instead (recompiling everything)
> and I get a segfault in PETSc in that case (still using GPU-aware MPI I
> think, at least not explicitly
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
> I still get the same error when deactivating GPU-aware MPI.
>
> I also tried unloading spectrum MPI and using openMPI instead (recompiling
> everything) and I get a segfault in PETSc in that case (still using
> GPU-aware MPI I think, at least not explicitly turning it off):
>

For this case, can you get a stack trace?

  Thanks,

 Matt


> 0 TS dt 1e-12 time 0.
>
> [ERROR] [0]PETSC ERROR:
>
> [ERROR]
> 
>
> [ERROR] [0]PETSC ERROR:
>
> [ERROR] Caught signal number 11 SEGV: Segmentation Violation, probably
> memory access out of range
>
> [ERROR] [0]PETSC ERROR:
>
> [ERROR] Try option -start_in_debugger or -on_error_attach_debugger
>
> [ERROR] [0]PETSC ERROR:
>
> [ERROR] or see 
> https://urldefense.us/v3/__https://petsc.org/release/faq/*valgrind__;Iw!!G_uCfscf7eWS!ealcEfAPe2Mm8ybP__T3FwtT82O3mdXnxfl73KX3wF-_aFOo6hEarAdd0WryuSjgrwh3QbmG-_5WBZSRV5Ft$
>  
> 
>  and
> https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!ealcEfAPe2Mm8ybP__T3FwtT82O3mdXnxfl73KX3wF-_aFOo6hEarAdd0WryuSjgrwh3QbmG-_5WBevQQa9s$
>  
> 
>
> [ERROR] [0]PETSC ERROR:
>
> [ERROR] or try 
> https://urldefense.us/v3/__https://docs.nvidia.com/cuda/cuda-memcheck/index.html__;!!G_uCfscf7eWS!ealcEfAPe2Mm8ybP__T3FwtT82O3mdXnxfl73KX3wF-_aFOo6hEarAdd0WryuSjgrwh3QbmG-_5WBSptcohd$
>  
> 
>  on
> NVIDIA CUDA systems to find memory corruption errors
>
> [ERROR] [0]PETSC ERROR:
>
> [ERROR] configure using --with-debugging=yes, recompile, link, and run
>
> [ERROR] [0]PETSC ERROR:
>
> [ERROR] to get more information on the crash.
>
> [ERROR] [0]PETSC ERROR:
>
> [ERROR] Run with -malloc_debug to check if memory corruption is causing
> the crash.
>
> --
>
> Best,
>
> Sophie
> --
> *From:* Blondel, Sophie via Xolotl-psi-development <
> xolotl-psi-developm...@lists.sourceforge.net>
> *Sent:* Thursday, February 29, 2024 10:17
> *To:* xolotl-psi-developm...@lists.sourceforge.net <
> xolotl-psi-developm...@lists.sourceforge.net>; petsc-users@mcs.anl.gov <
> petsc-users@mcs.anl.gov>
> *Subject:* [Xolotl-psi-development] PAMI error on Summit
>
> Hi,
>
> I am using PETSc build with the Kokkos CUDA backend on Summit but when I
> run my code with multiple MPI tasks I get the following error:
> 0 TS dt 1e-12 time 0.
> errno 14 pid 864558
> xolotl:
> /__SMPI_build_dir__/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164:
> size_t PAMI::Dev
> ice::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t,
> PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed.
> errno 14 pid 864557
> xolotl:
> /__SMPI_build_dir__/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164:
> size_t PAMI::Dev
> ice::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t,
> PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed.
> [e28n07:864557] *** Process received signal ***
> [e28n07:864557] Signal: Aborted (6)
> [e28n07:864557] Signal code:  (-6)
> [e28n07:864557] [ 0]
> linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x200604d8]
> [e28n07:864557] [ 1] 

Re: [petsc-users] PAMI error on Summit

[Matthew Knepley]

On Thu, Feb 29, 2024 at 11:03 AM Blondel, Sophie via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Thank you Junchao, Yes, I am using gpu-aware MPI. Is "-use_gpu_aware_mpi
> 0" a runtime option or a compile option? Best, Sophie From: Junchao Zhang
>  Sent: Thursday, February 29, 2024 10: 50 To:
> Blondel,
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
> Thank you Junchao,
>
> Yes, I am using gpu-aware MPI.
>
> Is "-use_gpu_aware_mpi 0" a runtime option or a compile option?
>

That is a configure option, so

  cd $PETSC_DIR
  ./${PETSC_ARCH}/lib./petsc/conf/reconfigure-${PETSC_ARCH}.py
-use-gpu_aware_mpi 0
  make all

  Thanks,

 Matt


> Best,
>
> Sophie
> --
> *From:* Junchao Zhang 
> *Sent:* Thursday, February 29, 2024 10:50
> *To:* Blondel, Sophie 
> *Cc:* xolotl-psi-developm...@lists.sourceforge.net <
> xolotl-psi-developm...@lists.sourceforge.net>; petsc-users@mcs.anl.gov <
> petsc-users@mcs.anl.gov>
> *Subject:* Re: [petsc-users] PAMI error on Summit
>
> You don't often get email from junchao.zh...@gmail.com. Learn why this is
> important
> 
> Hi Sophie,
>   PetscSFBcastEnd() was calling MPI_Waitall() to finish the communication
> in DMGlobalToLocal.
>   I guess you used gpu-aware MPI.  The error you saw might be due to it.
> You can try without it with a petsc option  -use_gpu_aware_mpi 0
>   But we generally recommend gpu-aware mpi.  You can try on other GPU
> machines to see if it is just an IBM Spectrum MPI problem.
>
>Thanks.
> --Junchao Zhang
>
>
> On Thu, Feb 29, 2024 at 9:17 AM Blondel, Sophie via petsc-users <
> petsc-users@mcs.anl.gov> wrote:
>
> Hi, I am using PETSc build with the Kokkos CUDA backend on Summit but when
> I run my code with multiple MPI tasks I get the following error: 0 TS dt
> 1e-12 time 0. errno 14 pid 864558 xolotl:
> /__SMPI_build_dir__/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/.
> . /pami/components/devices/shmem/shaddr/CMAShaddr. h: 164:
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
> Hi,
>
> I am using PETSc build with the Kokkos CUDA backend on Summit but when I
> run my code with multiple MPI tasks I get the following error:
> 0 TS dt 1e-12 time 0.
> errno 14 pid 864558
> xolotl:
> /__SMPI_build_dir__/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164:
> size_t PAMI::Dev
> ice::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t,
> PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed.
> errno 14 pid 864557
> xolotl:
> /__SMPI_build_dir__/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164:
> size_t PAMI::Dev
> ice::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t,
> PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed.
> [e28n07:864557] *** Process received signal ***
> [e28n07:864557] Signal: Aborted (6)
> [e28n07:864557] Signal code:  (-6)
> [e28n07:864557] [ 0]
> linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x200604d8]
> [e28n07:864557] [ 1] /lib64/glibc-hwcaps/power9/libc-2.28.so
> 
> (gsignal+0xd8)[0x25d796f8]
> [e28n07:864557] [ 2] /lib64/glibc-hwcaps/power9/libc-2.28.so
> 
> (abort+0x164)[0x25d53ff4]
> [e28n07:864557] [ 3] /lib64/glibc-hwcaps/power9/libc-2.28.so
> 
> (+0x3d280)[0x25d6d280]
> [e28n07:864557] [ 4] [e28n07:864558] *** Process received signal ***
> [e28n07:864558] Signal: Aborted (6)
> [e28n07:864558] Signal code:  (-6)
> [e28n07:864558] [ 0]
> linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x200604d8]
> [e28n07:864558] [ 1] /lib64/glibc-hwcaps/power9/libc-2.28.so
> 
> (gsignal+0xd8)[0x25d796f8]
> [e28n07:864558] [ 2] /lib64/glibc-hwcaps/power9/libc-2.28.so
> 
> (abort+0x164)[0x25d53ff4]
> [e28n07:864558] [ 3] 

Re: [petsc-users] Ghost points in DMSwarm

[Matthew Knepley]

On Sun, Feb 25, 2024 at 1:00 PM MIGUEL MOLINOS PEREZ  wrote:

> I see, so I'm the one in charge of the synchronization between the
> particles and the replicas associated, right?
>

Yes. We could conceivably put this in during our rewrite of DMSwarm, but it
would change the semantics. Right now, there is no particle identity.

  Thanks,

Matt


> Best,
> Miguel
>
> On Feb 25, 2024, at 6:53 PM, Matthew Knepley  wrote:
>
> On Sun, Feb 25, 2024 at 7:21 AM MIGUEL MOLINOS PEREZ 
> wrote:
>
>> Dear Matthew,
>>
>> Thank you for answer me out of office hours and for the suggestion. If I
>> interpret it correctly, your point is using the background mesh halo to
>> identify and create ghosts particles in the overlapping region using
>> DMSwarmMigrate().
>>
>> Then two questions hits me:
>>
>>- If I change a variable in the “real” particle (say the position),
>>it should be updated in the replicas on the overlapped regions, right? 
>> This
>>sort of communicator is included in DMSwarmMigrate()?
>>
>> No, the particles that are migrated are copies. There is no current
> analogue of DMGlobalToLocal(). It would need to be constructed. It is
> possible, but would take writing some code. All applications we have
> supported to date do not use replicated particles.
>
>>
>>- The function DMSwarmCreateLocalVectorFromField() does not include
>>extra space for ghost particles. Therefore, I should resize the local size
>>for each rank to accommodate the replicas manually. Right?
>>
>> It uses all particles on the process. If some of those are replicas, they
> will be included.
>
>   Thanks,
>
>  Matt
>
>
>> Anyway, I think this this will take me more time I expected originally.
>> Meanwhile,  I will go ahead with my “humble” parallelization scheme which
>> consist in using AllReduce to create a serial vectors with the variables I
>> need to evaluate rank-wise the residual equations of my problem  (please,
>> don’t judge me, publish or perish...)
>>
>> Many thanks,
>> Miguel
>>
>>
>> On Feb 24, 2024, at 4:24 PM, Matthew Knepley  wrote:
>>
>> On Fri, Feb 23, 2024 at 5:14 PM MIGUEL MOLINOS PEREZ 
>> wrote:
>>
>>> Dear all, I am struggling on how to include ghost points in DMSwarm
>>> local vectors. According to PETSc documentation it seems “straightforward”
>>> for a DMDA mesh. However, I am not so sure on how to do it for a DMSwarm.
>>> In fact, if I add the result
>>> ZjQcmQRYFpfptBannerStart
>>> This Message Is From an External Sender
>>> This message came from outside your organization.
>>>
>>> ZjQcmQRYFpfptBannerEnd
>>> Dear all,
>>>
>>> I am struggling on how to include ghost points in DMSwarm local vectors.
>>> According to PETSc documentation it seems “straightforward” for a DMDA
>>> mesh. However, I am not so sure on how to do it for a DMSwarm. In fact,
>>> if I add the result of DMSwarmGetLocalSize for each rank, the result is
>>> the exact number of particles in the system. Which means that the halo
>>> is zero.
>>>
>>> Since there is a function called DMSwarmCreateLocalVectorFromField,I
>>> was wandering if there is a function already implemented in petsc (and
>>> I’m missing it) to include ghost points in DMSwarm and therefore don’t
>>> have to reinvent the wheel. If so, is there any example out there I can
>>> follow?
>>>
>>
>> DMSwarm is currently different from the other DMs. There is no idea of
>> identity for particles, so there is no idea of a shared particle. You can
>> create this by adding an ID to your particle fields, but it would be a
>> manual process. DMSwarmMigrate() can duplicate particles if you have an
>> overlapping mesh, which would mimic shared particles.
>>
>>   THanks,
>>
>>  Matt
>>
>>
>>> Thanks in advance.
>>>
>>> Regards,
>>> Miguel
>>>
>>> petsc.org
>>> <https://urldefense.us/v3/__https://petsc.org/release/manual/vec/__;!!G_uCfscf7eWS!f6U_z6q8Rk6XrOxvbmE8KyNErlmqYwpGzkxCQ56xX0agWaCG0tLVLh1Cml6fTtqvve0aL3HGiAhZn-hDgIoH5w$>
>>>
>>> <https://urldefense.us/v3/__https://petsc.org/release/manual/vec/__;!!G_uCfscf7eWS!f6U_z6q8Rk6XrOxvbmE8KyNErlmqYwpGzkxCQ56xX0agWaCG0tLVLh1Cml6fTtqvve0aL3HGiAhZn-hDgIoH5w$>
>>> <https://urldefense.us/v3/__https://petsc.org/release/manual/vec/__;!!G_uCfscf7eWS!f6U_z6q8Rk6XrOxvbmE8KyNErlmqYwpGzkxCQ56xX0agWaCG0tLVLh1Cml6fTtqvve0aL3HGiAhZn-hDgIoH5w$>
>>>
>&

Re: [petsc-users] Ghost points in DMSwarm

[Matthew Knepley]

On Sun, Feb 25, 2024 at 7:21 AM MIGUEL MOLINOS PEREZ  wrote:

> Dear Matthew,
>
> Thank you for answer me out of office hours and for the suggestion. If I
> interpret it correctly, your point is using the background mesh halo to
> identify and create ghosts particles in the overlapping region using
> DMSwarmMigrate().
>
> Then two questions hits me:
>
>- If I change a variable in the “real” particle (say the position), it
>should be updated in the replicas on the overlapped regions, right? This
>sort of communicator is included in DMSwarmMigrate()?
>
> No, the particles that are migrated are copies. There is no current
analogue of DMGlobalToLocal(). It would need to be constructed. It is
possible, but would take writing some code. All applications we have
supported to date do not use replicated particles.

>
>- The function DMSwarmCreateLocalVectorFromField() does not include
>extra space for ghost particles. Therefore, I should resize the local size
>for each rank to accommodate the replicas manually. Right?
>
> It uses all particles on the process. If some of those are replicas, they
will be included.

  Thanks,

 Matt


> Anyway, I think this this will take me more time I expected originally.
> Meanwhile,  I will go ahead with my “humble” parallelization scheme which
> consist in using AllReduce to create a serial vectors with the variables I
> need to evaluate rank-wise the residual equations of my problem  (please,
> don’t judge me, publish or perish...)
>
> Many thanks,
> Miguel
>
>
> On Feb 24, 2024, at 4:24 PM, Matthew Knepley  wrote:
>
> On Fri, Feb 23, 2024 at 5:14 PM MIGUEL MOLINOS PEREZ 
> wrote:
>
>> Dear all, I am struggling on how to include ghost points in DMSwarm local
>> vectors. According to PETSc documentation it seems “straightforward” for a
>> DMDA mesh. However, I am not so sure on how to do it for a DMSwarm. In
>> fact, if I add the result
>> ZjQcmQRYFpfptBannerStart
>> This Message Is From an External Sender
>> This message came from outside your organization.
>>
>> ZjQcmQRYFpfptBannerEnd
>> Dear all,
>>
>> I am struggling on how to include ghost points in DMSwarm local vectors.
>> According to PETSc documentation it seems “straightforward” for a DMDA
>> mesh. However, I am not so sure on how to do it for a DMSwarm. In fact,
>> if I add the result of DMSwarmGetLocalSize for each rank, the result is
>> the exact number of particles in the system. Which means that the halo
>> is zero.
>>
>> Since there is a function called DMSwarmCreateLocalVectorFromField,I was
>> wandering if there is a function already implemented in petsc (and I’m
>> missing it) to include ghost points in DMSwarm and therefore don’t have
>> to reinvent the wheel. If so, is there any example out there I can follow?
>>
>
> DMSwarm is currently different from the other DMs. There is no idea of
> identity for particles, so there is no idea of a shared particle. You can
> create this by adding an ID to your particle fields, but it would be a
> manual process. DMSwarmMigrate() can duplicate particles if you have an
> overlapping mesh, which would mimic shared particles.
>
>   THanks,
>
>  Matt
>
>
>> Thanks in advance.
>>
>> Regards,
>> Miguel
>>
>> petsc.org
>> <https://urldefense.us/v3/__https://petsc.org/release/manual/vec/__;!!G_uCfscf7eWS!f6U_z6q8Rk6XrOxvbmE8KyNErlmqYwpGzkxCQ56xX0agWaCG0tLVLh1Cml6fTtqvve0aL3HGiAhZn-hDgIoH5w$>
>>
>> <https://urldefense.us/v3/__https://petsc.org/release/manual/vec/__;!!G_uCfscf7eWS!f6U_z6q8Rk6XrOxvbmE8KyNErlmqYwpGzkxCQ56xX0agWaCG0tLVLh1Cml6fTtqvve0aL3HGiAhZn-hDgIoH5w$>
>> <https://urldefense.us/v3/__https://petsc.org/release/manual/vec/__;!!G_uCfscf7eWS!f6U_z6q8Rk6XrOxvbmE8KyNErlmqYwpGzkxCQ56xX0agWaCG0tLVLh1Cml6fTtqvve0aL3HGiAhZn-hDgIoH5w$>
>>
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bRyZjT28s2cAZIFj69F3Bmv2X5NDZvT-prHmzArFuDq6aqqCaDpfaHH0ip-K53DfXi574ejpx04DurcseuAb$
>  
> <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bRyZjT28s2cAZIFj69F3Bmv2X5NDZvT-prHmzArFuDq6aqqCaDpfaHH0ip-K53DfXi574ejpx04Dukm3lyRJ$
>  >
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bRyZjT28s2cAZIFj69F3Bmv2X5NDZvT-prHmzArFuDq6aqqCaDpfaHH0ip-K53DfXi574ejpx04DurcseuAb$
  
<https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bRyZjT28s2cAZIFj69F3Bmv2X5NDZvT-prHmzArFuDq6aqqCaDpfaHH0ip-K53DfXi574ejpx04Dukm3lyRJ$
 >

Re: [petsc-users] Ghost points in DMSwarm

[Matthew Knepley]

On Fri, Feb 23, 2024 at 5:14 PM MIGUEL MOLINOS PEREZ  wrote:

> Dear all, I am struggling on how to include ghost points in DMSwarm local
> vectors. According to PETSc documentation it seems “straightforward” for a
> DMDA mesh. However, I am not so sure on how to do it for a DMSwarm. In
> fact, if I add the result
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
> Dear all,
>
> I am struggling on how to include ghost points in DMSwarm local vectors.
> According to PETSc documentation it seems “straightforward” for a DMDA
> mesh. However, I am not so sure on how to do it for a DMSwarm. In fact,
> if I add the result of DMSwarmGetLocalSize for each rank, the result is
> the exact number of particles in the system. Which means that the halo is
> zero.
>
> Since there is a function called DMSwarmCreateLocalVectorFromField,I was
> wandering if there is a function already implemented in petsc (and I’m
> missing it) to include ghost points in DMSwarm and therefore don’t have
> to reinvent the wheel. If so, is there any example out there I can follow?
>

DMSwarm is currently different from the other DMs. There is no idea of
identity for particles, so there is no idea of a shared particle. You can
create this by adding an ID to your particle fields, but it would be a
manual process. DMSwarmMigrate() can duplicate particles if you have an
overlapping mesh, which would mimic shared particles.

  THanks,

 Matt


> Thanks in advance.
>
> Regards,
> Miguel
>
> petsc.org
> 
>
> 
> 
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aLpar0ltcRArMpZIgCFcyKDnezb9G3tPzvVdBNtiSHZD2-37Vdmfl-Kp4YrSLNP7wO3gZjhPcXCq2BitoRRm$
  

Re: [petsc-users] How to fetch right values of a local vector

[Matthew Knepley]

On Fri, Feb 23, 2024 at 9:54 AM 袁煕  wrote:

> Hello, I cannot fetch right values of a local vector after a second time
> calling DMSetLocalSection.   My program runs like  call
> PetscSectionSetup(section1,ierr) call
> DMSetLocalSection(dm_mesh,section1,ierr)call 
> PetscSectionDestroy(section1,ierr)call
>
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
> Hello,
>
> I cannot fetch right values of a local vector after a second time calling
> DMSetLocalSection.  My program runs like
>
> 
> call PetscSectionSetup(section1,ierr)
> call DMSetLocalSection(dm_mesh,section1,ierr)
> call PetscSectionDestroy(section1,ierr)
>
> call DMCreateGlobalVector(dm_mesh,vec_a,ierr)
>
> call VecDestroy(vec_a,ierr)
> call DMClearLocalVectors(dm_mesh, ierr);
>

You also need to wipe out the SF,

  call DMSetSectionSF(dm, PETSC_NULL_PETSCSF, ierr)


> call PetscSectionSetup(section2,ierr)
> call DMSetLocalSection(dm_mesh,section2,ierr)
>
> call DMCreateGlobalVector(dm_mesh,vec_a,ierr)
> call DMGetLocalVector(dm_mesh,lvec,ierr);
> call DMGlobalToLocal(dm_mesh,vec_a,INSERT_VALUES,lvec,ierr)
> call VecGetArrayReadF90(lvec,tempv,ierr)
> call VecGetArrayReadF90(vec_a,tempv1,ierr)
> 
>
> In this program, I got two Fortran style arrays, *tempv* and *tempv1*,
> from a Vec *vec_a*. When only 1 cpu is used, those two arrays should be
> completely the same. Am I right? But I found there are differences as
> follows:
>
> *  In array *tempv*,  dofs newly introduced by section2 are zero valued.
> Looks like those values are ignored by DMGlobalToLocal. But in array
> *tempv1*, all dofs values are set correctly.
>
> Did I do something wrong in the above code?
>

We do not usually reset the section on a DM. Instead, when we want to use a
different function space,
we clone the DM, using DMClone(), and make a new function space on the new
DM.

  Thanks,

 MAtt

Much thanks for your help.
>
> X. Yuan, Ph.D. in Solid Mechanics
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bAsyX_068Z0c_UIQmCwlZXop22TY8QcA8rRgki-yA9aSOze_tjnsWkrWUfzE8KpRNzzHsqLXQwFSAMO0WIRk$
  

Re: [petsc-users] Fortran Interface NULL object / Casting

[Matthew Knepley]

On Thu, Dec 8, 2022 at 2:56 PM Barry Smith  wrote:

>
>You would use *PETSC_NULL_DM*LABEL but Matt needs to customize the
> PETSc Fortran stub for DMAddField() for you to handle accepting the NULL
> from PETSc.
>

Sorry, I missed this message. The customization is done.

  Thanks,

 Matt


>Barry
>
>
>
>
>
> On Dec 8, 2022, at 1:05 PM, Nicholas Arnold-Medabalimi 
> wrote:
>
> Hi Petsc Users
>
> I am trying to use DMAddField in a Fortran code. I had some questions on
> casting/passing NULL. I follow how to pass NULL for standard types (INT,
> CHAR, etc).
>
> Is there a method/best practice for passing NULL for Petsc type arguments?
> (In this case DMAddLabel I'd want to pass NULL to a DMLabel Parameter not
> an intrinsic type)
>
> For the 2nd question, In some cases in C we would cast a more specific
> object to Petsc Object
> DMAddField(dm, NULL, (PetscObject)fvm); (where fvm is a PetscFVM type)
> Is there a method or practice to do the same in Fortran.
>
> Thanks
> Nicholas
>
> --
> Nicholas Arnold-Medabalimi
>
> Ph.D. Candidate
> Computational Aeroscience Lab
> University of Michigan
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!Y7V3ljA2ftwwmUl996o8LBEkPFj2p3sNz4c1MmdMRf8S5xQVjNjWnUACSD3Ys6A2ER8jLncCxWS5fZjxqnnF$
  

Re: [petsc-users] Difference between DMGetLocalVector and DMCreateLocalVector

[Matthew Knepley]

You are right. I was not clearing the caches when the section was reset.
Stefano ahs fixed this now:

  
https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7314__;!!G_uCfscf7eWS!ZPkq_k9vdzTZxeW53aWVTIcDgeGnywSm0tGKDJkJqdwgTomfYjw1lsTMyRtAjtruSslcgLIoKGg7IILlBxbJ$
 

  Thanks,

 Matt

On Thu, Feb 22, 2024 at 4:24 AM Stefano Zampini 
wrote:

> I guess we are missing to clear the internal scratch vectors, matt can
> confirm
>
> Can you please add this call AFTER you set the local section to confirm?
>
>   call DMClearLocalVectors(dm_mesh, ierr);
>
>
> Il giorno gio 22 feb 2024 alle ore 11:36 袁煕  ha
> scritto:
>
>> Hello, I found DMGetLocalVector and DMCreateLocalVector generate Vec of
>> different size in my following codes  call
>> PetscSectionSetup(section,ierr) call
>> DMSetLocalSection(dm_mesh,section,ierr) .. . . . . do somethingcall
>> PetscSectionSetup(section,ierr)call
>> ZjQcmQRYFpfptBannerStart
>> This Message Is From an External Sender
>> This message came from outside your organization.
>>
>> ZjQcmQRYFpfptBannerEnd
>>  Hello,
>>
>> I found DMGetLocalVector and DMCreateLocalVector generate Vec of
>> different size in my following codes
>>
>> 
>> call PetscSectionSetup(section,ierr)
>> call DMSetLocalSection(dm_mesh,section,ierr)
>> ..do something
>>
>> call PetscSectionSetup(section,ierr)
>> call DMSetLocalSection(dm_mesh,section,ierr)
>> ..
>> call DMCreateLocalVector(dm_mesh,lvec,ierr);
>> call DMGetLocalVector(dm_mesh,llvec,ierr);
>> call VecGetSize(lvec,off,ierr)
>> call VecGetSize(llvec,offl,ierr)
>> call VecDestroy(lvec,ierr)
>> call DMRestoreLocalVector(dm_mesh,llvec,ierr);
>> 
>> The pointer in the above program is that DMSetLocalSection is called two
>> times. As the petsc manual indicates, "Any existing Section will be
>> destroyed" after calling this function and To my understanding, a new
>> section should be set. Therefore,  The above program should obtain
>> values of *off* and *offl *the same. However, I found that they are
>> different, with *off *equal to the size defined by the new section and
>> *offl* equal to the size defined by the first one.
>>
>> Any problem in above codes? And how can I obtain a Vector of the same
>> size by DMGetLocalVector and DMCreateLocalVector?
>>
>> Much thanks for your help.
>>
>> X. Yuan, Ph.D. in Solid Mechanics
>>
>
>
> --
> Stefano
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!ZPkq_k9vdzTZxeW53aWVTIcDgeGnywSm0tGKDJkJqdwgTomfYjw1lsTMyRtAjtruSslcgLIoKGg7IIE6dMV7$
  

Re: [petsc-users] Near nullspace lost in fieldsplit

[Matthew Knepley]

On Mon, Feb 12, 2024 at 7:10 AM Jeremy Theler (External) <
jeremy.theler-...@ansys.com> wrote:

> Hi Barry
>
> >   The bug fix for 2 is availabel in
> https://gitlab.com/petsc/petsc/-/merge_requests/7279
>
> Note that our code goes through   MatZeroRowsColumns_MPIAIJ() not through
> MatZeroRows_MPIAIJ() so this fix does not change anything for the case I
> mentioned.
>

The other fixes are in https://gitlab.com/petsc/petsc/-/merge_requests/7279

  Thanks,

 Matt


> --
> jeremy
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Near nullspace lost in fieldsplit

[Matthew Knepley]

On Fri, Feb 9, 2024 at 7:04 AM Jeremy Theler (External) <
jeremy.theler-...@ansys.com> wrote:

> Hi all
>
> Because of a combination of settings, our code passes through this line:
>
>
> https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/pc/impls/fieldsplit/fieldsplit.c?ref_type=heads#L692
>
> i.e. the matrices associated with each of the sub-KSPs of a fieldsplit are
> destroyed and then re-created later.
> The thing is that one of these destroyed matrices had a near nullspace
> attached, which is lost because the new matrix does not have it anymore.
>
> Is this a bug or are we missing something?
>

I just want to get a clear picture. You create a PCFIELDSPLIT, set it up,
then pull out the matrices and attach a nullspace before the solve. At a
later time, you start another solve with this PC, and it has the
DIFFERENT_NONZERO_PATTERN flag, so it recreates these matrices and loses
your attached nullspace.

First, does the matrix really change?

Second, I had the same problem. I added callbacks to DM which allow a
nullspace to be automatically attached if you extract a certain subfield.
Are you using a DM?

  Thanks,

Matt


> I'll need some time to come up with a MWE showing how the near nullspace
> is lost, but if its needed I can take a stab at it.
>
> --
> jeremy
>
>
> 
> src/ksp/pc/impls/fieldsplit/fieldsplit.c · main · PETSc / petsc · GitLab
> 
> PETSc, pronounced PET-see (the S is silent), is a suite of data structures
> and routines for the scalable (parallel) solution of scientific
> applications modeled by partial differential equations.
> gitlab.com
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Unique number in each element of a DMPlex mesh

[Matthew Knepley]

On Thu, Feb 8, 2024 at 9:54 AM Berend van Wachem 
wrote:

> Dear Matt,
>
> I have now written a code to transform a DMPlex coming from a DMForest
> to a DMPlex of type DM_POLY, by removing the double faces and edges.
> Also, I can successfully write this transformed DMPlex of type DM_POLY
> to a file.
>
> However, the code to perform the transformation only works on a single
> process. Therefore, I distribute the original DMForest to process 0,
> perform the transformation to obtain a DMPlex of type DM_POLY.
>
> Is there a way, I can redistribute the transformed DMPlex back to the
> original distribution based on the cells' distribution of the original
> DMPlex? Can I somehow "store" the distribution of a DMPlex based on how
> the cells are divided over the processes, and distribute back to this
> original distribution?
>

Yes, in fact that exact code exists and is tested. The "shell" type for
PetscPartitioner can take in the points intended for each process. As long
as point numbers do not change, this works fine.

Second, I bet I could make your conversion routine parallel.

Third, I think Toby has fixed that other bug with coordinates (finger
crossed). We are running the tests now.

  Thanks,

 Matt


> Thanks, best regards,
> Berend.
>
> p.s. If someone is interested in the code to transform a DMForest to a
> DMPlex of type DM_POLY, I am more than happy to share.
>
> On 1/22/24 20:30, Matthew Knepley wrote:
> > On Mon, Jan 22, 2024 at 2:26 PM Berend van Wachem
> > mailto:berend.vanwac...@ovgu.de>> wrote:
> >
> > Dear Matt,
> >
> > The problem is that I haven't figured out how to write a polyhedral
> > DMplex in parallel. So, currently, I can write the Vec data
> > in parallel, but the cones for the cells/faces/edges/nodes for the
> > mesh from just one process to a file (after gathering the
> > DMplex to a single process).
> >
> >
> > Ah shoot. Can you send me a polyhedral mesh (or code to generate one) so
> > I can fix the parallel write problem? Or maybe it is already an issue
> > and I forgot?
> >
> >   From the restart, I can then read the cone information from one
> > process from the file, recreate the DMPlex, and then
> > redistribute it. In this scenario, the Vec data I read in (in
> > parallel) will not match the correct cells of the DMplex. Hence, I
> > need to put it in the right place afterwards.
> >
> >
> > Yes, then searching makes sense. You could call DMLocatePoints(), but
> > maybe you are doing that.
> >
> >Thanks,
> >
> >   Matt
> >
> > Best, Berend.
> >
> > On 1/22/24 20:03, Matthew Knepley wrote:
> >  > On Mon, Jan 22, 2024 at 1:57 PM Berend van Wachem
> > mailto:berend.vanwac...@ovgu.de>
> > <mailto:berend.vanwac...@ovgu.de <mailto:berend.vanwac...@ovgu.de>>>
> > wrote:
> >  >
> >  > Dear Matt,
> >  >
> >  > Thanks for your quick response.
> >  > I have a DMPlex with a polyhedral mesh, and have defined a
> > number of vectors with data at the cell center. I have generated
> >  > data
> >  > for a number of timesteps, and I write the data for each
> > point to a file together with the (x,y,z) co-ordinate of the cell
> >  > center.
> >  >
> >  > When I want to do a restart from the DMPlex, I recreate the
> > DMplex with the polyhedral mesh, redistribute it, and for each cell
> >  > center find the corresponding (x,y,z) co-ordinate and insert
> > the data that corresponds to it. This is quite expensive, as it
> >  > means I need to compare doubles very often.
> >  >
> >  > But reading your response, this may not be a bad way of doing
> it?
> >  >
> >  >
> >  > It always seems to be a game of "what do you want to assume?". I
> > tend to assume that I wrote the DM and Vec in the same order,
> >  > so when I load them they match. This is how Firedrake I/O works,
> > so that you can load up on a different number of processes
> >  > (https://arxiv.org/abs/2401.05868
> > <https://arxiv.org/abs/2401.05868> <https://arxiv.org/abs/2401.05868
> > <https://arxiv.org/abs/2401.05868>>).
> >  >
> >  > So, are you writing a Vec, and then redistributing and writing
> > another Vec? In the scheme above, you would have to wr

Re: [petsc-users] Creating DMPlex from the subsection of a DM

[Matthew Knepley]

On Wed, Feb 7, 2024 at 8:30 AM Mark Adams  wrote:

> Wait for Matt to chime in, but I think you will need to start from scratch
> and use
> https://petsc.org/release/manualpages/DMPlex/DMPlexBuildFromCellList
>

Let me restate things to make sure I get it. You want to select a submesh
that retains its labels? There are many different kinds of submeshes. The
easiest one is

  https://petsc.org/main/manualpages/DMPlex/DMPlexFilter/

The selects a subset of cells. Since you always get the closure of any mesh
point, I am not sure it makes sense to select vertices independently of
cells, but maybe you can explain that to me. You just mark the cells you
want with another label and call that function. All labels will be
propagated (along with everything else).

  Thanks,

 Matt


> Mark
>
> On Wed, Feb 7, 2024 at 5:36 AM Vilmer Dahlberg via petsc-users <
> petsc-users@mcs.anl.gov> wrote:
>
>> Dear PETSc dev team,
>>
>> I am working on an application which makes use of DMPlex. I would like to
>> extract a subsection of the dm based on some field metric defined on its
>> vertices or cells. I am using DMLabel to mark my boundary conditions and
>> would like the child mesh to retain some of the labels. Is there a simple
>> way of doing this? I could spend some time writing my own implementation,
>> looping over cells and building a new DM from scratch or using something
>> like DMPlexBuildFromCellListParallel, but I feel like there should exist a
>> solution. Could you point me in the right direction?
>>
>> Thanks in advance!
>>
>> Best,
>> Vilmer
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Advice for a non conform boundary coupling with PETSc

[Matthew Knepley]

On Mon, Feb 5, 2024 at 10:10 AM GUBERNATIS Pierre 
wrote:

> Hello,
>
> I would appreciate an advice on this task, if possible.
>
>
>
> So far, we use PETSc to ‘couple’ two computations of the same simulation
> code, in a single MPI run.
>
> I simplify but roughly, we have 2 Cartesian domains on which we discretize
> the same evolution problem.
>
> These two computations have a common boundary and we want to solve the
> global problem as if it were a single domain problem.
>
>
>
> We use the ‘ghost cells’ paradigm to handle the boundary conditions.
>
> At each required step of the time algorithm, we update the values in the
> ghost cells of both domains.
>
>
>
> Thanks to these exchanges, we can make a global resolution (as if it were
> a unique domain) by using PETSc : Each domain builds its part of the main
> operator and its part of the RHS. PETSc solves the global problem and
> broadcasts the partial solutions to each domain.
>
> This ‘boundary coupling’ of 2 computations works perfectly well with
> conforming meshes.
>
> (we can consider this type of coupling as a domain decomposition process,
> but instead of cutting a domain in two pieces, we merge two domains
> together).
>
>
>
> We are extending this method to the non-conforming case (we use a specific
> library to make the interpolations/mapping from one grid onto the other).
>
> The non-conforming case doesn’t work so far. Our guess is that the error
> is in the way we fill the data array of PETSc and we keep on working on it.
>
>
>
> Would you see a problem we are missing in this extension to the
> non-conforming situation ? Do you know a similar work we could look at ?
>

Maybe I am saying things you have already thought of, but I would start by
checking the error in my transfer operators. For instance, if it is
supposed to preserve constants, if you put a constant field on the two
domains, go to the global domain, and then back, it is still the same
constant? Does it work with higher polynomials? Once that is assured, we
could look at MMS problems.

  Thanks,

 Matt


> Thanks for any advice….
>
> *Pierre*
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Bound exceed when running code with PETSc

[Matthew Knepley]

On Mon, Feb 5, 2024 at 8:48 AM TAY Wee Beng  wrote:

> Hi all,
>
> Not in my code, nor in PETSc.
>
I would try this. Of course, it would be best to see where exactly the
overflow is happening.

   THanks,

 Matt


> On 5/2/2024 8:21 pm, Matthew Knepley wrote:
>
> On Mon, Feb 5, 2024 at 4:29 AM TAY Wee Beng  wrote:
>
>> Hi,
>>
>> I have a code which uses PETSc and it was working fine. However, I
>> haven't used the code for a while and now we've migrated to a new cray
>> cluster so the compilers have been updated. I'm using gnu fortran.
>>
>> When running the code now, I got the error:
>>
>> Error termination. Backtrace:
>> At line 294 of file
>>
>> /home/users/nus/tsltaywb/myprojects/ibm2d_high_Re_staggered_coef/PETSc_solvers.F90
>> Fortran runtime error: Index '-17591876251192' of dimension 1 of array
>> 'ppv' below lower bound of 1
>>
>> I have attached my makefile. Wonder what I should change to prevent the
>> above error.
>>
>
> This looks like an integer overflow error. DId you configure with 64-bit
> ints?
>
>Thanks,
>
>   Matt
>
>
>> Compiler should be v11.2
>>
>> --
>> Thank you very much.
>>
>> Yours sincerely,
>>
>> 
>> TAY Wee-Beng 郑伟明 (Zheng Weiming)
>> 
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
> --
>
> Thank you very much.
>
> Yours sincerely,
>
> 
> TAY Wee-Beng 郑伟明 (Zheng Weiming)
> 
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-users] Bound exceed when running code with PETSc

[Matthew Knepley]

On Mon, Feb 5, 2024 at 4:29 AM TAY Wee Beng  wrote:

> Hi,
>
> I have a code which uses PETSc and it was working fine. However, I
> haven't used the code for a while and now we've migrated to a new cray
> cluster so the compilers have been updated. I'm using gnu fortran.
>
> When running the code now, I got the error:
>
> Error termination. Backtrace:
> At line 294 of file
>
> /home/users/nus/tsltaywb/myprojects/ibm2d_high_Re_staggered_coef/PETSc_solvers.F90
> Fortran runtime error: Index '-17591876251192' of dimension 1 of array
> 'ppv' below lower bound of 1
>
> I have attached my makefile. Wonder what I should change to prevent the
> above error.
>

This looks like an integer overflow error. DId you configure with 64-bit
ints?

   Thanks,

  Matt


> Compiler should be v11.2
>
> --
> Thank you very much.
>
> Yours sincerely,
>
> 
> TAY Wee-Beng 郑伟明 (Zheng Weiming)
> 
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] PetscSection: Fortran interfaces

[Matthew Knepley]

Can you give us a simple code that is not working for you?  My test should
work

  src/dm/impls/plex/tests/ex26f90

  Thanks,

 Matt

On Sat, Feb 3, 2024 at 11:36 AM Barry Smith  wrote:

>
>   The Fortran "stubs" (subroutines) should be in
> $PETSC_ARCH/src/vec/is/section/interface/ftn-auto/sectionf.c and compiled
> and linked into the PETSc library.
>
>   The same tool that builds the interfaces in
> $PETSC_ARCH/src/vec/f90-mod/ftn-auto-interfaces/petscpetscsection.h90,
> also builds the stubs so it is surprising one would exist but not the other.
>
>   Barry
>
>
> > On Feb 3, 2024, at 11:27 AM, Martin Diehl 
> wrote:
> >
> > Dear PETSc team,
> >
> > I currently can't make use of Fortran interfaces for "section".
> > In particular, I can't see how to use
> >
> > PetscSectionGetFieldComponents
> > PetscSectionGetFieldDof
> > PetscSectionGetFieldOffset
> >
> > The interfaces for them are created in $PETSC_ARCH/src/vec/f90-mod/ftn-
> > auto-interfaces/petscpetscsection.h90, but it seems that they are not
> > exposed to the public.
> >
> > Could you give me a hint how to use them or fix this?
> >
> > with best regards,
> > Martin
> >
> >
> > --
> > KU Leuven
> > Department of Computer Science
> > Department of Materials Engineering
> > Celestijnenlaan 200a
> > 3001 Leuven, Belgium
> >
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] PETSc init question

[Matthew Knepley]

On Wed, Jan 31, 2024 at 10:10 AM Alain O' Miniussi 
wrote:

> Hi,
>
> It is indicated in:
> https://petsc.org/release/manualpages/Sys/PetscInitialize/
> that the init function will call MPI_Init.
>
> What if MPI_Init was already called (as it is the case in my application)


>From the page: " PetscInitialize() calls MPI_Init() if that has yet to be
called,". Also
"Note
If for some reason you must call MPI_Init() separately, call it before
PetscInitialize()."


> and what about MPI_Init_thread.
>

https://petsc.org/release/manualpages/Sys/PETSC_MPI_THREAD_REQUIRED/


> Wouldn't it be more convenient to have a Petsc init function taking a
> already initialized communicator as argument ?
>

Probably not.


> Also, that initialization seems to imply that it is not possible to have
> multiple instance of PETSc on different communicators. Is that the case ?
>

No, this is possible. We have examples for this. You call MPI_init()
yourself, set PETSC_COMM_WORLD to the communicator you want, and then call
PetscInitialize(). See

  https://petsc.org/release/manualpages/Sys/PETSC_COMM_WORLD/

  Thanks,

  Matt

  Thanks,

 Matt


> Thanks
>
> 
> Alain Miniussi
> DSI, Pôles Calcul et Genie Log.
> Observatoire de la Côte d'Azur
> Tél. : +33609650665
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Preconditioning of Liouvillian Superoperator

[Matthew Knepley]

On Wed, Jan 31, 2024 at 8:21 AM Mark Adams  wrote:

> Iterative solvers have to be designed for your particular operator.
> You want to look in your field to see how people solve these problems.
> (eg, zeros on the diagonal will need something like a block solver or maybe
> ILU with a particular ordering)
> I don't personally know anything about this operator. Perhaps someone else
> can help you, but you will probably need to find this yourself.
> Also, hypre's ILUTP is not well supported. You could use our (serial) ILU
> on one processor to experiment with (
> https://petsc.org/main/manualpages/PC/PCILU).
>

As Mark says, understanding your operator is key here. However, some
comments about GMRES. Full
GMRES is guaranteed to converge. By default you are using GMRES(30), which
has no guarantees. You could look at the effect of increasing the subspace
size. This is probably not worth it without first understanding at least
the spectrum of the operator, and other analytic characteristics (say is it
a PDE, or BIE, etc)

  Thanks,

 Matt


> Mark
>
>
> On Wed, Jan 31, 2024 at 6:51 AM Niclas Götting <
> ngoett...@itp.uni-bremen.de> wrote:
>
>> Hi all,
>>
>> I've been trying for the last couple of days to solve a linear system
>> using iterative methods. The system size itself scales exponentially
>> (64^N) with the number of components, so I receive sizes of
>>
>> * (64, 64) for one component
>> * (4096, 4096) for two components
>> * (262144, 262144) for three components
>>
>> I can solve the first two cases with direct solvers and don't run into
>> any problems; however, the last case is the first nontrivial and it's
>> too large for a direct solution, which is why I believe that I need an
>> iterative solver.
>>
>> As I know the solution for the first two cases, I tried to reproduce
>> them using GMRES and failed on the second, because GMRES didn't converge
>> and seems to have been going in the wrong direction (the vector to which
>> it "tries" to converge is a totally different one than the correct
>> solution). I went as far as -ksp_max_it 100, which takes orders of
>> magnitude longer than the LU solution and I'd intuitively think that
>> GMRES should not take *that* much longer than LU. Here is the
>> information I have about this (4096, 4096) system:
>>
>> * not symmetric (which is why I went for GMRES)
>> * not singular (SVD: condition number 1.427743623238e+06, 0 of 4096
>> singular values are (nearly) zero)
>> * solving without preconditioning does not converge (DIVERGED_ITS)
>> * solving with iLU and natural ordering fails due to zeros on the diagonal
>> * solving with iLU and RCM ordering does not converge (DIVERGED_ITS)
>>
>> After some searching I also found
>> [this](http://arxiv.org/abs/1504.06768) paper, which mentions the use of
>> ILUTP, which I believe in PETSc should be used via hypre, which,
>> however, threw a SEGV for me, and I'm not sure if it's worth debugging
>> at this point in time, because I might be missing something entirely
>> different.
>>
>> Does anybody have an idea how this system could be solved in finite
>> time, such that the method also scales to the three component problem?
>>
>> Thank you all very much in advance!
>>
>> Best regards
>> Niclas
>>
>>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] pc_redistribute issue

[Matthew Knepley]

Okay, so the tolerance is right. It must be using ||b|| instead of ||r0||.
Run with

  -redistribute_ksp_monitor_true_residual

You might have to force r0.

  Thanks,

 Matt

On Sat, Jan 27, 2024 at 11:44 AM Mark Adams  wrote:

> KSP Object: (redistribute_) 1 MPI process
>   type: gmres
> restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> happy breakdown tolerance 1e-30
>   maximum iterations=1, initial guess is zero
>
> *  tolerances:  relative=1e-12, absolute=1e-50, divergence=1.*  left
> preconditioning
>   using PRECONDITIONED norm type for convergence test
> PC Object: (redistribute_) 1 MPI process
>   type: bjacobi
> number of blocks = 1
> Local solver information for first block is in the following KSP and
> PC objects on rank 0:
> Use -redistribute_ksp_view ::ascii_info_detail to display information
> for all blocks
> KSP Object: (redistribute_sub_) 1 MPI process
>   type: preonly
>   maximum iterations=1, initial guess is zero
>   tolerances:  relative=1e-05, absolute=1e-50, divergence=1.
>   left preconditioning
>   using NONE norm type for convergence test
> PC Object: (redistribute_sub_) 1 MPI process
>   type: lu
> out-of-place factorization
>     tolerance for zero pivot 2.22045e-14
>
> On Sat, Jan 27, 2024 at 10:24 AM Matthew Knepley 
> wrote:
>
>> View the solver.
>>
>>   Matt
>>
>> On Sat, Jan 27, 2024 at 9:43 AM Mark Adams  wrote:
>>
>>> I am not getting ksp_rtol 1e-12 into pc_redistribute correctly?
>>>
>>>
>>>
>>> * Linear redistribute_ solve converged due to CONVERGED_RTOL iterations
>>> 1  0 KSP Residual norm 2.182384017537e+02   1 KSP Residual norm
>>> 1.889764161573e-04 *
>>> Number of iterations =   1 N = 47628
>>> Residual norm 8.65917e-07
>>> #PETSc Option Table entries:
>>> -f S.bin # (source: command line)
>>> -ksp_monitor # (source: command line)
>>> -ksp_type preonly # (source: command line)
>>> -mat_block_size 36 # (source: command line)
>>> -mat_view ascii::ascii_info # (source: command line)
>>> -options_left # (source: command line)
>>> -pc_type redistribute # (source: command line)
>>> -redistribute_ksp_converged_reason # (source: command line)
>>>
>>> *-redistribute_ksp_rtol 1e-12 # (source: command 
>>> line)*-redistribute_ksp_type
>>> gmres # (source: command line)
>>> -redistribute_pc_type bjacobi # (source: command line)
>>> -redistribute_sub_pc_factor_mat_solver_type mumps # (source: command
>>> line)
>>> -redistribute_sub_pc_type lu # (source: command line)
>>> #End of PETSc Option Table entries
>>> *There are no unused options.*
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-users] pc_redistribute issue

[Matthew Knepley]

View the solver.

  Matt

On Sat, Jan 27, 2024 at 9:43 AM Mark Adams  wrote:

> I am not getting ksp_rtol 1e-12 into pc_redistribute correctly?
>
>
>
> * Linear redistribute_ solve converged due to CONVERGED_RTOL iterations 1
> 0 KSP Residual norm 2.182384017537e+02   1 KSP Residual norm
> 1.889764161573e-04 *
> Number of iterations =   1 N = 47628
> Residual norm 8.65917e-07
> #PETSc Option Table entries:
> -f S.bin # (source: command line)
> -ksp_monitor # (source: command line)
> -ksp_type preonly # (source: command line)
> -mat_block_size 36 # (source: command line)
> -mat_view ascii::ascii_info # (source: command line)
> -options_left # (source: command line)
> -pc_type redistribute # (source: command line)
> -redistribute_ksp_converged_reason # (source: command line)
>
> *-redistribute_ksp_rtol 1e-12 # (source: command line)*-redistribute_ksp_type
> gmres # (source: command line)
> -redistribute_pc_type bjacobi # (source: command line)
> -redistribute_sub_pc_factor_mat_solver_type mumps # (source: command line)
> -redistribute_sub_pc_type lu # (source: command line)
> #End of PETSc Option Table entries
> *There are no unused options.*
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Question about a parallel implementation of PCFIELDSPLIT

[Matthew Knepley]

On Tue, Jan 23, 2024 at 11:06 AM Pantelis Moschopoulos <
pmoschopou...@outlook.com> wrote:

> Dear Matt,
>
> Thank you for your explanation. The new methodology is straightforward to
> implement.
> Still, I have one more question . When I use the option
> -pc_fieldsplit_schur_precondition full, PETSc computes internally the exact
> Schur complement matrix representation. Based on the example matrix that
> you send, the Schur complement is: S = -v^t (A^-1) u. How will PETSc will
> calculate the vector (A^-1) u ? Or it calculates the exact Schur complement
> matrix differently?
>

FULL calls MatSchurComplementComputeExplicitOperator(), which
calls KSPMatSolve() to compute A^{-1} u, which default to KSPSolve for each
column if no specialized code is available. So it should just use your
solver for the (0,0) block, and then take the dot product with v.

  Thanks,

 Matt


> Thanks,
> Pantelis
> ------
> *Από:* Matthew Knepley 
> *Στάλθηκε:* Τρίτη, 23 Ιανουαρίου 2024 5:21 μμ
> *Προς:* Pantelis Moschopoulos 
> *Κοιν.:* Barry Smith ; petsc-users@mcs.anl.gov <
> petsc-users@mcs.anl.gov>
> *Θέμα:* Re: [petsc-users] Question about a parallel implementation of
> PCFIELDSPLIT
>
> On Tue, Jan 23, 2024 at 9:45 AM Pantelis Moschopoulos <
> pmoschopou...@outlook.com> wrote:
>
> Dear Matt,
>
> I read about the MATLRC. However, its correct usage is not clear to me so
> I have the following questions:
>
>1. The U and V input matrices should be created as dense using
>MatCreateDense?
>
>
> Yes. If you have one row, it looks like a vector, or a matrix with one
> column.
>
> If you have 1 row on the bottom, then
>
>   U = [0, 0, ..., 0, 1]
>   V = [the row]
>   C = [1]
>
> will give you that. However, you have an extra row and column?
>
>
>
>1. I use the command MatCreateLRC just to declare the matrix and then
>MatLRCSetMats to pass the values of the constituents?
>
> You can use
>
>   MatCreate(comm, )
>   MatSetSizes(M, ...)
>   MatSetType(M, MATLRC)
>   MatLRCSetMats(M, ...)
>
> However, you are right that it is a little more complicated, because A is
> not just the upper block here.
>
>
>1.
>
> Then, how do I proceed? How I apply the step of Sherman-Morrison-Woobury
> formula? I intend to use iterative solvers for A (main matrix) so I will
> not have its A^-1 at hand which I think is what the
> Sherman-Morrison-Woobury formula needs.
>
>
> I think I was wrong. MatLRC is not the best fit. We should use MatNest
> instead. Then you could have
>
>   A   u
>   v^t 0
>
> as your matrix. We could still get an explicit Schur complement
> automatically using nested FieldSplit. So
>
>   1) Make the MATNEST matrix as shown above
>
>   2) Use PCFIELDSPLIT  for it. This should be an easy IS, since there is
> only one row in the second one.
>
>   3) Select the full Schur complement
>
> -pc_type fieldsplit
> -pc_fieldsplit_type schur
> -pc_fieldsplit_schur_fact_type full
> -pc_fieldsplit_schur_precondition full
>
>   4) Use a recursive FieldSplit (might be able to use
> -fieldsplit_0_pc_fieldsplit_detect_saddle_point)
>
> -fieldsplit_0_pc_type fieldsplit
> -fieldsplit_0_pc_fieldsplit_0_fields 0,1,2
> -fieldsplit_0_pc_fieldsplit_1_fields 3
>
> I think this does what you want, and should be much easier than getting
> MatLRC to do it.
>
>   Thanks,
>
>  Matt
>
>
> Thanks,
> Pantelis
> --
> *Από:* Matthew Knepley 
> *Στάλθηκε:* Τρίτη, 23 Ιανουαρίου 2024 3:20 μμ
> *Προς:* Pantelis Moschopoulos 
> *Κοιν.:* Barry Smith ; petsc-users@mcs.anl.gov <
> petsc-users@mcs.anl.gov>
> *Θέμα:* Re: [petsc-users] Question about a parallel implementation of
> PCFIELDSPLIT
>
> On Tue, Jan 23, 2024 at 8:16 AM Pantelis Moschopoulos <
> pmoschopou...@outlook.com> wrote:
>
> Dear Matt,
>
> Thank you for your response. This is an idealized setup where I have only
> one row/column. Sometimes we might need two or even three constraints based
> on the application. Thus, I will pursue the user-defined IS.
>
>
> Anything < 50 I would use MatLRC. The bottleneck is the inversion of a
> dense matrix of size k x k, where k is the number of constraints. Using an
> IS is definitely fine, but dense rows can detract from iterative
> convergence.
>
>
> When I supply the IS using the command PCFieldSplitSetIS, I do not specify
> anything in the matrix set up right?
>
>
> You should just need to specify the rows for each field as an IS.
>
>   Thanks,
>
>  Matt
>
>
> Thanks,
> Pantelis
> --

Re: [petsc-users] Question about a parallel implementation of PCFIELDSPLIT

[Matthew Knepley]

On Tue, Jan 23, 2024 at 9:45 AM Pantelis Moschopoulos <
pmoschopou...@outlook.com> wrote:

> Dear Matt,
>
> I read about the MATLRC. However, its correct usage is not clear to me so
> I have the following questions:
>
>1. The U and V input matrices should be created as dense using
>MatCreateDense?
>
>
Yes. If you have one row, it looks like a vector, or a matrix with one
column.

If you have 1 row on the bottom, then

  U = [0, 0, ..., 0, 1]
  V = [the row]
  C = [1]

will give you that. However, you have an extra row and column?


>
>1. I use the command MatCreateLRC just to declare the matrix and then
>MatLRCSetMats to pass the values of the constituents?
>
> You can use

  MatCreate(comm, )
  MatSetSizes(M, ...)
  MatSetType(M, MATLRC)
  MatLRCSetMats(M, ...)

However, you are right that it is a little more complicated, because A is
not just the upper block here.


>1.
>
> Then, how do I proceed? How I apply the step of Sherman-Morrison-Woobury
> formula? I intend to use iterative solvers for A (main matrix) so I will
> not have its A^-1 at hand which I think is what the
> Sherman-Morrison-Woobury formula needs.
>

I think I was wrong. MatLRC is not the best fit. We should use MatNest
instead. Then you could have

  A   u
  v^t 0

as your matrix. We could still get an explicit Schur complement
automatically using nested FieldSplit. So

  1) Make the MATNEST matrix as shown above

  2) Use PCFIELDSPLIT  for it. This should be an easy IS, since there is
only one row in the second one.

  3) Select the full Schur complement

-pc_type fieldsplit
-pc_fieldsplit_type schur
-pc_fieldsplit_schur_fact_type full
-pc_fieldsplit_schur_precondition full

  4) Use a recursive FieldSplit (might be able to use
-fieldsplit_0_pc_fieldsplit_detect_saddle_point)

-fieldsplit_0_pc_type fieldsplit
-fieldsplit_0_pc_fieldsplit_0_fields 0,1,2
-fieldsplit_0_pc_fieldsplit_1_fields 3

I think this does what you want, and should be much easier than getting
MatLRC to do it.

  Thanks,

 Matt


> Thanks,
> Pantelis
> --
> *Από:* Matthew Knepley 
> *Στάλθηκε:* Τρίτη, 23 Ιανουαρίου 2024 3:20 μμ
> *Προς:* Pantelis Moschopoulos 
> *Κοιν.:* Barry Smith ; petsc-users@mcs.anl.gov <
> petsc-users@mcs.anl.gov>
> *Θέμα:* Re: [petsc-users] Question about a parallel implementation of
> PCFIELDSPLIT
>
> On Tue, Jan 23, 2024 at 8:16 AM Pantelis Moschopoulos <
> pmoschopou...@outlook.com> wrote:
>
> Dear Matt,
>
> Thank you for your response. This is an idealized setup where I have only
> one row/column. Sometimes we might need two or even three constraints based
> on the application. Thus, I will pursue the user-defined IS.
>
>
> Anything < 50 I would use MatLRC. The bottleneck is the inversion of a
> dense matrix of size k x k, where k is the number of constraints. Using an
> IS is definitely fine, but dense rows can detract from iterative
> convergence.
>
>
> When I supply the IS using the command PCFieldSplitSetIS, I do not specify
> anything in the matrix set up right?
>
>
> You should just need to specify the rows for each field as an IS.
>
>   Thanks,
>
>  Matt
>
>
> Thanks,
> Pantelis
> --
> *Από:* Matthew Knepley 
> *Στάλθηκε:* Τρίτη, 23 Ιανουαρίου 2024 2:51 μμ
> *Προς:* Pantelis Moschopoulos 
> *Κοιν.:* Barry Smith ; petsc-users@mcs.anl.gov <
> petsc-users@mcs.anl.gov>
> *Θέμα:* Re: [petsc-users] Question about a parallel implementation of
> PCFIELDSPLIT
>
> On Tue, Jan 23, 2024 at 4:23 AM Pantelis Moschopoulos <
> pmoschopou...@outlook.com> wrote:
>
> Dear Matt and Dear Barry,
>
> I have some follow up questions regarding FieldSplit.
> Let's assume that I solve again the 3D Stokes flow but now I have also a
> global constraint that controls the flow rate at the inlet. Now, the matrix
> has the same unknowns as before, i.e. ux0,uy0,uz0,p0//ux1,uy1,uz1,p1//...,
> but the last line (and the last column) corresponds to the contribution of
> the global constraint equation. I want to incorporate the last line (and
> last column)  into the local block of velocities (split 0) and the
> pressure. The problem is how I do that. I have two questions:
>
>1. Now, the block size should be 5 in the matrix and vector creation
>for this problem?
>
> No. Blocksize is only useful when the vector/matrix layout is completely
> regular, meaning _every_ block looks the same. Here you have a single row
> to be added in.
>
>
>1. I have to rely entirely on PCFieldSplitSetIS to create the two
>blocks? Can I augment simply the previously defined block 0 with the last
>line of the matrix?
>
> If you want to add in a s

Re: [petsc-users] Question about a parallel implementation of PCFIELDSPLIT

[Matthew Knepley]

On Tue, Jan 23, 2024 at 8:16 AM Pantelis Moschopoulos <
pmoschopou...@outlook.com> wrote:

> Dear Matt,
>
> Thank you for your response. This is an idealized setup where I have only
> one row/column. Sometimes we might need two or even three constraints based
> on the application. Thus, I will pursue the user-defined IS.
>

Anything < 50 I would use MatLRC. The bottleneck is the inversion of a
dense matrix of size k x k, where k is the number of constraints. Using an
IS is definitely fine, but dense rows can detract from iterative
convergence.


> When I supply the IS using the command PCFieldSplitSetIS, I do not specify
> anything in the matrix set up right?
>

You should just need to specify the rows for each field as an IS.

  Thanks,

 Matt


> Thanks,
> Pantelis
> ------
> *Από:* Matthew Knepley 
> *Στάλθηκε:* Τρίτη, 23 Ιανουαρίου 2024 2:51 μμ
> *Προς:* Pantelis Moschopoulos 
> *Κοιν.:* Barry Smith ; petsc-users@mcs.anl.gov <
> petsc-users@mcs.anl.gov>
> *Θέμα:* Re: [petsc-users] Question about a parallel implementation of
> PCFIELDSPLIT
>
> On Tue, Jan 23, 2024 at 4:23 AM Pantelis Moschopoulos <
> pmoschopou...@outlook.com> wrote:
>
> Dear Matt and Dear Barry,
>
> I have some follow up questions regarding FieldSplit.
> Let's assume that I solve again the 3D Stokes flow but now I have also a
> global constraint that controls the flow rate at the inlet. Now, the matrix
> has the same unknowns as before, i.e. ux0,uy0,uz0,p0//ux1,uy1,uz1,p1//...,
> but the last line (and the last column) corresponds to the contribution of
> the global constraint equation. I want to incorporate the last line (and
> last column)  into the local block of velocities (split 0) and the
> pressure. The problem is how I do that. I have two questions:
>
>1. Now, the block size should be 5 in the matrix and vector creation
>for this problem?
>
> No. Blocksize is only useful when the vector/matrix layout is completely
> regular, meaning _every_ block looks the same. Here you have a single row
> to be added in.
>
>
>1. I have to rely entirely on PCFieldSplitSetIS to create the two
>blocks? Can I augment simply the previously defined block 0 with the last
>line of the matrix?
>
> If you want to add in a single row, then you have to specify the IS
> yourself since we cannot generate it from the regular pattern.
>
> However, if you know that you will only ever have a single constraint row
> (which I assume is fairly dense), then I would suggest instead using
> MatLRC, which Jose developed for SLEPc. This handles the last row/col as a
> low-rank correction. One step of Sherman-Morrison-Woobury solves this
> exactly. It requires a solve for A, for which you can use FieldSplit as
> normal.
>
>   Thanks,
>
>  Matt
>
>
> Up to this moment, I use the following commands to create the Field split:
> ufields(3) = [0, 1, 2]
> pfields(1) = [3]
>
> call PCSetType(pc, PCFIELDSPLIT, ierr)
> call PCFieldSplitSetBlockSize(pc, 4,ierr)
>  call PCFieldSplitSetFields(pc, "0", 3, ufields, ufields,ierr)
>  call PCFieldSplitSetFields(pc, "1", 1, pfields, pfields,ierr)
>
> Thanks,
> Pantelis
>
>
> --
> *Από:* Matthew Knepley 
> *Στάλθηκε:* Παρασκευή, 19 Ιανουαρίου 2024 11:31 μμ
> *Προς:* Barry Smith 
> *Κοιν.:* Pantelis Moschopoulos ;
> petsc-users@mcs.anl.gov 
> *Θέμα:* Re: [petsc-users] Question about a parallel implementation of
> PCFIELDSPLIT
>
> On Fri, Jan 19, 2024 at 4:25 PM Barry Smith  wrote:
>
>
>Generally fieldsplit is used on problems that have a natural "split" of
> the variables into two or more subsets. For example u0,v0,u1,v1,u2,v2,u3,v4
> This is often indicated in the vectors and matrices with the "blocksize"
> argument, 2 in this case. DM also often provides this information.
>
>When laying out a vector/matrix with a blocksize one must ensure that
> an equal number of of the subsets appears on each MPI process. So, for
> example, if the above vector is distributed over 3 MPI processes one could
> use   u0,v0,u1,v1   u2,v2  u3,v3  but one cannot use u0,v0,u1
>  v1,u2,v2   u3,v3.  Another way to think about it is that one must split up
> the vector as indexed by block among the processes. For most multicomponent
> problems this type of decomposition is very natural in the logic of the
> code.
>
>
> This blocking is only convenient, not necessary. You can specify your own
> field division using PCFieldSplitSetIS().
>
>   Thanks,
>
>  Matt
>
>
>   Barry
>
>
> On Jan 19, 2024, at 3:19 AM, Pantelis Moschopoulos <
> pmoschopou...@outlook.com> wrot

Re: [petsc-users] Question about a parallel implementation of PCFIELDSPLIT

[Matthew Knepley]

On Tue, Jan 23, 2024 at 4:23 AM Pantelis Moschopoulos <
pmoschopou...@outlook.com> wrote:

> Dear Matt and Dear Barry,
>
> I have some follow up questions regarding FieldSplit.
> Let's assume that I solve again the 3D Stokes flow but now I have also a
> global constraint that controls the flow rate at the inlet. Now, the matrix
> has the same unknowns as before, i.e. ux0,uy0,uz0,p0//ux1,uy1,uz1,p1//...,
> but the last line (and the last column) corresponds to the contribution of
> the global constraint equation. I want to incorporate the last line (and
> last column)  into the local block of velocities (split 0) and the
> pressure. The problem is how I do that. I have two questions:
>
>1. Now, the block size should be 5 in the matrix and vector creation
>for this problem?
>
> No. Blocksize is only useful when the vector/matrix layout is completely
regular, meaning _every_ block looks the same. Here you have a single row
to be added in.

>
>1. I have to rely entirely on PCFieldSplitSetIS to create the two
>blocks? Can I augment simply the previously defined block 0 with the last
>line of the matrix?
>
> If you want to add in a single row, then you have to specify the IS
yourself since we cannot generate it from the regular pattern.

However, if you know that you will only ever have a single constraint row
(which I assume is fairly dense), then I would suggest instead using
MatLRC, which Jose developed for SLEPc. This handles the last row/col as a
low-rank correction. One step of Sherman-Morrison-Woobury solves this
exactly. It requires a solve for A, for which you can use FieldSplit as
normal.

  Thanks,

 Matt


> Up to this moment, I use the following commands to create the Field split:
> ufields(3) = [0, 1, 2]
> pfields(1) = [3]
>
> call PCSetType(pc, PCFIELDSPLIT, ierr)
> call PCFieldSplitSetBlockSize(pc, 4,ierr)
>  call PCFieldSplitSetFields(pc, "0", 3, ufields, ufields,ierr)
>  call PCFieldSplitSetFields(pc, "1", 1, pfields, pfields,ierr)
>
> Thanks,
> Pantelis
>
>
> --
> *Από:* Matthew Knepley 
> *Στάλθηκε:* Παρασκευή, 19 Ιανουαρίου 2024 11:31 μμ
> *Προς:* Barry Smith 
> *Κοιν.:* Pantelis Moschopoulos ;
> petsc-users@mcs.anl.gov 
> *Θέμα:* Re: [petsc-users] Question about a parallel implementation of
> PCFIELDSPLIT
>
> On Fri, Jan 19, 2024 at 4:25 PM Barry Smith  wrote:
>
>
>Generally fieldsplit is used on problems that have a natural "split" of
> the variables into two or more subsets. For example u0,v0,u1,v1,u2,v2,u3,v4
> This is often indicated in the vectors and matrices with the "blocksize"
> argument, 2 in this case. DM also often provides this information.
>
>When laying out a vector/matrix with a blocksize one must ensure that
> an equal number of of the subsets appears on each MPI process. So, for
> example, if the above vector is distributed over 3 MPI processes one could
> use   u0,v0,u1,v1   u2,v2  u3,v3  but one cannot use u0,v0,u1
>  v1,u2,v2   u3,v3.  Another way to think about it is that one must split up
> the vector as indexed by block among the processes. For most multicomponent
> problems this type of decomposition is very natural in the logic of the
> code.
>
>
> This blocking is only convenient, not necessary. You can specify your own
> field division using PCFieldSplitSetIS().
>
>   Thanks,
>
>  Matt
>
>
>   Barry
>
>
> On Jan 19, 2024, at 3:19 AM, Pantelis Moschopoulos <
> pmoschopou...@outlook.com> wrote:
>
> Dear all,
>
> When I am using PCFIELDSPLIT and pc type "schur" in serial mode everything
> works fine. When I turn now to parallel, I observe that the number of ranks
> that I can use must divide the number of N without any remainder, where N
> is the number of unknowns. Otherwise, an error of the following form
> emerges: "Local columns of A10 3473 do not equal local rows of A00 3471".
>
> Can I do something to overcome this?
>
> Thanks,
> Pantelis
>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-users] Unique number in each element of a DMPlex mesh

[Matthew Knepley]

On Mon, Jan 22, 2024 at 2:26 PM Berend van Wachem 
wrote:

> Dear Matt,
>
> The problem is that I haven't figured out how to write a polyhedral DMplex
> in parallel. So, currently, I can write the Vec data
> in parallel, but the cones for the cells/faces/edges/nodes for the mesh
> from just one process to a file (after gathering the
> DMplex to a single process).
>

Ah shoot. Can you send me a polyhedral mesh (or code to generate one) so I
can fix the parallel write problem? Or maybe it is already an issue and I
forgot?


>  From the restart, I can then read the cone information from one process
> from the file, recreate the DMPlex, and then
> redistribute it. In this scenario, the Vec data I read in (in parallel)
> will not match the correct cells of the DMplex. Hence, I
> need to put it in the right place afterwards.
>

Yes, then searching makes sense. You could call DMLocatePoints(), but maybe
you are doing that.

  Thanks,

 Matt


> Best, Berend.
>
> On 1/22/24 20:03, Matthew Knepley wrote:
> > On Mon, Jan 22, 2024 at 1:57 PM Berend van Wachem <
> berend.vanwac...@ovgu.de <mailto:berend.vanwac...@ovgu.de>> wrote:
> >
> > Dear Matt,
> >
> > Thanks for your quick response.
> > I have a DMPlex with a polyhedral mesh, and have defined a number of
> vectors with data at the cell center. I have generated
> > data
> > for a number of timesteps, and I write the data for each point to a
> file together with the (x,y,z) co-ordinate of the cell
> > center.
> >
> > When I want to do a restart from the DMPlex, I recreate the DMplex
> with the polyhedral mesh, redistribute it, and for each cell
> > center find the corresponding (x,y,z) co-ordinate and insert the
> data that corresponds to it. This is quite expensive, as it
> > means I need to compare doubles very often.
> >
> > But reading your response, this may not be a bad way of doing it?
> >
> >
> > It always seems to be a game of "what do you want to assume?". I tend to
> assume that I wrote the DM and Vec in the same order,
> > so when I load them they match. This is how Firedrake I/O works, so that
> you can load up on a different number of processes
> > (https://arxiv.org/abs/2401.05868 <https://arxiv.org/abs/2401.05868>).
> >
> > So, are you writing a Vec, and then redistributing and writing another
> Vec? In the scheme above, you would have to write both
> > DMs. Are you trying to avoid this?
> >
> >Thanks,
> >
> >   Matt
> >
> > Thanks,
> >
> > Berend.
> >
> > On 1/22/24 18:58, Matthew Knepley wrote:
> >  > On Mon, Jan 22, 2024 at 10:49 AM Berend van Wachem <
> berend.vanwac...@ovgu.de <mailto:berend.vanwac...@ovgu.de>
> > <mailto:berend.vanwac...@ovgu.de <mailto:berend.vanwac...@ovgu.de>>>
> wrote:
> >  >
> >  > Dear Petsc-Team,
> >  >
> >  > Is there a good way to define a unique integer number in each
> element
> >  > (e.g. a cell) of a DMPlex mesh, which is in the same location,
> >  > regardless of the number of processors or the distribution of
> the mesh
> >  > over the processors?
> >  >
> >  > So, for instance, if I have a DMPlex box mesh, the
> top-right-front
> >  > corner element (e.g. cell) will always have the same unique
> number,
> >  > regardless of the number of processors the mesh is
> distributed over?
> >  >
> >  > I want to be able to link the results I have achieved with a
> mesh from
> >  > DMPlex on a certain number of cores to the same mesh from a
> DMPlex on a
> >  > different number of cores.
> >  >
> >  > Of course, I could make a tree based on the distance of each
> element to
> >  > a certain point (based on the X,Y,Z co-ordinates of the
> element), and go
> >  > through this tree in the same way and define an integer based
> on this,
> >  > but that seems rather cumbersome.
> >  >
> >  >
> >  > I think this is harder than it sounds. The distance will not work
> because it can be very degenerate.
> >  > You could lexicographically sort the coordinates, but this is
> hard in parallel. It is fine if you are willing
> >  > to gather everything on one process. You could put down a p4est,
> use the Morton order to number them since this is stable
> > for a
> >  > given refinemen

Re: [petsc-users] Unique number in each element of a DMPlex mesh

[Matthew Knepley]

On Mon, Jan 22, 2024 at 1:57 PM Berend van Wachem 
wrote:

> Dear Matt,
>
> Thanks for your quick response.
> I have a DMPlex with a polyhedral mesh, and have defined a number of
> vectors with data at the cell center. I have generated data
> for a number of timesteps, and I write the data for each point to a file
> together with the (x,y,z) co-ordinate of the cell center.
>
> When I want to do a restart from the DMPlex, I recreate the DMplex with
> the polyhedral mesh, redistribute it, and for each cell
> center find the corresponding (x,y,z) co-ordinate and insert the data that
> corresponds to it. This is quite expensive, as it
> means I need to compare doubles very often.
>
> But reading your response, this may not be a bad way of doing it?
>

It always seems to be a game of "what do you want to assume?". I tend to
assume that I wrote the DM and Vec in the same order, so when I load them
they match. This is how Firedrake I/O works, so that you can load up on a
different number of processes (https://arxiv.org/abs/2401.05868).

So, are you writing a Vec, and then redistributing and writing another Vec?
In the scheme above, you would have to write both DMs. Are you trying to
avoid this?

  Thanks,

 Matt


> Thanks,
>
> Berend.
>
> On 1/22/24 18:58, Matthew Knepley wrote:
> > On Mon, Jan 22, 2024 at 10:49 AM Berend van Wachem <
> berend.vanwac...@ovgu.de <mailto:berend.vanwac...@ovgu.de>> wrote:
> >
> > Dear Petsc-Team,
> >
> > Is there a good way to define a unique integer number in each element
> > (e.g. a cell) of a DMPlex mesh, which is in the same location,
> > regardless of the number of processors or the distribution of the
> mesh
> > over the processors?
> >
> > So, for instance, if I have a DMPlex box mesh, the top-right-front
> > corner element (e.g. cell) will always have the same unique number,
> > regardless of the number of processors the mesh is distributed over?
> >
> > I want to be able to link the results I have achieved with a mesh
> from
> > DMPlex on a certain number of cores to the same mesh from a DMPlex
> on a
> > different number of cores.
> >
> > Of course, I could make a tree based on the distance of each element
> to
> > a certain point (based on the X,Y,Z co-ordinates of the element),
> and go
> > through this tree in the same way and define an integer based on
> this,
> > but that seems rather cumbersome.
> >
> >
> > I think this is harder than it sounds. The distance will not work
> because it can be very degenerate.
> > You could lexicographically sort the coordinates, but this is hard in
> parallel. It is fine if you are willing
> > to gather everything on one process. You could put down a p4est, use the
> Morton order to number them since this is stable for a
> > given refinement. And then within each box lexicographically sort the
> centroids. This is definitely cumbersome, but I cannot
> > think of anything else. This also might have parallel problems since you
> need to know how much overlap you need to fill each box.
> >
> >Thanks,
> >
> >Matt
> >
> > Thanks and best regards, Berend.
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to
> > which their experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/ <
> http://www.cse.buffalo.edu/~knepley/>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-users] Unique number in each element of a DMPlex mesh

[Matthew Knepley]

On Mon, Jan 22, 2024 at 10:49 AM Berend van Wachem 
wrote:

> Dear Petsc-Team,
>
> Is there a good way to define a unique integer number in each element
> (e.g. a cell) of a DMPlex mesh, which is in the same location,
> regardless of the number of processors or the distribution of the mesh
> over the processors?
>
> So, for instance, if I have a DMPlex box mesh, the top-right-front
> corner element (e.g. cell) will always have the same unique number,
> regardless of the number of processors the mesh is distributed over?
>
> I want to be able to link the results I have achieved with a mesh from
> DMPlex on a certain number of cores to the same mesh from a DMPlex on a
> different number of cores.
>
> Of course, I could make a tree based on the distance of each element to
> a certain point (based on the X,Y,Z co-ordinates of the element), and go
> through this tree in the same way and define an integer based on this,
> but that seems rather cumbersome.
>

I think this is harder than it sounds. The distance will not work because
it can be very degenerate.
You could lexicographically sort the coordinates, but this is hard in
parallel. It is fine if you are willing
to gather everything on one process. You could put down a p4est, use the
Morton order to number them since this is stable for a given refinement.
And then within each box lexicographically sort the centroids. This is
definitely cumbersome, but I cannot think of anything else. This also might
have parallel problems since you need to know how much overlap you need to
fill each box.

  Thanks,

  Matt


> Thanks and best regards, Berend.

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Question about a parallel implementation of PCFIELDSPLIT

[Matthew Knepley]

On Fri, Jan 19, 2024 at 4:25 PM Barry Smith  wrote:

>
>Generally fieldsplit is used on problems that have a natural "split" of
> the variables into two or more subsets. For example u0,v0,u1,v1,u2,v2,u3,v4
> This is often indicated in the vectors and matrices with the "blocksize"
> argument, 2 in this case. DM also often provides this information.
>
>When laying out a vector/matrix with a blocksize one must ensure that
> an equal number of of the subsets appears on each MPI process. So, for
> example, if the above vector is distributed over 3 MPI processes one could
> use   u0,v0,u1,v1   u2,v2  u3,v3  but one cannot use u0,v0,u1
>  v1,u2,v2   u3,v3.  Another way to think about it is that one must split up
> the vector as indexed by block among the processes. For most multicomponent
> problems this type of decomposition is very natural in the logic of the
> code.
>

This blocking is only convenient, not necessary. You can specify your own
field division using PCFieldSplitSetIS().

  Thanks,

 Matt


>   Barry
>
>
> On Jan 19, 2024, at 3:19 AM, Pantelis Moschopoulos <
> pmoschopou...@outlook.com> wrote:
>
> Dear all,
>
> When I am using PCFIELDSPLIT and pc type "schur" in serial mode everything
> works fine. When I turn now to parallel, I observe that the number of ranks
> that I can use must divide the number of N without any remainder, where N
> is the number of unknowns. Otherwise, an error of the following form
> emerges: "Local columns of A10 3473 do not equal local rows of A00 3471".
>
> Can I do something to overcome this?
>
> Thanks,
> Pantelis
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] HashMap Error when populating AIJCUSPARSE matrix

[Matthew Knepley]

On Thu, Jan 18, 2024 at 4:18 PM Yesypenko, Anna 
wrote:

> Hi Matt, Barry,
>
> Apologies for the extra dependency on scipy. I can replicate the error by
> calling setValue (i,j,v) in a loop as well.
> In roughly half of 10 runs, the following script fails because of an error
> in hashmapijv – the same as my original post.
> It successfully runs without error the other times.
>
> Barry is right that it's CUDA specific. The script runs fine on the CPU.
> Do you have any suggestions or example scripts on assigning entries to a
> AIJCUSPARSE matrix?
>

Oh, you definitely do not want to be doing this. I believe you would rather

1) Make the CPU matrix and then convert to AIJCUSPARSE. This is efficient.

2) Produce the values on the GPU and call

  https://petsc.org/main/manualpages/Mat/MatSetPreallocationCOO/
  https://petsc.org/main/manualpages/Mat/MatSetValuesCOO/

  This is what most people do who are forming matrices directly on the GPU.

What you are currently doing is incredibly inefficient, and I think
accounts for you running out of memory.
It talks back and forth between the CPU and GPU.

  Thanks,

 Matt

Here is a minimum snippet that doesn't depend on scipy.
> ```
> from petsc4py import PETSc
> import numpy as np
>
> n = int(5e5);
> nnz = 3 * np.ones(n, dtype=np.int32)
> nnz[0] = nnz[-1] = 2
> A = PETSc.Mat(comm=PETSc.COMM_WORLD)
> A.createAIJ(size=[n,n],comm=PETSc.COMM_WORLD,nnz=nnz)
> A.setType('aijcusparse')
>
> A.setValue(0, 0, 2)
> A.setValue(0, 1, -1)
> A.setValue(n-1, n-2, -1)
> A.setValue(n-1, n-1, 2)
>
> for index in range(1, n - 1):
>  A.setValue(index, index - 1, -1)
>  A.setValue(index, index, 2)
>  A.setValue(index, index + 1, -1)
> A.assemble()
> ```
> If it means anything to you, when the hash error occurs, it is for index
> 67283 after filling 201851 nonzero values.
>
> Thank you for your help and suggestions!
> Anna
>
> --
> *From:* Barry Smith 
> *Sent:* Thursday, January 18, 2024 2:35 PM
> *To:* Yesypenko, Anna 
> *Cc:* petsc-users@mcs.anl.gov 
> *Subject:* Re: [petsc-users] HashMap Error when populating AIJCUSPARSE
> matrix
>
>
>Do you ever get a problem with 'aij` ?   Can you run in a loop with
> 'aij' to confirm it doesn't fail then?
>
>
>
>Barry
>
>
> On Jan 17, 2024, at 4:51 PM, Yesypenko, Anna  wrote:
>
> Dear Petsc users/developers,
>
> I'm experiencing a bug when using petsc4py with GPU support. It may be my
> mistake in how I set up a AIJCUSPARSE matrix.
> For larger matrices, I sometimes encounter a error in assigning matrix
> values; the error is thrown in PetscHMapIJVQuerySet().
> Here is a minimum snippet that populates a sparse tridiagonal matrix.
>
> ```
> from petsc4py import PETSc
> from scipy.sparse import diags
> import numpy as np
>
> n = int(5e5);
>
> nnz = 3 * np.ones(n, dtype=np.int32); nnz[0] = nnz[-1] = 2
> A = PETSc.Mat(comm=PETSc.COMM_WORLD)
> A.createAIJ(size=[n,n],comm=PETSc.COMM_WORLD,nnz=nnz)
> A.setType('aijcusparse')
> tmp = diags([-1,2,-1],[-1,0,+1],shape=(n,n)).tocsr()
> A.setValuesCSR(tmp.indptr,tmp.indices,tmp.data)
> ### this is the line where the error is thrown.
> A.assemble()
> ```
>
> The error trace is below:
> ```
> File "petsc4py/PETSc/Mat.pyx", line 2603, in
> petsc4py.PETSc.Mat.setValuesCSR
>   File "petsc4py/PETSc/petscmat.pxi", line 1039, in
> petsc4py.PETSc.matsetvalues_csr
>   File "petsc4py/PETSc/petscmat.pxi", line 1032, in
> petsc4py.PETSc.matsetvalues_ijv
> petsc4py.PETSc.Error: error code 76
> [0] MatSetValues() at
> /work/06368/annayesy/ls6/petsc/src/mat/interface/matrix.c:1497
> [0] MatSetValues_Seq_Hash() at
> /work/06368/annayesy/ls6/petsc/include/../src/mat/impls/aij/seq/seqhashmatsetvalues.h:52
> [0] PetscHMapIJVQuerySet() at
> /work/06368/annayesy/ls6/petsc/include/petsc/private/hashmapijv.h:10
> [0] Error in external library
> [0] [khash] Assertion: `ret >= 0' failed.
> ```
>
> If I run the same script a handful of times, it will run without errors
> eventually.
> Does anyone have insight on why it is behaving this way? I'm running on a
> node with 3x NVIDIA A100 PCIE 40GB.
>
> Thank you!
> Anna
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Swarm view HDF5

[Matthew Knepley]

On Thu, Jan 18, 2024 at 1:20 PM Mark Adams  wrote:

> I had this working at one point.
>
> Should I add PetscViewerHDF5PushTimestepping?
> I don't create a viewer now, but I could make one.
>

That will make it work. The real fix would be to check at swarm.c:45 to see
whether timestepping is set.

  Thanks,

Matt


> Thanks,
> Mark
>
>
> On Thu, Jan 18, 2024 at 11:26 AM Matthew Knepley 
> wrote:
>
>> On Thu, Jan 18, 2024 at 10:08 AM Mark Adams  wrote:
>>
>>> I am trying to view a DMSwarm with: -weights_view hdf5:part.h5
>>>
>>>  Vec f;
>>>   PetscCall(DMSetOutputSequenceNumber(sw, 0, 0.0));
>>>   PetscCall(DMSwarmCreateGlobalVectorFromField(sw, "w_q", ));
>>>   PetscCall(PetscObjectSetName((PetscObject)f, "particle weights"));
>>>   PetscCall(VecViewFromOptions(f, NULL, "-weights_view"));
>>>   PetscCall(DMSwarmDestroyGlobalVectorFromField(sw, "w_q", ));
>>>
>>> And I get this error. I had this working once and did not set
>>> PetscViewerHDF5PushTimestepping, so I wanted to check.
>>>
>>
>> We probably were not checking then. We might have to check there when we
>> set the timestep.
>>
>>   Thanks,
>>
>> Matt
>>
>>
>>> Thanks,
>>> Mark
>>>
>>>
>>> [0]PETSC ERROR: Object is in wrong state
>>> [0]PETSC ERROR: Timestepping has not been pushed yet. Call
>>> PetscViewerHDF5PushTimestepping() first
>>> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the
>>> program crashed before usage or a spelling mistake, etc!
>>> [0]PETSC ERROR:   Option left: name:-options_left (no value) source:
>>> command line
>>> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.20.3-461-g585a01bd688
>>>  GIT Date: 2024-01-16 23:32:45 +
>>> [0]PETSC ERROR: ./ex30k on a arch-macosx-gnu-O named MarksMac-302.local
>>> by markadams Thu Jan 18 10:05:53 2024
>>> [0]PETSC ERROR: Configure options CFLAGS="-g
>>> -Wno-deprecated-declarations " CXXFLAGS="-g -Wno-deprecated-declarations "
>>> COPTFLAGS=-O CXXOPTFLAGS=-O --with-cc=/usr/local/opt/llvm/bin/clang
>>> --with-cxx=/usr/local/opt/llvm/bin/clang++ --download-mpich
>>> --with-strict-petscerrorcode --download-triangle=1 --with-debugging=0
>>> --download-hdf5=1 PETSC_ARCH=arch-macosx-gnu-O
>>> [0]PETSC ERROR: #1 PetscViewerHDF5SetTimestep() at
>>> /Users/markadams/Codes/petsc/src/sys/classes/viewer/impls/hdf5/hdf5v.c:990
>>> [0]PETSC ERROR: #2 VecView_Swarm_HDF5_Internal() at
>>> /Users/markadams/Codes/petsc/src/dm/impls/swarm/swarm.c:45
>>> [0]PETSC ERROR: #3 VecView_Swarm() at
>>> /Users/markadams/Codes/petsc/src/dm/impls/swarm/swarm.c:86
>>> [0]PETSC ERROR: #4 VecView() at
>>> /Users/markadams/Codes/petsc/src/vec/vec/interface/vector.c:806
>>> [0]PETSC ERROR: #5 PetscObjectView() at
>>> /Users/markadams/Codes/petsc/src/sys/objects/destroy.c:76
>>> [0]PETSC ERROR: #6 PetscObjectViewFromOptions() at
>>> /Users/markadams/Codes/petsc/src/sys/objects/destroy.c:128
>>> [0]PETSC ERROR: #7 VecViewFromOptions() at
>>> /Users/markadams/Codes/petsc/src/vec/vec/interface/vector.c:691
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-users] Swarm view HDF5

[Matthew Knepley]

On Thu, Jan 18, 2024 at 10:08 AM Mark Adams  wrote:

> I am trying to view a DMSwarm with: -weights_view hdf5:part.h5
>
>  Vec f;
>   PetscCall(DMSetOutputSequenceNumber(sw, 0, 0.0));
>   PetscCall(DMSwarmCreateGlobalVectorFromField(sw, "w_q", ));
>   PetscCall(PetscObjectSetName((PetscObject)f, "particle weights"));
>   PetscCall(VecViewFromOptions(f, NULL, "-weights_view"));
>   PetscCall(DMSwarmDestroyGlobalVectorFromField(sw, "w_q", ));
>
> And I get this error. I had this working once and did not set
> PetscViewerHDF5PushTimestepping, so I wanted to check.
>

We probably were not checking then. We might have to check there when we
set the timestep.

  Thanks,

Matt


> Thanks,
> Mark
>
>
> [0]PETSC ERROR: Object is in wrong state
> [0]PETSC ERROR: Timestepping has not been pushed yet. Call
> PetscViewerHDF5PushTimestepping() first
> [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the
> program crashed before usage or a spelling mistake, etc!
> [0]PETSC ERROR:   Option left: name:-options_left (no value) source:
> command line
> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.20.3-461-g585a01bd688
>  GIT Date: 2024-01-16 23:32:45 +
> [0]PETSC ERROR: ./ex30k on a arch-macosx-gnu-O named MarksMac-302.local by
> markadams Thu Jan 18 10:05:53 2024
> [0]PETSC ERROR: Configure options CFLAGS="-g -Wno-deprecated-declarations
> " CXXFLAGS="-g -Wno-deprecated-declarations " COPTFLAGS=-O CXXOPTFLAGS=-O
> --with-cc=/usr/local/opt/llvm/bin/clang
> --with-cxx=/usr/local/opt/llvm/bin/clang++ --download-mpich
> --with-strict-petscerrorcode --download-triangle=1 --with-debugging=0
> --download-hdf5=1 PETSC_ARCH=arch-macosx-gnu-O
> [0]PETSC ERROR: #1 PetscViewerHDF5SetTimestep() at
> /Users/markadams/Codes/petsc/src/sys/classes/viewer/impls/hdf5/hdf5v.c:990
> [0]PETSC ERROR: #2 VecView_Swarm_HDF5_Internal() at
> /Users/markadams/Codes/petsc/src/dm/impls/swarm/swarm.c:45
> [0]PETSC ERROR: #3 VecView_Swarm() at
> /Users/markadams/Codes/petsc/src/dm/impls/swarm/swarm.c:86
> [0]PETSC ERROR: #4 VecView() at
> /Users/markadams/Codes/petsc/src/vec/vec/interface/vector.c:806
> [0]PETSC ERROR: #5 PetscObjectView() at
> /Users/markadams/Codes/petsc/src/sys/objects/destroy.c:76
> [0]PETSC ERROR: #6 PetscObjectViewFromOptions() at
> /Users/markadams/Codes/petsc/src/sys/objects/destroy.c:128
> [0]PETSC ERROR: #7 VecViewFromOptions() at
> /Users/markadams/Codes/petsc/src/vec/vec/interface/vector.c:691
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Undestanding how to increase the overlap

[Matthew Knepley]

On Thu, Jan 18, 2024 at 10:05 AM TARDIEU Nicolas 
wrote:

> Arrghhh ! Shame on me !
> Sorry for that and thank you for the help Matt.
> I works like a charm now.
>

No problem. I like when I figure it out :)

  Thanks,

 Matt


> --
> *De :* knep...@gmail.com 
> *Envoyé :* jeudi 18 janvier 2024 15:49
> *À :* TARDIEU Nicolas 
> *Cc :* petsc-users@mcs.anl.gov 
> *Objet :* Re: [petsc-users] Undestanding how to increase the overlap
>
> On Thu, Jan 18, 2024 at 9:46 AM TARDIEU Nicolas 
> wrote:
>
> Hi Matt,
> The isovl is in global numbering. I am pasting the output of the script  :
>
>
> I see. Is your matrix diagonal? If so, there is no overlap. You need
> connections to the other rows in order to have overlapping submatrices.
>
>   Thanks,
>
>  Matt
>
>
>
> #
> # The matrix
> # ---
> [1,0]:Mat Object: 2 MPI processes
> [1,0]:  type: mpiaij
> [1,0]:row 0: (0, 0.)
> [1,0]:row 1: (1, 1.)
> [1,0]:row 2: (2, 2.)
> [1,0]:row 3: (3, 3.)
> [1,0]:row 4: (4, 4.)
> [1,0]:row 5: (5, 5.)
>
> #
> # The IS isovl before the call to increaseOverlap
> # ---
> [1,0]:locSize before = 3
> [1,0]:IS Object: 1 MPI processes
> [1,0]:  type: stride
> [1,0]:Number of indices in (stride) set 3
> [1,0]:0 0
> [1,0]:1 1
> [1,0]:2 2
> [1,1]:locSize before = 3
> [1,1]:IS Object: 1 MPI processes
> [1,1]:  type: stride
> [1,1]:Number of indices in (stride) set 3
> [1,1]:0 3
> [1,1]:1 4
> [1,1]:2 5
>
> #
> # The IS isovl after the call to increaseOverlap
> # ---
> [1,0]:locSize after = 3
> [1,0]:IS Object: 1 MPI processes
> [1,0]:  type: general
> [1,0]:Number of indices in set 3
> [1,0]:0 0
> [1,0]:1 1
> [1,0]:2 2
> [1,1]:locSize after = 3
> [1,1]:IS Object: 1 MPI processes
> [1,1]:  type: general
> [1,1]:Number of indices in set 3
> [1,1]:0 3
> [1,1]:1 4
> [1,1]:2 5
>
> #
>
> Regards,
> Nicolas
> --
> Nicolas Tardieu
> Ing PhD Computational Mechanics
> EDF - R Dpt ERMES
> PARIS-SACLAY, FRANCE
> --
> *De :* knep...@gmail.com 
> *Envoyé :* jeudi 18 janvier 2024 15:29
> *À :* TARDIEU Nicolas 
> *Cc :* petsc-users@mcs.anl.gov 
> *Objet :* Re: [petsc-users] Undestanding how to increase the overlap
>
> On Thu, Jan 18, 2024 at 9:24 AM TARDIEU Nicolas via petsc-users <
> petsc-users@mcs.anl.gov> wrote:
>
> Dear PETSc Team,
>
> I am trying to understand how to increase the overlap of a matrix.
> I wrote the attached petsc4py script where I build a simple matrix and
> play with the increaseOverlap method. Unfortunately, before and after the
> call, nothing changes in the index set. FYI, I have tried to mimic
> src/ksp/ksp/tutorials/ex82.c
>  line 72:76.
> Here is how I run the script : "mpiexec -n 2 python test_overlap.py"
>
> Could you please indicate what I am missing ?
>
>
> Usually matrix functions like this take input in global indices. It looks
> like your isovl is in local indices. Am I reading that correctly?
>
>   Thanks,
>
>  Matt
>
>
> Regards,
> Nicolas
> --
> Nicolas Tardieu
> Ing PhD Computational Mechanics
> EDF - R Dpt ERMES
> PARIS-SACLAY, FRANCE
>
>
> Ce message et toutes les pièces jointes (ci-après le 'Message') sont
> établis à l'intention exclusive des destinataires et les informations qui y
> figurent sont strictement confidentielles. Toute utilisation de ce Message
> non conforme à sa destination, toute diffusion ou toute publication totale
> ou partielle, est interdite sauf autorisation expresse.
>
> Si vous n'êtes pas le destinataire de ce Message, il vous est interdit de
> le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou
> partie. Si vous avez reçu ce Message par erreur, merci de le supprimer de
> votre système, ainsi que toutes ses copies, et de n'en garder aucune trace
> sur quelque support que ce soit. Nous vous remercions également d'en
> avertir immédiatement l'expéditeur par retour du message.
>
> Il est impossible de garantir que les communications par messagerie
> électronique arrivent en temps utile, sont sécurisées ou dénuées de toute
> erreur ou virus.
> 
>
> This message and any attachments (the 'Message') are intended solely for
> the addressees. The information contained in this Message is confidential.
> Any use of information contained in this Message not in accord with its
> purpose, any dissemination or disclosure, either whole or partial, is
> prohibited except formal approval.
>
> If you are not the addressee, you may not copy, forward, disclose or use
> any part of it. If you have received this message in error, please delete
> it and all copies from your system and 

Re: [petsc-users] Undestanding how to increase the overlap

[Matthew Knepley]

On Thu, Jan 18, 2024 at 9:46 AM TARDIEU Nicolas 
wrote:

> Hi Matt,
> The isovl is in global numbering. I am pasting the output of the script  :
>

I see. Is your matrix diagonal? If so, there is no overlap. You need
connections to the other rows in order to have overlapping submatrices.

  Thanks,

 Matt


>
> #
> # The matrix
> # ---
> [1,0]:Mat Object: 2 MPI processes
> [1,0]:  type: mpiaij
> [1,0]:row 0: (0, 0.)
> [1,0]:row 1: (1, 1.)
> [1,0]:row 2: (2, 2.)
> [1,0]:row 3: (3, 3.)
> [1,0]:row 4: (4, 4.)
> [1,0]:row 5: (5, 5.)
>
> #
> # The IS isovl before the call to increaseOverlap
> # ---
> [1,0]:locSize before = 3
> [1,0]:IS Object: 1 MPI processes
> [1,0]:  type: stride
> [1,0]:Number of indices in (stride) set 3
> [1,0]:0 0
> [1,0]:1 1
> [1,0]:2 2
> [1,1]:locSize before = 3
> [1,1]:IS Object: 1 MPI processes
> [1,1]:  type: stride
> [1,1]:Number of indices in (stride) set 3
> [1,1]:0 3
> [1,1]:1 4
> [1,1]:2 5
>
> #
> # The IS isovl after the call to increaseOverlap
> # ---
> [1,0]:locSize after = 3
> [1,0]:IS Object: 1 MPI processes
> [1,0]:  type: general
> [1,0]:Number of indices in set 3
> [1,0]:0 0
> [1,0]:1 1
> [1,0]:2 2
> [1,1]:locSize after = 3
> [1,1]:IS Object: 1 MPI processes
> [1,1]:  type: general
> [1,1]:Number of indices in set 3
> [1,1]:0 3
> [1,1]:1 4
> [1,1]:2 5
>
> #
>
> Regards,
> Nicolas
> --
> Nicolas Tardieu
> Ing PhD Computational Mechanics
> EDF - R Dpt ERMES
> PARIS-SACLAY, FRANCE
> --
> *De :* knep...@gmail.com 
> *Envoyé :* jeudi 18 janvier 2024 15:29
> *À :* TARDIEU Nicolas 
> *Cc :* petsc-users@mcs.anl.gov 
> *Objet :* Re: [petsc-users] Undestanding how to increase the overlap
>
> On Thu, Jan 18, 2024 at 9:24 AM TARDIEU Nicolas via petsc-users <
> petsc-users@mcs.anl.gov> wrote:
>
> Dear PETSc Team,
>
> I am trying to understand how to increase the overlap of a matrix.
> I wrote the attached petsc4py script where I build a simple matrix and
> play with the increaseOverlap method. Unfortunately, before and after the
> call, nothing changes in the index set. FYI, I have tried to mimic
> src/ksp/ksp/tutorials/ex82.c
>  line 72:76.
> Here is how I run the script : "mpiexec -n 2 python test_overlap.py"
>
> Could you please indicate what I am missing ?
>
>
> Usually matrix functions like this take input in global indices. It looks
> like your isovl is in local indices. Am I reading that correctly?
>
>   Thanks,
>
>  Matt
>
>
> Regards,
> Nicolas
> --
> Nicolas Tardieu
> Ing PhD Computational Mechanics
> EDF - R Dpt ERMES
> PARIS-SACLAY, FRANCE
>
>
> Ce message et toutes les pièces jointes (ci-après le 'Message') sont
> établis à l'intention exclusive des destinataires et les informations qui y
> figurent sont strictement confidentielles. Toute utilisation de ce Message
> non conforme à sa destination, toute diffusion ou toute publication totale
> ou partielle, est interdite sauf autorisation expresse.
>
> Si vous n'êtes pas le destinataire de ce Message, il vous est interdit de
> le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou
> partie. Si vous avez reçu ce Message par erreur, merci de le supprimer de
> votre système, ainsi que toutes ses copies, et de n'en garder aucune trace
> sur quelque support que ce soit. Nous vous remercions également d'en
> avertir immédiatement l'expéditeur par retour du message.
>
> Il est impossible de garantir que les communications par messagerie
> électronique arrivent en temps utile, sont sécurisées ou dénuées de toute
> erreur ou virus.
> 
>
> This message and any attachments (the 'Message') are intended solely for
> the addressees. The information contained in this Message is confidential.
> Any use of information contained in this Message not in accord with its
> purpose, any dissemination or disclosure, either whole or partial, is
> prohibited except formal approval.
>
> If you are not the addressee, you may not copy, forward, disclose or use
> any part of it. If you have received this message in error, please delete
> it and all copies from your system and notify the sender immediately by
> return message.
>
> E-mail communication cannot be guaranteed to be timely secure, error or
> virus-free.
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> 
>
>
> Ce message et toutes les pièces jointes (ci-après le 

Re: [petsc-users] HashMap Error when populating AIJCUSPARSE matrix

[Matthew Knepley]

On Thu, Jan 18, 2024 at 9:04 AM Yesypenko, Anna 
wrote:

> Dear Petsc users/developers,
>
> I'm experiencing a bug when using petsc4py with GPU support. It may be my
> mistake in how I set up a AIJCUSPARSE matrix.
> For larger matrices, I sometimes encounter a error in assigning matrix
> values; the error is thrown in PetscHMapIJVQuerySet().
> Here is a minimum snippet that populates a sparse tridiagonal matrix.
>
> ```
> from petsc4py import PETSc
> from scipy.sparse import diags
> import numpy as np
>
> n = int(5e5);
>
> nnz = 3 * np.ones(n, dtype=np.int32); nnz[0] = nnz[-1] = 2
> A = PETSc.Mat(comm=PETSc.COMM_WORLD)
> A.createAIJ(size=[n,n],comm=PETSc.COMM_WORLD,nnz=nnz)
> A.setType('aijcusparse')
> tmp = diags([-1,2,-1],[-1,0,+1],shape=(n,n)).tocsr()
> A.setValuesCSR(tmp.indptr,tmp.indices,tmp.data)
> ### this is the line where the error is thrown.
> A.assemble()
> ```
>

I don't have scipy installed. Since the matrix is so small, can you
print tmp.indptr,tmp.indices,tmp.data when you run? It seems to be either
bad values there, or something is wrong with passing those pointers.

  Thanks,

 Matt


> The error trace is below:
> ```
> File "petsc4py/PETSc/Mat.pyx", line 2603, in
> petsc4py.PETSc.Mat.setValuesCSR
>   File "petsc4py/PETSc/petscmat.pxi", line 1039, in
> petsc4py.PETSc.matsetvalues_csr
>   File "petsc4py/PETSc/petscmat.pxi", line 1032, in
> petsc4py.PETSc.matsetvalues_ijv
> petsc4py.PETSc.Error: error code 76
> [0] MatSetValues() at
> /work/06368/annayesy/ls6/petsc/src/mat/interface/matrix.c:1497
> [0] MatSetValues_Seq_Hash() at
> /work/06368/annayesy/ls6/petsc/include/../src/mat/impls/aij/seq/seqhashmatsetvalues.h:52
> [0] PetscHMapIJVQuerySet() at
> /work/06368/annayesy/ls6/petsc/include/petsc/private/hashmapijv.h:10
> [0] Error in external library
> [0] [khash] Assertion: `ret >= 0' failed.
> ```
>
> If I run the same script a handful of times, it will run without errors
> eventually.
> Does anyone have insight on why it is behaving this way? I'm running on a
> node with 3x NVIDIA A100 PCIE 40GB.
>
> Thank you!
> Anna
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Undestanding how to increase the overlap

[Matthew Knepley]

On Thu, Jan 18, 2024 at 9:24 AM TARDIEU Nicolas via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Dear PETSc Team,
>
> I am trying to understand how to increase the overlap of a matrix.
> I wrote the attached petsc4py script where I build a simple matrix and
> play with the increaseOverlap method. Unfortunately, before and after the
> call, nothing changes in the index set. FYI, I have tried to mimic
> src/ksp/ksp/tutorials/ex82.c
>  line 72:76.
> Here is how I run the script : "mpiexec -n 2 python test_overlap.py"
>
> Could you please indicate what I am missing ?
>

Usually matrix functions like this take input in global indices. It looks
like your isovl is in local indices. Am I reading that correctly?

  Thanks,

 Matt


> Regards,
> Nicolas
> --
> Nicolas Tardieu
> Ing PhD Computational Mechanics
> EDF - R Dpt ERMES
> PARIS-SACLAY, FRANCE
>
>
> Ce message et toutes les pièces jointes (ci-après le 'Message') sont
> établis à l'intention exclusive des destinataires et les informations qui y
> figurent sont strictement confidentielles. Toute utilisation de ce Message
> non conforme à sa destination, toute diffusion ou toute publication totale
> ou partielle, est interdite sauf autorisation expresse.
>
> Si vous n'êtes pas le destinataire de ce Message, il vous est interdit de
> le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou
> partie. Si vous avez reçu ce Message par erreur, merci de le supprimer de
> votre système, ainsi que toutes ses copies, et de n'en garder aucune trace
> sur quelque support que ce soit. Nous vous remercions également d'en
> avertir immédiatement l'expéditeur par retour du message.
>
> Il est impossible de garantir que les communications par messagerie
> électronique arrivent en temps utile, sont sécurisées ou dénuées de toute
> erreur ou virus.
> 
>
> This message and any attachments (the 'Message') are intended solely for
> the addressees. The information contained in this Message is confidential.
> Any use of information contained in this Message not in accord with its
> purpose, any dissemination or disclosure, either whole or partial, is
> prohibited except formal approval.
>
> If you are not the addressee, you may not copy, forward, disclose or use
> any part of it. If you have received this message in error, please delete
> it and all copies from your system and notify the sender immediately by
> return message.
>
> E-mail communication cannot be guaranteed to be timely secure, error or
> virus-free.
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] MatAssemblyBegin freezes during MPI communication

[Matthew Knepley]

On Thu, Jan 18, 2024 at 2:47 AM 袁煕  wrote:

> Dear PETSc Experts,
>
> My FEM program works well generally, but in some specific cases with
> multiple CPUs are used, it freezes when calling MatAssemblyBegin where
> PMPI_Allreduce is called (see attached file).
>
> After some investigation, I found that it is most probably due to
>
> ・ MatSetValue is not called from all CPUs before MatAssemblyBegin
>
> For example, when 4 CPUs are used, if there are elements in CPU 0,1,2 but
> no elements in CPU 3, then all CPUs other than CPU 3 would call
> MatSetValue  function. I want to know
>
> 1. If my conjecture could be right? And If so
>

No, you do not have to call MatSetValue() from all processes.


> 2. Are there any convenient means to avoid this problem?
>

Are you calling MatAssemblyBegin() from all processes? This is necessary.

  Thanks,

 Matt


> Thanks,
> Xi YUAN, PhD Solid Mechanics
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Question about petsc4py with cuda

[Matthew Knepley]

On Mon, Jan 15, 2024 at 11:57 AM MIA via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Hi,
>
> I am trying to solve a large linear equation, which needs a GPU solver as
> comparison. I install a CUDA-enabled PETSc and petsc4py from sources using
> the release tarball. According to the test results after installation, the
> PETSc can successfully work with cuda.
>

Here is a How-To for GPUs: https://petsc.org/main/faq/#doc-faq-gpuhowto

  Thanks,

 Matt


> All my programs are written in python, so I turn to petsc4py. But I do not
> find any commands that define variables on coda device or define where the
> direct solver is executed. I check `nvidia-smi` and find my cuda does not
> work at all when executing my python script:
>
> from petsc4py import PETSc
> import numpy as np
>
> n = 1000
>
> nnz = 3 * np.ones(n, dtype=np.int32)
> nnz[0] = nnz[-1] = 2
>
> A = PETSc.Mat()
> A.createAIJ([n, n], nnz=nnz)
>
> # First set the first row
> A.setValue(0, 0, 2)
> A.setValue(0, 1, -1)
> # Now we fill the last row
> A.setValue(999, 998, -1)
> A.setValue(999, 999, 2)
>
> # And now everything else
> for index in range(1, n - 1):
> A.setValue(index, index - 1, -1)
> A.setValue(index, index, 2)
> A.setValue(index, index + 1, -1)
>
> A.assemble()
>
> indexptr, indices, data = A.getValuesCSR()
> b = A.createVecLeft()
> b.array[:] = 1
> for i in range(10):
> ksp = PETSc.KSP().create()
> ksp.setOperators(A)
> ksp.setType('preonly')
> ksp.setConvergenceHistory()
> ksp.getPC().setType('lu')
> x = A.createVecRight()
> ksp.solve(2*b, x)
> residual = A * x - 2*b
> if i % 10 == 0:
> print(f"The relative residual is: {residual.norm() / b.norm()}.”)
>
> What should I do to utlize GPU to execute the KSP task? Are there some
> settings to be modified?
>
> Looking forward to your early reply. Thanks a lot.
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] [Gmsh] Access both default sets and region names

[Matthew Knepley]

On Fri, Jan 12, 2024 at 5:26 AM Noam T.  wrote:

> Great.
>
> Thank you very much for the quick replies.
>

It has now merged to the main branch.

  Thanks,

Matt


> Noam
> On Thursday, January 11th, 2024 at 9:34 PM, Matthew Knepley <
> knep...@gmail.com> wrote:
>
> On Thu, Jan 11, 2024 at 12:53 PM Noam T.  wrote:
>
>> There could be some overlapping/redundancy between the default and the
>> user-defined groups, so perhaps that was the intended behavior. Glad to
>> hear it's possible to have access to everything.
>>
>
> Here is the MR: https://gitlab.com/petsc/petsc/-/merge_requests/7178
>
> If you build that branch, you can use -dm_plex_gmsh_use_generic to turn on
> those labels.
>
> Thanks,
>
> Matt
>
>> Thanks,
>> Noam
>> On Thursday, January 11th, 2024 at 6:31 PM, Matthew Knepley <
>> knep...@gmail.com> wrote:
>>
>> On Thu, Jan 11, 2024 at 11:59 AM Noam T. 
>> wrote:
>>
>>> Without using the flag -dm_plex_gmsh_use_regions, the DMGetNumLabels
>>> says there are 5, named celltype, depth, cell/face/vertex sets.
>>> With the flag, the labels are celltype, depth, my_vol, my_surface (using
>>> the same example as before).
>>> Am I misusing the flag somehow, and I should be able to access those of
>>> cell/face/vertex as well?
>>>
>>
>> Shoot, yes this changed after another request. Yes, we can put in a flag
>> for that. Should not take long.
>>
>> Thanks,
>>
>> Matt
>>
>>> PS: Using PETSc 3.20.3
>>>
>>> Thanks,
>>> Noam
>>> On Thursday, January 11th, 2024 at 5:28 PM, Matthew Knepley <
>>> knep...@gmail.com> wrote:
>>>
>>> On Thu, Jan 11, 2024 at 11:18 AM Noam T. via petsc-users <
>>> petsc-users@mcs.anl.gov> wrote:
>>>
>>>> Would it be feasible to have an option (e.g. new flag along the lines
>>>> of -dm_plex_gmsh_...) that allows the user to access both the default sets
>>>> (Cell / Face / Vertex) together with user-defined gorups (those under
>>>> $PhysicalNames, available when using -dm_plex_gmsh_use_regions)?
>>>>
>>>
>>> I am not sure I understand the question. When you turn on regions, it
>>> makes extra labels, but the generic labels still exist.
>>>
>>> Thanks,
>>>
>>> Matt
>>>
>>>> That is, with a *.msh file containing
>>>>
>>>> $PhysicalNames
>>>> 2
>>>> 2 100 "my_surface"
>>>> 3 200 "my_vol"
>>>>
>>>> the return of DMGetLabelName(dm, n, name) would be (order may differ)
>>>>
>>>> n = 0, name = "celltype"
>>>> n = 1, name = "depth"
>>>> n = 2, name = "Cell Sets"
>>>> n = 3, name = "my_vol"
>>>> n = 4, name = "Face Sets"
>>>> n = 5, name = "my_surface"
>>>> ...
>>>>
>>>> I poked into src/dm/impls/plex/plexgmsh.c and have managed to print all
>>>> the labels after changing a couple of variable values, so perhaps it is
>>>> doable.
>>>> The changes made are not a solution, simply naively set some variables
>>>> to skip checking for the use_regions flag, so it understandably crashes
>>>> soon after.
>>>>
>>>> Thanks, Noam
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-users] [Gmsh] Access both default sets and region names

[Matthew Knepley]

On Thu, Jan 11, 2024 at 12:53 PM Noam T.  wrote:

> There could be some overlapping/redundancy between the default and the
> user-defined groups, so perhaps that was the intended behavior. Glad to
> hear it's possible to have access to everything.
>

Here is the MR: https://gitlab.com/petsc/petsc/-/merge_requests/7178

If you build that branch, you can use -dm_plex_gmsh_use_generic to turn on
those labels.

  Thanks,

 Matt


> Thanks,
> Noam
> On Thursday, January 11th, 2024 at 6:31 PM, Matthew Knepley <
> knep...@gmail.com> wrote:
>
> On Thu, Jan 11, 2024 at 11:59 AM Noam T.  wrote:
>
>> Without using the flag -dm_plex_gmsh_use_regions, the DMGetNumLabels
>> says there are 5, named celltype, depth, cell/face/vertex sets.
>> With the flag, the labels are celltype, depth, my_vol, my_surface (using
>> the same example as before).
>> Am I misusing the flag somehow, and I should be able to access those of
>> cell/face/vertex as well?
>>
>
> Shoot, yes this changed after another request. Yes, we can put in a flag
> for that. Should not take long.
>
> Thanks,
>
> Matt
>
>> PS: Using PETSc 3.20.3
>>
>> Thanks,
>> Noam
>> On Thursday, January 11th, 2024 at 5:28 PM, Matthew Knepley <
>> knep...@gmail.com> wrote:
>>
>> On Thu, Jan 11, 2024 at 11:18 AM Noam T. via petsc-users <
>> petsc-users@mcs.anl.gov> wrote:
>>
>>> Would it be feasible to have an option (e.g. new flag along the lines of
>>> -dm_plex_gmsh_...) that allows the user to access both the default sets
>>> (Cell / Face / Vertex) together with user-defined gorups (those under
>>> $PhysicalNames, available when using -dm_plex_gmsh_use_regions)?
>>>
>>
>> I am not sure I understand the question. When you turn on regions, it
>> makes extra labels, but the generic labels still exist.
>>
>> Thanks,
>>
>> Matt
>>
>>> That is, with a *.msh file containing
>>>
>>> $PhysicalNames
>>> 2
>>> 2 100 "my_surface"
>>> 3 200 "my_vol"
>>>
>>> the return of DMGetLabelName(dm, n, name) would be (order may differ)
>>>
>>> n = 0, name = "celltype"
>>> n = 1, name = "depth"
>>> n = 2, name = "Cell Sets"
>>> n = 3, name = "my_vol"
>>> n = 4, name = "Face Sets"
>>> n = 5, name = "my_surface"
>>> ...
>>>
>>> I poked into src/dm/impls/plex/plexgmsh.c and have managed to print all
>>> the labels after changing a couple of variable values, so perhaps it is
>>> doable.
>>> The changes made are not a solution, simply naively set some variables
>>> to skip checking for the use_regions flag, so it understandably crashes
>>> soon after.
>>>
>>> Thanks, Noam
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-users] [Gmsh] Access both default sets and region names

[Matthew Knepley]

On Thu, Jan 11, 2024 at 11:59 AM Noam T.  wrote:

> Without using the flag -dm_plex_gmsh_use_regions, the DMGetNumLabels says
> there are  5, named celltype, depth, cell/face/vertex sets.
> With the flag, the labels are celltype, depth, my_vol, my_surface (using
> the same example as before).
> Am I misusing the flag somehow, and I should be able to access those of
> cell/face/vertex as well?
>

Shoot, yes this changed after another request. Yes, we can put in a flag
for that. Should not take long.

  Thanks,

 Matt


> PS: Using PETSc 3.20.3
>
> Thanks,
> Noam
> On Thursday, January 11th, 2024 at 5:28 PM, Matthew Knepley <
> knep...@gmail.com> wrote:
>
> On Thu, Jan 11, 2024 at 11:18 AM Noam T. via petsc-users <
> petsc-users@mcs.anl.gov> wrote:
>
>> Would it be feasible to have an option (e.g. new flag along the lines of
>> -dm_plex_gmsh_...) that allows the user to access both the default sets
>> (Cell / Face / Vertex) together with user-defined gorups (those under
>> $PhysicalNames, available when using -dm_plex_gmsh_use_regions)?
>>
>
> I am not sure I understand the question. When you turn on regions, it
> makes extra labels, but the generic labels still exist.
>
> Thanks,
>
> Matt
>
>> That is, with a *.msh file containing
>>
>> $PhysicalNames
>> 2
>> 2 100 "my_surface"
>> 3 200 "my_vol"
>>
>> the return of DMGetLabelName(dm, n, name) would be (order may differ)
>>
>> n = 0, name = "celltype"
>> n = 1, name = "depth"
>> n = 2, name = "Cell Sets"
>> n = 3, name = "my_vol"
>> n = 4, name = "Face Sets"
>> n = 5, name = "my_surface"
>> ...
>>
>> I poked into src/dm/impls/plex/plexgmsh.c and have managed to print all
>> the labels after changing a couple of variable values, so perhaps it is
>> doable.
>> The changes made are not a solution, simply naively set some variables to
>> skip checking for the use_regions flag, so it understandably crashes soon
>> after.
>>
>> Thanks, Noam
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-users] [Gmsh] Access both default sets and region names

[Matthew Knepley]

On Thu, Jan 11, 2024 at 11:18 AM Noam T. via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Would it be feasible to have an option (e.g. new flag along the lines of
> -dm_plex_gmsh_...) that allows the user to access both the default sets
> (Cell / Face / Vertex) together with user-defined gorups (those under
> $PhysicalNames, available when using -dm_plex_gmsh_use_regions)?
>

I am not sure I understand the question. When you turn on regions, it makes
extra labels, but the generic labels still exist.

  Thanks,

 Matt


> That is, with a *.msh file containing
>
> $PhysicalNames
> 2
> 2 100 "my_surface"
> 3 200 "my_vol"
>
> the return of DMGetLabelName(dm, n, name) would be (order may differ)
>
> n = 0, name = "celltype"
> n = 1, name = "depth"
> n = 2, name = "Cell Sets"
> n = 3, name =  "my_vol"
> n = 4, name = "Face Sets"
> n = 5, name = "my_surface"
> ...
>
> I poked into src/dm/impls/plex/plexgmsh.c and have managed to print all
> the labels after changing a couple of variable values, so perhaps it is
> doable.
> The changes made are not a solution, simply naively set some variables to
> skip checking for the use_regions flag, so it understandably crashes soon
> after.
>
> Thanks, Noam
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Matvecs and KSPSolves with multiple vectors

[Matthew Knepley]

On Thu, Dec 21, 2023 at 6:46 AM Sreeram R Venkat 
wrote:

> Ok, I think the error I'm getting has something to do with how the
> multiple solves are being done in succession. I'll try to see if there's
> anything I'm doing wrong there.
>
> One question about the -pc_type lu -ksp_type preonly method: do you know
> which parts of the solve (factorization/triangular solves) are done on host
> and which are done on device?
>

For SEQDENSE, I believe both the factorization and solve is on device. It
is hard to see, but I believe the dispatch code is here:


https://gitlab.com/petsc/petsc/-/blob/main/src/mat/impls/dense/seq/cupm/matseqdensecupm.hpp?ref_type=heads#L368

  Thanks,

 Matt


> Thanks,
> Sreeram
>
> On Sat, Dec 16, 2023 at 10:56 PM Pierre Jolivet  wrote:
>
>> Unfortunately, I am not able to reproduce such a failure with your input
>> matrix.
>> I’ve used ex79 that I linked previously and the system is properly solved.
>> $ ./ex79 -pc_type hypre -ksp_type hpddm -ksp_hpddm_type cg
>> -ksp_converged_reason -ksp_view_mat ascii::ascii_info -ksp_view_rhs
>> ascii::ascii_info
>> Linear solve converged due to CONVERGED_RTOL iterations 6
>> Mat Object: 1 MPI process
>>   type: seqaijcusparse
>>   rows=289, cols=289
>>   total: nonzeros=2401, allocated nonzeros=2401
>>   total number of mallocs used during MatSetValues calls=0
>> not using I-node routines
>> Mat Object: 1 MPI process
>>   type: seqdensecuda
>>   rows=289, cols=10
>>   total: nonzeros=2890, allocated nonzeros=2890
>>   total number of mallocs used during MatSetValues calls=0
>>
>> You mentioned in a subsequent email that you are interested in systems
>> with at most 1E6 unknowns, and up to 1E4 right-hand sides.
>> I’m not sure you can expect significant gains from using GPU for such
>> systems.
>> Probably, the fastest approach would indeed be -pc_type lu -ksp_type
>> preonly -ksp_matsolve_batch_size 100 or something, depending on the memory
>> available on your host.
>>
>> Thanks,
>> Pierre
>>
>> On 15 Dec 2023, at 9:52 PM, Sreeram R Venkat  wrote:
>>
>> Here are the ksp_view files.  I set the options
>> -ksp_error_if_not_converged to try to get the vectors that caused the
>> error. I noticed that some of the KSPMatSolves converge while others don't.
>> In the code, the solves are called as:
>>
>> input vector v --> insert data of v into a dense mat --> KSPMatSolve()
>> --> MatMatMult() --> KSPMatSolve() --> insert data of dense mat into output
>> vector w -- output w
>>
>> The operator used in the KSP is a Laplacian-like operator, and the
>> MatMatMult is with a Mass Matrix. The whole thing is supposed to be a solve
>> with a biharmonic-like operator. I can also run it with only the first
>> KSPMatSolve (i.e. just a Laplacian-like operator). In that case, the KSP
>> reportedly converges after 0 iterations (see the next line), but this
>> causes problems in other parts of the code later on.
>>
>> I saw that sometimes the first KSPMatSolve "converges" after 0 iterations
>> due to CONVERGED_RTOL. Then, the second KSPMatSolve produces a NaN/Inf. I
>> tried setting ksp_min_it, but that didn't seem to do anything.
>>
>> I'll keep trying different options and also try to get the MWE made (this
>> KSPMatSolve is pretty performance critical for us).
>>
>> Thanks for all your help,
>> Sreeram
>>
>> On Fri, Dec 15, 2023 at 1:01 AM Pierre Jolivet  wrote:
>>
>>>
>>> On 14 Dec 2023, at 11:45 PM, Sreeram R Venkat 
>>> wrote:
>>>
>>> Thanks, I will try to create a minimal reproducible example. This may
>>> take me some time though, as I need to figure out how to extract only the
>>> relevant parts (the full program this solve is used in is getting quite
>>> complex).
>>>
>>>
>>> You could just do -ksp_view_mat binary:Amat.bin -ksp_view_pmat
>>> binary:Pmat.bin -ksp_view_rhs binary:rhs.bin and send me those three files
>>> (I’m guessing your are using double-precision scalars with 32-bit PetscInt).
>>>
>>> I'll also try out some of the BoomerAMG options to see if that helps.
>>>
>>>
>>> These should work (this is where all “PCMatApply()-ready” PC are being
>>> tested):
>>> https://petsc.org/release/src/ksp/ksp/tutorials/ex79.c.html#line215
>>> You can see it’s also testing PCHYPRE + KSPHPDDM on device (but not with
>>> HIP).
>>> I’m aware the performance should not be optimal (see your comment about
>>> host/device copies), I’ve money to hire someone to work on this but: a) I
>>> need to find the correct engineer/post-doc, b) I currently don’t have good
>>> use cases (of course, I could generate a synthetic benchmark, for science).
>>> So even if you send me the three Mat, a MWE would be appreciated if the
>>> KSPMatSolve() is performance-critical for you (see point b) from above).
>>>
>>> Thanks,
>>> Pierre
>>>
>>> Thanks,
>>> Sreeram
>>>
>>> On Thu, Dec 14, 2023, 1:12 PM Pierre Jolivet  wrote:
>>>


 On 14 Dec 2023, at 8:02 PM, Sreeram R Venkat 
 wrote:

 Hello Pierre,

 Thank you for your reply. I tried out the 

Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

[Matthew Knepley]

On Wed, Dec 20, 2023 at 10:02 PM Shatanawi, Sawsan Muhammad <
sawsan.shatan...@wsu.edu> wrote:

> Hello Matthew,
>
> Thank you for your help. I am sorry that I keep coming back with my error
> messages, but I reached a point that I don't know how to fix them, and I
> don't understand them easily.
> The list of errors is getting shorter, now I am getting the attached error
> messages
>

You are overwriting memory somewhere, but we cannot see the code, and thus
cannot tell where.

You can figure this out by running in the debugger, using
-start_in_debugger, which launches a debugger window, and then 'cont' to
run until the error, and then 'where' to print the stack trace.

   Thanks,

 Matt


> Thank you again,
>
> Sawsan
> ------
> *From:* Matthew Knepley 
> *Sent:* Wednesday, December 20, 2023 6:54 PM
> *To:* Shatanawi, Sawsan Muhammad 
> *Cc:* Barry Smith ; petsc-users@mcs.anl.gov <
> petsc-users@mcs.anl.gov>
> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran
> Groundwater Flow Simulation Code
>
>
> *[EXTERNAL EMAIL]*
> On Wed, Dec 20, 2023 at 9:49 PM Shatanawi, Sawsan Muhammad via petsc-users
>  wrote:
>
> Hello Barry,
>
> Thank you a lot for your help, Now I am getting the attached error message.
>
>
> Do not destroy the PC from KSPGetPC()
>
>   THanks,
>
>  Matt
>
>
> Bests,
> Sawsan
> --
> *From:* Barry Smith 
> *Sent:* Wednesday, December 20, 2023 6:32 PM
> *To:* Shatanawi, Sawsan Muhammad 
> *Cc:* Mark Adams ; petsc-users@mcs.anl.gov <
> petsc-users@mcs.anl.gov>
> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran
> Groundwater Flow Simulation Code
>
>
> *[EXTERNAL EMAIL]*
>
> Instead of
>
> call PCCreate(PETSC_COMM_WORLD, pc, ierr)
> call PCSetType(pc, PCILU,ierr) ! Choose a preconditioner type (ILU)
> call KSPSetPC(ksp, pc,ierr)  ! Associate the preconditioner with the
> KSP solver
>
> do
>
> call KSPGetPC(ksp,pc,ierr)
> call PCSetType(pc, PCILU,ierr)
>
> Do not call KSPSetUp(). It will be taken care of automatically during the
> solve
>
>
>
> On Dec 20, 2023, at 8:52 PM, Shatanawi, Sawsan Muhammad via petsc-users <
> petsc-users@mcs.anl.gov> wrote:
>
> Hello,
> I don't think that I set preallocation values when I created the matrix,
> would you please have look at my code. It is just the petsc related part
> from my code.
> I was able to fix some of the error messages. Now I have a new set of
> error messages related to the KSP solver (attached)
>
> I appreciate your help
>
> Sawsan
> --
> *From:* Mark Adams 
> *Sent:* Wednesday, December 20, 2023 6:44 AM
> *To:* Shatanawi, Sawsan Muhammad 
> *Cc:* petsc-users@mcs.anl.gov 
> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran
> Groundwater Flow Simulation Code
>
> *[EXTERNAL EMAIL]*
> Did you set preallocation values when you created the matrix?
> Don't do that.
>
> On Wed, Dec 20, 2023 at 9:36 AM Shatanawi, Sawsan Muhammad <
> sawsan.shatan...@wsu.edu> wrote:
>
> Hello,
>
> I am trying to create a sparse matrix( which is as I believe a zero
> matrix) then adding some nonzero elements to it over a loop, then
> assembling it
>
> Get Outlook for iOS
> <https://urldefense.com/v3/__https://aka.ms/o0ukef__;!!JmPEgBY0HMszNaDT!uUJ_jeYf45gcXDGR_PeMjhU7hbd_fKcXJPn0pM9eb-YQihKNYuXMYM9x-hglsbXsCFIwNBWgHXdetHODupsOloE$>
> --
> *From:* Mark Adams 
> *Sent:* Wednesday, December 20, 2023 2:48 AM
> *To:* Shatanawi, Sawsan Muhammad 
> *Cc:* petsc-users@mcs.anl.gov 
> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran
> Groundwater Flow Simulation Code
>
> *[EXTERNAL EMAIL]*
> I am guessing that you are creating a matrix, adding to it, finalizing it
> ("assembly"), and then adding to it again, which is fine, but you are
> adding new non-zeros to the sparsity pattern.
> If this is what you want then you can tell the matrix to let you do that.
> Otherwise you have a bug.
>
> Mark
>
> On Tue, Dec 19, 2023 at 9:50 PM Shatanawi, Sawsan Muhammad via petsc-users
>  wrote:
>
> Hello everyone,
>
> I hope this email finds you well.
>
>  My Name is Sawsan Shatanawi, and I am currently working on developing a
> Fortran code for simulating groundwater flow in a 3D system. The code
> involves solving a nonlinear system, and I have created the matrix to be
> solved using the PCG solver and Picard iteration. However, when I tried
> to assign it as a PETSc matrix I started getting a lot of error messages.
>
> I am kindly 

Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

[Matthew Knepley]

On Wed, Dec 20, 2023 at 9:49 PM Shatanawi, Sawsan Muhammad via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Hello Barry,
>
> Thank you a lot for your help, Now I am getting the attached error message.
>

Do not destroy the PC from KSPGetPC()

  THanks,

 Matt


> Bests,
> Sawsan
> --
> *From:* Barry Smith 
> *Sent:* Wednesday, December 20, 2023 6:32 PM
> *To:* Shatanawi, Sawsan Muhammad 
> *Cc:* Mark Adams ; petsc-users@mcs.anl.gov <
> petsc-users@mcs.anl.gov>
> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran
> Groundwater Flow Simulation Code
>
>
> *[EXTERNAL EMAIL]*
>
> Instead of
>
> call PCCreate(PETSC_COMM_WORLD, pc, ierr)
> call PCSetType(pc, PCILU,ierr) ! Choose a preconditioner type (ILU)
> call KSPSetPC(ksp, pc,ierr)  ! Associate the preconditioner with the
> KSP solver
>
> do
>
> call KSPGetPC(ksp,pc,ierr)
> call PCSetType(pc, PCILU,ierr)
>
> Do not call KSPSetUp(). It will be taken care of automatically during the
> solve
>
>
>
> On Dec 20, 2023, at 8:52 PM, Shatanawi, Sawsan Muhammad via petsc-users <
> petsc-users@mcs.anl.gov> wrote:
>
> Hello,
> I don't think that I set preallocation values when I created the matrix,
> would you please have look at my code. It is just the petsc related part
> from my code.
> I was able to fix some of the error messages. Now I have a new set of
> error messages related to the KSP solver (attached)
>
> I appreciate your help
>
> Sawsan
> --
> *From:* Mark Adams 
> *Sent:* Wednesday, December 20, 2023 6:44 AM
> *To:* Shatanawi, Sawsan Muhammad 
> *Cc:* petsc-users@mcs.anl.gov 
> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran
> Groundwater Flow Simulation Code
>
> *[EXTERNAL EMAIL]*
> Did you set preallocation values when you created the matrix?
> Don't do that.
>
> On Wed, Dec 20, 2023 at 9:36 AM Shatanawi, Sawsan Muhammad <
> sawsan.shatan...@wsu.edu> wrote:
>
> Hello,
>
> I am trying to create a sparse matrix( which is as I believe a zero
> matrix) then adding some nonzero elements to it over a loop, then
> assembling it
>
> Get Outlook for iOS
> 
> --
> *From:* Mark Adams 
> *Sent:* Wednesday, December 20, 2023 2:48 AM
> *To:* Shatanawi, Sawsan Muhammad 
> *Cc:* petsc-users@mcs.anl.gov 
> *Subject:* Re: [petsc-users] Help with Integrating PETSc into Fortran
> Groundwater Flow Simulation Code
>
> *[EXTERNAL EMAIL]*
> I am guessing that you are creating a matrix, adding to it, finalizing it
> ("assembly"), and then adding to it again, which is fine, but you are
> adding new non-zeros to the sparsity pattern.
> If this is what you want then you can tell the matrix to let you do that.
> Otherwise you have a bug.
>
> Mark
>
> On Tue, Dec 19, 2023 at 9:50 PM Shatanawi, Sawsan Muhammad via petsc-users
>  wrote:
>
> Hello everyone,
>
> I hope this email finds you well.
>
>  My Name is Sawsan Shatanawi, and I am currently working on developing a
> Fortran code for simulating groundwater flow in a 3D system. The code
> involves solving a nonlinear system, and I have created the matrix to be
> solved using the PCG solver and Picard iteration. However, when I tried
> to assign it as a PETSc matrix I started getting a lot of error messages.
>
> I am kindly asking if someone can help me, I would be happy to share my
> code with him/her.
>
> Please find the attached file contains a list of errors I have gotten
>
> Thank you in advance for your time and assistance.
>
> Best regards,
>
>  Sawsan
>
> 
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code

[Matthew Knepley]

On Tue, Dec 19, 2023 at 9:50 PM Shatanawi, Sawsan Muhammad via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Hello everyone,
>
> I hope this email finds you well.
>
>  My Name is Sawsan Shatanawi, and I am currently working on developing a
> Fortran code for simulating groundwater flow in a 3D system. The code
> involves solving a nonlinear system, and I have created the matrix to be
> solved using the PCG solver and Picard iteration. However, when I tried
> to assign it as a PETSc matrix I started getting a lot of error messages.
>
> I am kindly asking if someone can help me, I would be happy to share my
> code with him/her.
>
> Please find the attached file contains a list of errors I have gotten
>

This error indicates that your preallocation is not sufficient for the
values you want to insert. Now in PETSc you can just remove your
preallocation, and PETSc will automatically allocate correctly.

  Thanks,

 Matt


> Thank you in advance for your time and assistance.
>
> Best regards,
>
>  Sawsan
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] [petsc-maint] DMSwarm on multiple processors

[Matthew Knepley]

On Wed, Dec 20, 2023 at 4:12 AM Joauma Marichal <
joauma.maric...@uclouvain.be> wrote:

> Hello,
>
>
>
> I used Address Sanitizer on my laptop and I have no leaks.
>
> I do have access to another machine (managed by the same people as the
> previous one) but I obtain similar errors…
>

Let me understand:

1) You have run the exact same problem on two different parallel machines,
and gotten the same error,
 meaning on the second machine, it printed

malloc_consolidate(): invalid chunk size

Is this true?

 2) You have run the exact same problem on the same number of processes on
your own machine under Address Sanitizer with no errors?

  Thanks,

 Matt


> Thanks again for your help.
>
>
>
> Best regards,
>
>
>
> Joauma
>
>
>
> *De : *Matthew Knepley 
> *Date : *mardi, 19 décembre 2023 à 14:30
> *À : *Joauma Marichal 
> *Cc : *petsc-ma...@mcs.anl.gov ,
> petsc-users@mcs.anl.gov 
> *Objet : *Re: [petsc-maint] DMSwarm on multiple processors
>
> On Tue, Dec 19, 2023 at 5:11 AM Joauma Marichal <
> joauma.maric...@uclouvain.be> wrote:
>
> Hello,
>
>
>
> I have used Address Sanitizer to check any memory errors. On my computer,
> no errors are found. Unfortunately, on the supercomputer that I am using, I
> get lots of errors… I attach my log files (running on 1 and 70 procs).
>
> Do you have any idea of what I could do?
>
>
>
> Run the same parallel configuration as you do on the supercomputer. If
> that is fine, I would suggest Address Sanitizer there. Something is
> corrupting the stack, and it appears that it is connected to that machine,
> rather than the library. Do you have access to a second parallel machine?
>
>
>
>   Thanks,
>
>
>
>  Matt
>
>
>
> Thanks a lot for your help.
>
>
>
> Best regards,
>
>
>
> Joauma
>
>
>
> *De : *Matthew Knepley 
> *Date : *lundi, 18 décembre 2023 à 12:00
> *À : *Joauma Marichal 
> *Cc : *petsc-ma...@mcs.anl.gov ,
> petsc-users@mcs.anl.gov 
> *Objet : *Re: [petsc-maint] DMSwarm on multiple processors
>
> On Mon, Dec 18, 2023 at 5:09 AM Joauma Marichal <
> joauma.maric...@uclouvain.be> wrote:
>
> Hello,
>
>
>
> Sorry for the delay. I attach the file that I obtain when running the code
> with the debug mode.
>
>
>
> Okay, we can now see where this is happening:
>
>
>
> malloc_consolidate(): invalid chunk size
> [cns263:3265170] *** Process received signal ***
> [cns263:3265170] Signal: Aborted (6)
> [cns263:3265170] Signal code:  (-6)
> [cns263:3265170] [ 0] /lib64/libc.so.6(+0x4eb20)[0x7f3bd9148b20]
> [cns263:3265170] [ 1] /lib64/libc.so.6(gsignal+0x10f)[0x7f3bd9148a9f]
> [cns263:3265170] [ 2] /lib64/libc.so.6(abort+0x127)[0x7f3bd911be05]
> [cns263:3265170] [ 3] /lib64/libc.so.6(+0x91037)[0x7f3bd918b037]
> [cns263:3265170] [ 4] /lib64/libc.so.6(+0x9819c)[0x7f3bd919219c]
> [cns263:3265170] [ 5] /lib64/libc.so.6(+0x98b68)[0x7f3bd9192b68]
> [cns263:3265170] [ 6] /lib64/libc.so.6(+0x9af18)[0x7f3bd9194f18]
> [cns263:3265170] [ 7] /lib64/libc.so.6(__libc_malloc+0x1e2)[0x7f3bd9196822]
> [cns263:3265170] [ 8] /lib64/libc.so.6(posix_memalign+0x3c)[0x7f3bd91980fc]
> [cns263:3265170] [ 9]
> /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(PetscMallocAlign+0x45)[0x7f3bda5f1625]
> [cns263:3265170] [10]
> /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(PetscMallocA+0x297)[0x7f3bda5f1b07]
> [cns263:3265170] [11]
> /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(DMCreate+0x5b)[0x7f3bdaa73c1b]
> [cns263:3265170] [12]
> /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(DMDACreate+0x9)[0x7f3bdab0a2f9]
> [cns263:3265170] [13]
> /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(DMDACreate3d+0x9a)[0x7f3bdab07dea]
> [cns263:3265170] [14] ./cobpor[0x402de8]
> [cns263:3265170] [15]
> /lib64/libc.so.6(__libc_start_main+0xf3)[0x7f3bd9134cf3]
> [cns263:3265170] [16] ./cobpor[0x40304e]
> [cns263:3265170] *** End of error message ***
>
>
>
> However, this is not great. First, the amount of memory being allocated is
> quite small, and this does not appear to be an Out of Memory error. Second,
> the error occurs in libc:
>
>
>
>   malloc_consolidate(): invalid chunk size
>
>
>
> which means something is wrong internally. I agree with this analysis (
> https://stackoverflow.com/questions/18760999/sample-example-program-to-get-the-malloc-consolidate-error)
> that says you have probably overwritten memory somewhere in your code. I
> recommend running under valgrind, or using Address Sanitizer from clang.
>
>
>
>   Thanks,
>
>
>
>  M

Re: [petsc-users] [petsc-maint] DMSwarm on multiple processors

[Matthew Knepley]

On Tue, Dec 19, 2023 at 5:11 AM Joauma Marichal <
joauma.maric...@uclouvain.be> wrote:

> Hello,
>
>
>
> I have used Address Sanitizer to check any memory errors. On my computer,
> no errors are found. Unfortunately, on the supercomputer that I am using, I
> get lots of errors… I attach my log files (running on 1 and 70 procs).
>
> Do you have any idea of what I could do?
>

Run the same parallel configuration as you do on the supercomputer. If that
is fine, I would suggest Address Sanitizer there. Something is corrupting
the stack, and it appears that it is connected to that machine, rather than
the library. Do you have access to a second parallel machine?

  Thanks,

 Matt


> Thanks a lot for your help.
>
>
>
> Best regards,
>
>
>
> Joauma
>
>
>
> *De : *Matthew Knepley 
> *Date : *lundi, 18 décembre 2023 à 12:00
> *À : *Joauma Marichal 
> *Cc : *petsc-ma...@mcs.anl.gov ,
> petsc-users@mcs.anl.gov 
> *Objet : *Re: [petsc-maint] DMSwarm on multiple processors
>
> On Mon, Dec 18, 2023 at 5:09 AM Joauma Marichal <
> joauma.maric...@uclouvain.be> wrote:
>
> Hello,
>
>
>
> Sorry for the delay. I attach the file that I obtain when running the code
> with the debug mode.
>
>
>
> Okay, we can now see where this is happening:
>
>
>
> malloc_consolidate(): invalid chunk size
> [cns263:3265170] *** Process received signal ***
> [cns263:3265170] Signal: Aborted (6)
> [cns263:3265170] Signal code:  (-6)
> [cns263:3265170] [ 0] /lib64/libc.so.6(+0x4eb20)[0x7f3bd9148b20]
> [cns263:3265170] [ 1] /lib64/libc.so.6(gsignal+0x10f)[0x7f3bd9148a9f]
> [cns263:3265170] [ 2] /lib64/libc.so.6(abort+0x127)[0x7f3bd911be05]
> [cns263:3265170] [ 3] /lib64/libc.so.6(+0x91037)[0x7f3bd918b037]
> [cns263:3265170] [ 4] /lib64/libc.so.6(+0x9819c)[0x7f3bd919219c]
> [cns263:3265170] [ 5] /lib64/libc.so.6(+0x98b68)[0x7f3bd9192b68]
> [cns263:3265170] [ 6] /lib64/libc.so.6(+0x9af18)[0x7f3bd9194f18]
> [cns263:3265170] [ 7] /lib64/libc.so.6(__libc_malloc+0x1e2)[0x7f3bd9196822]
> [cns263:3265170] [ 8] /lib64/libc.so.6(posix_memalign+0x3c)[0x7f3bd91980fc]
> [cns263:3265170] [ 9]
> /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(PetscMallocAlign+0x45)[0x7f3bda5f1625]
> [cns263:3265170] [10]
> /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(PetscMallocA+0x297)[0x7f3bda5f1b07]
> [cns263:3265170] [11]
> /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(DMCreate+0x5b)[0x7f3bdaa73c1b]
> [cns263:3265170] [12]
> /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(DMDACreate+0x9)[0x7f3bdab0a2f9]
> [cns263:3265170] [13]
> /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(DMDACreate3d+0x9a)[0x7f3bdab07dea]
> [cns263:3265170] [14] ./cobpor[0x402de8]
> [cns263:3265170] [15]
> /lib64/libc.so.6(__libc_start_main+0xf3)[0x7f3bd9134cf3]
> [cns263:3265170] [16] ./cobpor[0x40304e]
> [cns263:3265170] *** End of error message ***
>
>
>
> However, this is not great. First, the amount of memory being allocated is
> quite small, and this does not appear to be an Out of Memory error. Second,
> the error occurs in libc:
>
>
>
>   malloc_consolidate(): invalid chunk size
>
>
>
> which means something is wrong internally. I agree with this analysis (
> https://stackoverflow.com/questions/18760999/sample-example-program-to-get-the-malloc-consolidate-error)
> that says you have probably overwritten memory somewhere in your code. I
> recommend running under valgrind, or using Address Sanitizer from clang.
>
>
>
>   Thanks,
>
>
>
>  Matt
>
>
>
> Thanks for your help.
>
>
>
> Best regards,
>
>
>
> Joauma
>
>
>
> *De : *Matthew Knepley 
> *Date : *jeudi, 23 novembre 2023 à 15:32
> *À : *Joauma Marichal 
> *Cc : *petsc-ma...@mcs.anl.gov ,
> petsc-users@mcs.anl.gov 
> *Objet : *Re: [petsc-maint] DMSwarm on multiple processors
>
> On Thu, Nov 23, 2023 at 9:01 AM Joauma Marichal <
> joauma.maric...@uclouvain.be> wrote:
>
> Hello,
>
>
>
> My problem persists… Is there anything I could try?
>
>
>
> Yes. It appears to be failing from a call inside PetscSFSetUpRanks(). It
> does allocation, and the failure
>
> is in libc, and it only happens on larger examples, so I suspect some
> allocation problem. Can you rebuild with debugging and run this example?
> Then we can see if the allocation fails.
>
>
>
>   Thanks,
>
>  Matt
>
>
>
> Thanks a lot.
>
>
>
> Best regards,
>
>
>
> Joauma
>
>
>
> *De : *Matthew Knepley 
> *Date : *mercredi, 25 octobre 2023 à 14:45
> *À : *Joauma Mar

Re: [petsc-users] [petsc-maint] DMSwarm on multiple processors

[Matthew Knepley]

On Mon, Dec 18, 2023 at 5:09 AM Joauma Marichal <
joauma.maric...@uclouvain.be> wrote:

> Hello,
>
>
>
> Sorry for the delay. I attach the file that I obtain when running the code
> with the debug mode.
>

Okay, we can now see where this is happening:

malloc_consolidate(): invalid chunk size
[cns263:3265170] *** Process received signal ***
[cns263:3265170] Signal: Aborted (6)
[cns263:3265170] Signal code:  (-6)
[cns263:3265170] [ 0] /lib64/libc.so.6(+0x4eb20)[0x7f3bd9148b20]
[cns263:3265170] [ 1] /lib64/libc.so.6(gsignal+0x10f)[0x7f3bd9148a9f]
[cns263:3265170] [ 2] /lib64/libc.so.6(abort+0x127)[0x7f3bd911be05]
[cns263:3265170] [ 3] /lib64/libc.so.6(+0x91037)[0x7f3bd918b037]
[cns263:3265170] [ 4] /lib64/libc.so.6(+0x9819c)[0x7f3bd919219c]
[cns263:3265170] [ 5] /lib64/libc.so.6(+0x98b68)[0x7f3bd9192b68]
[cns263:3265170] [ 6] /lib64/libc.so.6(+0x9af18)[0x7f3bd9194f18]
[cns263:3265170] [ 7] /lib64/libc.so.6(__libc_malloc+0x1e2)[0x7f3bd9196822]
[cns263:3265170] [ 8] /lib64/libc.so.6(posix_memalign+0x3c)[0x7f3bd91980fc]
[cns263:3265170] [ 9]
/gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(PetscMallocAlign+0x45)[0x7f3bda5f1625]
[cns263:3265170] [10]
/gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(PetscMallocA+0x297)[0x7f3bda5f1b07]
[cns263:3265170] [11]
/gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(DMCreate+0x5b)[0x7f3bdaa73c1b]
[cns263:3265170] [12]
/gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(DMDACreate+0x9)[0x7f3bdab0a2f9]
[cns263:3265170] [13]
/gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(DMDACreate3d+0x9a)[0x7f3bdab07dea]
[cns263:3265170] [14] ./cobpor[0x402de8]
[cns263:3265170] [15]
/lib64/libc.so.6(__libc_start_main+0xf3)[0x7f3bd9134cf3]
[cns263:3265170] [16] ./cobpor[0x40304e]
[cns263:3265170] *** End of error message ***

However, this is not great. First, the amount of memory being allocated is
quite small, and this does not appear to be an Out of Memory error. Second,
the error occurs in libc:

  malloc_consolidate(): invalid chunk size

which means something is wrong internally. I agree with this analysis (
https://stackoverflow.com/questions/18760999/sample-example-program-to-get-the-malloc-consolidate-error)
that says you have probably overwritten memory somewhere in your code. I
recommend running under valgrind, or using Address Sanitizer from clang.

  Thanks,

 Matt

Thanks for your help.
>
>
>
> Best regards,
>
>
>
> Joauma
>
>
>
> *De : *Matthew Knepley 
> *Date : *jeudi, 23 novembre 2023 à 15:32
> *À : *Joauma Marichal 
> *Cc : *petsc-ma...@mcs.anl.gov ,
> petsc-users@mcs.anl.gov 
> *Objet : *Re: [petsc-maint] DMSwarm on multiple processors
>
> On Thu, Nov 23, 2023 at 9:01 AM Joauma Marichal <
> joauma.maric...@uclouvain.be> wrote:
>
> Hello,
>
>
>
> My problem persists… Is there anything I could try?
>
>
>
> Yes. It appears to be failing from a call inside PetscSFSetUpRanks(). It
> does allocation, and the failure
>
> is in libc, and it only happens on larger examples, so I suspect some
> allocation problem. Can you rebuild with debugging and run this example?
> Then we can see if the allocation fails.
>
>
>
>   Thanks,
>
>  Matt
>
>
>
> Thanks a lot.
>
>
>
> Best regards,
>
>
>
> Joauma
>
>
>
> *De : *Matthew Knepley 
> *Date : *mercredi, 25 octobre 2023 à 14:45
> *À : *Joauma Marichal 
> *Cc : *petsc-ma...@mcs.anl.gov ,
> petsc-users@mcs.anl.gov 
> *Objet : *Re: [petsc-maint] DMSwarm on multiple processors
>
> On Wed, Oct 25, 2023 at 8:32 AM Joauma Marichal via petsc-maint <
> petsc-ma...@mcs.anl.gov> wrote:
>
> Hello,
>
>
>
> I am using the DMSwarm library in some Eulerian-Lagrangian approach to
> have vapor bubbles in water.
>
> I have obtained nice results recently and wanted to perform bigger
> simulations. Unfortunately, when I increase the number of processors used
> to run the simulation, I get the following error:
>
>
>
> free(): invalid size
>
> [cns136:590327] *** Process received signal ***
>
> [cns136:590327] Signal: Aborted (6)
>
> [cns136:590327] Signal code:  (-6)
>
> [cns136:590327] [ 0] /lib64/libc.so.6(+0x4eb20)[0x7f56cd4c9b20]
>
> [cns136:590327] [ 1] /lib64/libc.so.6(gsignal+0x10f)[0x7f56cd4c9a9f]
>
> [cns136:590327] [ 2] /lib64/libc.so.6(abort+0x127)[0x7f56cd49ce05]
>
> [cns136:590327] [ 3] /lib64/libc.so.6(+0x91037)[0x7f56cd50c037]
>
> [cns136:590327] [ 4] /lib64/libc.so.6(+0x9819c)[0x7f56cd51319c]
>
> [cns136:590327] [ 5] /lib64/libc.so.6(+0x99aac)[0x7f56cd514aac]
>
> [cns136:590327] [ 6]
> /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(PetscSFSetUpRanks+0x4c4)[0x7f56cea71

Re: [petsc-users] Call to DMSetMatrixPreallocateSkip not changing allocation behavior

[Matthew Knepley]

On Thu, Dec 14, 2023 at 2:06 PM Fackler, Philip via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> I'm using the following sequence of functions related to the Jacobian
> matrix:
>
> DMDACreate1d(..., );
> DMSetFromOptions(da);
> DMSetUp(da);
> DMSetMatType(da, MATAIJKOKKOS);
> DMSetMatrixPreallocateSkip(da, PETSC_TRUE);
> Mat J;
> DMCreateMatrix(da, );
> MatSetPreallocationCOO(J, ...);
>
> I recently added the call to DMSetMatrixPreallocateSkip, hoping the
> allocation would be delayed to MatSetPreallocationCOO, and that it would
> require less memory. The documentation
>  says
> that the data structures will not be preallocated.
>

You are completely correct. DMDA is just ignoring this flag. We will fix it.

  Thanks for catching this.

 Matt


> The following data from heaptrack shows that the allocation is still
> happening in the call to DMCreateMatrix.
>
>
> Can someone help me understand this?
>
> Thanks,
>
>
> *Philip Fackler *
> Research Software Engineer, Application Engineering Group
> Advanced Computing Systems Research Section
> Computer Science and Mathematics Division
> *Oak Ridge National Laboratory*
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] 捕获

[Matthew Knepley]

On Thu, Dec 14, 2023 at 1:27 AM 291--- via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Dear SLEPc Developers,
>
> I a am student from Tongji University. Recently I am trying to write a c++
> program for matrix solving, which requires importing the PETSc library that
> you have developed. However a lot of errors occur in the cpp file when I
> use #include  directly. I also try to use extern "C" but it
> gives me the error in the picture below. Is there a good way to use the
> PETSc library in a c++ program? (I compiled using cmake and my compiler is
> g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)).
>
> My cmakelists.txt is:
>
> cmake_minimum_required(VERSION 3.1.0)
>
> set(CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE)
>
> set(PETSC $ENV{PETSC_DIR}/$ENV{PETSC_ARCH})
> set(SLEPC $ENV{SLEPC_DIR}/$ENV{PETSC_ARCH})
> set(ENV{PKG_CONFIG_PATH} ${PETSC}/lib/pkgconfig:${SLEPC}/lib/pkgconfig)
>
> set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11")
> set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -std=c99")
>
> project(test)
>
> add_executable(${PROJECT_NAME} eigen_test2.cpp)
> find_package(PkgConfig REQUIRED)
>
> pkg_search_module(PETSc REQUIRED IMPORTED_TARGET PETSc)
> target_link_libraries(${PROJECT_NAME} PkgConfig::PETSc)
>
> The testing code is:eigen_test2.cpp
>

First, get rid of the "extern C" in front of the headers.

  Thanks,

 Matt


> extern "C"{
>//#include 
>#include 
>#include 
>#include 
>#include 
> }
>
> int main(int argc,char **argv)
> {
>return 0;
> }
>
>
>
> Best regards
>
> Weijie Xu
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] DMSTAG Gathering Vector on single process

[Matthew Knepley]

On Wed, Dec 13, 2023 at 11:22 AM Colton Bryant <
coltonbryant2...@u.northwestern.edu> wrote:

> Hi,
>
> Thanks for the help last week. The suggestions made the implementation I
> had much cleaner. I had one follow up question. Is there a way to sort of
> undo this operation? I know the vec scatter can be done backwards to
> distribute the arrays but I didn't see an easy way to migrate the DMDA
> vectors back into the DMStag object.
>

It is not there. However, writing it would be straightforward. I would

  1) Expose DMStagMigrateVecDMDA(), which is not currently public

  2) Add a ScatterMode argument

  3) Put in code that calls DMStagSetValuesStencil(), rather than
GetValuesStencil()

We would be happy to take on MR on this, and could help in the review.

  Thanks,

 MAtt


> Thanks for any advice.
>
> -Colton
>
> On Wed, Dec 6, 2023 at 8:18 PM Barry Smith  wrote:
>
>>
>>
>> On Dec 6, 2023, at 8:35 PM, Matthew Knepley  wrote:
>>
>> On Wed, Dec 6, 2023 at 8:10 PM Barry Smith  wrote:
>>
>>>
>>>   Depending on the serial library you may not need to split the vector
>>> into DMDA vectors with DMStagVecSplitToDMDA() for each component. Just
>>> global to natural and scatter to zero on the full vector, now the full
>>> vector is on the first rank and you can access what you need in that one
>>> vector if possible.
>>>
>>
>> Does DMStag have a GlobalToNatural?
>>
>>
>>Good point, it does not appear to have such a thing, though it could.
>>
>> Also, the serial code would have to have identical interleaving.
>>
>>   Thanks,
>>
>>  Matt
>>
>>> On Dec 6, 2023, at 6:37 PM, Colton Bryant <
>>> coltonbryant2...@u.northwestern.edu> wrote:
>>>
>>> Ah excellent! I was not aware of the ability to preallocate the objects
>>> and migrate them each time.
>>>
>>> Thanks!
>>> -Colton
>>>
>>> On Wed, Dec 6, 2023 at 5:18 PM Matthew Knepley 
>>> wrote:
>>>
>>>> On Wed, Dec 6, 2023 at 5:54 PM Colton Bryant <
>>>> coltonbryant2...@u.northwestern.edu> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I am working on a code in which a DMSTAG object is used to solve a
>>>>> fluid flow problem and I need to gather this flow data on a single process
>>>>> to interact with an existing (serial) library at each timestep of my
>>>>> simulation. After looking around the solution I've tried is:
>>>>>
>>>>> -use DMStagVecSplitToDMDA to extract vectors of each component of the
>>>>> flow
>>>>> -use DMDACreateNaturalVector and DMDAGlobalToNatural to get the
>>>>> components naturally ordered
>>>>> -use VecScatterCreateToZero to set up and then do the scatter to
>>>>> gather on the single process
>>>>>
>>>>> Unless I'm misunderstanding something this method results in a lot of
>>>>> memory allocation/freeing happening at each step of the evolution and I 
>>>>> was
>>>>> wondering if there is a way to directly perform such a scatter from the
>>>>> DMSTAG object without splitting as I'm doing here.
>>>>>
>>>>
>>>> 1) You can see here:
>>>>
>>>>
>>>> https://petsc.org/main/src/dm/impls/stag/stagda.c.html#DMStagVecSplitToDMDA
>>>>
>>>> that this function is small. You can do the DMDA creation manually, and
>>>> then just call DMStagMigrateVecDMDA() each time, which will not create
>>>> anything.
>>>>
>>>> 2) You can create the natural vector upfront, and just scatter each
>>>> time.
>>>>
>>>> 3) You can create the serial vector upfront, and just scatter each time.
>>>>
>>>> This is some data movement. You can compress the g2n and 2zero scatters
>>>> using
>>>>
>>>>   https://petsc.org/main/manualpages/PetscSF/PetscSFCompose/
>>>>
>>>> as an optimization.
>>>>
>>>>   Thanks,
>>>>
>>>>  Matt
>>>>
>>>>
>>>>> Any advice would be much appreciated!
>>>>>
>>>>> Best,
>>>>> Colton Bryant
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>>
>>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-users] Domain decomposition in PETSc for Molecular Dynamics

[Matthew Knepley]

On Tue, Dec 12, 2023 at 7:28 PM MIGUEL MOLINOS PEREZ  wrote:

> I meant the list of atoms which lies inside of a sphere of radius R_cutoff
> centered at the mean position of a given atom.
>

Okay, this is possible in parallel, but would require hard work and I have
not done it yet, although I think all the tools are coded.

In serial, there are at least two ways to do it. First, you can use a k-d
tree implementation, since they usually have the radius query. I have not
put one of these in, because I did not like any implementation, but
Underworld and Firedrake have and it works fine. Second, you can choose a
grid size of R_cutoff for the background grid, and then check neighbors.
This is probably how I would start.

  Thanks,

 Matt


> Best,
> Miguel
>
> On 13 Dec 2023, at 01:14, Matthew Knepley  wrote:
>
> 
> On Tue, Dec 12, 2023 at 2:36 PM MIGUEL MOLINOS PEREZ 
> wrote:
>
>> Dear Matthew and Mark,
>>
>> Thank you for four useful guidance.  I have taken as a starting point the
>> example in "dm/tutorials/swarm_ex3.c" to build a first approximation for
>> domain decomposition in my molecular dynamics code (diffusive molecular
>> dynamic to be more precise :-) ). And I must say that I am very happy with
>> the result. However, in my journey integrating domain decomposition into my
>> code, I am facing some new (and expected) problems.  The first is in the
>> implementation of the nearest neighbor algorithm (list of atoms closest to
>> each atom).
>>
>
> Can you help me understand this? For a given atom, there should be a
> single "closest" atom (barring
> degeneracies in distance). What do you mean by the list of closest atoms?
>
>   Thanks,
>
>  Matt
>
>
>> My current approach to the problem is a brute force algorithm (double
>> loop through the list of atoms and calculate the distance). However, it
>> seems that if I call the "neighbours" function after the "DMSwarmMigrate"
>> function the search algorithm does not work correctly. My thoughts / hints
>> are:
>>
>>- The two nested for loops should be done on the global indexing of
>>the atoms instead of the local one (I don't know how to get this number).
>>- If I print the mean position of atom #0 (for example) each range
>>prints a different value of the average position. One of them is the
>>correct position corresponding to site #0, the others are different (but
>>identically labeled) atomic sites. Which means that the site_i index is 
>> not
>>bijective.
>>
>>
>> I believe that solving this problem will increase my understanding of the
>> domain decomposition approach and may allow me to fix the remaining parts
>> of my code.
>>
>> Any additional comments are greatly appreciated. For instance, I will be
>> happy to be pointed to any piece of code (petsc examples for example) with
>> solves a similar problem in order to self-learn learn by example.
>>
>> Many thanks in advance.
>>
>> Best,
>> Miguel
>>
>> This is the piece of code (simplified) which computes the list of
>> neighbours for each atomic site. DMD is a structure which contains the
>> atomistic information (DMSWARM), and the background mesh and bounding
>> cell (DMDA and DMShell)
>>
>> int neighbours(DMD* Simulation) {
>>
>> PetscFunctionBegin;
>> PetscCallMPI(MPI_Comm_rank(PETSC_COMM_WORLD, ));
>>
>> PetscCall(DMSwarmGetLocalSize(Simulation->atomistic_data, _atoms_local
>> ));
>>
>> //! Get array with the mean position of the atoms
>> DMSwarmGetField(Simulation->atomistic_data, DMSwarmPICField_coor, &
>> blocksize, NULL,
>> (void**)_q_ptr);
>> Eigen::Map mean_q(mean_q_ptr, n_atoms_local, dim);
>>
>> int* neigh = Simulation->neigh;
>> int* numneigh = Simulation->numneigh;
>>
>> for (unsigned int site_i = 0; site_i < n_atoms_local; site_i++) {
>>
>> //! Get mean position of site i
>> Eigen::Vector3d mean_q_i = mean_q.block<1, 3>(site_i, 0);
>>
>> //! Search neighbourhs in the main cell (+ periodic cells)
>> for (unsigned site_j = 0; site_j < n_atoms_local; site_j++) {
>> if (site_i != site_j) {
>> //! Get mean position of site j in the periodic box
>> Eigen::Vector3d mean_q_j = mean_q.block<1, 3>(site_j, 0);
>>
>> //! Check is site j is the neibourhood of the site i
>> double norm_r_ij = (mean_q_i - mean_q_j).norm();
>> if ((norm_r_ij <= r_cutoff_ADP) && (numneigh[site_i] < maxneigh)) {
>> neigh[site_i * maxneigh + numneigh[site_i]] = site_j;
>

Re: [petsc-users] difference in Face Sets in latest petsc release

[Matthew Knepley]

On Tue, Dec 12, 2023 at 7:13 PM Sharan Roongta  wrote:

> Thanks for the clarification.
> However, would it be more consistent to differentiate between 0D (vertex
> sets), 1D (edge sets), 2d (faces) and 3D (cell sets)?
> If I want to now apply boundary condition on a face with tag 1, it would
> contain the 4 edges making up that face and an additional edge with the
> same physical tag?
> Basically, I can’t differentiate between the two entities
>

When we do this in PyLith, we only tag the things we want tagged. In this
mesh, absolutely everything is tagged, and with overlapping tag ranges.
That seems counterproductive.

  Thanks,

 Matt


> Thanks,
> Sharan
>
> *Group - Theory & Simulation*
> *Department of Microstructure Physics & Alloy Design*
>
>
>
> * From: * Matthew Knepley 
> * To: * Sharan Roongta 
> * Cc: * "petsc-users@mcs.anl.gov" 
> * Sent: * 13/12/2023 12:51 AM
> * Subject: * Re: [petsc-users] difference in Face Sets in latest petsc
> release
>
> On Tue, Dec 12, 2023 at 5:50 PM Sharan Roongta  wrote:
>
> Hello,
>
> I see discrepancy in the size/value of the 'Face Sets' printed in the
> current release v3.20.2 , and v3.18.6
>
> Attached is the .msh file
>
> -dm_view with v3.18.6
> DM Object: Generated Mesh 1 MPI process
>   type: plex
> Generated Mesh in 3 dimensions:
>   Number of 0-cells per rank: 14
>   Number of 1-cells per rank: 49
>   Number of 2-cells per rank: 60
>   Number of 3-cells per rank: 24
> Labels:
>   celltype: 4 strata with value/size (0 (14), 6 (24), 3 (60), 1 (49))
>   depth: 4 strata with value/size (0 (14), 1 (49), 2 (60), 3 (24))
>   Cell Sets: 1 strata with value/size (1 (24))
>   Face Sets: 5 strata with value/size (1 (4), 2 (4), 3 (4), 4 (4), 5 (4))
>
>
> -dm_view with the current release (commit 4b9a870af96)
>
> DM Object: Generated Mesh 1 MPI process
>   type: plex
> Generated Mesh in 3 dimensions:
>   Number of 0-cells per rank: 14
>   Number of 1-cells per rank: 49
>   Number of 2-cells per rank: 60
>   Number of 3-cells per rank: 24
> Labels:
>   celltype: 4 strata with value/size (0 (14), 6 (24), 3 (60), 1 (49))
>   depth: 4 strata with value/size (0 (14), 1 (49), 2 (60), 3 (24))
>   Cell Sets: 1 strata with value/size (1 (24))
>   Face Sets: 12 strata with value/size (1 (5), 2 (5), 3 (5), 4 (5), 5 (5),
> 6 (1), 7 (1), 8 (1), 9 (1), 10 (1), 11 (1), 12 (1))
>
> I believe the older version printed the correct thing? Has something
> changed in the interpretation of Face Sets?
>
>
> Yes. In the older version, I was only labeling cells, faces, and vertices.
> There were complaints, so I put in the edge labels. If you check, all the
> additional labels are on edges, and checking your .msh file, those edges
> clearly have those labels.
>
>   Thanks,
>
>  Matt
>
>
> Thanks,
> Sharan
>
> *Group - Theory & Simulation*
> *Department of Microstructure Physics & Alloy Design*
>
>
> --
> -
> Stay up to date and follow us on LinkedIn, Twitter and YouTube.
>
> Max-Planck-Institut für Eisenforschung GmbH
> Max-Planck-Straße 1
> D-40237 Düsseldorf
>
> Handelsregister B 2533
> Amtsgericht Düsseldorf
>
> Geschäftsführung
> Prof. Dr. Gerhard Dehm
> Prof. Dr. Jörg Neugebauer
> Prof. Dr. Dierk Raabe
> Dr. Kai de Weldige
>
> Ust.-Id.-Nr.: DE 11 93 58 514
> Steuernummer: 105 5891 1000
>
>
> Please consider that invitations and e-mails of our institute are
> only valid if they end with …@mpie.de.
> If you are not sure of the validity please contact r...@mpie.de
>
> Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails
> aus unserem Haus nur mit der Endung …@mpie.de gültig sind.
> In Zweifelsfällen wenden Sie sich bitte an r...@mpie.de
> -
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
>
> --
> -
> Stay up to date and follow us on LinkedIn, Twitter and YouTube.
>
> Max-Planck-Institut für Eisenforschung GmbH
> Max-Planck-Straße 1
> D-40237 Düsseldorf
>
> Handelsregister B 2533
> Amtsgericht Düsseldorf
>
> Geschäftsführung
> Prof. Dr. Gerhard Dehm
> Prof. Dr. Jörg Neugebauer
> Prof. Dr. Dierk Raabe
> Dr. Kai de Weldige
>
> Ust.-Id.-Nr.: DE 11 93 58 514
> Steuernummer: 10

Re: [petsc-users] difference in Face Sets in latest petsc release

[Matthew Knepley]

On Tue, Dec 12, 2023 at 5:50 PM Sharan Roongta  wrote:

> Hello,
>
> I see discrepancy in the size/value of the 'Face Sets' printed in the
> current release v3.20.2 , and v3.18.6
>
> Attached is the .msh file
>
> -dm_view with v3.18.6
> DM Object: Generated Mesh 1 MPI process
>   type: plex
> Generated Mesh in 3 dimensions:
>   Number of 0-cells per rank: 14
>   Number of 1-cells per rank: 49
>   Number of 2-cells per rank: 60
>   Number of 3-cells per rank: 24
> Labels:
>   celltype: 4 strata with value/size (0 (14), 6 (24), 3 (60), 1 (49))
>   depth: 4 strata with value/size (0 (14), 1 (49), 2 (60), 3 (24))
>   Cell Sets: 1 strata with value/size (1 (24))
>   Face Sets: 5 strata with value/size (1 (4), 2 (4), 3 (4), 4 (4), 5 (4))
>
>
> -dm_view with the current release (commit 4b9a870af96)
>
> DM Object: Generated Mesh 1 MPI process
>   type: plex
> Generated Mesh in 3 dimensions:
>   Number of 0-cells per rank: 14
>   Number of 1-cells per rank: 49
>   Number of 2-cells per rank: 60
>   Number of 3-cells per rank: 24
> Labels:
>   celltype: 4 strata with value/size (0 (14), 6 (24), 3 (60), 1 (49))
>   depth: 4 strata with value/size (0 (14), 1 (49), 2 (60), 3 (24))
>   Cell Sets: 1 strata with value/size (1 (24))
>   Face Sets: 12 strata with value/size (1 (5), 2 (5), 3 (5), 4 (5), 5 (5),
> 6 (1), 7 (1), 8 (1), 9 (1), 10 (1), 11 (1), 12 (1))
>
> I believe the older version printed the correct thing? Has something
> changed in the interpretation of Face Sets?
>

Yes. In the older version, I was only labeling cells, faces, and vertices.
There were complaints, so I put in the edge labels. If you check, all the
additional labels are on edges, and checking your .msh file, those edges
clearly have those labels.

  Thanks,

 Matt


> Thanks,
> Sharan
>
> *Group - Theory & Simulation*
> *Department of Microstructure Physics & Alloy Design*
>
>
> --
> -
> Stay up to date and follow us on LinkedIn, Twitter and YouTube.
>
> Max-Planck-Institut für Eisenforschung GmbH
> Max-Planck-Straße 1
> D-40237 Düsseldorf
>
> Handelsregister B 2533
> Amtsgericht Düsseldorf
>
> Geschäftsführung
> Prof. Dr. Gerhard Dehm
> Prof. Dr. Jörg Neugebauer
> Prof. Dr. Dierk Raabe
> Dr. Kai de Weldige
>
> Ust.-Id.-Nr.: DE 11 93 58 514
> Steuernummer: 105 5891 1000
>
>
> Please consider that invitations and e-mails of our institute are
> only valid if they end with …@mpie.de.
> If you are not sure of the validity please contact r...@mpie.de
>
> Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails
> aus unserem Haus nur mit der Endung …@mpie.de gültig sind.
> In Zweifelsfällen wenden Sie sich bitte an r...@mpie.de
> -
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] DMPlex "Could not find orientation for quadrilateral"

[Matthew Knepley]

On Tue, Dec 12, 2023 at 12:22 PM onur.notonur via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Hi,
>
> I hope this email finds you well. I am currently working on importing an
> OpenFOAM PolyMesh into DMPlex, and I've encountered an issue. The PolyMesh
> format includes face owner cells/neighbor cells and face-to-vertex
> connectivity. I was using the "DMPlexCreateFromCellListPetsc()" function,
> which required cell-to-vertex connectivity. However, when attempting to
> create the cell connectivity using an edge loop [p_0, p_1, ..., p_7] (p_n
> and p_(n+1) are valid edges in my mesh), I encountered an error stating,
> "Could not find orientation for quadrilateral."
>
> (Actually at first, I generated the connectivity list by simply creating a
> cell-to-face list and then using that to create a cell-to-vertex list.
> (just map over the list and remove duplicates) This created a DMPlex
> successfully, however, resulted in a mesh that was incorrect when looking
> with ParaView. I think that was because of I stated wrong edge loop to
> create cells)
>
> I understand that I may need to follow a different format for
> connectivity, but I'm not sure what that format is. My current mesh is
> hexahedral, consisting of 8 corner elements(if important). I would
> appreciate any guidance on a general approach to address this issue.
>
Can you start by giving the PolyMesh format, or some URL with it documented?

  Thanks,

Matt

> Thank you for your time and assistance.
> Best,
> Onur
> Sent with Proton Mail secure email.
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Applying Natural Boundary Conditions using PETSc FEM Technology

[Matthew Knepley]

On Tue, Dec 12, 2023 at 12:23 PM Brandon Denton via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Good Afternoon,
>
> I am currently working on an Inviscid Navier-Stokes problem and would like
> to apply DM_BC_NATURAL boundary conditions to my domain. Looking through
> the example files on petsc.org, I noticed that in almost all cases there
> are the following series of calls.
>
> PetscCall(DMAddBoundary(dm, DM_BC_NATURAL, "wall", label, 1, , 0, 0,
> NULL, NULL, NULL, user, ));
> PetscCall(PetscDSGetBoundary(ds, bd, , NULL, NULL, NULL, NULL, NULL,
> NULL, NULL, NULL, NULL, NULL, NULL));
> PetscCall(PetscWeakFormSetIndexBdResidual(wf, label, id, 0, 0, 0, f0_bd_u,
> 0, NULL));
>
> Is this the standard way of applying Natural boundary conditions in PETSc
> for FEM?
>

Yes. The problem is that AddBoundary was designed just to deliver boundary
values, but inhomogeneous Neumann conditions really want weak forms, and
the weak form interface came later. It is a little clunky.


> Also, I noticed in the signature for the  f0_bd_u function, there is a
> const PetscReal n[] array. What is this array and what information does it
> hold. Is it the normal vector at the point?
>

That is the normal at the evaluation point.

  Thanks,

 Matt


> static void f0_bd_u(PetscInt dim, PetscInt Nf, PetscInt NfAux, const
> PetscInt uOff[], const PetscInt uOff_x[], const PetscScalar u[], const
> PetscScalar u_t[], const PetscScalar u_x[], const PetscInt aOff[], const
> PetscInt aOff_x[], const PetscScalar a[], const PetscScalar a_t[], const
> PetscScalar a_x[], PetscReal t, const PetscReal x[], const PetscReal n[],
> PetscInt numConstants, const PetscScalar constants[], PetscScalar f0[])
>
> Thank you in advance for your time.
> Brandon
>
>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Fortran Interface

[Matthew Knepley]

On Tue, Dec 12, 2023 at 11:27 AM Palmer, Bruce J via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Does documentation for the PETSc fortran interface still exist? I looked
> at the web pages for 3.20 (petsc.org/release) but if you go under the tab
> C/Fortran API, only descriptions for the C interface are there.
>

I think after the most recent changes, the interface was supposed to be
very close to C, so we just document the differences on specific pages, and
put the general stuff here:

  https://petsc.org/release/manual/fortran/

   Thanks,

 Matt


> Bruce Palmer
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Question on output vector in vtk file

[Matthew Knepley]

On Mon, Dec 11, 2023 at 4:32 AM Gong Yujie 
wrote:

> Dear PETSc developers,
>
> I have a DMPlex DM with 1 field 1dof. I'd like to output a vector with
> block size equals to 3. It seems that there is no response using command
> line option or using some code about PetscViewer.
>

I am not sure how we can do this. If you only have 1 dof per cell (I
assume), how can we have a blocksize of 3?

  Thanks,

 Matt


> The DM is generated with (note that I'm not using PetscFE for
> discretization, just for allocate dof.)
>
> *PetscCall(DMPlexCreateExodusFromFile(PETSC_COMM_WORLD,"tube.exo",interpolate,));*
>
> *PetscCall(PetscFECreateLagrange(PETSC_COMM_SELF,dim,1,PETSC_TRUE,1,PETSC_DETERMINE,));*
> *PetscCall(PetscObjectSetName((PetscObject)fe,"potential_field"));*
> *PetscCall(DMSetField(dm,0,NULL,(PetscObject)fe));*
> *PetscCall(DMPlexDistribute(dm,0,,));*
>
> The Vector is created using
> *PetscCall(DMCreateGlobalVector(dm,_1));*
> *PetscCall(VecSetLocalToGlobalMapping(phi_1,Itog));*
> *PetscCall(VecGetLocalSize(phi_1,_local_size_test));*
> *PetscCall(VecCreateMPI(PETSC_COMM_WORLD, vec_local_size_test*3,
> PETSC_DETERMINE, _grad_psi));*
> *PetscCall(VecSetBlockSize(u_grad_psi, 3));*
> *PetscCall(VecSetLocalToGlobalMapping(u_grad_psi,Itog));*
>
> The output command line option is just --vec_view vtk:test.vtk. The PETSc
> version I'm using is 3.19.5.
>
> Could you please give me some advice?
>
> Best Regards,
> Yujie
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Bug report VecNorm

[Matthew Knepley]

We already merged the fix.

  Thanks,

Matt

On Mon, Dec 11, 2023 at 6:00 AM Barry Smith  wrote:

>
>I don't fully understand your code and what it is trying to
> demonstrate, but VecGetArrayWrite is Logically Collective. Having
>
>   if(rank == 0)
>   {
> PetscCall(VecGetArrayWrite(vec, ));
> PetscCall(VecRestoreArrayWrite(vec, ));
>   }
>
> is not allowed.  The reason is that VecRestoreArrayWrite() changes the
> PetscObjectState of the vector, and this state must be changed consistently
> across all MPI processes that share the vector.
>
>
>
> On Dec 10, 2023, at 2:40 AM, Stephan Köhler <
> stephan.koeh...@math.tu-freiberg.de> wrote:
>
> Dear PETSc/Tao team,
>
> there is a bug in the voector interface:  In the function
> VecNorm, see, eg.
> https://petsc.org/release/src/vec/vec/interface/rvector.c.html#VecNorm
> line 197 the check for consistency in line 214 is done on the wrong
> communicator.  The  communicator should be PETSC_COMM_SELF.
> Otherwise the program may hang when PetscCheck is executed.
>
> Please find a minimal example attached.
>
>
> Kind regards,
> Stephan Köhler
>
> --
> Stephan Köhler
> TU Bergakademie Freiberg
> Institut für numerische Mathematik und Optimierung
>
> Akademiestraße 6
> 09599 Freiberg
> Gebäudeteil Mittelbau, Zimmer 2.07
>
> Telefon: +49 (0)3731 39-3188 (Büro)
>
> 
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Bug report VecNorm

[Matthew Knepley]

On Sun, Dec 10, 2023 at 12:57 PM Matthew Knepley  wrote:

> On Sun, Dec 10, 2023 at 12:54 PM Matthew Knepley 
> wrote:
>
>> On Sun, Dec 10, 2023 at 2:41 AM Stephan Köhler <
>> stephan.koeh...@math.tu-freiberg.de> wrote:
>>
>>> Dear PETSc/Tao team,
>>>
>>> there is a bug in the voector interface:  In the function
>>> VecNorm, see, eg.
>>> https://petsc.org/release/src/vec/vec/interface/rvector.c.html#VecNorm
>>> line 197 the check for consistency in line 214 is done on the wrong
>>> communicator.  The  communicator should be PETSC_COMM_SELF.
>>> Otherwise the program may hang when PetscCheck is executed.
>>>
>>> Please find a minimal example attached.
>>>
>>
>> This is entirely right. I will fix it.
>>
>
> Here is the MR.
>

https://gitlab.com/petsc/petsc/-/merge_requests/7102

  Thanks,

Matt


>   Thanks,
>
> Matt
>
>
>>   Thanks,
>>
>>  Matt
>>
>>
>>>
>>>
>>> Kind regards,
>>> Stephan Köhler
>>>
>>> --
>>> Stephan Köhler
>>> TU Bergakademie Freiberg
>>> Institut für numerische Mathematik und Optimierung
>>>
>>> Akademiestraße 6
>>> 09599 Freiberg
>>> Gebäudeteil Mittelbau, Zimmer 2.07
>>>
>>> Telefon: +49 (0)3731 39-3188 (Büro)
>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-users] Bug report VecNorm

[Matthew Knepley]

On Sun, Dec 10, 2023 at 12:54 PM Matthew Knepley  wrote:

> On Sun, Dec 10, 2023 at 2:41 AM Stephan Köhler <
> stephan.koeh...@math.tu-freiberg.de> wrote:
>
>> Dear PETSc/Tao team,
>>
>> there is a bug in the voector interface:  In the function
>> VecNorm, see, eg.
>> https://petsc.org/release/src/vec/vec/interface/rvector.c.html#VecNorm
>> line 197 the check for consistency in line 214 is done on the wrong
>> communicator.  The  communicator should be PETSC_COMM_SELF.
>> Otherwise the program may hang when PetscCheck is executed.
>>
>> Please find a minimal example attached.
>>
>
> This is entirely right. I will fix it.
>

Here is the MR.

  Thanks,

Matt


>   Thanks,
>
>  Matt
>
>
>>
>>
>> Kind regards,
>> Stephan Köhler
>>
>> --
>> Stephan Köhler
>> TU Bergakademie Freiberg
>> Institut für numerische Mathematik und Optimierung
>>
>> Akademiestraße 6
>> 09599 Freiberg
>> Gebäudeteil Mittelbau, Zimmer 2.07
>>
>> Telefon: +49 (0)3731 39-3188 (Büro)
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>