Re: [petsc-users] Copying PETSc Objects Across MPI Communicators

[Jed Brown]

You can place it in a parallel Mat (that has rows or columns on only one rank 
or a subset of ranks) and then MatCreateSubMatrix with all new rows/columns on 
a different rank or subset of ranks.

That said, you usually have a function that assembles the matrix and you can 
just call that on the new communicator.

Damyn Chipman  writes:

> Hi PETSc developers,
>
> In short, my question is this: Does PETSc provide a way to move or copy an 
> object (say a Mat) from one communicator to another?
>
> The more detailed scenario is this: I’m working on a linear algebra solver on 
> quadtree meshes (i.e., p4est). I use communicator subsets in order to 
> facilitate communication between siblings or nearby neighbors. When 
> performing linear algebra across siblings (a group of 4), I need to copy a 
> node’s data (i.e., a Mat object) from a sibling’s communicator to the 
> communicator that includes the four siblings. From what I can tell, I can 
> only copy a PETSc object onto the same communicator.
>
> My current approach will be to copy the raw data from the Mat on one 
> communicator to a new Mat on the new communicator, but I wanted to see if 
> there is a more “elegant” approach within PETSc.
>
> Thanks in advance,
>
> Damyn Chipman
> Boise State University
> PhD Candidate
> Computational Sciences and Engineering
> damynchip...@u.boisestate.edu

[petsc-users] Copying PETSc Objects Across MPI Communicators

[Damyn Chipman]

Hi PETSc developers,

In short, my question is this: Does PETSc provide a way to move or copy an 
object (say a Mat) from one communicator to another?

The more detailed scenario is this: I’m working on a linear algebra solver on 
quadtree meshes (i.e., p4est). I use communicator subsets in order to 
facilitate communication between siblings or nearby neighbors. When performing 
linear algebra across siblings (a group of 4), I need to copy a node’s data 
(i.e., a Mat object) from a sibling’s communicator to the communicator that 
includes the four siblings. From what I can tell, I can only copy a PETSc 
object onto the same communicator.

My current approach will be to copy the raw data from the Mat on one 
communicator to a new Mat on the new communicator, but I wanted to see if there 
is a more “elegant” approach within PETSc.

Thanks in advance,

Damyn Chipman
Boise State University
PhD Candidate
Computational Sciences and Engineering
damynchip...@u.boisestate.edu

Re: [petsc-users] Seeking Clarification on SNES Solver Behavior

[Matthew Knepley]

On Mon, Oct 23, 2023 at 10:23 PM Rene Chenard via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Hi!
>
> We have recently noticed some inconsistencies in the behavior of the SNES
> solver when using different solver types, and we would greatly appreciate
> your insights in resolving this matter.
>
> While working with SNESSolve in parallel, we have encountered a
> discrepancy in the behavior of the evaluation functions for the
> ComputeFunction and the JacobianFunction. Specifically, there seems to be
> an inconsistency in whether Vec x receives automatic updates to its ghosts
> or if manual updates are required (with calls to VecGhostUpdateBegin/End).
>
> For instance, when using the ngmres solver, the ghosts of Vec x are
> adequately updated. However, when employing the nrichardson solver, it
> appears that manual updates to the ghosts are necessary.
>
> It's important to note that we do not utilize the DM object in our
> implementation, as we have developed our own solution to manage models and
> discretization.
>
> To better understand the root cause of this behavior, we kindly request
> your assistance in determining if we may be overlooking something in our
> implementation, or if there are inherent inconsistencies in the SNES solver
> itself.
>
> Your expertise in this matter would be invaluable to us, and we thank you
> in advance for your consideration and support.
>

Since you are not using DM, does that mean that you register a callback with

  https://petsc.org/main/manualpages/SNES/DMSNESSetFunction/

If so, we do not do any kind of local-to-global calls. I am not sure how
NGMRES would populate local
vectors for you. My guess is that you have a ghost update call somewhere in
your callback, and this
gets hit in the NGMRES because it has extra residual evaluations.

We can be more specific with more details about the code.

   Thanks,

 Matt



> Warm regards,
>
> —René Chenard
> Research Professional at Université Laval
> rene.chenar...@ulaval.ca
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/