Re: [petsc-users] Copying PETSc Objects Across MPI Communicators

[Damyn Chipman]

Sounds good, thanks.

I’ve also been looking into Elemental, but the documentation seems outdated and 
I can’t find good examples on how to use it. I have the LLNL fork installed.

Thanks,
-Damyn

> On Oct 28, 2023, at 8:56 AM, Matthew Knepley  wrote:
> 
> On Fri, Oct 27, 2023 at 3:54 PM Damyn Chipman  > wrote:
>> Yeah, I’ll make an issue and use a modified version of this test routine.
>> 
>> Does anything change if I will be using MATSCALAPACK matrices instead of the 
>> built in MATDENSE?
> 
> No, that is likely worse.
>  
>> Like I said, I will be computing Schur complements and need to use a 
>> parallel and dense matrix format.
> 
> I do not understand the communication pattern, but it is possible that 
> Elemental would be slightly faster since it has some cool built-in 
> communication operations, however it might be more programming.
> 
>   Thanks,
> 
>  Matt
>  
>> -Damyn
>> 
>>> On Oct 26, 2023, at 10:01 AM, Matthew Knepley >> > wrote:
>>> 
>>> On Wed, Oct 25, 2023 at 11:55 PM Damyn Chipman 
>>> mailto:damynchip...@u.boisestate.edu>> 
>>> wrote:
 Great thanks, that seemed to work well. This is something my algorithm 
 will do fairly often (“elevating” a node’s communicator to a communicator 
 that includes siblings). The matrices formed are dense but low rank. With 
 MatCreateSubMatrix, it appears I do a lot of copying from one Mat to 
 another. Is there a way to do it with array copying or pointer movement 
 instead of copying entries?
>>> 
>>> We could make a fast path for dense that avoids MatSetValues(). Can you 
>>> make an issue for this? The number one thing that would make this faster is 
>>> to contribute a small test. Then we could run it continually when putting 
>>> in the fast path to make sure we are preserving correctness.
>>> 
>>>   Thanks,
>>> 
>>> Matt
>>>  
 -Damyn
 
> On Oct 24, 2023, at 9:51 PM, Jed Brown  > wrote:
> 
> You can place it in a parallel Mat (that has rows or columns on only one 
> rank or a subset of ranks) and then MatCreateSubMatrix with all new 
> rows/columns on a different rank or subset of ranks.
> 
> That said, you usually have a function that assembles the matrix and you 
> can just call that on the new communicator.
> 
> Damyn Chipman  > writes:
> 
>> Hi PETSc developers,
>> 
>> In short, my question is this: Does PETSc provide a way to move or copy 
>> an object (say a Mat) from one communicator to another?
>> 
>> The more detailed scenario is this: I’m working on a linear algebra 
>> solver on quadtree meshes (i.e., p4est). I use communicator subsets in 
>> order to facilitate communication between siblings or nearby neighbors. 
>> When performing linear algebra across siblings (a group of 4), I need to 
>> copy a node’s data (i.e., a Mat object) from a sibling’s communicator to 
>> the communicator that includes the four siblings. From what I can tell, 
>> I can only copy a PETSc object onto the same communicator.
>> 
>> My current approach will be to copy the raw data from the Mat on one 
>> communicator to a new Mat on the new communicator, but I wanted to see 
>> if there is a more “elegant” approach within PETSc.
>> 
>> Thanks in advance,
>> 
>> Damyn Chipman
>> Boise State University
>> PhD Candidate
>> Computational Sciences and Engineering
>> damynchip...@u.boisestate.edu 
 
>>> 
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their 
>>> experiments is infinitely more interesting than any results to which their 
>>> experiments lead.
>>> -- Norbert Wiener
>>> 
>>> https://www.cse.buffalo.edu/~knepley/ 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments 
> is infinitely more interesting than any results to which their experiments 
> lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Copying PETSc Objects Across MPI Communicators

[Matthew Knepley]

On Fri, Oct 27, 2023 at 3:54 PM Damyn Chipman 
wrote:

> Yeah, I’ll make an issue and use a modified version of this test routine.
>
> Does anything change if I will be using MATSCALAPACK matrices instead of
> the built in MATDENSE?
>

No, that is likely worse.


> Like I said, I will be computing Schur complements and need to use a
> parallel and dense matrix format.
>

I do not understand the communication pattern, but it is possible that
Elemental would be slightly faster since it has some cool built-in
communication operations, however it might be more programming.

  Thanks,

 Matt


> -Damyn
>
> On Oct 26, 2023, at 10:01 AM, Matthew Knepley  wrote:
>
> On Wed, Oct 25, 2023 at 11:55 PM Damyn Chipman <
> damynchip...@u.boisestate.edu> wrote:
>
>> Great thanks, that seemed to work well. This is something my algorithm
>> will do fairly often (“elevating” a node’s communicator to a communicator
>> that includes siblings). The matrices formed are dense but low rank. With
>> MatCreateSubMatrix, it appears I do a lot of copying from one Mat to
>> another. Is there a way to do it with array copying or pointer movement
>> instead of copying entries?
>>
>
> We could make a fast path for dense that avoids MatSetValues(). Can you
> make an issue for this? The number one thing that would make this faster is
> to contribute a small test. Then we could run it continually when putting
> in the fast path to make sure we are preserving correctness.
>
>   Thanks,
>
> Matt
>
>
>> -Damyn
>>
>> On Oct 24, 2023, at 9:51 PM, Jed Brown  wrote:
>>
>> You can place it in a parallel Mat (that has rows or columns on only one
>> rank or a subset of ranks) and then MatCreateSubMatrix with all new
>> rows/columns on a different rank or subset of ranks.
>>
>> That said, you usually have a function that assembles the matrix and you
>> can just call that on the new communicator.
>>
>> Damyn Chipman  writes:
>>
>> Hi PETSc developers,
>>
>> In short, my question is this: Does PETSc provide a way to move or copy
>> an object (say a Mat) from one communicator to another?
>>
>> The more detailed scenario is this: I’m working on a linear algebra
>> solver on quadtree meshes (i.e., p4est). I use communicator subsets in
>> order to facilitate communication between siblings or nearby neighbors.
>> When performing linear algebra across siblings (a group of 4), I need to
>> copy a node’s data (i.e., a Mat object) from a sibling’s communicator to
>> the communicator that includes the four siblings. From what I can tell, I
>> can only copy a PETSc object onto the same communicator.
>>
>> My current approach will be to copy the raw data from the Mat on one
>> communicator to a new Mat on the new communicator, but I wanted to see if
>> there is a more “elegant” approach within PETSc.
>>
>> Thanks in advance,
>>
>> Damyn Chipman
>> Boise State University
>> PhD Candidate
>> Computational Sciences and Engineering
>> damynchip...@u.boisestate.edu
>>
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> 
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Copying PETSc Objects Across MPI Communicators

[Jose E. Roman]

Currently MATSCALAPACK does not support MatCreateSubMatrix(). I guess it would 
not be difficult to implement.

Jose


> El 27 oct 2023, a las 21:53, Damyn Chipman  
> escribió:
> 
> Yeah, I’ll make an issue and use a modified version of this test routine.
> 
> Does anything change if I will be using MATSCALAPACK matrices instead of the 
> built in MATDENSE? Like I said, I will be computing Schur complements and 
> need to use a parallel and dense matrix format.
> 
> -Damyn
> 
>> On Oct 26, 2023, at 10:01 AM, Matthew Knepley  wrote:
>> 
>> On Wed, Oct 25, 2023 at 11:55 PM Damyn Chipman 
>>  wrote:
>> Great thanks, that seemed to work well. This is something my algorithm will 
>> do fairly often (“elevating” a node’s communicator to a communicator that 
>> includes siblings). The matrices formed are dense but low rank. With 
>> MatCreateSubMatrix, it appears I do a lot of copying from one Mat to 
>> another. Is there a way to do it with array copying or pointer movement 
>> instead of copying entries?
>> 
>> We could make a fast path for dense that avoids MatSetValues(). Can you make 
>> an issue for this? The number one thing that would make this faster is to 
>> contribute a small test. Then we could run it continually when putting in 
>> the fast path to make sure we are preserving correctness.
>> 
>>   Thanks,
>> 
>> Matt
>>  
>> -Damyn
>> 
>>> On Oct 24, 2023, at 9:51 PM, Jed Brown  wrote:
>>> 
>>> You can place it in a parallel Mat (that has rows or columns on only one 
>>> rank or a subset of ranks) and then MatCreateSubMatrix with all new 
>>> rows/columns on a different rank or subset of ranks.
>>> 
>>> That said, you usually have a function that assembles the matrix and you 
>>> can just call that on the new communicator.
>>> 
>>> Damyn Chipman  writes:
>>> 
 Hi PETSc developers,
 
 In short, my question is this: Does PETSc provide a way to move or copy an 
 object (say a Mat) from one communicator to another?
 
 The more detailed scenario is this: I’m working on a linear algebra solver 
 on quadtree meshes (i.e., p4est). I use communicator subsets in order to 
 facilitate communication between siblings or nearby neighbors. When 
 performing linear algebra across siblings (a group of 4), I need to copy a 
 node’s data (i.e., a Mat object) from a sibling’s communicator to the 
 communicator that includes the four siblings. From what I can tell, I can 
 only copy a PETSc object onto the same communicator.
 
 My current approach will be to copy the raw data from the Mat on one 
 communicator to a new Mat on the new communicator, but I wanted to see if 
 there is a more “elegant” approach within PETSc.
 
 Thanks in advance,
 
 Damyn Chipman
 Boise State University
 PhD Candidate
 Computational Sciences and Engineering
 damynchip...@u.boisestate.edu
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments 
>> is infinitely more interesting than any results to which their experiments 
>> lead.
>> -- Norbert Wiener
>> 
>> https://www.cse.buffalo.edu/~knepley/
> 

Re: [petsc-users] Copying PETSc Objects Across MPI Communicators

[Damyn Chipman]

Yeah, I’ll make an issue and use a modified version of this test routine.

Does anything change if I will be using MATSCALAPACK matrices instead of the 
built in MATDENSE? Like I said, I will be computing Schur complements and need 
to use a parallel and dense matrix format.

-Damyn

> On Oct 26, 2023, at 10:01 AM, Matthew Knepley  wrote:
> 
> On Wed, Oct 25, 2023 at 11:55 PM Damyn Chipman  > wrote:
>> Great thanks, that seemed to work well. This is something my algorithm will 
>> do fairly often (“elevating” a node’s communicator to a communicator that 
>> includes siblings). The matrices formed are dense but low rank. With 
>> MatCreateSubMatrix, it appears I do a lot of copying from one Mat to 
>> another. Is there a way to do it with array copying or pointer movement 
>> instead of copying entries?
> 
> We could make a fast path for dense that avoids MatSetValues(). Can you make 
> an issue for this? The number one thing that would make this faster is to 
> contribute a small test. Then we could run it continually when putting in the 
> fast path to make sure we are preserving correctness.
> 
>   Thanks,
> 
> Matt
>  
>> -Damyn
>> 
>>> On Oct 24, 2023, at 9:51 PM, Jed Brown >> > wrote:
>>> 
>>> You can place it in a parallel Mat (that has rows or columns on only one 
>>> rank or a subset of ranks) and then MatCreateSubMatrix with all new 
>>> rows/columns on a different rank or subset of ranks.
>>> 
>>> That said, you usually have a function that assembles the matrix and you 
>>> can just call that on the new communicator.
>>> 
>>> Damyn Chipman >> > writes:
>>> 
 Hi PETSc developers,
 
 In short, my question is this: Does PETSc provide a way to move or copy an 
 object (say a Mat) from one communicator to another?
 
 The more detailed scenario is this: I’m working on a linear algebra solver 
 on quadtree meshes (i.e., p4est). I use communicator subsets in order to 
 facilitate communication between siblings or nearby neighbors. When 
 performing linear algebra across siblings (a group of 4), I need to copy a 
 node’s data (i.e., a Mat object) from a sibling’s communicator to the 
 communicator that includes the four siblings. From what I can tell, I can 
 only copy a PETSc object onto the same communicator.
 
 My current approach will be to copy the raw data from the Mat on one 
 communicator to a new Mat on the new communicator, but I wanted to see if 
 there is a more “elegant” approach within PETSc.
 
 Thanks in advance,
 
 Damyn Chipman
 Boise State University
 PhD Candidate
 Computational Sciences and Engineering
 damynchip...@u.boisestate.edu 
>> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments 
> is infinitely more interesting than any results to which their experiments 
> lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Copying PETSc Objects Across MPI Communicators

[Matthew Knepley]

On Wed, Oct 25, 2023 at 11:55 PM Damyn Chipman <
damynchip...@u.boisestate.edu> wrote:

> Great thanks, that seemed to work well. This is something my algorithm
> will do fairly often (“elevating” a node’s communicator to a communicator
> that includes siblings). The matrices formed are dense but low rank. With
> MatCreateSubMatrix, it appears I do a lot of copying from one Mat to
> another. Is there a way to do it with array copying or pointer movement
> instead of copying entries?
>

We could make a fast path for dense that avoids MatSetValues(). Can you
make an issue for this? The number one thing that would make this faster is
to contribute a small test. Then we could run it continually when putting
in the fast path to make sure we are preserving correctness.

  Thanks,

Matt


> -Damyn
>
> On Oct 24, 2023, at 9:51 PM, Jed Brown  wrote:
>
> You can place it in a parallel Mat (that has rows or columns on only one
> rank or a subset of ranks) and then MatCreateSubMatrix with all new
> rows/columns on a different rank or subset of ranks.
>
> That said, you usually have a function that assembles the matrix and you
> can just call that on the new communicator.
>
> Damyn Chipman  writes:
>
> Hi PETSc developers,
>
> In short, my question is this: Does PETSc provide a way to move or copy an
> object (say a Mat) from one communicator to another?
>
> The more detailed scenario is this: I’m working on a linear algebra solver
> on quadtree meshes (i.e., p4est). I use communicator subsets in order to
> facilitate communication between siblings or nearby neighbors. When
> performing linear algebra across siblings (a group of 4), I need to copy a
> node’s data (i.e., a Mat object) from a sibling’s communicator to the
> communicator that includes the four siblings. From what I can tell, I can
> only copy a PETSc object onto the same communicator.
>
> My current approach will be to copy the raw data from the Mat on one
> communicator to a new Mat on the new communicator, but I wanted to see if
> there is a more “elegant” approach within PETSc.
>
> Thanks in advance,
>
> Damyn Chipman
> Boise State University
> PhD Candidate
> Computational Sciences and Engineering
> damynchip...@u.boisestate.edu
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Copying PETSc Objects Across MPI Communicators

[Damyn Chipman]

More like smaller pieces that need to be combined. Combining them (merging) 
means sharing the actual data across a sibling communicator and doing some 
linear algebra to compute the merged matrices (it involves computing a Schur 
complement of a combined system from the sibling matrices).

The solver is based on the Hierarchical Poincaré-Steklov (HPS) method, a direct 
method for solving elliptic PDEs. I had a conversation with Richard at this 
year’s ATPESC2023 about this idea.

For some more context, here’s the test routine I wrote based on the 
MatCreateSubMatrix idea. The actual implementation would be part of a recursive 
merge up a quadtree. Each node's communicator would be a sub-communicator of 
its parent, and so on. I want to spread the data and compute across any ranks 
that are involved in that node’s merging. The sizes involved start “small” at 
each leaf node (say, no more than 256x256), then are essentially doubled up the 
tree to the root node.

```
void TEST_petsc_matrix_comm() {

// Create local matrices on local communicator
int M_local = 4;
int N_local = 4;
Mat mat_local;
MatCreate(MPI_COMM_SELF, _local); // Note the MPI_COMM_SELF as a 
substitute for a sub-communicator of MPI_COMM_WORLD
MatSetSizes(mat_local, PETSC_DECIDE, PETSC_DECIDE, M_local, N_local);
MatSetFromOptions(mat_local);

// Set values in local matrix
int* row_indices = (int*) malloc(M_local*sizeof(int));
for (int i = 0; i < M_local; i++) {
row_indices[i] = i;
}

int* col_indices = (int*) malloc(N_local*sizeof(int));
for (int j = 0; j < M_local; j++) {;
col_indices[j] = j;
}

double* values = (double*) malloc(M_local*N_local*sizeof(double));
int v = M_local*N_local*rank;
for (int j = 0; j < N_local; j++) {
for (int i = 0; i < M_local; i++) {
values[i + j*N_local] = (double) v;
v++;
}
}
MatSetValues(mat_local, M_local, row_indices, N_local, col_indices, values, 
INSERT_VALUES);
MatAssemblyBegin(mat_local, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(mat_local, MAT_FINAL_ASSEMBLY);

// Create local matrices on global communicator
Mat mat_global;
IS is_row;
int idx[4] = {0, 1, 2, 3};
ISCreateGeneral(MPI_COMM_WORLD, M_local, idx, PETSC_COPY_VALUES, _row);
MatCreateSubMatrix(mat_local, is_row, NULL, MAT_INITIAL_MATRIX, 
_global);

// View each local mat on global communicator (sleep for `rank` seconds so 
output is ordered)
sleep(rank);
MatView(mat_global, 0);

// Create merged mat on global communicator
//For this test, I just put the four locally computed matrices on the 
diagonal of the merged matrix
//In the 4-to-1 merge, this would compute T_merged from T_alpha, 
T_beta, T_gamma, and T_omega (children)
int M_merged = M_local*size;
int N_merged = N_local*size;
Mat mat_merged;
MatCreate(MPI_COMM_WORLD, _merged);
MatSetSizes(mat_merged, PETSC_DECIDE, PETSC_DECIDE, M_merged, N_merged);
MatSetFromOptions(mat_merged);

// Get values of local matrix to put on diagonal
double* values_diag = (double*) malloc(M_local*N_local*sizeof(double));
MatGetValues(mat_global, M_local, row_indices, N_local, col_indices, 
values_diag);

// Put local matrix contributions into merged matrix (placeholder for 
computing merged matrix)
for (int i = 0; i < M_local; i++) {
row_indices[i] = i + M_local*rank;
}

for (int j = 0; j < N_local; j++) {
col_indices[j] = j + N_local*rank;
}
MatSetValues(mat_merged, M_local, row_indices, N_local, col_indices, 
values_diag, INSERT_VALUES);
MatAssemblyBegin(mat_merged, MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(mat_merged, MAT_FINAL_ASSEMBLY);

// View merged mat on global communicator
sleep(rank);
MatView(mat_merged, 0);

// Clean up
free(row_indices);
free(col_indices);
free(values);
free(values_diag);
MatDestroy(_local);
MatDestroy(_global);
MatDestroy(_merged);

}
```

With the following output :

```
(base) ➜  mpi git:(feature-parallel) ✗ mpirun -n 4 ./mpi_matrix
Mat Object: 1 MPI process
  type: seqaij
row 0: (0, 0.)  (1, 1.)  (2, 2.)  (3, 3.) 
row 1: (0, 4.)  (1, 5.)  (2, 6.)  (3, 7.) 
row 2: (0, 8.)  (1, 9.)  (2, 10.)  (3, 11.) 
row 3: (0, 12.)  (1, 13.)  (2, 14.)  (3, 15.) 
Mat Object: 1 MPI process
  type: seqaij
row 0: (0, 16.)  (1, 17.)  (2, 18.)  (3, 19.) 
row 1: (0, 20.)  (1, 21.)  (2, 22.)  (3, 23.) 
row 2: (0, 24.)  (1, 25.)  (2, 26.)  (3, 27.) 
row 3: (0, 28.)  (1, 29.)  (2, 30.)  (3, 31.) 
Mat Object: 1 MPI process
  type: seqaij
row 0: (0, 32.)  (1, 33.)  (2, 34.)  (3, 35.) 
row 1: (0, 36.)  (1, 37.)  (2, 38.)  (3, 39.) 
row 2: (0, 40.)  (1, 41.)  (2, 42.)  (3, 43.) 
row 3: (0, 44.)  (1, 45.)  (2, 46.)  (3, 47.) 
Mat Object: 1 MPI process
  type: seqaij
row 0: (0, 48.)  (1, 49.)  (2, 50.)  (3, 51.) 
row 1: (0, 52.)  (1, 53.)  (2, 54.)  (3, 55.) 
row 2: (0, 56.)  (1, 57.)  (2, 58.)  (3, 

Re: [petsc-users] Copying PETSc Objects Across MPI Communicators

[Barry Smith]

  If the matrices are stored as dense it is likely new code is the best way to 
go. 

  What pieces live on the sub communicator? Is it an m by N matrix where m is 
the number of rows (on that rank) and N is the total number of columns in the 
final matrix? Or are they smaller "chunks" that need to be combined together?

  Barry


> On Oct 25, 2023, at 3:38 PM, Damyn Chipman  
> wrote:
> 
> Great thanks, that seemed to work well. This is something my algorithm will 
> do fairly often (“elevating” a node’s communicator to a communicator that 
> includes siblings). The matrices formed are dense but low rank. With 
> MatCreateSubMatrix, it appears I do a lot of copying from one Mat to another. 
> Is there a way to do it with array copying or pointer movement instead of 
> copying entries?
> 
> -Damyn
> 
>> On Oct 24, 2023, at 9:51 PM, Jed Brown  wrote:
>> 
>> You can place it in a parallel Mat (that has rows or columns on only one 
>> rank or a subset of ranks) and then MatCreateSubMatrix with all new 
>> rows/columns on a different rank or subset of ranks.
>> 
>> That said, you usually have a function that assembles the matrix and you can 
>> just call that on the new communicator.
>> 
>> Damyn Chipman  writes:
>> 
>>> Hi PETSc developers,
>>> 
>>> In short, my question is this: Does PETSc provide a way to move or copy an 
>>> object (say a Mat) from one communicator to another?
>>> 
>>> The more detailed scenario is this: I’m working on a linear algebra solver 
>>> on quadtree meshes (i.e., p4est). I use communicator subsets in order to 
>>> facilitate communication between siblings or nearby neighbors. When 
>>> performing linear algebra across siblings (a group of 4), I need to copy a 
>>> node’s data (i.e., a Mat object) from a sibling’s communicator to the 
>>> communicator that includes the four siblings. From what I can tell, I can 
>>> only copy a PETSc object onto the same communicator.
>>> 
>>> My current approach will be to copy the raw data from the Mat on one 
>>> communicator to a new Mat on the new communicator, but I wanted to see if 
>>> there is a more “elegant” approach within PETSc.
>>> 
>>> Thanks in advance,
>>> 
>>> Damyn Chipman
>>> Boise State University
>>> PhD Candidate
>>> Computational Sciences and Engineering
>>> damynchip...@u.boisestate.edu
> 

Re: [petsc-users] Copying PETSc Objects Across MPI Communicators

[Damyn Chipman]

Great thanks, that seemed to work well. This is something my algorithm will do 
fairly often (“elevating” a node’s communicator to a communicator that includes 
siblings). The matrices formed are dense but low rank. With MatCreateSubMatrix, 
it appears I do a lot of copying from one Mat to another. Is there a way to do 
it with array copying or pointer movement instead of copying entries?

-Damyn

> On Oct 24, 2023, at 9:51 PM, Jed Brown  wrote:
> 
> You can place it in a parallel Mat (that has rows or columns on only one rank 
> or a subset of ranks) and then MatCreateSubMatrix with all new rows/columns 
> on a different rank or subset of ranks.
> 
> That said, you usually have a function that assembles the matrix and you can 
> just call that on the new communicator.
> 
> Damyn Chipman  writes:
> 
>> Hi PETSc developers,
>> 
>> In short, my question is this: Does PETSc provide a way to move or copy an 
>> object (say a Mat) from one communicator to another?
>> 
>> The more detailed scenario is this: I’m working on a linear algebra solver 
>> on quadtree meshes (i.e., p4est). I use communicator subsets in order to 
>> facilitate communication between siblings or nearby neighbors. When 
>> performing linear algebra across siblings (a group of 4), I need to copy a 
>> node’s data (i.e., a Mat object) from a sibling’s communicator to the 
>> communicator that includes the four siblings. From what I can tell, I can 
>> only copy a PETSc object onto the same communicator.
>> 
>> My current approach will be to copy the raw data from the Mat on one 
>> communicator to a new Mat on the new communicator, but I wanted to see if 
>> there is a more “elegant” approach within PETSc.
>> 
>> Thanks in advance,
>> 
>> Damyn Chipman
>> Boise State University
>> PhD Candidate
>> Computational Sciences and Engineering
>> damynchip...@u.boisestate.edu

Re: [petsc-users] Copying PETSc Objects Across MPI Communicators

[Jed Brown]

You can place it in a parallel Mat (that has rows or columns on only one rank 
or a subset of ranks) and then MatCreateSubMatrix with all new rows/columns on 
a different rank or subset of ranks.

That said, you usually have a function that assembles the matrix and you can 
just call that on the new communicator.

Damyn Chipman  writes:

> Hi PETSc developers,
>
> In short, my question is this: Does PETSc provide a way to move or copy an 
> object (say a Mat) from one communicator to another?
>
> The more detailed scenario is this: I’m working on a linear algebra solver on 
> quadtree meshes (i.e., p4est). I use communicator subsets in order to 
> facilitate communication between siblings or nearby neighbors. When 
> performing linear algebra across siblings (a group of 4), I need to copy a 
> node’s data (i.e., a Mat object) from a sibling’s communicator to the 
> communicator that includes the four siblings. From what I can tell, I can 
> only copy a PETSc object onto the same communicator.
>
> My current approach will be to copy the raw data from the Mat on one 
> communicator to a new Mat on the new communicator, but I wanted to see if 
> there is a more “elegant” approach within PETSc.
>
> Thanks in advance,
>
> Damyn Chipman
> Boise State University
> PhD Candidate
> Computational Sciences and Engineering
> damynchip...@u.boisestate.edu

[petsc-users] Copying PETSc Objects Across MPI Communicators

[Damyn Chipman]

Hi PETSc developers,

In short, my question is this: Does PETSc provide a way to move or copy an 
object (say a Mat) from one communicator to another?

The more detailed scenario is this: I’m working on a linear algebra solver on 
quadtree meshes (i.e., p4est). I use communicator subsets in order to 
facilitate communication between siblings or nearby neighbors. When performing 
linear algebra across siblings (a group of 4), I need to copy a node’s data 
(i.e., a Mat object) from a sibling’s communicator to the communicator that 
includes the four siblings. From what I can tell, I can only copy a PETSc 
object onto the same communicator.

My current approach will be to copy the raw data from the Mat on one 
communicator to a new Mat on the new communicator, but I wanted to see if there 
is a more “elegant” approach within PETSc.

Thanks in advance,

Damyn Chipman
Boise State University
PhD Candidate
Computational Sciences and Engineering
damynchip...@u.boisestate.edu