Re: [petsc-users] PetscObjectGetComm

[Marius Buerkle]

thanks everyone for the very thorough answers. 

 

marius

 

 


MPI_Comm are opaque handles in C and integers in Fortran, which is required by MPI standard. The same applies to other types, like MPI_Op, MPI_Win etc. MPICH and OpenMPI have different implements for the handles. In MPICH handles are integer/bitfield, with some bits being offset to an array of objects.  This makes it easy to do things like MPI_Comm_f2c(). In OpenMPI handles are pointers. OpenMPI has to transform pointers to integer offsets in MPI_Comm_c2f().
 

Running your tests with OpenMPI, you can see different pointers but same offsets

test_comms.c:

0 4 -1419258464 909680992 909680992
1 4 1152255392 -2144517440 -2144517440
2 4 -306719328 768197312 768197312
3 4 -1766709856 715374384 715374384

 

test_comms.f90:

           0           0           3

           1           0           3
           2           0           3
           3           0           3
 

Running with MPICH, you can see C/Fortran MPI_Comm's are the same.  But why ranks do not have the same integer/bitfield, I don't know. You need to dig into mpich code.

test_comms.c:

0 4 1140850688 -2080374780 -2080374780
1 4 1140850688 -2080374780 -2080374780
2 4 1140850688 -2080374782 -2080374782
3 4 1140850688 -2080374782 -2080374782

 
test_comms.f90:

           0  1140850688 -2080374780
           1  1140850688 -2080374780
           2  1140850688 -2080374782
           3  1140850688 -2080374782

 

In summary, users should not expect MPI_Comm variables are equal across ranks, and MPI_Send an MPI_Comm variable to remote ranks.




--Junchao Zhang




 


On Wed, Apr 22, 2020 at 8:56 AM Patrick Sanan  wrote:


Perhaps the confusion here is related to the fact that an MPI_Comm is not an integer identifying the communicator. Rather,
it's a pointer to a data structure which contains information about the communicator (I'm not positive but probably something like this). 

 

You're converting that pointer to an int and printing it out. The value happens to be the same on all ranks except 0, but this

doesn't directly tell you anything about equality of the MPI_comm objects that those pointers point to.

 


Am Mi., 22. Apr. 2020 um 15:28 Uhr schrieb Matthew Knepley :



On Wed, Apr 22, 2020 at 3:07 AM Marius Buerkle  wrote:





I see, but I am still puzzeled, why are the communicators different on different notes eventhough it is the same object.




 

This is the output of MPI_Comm_dup() on line 126 of tagm.c. Therefore, dup comms are not guaranteed to have the same id

across multiple processes.

 

  Thanks,

 

     Matt

 




 




 

PETSc creates a duplicate of the communicator during object creation.
https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscCommDuplicate.html

Jose


> El 22 abr 2020, a las 8:40, Marius Buerkle  escribió:
>
> Hi Dave,
>
> I want to use it in Fortran if possible. But I tried both C and Fortran just to see if it works in general. I am using MPICH 3.3.2. I attached the MWE for C and Fortran with the output I get.
>
> Marius
>
>
>
>
>
> Hi,
>
> What is PetscObjectGetComm expected to return?
>
> As Patrick said, it returns the communicator associated with the petsc object.
>
> I thought it would give the MPI communicator the object lives on. So if I create A matrix on PETSC_COMM_WORLD a call of PetscObjectGetComm for A it would return PETSC_COMM_WORLD? But it seems to return something else, and while most of the nodes return a similar communicator some are giving a different one.
>
> How are you actually comparing the communicators (send code snippet)? Which MPI implementation are you using? And when are comparing comms is the comparison code written in C it FORTRAN?
>
>
> That said, is there a way to get the MPI communicator a matrix lives on?
>
> You are using the correct function. There is a macro as well but it’s best to use the function.
>
> Thanks,
> Dave
>
>
>
>
> Best,
> Marius
> 
 







 

 
--







What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

 

https://www.cse.buffalo.edu/~knepley/

Re: [petsc-users] PetscObjectGetComm

[Junchao Zhang]

MPI_Comm are opaque handles in C and integers in Fortran, which is required
by MPI standard. The same applies to other types, like MPI_Op, MPI_Win etc.
MPICH and OpenMPI have different implements for the handles. In MPICH
handles are integer/bitfield, with some bits being offset to an array of
objects.  This makes it easy to do things like MPI_Comm_f2c(). In OpenMPI
handles are pointers. OpenMPI has to transform pointers to integer offsets
in MPI_Comm_c2f().

Running your tests with OpenMPI, you can see different pointers but same
offsets
test_comms.c:
0 4 -1419258464 909680992 909680992
1 4 1152255392 -2144517440 -2144517440
2 4 -306719328 768197312 768197312
3 4 -1766709856 715374384 715374384

test_comms.f90:
   0   0   3
   1   0   3
   2   0   3
   3   0   3

Running with MPICH, you can see C/Fortran MPI_Comm's are the same.  But why
ranks do not have the same integer/bitfield, I don't know. You need to dig
into mpich code.
test_comms.c:
0 4 1140850688 -2080374780 -2080374780
1 4 1140850688 -2080374780 -2080374780
2 4 1140850688 -2080374782 -2080374782
3 4 1140850688 -2080374782 -2080374782

test_comms.f90:
   0  1140850688 -2080374780
   1  1140850688 -2080374780
   2  1140850688 -2080374782
   3  1140850688 -2080374782

In summary, users should not expect MPI_Comm variables are equal across
ranks, and MPI_Send an MPI_Comm variable to remote ranks.
--Junchao Zhang


On Wed, Apr 22, 2020 at 8:56 AM Patrick Sanan 
wrote:

> Perhaps the confusion here is related to the fact that an MPI_Comm is not
> an integer identifying the communicator. Rather,
> it's a pointer to a data structure which contains information about the
> communicator (I'm not positive but probably something like this
> 
> ).
>
> You're converting that pointer to an int and printing it out. The value
> happens to be the same on all ranks except 0, but this
> doesn't directly tell you anything about equality of the MPI_comm objects
> that those pointers point to.
>
> Am Mi., 22. Apr. 2020 um 15:28 Uhr schrieb Matthew Knepley <
> knep...@gmail.com>:
>
>> On Wed, Apr 22, 2020 at 3:07 AM Marius Buerkle  wrote:
>>
>>> I see, but I am still puzzeled, why are the communicators different on
>>> different notes eventhough it is the same object.
>>>
>>
>> This is the output of MPI_Comm_dup() on line 126 of tagm.c. Therefore,
>> dup comms are not guaranteed to have the same id
>> across multiple processes.
>>
>>   Thanks,
>>
>>  Matt
>>
>>
>>>
>>>
>>> PETSc creates a duplicate of the communicator during object creation.
>>>
>>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscCommDuplicate.html
>>>
>>> Jose
>>>
>>>
>>> > El 22 abr 2020, a las 8:40, Marius Buerkle  escribió:
>>> >
>>> > Hi Dave,
>>> >
>>> > I want to use it in Fortran if possible. But I tried both C and
>>> Fortran just to see if it works in general. I am using MPICH 3.3.2. I
>>> attached the MWE for C and Fortran with the output I get.
>>> >
>>> > Marius
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > Hi,
>>> >
>>> > What is PetscObjectGetComm expected to return?
>>> >
>>> > As Patrick said, it returns the communicator associated with the petsc
>>> object.
>>> >
>>> > I thought it would give the MPI communicator the object lives on. So
>>> if I create A matrix on PETSC_COMM_WORLD a call of PetscObjectGetComm for A
>>> it would return PETSC_COMM_WORLD? But it seems to return something else,
>>> and while most of the nodes return a similar communicator some are giving a
>>> different one.
>>> >
>>> > How are you actually comparing the communicators (send code snippet)?
>>> Which MPI implementation are you using? And when are comparing comms is the
>>> comparison code written in C it FORTRAN?
>>> >
>>> >
>>> > That said, is there a way to get the MPI communicator a matrix lives
>>> on?
>>> >
>>> > You are using the correct function. There is a macro as well but it’s
>>> best to use the function.
>>> >
>>> > Thanks,
>>> > Dave
>>> >
>>> >
>>> >
>>> >
>>> > Best,
>>> > Marius
>>> > 
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> 
>>
>

Re: [petsc-users] PetscObjectGetComm

[Patrick Sanan]

Perhaps the confusion here is related to the fact that an MPI_Comm is not
an integer identifying the communicator. Rather,
it's a pointer to a data structure which contains information about the
communicator (I'm not positive but probably something like this

).

You're converting that pointer to an int and printing it out. The value
happens to be the same on all ranks except 0, but this
doesn't directly tell you anything about equality of the MPI_comm objects
that those pointers point to.

Am Mi., 22. Apr. 2020 um 15:28 Uhr schrieb Matthew Knepley <
knep...@gmail.com>:

> On Wed, Apr 22, 2020 at 3:07 AM Marius Buerkle  wrote:
>
>> I see, but I am still puzzeled, why are the communicators different on
>> different notes eventhough it is the same object.
>>
>
> This is the output of MPI_Comm_dup() on line 126 of tagm.c. Therefore, dup
> comms are not guaranteed to have the same id
> across multiple processes.
>
>   Thanks,
>
>  Matt
>
>
>>
>>
>> PETSc creates a duplicate of the communicator during object creation.
>>
>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscCommDuplicate.html
>>
>> Jose
>>
>>
>> > El 22 abr 2020, a las 8:40, Marius Buerkle  escribió:
>> >
>> > Hi Dave,
>> >
>> > I want to use it in Fortran if possible. But I tried both C and Fortran
>> just to see if it works in general. I am using MPICH 3.3.2. I attached the
>> MWE for C and Fortran with the output I get.
>> >
>> > Marius
>> >
>> >
>> >
>> >
>> >
>> > Hi,
>> >
>> > What is PetscObjectGetComm expected to return?
>> >
>> > As Patrick said, it returns the communicator associated with the petsc
>> object.
>> >
>> > I thought it would give the MPI communicator the object lives on. So if
>> I create A matrix on PETSC_COMM_WORLD a call of PetscObjectGetComm for A it
>> would return PETSC_COMM_WORLD? But it seems to return something else, and
>> while most of the nodes return a similar communicator some are giving a
>> different one.
>> >
>> > How are you actually comparing the communicators (send code snippet)?
>> Which MPI implementation are you using? And when are comparing comms is the
>> comparison code written in C it FORTRAN?
>> >
>> >
>> > That said, is there a way to get the MPI communicator a matrix lives on?
>> >
>> > You are using the correct function. There is a macro as well but it’s
>> best to use the function.
>> >
>> > Thanks,
>> > Dave
>> >
>> >
>> >
>> >
>> > Best,
>> > Marius
>> > 
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> 
>

Re: [petsc-users] PetscObjectGetComm

[Matthew Knepley]

On Wed, Apr 22, 2020 at 3:07 AM Marius Buerkle  wrote:

> I see, but I am still puzzeled, why are the communicators different on
> different notes eventhough it is the same object.
>

This is the output of MPI_Comm_dup() on line 126 of tagm.c. Therefore, dup
comms are not guaranteed to have the same id
across multiple processes.

  Thanks,

 Matt


>
>
> PETSc creates a duplicate of the communicator during object creation.
>
> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscCommDuplicate.html
>
> Jose
>
>
> > El 22 abr 2020, a las 8:40, Marius Buerkle  escribió:
> >
> > Hi Dave,
> >
> > I want to use it in Fortran if possible. But I tried both C and Fortran
> just to see if it works in general. I am using MPICH 3.3.2. I attached the
> MWE for C and Fortran with the output I get.
> >
> > Marius
> >
> >
> >
> >
> >
> > Hi,
> >
> > What is PetscObjectGetComm expected to return?
> >
> > As Patrick said, it returns the communicator associated with the petsc
> object.
> >
> > I thought it would give the MPI communicator the object lives on. So if
> I create A matrix on PETSC_COMM_WORLD a call of PetscObjectGetComm for A it
> would return PETSC_COMM_WORLD? But it seems to return something else, and
> while most of the nodes return a similar communicator some are giving a
> different one.
> >
> > How are you actually comparing the communicators (send code snippet)?
> Which MPI implementation are you using? And when are comparing comms is the
> comparison code written in C it FORTRAN?
> >
> >
> > That said, is there a way to get the MPI communicator a matrix lives on?
> >
> > You are using the correct function. There is a macro as well but it’s
> best to use the function.
> >
> > Thanks,
> > Dave
> >
> >
> >
> >
> > Best,
> > Marius
> > 
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] PetscObjectGetComm

[Marius Buerkle]

I see, but I am still puzzeled, why are the communicators different on different notes eventhough it is the same object.

 


 

PETSc creates a duplicate of the communicator during object creation.
https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscCommDuplicate.html

Jose


> El 22 abr 2020, a las 8:40, Marius Buerkle  escribió:
>
> Hi Dave,
>
> I want to use it in Fortran if possible. But I tried both C and Fortran just to see if it works in general. I am using MPICH 3.3.2. I attached the MWE for C and Fortran with the output I get.
>
> Marius
>
>
>
>
>
> Hi,
>
> What is PetscObjectGetComm expected to return?
>
> As Patrick said, it returns the communicator associated with the petsc object.
>
> I thought it would give the MPI communicator the object lives on. So if I create A matrix on PETSC_COMM_WORLD a call of PetscObjectGetComm for A it would return PETSC_COMM_WORLD? But it seems to return something else, and while most of the nodes return a similar communicator some are giving a different one.
>
> How are you actually comparing the communicators (send code snippet)? Which MPI implementation are you using? And when are comparing comms is the comparison code written in C it FORTRAN?
>
>
> That said, is there a way to get the MPI communicator a matrix lives on?
>
> You are using the correct function. There is a macro as well but it’s best to use the function.
>
> Thanks,
> Dave
>
>
>
>
> Best,
> Marius
> 
 

Re: [petsc-users] PetscObjectGetComm

[Jose E. Roman]

PETSc creates a duplicate of the communicator during object creation.
https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscCommDuplicate.html

Jose


> El 22 abr 2020, a las 8:40, Marius Buerkle  escribió:
> 
> Hi Dave,
>  
> I want to use it in Fortran if possible. But I tried both C and Fortran just 
> to see if it works in general. I am using MPICH 3.3.2. I attached the MWE for 
> C and Fortran with the output I get.
> 
> Marius 
>  
>  
>  
>  
>  
> Hi,
> 
> What is PetscObjectGetComm expected to return?
>  
> As Patrick said, it returns the communicator associated with the petsc object.
>  
> I thought it would give the MPI communicator the object lives on. So if I 
> create A matrix on PETSC_COMM_WORLD a call of PetscObjectGetComm for A it 
> would return PETSC_COMM_WORLD? But it seems to return something else, and 
> while most of the nodes return a similar communicator some are giving a 
> different one.
>  
> How are you actually comparing the communicators (send code snippet)? Which 
> MPI implementation are you using? And when are comparing comms is the 
> comparison code written in C it FORTRAN?
>  
>  
> That said, is there a way to get the MPI communicator a matrix lives on?
>  
> You are using the correct function. There is a macro as well but it’s best to 
> use the function.
>  
> Thanks,
> Dave
>  
>  
> 
> 
> Best,
> Marius
> 

Re: [petsc-users] PetscObjectGetComm

[Marius Buerkle]

Hi Dave,

 

I want to use it in Fortran if possible. But I tried both C and Fortran just to see if it works in general. I am using MPICH 3.3.2. I attached the MWE for C and Fortran with the output I get.


Marius 

 


 


 

 

 

Hi,

What is PetscObjectGetComm expected to return?

 

As Patrick said, it returns the communicator associated with the petsc object.

 

I thought it would give the MPI communicator the object lives on. So if I create A matrix on PETSC_COMM_WORLD a call of PetscObjectGetComm for A it would return PETSC_COMM_WORLD? But it seems to return something else, and while most of the nodes return a similar communicator some are giving a different one.

 

How are you actually comparing the communicators (send code snippet)? Which MPI implementation are you using? And when are comparing comms is the comparison code written in C it FORTRAN?

 

 

That said, is there a way to get the MPI communicator a matrix lives on?

 

You are using the correct function. There is a macro as well but it’s best to use the function.

 

Thanks,

Dave

 

 



Best,
Marius







test_comm.tar.gz
Description: Binary data

Re: [petsc-users] PetscObjectGetComm

[Dave May]

On Wed 22. Apr 2020 at 07:11, Marius Buerkle  wrote:

> Hi,
>
> What is PetscObjectGetComm expected to return?


As Patrick said, it returns the communicator associated with the petsc
object.

I thought it would give the MPI communicator the object lives on. So if I
> create A matrix on PETSC_COMM_WORLD a call of PetscObjectGetComm for A it
> would return PETSC_COMM_WORLD? But it seems to return something else, and
> while most of the nodes return a similar communicator some are giving a
> different one.


How are you actually comparing the communicators (send code snippet)? Which
MPI implementation are you using? And when are comparing comms is the
comparison code written in C it FORTRAN?


That said, is there a way to get the MPI communicator a matrix lives on?


You are using the correct function. There is a macro as well but it’s best
to use the function.

Thanks,
Dave



>
> Best,
> Marius
>

Re: [petsc-users] PetscObjectGetComm

[Marius Buerkle]

Actually I am using fortran. But I also tried with C and yes I am casting A to PetscObject.

 
 

 


To confirm, are you casting A to PetscObject? e.g.
 

     MPI_Comm comm;

     /* ... */
     PetscObjectGetComm((PetscObject)A,);

 

 


Am Mi., 22. Apr. 2020 um 07:11 Uhr schrieb Marius Buerkle :

Hi,

What is PetscObjectGetComm expected to return? I thought it would give the MPI communicator the object lives on. So if I create A matrix on PETSC_COMM_WORLD a call of PetscObjectGetComm for A it would return PETSC_COMM_WORLD? But it seems to return something else, and while most of the nodes return a similar communicator some are giving a different one. That said, is there a way to get the MPI communicator a matrix lives on?

Best,
Marius

Re: [petsc-users] PetscObjectGetComm

[Patrick Sanan]

To confirm, are you casting A to PetscObject? e.g.

 MPI_Comm comm;
 /* ... */
 PetscObjectGetComm((PetscObject)A,);


Am Mi., 22. Apr. 2020 um 07:11 Uhr schrieb Marius Buerkle :

> Hi,
>
> What is PetscObjectGetComm expected to return? I thought it would give the
> MPI communicator the object lives on. So if I create A matrix on
> PETSC_COMM_WORLD a call of PetscObjectGetComm for A it would return
> PETSC_COMM_WORLD? But it seems to return something else, and while most of
> the nodes return a similar communicator some are giving a different one.
> That said, is there a way to get the MPI communicator a matrix lives on?
>
> Best,
> Marius
>

[petsc-users] PetscObjectGetComm

[Marius Buerkle]

Hi,

What is PetscObjectGetComm expected to return? I thought it would give the MPI 
communicator the object lives on. So if I create A matrix on PETSC_COMM_WORLD a 
call of PetscObjectGetComm for A it would return PETSC_COMM_WORLD? But it seems 
to return something else, and while most of the nodes return a similar 
communicator some are giving a different one. That said, is there a way to get 
the MPI communicator a matrix lives on?

Best,
Marius