On Fri, Jul 9, 2021 at 12:13 PM Kozdon, Jeremy (CIV) <[email protected]> wrote:
> This is all super helpful! Thanks. > > It seems to me that we do not need to carry around a reference to the > communicator in Julia then. > > Mainly I wanted to use `PetscObjectGetComm` everywhere once `PetscObjects` > were created, but someone pointed out this might run in to problems with > the garbage collector. > > In my mind it makes sense to rely on `PetscObjectGetComm` since you don’t > know if this is some object that matches the object you originally created > or some derived object with a different processor distribution (such as > what I believe happens with multigrid). > I think you are right. > > > On Jul 9, 2021, at 8:52 AM, Junchao Zhang <[email protected]> > wrote: > > > > > > NPS WARNING: *external sender* verify before acting. > > > > > > > > > > > > On Thu, Jul 8, 2021 at 11:21 PM Barry Smith <[email protected]> wrote: > > > > Whenever PETSc is handed a communicator it looks for an attribute > inside of the communicator that contains the "PETSc" version of that > communicator. If it does not find the attribute it adds an attribute with a > new communicator in, if it does find one it increases its reference count > by one. The routine it uses to perform this is PetscCommDuplicate(). We do > it this was so that PETSc communication will never potentially interfere > with the users use of their communicators. PetscCommDestroy() decreases the > reference count of the inner communicator by one. So, for example, if you > use "comm" to create two PETSc objects, PETSc will create an attribute on > "comm" with a new communicator, when both objects are destroy then > PetscCommDestroy() will have been called twice and the inner (PETSc) > communicator will be destroyed. > > > > If someone did > > > > Use MPI to create a new communicator > > VecCreate(comm,...) > > Use MPI to destroy the new communicator > > .... > > VecDestroy() > > The code above will work correctly. In 'Use MPI to destroy the new > communicator', MPI finds out comm has an attribute Petsc_InnerComm_keyval, > so it invokes a petsc function Petsc_InnerComm_Attr_Delete_Fn (which was > given to MPI at PetscInitialize). > > In Petsc_InnerComm_Attr_Delete_Fn, it cuts the link between comm and its > inner petsc comm (which is still used by vec in this example). The inner > petsc comm is still valid and accessible via PetscObjectComm(). It will be > destroyed when its reference count (managed by petsc) reaches zero > (probably in VecDestroy). > > > > > > I am not sure what will happen since PETSc keeps a reference to the > outer communicator from its own inner communicator. And destroying the user > communicator will cause an attempt to destroy the attribute containing the > inner PETSc communicator. I had always just assumed the user would not be > deleting any MPI communicators they made and pass to PETSc until they were > done with PETSc. It may work correctly but may not. > > > > The reality is very few MPI codes have complicated life cycles for MPI > communicators. > > > > Barry > > > > > > > On Jul 8, 2021, at 10:17 PM, Kozdon, Jeremy (CIV) <[email protected]> > wrote: > > > > > > Sorry if this is clearly stated somewhere in the docs, I'm still > getting familiar with the petsc codebase and was also unable to find the > answer searching (nor could I determine where this would be done in the > source). > > > > > > Does petsc duplicate MPI communicators? Or does the users program need > to make sure that the communicator remains valid for the life of a petsc > object? > > > > > > The attached little test code seems to suggest that there is some > duplication of MPI communicators behind the scenes. > > > > > > This came up when working on Julia wrappers for petsc. (Julia has a > garbage collector so we need to make sure that references are properly kept > if needed.) > > > > > > <try.c> > > > >
