This is all super helpful! Thanks. It seems to me that we do not need to carry around a reference to the communicator in Julia then.
Mainly I wanted to use `PetscObjectGetComm` everywhere once `PetscObjects` were created, but someone pointed out this might run in to problems with the garbage collector. In my mind it makes sense to rely on `PetscObjectGetComm` since you don’t know if this is some object that matches the object you originally created or some derived object with a different processor distribution (such as what I believe happens with multigrid). > On Jul 9, 2021, at 8:52 AM, Junchao Zhang <[email protected]> wrote: > > > NPS WARNING: *external sender* verify before acting. > > > > > > On Thu, Jul 8, 2021 at 11:21 PM Barry Smith <[email protected]> wrote: > > Whenever PETSc is handed a communicator it looks for an attribute inside > of the communicator that contains the "PETSc" version of that communicator. > If it does not find the attribute it adds an attribute with a new > communicator in, if it does find one it increases its reference count by one. > The routine it uses to perform this is PetscCommDuplicate(). We do it this > was so that PETSc communication will never potentially interfere with the > users use of their communicators. PetscCommDestroy() decreases the reference > count of the inner communicator by one. So, for example, if you use "comm" to > create two PETSc objects, PETSc will create an attribute on "comm" with a new > communicator, when both objects are destroy then PetscCommDestroy() will have > been called twice and the inner (PETSc) communicator will be destroyed. > > If someone did > > Use MPI to create a new communicator > VecCreate(comm,...) > Use MPI to destroy the new communicator > .... > VecDestroy() > The code above will work correctly. In 'Use MPI to destroy the new > communicator', MPI finds out comm has an attribute Petsc_InnerComm_keyval, > so it invokes a petsc function Petsc_InnerComm_Attr_Delete_Fn (which was > given to MPI at PetscInitialize). > In Petsc_InnerComm_Attr_Delete_Fn, it cuts the link between comm and its > inner petsc comm (which is still used by vec in this example). The inner > petsc comm is still valid and accessible via PetscObjectComm(). It will be > destroyed when its reference count (managed by petsc) reaches zero (probably > in VecDestroy). > > > I am not sure what will happen since PETSc keeps a reference to the outer > communicator from its own inner communicator. And destroying the user > communicator will cause an attempt to destroy the attribute containing the > inner PETSc communicator. I had always just assumed the user would not be > deleting any MPI communicators they made and pass to PETSc until they were > done with PETSc. It may work correctly but may not. > > The reality is very few MPI codes have complicated life cycles for MPI > communicators. > > Barry > > > > On Jul 8, 2021, at 10:17 PM, Kozdon, Jeremy (CIV) <[email protected]> wrote: > > > > Sorry if this is clearly stated somewhere in the docs, I'm still getting > > familiar with the petsc codebase and was also unable to find the answer > > searching (nor could I determine where this would be done in the source). > > > > Does petsc duplicate MPI communicators? Or does the users program need to > > make sure that the communicator remains valid for the life of a petsc > > object? > > > > The attached little test code seems to suggest that there is some > > duplication of MPI communicators behind the scenes. > > > > This came up when working on Julia wrappers for petsc. (Julia has a garbage > > collector so we need to make sure that references are properly kept if > > needed.) > > > > <try.c> >
