"Teranishi, Keita" <[email protected]> writes: > Jed, > > I am curious how you use MPI Keyvalue. Do you have any document or > source files related to this?
It is used to manage inner/outer communicators. The first time a PETSc object is created on a communicator, we dup it, creating a reference-counted inner communicator that we attach via keyval to the outer and vice-versa. So any time we need to create an object, we check: * it is an inner comm -- use it * it is an outer comm that has an attached inner -- get out the inner and use it * it has not been seen before -- create the inner and attach The inner comm has a tag dispenser and a reference count. When the reference count goes to zero, it is detached from the outer and cleaned up. This is way better for library interfaces than having a wrapper class that goes around the outside of an MPI_Comm, because with our system, communicators can be passed back and forth between libraries easily, without needing to re-dup or find the appropriate container. A story: BG/Q V1R2M0 made MPI_Comm_dup use O(P) memory per rank each time it is called. This was a catastrophe for applications that used MPI_Comm_dup for each object that used a communicator. Nek had this problem, and V1R2M0 was pushed onto Mira shortly before SIAM CSE, in order to hit some milestone that IBM needed for business reasons. A later efix resolved the problem, but it was still disruptive for Paul who had a plenary to give. You'll find the bulk of the implementation it tagm.c: https://bitbucket.org/petsc/petsc/src/a0a914e661bf6402b8edabe0f5a2dad46323f69f/src/sys/objects/tagm.c?at=master
