________________________________________ From: petsc-dev-bounces at mcs.anl.gov [petsc-dev-bounces at mcs.anl.gov] On Behalf Of Jed Brown [[email protected]] Sent: Tuesday, June 07, 2011 2:40 PM To: For users of the development version of PETSc Subject: Re: [petsc-dev] Integrating PFLOTRAN, PETSC & SAMRAI
On Tue, Jun 7, 2011 at 20:24, Philip, Bobby <philipb at ornl.gov> wrote: > The overhead of constructing a native PETSc vector across all refinement > levels would involve writing a global ordering > scheme that numbers nodes/cells/edges etc (depending on the data centering) > and then doing maps back and forth. For my larger simulations > the cost for this would be too high because this would not be a one time > cost. > Did you profile this? The cost seems very dependent on the implementation. The fundamental part is basically a copy from contiguous to discontiguous data. That memcpy for each patch might not be very expensive compared to e.g. the traversal. The issue is that for a given patch only a subset of the cells which do not have finer refinement cells would do the memcpy. This would be a very irregular data operation. I have not profiled this as I would first have to write the global ordering etc. Is this whole hierarchy manipulated directly by the user, or is there an iterator for patches? Could an iterator be made to extract the patch with ghosts from a contiguous PETSc vector? Again, the subset of cells only being a part of the PETSc vector would be the issue that is complicating. > I would have to do this each time PETSc KSP made a call to > one of my multilevel preconditioners such as FAC. > It seems to me that setup should only need to happen when the mesh adapts. Are you adapting the mesh inside of a KSPSolve? (I think that would be strange.) Every apply call from KSP does need to initialize data over the whole hierarchy that is consistent across the levels.
