On Tue, 31 Aug 2010 13:24:44 -0600, Derek Gaston <derek.gas...@inl.gov> wrote:
> On Aug 31, 2010, at 1:11 PM, Jed Brown wrote:
>
> > If a VecGhost was passed in to SNES, then all the Krylov vectors
> > will also be VecGhost. If not, create a new VecGhost
> > (VecCreateGhost() or VecDuplicate() an existing VecGhost) and
> > VecCopy from the plain Vec to the ghost Vec.
>
> Well, that is odd then. Because I'm sure that libMesh is passing in
> ghosted vectors (hmmm... maybe the residual (rhs) isn't ghosted by
> default?). Maybe libMesh is failing at detecting that they are
> ghosted? It looks like libMesh is looking for a local to global
> mapping to decide if it's ghosted or not. Is that the proper way?
Not really, you can set a LocalToGlobalMapping even for unghosted, in
order to use VecSetValuesLocal, though libmesh itself may never use this
functionality. Looking at the source, there isn't a public way to check
if a Vec has a local form. This could be added, but I'm not convinced
it's necessary (or would be good design to use). If you really need it,
I can think of two solutions:
/* Nasty because this is only available when the source tree is around. */
#include <../src/vec/vec/impls/mpi/pvecimpl.h>
if (vec->localrep) { /* is ghosted */ }
OR
PetscPushErrorHandler(PetscReturnErrorHandler,0);
ierr = VecGhostGetLocalForm(X,&loc);
PetscPopErrorHandler();
if (!err) { /* is ghosted */ }
Note that all serial Vecs are trivially ghosted (the updates are
no-ops).
> Jed: how much overhead will be incurred by having all vectors in my
> SNES system be ghosted? Will it be a noticeable slowdown or
> communication?
No extra communication, the only price is the extra memory for the
ghosted part that isn't being used. That could be significant for very
small or poorly shaped subdomains. PETSc could and maybe should deal
with this by creating the Krylov space with non-ghosted vectors, but we
currently always use vanilla VecDuplicate() so that the vector
operations will be done with the appropriate types (e.g. VecCUDA).
> Yeah; the real trouble is that ghosted vectors have to fall back to
> serial when ghosting isn't available, e.g. on Trilinos. I'm going to
> assume that you don't want to implement ghosted support there?
I assume you mean working with a separate vector for the local form.
> Otherwise the only thing to do is add a non-default code path, where
> instead of letting PETSc work on solution while we work to keep
> current_local_solution sync'ed up we do the opposite.
PETSc is always going to work with a parallel vector (VecGhost is
parallel). The different code path is that with VecGhost, you would
normally VecGhostUpdate then VecGhostGetLocalForm instead of VecScatter
to an independent local vector. Are you talking about this or something
else?
Jed
------------------------------------------------------------------------------
This SF.net Dev2Dev email is sponsored by:
Show off your parallel programming skills.
Enter the Intel(R) Threading Challenge 2010.
http://p.sf.net/sfu/intel-thread-sfd
_______________________________________________
Libmesh-devel mailing list
Libmesh-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/libmesh-devel
