Fredrik,
This question belongs on petsc-dev at mcs.anl.gov since it involves
additions/extensions to PETSc so I am moving the discussion over to there.
We have not done the required work to have ghosted vectors work with CUSP
yet, so this will require some additions to PETSc. We can help you with that
process but since the PETSc team does not have a CUSP person developing PETSc
full time you will need to actual contribute some code but I'll try to guide
you in the right direction.
The first observation is that ghosted vectors in PETSc are actually
handled with largely the same code as VECMPI vectors (with just no ghost points
by default) so in theory little work needs to be done to get the functionality
you need. What makes the needed changes non-trivial is the current interface
where one calls VecCreateGhost() to create the vectors. This is one of our
"easy" interfaces and it is somewhat legacy in that there is no way to control
the types of the vectors since it creates everything about the vector in one
step. Note that we have the same issues with regard to the pthread versions
of the PETSc vectors and ghosting.
So before we even talk about what code to change/add we need to decide on
the interface. Presumably you want to be able to decide at runtime whether to
use regular VECMPI, VECMPICUSP or VECMPIPTHREAD in your ghosted vectors. How
do we get that information in there? An additional argument to VecCreateGhost()
(ugly?)? Options database (by calling VecSetFromOptions() ?), other ways? So
for example one could have:
VecCreateGhost(......)
VecSetFromOptions(......)
to set the specific type cusp or pthread? What about
VecCreateGhost(......)
VecSetType(......,VECMPICUSP);
which as you note doesn't currently work. Note that the PTHREAD version needs
to do its own memory allocation so essentially has to undo much of what
VecCreateGhost() already did, is that a bad thing?
Or do we get rid of VecCreateGhost() completely and change the model to
something like
VecCreate()
VecSetType()
VecSetGhosted()
or
VecCreate()
VecSetTypeFromOptions()
VecSetGhosted()
or even
VecCreate()
VecSetGhosted() which will just default to regular MPI ghosted.
this model allows a clean implementation that doesn't require undoing
previously built internals.
Everyone chime in with observations so we can figure out any refactorizations
needed.
Barry
On Feb 6, 2012, at 8:33 AM, Fredrik Heffer Valdmanis wrote:
> Hi,
>
> In FEniCS, we use ghosted vectors for parallel computations, with the
> functions
>
> VecCreateGhost
> VecGhostGetLocalForm
>
> As I am integrating the new GPU-based vectors and matrices in FEniCS, I want
> the ghosted vectors to be of type VECMPICUSP. I have tried to do this by
> calling VecSetType after creating the vector, but that makes
> VecGhostGetLocalForm give an error.
>
> Is it possible to set the type to be mpicusp when using ghost vectors,
> without changing much of the code? If so, how?
>
> If not, how would you recommend I proceed to work with mpicusp vectors in
> this context?
>
> Thanks!
>
> --
> Fredrik
