You'll never benefit from having coarser levels on the CPU but I guess you
need a general mechanism to try and see that for yourself.
I think it should be a property of the DM and never let the PCMG see it. So
something like DMSetUseGPUWhen(dm, gpuvectype, gpumattype,localsize) the
command line option -dm_use_gpu_when [gpumattype,gpuvectype],[localsize] where
the localsize defaults to what turns out to be generally the best in our
experiments (which may depend on the machine). Then instead of using
dm->mattype and dm->vectype in the code we would always inquire with
DMGetGetType(dm, *type) (same for vectors)
if !dm->usegpu *type = dm->mattype;
else if localsize > dm->gpulocalsize *type = dm->gpumattype
else *type = dm->mat type;
Note that the current model will also continue work so someone can still do
-dm_vec_type cuda
Alternative model would be to use the GPU vector and matrix types everywhere
and simply pin the coarser matrices and vecs to the CPU so have
DMPinToCPUWhen(dm,localsize) and then
DMCreateMatrix_YYY(dm,&mat) would look like
MatCreate()
MatSetType(*mat,dm->mat type);
if localsize < dm->localsize MatPinToCPU(*mat,PETSC_TRUE);
I actually like the second model better. Note that pinning means that no space
is used on the mat/vec on the GPU unless unpinned. But the mat/vec can be
unpinned at anytime. With the first model you are stuck with it always what you
chose initially stuck on the CPU. In fact you could pin at say 1000 localsize,
run for a while, and then the code could decide to pin or unpin more to find an
"optimal" value. Thus I really like pinning.
Also we don't want to hardwire codes/commend lines to particular matrix types
but allow any GPU that is available so have things like MATGPU that default to
what is available. So I can do -dm_mat_type gpu -dm_use_gpu_when 500 and it
uses cuda if that is available and ViennaCL if it is available.
Better ideas, even more general?
Note that if dm->localsize which is a property of the dm and its partition is
different on different processes that is fine, on some process you can have a
VecGPU and others not or pinned on some and not on others for the same vector.
If you play with either of these models please do it off of my branch, not
master.
> On Jun 12, 2019, at 11:38 AM, Mills, Richard Tran via petsc-dev
> <[email protected]> wrote:
>
> Colleagues,
>
> I think we ought to have a way to control which levels of a PETSc multigrid
> solve happen on the GPU vs. the CPU, as I'd like to keep coarse levels on the
> CPU, but run the calculations for finer levels on the GPU. Currently, for a
> code that is using a DM to manage its grid, one can use GPUs inside the
> application of PCMG by doing putting something like
>
> -dm_mat_type aijcusparse -dm_vec_type cuda
>
> on the command line. What I'd like to be able to do is to also control which
> levels get plain AIJ matrices and which get a GPU type, maybe via something
> like
>
> -mg_levels_N_dm_mat_type aijcusparse -mg_levels_N_dm_mat_type cuda
>
> for level N. (Being able to specify a range of levels would be even nicer,
> but let's start simple.)
>
> Maybe doing the above is as simple as making sure that DMSetFromOptions()
> gets called for the DM for each level. But I think I may be not understanding
> some sort of additional complications. Can someone who knows the PCMG
> framework better chime in? Or do others have ideas for a more elegant way of
> giving this sort of control to the user?
>
> Best regards,
> Richard
