Hi Barry, The functions MatCreate_SeqDense and MatSeqDenseSetPreallocation_SeqDense do not handle memory allocation on GPU. I can see 2 ways to implement this:
1) Modify these functions so that they allocate memory on host or device when the solver is magma. I don't know how to do this. 2) Copy the relevant functions from dense.c to magma.c. Make a magma matrix class which is very similar to the dense matrix class PETSc uses, with some additional flags. Modify the MatMatMult, MatGetFactor, MatSolve functions so that they use magma functions in the proper way. I can do this. Can you please advise on how I should progress? Thanks, Harshad ----- Original Message ----- From: "Barry Smith" <[email protected]> To: "Harshad Sahasrabudhe" <[email protected]> Cc: "For users of the development version of PETSc" <[email protected]> Sent: Tuesday, August 27, 2013 8:43:57 AM Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat Both _p_Mat and _p_Vec have void pointers that can be used for the gpu memory as well as flags that could potentially be used. See below struct _p_Mat { PETSCHEADER(struct _MatOps); …. PetscBool nooffprocentries,nooffproczerorows; #if defined(PETSC_HAVE_CUSP) PetscCUSPFlag valid_GPU_matrix; /* flag pointing to the matrix on the gpu*/ #endif #if defined(PETSC_HAVE_VIENNACL) PetscViennaCLFlag valid_GPU_matrix; /* flag pointing to the matrix on the gpu*/ #endif void *spptr; /* pointer for special library like SuperLU */ On Aug 27, 2013, at 12:20 AM, Harshad Sahasrabudhe <[email protected]> wrote: > Hi Barry, > > Thanks, MatGetFactor() now much more clear. > > However, I have run into trouble with the device memory allocation for > Mat_SeqDense. How do I figure out whether the allocated memory is on host or > device in MatSeqDenseSetPreallocation_SeqDense() ? > > For some functions such as magma_xgetrf_gpu and magma_xgetrs_gpu, I need the > memory allocated on the device. As the preallocation is done in > MatSeqDenseSetPreallocation_SeqDense for dense Matrices, I want to write a > similar function. But checking for if(!data) is not enough as I need to make > sure that the allocated memory is on the device. > > Is it a good idea to make another Mat class for magma which has 2 pointers, > one for the host memory and one for device memory (only one of which should > point to data at any given time) ? > > Harshad > > ----- Original Message ----- > From: "Barry Smith" <[email protected]> > To: "Harshad Sahasrabudhe" <[email protected]> > Cc: "For users of the development version of PETSc" <[email protected]> > Sent: Monday, August 26, 2013 6:19:21 PM > Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat > > > On Aug 26, 2013, at 4:24 PM, Harshad Sahasrabudhe <[email protected]> wrote: > >> Hi Barry, >> >> Here is my analysis, please correct if I'm wrong: >> >> For using the MatGetFactor() mechanism, I need to define MATSOLVERMAGMA in >> petscmat.h. I also need to define the functions >> MatFactorGetSolverPackage_SeqDense_magma(), MatGetFactor_SeqDense_magma() in >> dense.c >> >> I'm not creating a new matrix class, so I think I don't need to add anything >> to MatRegisterAll() >> >> I don't understand the following: >> How does PETSc figure out that there is an additional solver present? Do I >> need to define any other variables or add anything for this? > > In MatCreate_SeqDense() note the line > ierr = > PetscObjectComposeFunction((PetscObject)B,"MatGetFactor_petsc_C",MatGetFactor_seqdense_petsc);CHKERRQ(ierr); > > you will add an additional line (protected by ifdef for magma being available > that registers your routine > > ierr = > PetscObjectComposeFunction((PetscObject)B,"MatGetFactor_magma_C",MatGetFactor_seqdense_magma);CHKERRQ(ierr); > > Barrt > >> > >> What else do I need to change if I have to add the functions >> MatSolve_SeqDense_magma() and MatMatMult_magma() (for magma_xgemm dense >> matrix multiplication). >> >> Harshad >> >> >> ----- Original Message ----- >> From: "Barry Smith" <[email protected]> >> To: "Harshad Sahasrabudhe" <[email protected]> >> Cc: "For users of the development version of PETSc" <[email protected]> >> Sent: Tuesday, August 20, 2013 7:10:40 PM >> Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat >> >> >> How general do you plan for this "magma" matrix class to be? If it all it >> is for is to do LU/Cholesky factorizations then you do NOT need to introduce >> src/mat/impls/dense/seq/magma at all. Simply use the MatGetFactor() >> mechanism to "How will the user be able to access this function?". This is >> definitely the easy way to go. >> >> If you want to use the "magma" matrix class for all kinds of >> non-factorization matrix operations then you need to write a full magma >> matrix class. I don't recommend this. >> >> Barry >> >> On Aug 20, 2013, at 1:35 PM, Harshad Sahasrabudhe <[email protected]> >> wrote: >> >>> Hi, >>> >>> I am working on adding the Seq Dense MAGMA matrix type. For this, I have >>> created the following files and directories: >>> >>> src/mat/impls/dense/seq/magma >>> src/mat/impls/dense/seq/magma/magma.c >>> src/mat/impls/dense/seq/magma/makefile >>> >>> Right now I am just trying the make LU factorization work through MAGMA. >>> Barry had suggested looking at dense.c, as the MAGMA function for LU has >>> the same sequence as the LAPACK function getrf. MAGMA does the memory >>> allocation on the GPU inside it's function magma_?getrf. So can I directly >>> use the matrix type declared in dense.c, and just define a function >>> MatLUFactor_SeqDense_MAGMA magma.c which uses the same matrix type? >>> >>> How will the user be able to access this function? >>> >>> Thanks, >>> Harshad >> >
