If you have conditional code in dense.c - then you don't need "#requirespackage 'PETSC_HAVE_LIBMAGMA'" in the makefile
Are you getting errors when compiling with Magma or without magma? What errors? Satish On Wed, 28 Aug 2013, Harshad Sahasrabudhe wrote: > Hi Barry, > > I'm trying to compile dense.c with the MAGMA code in it. Now the MAGMA code > is protected by #if defined statements so it is only compiled when we have > MAGMA. > > When we have MAGMA, I get errors while compiling, if we don't have > #requirespackage 'PETSC_HAVE_LIBMAGMA' in the Makefile. We obviously don't > require MAGMA to compile dense.c. Is there any other way to tell make that > MAGMA lib has to be used for compiling dense.c when it is present? > > I have attached the makefile inside src/mat/impls/dense/seq folder. > > Thanks, > Harshad > > ----- Original Message ----- > From: "Barry Smith" <[email protected]> > To: "Harshad Sahasrabudhe" <[email protected]> > Cc: "For users of the development version of PETSc" <[email protected]> > Sent: Tuesday, August 27, 2013 11:20:44 PM > Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat > > > Does the LU factorization use magma on the gpu now work? You told us that for > factorization magma allocated the memory on the GPU and copied the data over > everything over automatically. If the factorization and solve stuff all works > then we can talk about the MatMult() and other stuff but otherwise let's get > the factorization all working before moving on to the "harder" stuff. > > barry > > > > On Aug 27, 2013, at 10:12 PM, Harshad Sahasrabudhe <[email protected]> > wrote: > > > Hi Barry, > > > > The functions MatCreate_SeqDense and MatSeqDenseSetPreallocation_SeqDense > > do not handle memory allocation on GPU. I can see 2 ways to implement this: > > > > 1) Modify these functions so that they allocate memory on host or device > > when the solver is magma. I don't know how to do this. > > > > 2) Copy the relevant functions from dense.c to magma.c. Make a magma matrix > > class which is very similar to the dense matrix class PETSc uses, with some > > additional flags. Modify the MatMatMult, MatGetFactor, MatSolve functions > > so that they use magma functions in the proper way. I can do this. > > > > Can you please advise on how I should progress? > > > > Thanks, > > Harshad > > > > ----- Original Message ----- > > From: "Barry Smith" <[email protected]> > > To: "Harshad Sahasrabudhe" <[email protected]> > > Cc: "For users of the development version of PETSc" <[email protected]> > > Sent: Tuesday, August 27, 2013 8:43:57 AM > > Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat > > > > > > Both _p_Mat and _p_Vec have void pointers that can be used for the gpu > > memory as well as flags that could potentially be used. See below > > > > struct _p_Mat { > > PETSCHEADER(struct _MatOps); > > …. > > PetscBool nooffprocentries,nooffproczerorows; > > #if defined(PETSC_HAVE_CUSP) > > PetscCUSPFlag valid_GPU_matrix; /* flag pointing to the matrix on > > the gpu*/ > > #endif > > #if defined(PETSC_HAVE_VIENNACL) > > PetscViennaCLFlag valid_GPU_matrix; /* flag pointing to the > > matrix on the gpu*/ > > #endif > > void *spptr; /* pointer for special library > > like SuperLU */ > > > > On Aug 27, 2013, at 12:20 AM, Harshad Sahasrabudhe <[email protected]> > > wrote: > > > >> Hi Barry, > >> > >> Thanks, MatGetFactor() now much more clear. > >> > >> However, I have run into trouble with the device memory allocation for > >> Mat_SeqDense. How do I figure out whether the allocated memory is on host > >> or device in MatSeqDenseSetPreallocation_SeqDense() ? > >> > >> For some functions such as magma_xgetrf_gpu and magma_xgetrs_gpu, I need > >> the memory allocated on the device. As the preallocation is done in > >> MatSeqDenseSetPreallocation_SeqDense for dense Matrices, I want to write a > >> similar function. But checking for if(!data) is not enough as I need to > >> make sure that the allocated memory is on the device. > >> > >> Is it a good idea to make another Mat class for magma which has 2 > >> pointers, one for the host memory and one for device memory (only one of > >> which should point to data at any given time) ? > >> > >> Harshad > >> > >> ----- Original Message ----- > >> From: "Barry Smith" <[email protected]> > >> To: "Harshad Sahasrabudhe" <[email protected]> > >> Cc: "For users of the development version of PETSc" <[email protected]> > >> Sent: Monday, August 26, 2013 6:19:21 PM > >> Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat > >> > >> > >> On Aug 26, 2013, at 4:24 PM, Harshad Sahasrabudhe <[email protected]> > >> wrote: > >> > >>> Hi Barry, > >>> > >>> Here is my analysis, please correct if I'm wrong: > >>> > >>> For using the MatGetFactor() mechanism, I need to define MATSOLVERMAGMA > >>> in petscmat.h. I also need to define the functions > >>> MatFactorGetSolverPackage_SeqDense_magma(), MatGetFactor_SeqDense_magma() > >>> in dense.c > >>> > >>> I'm not creating a new matrix class, so I think I don't need to add > >>> anything to MatRegisterAll() > >>> > >>> I don't understand the following: > >>> How does PETSc figure out that there is an additional solver present? Do > >>> I need to define any other variables or add anything for this? > >> > >> In MatCreate_SeqDense() note the line > >> ierr = > >> PetscObjectComposeFunction((PetscObject)B,"MatGetFactor_petsc_C",MatGetFactor_seqdense_petsc);CHKERRQ(ierr); > >> > >> you will add an additional line (protected by ifdef for magma being > >> available that registers your routine > >> > >> ierr = > >> PetscObjectComposeFunction((PetscObject)B,"MatGetFactor_magma_C",MatGetFactor_seqdense_magma);CHKERRQ(ierr); > >> > >> Barrt > >> > >>> > >> > >>> What else do I need to change if I have to add the functions > >>> MatSolve_SeqDense_magma() and MatMatMult_magma() (for magma_xgemm dense > >>> matrix multiplication). > >>> > >>> Harshad > >>> > >>> > >>> ----- Original Message ----- > >>> From: "Barry Smith" <[email protected]> > >>> To: "Harshad Sahasrabudhe" <[email protected]> > >>> Cc: "For users of the development version of PETSc" > >>> <[email protected]> > >>> Sent: Tuesday, August 20, 2013 7:10:40 PM > >>> Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat > >>> > >>> > >>> How general do you plan for this "magma" matrix class to be? If it all it > >>> is for is to do LU/Cholesky factorizations then you do NOT need to > >>> introduce src/mat/impls/dense/seq/magma at all. Simply use the > >>> MatGetFactor() mechanism to "How will the user be able to access this > >>> function?". This is definitely the easy way to go. > >>> > >>> If you want to use the "magma" matrix class for all kinds of > >>> non-factorization matrix operations then you need to write a full magma > >>> matrix class. I don't recommend this. > >>> > >>> Barry > >>> > >>> On Aug 20, 2013, at 1:35 PM, Harshad Sahasrabudhe <[email protected]> > >>> wrote: > >>> > >>>> Hi, > >>>> > >>>> I am working on adding the Seq Dense MAGMA matrix type. For this, I have > >>>> created the following files and directories: > >>>> > >>>> src/mat/impls/dense/seq/magma > >>>> src/mat/impls/dense/seq/magma/magma.c > >>>> src/mat/impls/dense/seq/magma/makefile > >>>> > >>>> Right now I am just trying the make LU factorization work through MAGMA. > >>>> Barry had suggested looking at dense.c, as the MAGMA function for LU has > >>>> the same sequence as the LAPACK function getrf. MAGMA does the memory > >>>> allocation on the GPU inside it's function magma_?getrf. So can I > >>>> directly use the matrix type declared in dense.c, and just define a > >>>> function MatLUFactor_SeqDense_MAGMA magma.c which uses the same matrix > >>>> type? > >>>> > >>>> How will the user be able to access this function? > >>>> > >>>> Thanks, > >>>> Harshad > >>> > >> > > > >
