For one change it to PETSC_HAVE_MAGMA And make sure you add it after a petsc include.
i.e: >>>>> #include <../src/mat/impls/dense/seq/dense.h> /*I "petscmat.h" I*/ #include <petscblaslapack.h> #include <../src/mat/impls/aij/seq/aij.h> #if defined(PETSC_HAVE_MAGMA) #if !defined(HAVE_CUBLAS) #define HAVE_CUBLAS #endif #include "magma.h" #endif <<<<<< Satish On Thu, 29 Aug 2013, Harshad Sahasrabudhe wrote: > Hi Satish, > > It worked out. The HAVE_CUBLAS problem was still there. I'm not really sure > why it wasn't defining HAVE_CUBLAS before. I tested the code by putting it > outside PETSC_HAVE_LIBAMAGMA #ifdef and it compiled. I was also getting some > BLAS LAPACK mangling errors, as ADD_ etc. was supposed to be defined. But > they went away after I put in HAVE_CUBLAS. > > I don't know why they have designed MAGMA like this. It's a very wrong way to > let the user set these flags. > > I got the code compiled and now I'm testing it. > > Thanks, > Harshad > > ----- Original Message ----- > From: "Satish Balay" <[email protected]> > To: "Harshad Sahasrabudhe" <[email protected]> > Cc: "petsc-dev" <[email protected]> > Sent: Thursday, August 29, 2013 12:01:41 PM > Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat > > Lets step back. > > Since you are adding conditional #ifdefs to dense.c - its not > appropriate to add #requirespackage to the corresponding makefile. > [as dense.c should be compiled with or without magma] > > Secondly - you should use PETSC_HAVE_MAGMA - not PETSC_HAVE_LIBMAGMA > > If you still have errors - send the logs again. > > Also send the following: > > dense.c > > preprocessed dense.c. i.e: > > cd src/mat/impl/dense/seq > make dense.o CFLAGS=-E > > [the preprocessed output might be in dense.o] > > Satish > > On Wed, 28 Aug 2013, Harshad Sahasrabudhe wrote: > > > I even tried this with a fresh git pull from Hg repository. I added > > magma.py for configuring with magma. Then I added only the following in > > dense.c > > > > #if defined(PETSC_HAVE_LIBMAGMA) > > #if !defined(HAVE_CUBLAS) > > #define HAVE_CUBLAS > > #endif > > #include "magma.h" > > #endif > > > > Then I followed the procedure I mentioned below, and I can reproduce the > > problem. Now I'm certain that removing #requirespackage is causing it. > > > > 1) remove #requirespackage from src/mat/impl/dense/seq/makefile > > 2) remove build directory > > 3) remove externalpackages directory > > 4) configure && make > > I get these errors. > > 5) add #requirespackage to src/mat/impl/dense/seq/makefile > > 6) remove build directory > > 7) remove externalpackages directory > > 8) configure && make > > No errors. > > > > Harshad > > > > > > > > >>>>>>>> > > /usr/bin/gcc -Dpetsc_EXPORTS -D__INSDIR__="" -malign-double -fPIC -g3 > > -fno-inline -O0 -fPIC -I/home/harshad/workspace/petsc-hg/include > > -I/home/harshad/workspace/petsc-hg/arch-linux2-c-debug/include > > -I/home/harshad/Downloads/magma-1.4.0/include > > -I/home/harshad/cuda-5.5/include > > -I/home/harshad/workspace/petsc-hg/include/mpiuni -o > > CMakeFiles/petsc.dir/src/mat/impls/dense/seq/dense.c.o -c > > /home/harshad/workspace/petsc-hg/src/mat/impls/dense/seq/dense.c > > In file included from > > /home/harshad/Downloads/magma-1.4.0/include/magmablas_z.h:14:0, > > from > > /home/harshad/Downloads/magma-1.4.0/include/magmablas.h:12, > > from > > /home/harshad/Downloads/magma-1.4.0/include/magma.h:15, > > from > > /home/harshad/workspace/petsc-hg/src/mat/impls/dense/seq/dense.c:10: > > /home/harshad/Downloads/magma-1.4.0/include/magma_types.h:146:6: error: > > #error "One of HAVE_CUBLAS, HAVE_clAmdBlas, or HAVE_MIC must be defined. > > This typically happens in Makefile.internal." > > <<<<<<< > > > > On Wed, 28 Aug 2013, Harshad Sahasrabudhe wrote: > > > > > Attached make.log > > > > > > ----- Original Message ----- > > > From: "Harshad Sahasrabudhe" <[email protected]> > > > To: "petsc-dev" <[email protected]> > > > Cc: "Barry Smith" <[email protected]> > > > Sent: Wednesday, August 28, 2013 4:33:23 PM > > > Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat > > > > > > I'm getting errors when compiling with MAGMA and without the > > > #requirespackage tag in the makefile. It builds correctly if I have > > > #requirespackage in the makefile. There are lots of these errors with > > > different type names: > > > > > > magma-1.4.0/include/magma_s.h:607:38: error: unknown type name > > > ‘magma_queue_t’ > > > > > > Harshad > > > > > > ----- Original Message ----- > > > From: "Satish Balay" <[email protected]> > > > To: "Harshad Sahasrabudhe" <[email protected]> > > > Cc: "Barry Smith" <[email protected]>, "For users of the development > > > version of PETSc" <[email protected]> > > > Sent: Wednesday, August 28, 2013 4:26:20 PM > > > Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat > > > > > > If you have conditional code in dense.c - then you don't need > > > "#requirespackage 'PETSC_HAVE_LIBMAGMA'" in the makefile > > > > > > Are you getting errors when compiling with Magma or without magma? > > > What errors? > > > > > > Satish > > > > > > > > > On Wed, 28 Aug 2013, Harshad Sahasrabudhe wrote: > > > > > > > Hi Barry, > > > > > > > > I'm trying to compile dense.c with the MAGMA code in it. Now the MAGMA > > > > code is protected by #if defined statements so it is only compiled when > > > > we have MAGMA. > > > > > > > > When we have MAGMA, I get errors while compiling, if we don't have > > > > #requirespackage 'PETSC_HAVE_LIBMAGMA' in the Makefile. We obviously > > > > don't require MAGMA to compile dense.c. Is there any other way to tell > > > > make that MAGMA lib has to be used for compiling dense.c when it is > > > > present? > > > > > > > > I have attached the makefile inside src/mat/impls/dense/seq folder. > > > > > > > > Thanks, > > > > Harshad > > > > > > > > ----- Original Message ----- > > > > From: "Barry Smith" <[email protected]> > > > > To: "Harshad Sahasrabudhe" <[email protected]> > > > > Cc: "For users of the development version of PETSc" > > > > <[email protected]> > > > > Sent: Tuesday, August 27, 2013 11:20:44 PM > > > > Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat > > > > > > > > > > > > Does the LU factorization use magma on the gpu now work? You told us > > > > that for factorization magma allocated the memory on the GPU and copied > > > > the data over everything over automatically. If the factorization and > > > > solve stuff all works then we can talk about the MatMult() and other > > > > stuff but otherwise let's get the factorization all working before > > > > moving on to the "harder" stuff. > > > > > > > > barry > > > > > > > > > > > > > > > > On Aug 27, 2013, at 10:12 PM, Harshad Sahasrabudhe > > > > <[email protected]> wrote: > > > > > > > > > Hi Barry, > > > > > > > > > > The functions MatCreate_SeqDense and > > > > > MatSeqDenseSetPreallocation_SeqDense do not handle memory allocation > > > > > on GPU. I can see 2 ways to implement this: > > > > > > > > > > 1) Modify these functions so that they allocate memory on host or > > > > > device when the solver is magma. I don't know how to do this. > > > > > > > > > > 2) Copy the relevant functions from dense.c to magma.c. Make a magma > > > > > matrix class which is very similar to the dense matrix class PETSc > > > > > uses, with some additional flags. Modify the MatMatMult, > > > > > MatGetFactor, MatSolve functions so that they use magma functions in > > > > > the proper way. I can do this. > > > > > > > > > > Can you please advise on how I should progress? > > > > > > > > > > Thanks, > > > > > Harshad > > > > > > > > > > ----- Original Message ----- > > > > > From: "Barry Smith" <[email protected]> > > > > > To: "Harshad Sahasrabudhe" <[email protected]> > > > > > Cc: "For users of the development version of PETSc" > > > > > <[email protected]> > > > > > Sent: Tuesday, August 27, 2013 8:43:57 AM > > > > > Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat > > > > > > > > > > > > > > > Both _p_Mat and _p_Vec have void pointers that can be used for the > > > > > gpu memory as well as flags that could potentially be used. See below > > > > > > > > > > struct _p_Mat { > > > > > PETSCHEADER(struct _MatOps); > > > > > …. > > > > > PetscBool nooffprocentries,nooffproczerorows; > > > > > #if defined(PETSC_HAVE_CUSP) > > > > > PetscCUSPFlag valid_GPU_matrix; /* flag pointing to the > > > > > matrix on the gpu*/ > > > > > #endif > > > > > #if defined(PETSC_HAVE_VIENNACL) > > > > > PetscViennaCLFlag valid_GPU_matrix; /* flag pointing to the > > > > > matrix on the gpu*/ > > > > > #endif > > > > > void *spptr; /* pointer for special > > > > > library like SuperLU */ > > > > > > > > > > On Aug 27, 2013, at 12:20 AM, Harshad Sahasrabudhe > > > > > <[email protected]> wrote: > > > > > > > > > >> Hi Barry, > > > > >> > > > > >> Thanks, MatGetFactor() now much more clear. > > > > >> > > > > >> However, I have run into trouble with the device memory allocation > > > > >> for Mat_SeqDense. How do I figure out whether the allocated memory > > > > >> is on host or device in MatSeqDenseSetPreallocation_SeqDense() ? > > > > >> > > > > >> For some functions such as magma_xgetrf_gpu and magma_xgetrs_gpu, I > > > > >> need the memory allocated on the device. As the preallocation is > > > > >> done in MatSeqDenseSetPreallocation_SeqDense for dense Matrices, I > > > > >> want to write a similar function. But checking for if(!data) is not > > > > >> enough as I need to make sure that the allocated memory is on the > > > > >> device. > > > > >> > > > > >> Is it a good idea to make another Mat class for magma which has 2 > > > > >> pointers, one for the host memory and one for device memory (only > > > > >> one of which should point to data at any given time) ? > > > > >> > > > > >> Harshad > > > > >> > > > > >> ----- Original Message ----- > > > > >> From: "Barry Smith" <[email protected]> > > > > >> To: "Harshad Sahasrabudhe" <[email protected]> > > > > >> Cc: "For users of the development version of PETSc" > > > > >> <[email protected]> > > > > >> Sent: Monday, August 26, 2013 6:19:21 PM > > > > >> Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat > > > > >> > > > > >> > > > > >> On Aug 26, 2013, at 4:24 PM, Harshad Sahasrabudhe > > > > >> <[email protected]> wrote: > > > > >> > > > > >>> Hi Barry, > > > > >>> > > > > >>> Here is my analysis, please correct if I'm wrong: > > > > >>> > > > > >>> For using the MatGetFactor() mechanism, I need to define > > > > >>> MATSOLVERMAGMA in petscmat.h. I also need to define the functions > > > > >>> MatFactorGetSolverPackage_SeqDense_magma(), > > > > >>> MatGetFactor_SeqDense_magma() in dense.c > > > > >>> > > > > >>> I'm not creating a new matrix class, so I think I don't need to add > > > > >>> anything to MatRegisterAll() > > > > >>> > > > > >>> I don't understand the following: > > > > >>> How does PETSc figure out that there is an additional solver > > > > >>> present? Do I need to define any other variables or add anything > > > > >>> for this? > > > > >> > > > > >> In MatCreate_SeqDense() note the line > > > > >> ierr = > > > > >> PetscObjectComposeFunction((PetscObject)B,"MatGetFactor_petsc_C",MatGetFactor_seqdense_petsc);CHKERRQ(ierr); > > > > >> > > > > >> you will add an additional line (protected by ifdef for magma being > > > > >> available that registers your routine > > > > >> > > > > >> ierr = > > > > >> PetscObjectComposeFunction((PetscObject)B,"MatGetFactor_magma_C",MatGetFactor_seqdense_magma);CHKERRQ(ierr); > > > > >> > > > > >> Barrt > > > > >> > > > > >>> > > > > >> > > > > >>> What else do I need to change if I have to add the functions > > > > >>> MatSolve_SeqDense_magma() and MatMatMult_magma() (for magma_xgemm > > > > >>> dense matrix multiplication). > > > > >>> > > > > >>> Harshad > > > > >>> > > > > >>> > > > > >>> ----- Original Message ----- > > > > >>> From: "Barry Smith" <[email protected]> > > > > >>> To: "Harshad Sahasrabudhe" <[email protected]> > > > > >>> Cc: "For users of the development version of PETSc" > > > > >>> <[email protected]> > > > > >>> Sent: Tuesday, August 20, 2013 7:10:40 PM > > > > >>> Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat > > > > >>> > > > > >>> > > > > >>> How general do you plan for this "magma" matrix class to be? If it > > > > >>> all it is for is to do LU/Cholesky factorizations then you do NOT > > > > >>> need to introduce src/mat/impls/dense/seq/magma at all. Simply use > > > > >>> the MatGetFactor() mechanism to "How will the user be able to > > > > >>> access this function?". This is definitely the easy way to go. > > > > >>> > > > > >>> If you want to use the "magma" matrix class for all kinds of > > > > >>> non-factorization matrix operations then you need to write a full > > > > >>> magma matrix class. I don't recommend this. > > > > >>> > > > > >>> Barry > > > > >>> > > > > >>> On Aug 20, 2013, at 1:35 PM, Harshad Sahasrabudhe > > > > >>> <[email protected]> wrote: > > > > >>> > > > > >>>> Hi, > > > > >>>> > > > > >>>> I am working on adding the Seq Dense MAGMA matrix type. For this, > > > > >>>> I have created the following files and directories: > > > > >>>> > > > > >>>> src/mat/impls/dense/seq/magma > > > > >>>> src/mat/impls/dense/seq/magma/magma.c > > > > >>>> src/mat/impls/dense/seq/magma/makefile > > > > >>>> > > > > >>>> Right now I am just trying the make LU factorization work through > > > > >>>> MAGMA. Barry had suggested looking at dense.c, as the MAGMA > > > > >>>> function for LU has the same sequence as the LAPACK function > > > > >>>> getrf. MAGMA does the memory allocation on the GPU inside it's > > > > >>>> function magma_?getrf. So can I directly use the matrix type > > > > >>>> declared in dense.c, and just define a function > > > > >>>> MatLUFactor_SeqDense_MAGMA magma.c which uses the same matrix type? > > > > >>>> > > > > >>>> How will the user be able to access this function? > > > > >>>> > > > > >>>> Thanks, > > > > >>>> Harshad > > > > >>> > > > > >> > > > > > > > > > > > > > > > > > > >
