It defines both - but they mean different things. PETSC_HAVE_MAGMA => configure found package 'MAGMA'
PETSC_HAVE_LIBMAGMA => configure found a library by the name libmagma [for all we know the next version could be libmagma2.a - and then the flag set by configure here would be PETSC_HAVE_LIBMAGMA2] Satish On Thu, 29 Aug 2013, [email protected] wrote: > But PETSc defines PETSC_HAVE_LIBMAGMA in petscconf.h by default. I thought it > would be PETSC_HAVE_MAGMA but it wasn't. > > ----- Reply message ----- > From: "Satish Balay" <[email protected]> > To: "Harshad Sahasrabudhe" <[email protected]> > Cc: "petsc-dev" <[email protected]> > Subject: [petsc-dev] Adding Seq Dense MAGMA Mat > Date: Thu, Aug 29, 2013 12:14 PM > > > For one change it to PETSC_HAVE_MAGMA > > And make sure you add it after a petsc include. > > i.e: > > >>>>> > > #include <../src/mat/impls/dense/seq/dense.h> /*I "petscmat.h" I*/ > #include <petscblaslapack.h> > > #include <../src/mat/impls/aij/seq/aij.h> > > #if defined(PETSC_HAVE_MAGMA) > #if !defined(HAVE_CUBLAS) > #define HAVE_CUBLAS > #endif > #include "magma.h" > #endif > <<<<<< > > Satish > > > On Thu, 29 Aug 2013, Harshad Sahasrabudhe wrote: > > > Hi Satish, > > > > It worked out. The HAVE_CUBLAS problem was still there. I'm not really sure > > why it wasn't defining HAVE_CUBLAS before. I tested the code by putting it > > outside PETSC_HAVE_LIBAMAGMA #ifdef and it compiled. I was also getting > > some BLAS LAPACK mangling errors, as ADD_ etc. was supposed to be defined. > > But they went away after I put in HAVE_CUBLAS. > > > > I don't know why they have designed MAGMA like this. It's a very wrong way > > to let the user set these flags. > > > > I got the code compiled and now I'm testing it. > > > > Thanks, > > Harshad > > > > ----- Original Message ----- > > From: "Satish Balay" <[email protected]> > > To: "Harshad Sahasrabudhe" <[email protected]> > > Cc: "petsc-dev" <[email protected]> > > Sent: Thursday, August 29, 2013 12:01:41 PM > > Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat > > > > Lets step back. > > > > Since you are adding conditional #ifdefs to dense.c - its not > > appropriate to add #requirespackage to the corresponding makefile. > > [as dense.c should be compiled with or without magma] > > > > Secondly - you should use PETSC_HAVE_MAGMA - not PETSC_HAVE_LIBMAGMA > > > > If you still have errors - send the logs again. > > > > Also send the following: > > > > dense.c > > > > preprocessed dense.c. i.e: > > > > cd src/mat/impl/dense/seq > > make dense.o CFLAGS=-E > > > > [the preprocessed output might be in dense.o] > > > > Satish > > > > On Wed, 28 Aug 2013, Harshad Sahasrabudhe wrote: > > > > > I even tried this with a fresh git pull from Hg repository. I added > > > magma.py for configuring with magma. Then I added only the following in > > > dense.c > > > > > > #if defined(PETSC_HAVE_LIBMAGMA) > > > #if !defined(HAVE_CUBLAS) > > > #define HAVE_CUBLAS > > > #endif > > > #include "magma.h" > > > #endif > > > > > > Then I followed the procedure I mentioned below, and I can reproduce the > > > problem. Now I'm certain that removing #requirespackage is causing it. > > > > > > 1) remove #requirespackage from src/mat/impl/dense/seq/makefile > > > 2) remove build directory > > > 3) remove externalpackages directory > > > 4) configure && make > > > I get these errors. > > > 5) add #requirespackage to src/mat/impl/dense/seq/makefile > > > 6) remove build directory > > > 7) remove externalpackages directory > > > 8) configure && make > > > No errors. > > > > > > Harshad > > > > > > > > > > > > >>>>>>>> > > > /usr/bin/gcc -Dpetsc_EXPORTS -D__INSDIR__="" -malign-double -fPIC -g3 > > > -fno-inline -O0 -fPIC -I/home/harshad/workspace/petsc-hg/include > > > -I/home/harshad/workspace/petsc-hg/arch-linux2-c-debug/include > > > -I/home/harshad/Downloads/magma-1.4.0/include > > > -I/home/harshad/cuda-5.5/include > > > -I/home/harshad/workspace/petsc-hg/include/mpiuni -o > > > CMakeFiles/petsc.dir/src/mat/impls/dense/seq/dense.c.o -c > > > /home/harshad/workspace/petsc-hg/src/mat/impls/dense/seq/dense.c > > > In file included from > > > /home/harshad/Downloads/magma-1.4.0/include/magmablas_z.h:14:0, > > > from > > > /home/harshad/Downloads/magma-1.4.0/include/magmablas.h:12, > > > from > > > /home/harshad/Downloads/magma-1.4.0/include/magma.h:15, > > > from > > > /home/harshad/workspace/petsc-hg/src/mat/impls/dense/seq/dense.c:10: > > > /home/harshad/Downloads/magma-1.4.0/include/magma_types.h:146:6: error: > > > #error "One of HAVE_CUBLAS, HAVE_clAmdBlas, or HAVE_MIC must be defined. > > > This typically happens in Makefile.internal." > > > <<<<<<< > > > > > > On Wed, 28 Aug 2013, Harshad Sahasrabudhe wrote: > > > > > > > Attached make.log > > > > > > > > ----- Original Message ----- > > > > From: "Harshad Sahasrabudhe" <[email protected]> > > > > To: "petsc-dev" <[email protected]> > > > > Cc: "Barry Smith" <[email protected]> > > > > Sent: Wednesday, August 28, 2013 4:33:23 PM > > > > Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat > > > > > > > > I'm getting errors when compiling with MAGMA and without the > > > > #requirespackage tag in the makefile. It builds correctly if I have > > > > #requirespackage in the makefile. There are lots of these errors with > > > > different type names: > > > > > > > > magma-1.4.0/include/magma_s.h:607:38: error: unknown type name > > > > ‘magma_queue_t’ > > > > > > > > Harshad > > > > > > > > ----- Original Message ----- > > > > From: "Satish Balay" <[email protected]> > > > > To: "Harshad Sahasrabudhe" <[email protected]> > > > > Cc: "Barry Smith" <[email protected]>, "For users of the development > > > > version of PETSc" <[email protected]> > > > > Sent: Wednesday, August 28, 2013 4:26:20 PM > > > > Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat > > > > > > > > If you have conditional code in dense.c - then you don't need > > > > "#requirespackage 'PETSC_HAVE_LIBMAGMA'" in the makefile > > > > > > > > Are you getting errors when compiling with Magma or without magma? > > > > What errors? > > > > > > > > Satish > > > > > > > > > > > > On Wed, 28 Aug 2013, Harshad Sahasrabudhe wrote: > > > > > > > > > Hi Barry, > > > > > > > > > > I'm trying to compile dense.c with the MAGMA code in it. Now the > > > > > MAGMA code is protected by #if defined statements so it is only > > > > > compiled when we have MAGMA. > > > > > > > > > > When we have MAGMA, I get errors while compiling, if we don't have > > > > > #requirespackage 'PETSC_HAVE_LIBMAGMA' in the Makefile. We obviously > > > > > don't require MAGMA to compile dense.c. Is there any other way to > > > > > tell make that MAGMA lib has to be used for compiling dense.c when it > > > > > is present? > > > > > > > > > > I have attached the makefile inside src/mat/impls/dense/seq folder. > > > > > > > > > > Thanks, > > > > > Harshad > > > > > > > > > > ----- Original Message ----- > > > > > From: "Barry Smith" <[email protected]> > > > > > To: "Harshad Sahasrabudhe" <[email protected]> > > > > > Cc: "For users of the development version of PETSc" > > > > > <[email protected]> > > > > > Sent: Tuesday, August 27, 2013 11:20:44 PM > > > > > Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat > > > > > > > > > > > > > > > Does the LU factorization use magma on the gpu now work? You told us > > > > > that for factorization magma allocated the memory on the GPU and > > > > > copied the data over everything over automatically. If the > > > > > factorization and solve stuff all works then we can talk about the > > > > > MatMult() and other stuff but otherwise let's get the factorization > > > > > all working before moving on to the "harder" stuff. > > > > > > > > > > barry > > > > > > > > > > > > > > > > > > > > On Aug 27, 2013, at 10:12 PM, Harshad Sahasrabudhe > > > > > <[email protected]> wrote: > > > > > > > > > > > Hi Barry, > > > > > > > > > > > > The functions MatCreate_SeqDense and > > > > > > MatSeqDenseSetPreallocation_SeqDense do not handle memory > > > > > > allocation on GPU. I can see 2 ways to implement this: > > > > > > > > > > > > 1) Modify these functions so that they allocate memory on host or > > > > > > device when the solver is magma. I don't know how to do this. > > > > > > > > > > > > 2) Copy the relevant functions from dense.c to magma.c. Make a > > > > > > magma matrix class which is very similar to the dense matrix class > > > > > > PETSc uses, with some additional flags. Modify the MatMatMult, > > > > > > MatGetFactor, MatSolve functions so that they use magma functions > > > > > > in the proper way. I can do this. > > > > > > > > > > > > Can you please advise on how I should progress? > > > > > > > > > > > > Thanks, > > > > > > Harshad > > > > > > > > > > > > ----- Original Message ----- > > > > > > From: "Barry Smith" <[email protected]> > > > > > > To: "Harshad Sahasrabudhe" <[email protected]> > > > > > > Cc: "For users of the development version of PETSc" > > > > > > <[email protected]> > > > > > > Sent: Tuesday, August 27, 2013 8:43:57 AM > > > > > > Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat > > > > > > > > > > > > > > > > > > Both _p_Mat and _p_Vec have void pointers that can be used for > > > > > > the gpu memory as well as flags that could potentially be used. See > > > > > > below > > > > > > > > > > > > struct _p_Mat { > > > > > > PETSCHEADER(struct _MatOps); > > > > > > …. > > > > > > PetscBool nooffprocentries,nooffproczerorows; > > > > > > #if defined(PETSC_HAVE_CUSP) > > > > > > PetscCUSPFlag valid_GPU_matrix; /* flag pointing to the > > > > > > matrix on the gpu*/ > > > > > > #endif > > > > > > #if defined(PETSC_HAVE_VIENNACL) > > > > > > PetscViennaCLFlag valid_GPU_matrix; /* flag pointing to > > > > > > the matrix on the gpu*/ > > > > > > #endif > > > > > > void *spptr; /* pointer for special > > > > > > library like SuperLU */ > > > > > > > > > > > > On Aug 27, 2013, at 12:20 AM, Harshad Sahasrabudhe > > > > > > <[email protected]> wrote: > > > > > > > > > > > >> Hi Barry, > > > > > >> > > > > > >> Thanks, MatGetFactor() now much more clear. > > > > > >> > > > > > >> However, I have run into trouble with the device memory allocation > > > > > >> for Mat_SeqDense. How do I figure out whether the allocated memory > > > > > >> is on host or device in MatSeqDenseSetPreallocation_SeqDense() ? > > > > > >> > > > > > >> For some functions such as magma_xgetrf_gpu and magma_xgetrs_gpu, > > > > > >> I need the memory allocated on the device. As the preallocation is > > > > > >> done in MatSeqDenseSetPreallocation_SeqDense for dense Matrices, I > > > > > >> want to write a similar function. But checking for if(!data) is > > > > > >> not enough as I need to make sure that the allocated memory is on > > > > > >> the device. > > > > > >> > > > > > >> Is it a good idea to make another Mat class for magma which has 2 > > > > > >> pointers, one for the host memory and one for device memory (only > > > > > >> one of which should point to data at any given time) ? > > > > > >> > > > > > >> Harshad > > > > > >> > > > > > >> ----- Original Message ----- > > > > > >> From: "Barry Smith" <[email protected]> > > > > > >> To: "Harshad Sahasrabudhe" <[email protected]> > > > > > >> Cc: "For users of the development version of PETSc" > > > > > >> <[email protected]> > > > > > >> Sent: Monday, August 26, 2013 6:19:21 PM > > > > > >> Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat > > > > > >> > > > > > >> > > > > > >> On Aug 26, 2013, at 4:24 PM, Harshad Sahasrabudhe > > > > > >> <[email protected]> wrote: > > > > > >> > > > > > >>> Hi Barry, > > > > > >>> > > > > > >>> Here is my analysis, please correct if I'm wrong: > > > > > >>> > > > > > >>> For using the MatGetFactor() mechanism, I need to define > > > > > >>> MATSOLVERMAGMA in petscmat.h. I also need to define the functions > > > > > >>> MatFactorGetSolverPackage_SeqDense_magma(), > > > > > >>> MatGetFactor_SeqDense_magma() in dense.c > > > > > >>> > > > > > >>> I'm not creating a new matrix class, so I think I don't need to > > > > > >>> add anything to MatRegisterAll() > > > > > >>> > > > > > >>> I don't understand the following: > > > > > >>> How does PETSc figure out that there is an additional solver > > > > > >>> present? Do I need to define any other variables or add anything > > > > > >>> for this? > > > > > >> > > > > > >> In MatCreate_SeqDense() note the line > > > > > >> ierr = > > > > > >> PetscObjectComposeFunction((PetscObject)B,"MatGetFactor_petsc_C",MatGetFactor_seqdense_petsc);CHKERRQ(ierr); > > > > > >> > > > > > >> you will add an additional line (protected by ifdef for magma > > > > > >> being available that registers your routine > > > > > >> > > > > > >> ierr = > > > > > >> PetscObjectComposeFunction((PetscObject)B,"MatGetFactor_magma_C",MatGetFactor_seqdense_magma);CHKERRQ(ierr); > > > > > >> > > > > > >> Barrt > > > > > >> > > > > > >>> > > > > > >> > > > > > >>> What else do I need to change if I have to add the functions > > > > > >>> MatSolve_SeqDense_magma() and MatMatMult_magma() (for magma_xgemm > > > > > >>> dense matrix multiplication). > > > > > >>> > > > > > >>> Harshad > > > > > >>> > > > > > >>> > > > > > >>> ----- Original Message ----- > > > > > >>> From: "Barry Smith" <[email protected]> > > > > > >>> To: "Harshad Sahasrabudhe" <[email protected]> > > > > > >>> Cc: "For users of the development version of PETSc" > > > > > >>> <[email protected]> > > > > > >>> Sent: Tuesday, August 20, 2013 7:10:40 PM > > > > > >>> Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat > > > > > >>> > > > > > >>> > > > > > >>> How general do you plan for this "magma" matrix class to be? If > > > > > >>> it all it is for is to do LU/Cholesky factorizations then you do > > > > > >>> NOT need to introduce src/mat/impls/dense/seq/magma at all. > > > > > >>> Simply use the MatGetFactor() mechanism to "How will the user be > > > > > >>> able to access this function?". This is definitely the easy way > > > > > >>> to go. > > > > > >>> > > > > > >>> If you want to use the "magma" matrix class for all kinds of > > > > > >>> non-factorization matrix operations then you need to write a full > > > > > >>> magma matrix class. I don't recommend this. > > > > > >>> > > > > > >>> Barry > > > > > >>> > > > > > >>> On Aug 20, 2013, at 1:35 PM, Harshad Sahasrabudhe > > > > > >>> <[email protected]> wrote: > > > > > >>> > > > > > >>>> Hi, > > > > > >>>> > > > > > >>>> I am working on adding the Seq Dense MAGMA matrix type. For > > > > > >>>> this, I have created the following files and directories: > > > > > >>>> > > > > > >>>> src/mat/impls/dense/seq/magma > > > > > >>>> src/mat/impls/dense/seq/magma/magma.c > > > > > >>>> src/mat/impls/dense/seq/magma/makefile > > > > > >>>> > > > > > >>>> Right now I am just trying the make LU factorization work > > > > > >>>> through MAGMA. Barry had suggested looking at dense.c, as the > > > > > >>>> MAGMA function for LU has the same sequence as the LAPACK > > > > > >>>> function getrf. MAGMA does the memory allocation on the GPU > > > > > >>>> inside it's function magma_?getrf. So can I directly use the > > > > > >>>> matrix type declared in dense.c, and just define a function > > > > > >>>> MatLUFactor_SeqDense_MAGMA magma.c which uses the same matrix > > > > > >>>> type? > > > > > >>>> > > > > > >>>> How will the user be able to access this function? > > > > > >>>> > > > > > >>>> Thanks, > > > > > >>>> Harshad > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >
