Hi Satish, The error goes away because dense.c and dense.h don't compile if the makefile has #requirespackage 'PETSC_HAVE_LIBMAGMA'. You are right, the #requirespackage makes no difference with the errors.
Harshad ----- Original Message ----- From: "Satish Balay" <[email protected]> To: "Harshad Sahasrabudhe" <[email protected]> Cc: "petsc-dev" <[email protected]> Sent: Wednesday, August 28, 2013 4:45:49 PM Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat Didn't we discuss this issue before? Perhaps dense.c is supporsed to have: #if !defined(HAVE_CUBLAS) #define HAVE_CUBLAS #endif #include "magma.h" Satish >>>>>>>> /usr/bin/gcc -Dpetsc_EXPORTS -D__INSDIR__="" -malign-double -fPIC -g3 -fno-inline -O0 -fPIC -I/home/harshad/workspace/petsc-hg/include -I/home/harshad/workspace/petsc-hg/arch-linux2-c-debug/include -I/home/harshad/Downloads/magma-1.4.0/include -I/home/harshad/cuda-5.5/include -I/home/harshad/workspace/petsc-hg/include/mpiuni -o CMakeFiles/petsc.dir/src/mat/impls/dense/seq/dense.c.o -c /home/harshad/workspace/petsc-hg/src/mat/impls/dense/seq/dense.c In file included from /home/harshad/Downloads/magma-1.4.0/include/magmablas_z.h:14:0, from /home/harshad/Downloads/magma-1.4.0/include/magmablas.h:12, from /home/harshad/Downloads/magma-1.4.0/include/magma.h:15, from /home/harshad/workspace/petsc-hg/src/mat/impls/dense/seq/dense.c:10: /home/harshad/Downloads/magma-1.4.0/include/magma_types.h:146:6: error: #error "One of HAVE_CUBLAS, HAVE_clAmdBlas, or HAVE_MIC must be defined. This typically happens in Makefile.internal." <<<<<<< On Wed, 28 Aug 2013, Harshad Sahasrabudhe wrote: > Attached make.log > > ----- Original Message ----- > From: "Harshad Sahasrabudhe" <[email protected]> > To: "petsc-dev" <[email protected]> > Cc: "Barry Smith" <[email protected]> > Sent: Wednesday, August 28, 2013 4:33:23 PM > Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat > > I'm getting errors when compiling with MAGMA and without the #requirespackage > tag in the makefile. It builds correctly if I have #requirespackage in the > makefile. There are lots of these errors with different type names: > > magma-1.4.0/include/magma_s.h:607:38: error: unknown type name ‘magma_queue_t’ > > Harshad > > ----- Original Message ----- > From: "Satish Balay" <[email protected]> > To: "Harshad Sahasrabudhe" <[email protected]> > Cc: "Barry Smith" <[email protected]>, "For users of the development version > of PETSc" <[email protected]> > Sent: Wednesday, August 28, 2013 4:26:20 PM > Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat > > If you have conditional code in dense.c - then you don't need > "#requirespackage 'PETSC_HAVE_LIBMAGMA'" in the makefile > > Are you getting errors when compiling with Magma or without magma? > What errors? > > Satish > > > On Wed, 28 Aug 2013, Harshad Sahasrabudhe wrote: > > > Hi Barry, > > > > I'm trying to compile dense.c with the MAGMA code in it. Now the MAGMA code > > is protected by #if defined statements so it is only compiled when we have > > MAGMA. > > > > When we have MAGMA, I get errors while compiling, if we don't have > > #requirespackage 'PETSC_HAVE_LIBMAGMA' in the Makefile. We obviously don't > > require MAGMA to compile dense.c. Is there any other way to tell make that > > MAGMA lib has to be used for compiling dense.c when it is present? > > > > I have attached the makefile inside src/mat/impls/dense/seq folder. > > > > Thanks, > > Harshad > > > > ----- Original Message ----- > > From: "Barry Smith" <[email protected]> > > To: "Harshad Sahasrabudhe" <[email protected]> > > Cc: "For users of the development version of PETSc" <[email protected]> > > Sent: Tuesday, August 27, 2013 11:20:44 PM > > Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat > > > > > > Does the LU factorization use magma on the gpu now work? You told us that > > for factorization magma allocated the memory on the GPU and copied the data > > over everything over automatically. If the factorization and solve stuff > > all works then we can talk about the MatMult() and other stuff but > > otherwise let's get the factorization all working before moving on to the > > "harder" stuff. > > > > barry > > > > > > > > On Aug 27, 2013, at 10:12 PM, Harshad Sahasrabudhe <[email protected]> > > wrote: > > > > > Hi Barry, > > > > > > The functions MatCreate_SeqDense and MatSeqDenseSetPreallocation_SeqDense > > > do not handle memory allocation on GPU. I can see 2 ways to implement > > > this: > > > > > > 1) Modify these functions so that they allocate memory on host or device > > > when the solver is magma. I don't know how to do this. > > > > > > 2) Copy the relevant functions from dense.c to magma.c. Make a magma > > > matrix class which is very similar to the dense matrix class PETSc uses, > > > with some additional flags. Modify the MatMatMult, MatGetFactor, MatSolve > > > functions so that they use magma functions in the proper way. I can do > > > this. > > > > > > Can you please advise on how I should progress? > > > > > > Thanks, > > > Harshad > > > > > > ----- Original Message ----- > > > From: "Barry Smith" <[email protected]> > > > To: "Harshad Sahasrabudhe" <[email protected]> > > > Cc: "For users of the development version of PETSc" > > > <[email protected]> > > > Sent: Tuesday, August 27, 2013 8:43:57 AM > > > Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat > > > > > > > > > Both _p_Mat and _p_Vec have void pointers that can be used for the gpu > > > memory as well as flags that could potentially be used. See below > > > > > > struct _p_Mat { > > > PETSCHEADER(struct _MatOps); > > > …. > > > PetscBool nooffprocentries,nooffproczerorows; > > > #if defined(PETSC_HAVE_CUSP) > > > PetscCUSPFlag valid_GPU_matrix; /* flag pointing to the matrix > > > on the gpu*/ > > > #endif > > > #if defined(PETSC_HAVE_VIENNACL) > > > PetscViennaCLFlag valid_GPU_matrix; /* flag pointing to the > > > matrix on the gpu*/ > > > #endif > > > void *spptr; /* pointer for special library > > > like SuperLU */ > > > > > > On Aug 27, 2013, at 12:20 AM, Harshad Sahasrabudhe <[email protected]> > > > wrote: > > > > > >> Hi Barry, > > >> > > >> Thanks, MatGetFactor() now much more clear. > > >> > > >> However, I have run into trouble with the device memory allocation for > > >> Mat_SeqDense. How do I figure out whether the allocated memory is on > > >> host or device in MatSeqDenseSetPreallocation_SeqDense() ? > > >> > > >> For some functions such as magma_xgetrf_gpu and magma_xgetrs_gpu, I need > > >> the memory allocated on the device. As the preallocation is done in > > >> MatSeqDenseSetPreallocation_SeqDense for dense Matrices, I want to write > > >> a similar function. But checking for if(!data) is not enough as I need > > >> to make sure that the allocated memory is on the device. > > >> > > >> Is it a good idea to make another Mat class for magma which has 2 > > >> pointers, one for the host memory and one for device memory (only one of > > >> which should point to data at any given time) ? > > >> > > >> Harshad > > >> > > >> ----- Original Message ----- > > >> From: "Barry Smith" <[email protected]> > > >> To: "Harshad Sahasrabudhe" <[email protected]> > > >> Cc: "For users of the development version of PETSc" > > >> <[email protected]> > > >> Sent: Monday, August 26, 2013 6:19:21 PM > > >> Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat > > >> > > >> > > >> On Aug 26, 2013, at 4:24 PM, Harshad Sahasrabudhe <[email protected]> > > >> wrote: > > >> > > >>> Hi Barry, > > >>> > > >>> Here is my analysis, please correct if I'm wrong: > > >>> > > >>> For using the MatGetFactor() mechanism, I need to define MATSOLVERMAGMA > > >>> in petscmat.h. I also need to define the functions > > >>> MatFactorGetSolverPackage_SeqDense_magma(), > > >>> MatGetFactor_SeqDense_magma() in dense.c > > >>> > > >>> I'm not creating a new matrix class, so I think I don't need to add > > >>> anything to MatRegisterAll() > > >>> > > >>> I don't understand the following: > > >>> How does PETSc figure out that there is an additional solver present? > > >>> Do I need to define any other variables or add anything for this? > > >> > > >> In MatCreate_SeqDense() note the line > > >> ierr = > > >> PetscObjectComposeFunction((PetscObject)B,"MatGetFactor_petsc_C",MatGetFactor_seqdense_petsc);CHKERRQ(ierr); > > >> > > >> you will add an additional line (protected by ifdef for magma being > > >> available that registers your routine > > >> > > >> ierr = > > >> PetscObjectComposeFunction((PetscObject)B,"MatGetFactor_magma_C",MatGetFactor_seqdense_magma);CHKERRQ(ierr); > > >> > > >> Barrt > > >> > > >>> > > >> > > >>> What else do I need to change if I have to add the functions > > >>> MatSolve_SeqDense_magma() and MatMatMult_magma() (for magma_xgemm dense > > >>> matrix multiplication). > > >>> > > >>> Harshad > > >>> > > >>> > > >>> ----- Original Message ----- > > >>> From: "Barry Smith" <[email protected]> > > >>> To: "Harshad Sahasrabudhe" <[email protected]> > > >>> Cc: "For users of the development version of PETSc" > > >>> <[email protected]> > > >>> Sent: Tuesday, August 20, 2013 7:10:40 PM > > >>> Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat > > >>> > > >>> > > >>> How general do you plan for this "magma" matrix class to be? If it all > > >>> it is for is to do LU/Cholesky factorizations then you do NOT need to > > >>> introduce src/mat/impls/dense/seq/magma at all. Simply use the > > >>> MatGetFactor() mechanism to "How will the user be able to access this > > >>> function?". This is definitely the easy way to go. > > >>> > > >>> If you want to use the "magma" matrix class for all kinds of > > >>> non-factorization matrix operations then you need to write a full magma > > >>> matrix class. I don't recommend this. > > >>> > > >>> Barry > > >>> > > >>> On Aug 20, 2013, at 1:35 PM, Harshad Sahasrabudhe <[email protected]> > > >>> wrote: > > >>> > > >>>> Hi, > > >>>> > > >>>> I am working on adding the Seq Dense MAGMA matrix type. For this, I > > >>>> have created the following files and directories: > > >>>> > > >>>> src/mat/impls/dense/seq/magma > > >>>> src/mat/impls/dense/seq/magma/magma.c > > >>>> src/mat/impls/dense/seq/magma/makefile > > >>>> > > >>>> Right now I am just trying the make LU factorization work through > > >>>> MAGMA. Barry had suggested looking at dense.c, as the MAGMA function > > >>>> for LU has the same sequence as the LAPACK function getrf. MAGMA does > > >>>> the memory allocation on the GPU inside it's function magma_?getrf. So > > >>>> can I directly use the matrix type declared in dense.c, and just > > >>>> define a function MatLUFactor_SeqDense_MAGMA magma.c which uses the > > >>>> same matrix type? > > >>>> > > >>>> How will the user be able to access this function? > > >>>> > > >>>> Thanks, > > >>>> Harshad > > >>> > > >> > > > > > > > >
