I even tried this with a fresh git pull from Hg repository. I added magma.py
for configuring with magma. Then I added only the following in dense.c
#if defined(PETSC_HAVE_LIBMAGMA)
#if !defined(HAVE_CUBLAS)
#define HAVE_CUBLAS
#endif
#include "magma.h"
#endif
Then I followed the procedure I mentioned below, and I can reproduce the
problem. Now I'm certain that removing #requirespackage is causing it.
1) remove #requirespackage from src/mat/impl/dense/seq/makefile
2) remove build directory
3) remove externalpackages directory
4) configure && make
I get these errors.
5) add #requirespackage to src/mat/impl/dense/seq/makefile
6) remove build directory
7) remove externalpackages directory
8) configure && make
No errors.
Harshad
>>>>>>>>
/usr/bin/gcc -Dpetsc_EXPORTS -D__INSDIR__="" -malign-double -fPIC -g3
-fno-inline -O0 -fPIC -I/home/harshad/workspace/petsc-hg/include
-I/home/harshad/workspace/petsc-hg/arch-linux2-c-debug/include
-I/home/harshad/Downloads/magma-1.4.0/include -I/home/harshad/cuda-5.5/include
-I/home/harshad/workspace/petsc-hg/include/mpiuni -o
CMakeFiles/petsc.dir/src/mat/impls/dense/seq/dense.c.o -c
/home/harshad/workspace/petsc-hg/src/mat/impls/dense/seq/dense.c
In file included from
/home/harshad/Downloads/magma-1.4.0/include/magmablas_z.h:14:0,
from
/home/harshad/Downloads/magma-1.4.0/include/magmablas.h:12,
from /home/harshad/Downloads/magma-1.4.0/include/magma.h:15,
from
/home/harshad/workspace/petsc-hg/src/mat/impls/dense/seq/dense.c:10:
/home/harshad/Downloads/magma-1.4.0/include/magma_types.h:146:6: error: #error
"One of HAVE_CUBLAS, HAVE_clAmdBlas, or HAVE_MIC must be defined. This
typically happens in Makefile.internal."
<<<<<<<
On Wed, 28 Aug 2013, Harshad Sahasrabudhe wrote:
> Attached make.log
>
> ----- Original Message -----
> From: "Harshad Sahasrabudhe" <[email protected]>
> To: "petsc-dev" <[email protected]>
> Cc: "Barry Smith" <[email protected]>
> Sent: Wednesday, August 28, 2013 4:33:23 PM
> Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat
>
> I'm getting errors when compiling with MAGMA and without the #requirespackage
> tag in the makefile. It builds correctly if I have #requirespackage in the
> makefile. There are lots of these errors with different type names:
>
> magma-1.4.0/include/magma_s.h:607:38: error: unknown type name ‘magma_queue_t’
>
> Harshad
>
> ----- Original Message -----
> From: "Satish Balay" <[email protected]>
> To: "Harshad Sahasrabudhe" <[email protected]>
> Cc: "Barry Smith" <[email protected]>, "For users of the development version
> of PETSc" <[email protected]>
> Sent: Wednesday, August 28, 2013 4:26:20 PM
> Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat
>
> If you have conditional code in dense.c - then you don't need
> "#requirespackage 'PETSC_HAVE_LIBMAGMA'" in the makefile
>
> Are you getting errors when compiling with Magma or without magma?
> What errors?
>
> Satish
>
>
> On Wed, 28 Aug 2013, Harshad Sahasrabudhe wrote:
>
> > Hi Barry,
> >
> > I'm trying to compile dense.c with the MAGMA code in it. Now the MAGMA code
> > is protected by #if defined statements so it is only compiled when we have
> > MAGMA.
> >
> > When we have MAGMA, I get errors while compiling, if we don't have
> > #requirespackage 'PETSC_HAVE_LIBMAGMA' in the Makefile. We obviously don't
> > require MAGMA to compile dense.c. Is there any other way to tell make that
> > MAGMA lib has to be used for compiling dense.c when it is present?
> >
> > I have attached the makefile inside src/mat/impls/dense/seq folder.
> >
> > Thanks,
> > Harshad
> >
> > ----- Original Message -----
> > From: "Barry Smith" <[email protected]>
> > To: "Harshad Sahasrabudhe" <[email protected]>
> > Cc: "For users of the development version of PETSc" <[email protected]>
> > Sent: Tuesday, August 27, 2013 11:20:44 PM
> > Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat
> >
> >
> > Does the LU factorization use magma on the gpu now work? You told us that
> > for factorization magma allocated the memory on the GPU and copied the data
> > over everything over automatically. If the factorization and solve stuff
> > all works then we can talk about the MatMult() and other stuff but
> > otherwise let's get the factorization all working before moving on to the
> > "harder" stuff.
> >
> > barry
> >
> >
> >
> > On Aug 27, 2013, at 10:12 PM, Harshad Sahasrabudhe <[email protected]>
> > wrote:
> >
> > > Hi Barry,
> > >
> > > The functions MatCreate_SeqDense and MatSeqDenseSetPreallocation_SeqDense
> > > do not handle memory allocation on GPU. I can see 2 ways to implement
> > > this:
> > >
> > > 1) Modify these functions so that they allocate memory on host or device
> > > when the solver is magma. I don't know how to do this.
> > >
> > > 2) Copy the relevant functions from dense.c to magma.c. Make a magma
> > > matrix class which is very similar to the dense matrix class PETSc uses,
> > > with some additional flags. Modify the MatMatMult, MatGetFactor, MatSolve
> > > functions so that they use magma functions in the proper way. I can do
> > > this.
> > >
> > > Can you please advise on how I should progress?
> > >
> > > Thanks,
> > > Harshad
> > >
> > > ----- Original Message -----
> > > From: "Barry Smith" <[email protected]>
> > > To: "Harshad Sahasrabudhe" <[email protected]>
> > > Cc: "For users of the development version of PETSc"
> > > <[email protected]>
> > > Sent: Tuesday, August 27, 2013 8:43:57 AM
> > > Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat
> > >
> > >
> > > Both _p_Mat and _p_Vec have void pointers that can be used for the gpu
> > > memory as well as flags that could potentially be used. See below
> > >
> > > struct _p_Mat {
> > > PETSCHEADER(struct _MatOps);
> > > ….
> > > PetscBool nooffprocentries,nooffproczerorows;
> > > #if defined(PETSC_HAVE_CUSP)
> > > PetscCUSPFlag valid_GPU_matrix; /* flag pointing to the matrix
> > > on the gpu*/
> > > #endif
> > > #if defined(PETSC_HAVE_VIENNACL)
> > > PetscViennaCLFlag valid_GPU_matrix; /* flag pointing to the
> > > matrix on the gpu*/
> > > #endif
> > > void *spptr; /* pointer for special library
> > > like SuperLU */
> > >
> > > On Aug 27, 2013, at 12:20 AM, Harshad Sahasrabudhe <[email protected]>
> > > wrote:
> > >
> > >> Hi Barry,
> > >>
> > >> Thanks, MatGetFactor() now much more clear.
> > >>
> > >> However, I have run into trouble with the device memory allocation for
> > >> Mat_SeqDense. How do I figure out whether the allocated memory is on
> > >> host or device in MatSeqDenseSetPreallocation_SeqDense() ?
> > >>
> > >> For some functions such as magma_xgetrf_gpu and magma_xgetrs_gpu, I need
> > >> the memory allocated on the device. As the preallocation is done in
> > >> MatSeqDenseSetPreallocation_SeqDense for dense Matrices, I want to write
> > >> a similar function. But checking for if(!data) is not enough as I need
> > >> to make sure that the allocated memory is on the device.
> > >>
> > >> Is it a good idea to make another Mat class for magma which has 2
> > >> pointers, one for the host memory and one for device memory (only one of
> > >> which should point to data at any given time) ?
> > >>
> > >> Harshad
> > >>
> > >> ----- Original Message -----
> > >> From: "Barry Smith" <[email protected]>
> > >> To: "Harshad Sahasrabudhe" <[email protected]>
> > >> Cc: "For users of the development version of PETSc"
> > >> <[email protected]>
> > >> Sent: Monday, August 26, 2013 6:19:21 PM
> > >> Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat
> > >>
> > >>
> > >> On Aug 26, 2013, at 4:24 PM, Harshad Sahasrabudhe <[email protected]>
> > >> wrote:
> > >>
> > >>> Hi Barry,
> > >>>
> > >>> Here is my analysis, please correct if I'm wrong:
> > >>>
> > >>> For using the MatGetFactor() mechanism, I need to define MATSOLVERMAGMA
> > >>> in petscmat.h. I also need to define the functions
> > >>> MatFactorGetSolverPackage_SeqDense_magma(),
> > >>> MatGetFactor_SeqDense_magma() in dense.c
> > >>>
> > >>> I'm not creating a new matrix class, so I think I don't need to add
> > >>> anything to MatRegisterAll()
> > >>>
> > >>> I don't understand the following:
> > >>> How does PETSc figure out that there is an additional solver present?
> > >>> Do I need to define any other variables or add anything for this?
> > >>
> > >> In MatCreate_SeqDense() note the line
> > >> ierr =
> > >> PetscObjectComposeFunction((PetscObject)B,"MatGetFactor_petsc_C",MatGetFactor_seqdense_petsc);CHKERRQ(ierr);
> > >>
> > >> you will add an additional line (protected by ifdef for magma being
> > >> available that registers your routine
> > >>
> > >> ierr =
> > >> PetscObjectComposeFunction((PetscObject)B,"MatGetFactor_magma_C",MatGetFactor_seqdense_magma);CHKERRQ(ierr);
> > >>
> > >> Barrt
> > >>
> > >>>
> > >>
> > >>> What else do I need to change if I have to add the functions
> > >>> MatSolve_SeqDense_magma() and MatMatMult_magma() (for magma_xgemm dense
> > >>> matrix multiplication).
> > >>>
> > >>> Harshad
> > >>>
> > >>>
> > >>> ----- Original Message -----
> > >>> From: "Barry Smith" <[email protected]>
> > >>> To: "Harshad Sahasrabudhe" <[email protected]>
> > >>> Cc: "For users of the development version of PETSc"
> > >>> <[email protected]>
> > >>> Sent: Tuesday, August 20, 2013 7:10:40 PM
> > >>> Subject: Re: [petsc-dev] Adding Seq Dense MAGMA Mat
> > >>>
> > >>>
> > >>> How general do you plan for this "magma" matrix class to be? If it all
> > >>> it is for is to do LU/Cholesky factorizations then you do NOT need to
> > >>> introduce src/mat/impls/dense/seq/magma at all. Simply use the
> > >>> MatGetFactor() mechanism to "How will the user be able to access this
> > >>> function?". This is definitely the easy way to go.
> > >>>
> > >>> If you want to use the "magma" matrix class for all kinds of
> > >>> non-factorization matrix operations then you need to write a full magma
> > >>> matrix class. I don't recommend this.
> > >>>
> > >>> Barry
> > >>>
> > >>> On Aug 20, 2013, at 1:35 PM, Harshad Sahasrabudhe <[email protected]>
> > >>> wrote:
> > >>>
> > >>>> Hi,
> > >>>>
> > >>>> I am working on adding the Seq Dense MAGMA matrix type. For this, I
> > >>>> have created the following files and directories:
> > >>>>
> > >>>> src/mat/impls/dense/seq/magma
> > >>>> src/mat/impls/dense/seq/magma/magma.c
> > >>>> src/mat/impls/dense/seq/magma/makefile
> > >>>>
> > >>>> Right now I am just trying the make LU factorization work through
> > >>>> MAGMA. Barry had suggested looking at dense.c, as the MAGMA function
> > >>>> for LU has the same sequence as the LAPACK function getrf. MAGMA does
> > >>>> the memory allocation on the GPU inside it's function magma_?getrf. So
> > >>>> can I directly use the matrix type declared in dense.c, and just
> > >>>> define a function MatLUFactor_SeqDense_MAGMA magma.c which uses the
> > >>>> same matrix type?
> > >>>>
> > >>>> How will the user be able to access this function?
> > >>>>
> > >>>> Thanks,
> > >>>> Harshad
> > >>>
> > >>
> > >
> >
> >
>
